Assignment 01 Partition Linux OS and Config

Assignment 01:  Partition Linux OS and Config

Azure VM

Choose Ubuntu version 16.04 LTS

Step 1:


Step 2:

  • VM Size B4MS

Step 3:

  • Defaults for most settings
  • Enable auto-shutdown with a time that makes sense for you

Step 4:

  • Create and wait



Data Disk

Add data disk

  • Name
    • [NETID]01_data1
  • Resource Group
    • Use existing: Big_Data_Technologies
  • Account type
    • SSD
  • Source Type
    • None
  • Size
    • 64GB

Make sure to "SAVE" after creating!!!

Parition/Format/Mount Data Disk

Create partition table

  • sudo fdisk /dev/sdc
  • 'g' to create new label
  • 'n' to create new partition table
    • use enter for defaults
    • partition number 1
    • first and last sector
  • 'w' to write partition table to disk

Format Disk

  • sudo mkfs -t xfs /dev/sdc1

Mount Disk

  • sudo mkdir /data
  • sudo mount /dev/sdc1 /data

Add to fstab to persist reboots

  • sudo blkid /dev/sdc1
  • edit /etc/fstab
  • add a line that looks somethign like the following, with the UUID from your blkid output substituted in
    • UUID=b8feed20-d520-4130-8cf3-d0633f63a2d8   /data xfs defaults,nofail 1 2

Exit SSH session and shut down VM when done!

Comments

Post a Comment

Popular posts from this blog

Final Project: NYC Parking Tickets

Final Project Big Data Technologies 201A This is where the big data file is and the sample data https://www.kaggle.com/new-york-city/nyc-parking-tickets/data This is a cool map I couldn't get to work. Maybe next course I will attempt it http://www.bigendiandata.com/2017-06-27-Mapping_in_Jupyter/ This is a way to split csv files with a windows machine https://www.addictivetips.com/windows-tips/csv-splitter-for-windows/ Moving local csv file from Local Drive to VM (Docker Container) Command Prompt Move to Linux server pscp -i c:\BigDataTechnolgies\ServerKey\tjpauley_azure.ppk C:\data\DateDim.csv tjpauley@ IPAddress : Note: I used the FAQ at putty's website Linux Command Copy to sandbox data folder   sudo cp DateDim .csv /data Zeppelin Notebook Create data frame case class DateDim ( DateNo: String, Weekday: String, Year: Integer, Month: Integer, Day: Integer) val ParkingFourteens = spark.read.option("inferSchema", "true...
Read more

Assignment 03 Fun with Spark Part II

Image
Assignment 03 Description Fun with Spark Lab Prepare Sample Data For this assignment we are going to use two sample data sets.  One is a text file version of the classic book “War and Peace”, the other is the same housing sales data from Week 2. If you have not already done so, please put home_data.csv onto your VM and then also war_and_peace.txt.  To get the data onto your VM you can either download to your local machine and then use the SCP command to copy the files e.g: “scp home_data.csv username@ip:   (the colon is necessary) Or use the wget command and pass in the link directly, similar to how you downloaded the HDP sandbox.   Note that the “stylized” single quote marks in this doc might cause problems if you copy/paste directly into command line. wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2&id=0B0Ntj7VtxrluZG9xRkc0NmZ4Q0E&export=download ’ -O war_and_peace.txt wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2...
Read more