Final Project: NYC Parking Tickets

Final Project Big Data Technologies 201A

 This is where the big data file is and the sample data


This is a cool map I couldn't get to work. Maybe next course I will attempt it


This is a way to split csv files with a windows machine

Moving local csv file from Local Drive to VM (Docker Container)

Command Prompt
Move to Linux server
pscp -i c:\BigDataTechnolgies\ServerKey\tjpauley_azure.ppk C:\data\DateDim.csv tjpauley@IPAddress:

 Note: I used the FAQ at putty's website

 Linux Command
Copy to sandbox data folder  

 sudo cp DateDim.csv /data

Zeppelin Notebook
Create data frame

case class DateDim ( DateNo: String, Weekday: String, Year: Integer, Month: Integer, Day: Integer)
val ParkingFourteens = spark.read.option("inferSchema", "true").option("header","true").csv("file:///data/DateDim.csv")
DateDims.toDF().registerTempTable("DateDim")
DateDims.show(1)

 Zeppelin Notebook Commands

 %md = markdown
%sql = SQL
http://IPAddress:9995 Note: Open port 9995

Jupyter Notebook Commands

 http://IPAddress:9999 Note: Open port 9999

Comments

Post a Comment

Popular posts from this blog

Assignment 03 Fun with Spark Part II

Image
Assignment 03 Description Fun with Spark Lab Prepare Sample Data For this assignment we are going to use two sample data sets.  One is a text file version of the classic book “War and Peace”, the other is the same housing sales data from Week 2. If you have not already done so, please put home_data.csv onto your VM and then also war_and_peace.txt.  To get the data onto your VM you can either download to your local machine and then use the SCP command to copy the files e.g: “scp home_data.csv username@ip:   (the colon is necessary) Or use the wget command and pass in the link directly, similar to how you downloaded the HDP sandbox.   Note that the “stylized” single quote marks in this doc might cause problems if you copy/paste directly into command line. wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2&id=0B0Ntj7VtxrluZG9xRkc0NmZ4Q0E&export=download ’ -O war_and_peace.txt wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2&id=0B0Ntj7VtxrluN1dFWlRiY
Read more

Assignment 01 Partition Linux OS and Config

Assignment 01:  Partition Linux OS and Config Azure VM Choose Ubuntu version 16.04 LTS Step 1: Name [NETID]01 i.e. jkolter01 Authentication SSH key pair; paste public key in this window https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html  (Links to an external site.) Links to an external site.  for Windows PuTTY Resource Group: Use existing Big_Data_Technologies Location West US 2 Step 2: VM Size B4MS Step 3: Defaults for most settings Enable auto-shutdown with a time that makes sense for you Step 4: Create and wait Data Disk Add data disk Name [NETID]01_data1 Resource Group Use existing: Big_Data_Technologies Account type SSD Source Type None Size 64GB Make sure to "SAVE" after creating!!! Parition/Format/Mount Data Disk Create partition table sudo fdisk /dev/sdc 'g' to create new label 'n' to create new partition table
Read more