Final Project: NYC Parking Tickets

Final Project Big Data Technologies 201A

 This is where the big data file is and the sample data


This is a cool map I couldn't get to work. Maybe next course I will attempt it


This is a way to split csv files with a windows machine

Moving local csv file from Local Drive to VM (Docker Container)

Command Prompt
Move to Linux server
pscp -i c:\BigDataTechnolgies\ServerKey\tjpauley_azure.ppk C:\data\DateDim.csv tjpauley@IPAddress:

 Note: I used the FAQ at putty's website

 Linux Command
Copy to sandbox data folder  

 sudo cp DateDim.csv /data

Zeppelin Notebook
Create data frame

case class DateDim ( DateNo: String, Weekday: String, Year: Integer, Month: Integer, Day: Integer)
val ParkingFourteens = spark.read.option("inferSchema", "true").option("header","true").csv("file:///data/DateDim.csv")
DateDims.toDF().registerTempTable("DateDim")
DateDims.show(1)

 Zeppelin Notebook Commands

 %md = markdown
%sql = SQL
http://IPAddress:9995 Note: Open port 9995

Jupyter Notebook Commands

 http://IPAddress:9999 Note: Open port 9999

Comments

Post a Comment

Popular posts from this blog

Assignment 03 Fun with Spark Part II

Image
Assignment 03 Description Fun with Spark Lab Prepare Sample Data For this assignment we are going to use two sample data sets.  One is a text file version of the classic book “War and Peace”, the other is the same housing sales data from Week 2. If you have not already done so, please put home_data.csv onto your VM and then also war_and_peace.txt.  To get the data onto your VM you can either download to your local machine and then use the SCP command to copy the files e.g: “scp home_data.csv username@ip:   (the colon is necessary) Or use the wget command and pass in the link directly, similar to how you downloaded the HDP sandbox.   Note that the “stylized” single quote marks in this doc might cause problems if you copy/paste directly into command line. wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2&id=0B0Ntj7VtxrluZG9xRkc0NmZ4Q0E&export=download ’ -O war_and_peace.txt wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2...
Read more

Assignment 05 Fun with Spark III

Image
Week 5 Lab This week’s work outside of class is optional, but highly recommended.  We will walk through installing Jupyter as well as make some sandbox modifications to support running Jupyter as well as Zeppelin and the Spark Web UI and I’ve also provided some additional fun Spark exercises for those of you who are especially interested in the software part of our class. Docker Port Forwarding Modifications In this step we are going to go through process we talked about in class this week about adding additional port forwards to your sandbox-hdp Docker container.  Remember that we can only set up these proxies when the container is created, so we need to commit an image of our current setup and recreate a new container with the new mappings. SSH into your VM: Stop sandbox-hdp Docker container sudo docker stop sandbox-hdp Confirm  not running using Docker ‘ps” sudo docker ps Crate image of the current container state sudo docker com...
Read more