Assignment 04 Fun with Spark Part III

Assignment 04 Description


Assignment 4 - More Fun with Spark

No environment setup this week!  Use the Spark shell (or notebook if you have set it up) to do this weeks assignment in either Scala or Python.  Unless specified otherwise, please use the DataFrame API for this week’s work; do not just use SQL string queries.

If you are unable to come up with code to answer a question please describe how you think you would solve the problem in a “sparkified” way based on what we have learned so far.

  1. From home_data.csv, how many houses sold were built prior to 1979? (I promise that is the last time I’m going to ask this question)
  2. From home_data.csv, what is the most expensive zipcode in the data set, defined as highest average sales price?
  3. How many unique zipcodes have sales data in the home_data.csv data set?
  4. Demonstrate how to drop the “sqft_living15” and “sqft_lot15” columns from your dataset.
  5. Access the zipcode table stored in Hive (ok to use a SQL string query here) and demonstrate joining that data with a dataframe created from home_data.csv.

Bonus Exercise

Remember, bonus exercise is meant for those either with prior knowledge on the topic or the interest (and time) to do some research beyond what we covered in class and is optional.

Exercise 1:

You can register a user-defined function (UDF) in your code that can then be used like any built-in Spark functions.  

In the airport_codes.csv file used in class this week, the data contains a field “iso_region” that has a syntax like this for rows in the US: “US-StateCode”.  So for example Seattle is “US-WA”.
Write a UDF to return the state from this column.

Exercise 2:

It is possible to programmatically apply a schema to a DataFrame, either if you don’t have a schema in advance, or maybe you want to provide a different projection of the data to create multiple views of a base data set, or most commonly when going from a RDD to a DataFrame.  Create a simple RDD, and then programmatically apply a schema to it to create a DataFrame.

Assignment 04 Answers


Comments

Post a Comment

Popular posts from this blog

Final Project: NYC Parking Tickets

Final Project Big Data Technologies 201A This is where the big data file is and the sample data https://www.kaggle.com/new-york-city/nyc-parking-tickets/data This is a cool map I couldn't get to work. Maybe next course I will attempt it http://www.bigendiandata.com/2017-06-27-Mapping_in_Jupyter/ This is a way to split csv files with a windows machine https://www.addictivetips.com/windows-tips/csv-splitter-for-windows/ Moving local csv file from Local Drive to VM (Docker Container) Command Prompt Move to Linux server pscp -i c:\BigDataTechnolgies\ServerKey\tjpauley_azure.ppk C:\data\DateDim.csv tjpauley@ IPAddress : Note: I used the FAQ at putty's website Linux Command Copy to sandbox data folder   sudo cp DateDim .csv /data Zeppelin Notebook Create data frame case class DateDim ( DateNo: String, Weekday: String, Year: Integer, Month: Integer, Day: Integer) val ParkingFourteens = spark.read.option("inferSchema", "true&
Read more

Assignment 03 Fun with Spark Part II

Image
Assignment 03 Description Fun with Spark Lab Prepare Sample Data For this assignment we are going to use two sample data sets.  One is a text file version of the classic book “War and Peace”, the other is the same housing sales data from Week 2. If you have not already done so, please put home_data.csv onto your VM and then also war_and_peace.txt.  To get the data onto your VM you can either download to your local machine and then use the SCP command to copy the files e.g: “scp home_data.csv username@ip:   (the colon is necessary) Or use the wget command and pass in the link directly, similar to how you downloaded the HDP sandbox.   Note that the “stylized” single quote marks in this doc might cause problems if you copy/paste directly into command line. wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2&id=0B0Ntj7VtxrluZG9xRkc0NmZ4Q0E&export=download ’ -O war_and_peace.txt wget ‘ https://drive.google.com/a/uw.edu/uc?authuser=2&id=0B0Ntj7VtxrluN1dFWlRiY
Read more

Assignment 01 Partition Linux OS and Config

Assignment 01:  Partition Linux OS and Config Azure VM Choose Ubuntu version 16.04 LTS Step 1: Name [NETID]01 i.e. jkolter01 Authentication SSH key pair; paste public key in this window https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html  (Links to an external site.) Links to an external site.  for Windows PuTTY Resource Group: Use existing Big_Data_Technologies Location West US 2 Step 2: VM Size B4MS Step 3: Defaults for most settings Enable auto-shutdown with a time that makes sense for you Step 4: Create and wait Data Disk Add data disk Name [NETID]01_data1 Resource Group Use existing: Big_Data_Technologies Account type SSD Source Type None Size 64GB Make sure to "SAVE" after creating!!! Parition/Format/Mount Data Disk Create partition table sudo fdisk /dev/sdc 'g' to create new label 'n' to create new partition table
Read more