Recipes

These sequential tutorial recipes are intended for developers just getting started with Spark. Developers should have some familiarity with the Linux command line and common tasks like running shell scripts.

• Tutorial #0: Sparkour Conventions
• Tutorial #1: Apache Spark in a Nutshell
• Tutorial #2: Installing Spark on Amazon EC2
• Tutorial #3: Managing Spark Clusters
• Tutorial #4: Writing and Submitting a Spark Application
• Tutorial #5: Working with Spark Resilient Distributed Datasets (RDDs)

These recipes are targeted, adaptable examples of useful Spark code addressing common developer needs. They assume some familiarity with the concepts explained in the tutorials.

• Libraries: Spark GraphX
• Libraries: Spark MLlib Library
• Libraries: Spark Streaming
• Libraries: Spark SQL
  • Working with Spark DataFrames
  • Using SQL and User-Defined Functions with Spark DataFrames
  • Using JDBC with Spark DataFrames
  • Controlling the Schema of a Spark DataFrame
  • Understanding the SparkSession in Spark 2.0
  • Using Hive Tables with Spark DataFrames
• Practices
  • Building Spark Applications with Maven
  • Building Spark Applications with SBT
  • Aggregating Results with Spark Accumulators
  • Improving Spark Performance with Broadcast Variables
  • Simplifying Java Applications with Lambda Expressions
  • Writing Unit Tests

Unlinked titles depict the roadmap of recipes I would like to tackle next -- the titles will become links as they are published. Suggestions for new recipes can be submitted through the Issues page.

These recipes are targeted, adaptable examples of useful Spark code addressing common developer needs. They assume some familiarity with the concepts explained in the tutorials.

• Clusters
  • Managing a Spark Cluster with the spark-ec2 Script
  • Configuring Job Scheduling
  • Using Apache Hadoop YARN as a Cluster Manager
  • Using Apache Mesos as a Cluster Manager
• Data Sources: S3
  • Configuring Amazon S3 as a Spark Data Source
  • Configuring Spark to Use Amazon S3
  • Configuring an S3 VPC Endpoint for a Spark Cluster
• Data Sources: Other
  • Using HDFS as a Data Store
• Logging and Monitoring
  • Enabling History in Your Master UI
  • Monitoring Your Cluster with the REST API
  • Capturing Metrics with slf4j
• Notebooks
  • Installing and Configuring Apache Zeppelin
• Security
  • Adding Authentication to Spark UIs
  • Running Spark over SSL
• Tuning
  • Configuring Shuffle Behavior
  • Using Amazon CloudWatch to Profile Your Workload
  • Using Kryo for Memory-Efficient Data Serialization
  • Using Spark Metrics to Profile Your Workload

Unlinked titles depict the roadmap of recipes I would like to tackle next -- the titles will become links as they are published. Suggestions for new recipes can be submitted through the Issues page.