Tutorial #4: Writing and Submitting a Spark Application

⇖ Installing a Programming Language

Spark imposes no special restrictions on where you can do your development. The Sparkour recipes will continue to use the EC2 instance created in a previous tutorial as a development environment, so that each recipe can start from the same baseline configuration. However, you probably already have a development environment tuned just the way you like it, so you can use it instead if you prefer. You'll just need to get your build dependencies in order.

1. Regardless of which language you use, you'll need Apache Spark and a Java Runtime Environment (8 or higher) installed. These components allow you to submit your application to a Spark cluster (or run it in Local mode).
2. You also need the development kit for your language. If developing for Spark 2.x, you would want a minimum of Java Development Kit (JDK) 8, Python 3.0, R 3.1, or Scala 2.11, respectively. You probably already have the development kit for your language installed in your development environment.
3. Finally, you need to link or include the core Spark libraries with your application. If you are using an Integrated Development Environment (IDE) like Eclipse or IntelliJ, the official Spark documentation provides instructions for adding Spark dependencies in Maven. If you don't use Maven, you can manually track down the dependencies in your installed Spark directory:
   • Java and Scala dependencies can be found in jars (or lib for Spark 1.6).
   • Python dependencies can be found in python/pyspark.
   • R dependencies can be found in R/lib.

Instructions for installing each programming language on your EC2 instance are shown below. If your personal development environment (outside of EC2) does not already have the right components installed, you should be able to review these instructions and adapt them to your unique environment.

These instructions do not cover Java and Scala build tools (such as Maven and SBT) which simplify the compiling and bundling steps of the development lifecycle. Build tools are covered in Building Spark Applications with Maven and Building Spark Applications with SBT.

⇖ Writing a Spark Application

1. Download and unzip the example source code for this tutorial. This ZIP archive contains source code in all supported languages. Here's how you would do this on our EC2 instance:
2. The example source code for each language is in a subdirectory of src/main with that language's name. Locate the source code for your programming language and open the file in a text editor.
• Java
• Python
• R
• Scala
• 💾Default language saved.
3. The application is decidedly uninteresting, merely initializing a SparkSession, printing a message, and stopping the SparkContext. We are more concerned with getting the application executed right now, and will create more compelling programs in the next tutorial.

⇖ Bundling Dependencies

When you submit an application to a Spark cluster, the cluster manager distributes the application code to each worker so it can be executed locally. This means that all dependencies need to be included (except for Spark and Hadoop dependencies, which the workers already have copies of). There are multiple approaches for bundling dependencies, depending on your programming language:

In the case of the --py-files or --jars parameters, you can also use URL schemes such as ftp: or hdfs: to reference files stored elsewhere, or local: to specify files that are already stored on the workers. See the documentation on Advanced Dependency Management for more details.

The simple application in this tutorial has no dependencies besides Spark itself. We cover the creation of an assembly JAR in Building Spark Applications with Maven and Building Spark Applications with SBT.

⇖ Submitting the Application

Here are examples of how you would submit an application in each of the supported languages. For JAR-based languages (Java and Scala), we would pass in an application JAR file and a class name. For interpreted languages, we just need the top-level module or script name. Everything after the expected spark-submit arguments are passed into the application itself.

1. It's time to submit our simple application. It has no dependencies or application arguments, and we'll run it in Local mode, which means that a brand new Spark engine spins up to execute the code, rather than relying upon an existing Spark cluster.
• Java
• Python
• R
• Scala
• 💾Default language saved.
2. The complete set of low-level commands required to build and submit are explicitly shown here for your awareness. In tutorials and recipes that follow, these commands are simplified through a helpful build script.
3. Submitting your application should result in a brief statement about Spark.

Deploy Modes

The spark-submit script accepts a --deploy-mode parameter which dictates how the driver is set up. Recall from the previous recipe that the driver contains your application and relies upon the completed tasks of workers to successfully execute your code. It's best if the driver is physically co-located near the workers to reduce any network latency.

• If the location where you are running spark-submit is sufficiently close to the cluster already, the client deploy mode simply places the driver on the same instance that the script was run. This is the default approach.
• If the spark-submit location is very far from the cluster (e.g., your cluster is in another AWS region), you can reduce network latency by placing the driver on a node within the cluster with the cluster deploy mode.

You can visualize these modes in the image below.

Configuring spark-submit

If you find yourself often reusing the same configuration parameters, you can create a conf/spark-defaults.conf file in the Spark home directory of your development environment. Properties explicitly set within a Spark application (on the SparkConf object) have the highest priority, followed by properties passed into the spark-submit script, and finally the defaults file.

Here is an example of setting the master URL in a defaults file. More detail on the available properties can be found in the official documentation.

⇖ Spark Distribution Examples

Now that we have successfully submitted and executed an application, let's take a look at one of the examples included in the Spark distribution.

1. The source code for the JavaWordCount application can be found in the org.apache.spark.examples package. You can view the source code, but be aware that this is just a reference copy. Any changes you make to it will not affect the already compiled application JAR.
2. To submit this application in Local mode, you use the spark-submit script, just as we did with the Python application.
3. Spark also includes a quality-of-life script that makes running Java and Scala examples simpler. Under the hood, this script ultimately calls spark-submit.
4. Source code for the Python examples can be found in $SPARK_HOME/examples/src/main/python/. These examples have no dependent packages, other than the basic pyspark library.
5. Source code for the R examples can be found in $SPARK_HOME/examples/src/main/r/. These examples have no dependent packages, other than the basic SparkR library.
6. Source code for the Scala examples can be found in $SPARK_HOME/examples/src/main/scala/. These examples follow the same patterns and directory organization as the Java examples.

⇖ Conclusion

You have now been exposed to the application lifecycle for applications that use Spark for data processing. It's time to stop poking around the edges of Spark and actually use the Spark APIs to do something useful to data. In the next tutorial, Tutorial #5: Working with Spark Resilient Distributed Datasets (RDDs), we dive into the Spark Core API and work with some common transformations and actions. If you are done playing with Spark for now, make sure that you stop your EC2 instance so you don't incur unexpected charges.

Spot any inconsistencies or errors? See things that could be explained better or code that could be written more idiomatically? If so, please help me improve Sparkour by opening a ticket on the Issues page. You can also discuss this recipe with others in the Sparkour community on the Discussion page.