Running/Debugging Docker based Apache Spark tests on your local environment

Apache Spark has docker based integration tests for the different JDBC Dialects currently supported, and having the ability to run and debug these tests directly from your IDE, in my case IntelliJ, can save a lot of time when investigating issues related to these data sources.

For this post, I will assume that :

• Docker is already installed, by following these installation steps.
• Apache Spark source is forked and loaded to your IntelliJ IDE as described in the Apache Spark wiki.

First, we will need to retrieve the environment settings for your docker installation by loading your Docker quickstart terminal, and issue the command below:

$ docker-machine env default

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/lresende/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"

Now let's configure IntelliJ to have the proper docker environment information so that I knows how to connect with the docker infrastructure for running the tests. Note the tests are available under the project "external/docker-integration-tests"

• Create/Edit the test configuration and add one each environment variable retrieved with the docker command above.

Now you should be ready to run the tests. If this is the first time, you might experience some delay or timeouts as the docker images required by the tests are downloaded.

To force the download of the images, utilize the commands below:

$ docker pull lresende/db2express-c:10.5.0.5-3.10.0
$ docker pull mysql:5.7.9
$ docker pull wnameless/oracle-xe-11g:latest
$ docker pull postgres:9.4.5

Hope this is useful and get you eager to start contributing to Apache Spark.