Enabling HTTPS with Apache Livy - Amazon EMR

Enabling HTTPS with Apache Livy

  1. Provision an Amazon EMR cluster with transit encryption enabled. To learn more about encryption, see Encrypt data at rest and in transit.

  2. Create a file called livy_ssh.sh with the following contents.

    #!/bin/bash KEYSTORE_FILE=`awk '/ssl.server.keystore.location/{getline; print}' /etc/hadoop/conf/ssl-server.xml | sed -e 's/<[^>]*>//g' | tr -d ' \t\n\r\f'` KEYSTORE_PASS=`awk '/ssl.server.keystore.password/{getline; print}' /etc/hadoop/conf/ssl-server.xml | sed -e 's/<[^>]*>//g' | tr -d ' \t\n\r\f'` KEY_PASS=`awk '/ssl.server.keystore.keypassword/{getline; print}' /etc/hadoop/conf/ssl-server.xml | sed -e 's/<[^>]*>//g' | tr -d ' \t\n\r\f'` echo "livy.keystore $KEYSTORE_FILE livy.keystore.password $KEYSTORE_PASS livy.key-password $KEY_PASS" | sudo tee -a /etc/livy/conf/livy.conf >/dev/null sudo systemctl restart livy-server.service
  3. Run the following script as an Amazon EMR step. This script modifies /etc/livy/conf/livy.conf to activate SSL.

    --steps '[{"Args":["s3://DOC-EXAMPLE-BUCKET/livy_ssl.sh"],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar","Properties":"","Name":"Custom JAR"}]'
  4. Restart the Apache Livy service so that the changes take effect. To restart Apache Livy, see Stopping and restarting processes.

  5. Test that the clients can now communicate using HTTPS. To submit a job, for example, run the following code.

    curl -k -X POST --data '{"file": "local:///usr/lib/spark/examples/jars/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi"}' \ -H "Content-Type: application/json" \ https://EMR_Master_Node_Host:8998/batches

    If you've enabled HTTPS successfully, Livy sends a response indicating that the command was accepted and that the batch job was submitted.

    {"id":1,"name":null,"owner":null,"proxyUser":null,"state":"starting","appId":null,"appInfo": {"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","\nYARN Diagnostics: "]}