Lab8-EMR-Spark-New.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WARNING: MAKE SURE YOU TERMINATE YOUR AWS INSTANCES WHEN YOU'RE DONE!!! OTHERWISE YOU WILL GET CHARGED (PROBABLY A LOT) BY AMAZON."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an AWS account\n",
    "\n",
    "* Go to https://aws.amazon.com acn click on the \"Sign In to the Console\" button on the upper right.\n",
    "\n",
    "---\n",
    "\n",
    "## Create an IAM user\n",
    "\n",
    "1. Log in (or create a new account) and click on the _Identity & Access Management_ link under the __Security & Identity__ section.\n",
    "2. Select _Users_ on the menu on the left and click on the _Create New Users_ button.\n",
    "3. Use __cs109__ for the user name and make sure the _Generate an access key for each user_ checkbox is selected.\n",
    "4. Click the _Download Credentials_ button to get the new credential keys. Once the file is downloaded, click the _Close_ button.\n",
    "5. Click on the newly created __cs109__ user.\n",
    "6. Scroll down until you see the _Attach Policy_ button and click on it.\n",
    "7. Search and select the __AdministratorAccess__ policy, and click the _Attach Policy_ button.\n",
    "\n",
    "#### You should now have a file called __credentials.csv__ on your default download path and a new user with the necessary permissions to perform the next actions.\n",
    "\n",
    "---\n",
    "\n",
    "## Download and configure the AWS CLI tools\n",
    "\n",
    "1. Go to http://docs.aws.amazon.com/cli/latest/userguide/installing.html and find the instructions for the platform you're using.\n",
    "2. Run the following on the command line: ```aws configure```\n",
    "3. Fill out the requested information (replace the ??? bellow with the values from the credentials file):\n",
    "```\n",
    "AWS Access Key ID [None]: ???\n",
    "AWS Secret Access Key [None]: ???\n",
    "Default region name [None]: us-east-1\n",
    "Default output format [None]: json\n",
    "```\n",
    "4. Run the following on the command line: ```aws emr create-default-roles```\n",
    "\n",
    "#### You should get a big JSON string as the output of this command.\n",
    "\n",
    "---\n",
    "\n",
    "## The previous two steps only need to be performed once. The next steps must to be performed every time you create a new cluster (unless noted otherwise).\n",
    "\n",
    "---\n",
    "\n",
    "## Create an EC2 SSH key pair\n",
    "\n",
    "### Only perform the step bellow if this is the first time you are creating the SSH keys, or if you deleted the keys created on a previous iteration of these instructions.\n",
    "\n",
    "* Run the following on the command line:\n",
    "\n",
    "```\n",
    "aws ec2 create-key-pair --key-name CS109 --query 'KeyMaterial' --output text > CS109.pem\n",
    "chmod 400 CS109.pem\n",
    "```\n",
    "\n",
    "#### You should now have a file called __CS109.pem__ on your current directory.\n",
    "\n",
    "---\n",
    "\n",
    "## Create the Spark cluster\n",
    "\n",
    "* Run the following on the command line:\n",
    "```\n",
    "export CLUSTER_ID=`aws emr create-cluster --name \"CS109 Spark cluster\" \\\n",
    "--release-label emr-4.1.0 --applications Name=Spark --ec2-attributes KeyName=CS109 \\\n",
    "--instance-type m1.large --instance-count 3 --use-default-roles \\\n",
    "--bootstrap-actions Path=s3://cs109-2015/install-anaconda-emr,Name=Install_Anaconda \\\n",
    "--query 'ClusterId' --output text` && echo $CLUSTER_ID\n",
    "```\n",
    "\n",
    "The output of this command will be something like the following (your actual value will be different):\n",
    "```\n",
    "j-33S87OUETACNK\n",
    "```\n",
    "\n",
    "It will take a few minutes for your cluster to be ready. Go watch some cat videos on YouTube and come back in 20 minutes or so.\n",
    "\n",
    "---\n",
    "\n",
    "## Check cluster status\n",
    "\n",
    "* Run the following on the command line:\n",
    "```\n",
    "aws emr describe-cluster --cluster-id $CLUSTER_ID --query 'Cluster.Status.State' --output text\n",
    "```\n",
    "\n",
    "If the output is anything but _WAITING_, your cluster is not ready yet. Wait a few more minutes and run the command again. Do not proceeed until the cluster is ready.\n",
    "\n",
    "---\n",
    "\n",
    "## Connect to the iPython Notebook:\n",
    "\n",
    "#### Make sure the cluster is ready before proceedeing.\n",
    "\n",
    "1. Get the cluster master's IP:\n",
    "```\n",
    "export DNS_NAME=`aws emr describe-cluster --cluster-id $CLUSTER_ID \\\n",
    "--query 'Cluster.MasterPublicDnsName' --output text` && echo $DNS_NAME\n",
    "```\n",
    "\n",
    "2. Run the script to configure Spark:\n",
    "```\n",
    "ssh -o ServerAliveInterval=10 -i CS109.pem hadoop@$DNS_NAME 'sh configure-spark.sh'\n",
    "```\n",
    "\n",
    "3. Create an SSH tunel to the AWS box (this assumes your SSH key is on the same directory you are invoking the SSH command from).\n",
    "```\n",
    "ssh -o ServerAliveInterval=10 -i CS109.pem hadoop@$DNS_NAME -L 8989:localhost:8888\n",
    "```\n",
    "\n",
    "4. The previous command will create an SSH connection to the Spark cluster and a tunnel to access the notebook. Run the following command on the SSH section:\n",
    "```\n",
    "pyspark\n",
    "```\n",
    "\n",
    "5. Open your browser and got to http://localhost:8989 (or just click on this link)\n",
    "\n",
    "---\n",
    "\n",
    "## Do your thing...\n",
    "\n",
    "---\n",
    "\n",
    "## Terminate the Spark cluster\n",
    "\n",
    "1. Press _CTRL-C_ twice to terminate iPython.\n",
    "\n",
    "2. Type _exit_ on the command line to exit the SSH session.\n",
    "\n",
    "3. Run the following on the command line:\n",
    "```\n",
    "aws emr terminate-clusters --cluster-ids $CLUSTER_ID\n",
    "```\n",
    "\n",
    "* Optionally, run the following two commands to remove your keys. This is only needed if you don't plan on creating new clusters.\n",
    "\n",
    "```\n",
    "aws ec2 delete-key-pair --key-name CS109\n",
    "rm CS109.pem\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Make sure the cluster was terminated\n",
    "\n",
    "* Run the following on the command line:\n",
    "```\n",
    "aws emr describe-cluster --cluster-id $CLUSTER_ID --query 'Cluster.Status.State' --output text\n",
    "```\n",
    "\n",
    "If the output is anything different from _TERMINATING_ or _TERMINATED_, re-run the command above or go to the AWS console and terminate the cluster manually.\n",
    "\n",
    "### YOU MUST TERMINATE THE CLUSTER OR YOU WILL BE CHARGED!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}