0a - unit 05.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3710jvsc74a57bd0a970c7e51787c6ef7f3c805442fbb5a30ae2745635e1d2514e62d936ba85c55c",
   "display_name": "Python 3.7.10 64-bit ('ml_crash_course': conda)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Exercise: Datasets in Python\n",
    "\n",
    "In the previous exercise we loaded some data and fit a model to it. Several aspects of this were simplified - particularly that the data was hard-coded into our python script, and we didn't spend any time really looking at the data itself.\n",
    "\n",
    "Here we will load data from file, filter it, and graph it. Doing so is a very important first step in order to build proper models, or to understand their limitations.\n",
    "\n",
    "As normal, in the examples below, there is no need to edit any code. Try to read it, understand it, then press the run button to run it. As always, it is vitally important that these code blocks are run in the correct order, and nothing is missed.\n",
    "\n",
    "## Loading data with Pandas\n",
    "\n",
    "There are large variety of libraries that help you work with data. In Python, one of the most common is _Pandas_. We used pandas briefly in the previous exercise. Pandas can open data saved as text files and store them in an organised table called a `DataFrame`.\n",
    "\n",
    "Let's open some text data that is stored on disk. Our data is saved in a file called `doggy-boot-harness.csv`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "   boot_size  harness_size     sex  age_years\n",
       "0         39            58    male       12.0\n",
       "1         38            58    male        9.6\n",
       "2         37            52  female        8.6\n",
       "3         39            58    male       10.2\n",
       "4         38            57    male        7.8"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>boot_size</th>\n      <th>harness_size</th>\n      <th>sex</th>\n      <th>age_years</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>39</td>\n      <td>58</td>\n      <td>male</td>\n      <td>12.0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>38</td>\n      <td>58</td>\n      <td>male</td>\n      <td>9.6</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>37</td>\n      <td>52</td>\n      <td>female</td>\n      <td>8.6</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>39</td>\n      <td>58</td>\n      <td>male</td>\n      <td>10.2</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>38</td>\n      <td>57</td>\n      <td>male</td>\n      <td>7.8</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "source": [
    "import pandas\n",
    "\n",
    "# Read the text file containing data using pandas\n",
    "dataset = pandas.read_csv('Data/doggy-boot-harness.csv')\n",
    "\n",
    "# Print the data\n",
    "# Because there are a lot of data, use head() to only print the first few rows\n",
    "dataset.head()"
   ]
  },
  {
   "source": [
    "As you can see, this dataset contains information about dogs, including their doggy boot size, harness size, sex, and age in years.\n",
    "\n",
    "Data are stored as columns and rows, similar to a table you might see in Excel.\n",
    "\n",
    "## Filtering data by Columns\n",
    "\n",
    "Data are easy to filter by columns. We can either type this directly, like `dataset.my_column_name`, or like so: `dataset[\"my_column_name\"]`.\n",
    "\n",
    "We can use this to either extract data, or to delete data.\n",
    "\n",
    "Lets take a look at the harness sizes, and delete the `sex` and `age_years` columns. "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Harness sizes\n0     58\n1     58\n2     52\n3     58\n4     57\n5     52\n6     55\n7     53\n8     49\n9     54\n10    59\n11    56\n12    53\n13    58\n14    57\n15    58\n16    56\n17    51\n18    50\n19    59\n20    59\n21    59\n22    55\n23    50\n24    55\n25    52\n26    53\n27    54\n28    61\n29    56\n30    55\n31    60\n32    57\n33    56\n34    61\n35    58\n36    53\n37    57\n38    57\n39    55\n40    60\n41    51\n42    52\n43    56\n44    55\n45    57\n46    58\n47    57\n48    51\n49    59\nName: harness_size, dtype: int64\n\nAvailable columns after deleting sex and age information:\n['boot_size' 'harness_size']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Look at the harness sizes\n",
    "print(\"Harness sizes\")\n",
    "print(dataset.harness_size)\n",
    "\n",
    "# Remove the sex and age-in-years columns.\n",
    "del dataset[\"sex\"]\n",
    "del dataset[\"age_years\"]\n",
    "\n",
    "# Print the column names\n",
    "print(\"\\nAvailable columns after deleting sex and age information:\")\n",
    "print(dataset.columns.values)\n"
   ]
  },
  {
   "source": [
    "\n",
    "## Filtering data by Rows\n",
    "\n",
    "We can get data from the top of the table by using the `head()` function, or from the bottom of the table by using the `tail()` function. \n",
    "\n",
    "Both functions make a shallow copy of a section of our dataframe. Here we sending these copies to the `print()` function, the head and tail views can also be used for other purposes, such as for use in analyses or graphs."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "# Print the data at the top of the table\n",
    "print(\"TOP OF TABLE\")\n",
    "print(dataset.head())\n",
    "\n",
    "# print the data at the bottom of the table\n",
    "print(\"\\nBOTTOM OF TABLE\")\n",
    "print(dataset.tail())"
   ],
   "cell_type": "code",
   "metadata": {},
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "TOP OF TABLE\n   boot_size  harness_size\n0         39            58\n1         38            58\n2         37            52\n3         39            58\n4         38            57\n\nBOTTOM OF TABLE\n    boot_size  harness_size\n45         41            57\n46         39            58\n47         39            57\n48         35            51\n49         39            59\n"
     ]
    }
   ]
  },
  {
   "source": [
    "We can also filter logically. For example, we can look at data for dogs who have a harness smaller than a size 55.\n",
    "\n",
    "This works by calculating a `True` or `False` value for each row, then keeping only those rows where the value is `True`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "We have 50 rows of data\n\nWhether the dog's harness was smaller than size 55:\n0     False\n1     False\n2      True\n3     False\n4     False\n5      True\n6     False\n7      True\n8      True\n9      True\n10    False\n11    False\n12     True\n13    False\n14    False\n15    False\n16    False\n17     True\n18     True\n19    False\n20    False\n21    False\n22    False\n23     True\n24    False\n25     True\n26     True\n27     True\n28    False\n29    False\n30    False\n31    False\n32    False\n33    False\n34    False\n35    False\n36     True\n37    False\n38    False\n39    False\n40    False\n41     True\n42     True\n43    False\n44    False\n45    False\n46    False\n47    False\n48     True\n49    False\nName: harness_size, dtype: bool\n\nData for dogs with harness smaller than size 55:\n    boot_size  harness_size\n2          37            52\n5          35            52\n7          36            53\n8          35            49\n9          40            54\n12         38            53\n17         36            51\n18         35            50\n23         35            50\n25         36            52\n26         35            53\n27         39            54\n36         36            53\n41         35            51\n42         36            52\n48         35            51\n\nNumber of dogs with harness size less than 55: 16\n"
     ]
    }
   ],
   "source": [
    "# Print how many rows of data we have\n",
    "print(f\"We have {len(dataset)} rows of data\")\n",
    "\n",
    "# Determine whether each avalanche dog's harness size is < 55\n",
    "# This creates a True or False value for each row where True means \n",
    "# they are smaller than 55\n",
    "is_small = dataset.harness_size < 55\n",
    "print(\"\\nWhether the dog's harness was smaller than size 55:\")\n",
    "print(is_small)\n",
    "\n",
    "# Now apply this 'mask' to our data to keep the smaller dogs\n",
    "data_from_small_dogs = dataset[is_small]\n",
    "print(\"\\nData for dogs with harness smaller than size 55:\")\n",
    "print(data_from_small_dogs)\n",
    "\n",
    "# Print the number of small dogs\n",
    "print(f\"\\nNumber of dogs with harness size less than 55: {len(data_from_small_dogs)}\")"
   ]
  },
  {
   "source": [
    "This looks like a lot of code, but we can compress the important parts into a single line.\n",
    "\n",
    "Let's do something similar - restrict our data to only those with boot sizes smaller than 40. "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "We now have 34 rows in our dataset. The last few rows are:\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "    boot_size  harness_size\n",
       "42         36            52\n",
       "46         39            58\n",
       "47         39            57\n",
       "48         35            51\n",
       "49         39            59"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>boot_size</th>\n      <th>harness_size</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>42</th>\n      <td>36</td>\n      <td>52</td>\n    </tr>\n    <tr>\n      <th>46</th>\n      <td>39</td>\n      <td>58</td>\n    </tr>\n    <tr>\n      <th>47</th>\n      <td>39</td>\n      <td>57</td>\n    </tr>\n    <tr>\n      <th>48</th>\n      <td>35</td>\n      <td>51</td>\n    </tr>\n    <tr>\n      <th>49</th>\n      <td>39</td>\n      <td>59</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "source": [
    "# Make a copy of the dataset that only contains dogs with \n",
    "# a boot size below size 40\n",
    "# The call to copy() is optional but can help avoid unexpected\n",
    "# behaviour in more complex scenarios\n",
    "data_smaller_paws = dataset[dataset.boot_size < 40].copy()\n",
    "\n",
    "\n",
    "# Print information about this\n",
    "print(f\"We now have {len(data_smaller_paws)} rows in our dataset. The last few rows are:\")\n",
    "data_smaller_paws.tail()"
   ]
  },
  {
   "source": [
    "## Graphing Data\n",
    "\n",
    "Graphing data is often the easiest way to understand it. \n",
    "\n",
    "In these exercises, we usually make our graphs using code in a custom file we have created, called `graphing.py`, which you can look at on our github page.\n",
    "\n",
    "Here, we will practice making a graph without this custom code, however.\n",
    "\n",
    "Lets make a simple graph of harness size versus boot size, for our avalanche dogs with smaller feet. "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "harness_size=%{x}<br>boot_size=%{y}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "",
         "orientation": "v",
         "showlegend": false,
         "type": "scatter",
         "x": [
          58,
          58,
          52,
          58,
          57,
          52,
          55,
          53,
          49,
          56,
          53,
          58,
          56,
          51,
          50,
          59,
          55,
          50,
          52,
          53,
          54,
          56,
          55,
          57,
          53,
          57,
          57,
          55,
          51,
          52,
          58,
          57,
          51,
          59
         ],
         "xaxis": "x",
         "y": [
          39,
          38,
          37,
          39,
          38,
          35,
          37,
          36,
          35,
          36,
          38,
          39,
          36,
          36,
          35,
          38,
          37,
          35,
          36,
          35,
          39,
          37,
          35,
          39,
          36,
          37,
          37,
          39,
          35,
          36,
          39,
          39,
          35,
          39
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "tracegroupgap": 0
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "white",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "white",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "box": [
           {
            "boxpoints": "outliers",
            "notched": false,
            "type": "box"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "#C8D4E3",
             "linecolor": "#C8D4E3",
             "minorgridcolor": "#C8D4E3",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "#C8D4E3",
             "linecolor": "#C8D4E3",
             "minorgridcolor": "#C8D4E3",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             },
             "line": {
              "width": 1
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             },
             "opacity": 0.8
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             },
             "opacity": 0.8
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "white",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "#C8D4E3"
          },
          "height": 400,
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "margin": {
           "b": 2,
           "l": 2,
           "r": 2,
           "t": 30
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "white",
          "polar": {
           "angularaxis": {
            "gridcolor": "#EBF0F8",
            "linecolor": "#EBF0F8",
            "ticks": ""
           },
           "bgcolor": "white",
           "radialaxis": {
            "gridcolor": "#EBF0F8",
            "linecolor": "#EBF0F8",
            "ticks": ""
           }
          },
          "scene": {
           "camera": {
            "eye": {
             "x": 1.5,
             "y": 1.5,
             "z": 0.1
            }
           },
           "xaxis": {
            "backgroundcolor": "white",
            "gridcolor": "#DFE8F3",
            "gridwidth": 2,
            "linecolor": "#EBF0F8",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "#EBF0F8"
           },
           "yaxis": {
            "backgroundcolor": "white",
            "gridcolor": "#DFE8F3",
            "gridwidth": 2,
            "linecolor": "#EBF0F8",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "#EBF0F8"
           },
           "zaxis": {
            "backgroundcolor": "white",
            "gridcolor": "#DFE8F3",
            "gridwidth": 2,
            "linecolor": "#EBF0F8",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "#EBF0F8"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "#DFE8F3",
            "linecolor": "#A2B1C6",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "#DFE8F3",
            "linecolor": "#A2B1C6",
            "ticks": ""
           },
           "bgcolor": "white",
           "caxis": {
            "gridcolor": "#DFE8F3",
            "linecolor": "#A2B1C6",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.5
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "#EBF0F8",
           "linecolor": "#EBF0F8",
           "linewidth": 2,
           "showline": true,
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "#EBF0F8",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "#EBF0F8",
           "linecolor": "#EBF0F8",
           "linewidth": 2,
           "showline": true,
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "#EBF0F8",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "harness_size"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "boot_size"
         }
        }
       }
      }
     },
     "metadata": {}
    }
   ],
   "source": [
    "# Load and prepare plotly to create our graphs\n",
    "import plotly.express\n",
    "import graphing # this is a custom file you can find in our code on github\n",
    "\n",
    "# Show a graph of harness size by boot size:\n",
    "plotly.express.scatter(data_smaller_paws, x=\"harness_size\", y=\"boot_size\")"
   ]
  },
  {
   "source": [
    "## Creating New Columns\n",
    "\n",
    "The graph above shows the relationship we want to investigate for our store, but some customers might want harness sizes lists in inches, not centimeters. How can we view these harness sizes in imperial units?\n",
    "\n",
    "To do this, we will need to create a new column - `harness_size_imperial` - and put that on the X axis instead.\n",
    "\n",
    "Creating new columns uses very similar syntax to what we've seen before."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "harness_size_imperial=%{x}<br>boot_size=%{y}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "",
         "orientation": "v",
         "showlegend": false,
         "type": "scatter",
         "x": [
          22.834645669291337,
          22.834645669291337,
          20.47244094488189,
          22.834645669291337,
          22.440944881889763,
          20.47244094488189,
          21.653543307086615,
          20.866141732283463,
          19.291338582677167,
          22.04724409448819,
          20.866141732283463,
          22.834645669291337,
          22.04724409448819,
          20.078740157480315,
          19.68503937007874,
          23.228346456692915,
          21.653543307086615,
          19.68503937007874,
          20.47244094488189,
          20.866141732283463,
          21.25984251968504,
          22.04724409448819,
          21.653543307086615,
          22.440944881889763,
          20.866141732283463,
          22.440944881889763,
          22.440944881889763,
          21.653543307086615,
          20.078740157480315,
          20.47244094488189,
          22.834645669291337,
          22.440944881889763,
          20.078740157480315,
          23.228346456692915
         ],
         "xaxis": "x",
         "y": [
          39,
          38,
          37,
          39,
          38,
          35,
          37,
          36,
          35,
          36,
          38,
          39,
          36,
          36,
          35,
          38,
          37,
          35,
          36,
          35,
          39,
          37,
          35,
          39,
          36,
          37,
          37,
          39,
          35,
          36,
          39,
          39,
          35,
          39
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "tracegroupgap": 0
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "white",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "white",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "box": [
           {
            "boxpoints": "outliers",
            "notched": false,
            "type": "box"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "#C8D4E3",
             "linecolor": "#C8D4E3",
             "minorgridcolor": "#C8D4E3",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "#C8D4E3",
             "linecolor": "#C8D4E3",
             "minorgridcolor": "#C8D4E3",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             },
             "line": {
              "width": 1
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             },
             "opacity": 0.8
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             },
             "opacity": 0.8
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "white",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "#C8D4E3"
          },
          "height": 400,
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "margin": {
           "b": 2,
           "l": 2,
           "r": 2,
           "t": 30
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "white",
          "polar": {
           "angularaxis": {
            "gridcolor": "#EBF0F8",
            "linecolor": "#EBF0F8",
            "ticks": ""
           },
           "bgcolor": "white",
           "radialaxis": {
            "gridcolor": "#EBF0F8",
            "linecolor": "#EBF0F8",
            "ticks": ""
           }
          },
          "scene": {
           "camera": {
            "eye": {
             "x": 1.5,
             "y": 1.5,
             "z": 0.1
            }
           },
           "xaxis": {
            "backgroundcolor": "white",
            "gridcolor": "#DFE8F3",
            "gridwidth": 2,
            "linecolor": "#EBF0F8",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "#EBF0F8"
           },
           "yaxis": {
            "backgroundcolor": "white",
            "gridcolor": "#DFE8F3",
            "gridwidth": 2,
            "linecolor": "#EBF0F8",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "#EBF0F8"
           },
           "zaxis": {
            "backgroundcolor": "white",
            "gridcolor": "#DFE8F3",
            "gridwidth": 2,
            "linecolor": "#EBF0F8",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "#EBF0F8"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "#DFE8F3",
            "linecolor": "#A2B1C6",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "#DFE8F3",
            "linecolor": "#A2B1C6",
            "ticks": ""
           },
           "bgcolor": "white",
           "caxis": {
            "gridcolor": "#DFE8F3",
            "linecolor": "#A2B1C6",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.5
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "#EBF0F8",
           "linecolor": "#EBF0F8",
           "linewidth": 2,
           "showline": true,
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "#EBF0F8",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "#EBF0F8",
           "linecolor": "#EBF0F8",
           "linewidth": 2,
           "showline": true,
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "#EBF0F8",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "harness_size_imperial"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "boot_size"
         }
        }
       }
      }
     },
     "metadata": {}
    }
   ],
   "source": [
    "# Convert harness sizes from metric to imperial units \n",
    "# and save the result to a new column\n",
    "data_smaller_paws['harness_size_imperial'] = data_smaller_paws.harness_size / 2.54\n",
    "\n",
    "# Show a graph but not in imperial units\n",
    "plotly.express.scatter(data_smaller_paws, x=\"harness_size_imperial\", y=\"boot_size\")"
   ]
  },
  {
   "source": [
    "We've now graphed our new column of data (`harness_size_imperial`) against boot size for dogs with small paws.\n",
    "\n",
    "## Summary\n",
    "\n",
    "We've introduced working with data in Python. This has included:\n",
    "* Opening tablated data from a file into a `DataFrame` (table)\n",
    "* Inspecting the top and bottom of the dataframe\n",
    "* Adding and removing columns of data\n",
    "* Removing rows of data based on criteria\n",
    "* Graphing data to understand trends\n",
    "\n",
    "Learning to work with dataframes can feel tedious or dry, but keep going because these basic skills are critical to unlocking exciting machine learning techniques that we will cover in later modules."
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ]
}