Week4_DataFrame_Part1_Ex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 5\n",
    "# Getting Started with Pandas (Part 1)\n",
    "\n",
    "Pandas is a major tool of data scientists. It contains data structures and data manipulation tools designed to make data cleaning and analysis fast and easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Download and extract MovieLens data\n",
    "\n",
    "- [GroupLens Research](https://grouplens.org) is a human-computer interaction research lab at University of Minnesota, Twin Cities. \n",
    "- [MovieLens](https://grouplens.org/datasets/movielens/) is a web-based recommender system and virtual community that recommends movies for its users to watch, based on their file preferences using collaborative filtering of members' movie ratings and movie reviews. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Data/ml-latest-small.zip', <http.client.HTTPMessage at 0x1c008ab9128>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download ml-latest-small.zip from https://grouplens.org/datasets/movielens/\n",
    "import urllib.request\n",
    "url = \"http://files.grouplens.org/datasets/movielens/ml-latest-small.zip\"\n",
    "file_path = \"Data/\" # you need to create this folder first\n",
    "\n",
    "import os\n",
    "os.mkdir(file_path)\n",
    "\n",
    "zip_file = \"ml-latest-small.zip\"\n",
    "urllib.request.urlretrieve(url, file_path + zip_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Name                                             Modified             Size\n",
      "ml-latest-small/                               2018-09-26 15:50:12            0\n",
      "ml-latest-small/links.csv                      2018-09-26 15:50:10       197979\n",
      "ml-latest-small/tags.csv                       2018-09-26 15:49:40       118660\n",
      "ml-latest-small/ratings.csv                    2018-09-26 15:49:38      2483723\n",
      "ml-latest-small/README.txt                     2018-09-26 15:50:12         8342\n",
      "ml-latest-small/movies.csv                     2018-09-26 15:49:56       494431\n"
     ]
    }
   ],
   "source": [
    "# Decompress the zip file\n",
    "import zipfile\n",
    "f = zipfile.ZipFile(file_path + zip_file, \"r\")\n",
    "f.printdir()\n",
    "f.extractall(file_path)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Since we only need the zip file in this step, \n",
    "# we can use the following strcuture:\n",
    "with zipfile.ZipFile(file_path + zip_file, \"r\") as f:\n",
    "    f.printdir()\n",
    "    f.extractall(file_path)\n",
    "# This cell does exactly the same thing as the above cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['links.csv', 'movies.csv', 'ratings.csv', 'ratings_processed.csv', 'ratings_processed.feather', 'README.txt', 'tags.csv']\n"
     ]
    }
   ],
   "source": [
    "# Check that files has been unzipped\n",
    "import os\n",
    "print(os.listdir(file_path + \"ml-latest-small\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cereals', 'creditcardfraud', 'ml-20m', 'ml-20m.zip', 'ml-latest-small']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete the zip file\n",
    "os.remove(file_path + zip_file)\n",
    "os.listdir(file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Write a function to download and decompress file\n",
    "import os\n",
    "import urllib.request\n",
    "import zipfile\n",
    "\n",
    "def get_movielens(file_path, file_name, delete_zip_file=False):\n",
    "    url = \"http://files.grouplens.org/datasets/movielens/\" + file_name\n",
    "    if not os.path.exists(file_path + file_name):\n",
    "        urllib.request.urlretrieve(url, file_path + file_name)\n",
    "        print(\"File\", file_name, \"downloaded.\")\n",
    "    with zipfile.ZipFile(file_path + file_name, \"r\") as f:\n",
    "        f.extractall(file_path)\n",
    "        print(\"Files extracted:\")\n",
    "        print(f.printdir())\n",
    "    if delete_zip_file:\n",
    "        os.remove(file_path + file_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File ml-latest-small.zip downloaded.\n",
      "Files extracted:\n",
      "File Name                                             Modified             Size\n",
      "ml-latest-small/                               2018-09-26 15:50:12            0\n",
      "ml-latest-small/links.csv                      2018-09-26 15:50:10       197979\n",
      "ml-latest-small/tags.csv                       2018-09-26 15:49:40       118660\n",
      "ml-latest-small/ratings.csv                    2018-09-26 15:49:38      2483723\n",
      "ml-latest-small/README.txt                     2018-09-26 15:50:12         8342\n",
      "ml-latest-small/movies.csv                     2018-09-26 15:49:56       494431\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "file_path = \"Data/\"\n",
    "zip_file = \"ml-latest-small.zip\"\n",
    "get_movielens(file_path, zip_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cereals', 'creditcardfraud', 'ml-latest-small', 'ml-latest-small.zip']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A larger dataset is ml-20m\n",
    "# get_movielens(file_path, \"ml-20m.zip\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Load ratings data as a pandas DataFrame\n",
    "- Load csv file as a DataFrame\n",
    "- head()\n",
    "- shape, columns, dtypes\n",
    "- indexing with loc[]\n",
    "- Handle date and time data\n",
    "- Data transformation with apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp\n",
       "0       1        1     4.0  964982703\n",
       "1       1        3     4.0  964981247\n",
       "2       1        6     4.0  964982224\n",
       "3       1       47     5.0  964983815\n",
       "4       1       50     5.0  964982931"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "ratings_df = pd.read_csv(\"Data/ml-latest-small/ratings.csv\",\n",
    "                         delimiter=\",\")\n",
    "ratings_df.head()  # Show the first several rows of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100831</th>\n",
       "      <td>610</td>\n",
       "      <td>166534</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1493848402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100832</th>\n",
       "      <td>610</td>\n",
       "      <td>168248</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1493850091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100833</th>\n",
       "      <td>610</td>\n",
       "      <td>168250</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1494273047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100834</th>\n",
       "      <td>610</td>\n",
       "      <td>168252</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1493846352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100835</th>\n",
       "      <td>610</td>\n",
       "      <td>170875</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1493846415</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        userId  movieId  rating   timestamp\n",
       "100831     610   166534     4.0  1493848402\n",
       "100832     610   168248     5.0  1493850091\n",
       "100833     610   168250     5.0  1494273047\n",
       "100834     610   168252     5.0  1493846352\n",
       "100835     610   170875     3.0  1493846415"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns: userId, movieId, rating, timestamp\n"
     ]
    }
   ],
   "source": [
    "print(\"Columns:\", \", \".join(ratings_df.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (100836, 4)\n"
     ]
    }
   ],
   "source": [
    "print(\"Shape:\", ratings_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data types:\n",
      " userId         int64\n",
      "movieId        int64\n",
      "rating       float64\n",
      "timestamp      int64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(\"Data types:\\n\", ratings_df.dtypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "964982703\n",
      "2000-07-30 14:45:03\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp    Year  Month   Day  Hour  Minute  \\\n",
       "0       1        1     4.0  964982703  2000.0    7.0  30.0  14.0    45.0   \n",
       "1       1        3     4.0  964981247     NaN    NaN   NaN   NaN     NaN   \n",
       "2       1        6     4.0  964982224     NaN    NaN   NaN   NaN     NaN   \n",
       "3       1       47     5.0  964983815     NaN    NaN   NaN   NaN     NaN   \n",
       "4       1       50     5.0  964982931     NaN    NaN   NaN   NaN     NaN   \n",
       "\n",
       "   Second  \n",
       "0     3.0  \n",
       "1     NaN  \n",
       "2     NaN  \n",
       "3     NaN  \n",
       "4     NaN  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert time stamp to a readable format\n",
    "\n",
    "# Example: convert the first time stamp\n",
    "timestamp = ratings_df.loc[0, \"timestamp\"]\n",
    "print(timestamp)\n",
    "\n",
    "from datetime import datetime\n",
    "dt = datetime.fromtimestamp(timestamp)\n",
    "print(dt)\n",
    "\n",
    "ratings_df.loc[0, \"Year\"] = dt.year\n",
    "ratings_df.loc[0, \"Month\"] = dt.month\n",
    "ratings_df.loc[0, \"Day\"] = dt.day\n",
    "ratings_df.loc[0, \"Hour\"] = dt.hour\n",
    "ratings_df.loc[0, \"Minute\"] = dt.minute\n",
    "ratings_df.loc[0, \"Second\"] = dt.second\n",
    "ratings_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000-07-30 14:45:03\n",
      "2000 - Jul - 30\n"
     ]
    }
   ],
   "source": [
    "# change the format of datetime\n",
    "print(dt)\n",
    "dt2 = dt.strftime('%Y - %b - %d')\n",
    "print(dt2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2000-07-30 14:45:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:20:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:37:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 15:03:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:48:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp    Year  Month   Day  Hour  Minute  \\\n",
       "0       1        1     4.0  964982703  2000.0    7.0  30.0  14.0    45.0   \n",
       "1       1        3     4.0  964981247     NaN    NaN   NaN   NaN     NaN   \n",
       "2       1        6     4.0  964982224     NaN    NaN   NaN   NaN     NaN   \n",
       "3       1       47     5.0  964983815     NaN    NaN   NaN   NaN     NaN   \n",
       "4       1       50     5.0  964982931     NaN    NaN   NaN   NaN     NaN   \n",
       "\n",
       "   Second                  DT  \n",
       "0     3.0 2000-07-30 14:45:03  \n",
       "1     NaN 2000-07-30 14:20:47  \n",
       "2     NaN 2000-07-30 14:37:04  \n",
       "3     NaN 2000-07-30 15:03:35  \n",
       "4     NaN 2000-07-30 14:48:51  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use apply() to apply a function to all rows\n",
    "ratings_df[\"DT\"] = ratings_df[\"timestamp\"].apply(datetime.fromtimestamp)\n",
    "ratings_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>14.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2000-07-30 14:45:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:20:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:37:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 15:03:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:48:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp  Year  Month  Day  Hour  Minute  Second  \\\n",
       "0       1        1     4.0  964982703  2000      7   30  14.0    45.0     3.0   \n",
       "1       1        3     4.0  964981247  2000      7   30   NaN     NaN     NaN   \n",
       "2       1        6     4.0  964982224  2000      7   30   NaN     NaN     NaN   \n",
       "3       1       47     5.0  964983815  2000      7   30   NaN     NaN     NaN   \n",
       "4       1       50     5.0  964982931  2000      7   30   NaN     NaN     NaN   \n",
       "\n",
       "                   DT  \n",
       "0 2000-07-30 14:45:03  \n",
       "1 2000-07-30 14:20:47  \n",
       "2 2000-07-30 14:37:04  \n",
       "3 2000-07-30 15:03:35  \n",
       "4 2000-07-30 14:48:51  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fill the year column\n",
    "ratings_df['Year'] = ratings_df['DT'].dt.year\n",
    "ratings_df['Month'] = ratings_df['DT'].dt.month\n",
    "ratings_df['Day'] = ratings_df['DT'].dt.day\n",
    "\n",
    "ratings_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply() can be used to apply user-defined functions\n",
    "\n",
    "# Exercise: define a function that convert month to Q1 - Q4, and \n",
    "# use this function to create a \"Quarter\" column:\n",
    "\n",
    "def convertMonToQuarter(month):\n",
    "    if(month < 4):\n",
    "        quarter = 1\n",
    "    if (month>= 4 and month<7):\n",
    "        quarter = 2\n",
    "    if (month >= 7 and month < 9):\n",
    "        quarter = 3\n",
    "    if(month >= 9):\n",
    "        quarter = 4\n",
    "    return quarter    \n",
    "convertMonToQuarter(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ratings_df[\"Quarter\"] = ratings_df['Month'].apply(convertMonToQuarter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "      <th>Quarter</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100831</th>\n",
       "      <td>610</td>\n",
       "      <td>166534</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1493848402</td>\n",
       "      <td>2017</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:53:22</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100832</th>\n",
       "      <td>610</td>\n",
       "      <td>168248</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1493850091</td>\n",
       "      <td>2017</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 18:21:31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100833</th>\n",
       "      <td>610</td>\n",
       "      <td>168250</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1494273047</td>\n",
       "      <td>2017</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-08 15:50:47</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100834</th>\n",
       "      <td>610</td>\n",
       "      <td>168252</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1493846352</td>\n",
       "      <td>2017</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:19:12</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100835</th>\n",
       "      <td>610</td>\n",
       "      <td>170875</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1493846415</td>\n",
       "      <td>2017</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:20:15</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        userId  movieId  rating   timestamp  Year  Month  Day  Hour  Minute  \\\n",
       "100831     610   166534     4.0  1493848402  2017      5    3   NaN     NaN   \n",
       "100832     610   168248     5.0  1493850091  2017      5    3   NaN     NaN   \n",
       "100833     610   168250     5.0  1494273047  2017      5    8   NaN     NaN   \n",
       "100834     610   168252     5.0  1493846352  2017      5    3   NaN     NaN   \n",
       "100835     610   170875     3.0  1493846415  2017      5    3   NaN     NaN   \n",
       "\n",
       "        Second                  DT  Quarter  \n",
       "100831     NaN 2017-05-03 17:53:22        2  \n",
       "100832     NaN 2017-05-03 18:21:31        2  \n",
       "100833     NaN 2017-05-08 15:50:47        2  \n",
       "100834     NaN 2017-05-03 17:19:12        2  \n",
       "100835     NaN 2017-05-03 17:20:15        2  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Test1</th>\n",
       "      <th>Test2</th>\n",
       "      <th>Test3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Alice</th>\n",
       "      <td>70</td>\n",
       "      <td>75</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bob</th>\n",
       "      <td>80</td>\n",
       "      <td>85</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Clare</th>\n",
       "      <td>90</td>\n",
       "      <td>95</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Test1  Test2  Test3\n",
       "Alice     70     75     80\n",
       "Bob       80     85     90\n",
       "Clare     90     95    100"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\"Test1\": [70, 80, 90],\n",
    "                   \"Test2\": [75, 85, 95],\n",
    "                   \"Test3\": [80, 90, 100]},\n",
    "                  index=[\"Alice\", \"Bob\", \"Clare\"])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test1       90.000000\n",
      "Test2       95.000000\n",
      "Test3      100.000000\n",
      "Test1sq      9.486833\n",
      "Mean        78.018090\n",
      "Max        100.000000\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# Exercise: apply np.sqrt() to calculate the square root of test 1 scores\n",
    "df['Test1sq'] = df['Test1'].apply(np.sqrt)\n",
    "df.head()\n",
    "\n",
    "# Exercise: apply np.mean() to create an \"Average\" column\n",
    "df['Mean'] = df.apply(np.mean, axis=1)\n",
    "df.head()\n",
    "\n",
    "# Exercise: create a \"Max\" column that stores the maximum value from each row\n",
    "df['Max'] = df.apply(np.max, axis=1)\n",
    "df.head()\n",
    "\n",
    "# Without axis=1, the function is applied to each column of the data frame.\n",
    "print(df.apply(np.max))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Test1</th>\n",
       "      <th>Test2</th>\n",
       "      <th>Test3</th>\n",
       "      <th>sqrt from math</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Alice</th>\n",
       "      <td>70</td>\n",
       "      <td>75</td>\n",
       "      <td>80</td>\n",
       "      <td>8.366600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bob</th>\n",
       "      <td>80</td>\n",
       "      <td>85</td>\n",
       "      <td>90</td>\n",
       "      <td>8.944272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Clare</th>\n",
       "      <td>90</td>\n",
       "      <td>95</td>\n",
       "      <td>100</td>\n",
       "      <td>9.486833</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Test1  Test2  Test3  sqrt from math\n",
       "Alice     70     75     80        8.366600\n",
       "Bob       80     85     90        8.944272\n",
       "Clare     90     95    100        9.486833"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "# math.sqrt(16)\n",
    "df['sqrt from math'] = df['Test1'].apply(math.sqrt)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Save the processed ratings data\n",
    "\n",
    "Data can be saved in many different formats. Today we will study two commonly used formats: csv and feather.\n",
    "- csv: High readability and stability\n",
    "- feather: High read and write performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Save the processed data as a .csv file\n",
    "csv_file = \"Data/ml-latest-small/ratings_processed.csv\"\n",
    "ratings_df.to_csv(csv_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: pip in c:\\users\\ch002\\anaconda3\\lib\\site-packages (19.2.3)\n",
      "Requirement already satisfied: feather-format in c:\\users\\ch002\\anaconda3\\lib\\site-packages (0.4.0)\n",
      "Requirement already satisfied: pyarrow>=0.4.0 in c:\\users\\ch002\\anaconda3\\lib\\site-packages (from feather-format) (0.14.0)\n",
      "Requirement already satisfied: six>=1.0.0 in c:\\users\\ch002\\anaconda3\\lib\\site-packages (from pyarrow>=0.4.0->feather-format) (1.11.0)\n",
      "Requirement already satisfied: numpy>=1.14 in c:\\users\\ch002\\anaconda3\\lib\\site-packages (from pyarrow>=0.4.0->feather-format) (1.14.2)\n"
     ]
    }
   ],
   "source": [
    "# Use Python package installer pip to install feather\n",
    "!pip install --upgrade pip\n",
    "!pip install feather-format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import feather\n",
    "# Save the processed data as a .feather file\n",
    "feather_file = \"Data/ml-latest-small/ratings_processed.feather\"\n",
    "# feather.write_dataframe(ratings_df, feather_file)\n",
    "ratings_df.to_feather(feather_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>14.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2000-07-30 14:45:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:20:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:37:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 15:03:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "      <td>2000</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:48:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp  Year  Month  Day  Hour  Minute  Second  \\\n",
       "0       1        1     4.0  964982703  2000      7   30  14.0    45.0     3.0   \n",
       "1       1        3     4.0  964981247  2000      7   30   NaN     NaN     NaN   \n",
       "2       1        6     4.0  964982224  2000      7   30   NaN     NaN     NaN   \n",
       "3       1       47     5.0  964983815  2000      7   30   NaN     NaN     NaN   \n",
       "4       1       50     5.0  964982931  2000      7   30   NaN     NaN     NaN   \n",
       "\n",
       "                   DT  \n",
       "0 2000-07-30 14:45:03  \n",
       "1 2000-07-30 14:20:47  \n",
       "2 2000-07-30 14:37:04  \n",
       "3 2000-07-30 15:03:35  \n",
       "4 2000-07-30 14:48:51  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load .feather file as DataFrame\n",
    "# ratings_df2 = feather.read_dataframe(feather_file)\n",
    "ratings_df2 = pd.read_feather(feather_file)\n",
    "ratings_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1568665170.9880412"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to a csv file costs: 2.6284680366516113\n",
      "File size: 6399957\n",
      "Writing to a feather file costs: 0.01993727684020996\n",
      "File size: 8912432\n"
     ]
    }
   ],
   "source": [
    "# Time the writing speed\n",
    "import time\n",
    "start = time.time()\n",
    "ratings_df.to_csv(csv_file)\n",
    "end = time.time()\n",
    "print(\"Writing to a csv file costs:\", (end - start))\n",
    "print(\"File size:\", os.path.getsize(csv_file))\n",
    "\n",
    "start = time.time()\n",
    "ratings_df.to_feather(feather_file)\n",
    "end = time.time()\n",
    "print(\"Writing to a feather file costs:\", (end - start))\n",
    "print(\"File size:\", os.path.getsize(feather_file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.29296183586120605\n",
      "0.01935577392578125\n"
     ]
    }
   ],
   "source": [
    "# Exercise: Compare the loading time of csv and feather format\n",
    "\n",
    "start = time.time()\n",
    "ratings_df = pd.read_csv(csv_file,\n",
    "                        delimiter=\",\")\n",
    "end = time.time()\n",
    "print(end-start)\n",
    "\n",
    "start  = time.time()\n",
    "ratings_df = pd.read_feather(feather_file)\n",
    "end = time.time()\n",
    "print(end - start)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. pd.Series and pd.DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Two important pandas data structure: Series (1D) and DataFrame (2D)\n",
    "userId = ratings_df['userId']\n",
    "# ?userId\n",
    "# ?ratings_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]\n",
      "[610 610 610 610 610 610 610 610 610 610 610 610 610 610 610 610 610 610\n",
      " 610 610]\n"
     ]
    }
   ],
   "source": [
    "print(userId.values[:20])\n",
    "print(userId.values[-20:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "610\n"
     ]
    }
   ],
   "source": [
    "# Exercise: How many unique user IDs are there?\n",
    "userId_set = set(userId)\n",
    "print(len(userId_set))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RangeIndex(start=0, stop=20, step=1)\n"
     ]
    }
   ],
   "source": [
    "# Index of userId\n",
    "print(userId.index[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d    4\n",
       "b    7\n",
       "a   -5\n",
       "c    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a Series from scratch\n",
    "series = pd.Series([4, 7, -5, 3],\n",
    "                     index=['d', 'b', 'a', 'c'])\n",
    "series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "series['a']:\n",
      "-5\n",
      "\n",
      "series[['a', 'b', 'c']]:\n",
      "a   -5\n",
      "b    7\n",
      "c    3\n",
      "dtype: int64\n",
      "\n",
      "series[series > 0]:\n",
      "d    4\n",
      "b    7\n",
      "c    3\n",
      "dtype: int64\n",
      "[False False  True False]\n",
      "a   -5\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# slicing a Series\n",
    "print(\"series['a']:\")\n",
    "print(series['a'])\n",
    "print(\"\\nseries[['a', 'b', 'c']]:\")\n",
    "print(series[['a', 'b', 'c']])\n",
    "print(\"\\nseries[series > 0]:\")\n",
    "print(series[series > 0])\n",
    "# boolean values regarding index\n",
    "print(series.index == 'a')\n",
    "print(series[series.index == 'a'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "series * 2:\n",
      "d     8\n",
      "b    14\n",
      "a   -10\n",
      "c     6\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Arithmetic operations\n",
    "print(\"series * 2:\")\n",
    "print(series * 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio      35000\n",
       "Oregon    16000\n",
       "Texas     71000\n",
       "Utah       5000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a Series from a dictionary\n",
    "dictionary = {'Ohio': 35000,\n",
    "              'Texas': 71000,\n",
    "              'Oregon': 16000,\n",
    "              'Utah': 5000}\n",
    "series2 = pd.Series(dictionary)\n",
    "series2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio        35000\n",
       "Oregon      16000\n",
       "Texas       71000\n",
       "Utah         5000\n",
       "New York    12345\n",
       "dtype: int64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add values\n",
    "series2['New York'] = 12345\n",
    "series2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio      35000\n",
       "Oregon    16000\n",
       "Texas     71000\n",
       "Utah       5000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove values\n",
    "series2.drop('New York', inplace=True)\n",
    "series2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Oregon    16000\n",
       "Texas     71000\n",
       "Utah       5000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# without inplace=True\n",
    "series3 = series2.drop(\"Ohio\")\n",
    "series3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ohio      35000\n",
       "Oregon    16000\n",
       "Texas     71000\n",
       "Utah       5000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.181044</td>\n",
       "      <td>0.697178</td>\n",
       "      <td>0.461743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.976684</td>\n",
       "      <td>0.529399</td>\n",
       "      <td>0.995427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Feature1  Feature2  Feature3\n",
       "0  0.181044  0.697178  0.461743\n",
       "1  0.976684  0.529399  0.995427\n",
       "2  0.055476  0.402942  0.398853\n",
       "3  0.133120  0.672739  0.045918\n",
       "4  0.830402  0.106249  0.594716"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "# Define a DataFrame from scratch\n",
    "df1 = pd.DataFrame(np.random.rand(5, 3),\n",
    "                   columns=['Feature1', 'Feature2', 'Feature3'])\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "      <th>Feature4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.181044</td>\n",
       "      <td>0.697178</td>\n",
       "      <td>0.461743</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.976684</td>\n",
       "      <td>0.529399</td>\n",
       "      <td>0.995427</td>\n",
       "      <td>1.234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Feature1  Feature2  Feature3  Feature4\n",
       "0  0.181044  0.697178  0.461743       NaN\n",
       "1  0.976684  0.529399  0.995427     1.234\n",
       "2  0.055476  0.402942  0.398853       NaN\n",
       "3  0.133120  0.672739  0.045918       NaN\n",
       "4  0.830402  0.106249  0.594716       NaN"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add values\n",
    "df1.loc[1, 'Feature4'] = 1.234\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Feature1  Feature2  Feature3\n",
       "0  0.055476  0.402942  0.398853\n",
       "1  0.133120  0.672739  0.045918\n",
       "2  0.830402  0.106249  0.594716"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.04591826720906389"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.iloc[1, 2] # use the intrinsic index, not recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "      <th>Feature4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Feature1  Feature2  Feature3  Feature4\n",
       "0     False     False     False      True\n",
       "1     False     False     False     False\n",
       "2     False     False     False      True\n",
       "3     False     False     False      True\n",
       "4     False     False     False      True"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Identify null values\n",
    "df1.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df1[df1.isnull()] = 0\n",
    "# df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Feature1    0\n",
       "Feature2    0\n",
       "Feature3    0\n",
       "Feature4    4\n",
       "dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cound number of missing values for each column\n",
    "np.sum(df1.isnull(), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "      <th>Feature4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Feature1  Feature2  Feature3  Feature4\n",
       "2  0.055476  0.402942  0.398853       NaN\n",
       "3  0.133120  0.672739  0.045918       NaN\n",
       "4  0.830402  0.106249  0.594716       NaN"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove rows\n",
    "df1.drop([0, 1], axis=0, inplace=True)\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "      <th>Feature4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  Feature1  Feature2  Feature3  Feature4\n",
       "0      2  0.055476  0.402942  0.398853       NaN\n",
       "1      3  0.133120  0.672739  0.045918       NaN\n",
       "2      4  0.830402  0.106249  0.594716       NaN"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reset index\n",
    "df1.reset_index(inplace=True) # set drop=True to remove original index\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  Feature1  Feature2  Feature3\n",
       "0      2  0.055476  0.402942  0.398853\n",
       "1      3  0.133120  0.672739  0.045918\n",
       "2      4  0.830402  0.106249  0.594716"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove columns\n",
    "df1.drop(['Feature4'], axis=1, inplace=True)\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Feature1</th>\n",
       "      <th>Feature2</th>\n",
       "      <th>Feature3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.055476</td>\n",
       "      <td>0.402942</td>\n",
       "      <td>0.398853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.133120</td>\n",
       "      <td>0.672739</td>\n",
       "      <td>0.045918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.830402</td>\n",
       "      <td>0.106249</td>\n",
       "      <td>0.594716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Feature1  Feature2  Feature3\n",
       "0  0.055476  0.402942  0.398853\n",
       "1  0.133120  0.672739  0.045918\n",
       "2  0.830402  0.106249  0.594716"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.drop(['index'], axis=1, inplace=True)\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Exercise: Add one row to ratings_df and remove it.\n",
    "ratings_df.loc[100836, ['userId', 'movieId', 'rating']] = [1.1, 2.2, 3.3]\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100832</th>\n",
       "      <td>610.0</td>\n",
       "      <td>168248.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.493850e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 18:21:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100833</th>\n",
       "      <td>610.0</td>\n",
       "      <td>168250.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.494273e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-08 15:50:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100834</th>\n",
       "      <td>610.0</td>\n",
       "      <td>168252.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.493846e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:19:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100835</th>\n",
       "      <td>610.0</td>\n",
       "      <td>170875.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.493846e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:20:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100836</th>\n",
       "      <td>1.1</td>\n",
       "      <td>2.2</td>\n",
       "      <td>3.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        userId   movieId  rating     timestamp    Year  Month  Day  Hour  \\\n",
       "100832   610.0  168248.0     5.0  1.493850e+09  2017.0    5.0  3.0   NaN   \n",
       "100833   610.0  168250.0     5.0  1.494273e+09  2017.0    5.0  8.0   NaN   \n",
       "100834   610.0  168252.0     5.0  1.493846e+09  2017.0    5.0  3.0   NaN   \n",
       "100835   610.0  170875.0     3.0  1.493846e+09  2017.0    5.0  3.0   NaN   \n",
       "100836     1.1       2.2     3.3           NaN     NaN    NaN  NaN   NaN   \n",
       "\n",
       "        Minute  Second                  DT  \n",
       "100832     NaN     NaN 2017-05-03 18:21:31  \n",
       "100833     NaN     NaN 2017-05-08 15:50:47  \n",
       "100834     NaN     NaN 2017-05-03 17:19:12  \n",
       "100835     NaN     NaN 2017-05-03 17:20:15  \n",
       "100836     NaN     NaN                 NaT  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ratings_df.drop([100836], axis=0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100831</th>\n",
       "      <td>610.0</td>\n",
       "      <td>166534.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.493848e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:53:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100832</th>\n",
       "      <td>610.0</td>\n",
       "      <td>168248.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.493850e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 18:21:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100833</th>\n",
       "      <td>610.0</td>\n",
       "      <td>168250.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.494273e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-08 15:50:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100834</th>\n",
       "      <td>610.0</td>\n",
       "      <td>168252.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.493846e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:19:12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100835</th>\n",
       "      <td>610.0</td>\n",
       "      <td>170875.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.493846e+09</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017-05-03 17:20:15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        userId   movieId  rating     timestamp    Year  Month  Day  Hour  \\\n",
       "100831   610.0  166534.0     4.0  1.493848e+09  2017.0    5.0  3.0   NaN   \n",
       "100832   610.0  168248.0     5.0  1.493850e+09  2017.0    5.0  3.0   NaN   \n",
       "100833   610.0  168250.0     5.0  1.494273e+09  2017.0    5.0  8.0   NaN   \n",
       "100834   610.0  168252.0     5.0  1.493846e+09  2017.0    5.0  3.0   NaN   \n",
       "100835   610.0  170875.0     3.0  1.493846e+09  2017.0    5.0  3.0   NaN   \n",
       "\n",
       "        Minute  Second                  DT  \n",
       "100831     NaN     NaN 2017-05-03 17:53:22  \n",
       "100832     NaN     NaN 2017-05-03 18:21:31  \n",
       "100833     NaN     NaN 2017-05-08 15:50:47  \n",
       "100834     NaN     NaN 2017-05-03 17:19:12  \n",
       "100835     NaN     NaN 2017-05-03 17:20:15  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Analyzing MovieLens Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A. How are the ratings distributed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0    26818\n",
      "3.0    20047\n",
      "5.0    13211\n",
      "3.5    13136\n",
      "4.5     8551\n",
      "2.0     7551\n",
      "2.5     5550\n",
      "1.0     2811\n",
      "1.5     1791\n",
      "0.5     1370\n",
      "Name: rating, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Use value_counts() to find the rating frequencies\n",
    "rating_frequencies = ratings_df['rating'].value_counts()\n",
    "print(rating_frequencies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1c00e4c9908>]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEUVJREFUeJzt3X+IXWedx/H3Z6fGFV1p3Y4lNHUj\nS1is+0etl9qhsMyapaZd2VZQqLA2SCEiFSorrK3/1LVC9A91KWihuxZb1rVbVqXFrdaQ7UWEae1E\nu/1hdmnQamNDE4k/KoKh8bt/3DP0Js8kM5lkcma87xdczp3vfc6933sD85nznOfcpKqQJGncH/Xd\ngCRp7TEcJEkNw0GS1DAcJEkNw0GS1DAcJEkNw0GS1DAcJEkNw0GS1Din7wZW6vzzz6/Nmzf33YYk\nrSt79uz5eVVNLzVu3YbD5s2bmZ+f77sNSVpXkvxkOeOcVpIkNQwHSVLDcJAkNQwHSVLDcJAkNQwH\nSVLDcJC0ps3Nwc6do63OnnV7nYOkP3xzc7B1Kxw5Ahs2wO7dMDPTd1eTwSMHSWvWcDgKhqNHR9vh\nsO+OJofhIGnNmp0dHTFMTY22s7N9dzQ5nFaStGbNzIymkobDUTA4pXT2GA6S1rSZGUOhD04rSZIa\nhoMkqWE4SJIahoMkqWE4SJIahoMkqWE4SJIaS4ZDkouSPJxkb5Knk9zU1T+e5GdJHu9uV4/tc0uS\nfUn+L8k7xurbutq+JDeP1d+Y5NEkzyT5jyQbzvQblSQt33KOHF4CPlJVbwIuB25McnH32Oeq6pLu\n9iBA99h1wJuBbcAXkkwlmQI+D1wFXAy8d+x5Pt091xbgF8ANZ+j9SZJWYMlwqKoDVfX97v6LwF7g\nwpPscg1wb1X9rqp+DOwDLutu+6rqR1V1BLgXuCZJgLcD/9ntfzdw7UrfkCTp9J3SOYckm4G3AI92\npQ8leSLJXUnO62oXAs+N7ba/q52o/qfAL6vqpePqi73+jiTzSeYPHTp0Kq1Lkk7BssMhyWuArwIf\nrqpfA3cAfw5cAhwAPrMwdJHdawX1tlh1Z1UNqmowPT293NYlSadoWV+8l+QVjILhy1X1NYCqemHs\n8X8BvtH9uB+4aGz3TcDz3f3F6j8Hzk1yTnf0MD5ektSD5axWCvBFYG9VfXasvnFs2LuAp7r7DwDX\nJXllkjcCW4DvAY8BW7qVSRsYnbR+oKoKeBh4d7f/duD+03tbkqTTsZwjhyuA9wFPJnm8q32M0Wqj\nSxhNAT0LfACgqp5Och/wQ0YrnW6sqqMAST4EPARMAXdV1dPd830UuDfJJ4EfMAojSVJPMvrDff0Z\nDAY1Pz/fdxuStK4k2VNVg6XGeYW0JKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaD\nJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlh\nOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGoaDJKlhOEiSGkuGQ5KLkjycZG+Sp5Pc1NVf\nl2RXkme67XldPUluT7IvyRNJLh17ru3d+GeSbB+rvzXJk90+tyfJarxZSdLyLOfI4SXgI1X1JuBy\n4MYkFwM3A7uraguwu/sZ4CpgS3fbAdwBozABbgXeBlwG3LoQKN2YHWP7bTv9tyatT3NzsHPnaCv1\n5ZylBlTVAeBAd//FJHuBC4FrgNlu2N3AEPhoV7+nqgp4JMm5STZ2Y3dV1WGAJLuAbUmGwGuraq6r\n3wNcC3zzzLxFaf2Ym4OtW+HIEdiwAXbvhpmZvrvSJDqlcw5JNgNvAR4FLuiCYyFAXt8NuxB4bmy3\n/V3tZPX9i9QXe/0dSeaTzB86dOhUWpfWheFwFAxHj462w2HfHWlSLTsckrwG+Crw4ar69cmGLlKr\nFdTbYtWdVTWoqsH09PRSLUvrzuzs6Ihhamq0nZ3tuyNNqiWnlQCSvIJRMHy5qr7WlV9IsrGqDnTT\nRge7+n7gorHdNwHPd/XZ4+rDrr5pkfHSxJmZGU0lDYejYHBKSX1ZzmqlAF8E9lbVZ8ceegBYWHG0\nHbh/rH59t2rpcuBX3bTTQ8CVSc7rTkRfCTzUPfZiksu717p+7LmkiTMzA7fcYjCoX8s5crgCeB/w\nZJLHu9rHgE8B9yW5Afgp8J7usQeBq4F9wG+B9wNU1eEktwGPdeM+sXByGvgg8CXgVYxORHsyWpJ6\nlNGiovVnMBjU/Px8321I0rqSZE9VDZYa5xXSkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG\n4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJ\nahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqTGkuGQ5K4kB5M8NVb7eJKf\nJXm8u1099tgtSfYl+b8k7xirb+tq+5LcPFZ/Y5JHkzyT5D+SbDiTb1CSdOqWc+TwJWDbIvXPVdUl\n3e1BgCQXA9cBb+72+UKSqSRTwOeBq4CLgfd2YwE+3T3XFuAXwA2n84YkSadvyXCoqu8Ah5f5fNcA\n91bV76rqx8A+4LLutq+qflRVR4B7gWuSBHg78J/d/ncD157ie5AknWGnc87hQ0me6KadzutqFwLP\njY3Z39VOVP9T4JdV9dJxdUlSj1YaDncAfw5cAhwAPtPVs8jYWkF9UUl2JJlPMn/o0KFT61iStGwr\nCoeqeqGqjlbV74F/YTRtBKO//C8aG7oJeP4k9Z8D5yY557j6iV73zqoaVNVgenp6Ja1LkpZhReGQ\nZOPYj+8CFlYyPQBcl+SVSd4IbAG+BzwGbOlWJm1gdNL6gaoq4GHg3d3+24H7V9KTJOnMOWepAUm+\nAswC5yfZD9wKzCa5hNEU0LPABwCq6ukk9wE/BF4Cbqyqo93zfAh4CJgC7qqqp7uX+Chwb5JPAj8A\nvnjG3p0kaUUy+uN9/RkMBjU/P993G5K0riTZU1WDpcZ5hbQkqWE4SJIahoMkqWE4SJIahoMkqWE4\nSJIahoMkqWE4SNI6MTcHO3eOtqttySukJU2muTkYDmF2FmZm+u5Gc3OwdSscOQIbNsDu3av772I4\nSGqc7V9EWtpwOPr3OHp0tB0OV/ffxGklSY3FfhGpX7Ozo6CemhptZ2dX9/U8cpDUWPhFtHDksNq/\niLS0mZnREdzZmuozHCQ1zvYvIi3PzMzZ+7cwHCQt6mz+ItLa4zkHSVLDcJAkNQwHSVLDcJAkNQwH\nSVLDcJAkNQwHSVLDcJAkNQwHSVLDcJAkNQwHSVLDcJAkNQwHSVLDcJAkNQwHSVLDcJAkNZYMhyR3\nJTmY5Kmx2uuS7EryTLc9r6snye1J9iV5IsmlY/ts78Y/k2T7WP2tSZ7s9rk9Sc70m5QknZrlHDl8\nCdh2XO1mYHdVbQF2dz8DXAVs6W47gDtgFCbArcDbgMuAWxcCpRuzY2y/419LknSWLRkOVfUd4PBx\n5WuAu7v7dwPXjtXvqZFHgHOTbATeAeyqqsNV9QtgF7Cte+y1VTVXVQXcM/ZckqSerPScwwVVdQCg\n276+q18IPDc2bn9XO1l9/yJ1SVKPzvQJ6cXOF9QK6os/ebIjyXyS+UOHDq2wRUnSUlYaDi90U0J0\n24NdfT9w0di4TcDzS9Q3LVJfVFXdWVWDqhpMT0+vsHVJ0lJWGg4PAAsrjrYD94/Vr+9WLV0O/Kqb\ndnoIuDLJed2J6CuBh7rHXkxyebdK6fqx55Ik9eScpQYk+QowC5yfZD+jVUefAu5LcgPwU+A93fAH\ngauBfcBvgfcDVNXhJLcBj3XjPlFVCye5P8hoRdSrgG92N0lSjzJaJLT+DAaDmp+f77sNSVpXkuyp\nqsFS47xCWpLUMBwkSQ3DQZLUMBwkSQ3DQZKWMDcHO3eOtpNiyaWskjTJ5uZg61Y4cgQ2bIDdu2Fm\npu+uVp9HDpJ0EsPhKBiOHh1th8O+Ozo7DAdJOonZ2dERw9TUaDs723dHZ4fTSpJ0EjMzo6mk4XAU\nDJMwpQSGgyQtaWZmckJhgdNKkqSG4SBJahgO6t0kriGX1jrPOahXk7qGXFrrPHJQryZ1Dbm01hkO\n6tWkriGX1jqnldSrSV1DLq11hoN6N4lryKW1zmklSVLDcJAkNQwHSVLDcJAkNQwHSVLDcJAkNQwH\nSVLDcJAkNQwHSVLDcJA6fnW49DK/PkPCrw6XjueRg4RfHS4d77TCIcmzSZ5M8niS+a72uiS7kjzT\nbc/r6klye5J9SZ5IcunY82zvxj+TZPvpvSXp1PnV4dKxzsSRw19X1SVVNeh+vhnYXVVbgN3dzwBX\nAVu62w7gDhiFCXAr8DbgMuDWhUCRzpaFrw6/7TanlCRYnXMO1wCz3f27gSHw0a5+T1UV8EiSc5Ns\n7MbuqqrDAEl2AduAr6xCb9IJ+dXh0stO98ihgG8n2ZNkR1e7oKoOAHTb13f1C4Hnxvbd39VOVJck\n9eR0jxyuqKrnk7we2JXkf08yNovU6iT19glGAbQD4A1veMOp9ipJWqbTOnKoque77UHg64zOGbzQ\nTRfRbQ92w/cDF43tvgl4/iT1xV7vzqoaVNVgenr6dFqXJJ3EisMhyauT/MnCfeBK4CngAWBhxdF2\n4P7u/gPA9d2qpcuBX3XTTg8BVyY5rzsRfWVXkyT15HSmlS4Avp5k4Xn+vaq+leQx4L4kNwA/Bd7T\njX8QuBrYB/wWeD9AVR1OchvwWDfuEwsnpyVJ/cho8dD6MxgMan5+vu82JGldSbJn7NKDE/IKaUlS\nw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMh57MzcHOnaOtJK01q/E/\nwWkJc3OwdevoP7LfsMH/llLS2uORQw+Gw1EwHD062g6HfXckSccyHHowOzs6YpiaGm1nZ/vuSJKO\n5bRSD2ZmRlNJw+EoGJxSkrTWGA49mZkxFCStXU4rSZIahoMkqWE4TDivt5C0GM85TDCvt5B0Ih45\nTDCvt5B0IhMXDk6jvMzrLSSdyERNKzmNciyvt5B0IhMVDotNo0z6L0Svt5C0mImaVnIaRZKWZ6KO\nHJxGkaTlmahwAKdRJGk5JmpaSZK0PIaDJKlhOEiSGoaDJKlhOEiSGoaDJKmRquq7hxVJcgj4Sd99\nnKbzgZ/33cQa4WdxLD+PY/l5vOx0P4s/q6rppQat23D4Q5BkvqoGffexFvhZHMvP41h+Hi87W5+F\n00qSpIbhIElqGA79urPvBtYQP4tj+Xkcy8/jZWfls/CcgySp4ZGDJKlhOPQgyV1JDiZ5qu9e+pbk\noiQPJ9mb5OkkN/XdU5+S/HGS7yX5n+7z+Ke+e+pbkqkkP0jyjb576VuSZ5M8meTxJPOr+lpOK519\nSf4K+A1wT1X9Zd/99CnJRmBjVX0/yZ8Ae4Brq+qHPbfWiyQBXl1Vv0nyCuC7wE1V9UjPrfUmyT8A\nA+C1VfXOvvvpU5JngUFVrfo1Hx459KCqvgMc7ruPtaCqDlTV97v7LwJ7gQv77ao/NfKb7sdXdLeJ\n/QsuySbgb4F/7buXSWM4aM1Ishl4C/Bov530q5tGeRw4COyqqkn+PP4Z+Efg9303skYU8O0ke5Ls\nWM0XMhy0JiR5DfBV4MNV9eu+++lTVR2tqkuATcBlSSZy6jHJO4GDVbWn717WkCuq6lLgKuDGbop6\nVRgO6l03t/5V4MtV9bW++1krquqXwBDY1nMrfbkC+Ltunv1e4O1J/q3flvpVVc9324PA14HLVuu1\nDAf1qjsB+0Vgb1V9tu9++pZkOsm53f1XAX8D/G+/XfWjqm6pqk1VtRm4Dvjvqvr7ntvqTZJXd4s2\nSPJq4Epg1VY8Gg49SPIVYA74iyT7k9zQd089ugJ4H6O/Ch/vblf33VSPNgIPJ3kCeIzROYeJX8Ip\nAC4Avpvkf4DvAf9VVd9arRdzKaskqeGRgySpYThIkhqGgySpYThIkhqGgySpYThIkhqGgySpYThI\nkhr/D9TSAWgtaRXKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c00e31c4a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(rating_frequencies.index,\n",
    "         rating_frequencies.values,\n",
    "         'b.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply value_counts() to a data frame will give an error\n",
    "# ratings_df.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the sub data frame: (6413, 11)\n",
      "2018.0    6413\n",
      "Name: Year, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# What are the rating frequencies in 2018?\n",
    "\n",
    "# Find the slice that correspond to 2018 ratings\n",
    "index_2018 = (ratings_df['Year'] == 2018)\n",
    "# print(index_2018)\n",
    "\n",
    "# Extract all 2018 ratings\n",
    "ratings_2018 = ratings_df[index_2018]\n",
    "ratings_2018.head()\n",
    "print(\"Shape of the sub data frame:\", ratings_2018.shape)\n",
    "print(ratings_2018['Year'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0    1345\n",
      "3.5     971\n",
      "3.0     857\n",
      "4.5     770\n",
      "5.0     763\n",
      "2.5     556\n",
      "2.0     521\n",
      "1.5     268\n",
      "0.5     182\n",
      "1.0     180\n",
      "Name: rating, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# find rating frequencies of the subset\n",
    "rating_freq_2018 = ratings_2018['rating'].value_counts()\n",
    "print(rating_freq_2018)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Relative Frequency')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHidJREFUeJzt3XuUHWWd7vHvQychaAC5BAiEmCCX\nEyIYoLn04SzTTARRFBzgKKgjnmEGFBEUc86B452egTh4AWZYCzJCG0VF8UZmFBgMNBjSQTqkRUjQ\nxHBJJx2ISYCEW5Od3/mjqkMTuntXk66u6uzns9Zee1ftqtq/3gvy7Pd9q95SRGBmZtafHYouwMzM\nys9hYWZmVTkszMysKoeFmZlV5bAwM7OqHBZmZlaVw8LMzKpyWJiZWVUOCzMzq2pE0QUMlj333DMm\nTpxYdBlmZsPKwoUL/xoRY6ttt92ExcSJE2lrayu6DDOzYUXSk1m2czeUmZlV5bAwM7OqHBZmZlbV\ndjNmYWY2EK+++iodHR28/PLLRZcyJEaPHs348eMZOXLkm9rfYWFmNamjo4Odd96ZiRMnIqnocnIV\nEaxdu5aOjg4mTZr0po7hbigzq0kvv/wye+yxx3YfFACS2GOPPbapFeWwMLNhpbUVrrwyed5WtRAU\n3bb1b3U3lJkNG62tMH06dHXBqFEwdy40NBRdVW1wy8LMho2WliQoKpXkuaWl6Iq2zYoVKzjhhBOY\nPHkyU6ZM4ZprrgFg3bp1nHjiiRx00EGceOKJrF+/HkjGHi666CIOPPBADj/8cB566CEAnnzySY46\n6iimTp3KlClTuP766we9VoeFmQ0bjY1Ji6KuLnlubCy6om0zYsQIvvWtb7FkyRIWLFjAddddx+LF\ni5k5cybTp09n6dKlTJ8+nZkzZwJw++23s3TpUpYuXcqsWbP49Kc/DcC4ceOYP38+7e3tPPDAA8yc\nOZNVq1YNaq0OCzMbNhoakq6npqZiuqAGc7wEkn/kjzzySAB23nlnJk+ezMqVK7nttts455xzADjn\nnHP41a9+BcBtt93GJz7xCSRx3HHH8eyzz9LZ2cmoUaPYcccdAXjllVfYvHnz4BTYg8cszGxYaWgo\nZpwi7/GSJ554gkWLFnHsscfy9NNPM27cOCAJlGeeeQaAlStXsv/++2/ZZ/z48axcuZJx48axYsUK\nTjnlFJYtW8ZVV13FvvvuO3jF4ZaFmVkmeY6XbNy4kTPOOIOrr76aXXbZpc/tIuIN67rPctp///15\n+OGHWbZsGbNnz+bpp58evAJxWJiZZZLXeMmrr77KGWecwcc+9jFOP/10APbee286OzsB6OzsZK+9\n9gKSlsSKFSu27NvR0fGGFsS+++7LlClT+N3vfjc4BaYcFmZmGeQxXhIRnHvuuUyePJlLLrlky/pT\nTz2V2bNnAzB79mxOO+20Leu///3vExEsWLCAXXfdlXHjxtHR0cFLL70EwPr167n//vs55JBDtr3A\nHnIds5B0MnANUAd8NyJmbvX+JcA/AJuANcDfR8ST6XsV4I/ppk9FxKl51mpmVs1gj5fcf//9/OAH\nP+Cwww5j6tSpAFxxxRVceumlfPjDH+bGG29kwoQJ3HrrrQC8//3v5ze/+Q0HHnggb3nLW2hubgZg\nyZIlfOELX0ASEcGMGTM47LDDBq9QQL31gQ3KgaU64M/AiUAH8CBwdkQs7rHNCcADEfGipE8DjRHx\nkfS9jRExJuvn1dfXh29+ZGZZLVmyhMmTJxddxpDq7W+WtDAi6qvtm2c31DHAsohYHhFdwC3AaT03\niIh7IuLFdHEBMD7HeszM7E3KMyz2A1b0WO5I1/XlXOD2HsujJbVJWiDpQ73tIOm8dJu2NWvWbHvF\nZmbWqzzHLHqbtarXPi9JHwfqgWk9Vk+IiFWSDgDulvTHiPjL6w4WMQuYBUk31OCUbWZmW8uzZdEB\n7N9jeTzwhuvPJb0H+CJwakS80r0+Ilalz8uBFuCIHGs1M7N+5BkWDwIHSZokaRRwFjCn5waSjgBu\nIAmKZ3qs303SjunrPYHjgcWYmVkhcuuGiohNki4E7iQ5dfamiHhU0uVAW0TMAa4CxgC3plchdp8i\nOxm4QdJmkkCb2fMsKjMzG1q5XpQXEb+JiIMj4h0R8c/puq+kQUFEvCci9o6Iqenj1HT9/Ig4LCLe\nlT7fmGedZmZFGOgU5Y899hgNDQ3suOOOfPOb33zdsb7zne8wZcoU3vnOd3L22WcP+r3FfQW3mVlG\nnRs6mfa9aazeuHpQjjfQKcp33313rr32WmbMmPG646xcuZJrr72WtrY2HnnkESqVCrfccsug1NjN\nYWFmllHTfU3Me2oeTfc2DcrxBjpF+V577cXRRx/NyJEj33CsTZs28dJLL7Fp0yZefPFFzzprZlaE\nzg2dNLc3szk209zePGiti25Zpijvy3777ceMGTOYMGEC48aNY9ddd+Wkk04a1PocFmZmGTTd18Tm\nSG4qVInKoLUuIPsU5X1Zv349t912G48//jirVq3ihRde4Oabbx60+sBhYWZWVXeroqvSBUBXpWvQ\nWhcDmaK8L7/97W+ZNGkSY8eOZeTIkZx++unMnz9/m2vryWFhZlZFz1ZFt8FoXQx0ivK+TJgwgQUL\nFvDiiy8SEcydO3fQJ0n0bVXNzKpo7Wjd0qro1lXpYn7Htv16H+gU5atXr6a+vp7nn3+eHXbYgauv\nvprFixdz7LHHcuaZZ3LkkUcyYsQIjjjiCM4777xtqm1ruU1RPtQ8RbmZDYSnKE+UYYpyMzPbTjgs\nzMysKoeFmdWs7aUbPott/VsdFmZWk0aPHs3atWtrIjAigrVr1zJ69Og3fQyfDWVmNWn8+PF0dHRQ\nK3fZHD16NOPHv/k7VzsszKwmjRw5kkmTJhVdxrDhbigzM6vKYWFmZlU5LMzMrCqHhZmZVeWwMDOz\nqhwWZmZWlcPCzMyqcliYmVlVDgszM6vKYWFmZlU5LMxsWOnc0Mm0700blPtfW3YOCzMbVprua2Le\nU/O2+f7XNjAOCzMbNjo3dNLc3szm2Exze7NbF0PIYWFmw0bTfU1sjs0AVKLi1sUQcliY2bDQ3aro\nqnQB0FXpcutiCDkszGxY6Nmq6ObWxdBxWJjZsNDa0bqlVdGtq9LF/I75BVVUW3K9U56kk4FrgDrg\nuxExc6v3LwH+AdgErAH+PiKeTN87B/hSuuk/RcTsPGs1s3JbdP6iokuoabm1LCTVAdcB7wMOBc6W\ndOhWmy0C6iPicOBnwL+k++4OfBU4FjgG+Kqk3fKq1czM+pdnN9QxwLKIWB4RXcAtwGk9N4iIeyLi\nxXRxAdB9N/H3AndFxLqIWA/cBZycY61mZtaPPMNiP2BFj+WOdF1fzgVuH8i+ks6T1Capbc2aNdtY\nrpmZ9SXPsFAv66LXDaWPA/XAVQPZNyJmRUR9RNSPHTv2TRdqZmb9qxoW6fjBm9EB7N9jeTywqpfj\nvwf4InBqRLwykH3NzGxoZGlZPCDpVknvl9TbL/6+PAgcJGmSpFHAWcCcnhtIOgK4gSQonunx1p3A\nSZJ2Swe2T0rXmZlZAbKExcHALODvgGWSrpB0cLWdImITcCHJP/JLgJ9GxKOSLpd0arrZVcAY4FZJ\n7ZLmpPuuA5pIAudB4PJ0nVlNam2FK69Mns2KoIhehxF631g6AbgZeCvwB+DSiCjFf7719fXR1tZW\ndBlmg661FaZPh64uGDUK5s6Fhoaiq7LthaSFEVFfbbssYxZ7SLpYUhswA/gssCfwBeBH21ypmfWr\npSUJikoleW5pKboiq0VZruBuBX4AfCgiOnqsb5N0fT5lmVm3xsakRdHdsmhsLLoiq0VZwuKQ6KOv\nKiK+Mcj1mNlWGhqSrqeWliQo3AVlRcgSFv8l6X9GxLMA6dlJt0TEe/Mtzcy6NTQ4JKxYWc6GGtsd\nFADp9Bt75VeSmZmVTZawqEia0L0g6e30cSW2mZltn7J0Q30RmCfp3nT53cB5+ZVkZmZlUzUsIuIO\nSUcCx5HM2fT5iPhr7pWZmVlpZL350Y7AunT7QyUREfflV5aZmZVJ1bCQ9A3gI8CjQPcNcANwWJiZ\n1YgsLYsPkVxr8UrVLc3MbLuU5Wyo5cDIvAsxM7PyytKyeBFolzQX2NK6iIiLcqvKzMxKJUtYzGGr\n+1CYmVltyXLq7GxJOwETIuJPQ1CTmZmVTJYpyj8ItAN3pMtTu29SZGZmtSHLAPfXgGOAZwEioh2Y\nlGNNZmZWMlnCYlNEPLfVOs8NZWZWQ7IMcD8i6aNAnaSDgIuA+fmWZWZmZZKlZfFZYArJabM/Bp4H\nPpdnUWZmVi5ZzoZ6kWTm2S/mX46ZmZVRlrmh7qGXMYqI+JtcKjIzs9LJMmYxo8fr0cAZwKZ8yjEz\nszLK0g21cKtV9/e4EZKZmdWALN1Qu/dY3AE4Ctgnt4rMzKx0snRDLSQZsxBJ99PjwLl5FmVmZuWS\npRvKV2ubmdW4LN1Qp/f3fkT8YvDKMTOzMsrSDXUu8N+Bu9PlE4AW4DmS7imHhZnZdi5LWARwaER0\nAkgaB1wXEf8r18rMzKw0skz3MbE7KFJPAwdnObikkyX9SdIySZf28v67JT0kaZOkM7d6ryKpPX14\nSnSzEujc0Mm0701j9cbVRZdiQyxLWLRIulPSJyWdA/wauKfaTpLqgOuA9wGHAmdLOnSrzZ4CPgn8\nqJdDvBQRU9PHqRnqNLOcNd3XxLyn5tF0b1PRpdgQqxoWEXEhcD3wLmAqMCsiPpvh2McAyyJieUR0\nAbcAp2117Cci4mFg84ArN7Mh1bmhk+b2ZjbHZprbm926qDFZWhYADwG/jojPA3dK2jnDPvsBK3os\nd6TrshotqU3SAkkfGsB+ZpaDpvua2BzJ77pKVNy6qDFZbqv6j8DPgBvSVfsBv8pwbPWybiA3TZoQ\nEfXAR4GrJb2jl9rOSwOlbc2aNQM4tJkNRHeroqvSBUBXpcutixqTpWXxGeB4kvtYEBFLgb0y7NcB\n7N9jeTywKmthEbEqfV5OcqruEb1sMysi6iOifuzYsVkPbWYD1LNV0c2ti9qSJSxeScccAJA0gmwt\nhAeBgyRNkjQKOAvIdFaTpN0k7Zi+3pMkrBZn2dfMBl9rR+uWVkW3rkoX8zt808xakeU6i3sl/T9g\nJ0knAhcA/1Ftp4jYJOlC4E6gDrgpIh6VdDnQFhFzJB0N/BLYDfigpK9HxBRgMnCDpM0kgTYzIhwW\nZgVZdP6iokuwgimi/0aCpB1IruI+iWQc4k7gu1FtxyFWX18fbW1tRZdhZjasSFqYjg/3q9+WRXqt\nxOyI+Djw74NVnJmZDS/9jllERAUYm445mJlZjcoyZvEEyd3x5gAvdK+MiG/nVZSZmZVLlrBYlT52\nALJcjGdmZtuZPsNC0oiI2BQRXx/KgszMrHz6G7P4ffcLSf86BLWYWR8826sVrb+w6Dldx/F5F2Jm\nffNsr1a0/sKiVNdRmNUqz/Zq/WlthSuvTJ7z1N8A93+T9DBJC+Md6WvS5YiIw/Mtzcyg99lerzvl\nuoKrsjJobYXp06GrC0aNgrlzoaEhn8/qLywm5/ORZpZVX7O9fnnal9lnzD4FV2dFa2lJgqJSSZ5b\nWvILiz67oSLiyf4e+ZRjZj15tlfrT2Nj0qKoq0ueGxvz+6ws11mYWUE826v1p6Eh6XpqaUmCIq9W\nBWSYSHC48ESCZmYDl3UiwUy3VZW0k6RDtr0sMzMbjrLcVvWDQDtwR7o8NZ0nyszMakSWlsXXgGOA\nZwEioh2YmF9JZlZWQ3VOv5VPlrDYFBHP5V6JmZVa9zn9X/5y8lzrgVFrU7BkCYtHJH0UqJN0UDpP\nlE/FMKsxvZ3TX8vKMgXLUIVWlrD4LDAFeAX4EfAc8Lk8izKz8hnKc/rLrkxTsAxVaGUJi0Mi4osR\ncXT6+FJEvJxrVWZWOt3n9Dc15TutxHDQ2xQsRRjK0MoSFt+W9JikJklTcqvEzEqvoQEuu6y2g6Kv\nKViKaF0MZWhVDYuIOAFoBNYAsyT9UdKXcqvIzKzEyjIFy1CHVqaL8iJidURcC3yK5JqLr+RSjZlZ\nyZVlCpahDq2qc0NJmgx8BDgTWAvcAnwhl2rMzEpu0fmLii4BGPrQyjKRYDPwY+CkiFiVSxVmZjYg\nQx1aVcMiIo4bikLMzKy8+gwLST+NiA9L+iOvv8Wq75RnZlZj+mtZXJw+f2AoCjEzs/Lq7055nenL\nC3q5S94FQ1OemZmVQZZTZ0/sZd37BrsQMzMrr/7GLD5N0oI4QNLDPd7aGbg/78LMzKw8+mtZ/Aj4\nIDAnfe5+HBURH89ycEknS/qTpGWSLu3l/XdLekjSJklnbvXeOZKWpo9zMv9FZmY26PpsWaT3sHgO\nOBtA0l7AaGCMpDER8VR/B5ZUB1xH0o3VATwoaU5ELO6x2VPAJ4EZW+27O/BVoJ7kTKyF6b7rB/bn\nmZnZYMh0W1VJS4HHgXuBJ4DbMxz7GGBZRCyPiC6SK79P67lBRDwREQ8Dm7fa973AXRGxLg2Iu4CT\nM3ymmZnlIMsA9z8BxwF/johJwHSyjVnsB6zosdyRrstiW/Y1M7NBliUsXo2ItcAOknaIiHuAqRn2\nUy/ropd1b3pfSedJapPUtmbNmoyHNjOzgcoSFs9KGgPcB/xQ0jXApgz7dQD791geD2SdWyrTvhEx\nKyLqI6J+7NixGQ9tZmYDlSUsTgNeAj4P3AH8heSsqGoeBA6SNEnSKOAskjOrsrgTOEnSbpJ2A05K\n15mZWQGyTCT4Qo/F2VkPHBGbJF1I8o98HXBTRDwq6XKgLSLmSDoa+CWwG/BBSV+PiCkRsU5SE0ng\nAFweEeuyfraZmQ0uRfQ+jCBpA71MIMhrEwnukn952dXX10dbW1vRZdggaW2FlhZobKztW3ia5U3S\nwoior7Zdf9dZ7Dy4JZll09oK06dDVxeMGgVz5zowrHxq7QdNlpsfIel/AAdFRLOkPYGdI+LxfEuz\nWtXSkgRFpZI8t7TUxv+MNnzU4g+aLBflfRX4v8Bl6apRwM15FmW1rbEx+R+wri55bmwsrpbODZ1M\n+940Vm9cXVwRVjq9/aDZ3mU5G+pvgVOBFwDSW6u6i8py09CQ/FJrair+F1vTfU3Me2oeTfc2FVeE\nlU6ZftAMlT4HuLdsIP0+Io6R9FBEHCnprUBr2e6U5wFuG2ydGzo54NoDeHnTy+w0YieWX7ycfcbs\nU3RZVhLby5hF1gHuLC2Ln0q6AXibpH8Efgt8d1sLNCu7pvua2BzJtGWVqLh1Ya/T0ACXXTa8g2Ig\nqoZFRHwT+Bnwc+AQ4CsRcW3ehZkVqXNDJ83tzXRVugDoqnTR3N7ssQurWVlaFkTEXRHxvyNiBnC3\npI/lXJfVuKIHlnu2Krq5dWG1rM+wkLSLpMsk/Zukk5S4EFgOfHjoSrRaVPTAcmtH65ZWRbeuShfz\nO+YXUo9Z0fq7gvs2YD3QSjIt+W4kp81eHBHtQ1ZhRh7g3n54YNls6AzGAPcBEfHJiLiB5G559cAH\nyhgUtn3xwLJZ+fQXFq92v4iICvB4RGzIvySrZR5YNiun/sLiXZKeTx8bgMO7X0t6fqgKtNrigWWz\ncupvIsG6oSzEDDywbFZWmSYSNBsqi85fVHQJZtaLTNdZmJlZbXNYmJlZVQ4LMzOrymFhZmZVOSzM\nzKwqh4WZmVXlsDAzs6ocFmZmVpXDwszMqnJYmJlZVQ4LMzOrymFhZmZVOSzMzKwqh4WZmVXlsDAz\ns6ocFmZmVlWuYSHpZEl/krRM0qW9vL+jpJ+k7z8gaWK6fqKklyS1p4/r86zTzMz6l9ud8iTVAdcB\nJwIdwIOS5kTE4h6bnQusj4gDJZ0FfAP4SPreXyJial71mZlZdnm2LI4BlkXE8ojoAm4BTttqm9OA\n2enrnwHTJSnHmswGpLUVrrwyeTarZXneg3s/YEWP5Q7g2L62iYhNkp4D9kjfmyRpEfA88KWI+F2O\ntZq9QWsrTJ8OXV0wahTMnQsNDUVXZVaMPFsWvbUQIuM2ncCEiDgCuAT4kaRd3vAB0nmS2iS1rVmz\nZpsLNuuppSUJikoleW5pKbois+LkGRYdwP49lscDq/raRtIIYFdgXUS8EhFrASJiIfAX4OCtPyAi\nZkVEfUTUjx07Noc/wWpZY2PSoqirS54bG4uuyKw4eXZDPQgcJGkSsBI4C/joVtvMAc4BWoEzgbsj\nIiSNJQmNiqQDgIOA5TnWavYGDQ1J11NLSxIU7oKyWpZbWKRjEBcCdwJ1wE0R8aiky4G2iJgD3Aj8\nQNIyYB1JoAC8G7hc0iagAnwqItblVau9pnNDJ2f9/Cx+cuZP2GfMPkWXU7iGBoeEGYAith5GGJ7q\n6+ujra2t6DKGvQt+fQE3LLyBTx31Ka475bqiyzGznElaGBH11bbzFdy2ReeGTprbm9kcm2lub2b1\nxtVFl2RmJeGwsC2a7mtic2wGoBIVmu5tKrgiMysLh4UBr7UquipdAHRVuty6MLMtHBYGvL5V0c2t\nCzPr5rAwAFo7Wre0Krp1VbqY3zG/oIrMrEzyvM7ChpFF5y8qugQzKzG3LMzMrCqHhZmZVeWwMDOz\nqhwWJKeNTvvetEJPEy1DDWZmfXFYkJw2Ou+peYWeJlqGGszM+lLzYVGGKS7KUIOZWX9qPizKMMVF\nGWowM+tPTYdFGaa4KEMNZmbV1HRYlGGKizLUYGZWTU2HRRmmuChDDWZm1dT0dB9lmOJi0fmLaG2F\n6dOhqyu51/PcucXdna211bcRNbM3qumwKIuWliQoKpXkuaWlmH+oyxRaZlYuNd0NVRaNjck/znV1\nyXNjYzF19BZaZmbglkUpNDQkv+KL7v7pDq3ulkVRoWVm5eOwKImGhuK7fMoSWmZWPg4LPKjbUxlC\ny8zKp+bDwoO6ZmbV1fwAtwd1zcyqq/mwKMuZSGZmZVbz3VAe1DUzq67mwwI8qGtmVk3Nd0OZmVl1\nDgszM6vKYWFmZlU5LMzMrCqHhZmZVeWwMDOzqhQRRdcwKCStAZ4suo5ttCfw16KLKBF/H6/n7+M1\n/i5eb1u+j7dHxNhqG203YbE9kNQWEfVF11EW/j5ez9/Ha/xdvN5QfB/uhjIzs6ocFmZmVpXDolxm\nFV1Ayfj7eD1/H6/xd/F6uX8fHrMwM7Oq3LIwM7OqHBYlIOkmSc9IeqToWspA0v6S7pG0RNKjki4u\nuqaiSBot6feS/pB+F18vuqYykFQnaZGk/yy6lqJJekLSHyW1S2rL7XPcDVU8Se8GNgLfj4h3Fl1P\n0SSNA8ZFxEOSdgYWAh+KiMUFlzbkJAl4a0RslDQSmAdcHBELCi6tUJIuAeqBXSLiA0XXUyRJTwD1\nEZHrdSduWZRARNwHrCu6jrKIiM6IeCh9vQFYAuxXbFXFiMTGdHFk+qjpX3iSxgOnAN8tupZa4rCw\nUpM0ETgCeKDYSoqTdrm0A88Ad0VEzX4XqauB/wNsLrqQkgjgvyQtlHReXh/isLDSkjQG+DnwuYh4\nvuh6ihIRlYiYCowHjpFUs12Vkj4APBMRC4uupUSOj4gjgfcBn0m7tQedw8JKKe2f/znww4j4RdH1\nlEFEPAu0ACcXXEqRjgdOTfvpbwH+RtLNxZZUrIhYlT4/A/wSOCaPz3FYWOmkg7o3Aksi4ttF11Mk\nSWMlvS19vRPwHuCxYqsqTkRcFhHjI2IicBZwd0R8vOCyCiPprelJIEh6K3ASkMtZlQ6LEpD0Y6AV\nOERSh6Rzi66pYMcDf0fyq7E9fby/6KIKMg64R9LDwIMkYxY1f7qobbE3ME/SH4DfA7+OiDvy+CCf\nOmtmZlW5ZWFmZlU5LMzMrCqHhZmZVeWwMDOzqhwWZmZWlcPCLCNJlfQ03kck/Uf39Q/9bP82SRf0\nWN5X0s/yr9Rs8PnUWbOMJG2MiDHp69nAnyPin/vZfiLwn55J2LYHblmYvTmtpDPhShojaa6kh9L7\nCpyWbjMTeEfaGrlK0sTue5ZI+qSkX0i6Q9JSSf/SfWBJ50r6s6QWSf8u6d+G/K8z28qIogswG24k\n1QHTSaYkAXgZ+NuIeF7SnsACSXOAS4F3ppMAdrc0eppKMqPuK8CfJP0rUAG+DBwJbADuBv6Q6x9k\nloHDwiy7ndKpwieS3JDprnS9gCvS2T43k7Q49s5wvLkR8RyApMXA24E9gXsjYl26/lbg4MH8I8ze\nDHdDmWX3UtpKeDswCvhMuv5jwFjgqPT9p4HRGY73So/XFZIfbxq8cs0Gj8PCbIDS1sBFwIx0KvVd\nSe6x8KqkE0jCBJJupJ0HePjfA9Mk7SZpBHDGYNVtti0cFmZvQkQsIhlLOAv4IVAvqY2klfFYus1a\n4P70VNurMh53JXAFyZ0BfwssBp4b/L/AbGB86qxZyUgaExEb05bFL4GbIuKXRddltc0tC7Py+Vo6\nkP4I8Djwq4LrMXPLwszMqnPLwszMqnJYmJlZVQ4LMzOrymFhZmZVOSzMzKwqh4WZmVX1/wHrLWCl\ndZ8bEQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c00e621898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What are the rating frequencies in 2003?\n",
    "rating_freq_2003 = ratings_df[ratings_df['Year'] == 2003]['rating'].value_counts()\n",
    "plt.plot(rating_freq_2003 / np.sum(rating_freq_2003), 'b.', label=\"2003\")\n",
    "plt.plot(rating_freq_2018 / np.sum(rating_freq_2018), 'g^', label=\"2018\")\n",
    "plt.legend()\n",
    "plt.xlabel(\"Rating\")\n",
    "plt.ylabel(\"Relative Frequency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### B. How consistant are the average ratings over time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# How consistent are the average ratings over time?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### C. How fast does the movie library grow?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.3891314517386557\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0, 5)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFwVJREFUeJzt3XuYHFWdxvH3lxskMUogEwSSFRBc\nQFcEJiAERUQuG0B0QYUsmIdFs+sKonthUVS8gCC6PBIeXS4CK0RQlFWULGggZjEil0lICCFcQwIh\nIZMI5sKGhMm8+8ep2ZlkbjUz3TNzJt/P8/Qz3dXVVafOVL996tSp7rAtAEA+BvV1AQAAXUNwA0Bm\nCG4AyAzBDQCZIbgBIDMENwBkZkiZmSJiqaT1krZIarBdW81CAQDaVyq4C0fbXlO1kgAASqGrBAAy\nE2WunIyI5yW9KsmSrrV9XRvzTJU0VZJGjhx5yH777VfhogLAwDV37tw1tmvKzFs2uHe3vSIixkqa\nKek82/e3N39tba3r6upKFxgAtncRMbfs+cNSXSW2VxR/6yX9QtKh3S8eAKAnOg3uiBgZEaOa7ks6\nTtLj1S4YAKBtZUaV7CrpFxHRNP+ttu+paqkAAO3qNLhtL5F0YC+UBQBQAsMBASAzBDcAZIbgBoDM\nENwAkBmCGwAyQ3ADQGYIbgDIDMENAJkhuAEgMwQ3AGSG4AaAzBDcAJAZghsAMkNwA0BmCG4AyAzB\nDQCZIbgBIDMENwBkhuAGgMwQ3ACQGYIbADJDcANAZghuAMgMwQ0AmSG4ASAzBDcAZIbgBoDMENwA\nkBmCGwAyQ3ADQGYIbgDIDMENAJkhuAEgMwQ3AGSG4AaAzJQO7ogYHBGPRsRd1SwQAKBjXWlxny9p\ncbUKAgAop1RwR8Q4SSdK+mF1iwMA6EzZFvf3JF0gqbG9GSJiakTURUTd6tWrK1I4AEBrnQZ3RJwk\nqd723I7ms32d7VrbtTU1NRUrIABga2Va3BMlfTgilkr6iaQPRsT0qpYKANCuToPb9hdtj7O9p6TT\nJc2yfWbVSwYAaBPjuAF0iS3ddps0Y0Zfl2T71aXgtj3b9knVKgyA/m3dOumMM6TJk6U1a/q6NNuv\nIX1dAAB5mDtX+sQnpKVLpcsuk846K03fuFEaPrxPi7bdoasEQKdmzJAOP1zavFn6n/+RLrxQGjRI\nuvlm6YADpJde6usSbl8IbmAAWLZMmjZNqq+vzvKPOEI6+2zp0UeliRObpx94YOoyOekkaf366qwb\nrRHc3WBLXGOE/uL556X3v186/3zpwQfTNLvny/3jH6VTT02t7NGjpWuvlXbZZet5DjxQ+tnPpIUL\nUzdKQ0PP14vODYjg3rKld9bz8MPp77nnSkceKb36au+sF2jP0qXS0Uen1u4vf5lavpL01a9KJ54o\nzZrV9RBvbJS+/W3pfe9LLezlyzue/4QTpB/8QLr7bum88yrzoYGODYjgPv546aKLqrvDXHutdNhh\n0k9/Kp1+emrlfPzj0htvVG+dQEdefDGF9tq10r33SqeckvqdJWnMGKmuTjrmGOngg6Xp01PLuTP1\n9dKkSakP+6MfTcG9996dv27q1PSa8eN7tk0oyXbFb4cccoir7de/tletSvcvv9yW7E9/2m5oqPy6\npk+3I+yTTrI3b07TbroprfMf/7Hy6wPKePVV+/jj7bq6tp/fuNG+/np7//3TvvqpT3W+zA99yN5h\nB/s//sNubOx+2TZu7P5rK6Wx0X7ssZ5tR2+SVOeSGZtdcC9ZYp98cir5F7+YpjU22l/+cpr2N39T\n2Z3ml7+0Bw+2jz7a/t//3fq5f/3XtM6rr67c+tD/vPFGX5dgay+91Hpf7MiWLfZ//7e9YEF6/OST\n9uc+l95LdmrsNC1v0SJ7/vyele/hh+3ddrPnzOnZcirh5JPtf/u3vi5FOQMyuDdutL/+dXvHHe2R\nI+3vfMfetGnrea66Km3R8cdX5lN25Up7+HD7sMPsdetaP9/QYE+ZYs+a1fN1of/6+79PR1tr1/Z1\nSewXX7Tf/nb71FO7v4ybbrKHDLEHDbJPOy01SqZMqVQJ7dWr7X33tXfZxX766cott4yHHkpHDU8/\nnTLg/PNTJnzrW71bju4YkMH9uc+l0n7iE2nnbc/06fZPflK59f7qV/af/lRu3tdeq9x60X/84Q9p\n3/urv7KXLeu7cixfbu+zjz1qlP3ggz1f1oUX2jvtZI8YkcK8kp55JgX3PvukIK+2Z56xP/ax9H+q\nqbHvuSdN37LFnjw5jyPjARPczz+f/iF22tFmzuza6++8My2jq+bNa/7Hl3XVVWknXbOm6+urpKee\nsq+4wj72WPvss+27727ul0d5a9faZ51lr1iRHv/2t/Zb3mLvumtq1fW25ctTK3bUKPuBByq33Nde\na/toshL+8IfUX37EEdXt8/7CF9IRxMiR9sUXt96ezZvtD384pd306dUrR09lH9yvv25fcknqppg0\nqXvL2LAhvcl2281euLD86554wh4zJoVwVwLvgQfSTnrUUa27cKpp06bUVdPUNXT22em/+s532m9+\nc7p/2GHN8+dyoqYvbdyYug+GDNm6sbBokb3XXqm7rjsNgu5qbLQnTrTf9KYUhjm5/Xb79NMrH9wt\nl/dP/2R/5jOpa7Oj+c84o7mfvz/KOrjvuSe1LKTU//bCC91elBcutHffPR0OljlRsmSJvcceKfC7\n0zc3fXoq9znnVDcgV61Kh7annppaYFI6e27bzz3XXGcbN6aTq7ffnh5v2mTvt1/ayX/3u+qMwMld\nQ0Oq1/ZaZ/X19g9+0PvlWriwf5zs64lKhPfmzan+d9017cN2199rjY3NJ2b7k2yD+5ZbUon23df+\nzW+6tYhWnn8+LW/4cPuuu9qf76WX7L33tkePbg7B7rjoorQNV17Z/WVsq7ExHYXY9n33paGJUvpQ\nmjo19cOXGWXw8sv2xz+e+jQl+61vtc89N3WvINXzP/xD+f9fXV06cVmtI6yVK+3vfW9gHCW9/HIa\nlnjNNd17fWOj/fOf2+94R/r/vO99qUuzO6ZNS++B/nb0km1wb9iQ3jBNIVUpq1bZBx/cPHywLd/4\nRjoU7Wn/5ZYt6dDw+9/v2XIaG+25c9NJ2fHj7UsvTdPXrbO/9rX0XHff0Bs22D/9aRo6ueOO9u9/\nn6Y//XRq1T35ZGrhzZ27dX089lj6kLjjDvu22+ybb04ftk1Wr+5/Q+e64tVXU7hccEG5+a+8Mr2D\njjqq8uc2moJuxIj+2TrsqjfeSN2egwenLovJk+0zz2x+/qqr0tC9E0+0TzjBPu64tH82mTo11fUB\nB6R9sCcfZi+/nD4A3vKW7od/NWQb3NX02mvN/+z6+tbPNzZWbuhSy52qO90R06bZ7353+u/ssIP9\nkY90fLTQE+vWpQ8b2z7vvLTObW9N2/PpT7d+bsSI5mWddlo6QXTMMekk0cyZlTvx9frraXzxzTen\n8fMXX5yCttLWru1aKEyfbg8blo7qKrX/rFqVAmrECHv27Mossz9Yvz4Nq9xrrzSkcf/9m5/7ylfs\n97zHPuQQe8IE+73vTcN6m1xyiX3DDZXr3lu2LDWIamrsxYsrs8yeIrg78OKL9tix9pe+lFqekyc3\nj1yptBkz7He9q/kKz/Zs2mTff3/z40mT7EMPTX15r7xSnbK15c9/Tq3pW2+1f/az1D8+Y0ZzsC9Z\nYj/ySArQJ55I9dZyeNxdd6XwP+igNEZYsmtrm5+fPbvjE0h2Cs1ly9KVsd/6Vuqzt+0bb2z+sBg2\nLHUX7bJL53Vbxs9/no6SunukN2dOOqE9enTPQ2D+/HRiefjw5j5cVMdTT6UseNvbunZBU7UQ3B1o\naGg+7NpjjxQwd9xRnXU98kjqipg4se1QmD/f/vzn05s+ojkE+8NO1FNr16bzFDNmpMebNzf3rb/9\n7emCj+uvT4etdgq8I49Mh68tW/RNJ1ZfeCF1zyxalJb16KP2N7/ZvL7HH+/e4fOsWemD4IgjejYO\n/7nn7M9+tnxXUUND2oarr07XJjR1V82cmfaH++7rfllQ3oIFqZHSHxDcnWhsTCcRBw2yf/jD6q7r\n9ttTLX/yk83BMm9eapVK9tChqYvhrrvy7h/uTENDumjku99NXT9jxqTtv+yy9PzKlekD7jOfSd+T\nMWdOOgIo46mnUt/pMcd07XLtefPSqJx3vrP8RVZlrFiRvj+n6UilpVWrUh9u01BNyR43rvkDqqGB\n0T595b77+vY6DIK7pA0bemc93/hGqulLLkmPX345dYVcfXXfX7DTVxob00nQnozgabJ5czovsPPO\n6cjlU5/qvEvm2WfTkLLx4zu+Erc7rrgi/b8/+tHUH3/44WmssZ0+nCdMSKNXpk+3ly6t7LrRPWvW\npMEJEyZ0/bzM5s1pf7r33p4NrCC4+5nGxtQ10PIsOSrvlVdSQA4dmkJ8/fr2533ggXSRVTVOTDU2\n2v/+7+lDZOjQ1A0zbVrl14PKuvPOdOR21FFbd1c2NKQP2Nmz0/UTF1+c3s9NI1Kajqql1JXXXV0J\n7kjzV1Ztba3r6uoqvlygjGefTb/e0vRjtrNmSR/4QPqu6oYGaUjxE9kt71dDfb00ahQ/pJuTW2+V\nzjwzfYf5ZZdJxx6bflXo8MOb54mQxo2TrrkmfXf58uXSzJnSXntJEyZII0d2b90RMdd2bZl5+ZV3\nDDj77JNukjRnTvoxgUMPlS6/XLrkkvTDGxdcUN3QlqSxY6u7fFTe5MnS669Ll14qvfBCmrb//tL1\n10t77pnCefx4adiw5teMG5d+j7M30eLGgNbYKN1yi/SlL0krVqRpt9ySWlVAf0KLGygMGiRNmSKd\ndpp09dXS7rsT2sgfwY3twsiR6TcRgYFgQPxYMABsTwhuAMgMwQ0AmSG4ASAzBDcAZIbgBoDMENwA\nkBmCGwAy02lwR8SOEfFwRCyIiEUR8fXeKBgAoG1lrpzcJOmDtjdExFBJcyLibtsPVrlsAIA2dBrc\nxffEbigeDi1ulf9mKgBAKaX6uCNicETMl1Qvaabth9qYZ2pE1EVE3erVqytdTgBAoVRw295i+z2S\nxkk6NCLe1cY819mutV1bU1NT6XICAApdGlVi+8+SZks6oSqlAQB0qsyokpqI2Km4P1zShyQ9We2C\nAQDaVmZUyW6SfhQRg5WC/nbbd1W3WACA9pQZVfKYpIN6oSwAgBK4chIAMkNwA0BmCG4AyAzBDQCZ\nIbgBIDMENwBkhuAGgMwQ3ACQGYIbADJDcANAZghuAMgMwQ0AmSG4ASAzBDcAZIbgBoDMENwAkBmC\nGwAyQ3ADQGYIbgDIDMENAJkhuAEgMwQ3AGSG4AaAzBDcAJAZghsAMkNwA0BmCG4AyAzBDQCZIbgB\nIDMENwBkhuAGgMwQ3ACQGYIbADJDcANAZghuAMhMp8EdEeMj4ncRsTgiFkXE+b1RMABA24aUmKdB\n0j/bnhcRoyTNjYiZtp+octkAAG3otMVte6XtecX99ZIWS9qj2gUDALStS33cEbGnpIMkPdTGc1Mj\noi4i6lavXl2Z0gEAWikd3BHxJkl3SPq87XXbPm/7Otu1tmtramoqWUYAQAulgjsihiqF9o9t/1d1\niwQA6EiZUSUh6QZJi21fWf0iAQA6UqbFPVHSWZI+GBHzi9ukKpcLANCOTocD2p4jKXqhLACAErhy\nEgAyQ3ADQGYIbgDIDMENAJkhuAEgMwQ3AGSG4AaAzBDcAJAZghsAMkNwA0BmCG4AyAzBDQCZIbgB\nIDMENwBkhuAGgMwQ3ACQGYIbADJDcANAZghuAMgMwQ0AmSG4ASAzBDcAZIbgBoDMENwAkBmCGwAy\nQ3ADQGYIbgDIDMENAJkhuAEgMwQ3AGSG4AaAzBDcAJAZghsAMkNwA0BmOg3uiLgxIuoj4vHeKBAA\noGNlWtz/KemEKpcDAFBSp8Ft+35Jr/RCWQAAJVSsjzsipkZEXUTUrV69ulKLBQBso2LBbfs627W2\na2tqaiq1WADANhhVAgCZIbgBIDNlhgPeJumPkv4yIpZHxDnVLxYAoD1DOpvB9hm9URAAQDl0lQBA\nZghuAMgMwQ0AmSG4ASAzBDcAZIbgBoDMENwAkBmCGwAyQ3ADQGYIbgDIDMENAJkhuAEgMwQ3AGSG\n4AaAzBDcAJAZghsAMkNwA0BmCG4AyAzBDQCZIbgBIDMENwBkhuAGgMwQ3ACQGYIbADJDcANAZghu\nAMgMwQ0AmSG4ASAzBDcAZIbgBoDMENwAkBmCGwAyQ3ADQGYIbgDIDMENAJkpFdwRcUJEPBURz0bE\nhdUuFACgfZ0Gd0QMlvR9SX8t6QBJZ0TEAdUuGACgbWVa3IdKetb2EtubJf1E0inVLRYAoD1DSsyz\nh6QXWzxeLumwbWeKiKmSphYPN0TEnySt6XEJB5Yxok62RZ20Rp20baDXy9vKzlgmuKONaW41wb5O\n0nX//6KIOtu1ZQuyPaBOWqNOWqNO2ka9NCvTVbJc0vgWj8dJWlGd4gAAOlMmuB+RtG9E7BURwySd\nLulX1S0WAKA9nXaV2G6IiHMl/UbSYEk32l5UYtnXdT7Ldoc6aY06aY06aRv1Ugi7VXc1AKAf48pJ\nAMgMwQ0AmSkd3BFxY0TUR8TjLaYdGBF/jIiFEfHriHhzMX1YRNxUTF8QER9o8ZrZxeXz84vb2Ipu\nUS+KiPER8buIWBwRiyLi/GL6zhExMyKeKf6OLqZHREwrvjrgsYg4uMWyphTzPxMRU/pqm3qqwnWy\npcV+kvUJ8W7Uy37Fe2tTRPzLNssaEF9BUeE6WVrkzfyIqOuL7elVtkvdJL1f0sGSHm8x7RFJRxX3\n/07SN4v7n5V0U3F/rKS5kgYVj2dLqi273v58k7SbpIOL+6MkPa30tQBXSLqwmH6hpG8X9ydJultp\nbPx7JT1UTN9Z0pLi7+ji/ui+3r6+rJPiuQ19vT19WC9jJU2QdKmkf2mxnMGSnpO0t6RhkhZIOqCv\nt68v66R4bqmkMX29Tb11K93itn2/pFe2mfyXku4v7s+UdGpx/wBJ9xWvq5f0Z0kDbuC87ZW25xX3\n10tarHSl6SmSflTM9iNJHynunyLpZicPStopInaTdLykmbZfsf2qUl2e0IubUjEVrJMBpav1Yrve\n9iOS3thmUQPmKygqWCfbnZ72cT8u6cPF/Y+p+UKdBZJOiYghEbGXpEO09UU8NxWHNF+JiLauzMxO\nROwp6SBJD0na1fZKKe2cSi0Fqe2vD9ijg+lZ62GdSNKOEVEXEQ9GxEc0QJSsl/Zsz/tKRyzptxEx\nt/j6jQGtzCXvHfk7SdMi4qtKF+VsLqbfKGl/SXWSlkl6QFJD8dzf2n4pIkZJukPSWZJu7mE5+lRE\nvElpWz5ve10Hn0XtfX1Aqa8VyEkF6kSS/sL2iojYW9KsiFho+7kqFLfXdKFe2l1EG9O2l32lIxOL\nfWWspJkR8WTRSzAg9ajFbftJ28fZPkTSbUp9b7LdYPsLtt9j+xRJO0l6pnjupeLvekm3Kh36ZSsi\nhirtdD+2/V/F5FVNh/vF3/pientfHzCgvlagQnUi201/lyidGzmo6oWvoi7WS3u2532lXS32lXpJ\nv1DmudKZHgV304iQiBgk6cuSrikej4iIkcX9YyU12H6i6DoZU0wfKukkpe6WLBXdPDdIWmz7yhZP\n/UpS08iQKZLubDH9k8VIivdKWlscCv5G0nERMbo4g35cMS07laqToi52KJY5RtJESU/0ykZUQTfq\npT0D5isoKlUnETGyOIJXkTvHKeNcKaXsWUylFvVKpRMDyyWdI+l8pTPBT0u6XM1XYu4p6Smlkw33\nSnpbMX2k0giTxyQtknSVpMF9fYa2uzdJRyodpj4maX5xmyRpF6WTs88Uf3cu5g+lH6V4TtJCtRhd\no9Tt9GxxO7uvt62v60TSEcXjBcXfc/p623q5Xt5avM/WKZ3cXy7pzcVzk4r33HOSLurrbevrOlEa\nYbOguC3KuU7K3rjkHQAyw5WTAJAZghsAMkNwA0BmCG4AyAzBDQCZIbgBIDMENwBk5v8AvjxaGtti\nJlMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c00e325208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# How many movies were rated in 2003? in 2018?\n",
    "\n",
    "# Example: average rating of 2018\n",
    "index_2018 = (ratings_df['Year'] == 2018)\n",
    "ratings_2018 = ratings_df[index_2018]\n",
    "avg_rating = np.mean(ratings_2018['rating'])\n",
    "print(avg_rating)\n",
    "\n",
    "# Get the set of all years in the dataset\n",
    "# set(ratings_df['Year'])\n",
    "years = np.sort(ratings_df['Year'].unique())\n",
    "all_avg_ratings = []\n",
    "for year in years:\n",
    "    index_year = (ratings_df['Year'] == year)\n",
    "    ratings_year = ratings_df[index_year]\n",
    "    avg_rating = np.mean(ratings_year['rating'])\n",
    "    all_avg_ratings.append(avg_rating)\n",
    "plt.plot(years, all_avg_ratings, 'b--')\n",
    "plt.ylim([0, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Do number of movies increase over time?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### D. How fast does the user base grow?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# How many ratings were submitted per year?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# How many users gave ratings per year?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### E. Popular Movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Which movie received the highest number of ratings?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Which movie received the highest number of 5.0?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Which five movies have the highest average rating?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### F. Time of ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>Hour</th>\n",
       "      <th>Minute</th>\n",
       "      <th>Second</th>\n",
       "      <th>DT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2000-07-30 14:45:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:20:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:37:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>47.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 15:03:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000-07-30 14:48:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating    timestamp    Year  Month   Day  Hour  Minute  \\\n",
       "0     1.0      1.0     4.0  964982703.0  2000.0    7.0  30.0  14.0    45.0   \n",
       "1     1.0      3.0     4.0  964981247.0  2000.0    7.0  30.0   NaN     NaN   \n",
       "2     1.0      6.0     4.0  964982224.0  2000.0    7.0  30.0   NaN     NaN   \n",
       "3     1.0     47.0     5.0  964983815.0  2000.0    7.0  30.0   NaN     NaN   \n",
       "4     1.0     50.0     5.0  964982931.0  2000.0    7.0  30.0   NaN     NaN   \n",
       "\n",
       "   Second                  DT  \n",
       "0     3.0 2000-07-30 14:45:03  \n",
       "1     NaN 2000-07-30 14:20:47  \n",
       "2     NaN 2000-07-30 14:37:04  \n",
       "3     NaN 2000-07-30 15:03:35  \n",
       "4     NaN 2000-07-30 14:48:51  "
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('1996-03-29 14:36:55')"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# When was the first rating given?\n",
    "\n",
    "ratings_df['DT'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2018-09-24 10:27:30')"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# When was the last rating given?\n",
    "ratings_df['DT'].max()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# During what time of day do most users give ratings?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Week 4 Homework: \n",
    "\n",
    "Create a new Jupyter notebook and load ratings.csv as a pandas data frame.\n",
    "\n",
    "For movie (ID 260: Star Wars: A New Hope) and (ID 9645: Star Wars: The Last Jedi), find:\n",
    "\n",
    "1. how many ratings were given to this movie\n",
    "2. What is the highest and lowest rating\n",
    "3. When was the first rating given?\n",
    "4. When was the last rating given?\n",
    "5. What is the average rating?\n",
    "6. Plot the frequency distribution of the ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}