third_model_attempt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning\n",
    "\n",
    "First, let's load and mop up the data a little bit. We want to drop anything where the headline is `NaN`. Fortunately, there aren't too many of those.\n",
    "\n",
    "Secondly, a lot of the headline categories have many sub-categories. We end up with a ridiculously long list of category labels this way. Each of these likely only has so many headlines, and it's far more detailed than I'm concerned with here anyhow. I'll simplify those to be just the top-level categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>publish_date</th>\n",
       "      <th>headline_category</th>\n",
       "      <th>headline_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19960102</td>\n",
       "      <td>business</td>\n",
       "      <td>Smurfit's share price in retreat despite recor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19960102</td>\n",
       "      <td>business</td>\n",
       "      <td>Jamont plans £5m investment to update plant</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19960102</td>\n",
       "      <td>business</td>\n",
       "      <td>Management is blamed for most company failures</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19960102</td>\n",
       "      <td>business</td>\n",
       "      <td>Forte expected to announce a special dividend ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19960102</td>\n",
       "      <td>business</td>\n",
       "      <td>Accountancy firm adopts name change</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   publish_date headline_category  \\\n",
       "0      19960102          business   \n",
       "1      19960102          business   \n",
       "2      19960102          business   \n",
       "3      19960102          business   \n",
       "4      19960102          business   \n",
       "\n",
       "                                       headline_text  \n",
       "0  Smurfit's share price in retreat despite recor...  \n",
       "1        Jamont plans £5m investment to update plant  \n",
       "2     Management is blamed for most company failures  \n",
       "3  Forte expected to announce a special dividend ...  \n",
       "4                Accountancy firm adopts name change  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('data/irishtimes-date-text.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove blank headlines\n",
    "df.dropna(axis='index', how='any', inplace=True)\n",
    "\n",
    "# Simplify the category names\n",
    "df['simple_category'] = df.headline_category.str.split('.').str.get(0)\n",
    "\n",
    "# Drop the over-represented 'news' category\n",
    "df = df.loc[df['simple_category'] != 'news']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepping the Model\n",
    "\n",
    "We'll create a small pipeline here with a bag-of-words model that we can categorize with.\n",
    "\n",
    "We'll get a list of the unique category names and then add the labels. Then we'll create a variable with the headline text for predicting and a `dict` of labels, only of which will be `True` for a given headline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# Create the model\n",
    "nlp = spacy.blank('en')\n",
    "\n",
    "categorizer = nlp.create_pipe('textcat', config={'exclusive_classes': True,\n",
    "                                                 # Try the ensemble again later\n",
    "                                                 'architecture': 'bow',\n",
    "                                                 'ngram_size': 3})\n",
    "nlp.add_pipe(categorizer)\n",
    "\n",
    "# Add the labels\n",
    "cat_names = df['simple_category'].unique()\n",
    "\n",
    "for cat in cat_names:\n",
    "    categorizer.add_label(cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from multiprocessing import Pool\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "# Tokenize and remove stopwords\n",
    "stops = stopwords.words('english')\n",
    "\n",
    "def clean_headlines(doc):\n",
    "    return ' '.join([x.text.lower() for x in doc \n",
    "                     if x.text.lower() not in stops and not x.is_punct])\n",
    "\n",
    "# Add some data cleaning before categorization.\n",
    "nlp.add_pipe(clean_headlines, before='textcat')\n",
    "\n",
    "# Parallelize...go!\n",
    "# with Pool(16) as pool:\n",
    "#     headlines = pool.map(clean_headlines, df['headline_text'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df['headline_text'], \n",
    "                                                    df['simple_category'],\n",
    "                                                    test_size=0.2, train_size=0.8,\n",
    "                                                    random_state=777, \n",
    "                                                    stratify=df['simple_category'])\n",
    "\n",
    "def generate_label_dict(to_label):\n",
    "    return {'cats': \n",
    "            {'business': to_label == 'business',\n",
    "             'culture': to_label == 'culture',\n",
    "             'news': to_label == 'news',\n",
    "             'opinion': to_label == 'opinion',\n",
    "             'sport': to_label == 'sport',\n",
    "             'lifestyle': to_label == 'lifestyle'}}\n",
    "\n",
    "train_labels = [generate_label_dict(label) for label in y_train]\n",
    "test_labels = [generate_label_dict(label) for label in y_test]\n",
    "train = list(zip(X_train, train_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'textcat': 680.3957293283893}\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-7f3b7e06bcb6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     17\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mbatch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mbatches\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m         \u001b[0mtexts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_labels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 19\u001b[0;31m         \u001b[0mnlp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtexts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_labels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdrop\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdropout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msgd\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlosses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlosses\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     20\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlosses\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/spacy/language.py\u001b[0m in \u001b[0;36mupdate\u001b[0;34m(self, docs, golds, drop, sgd, losses, component_cfg)\u001b[0m\n\u001b[1;32m    519\u001b[0m             \u001b[0mproc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdocs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgolds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msgd\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mget_grads\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlosses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlosses\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    520\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mW\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdW\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgrads\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 521\u001b[0;31m                 \u001b[0msgd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mW\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdW\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    522\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    523\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrehearse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdocs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msgd\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlosses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import random\n",
    "from spacy.util import minibatch, fix_random_seed, decaying, compounding\n",
    "\n",
    "# Make it reproducible, yo!\n",
    "fix_random_seed(777)\n",
    "\n",
    "optimizer = nlp.begin_training()\n",
    "\n",
    "losses = {}\n",
    "dropout = decaying(0.8, 0.2, 1e-4)\n",
    "batch_sizes = compounding(1, 32, 1.001)\n",
    "\n",
    "for epoch in range(3):\n",
    "    random.shuffle(train)\n",
    "    batches = minibatch(train, size=batch_sizes)\n",
    "\n",
    "    for batch in batches:\n",
    "        texts, train_labels = zip(*batch)\n",
    "        nlp.update(texts, train_labels, drop=next(dropout), sgd=optimizer, losses=losses)\n",
    "    print(losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Let's save the model so that we can run it again later when we have more memory to spare.\n",
    "nlp.to_disk('/home/sean/Code/irish_times/bow_3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_label(match):\n",
    "    return 1 if match[0] == match[1] else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drumroll, please!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model is 14.14% accurate on the test data.\n"
     ]
    }
   ],
   "source": [
    "test_test_headlines = [nlp.tokenizer(txt) for txt in X_test]\n",
    "test_categorizer = nlp.get_pipe('textcat')\n",
    "test_scores, _ = test_categorizer.predict(test_test_headlines)\n",
    "test_pred_labels = test_scores.argmax(axis=1)\n",
    "test_predictions = [test_categorizer.labels[label] for label in test_pred_labels]\n",
    "compare_test_labels = list(zip(test_predictions, y_test))\n",
    "\n",
    "# ...aaaaaannnnd the scores!\n",
    "test_correct = sum([compare_label(m) for m in compare_test_labels])\n",
    "print(f'The model is {(test_correct / df.shape[0]) * 100:.2f}% accurate on the test data.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Notes\n",
    "It's looking like there are some repeatable things here that would work well in a Hy macro or three.\n",
    "Consider testing with methods other than `minibatch` as well as `minibatch` using `sklearn.model_selection.cross_val_predict` and train them all at once with the cool multiprocessing thing that I learned."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}