02_03_mwt_with_finditer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Working with Multi-Word Token Entities and RegEx in spaCy 3</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>Dr. W.J.B. Mattingly</center>\n",
    "\n",
    "<center>Smithsonian Data Science Lab and United States Holocaust Memorial Museum</center>\n",
    "\n",
    "<center>January 2021</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Concepts in this Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Working with Multi-Word Tokens and RegEx in spaCy 3x<br>\n",
    "2) RegEx's Finditer<br>\n",
    "3) Spans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problems with Multi-Word Tokens in spaCy as Entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw in **01.03: Rules-Based NER**, we can use spaCy's Matcher to grab multi-word tokens, or tokens that span multiple tokens. The main problem with this, however, is that these multi-word tokens are not placed into the doc.ents. This means that we cannot access them the same way we would other entities. In this notebook, we will figure out how to solve that problem with a simple workflow:\n",
    "\n",
    "1) Extract Multi-Word Tokens with re.finditer()<br>\n",
    "2) Reconstruct the spans in the spaCy doc<br>\n",
    "3) Give priority to longer spans (Optional)<br>\n",
    "4) Inject the Spans into doc.ents\n",
    "\n",
    "We will cover each of these steps in turn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Multi-Word Tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to grab the multi-word tokens. In this notebook, we are going to try and grab a multi-word token. In this case, a person whose first name begins with Paul. In the RegEx below, we specify that we are looking for any string that starts with \"Paul\" and then is followed by a capitalized letter. We then tell it to grab the entire second word until the end of the word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 11), match='Paul Newman'>\n",
      "<re.Match object; span=(39, 53), match='Paul Hollywood'>\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "text = \"Paul Newman was an American actor, but Paul Hollywood is a British TV Host. The name Paul is quite common.\"\n",
    "\n",
    "pattern = r\"Paul [A-Z]\\w+\"\n",
    "\n",
    "matches = re.finditer(pattern, text)\n",
    "\n",
    "for match in matches:\n",
    "    print (match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have not grabbed the final \"Paul\" which is not followed by a last name. In this case, we are not interested in that Paul. Now that we know how to grab the multi-word tokens, we need to have a way to parse them in spaCy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reconstruct Spans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next stage is a bit more complicated, but works quite well once you understand the process. First, we need to import the libraries we will need. Note that we are also adding Span from spacy.tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Enabling eager execution\n",
      "INFO:tensorflow:Enabling v2 tensorshape\n",
      "INFO:tensorflow:Enabling resource variables\n",
      "INFO:tensorflow:Enabling tensor equality\n",
      "INFO:tensorflow:Enabling control flow v2\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import spacy\n",
    "from spacy.tokens import Span"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will do the same thing that we did above with our text and our pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Paul Newman was an American actor, but Paul Hollywood is a British TV Host. The name Paul is quite common.\"\n",
    "pattern = r\"Paul [A-Z]\\w+\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will create a blank spaCy English model and create the doc object of the text. It will have no entities in it because we are working with a blank model that does not have an \"ner\" component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.blank(\"en\")\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though this part is unnecessary, it is good to do it here because in other situations you will have entities. If you do, you need to store them as a separate list to which we will append things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_ents = list(doc.ents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's iterate over the results from re.finditer(). In this cell, we are goingg to grab the start and end from each match. we will then create a temporary span that will be equal to where the characters start and end in the doc object. This is important because tokens and characters do not always align correctly. Finally, we append to mwt_ents, the start, end, and text. The text is not necessary but it will help with debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mwt_ents = []\n",
    "for match in re.finditer(pattern, doc.text):\n",
    "    start, end = match.span()\n",
    "    span = doc.char_span(start, end)\n",
    "    if span is not None:\n",
    "        mwt_ents.append((span.start, span.end, span.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inject the Spans into the doc.ents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that data, we can iterate over each entity and identify where it begins and ends in spaCy. Note, we are using the spaCy Span class. This allows us to create a span object and assign it a custom label. With this data, we can append each Span to original_ents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ent in mwt_ents:\n",
    "    start, end, name = ent\n",
    "    per_ent = Span(doc, start, end, label=\"PERSON\")\n",
    "    original_ents.append(per_ent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we set doc.ents equal to original_ents. This effectively loads the spans back into the spaCy doc.ents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc.ents = original_ents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's iterate over the ents as we normally would."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Newman PERSON\n",
      "Paul Hollywood PERSON\n"
     ]
    }
   ],
   "source": [
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that these are now properly identified entities in our doc.ents class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Give priority to Longer Spans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, the situation is not so neat. Sometimes our custom RegEx entities will overlap with spaCy's Entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Newman PERSON\n",
      "American NORP\n",
      "Paul Hollywood PERSON\n",
      "British NORP\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import spacy\n",
    "\n",
    "text = \"Paul Newman was an American actor, but Paul Hollywood is a British TV Host.\"\n",
    "pattern = r\"Hollywood\"\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "doc = nlp(text)\n",
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say that we create a new entity. Maybe words associated with Cinema. So, we want to classify Hollywood as a tag \"CINEMA\". Now, in the above text, Hollywood is clearly associated with Paul Hollywood, but let's imagine for a moment that it is not. Let's try and run the same code as above. If we do, we notice that we get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(44, 53), match='Hollywood'>\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "[E1010] Unable to set entity information for token 9 which is included in more than one span in entities, blocked, missing or outside.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-11-425d356ded45>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     12\u001b[0m     \u001b[0moriginal_ents\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mper_ent\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 14\u001b[1;33m \u001b[0mdoc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0ments\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0moriginal_ents\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mc:\\users\\wma22\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\spacy\\tokens\\doc.pyx\u001b[0m in \u001b[0;36mspacy.tokens.doc.Doc.ents.__set__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;32mc:\\users\\wma22\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\spacy\\tokens\\doc.pyx\u001b[0m in \u001b[0;36mspacy.tokens.doc.Doc.set_ents\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: [E1010] Unable to set entity information for token 9 which is included in more than one span in entities, blocked, missing or outside."
     ]
    }
   ],
   "source": [
    "mwt_ents = []\n",
    "original_ents = list(doc.ents)\n",
    "for match in re.finditer(pattern, doc.text):\n",
    "    print (match)\n",
    "    start, end = match.span()\n",
    "    span = doc.char_span(start, end)\n",
    "    if span is not None:\n",
    "        mwt_ents.append((span.start, span.end, span.text))\n",
    "for ent in mwt_ents:\n",
    "    start, end, name = ent\n",
    "    per_ent = Span(doc, start, end, label=\"CINEMA\")\n",
    "    original_ents.append(per_ent)\n",
    "\n",
    "doc.ents = original_ents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This error tells us that one of our tokens from the finditer() overlapped with one that our \"ner\" component found. This is a problem that can be rectified with spaCy's filter_spans. This gives primacy to longer spans. Notice how we have allowed the Paul Hollywood entity to be a PERSON, rather than CINEMA. This is because Hollywood is shorter than Paul Hollywood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paul Newman PERSON\n",
      "American NORP\n",
      "Paul Hollywood PERSON\n",
      "British NORP\n"
     ]
    }
   ],
   "source": [
    "from spacy.util import filter_spans\n",
    "filtered = filter_spans(original_ents)\n",
    "doc.ents = filtered\n",
    "for ent in doc.ents:\n",
    "    print (ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out the video below on RegEx and Multi-Word Tokens in spaCy 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<div align=\"center\">\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/wpyCzodvO3A\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}