Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Use the Hanover Tagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "* [Installation and Import](#sec-instal)\n",
    "* [German](#sec-german)\n",
    "* [Dutch](#sec-dutch)\n",
    "* [English](#sec-english)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation and Import<a class=\"anchor\" id=\"sec-installation\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: HanTa in c:\\users\\wartena\\anaconda3\\lib\\site-packages (1.0.0)\n",
      "Collecting HanTa\n",
      "  Downloading HanTa-1.1.0-py3-none-any.whl (15.0 MB)\n",
      "Installing collected packages: HanTa\n",
      "  Attempting uninstall: HanTa\n",
      "    Found existing installation: HanTa 1.0.0\n",
      "    Uninstalling HanTa-1.0.0:\n",
      "      Successfully uninstalled HanTa-1.0.0\n",
      "Successfully installed HanTa-1.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade HanTa "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from HanTa import HanoverTagger as ht"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## German<a class=\"anchor\" id=\"sec-german\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a trained model. E.g. the model on Github trained on the TIGER-Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger = ht.HanoverTagger('morphmodel_ger.pgz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing a word\n",
    "\n",
    "The method analyze gives the most probable part of speech, the lemma and a morphological analysis of a word.  By using the optional parameter taglevel, we can very the amount of information shown:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Fachmarkt', 'NN')\n",
      "NN\n",
      "('Fachmarkt', 'NN')\n",
      "('fach+markt+e', 'NN')\n",
      "('fachmarkt', [('fach', 'NN'), ('märkt', 'NN_VAR'), ('e', 'SUF_NN')], 'NN')\n"
     ]
    }
   ],
   "source": [
    "print(tagger.analyze('Fachmärkte'))\n",
    "print(tagger.analyze('Fachmärkte',taglevel=0))\n",
    "print(tagger.analyze('Fachmärkte',taglevel=1))\n",
    "print(tagger.analyze('Fachmärkte',taglevel=2))\n",
    "print(tagger.analyze('Fachmärkte',taglevel=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the taglevel is set to 1 the Hanover Tagger tries to generate the correct lemma. For the levels 2 and 3 the stem of te word is given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('werfen', 'VV(FIN)')\n",
      "('werf+t', 'VV(FIN)')\n",
      "('werf', [('wirf', 'VV_VAR'), ('t', 'SUF_FIN')], 'VV(FIN)')\n"
     ]
    }
   ],
   "source": [
    "print(tagger.analyze('wirft',taglevel=1))\n",
    "print(tagger.analyze('wirft',taglevel=2))\n",
    "print(tagger.analyze('wirft',taglevel=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the parameter *pos* we can force to give the most likely analysis for the given part of speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('vertrau', [('vertrau', 'VVnp'), ('te', 'SUF_FIN')], 'VV(FIN)')\n",
      "('vertraut', [('vertrau', 'VVnp'), ('t', 'SUF_PP'), ('e', 'SUF_ADJ')], 'ADJ(D)')\n",
      "('vertraut', [('vertrau', 'VVnp'), ('t', 'SUF_PP'), ('e', 'SUF_ADJ')], 'NNA')\n"
     ]
    }
   ],
   "source": [
    "print(tagger.analyze('vertraute',taglevel=3,pos='VV(FIN)'))\n",
    "print(tagger.analyze('vertraute',taglevel=3,pos='ADJ(D)'))\n",
    "print(tagger.analyze('vertraute',taglevel=3,pos='NNA'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tagging a word\n",
    "\n",
    "With the method tag_word we can get the most probable POS-tags for a word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', -13.007050108561302),\n",
       " ('NNI', -13.704188662004706),\n",
       " ('NE', -19.594898957386754),\n",
       " ('VV(INF)', -23.595031081236865),\n",
       " ('VV(FIN)', -25.169976341439774)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger.tag_word('Angeln')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The numbers are the natural logarithm of the probability that the word is found with the given POS, as estimated by the underlying Hidden Markov Model. Here e.g. the probability word 'Angeln'  is found together with the tag 'NN' is $e^{-13} = 2.26 \\cdot 10^{-6}$.\n",
    "\n",
    "Using the Parameter cutoff we can get more or less results. Cutoff give the maximal difference of the logprob of the last result with the best result. The cutoff Parameter does not apply to frequent words with cached analyses! The aim of the cutoff is to exclude impossible analyses. Each cached analysis, however, has been obeserved and is possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('NN', -18.757361480409173)]\n",
      "[('NN', -18.757361480409173), ('VV(FIN)', -25.868323558371134), ('ADJ(A)', -28.178979842637556)]\n",
      "[('NN', -18.757361480409173), ('VV(FIN)', -25.868323558371134), ('ADJ(A)', -28.178979842637556), ('ADV', -34.50301836881482), ('ADJ(D)', -34.836595953773134), ('NNA', -35.32902383199287), ('FM', -35.82508555170638), ('NE', -35.997194162564966), ('VV(IMP)', -36.16169684930409)]\n"
     ]
    }
   ],
   "source": [
    "print(tagger.tag_word('verdachte',cutoff=0))\n",
    "print(tagger.tag_word('verdachte',cutoff=10))\n",
    "print(tagger.tag_word('verdachte',cutoff=20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the optional Parameter casesensitive is set to True (the default value) uppercase is used to guess the most likely part of speech, mainly favouring proper noun readings and for German noun readings over other possibilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', -13.0038),\n",
       " ('NNI', -13.6969),\n",
       " ('VV(INF)', -16.8673),\n",
       " ('VV(FIN)', -18.3327),\n",
       " ('NE', -19.5531)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger.tag_word('angeln',casesensitive=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('VV(INF)', -16.868497963605382),\n",
       " ('VV(FIN)', -18.333773598068934),\n",
       " ('NNI', -18.621977405734608),\n",
       " ('NN', -18.734491493868266),\n",
       " ('NE', -22.74881056438149)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger.tag_word('angeln',casesensitive=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tagging a sentence\n",
    "\n",
    "The Hanover Tagger also can tag all words in a sentence at once. First probabilities for each word and POS are computed. Then a trigramm sentence model is used to disambiguate the tags and select the contextual most approriate POS. Finally, the words are analysed again, and the analysis leading to the contextually best PoS is given. \n",
    "\n",
    "Here we can again use the parameters taglevel and casesensitive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Die', 'der', 'ART'),\n",
      " ('Europawahl', 'Europawahl', 'NN'),\n",
      " ('in', 'in', 'APPR'),\n",
      " ('den', 'der', 'ART'),\n",
      " ('Niederlanden', 'Niederlanden', 'NE'),\n",
      " ('findet', 'finden', 'VV(FIN)'),\n",
      " ('immer', 'immer', 'ADV'),\n",
      " ('donnerstags', 'donnerstags', 'ADV'),\n",
      " ('statt', 'statt', 'PTKVZ'),\n",
      " ('.', '.', '$.')]\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from pprint import pprint\n",
    "\n",
    "sent = \"Die Europawahl in den Niederlanden findet immer donnerstags statt.\"\n",
    "\n",
    "words = nltk.word_tokenize(sent)\n",
    "lemmata = tagger.tag_sent(words)\n",
    "pprint(lemmata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Die', 'der', [('die', 'ART')], 'ART'),\n",
      " ('Sozialdemokraten',\n",
      "  'sozialdemokrat',\n",
      "  [('sozialdemokrat', 'NN'), ('en', 'SUF_NN')],\n",
      "  'NN'),\n",
      " ('haben', 'hab', [('hab', 'VA'), ('en', 'SUF_FIN')], 'VA(FIN)'),\n",
      " ('ersten', 'erst', [('erst', 'ADJ'), ('en', 'SUF_ADJ')], 'ADJ(A)'),\n",
      " ('Prognosen', 'prognose', [('prognose', 'NN'), ('n', 'SUF_NN')], 'NN'),\n",
      " ('zufolge', 'zufolge', [('zufolge', 'APPO')], 'APPO'),\n",
      " ('die', 'der', [('die', 'ART')], 'ART'),\n",
      " ('Europawahl', 'europawahl', [('europawahl', 'NN')], 'NN'),\n",
      " ('in', 'in', [('in', 'APPR')], 'APPR'),\n",
      " ('den', 'der', [('den', 'ART')], 'ART'),\n",
      " ('Niederlanden', 'niederlanden', [('niederlanden', 'NE')], 'NE'),\n",
      " ('gewonnen',\n",
      "  'gewinn',\n",
      "  [('gewonn', 'VVnp_VAR_PP'), ('en', 'SUF_PP')],\n",
      "  'VV(PP)'),\n",
      " ('.', '.', [('.', '$.')], '$.')]\n"
     ]
    }
   ],
   "source": [
    "sent = \"Die Sozialdemokraten haben ersten Prognosen zufolge die Europawahl in den Niederlanden gewonnen.\"\n",
    "\n",
    "words = nltk.word_tokenize(sent)\n",
    "lemmata = tagger.tag_sent(words,taglevel = 3)\n",
    "pprint(lemmata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ART', 'ADJ(A)', 'NN', 'NE', 'NE', 'VA(FIN)', 'ART', 'ADJ(A)', 'NN', 'APPRART', 'ADJ(A)', 'NN', '$,', 'PRELAT', 'NN', 'APPR', 'PIAT', 'NN', 'ADJ(A)', 'KON', 'ADJ(A)', 'NN', 'PTKVZ', 'VA(PP)', 'VA(FIN)', '$.']\n"
     ]
    }
   ],
   "source": [
    "sent = \"Der palästinensische Schriftsteller Emil Habibi ist der einzige Autor im Nahen Osten, dessen Werk von allen Seiten größte und offizielle Anerkennung zuteil geworden ist.\"\n",
    "\n",
    "words = nltk.word_tokenize(sent)\n",
    "tags = tagger.tag_sent(words,taglevel= 0)\n",
    "print(tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If only the part of speech is needed, it is recommended to set taglevel = 0, since this will be much faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some information on the underlying tagging model\n",
    "\n",
    "The German model was trained on data derived from the Tiger Corpus. Hence the POS-tags are almost the same as in the Tiger Corpus, sc. the tags from the STuttgart Tübingen Tagset. See https://www.ims.uni-stuttgart.de/documents/ressourcen/korpora/tiger-corpus/annotation/tiger_scheme-morph.pdf (esp. pp 26/27). A general description of the tagset is available e.g. here: https://www.ims.uni-stuttgart.de/forschung/ressourcen/lexika/germantagsets/#id-cfcbf0a7-0 or here: https://homepage.ruhr-uni-bochum.de/stephen.berman/Korpuslinguistik/Tagsets-STTS.html\n",
    "\n",
    "The tags used for the morphemes are derived from the POS tags. \n",
    "\n",
    "HanTa can list all the POS tags and the tags for morphemes with some random examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$(\t/, (, ,, ), \", ..., :, -\n",
      "$,\t,\n",
      "$.\t!, :, ?, ., ;\n",
      "ADJ(A)\tgrünen, weiterer, allgemeinen, russische, erhebliche, halbe, westlichen, gemeinsame, britischen, großer\n",
      "ADJ(D)\teindeutig, bewußt, nötig, größer, neu, eng, umstritten, verstärkt, angemessen, interessiert\n",
      "ADV\tdort, zuerst, ohnehin, erst, spätestens, weit, zweimal, außen, etwa, aber\n",
      "APPO\tnach, durch, über, gegenüber, wegen, entgegen, entlang, voran, halber, zufolge\n",
      "APPR\thinsichtlich, als, inklusive, entlang, entgegen, bei, außerhalb, statt, jenseits, v.\n",
      "APPRART\tunters, aufs, vom, am, z., beim, ins, ums, zur, fürs\n",
      "APZR\thinein, vorbei, aus, hin, herum, willen, entlang, hinaus, an, heraus\n",
      "ART\tden, ein, einen, 'n, einer, s, die, der, eines, eine\n",
      "CARD\t26, 1977, tausend, 60, 35, 1972, sieben, 1987, 54, 65\n",
      "FM\taustria, first, nouveau, akbar, tiger, par, puncto, parks, il, labour\n",
      "ITJ\tach, ja, o, mann, na\n",
      "KOKOM\tdenn, wie, als\n",
      "KON\tauch, weder, aber, denn, wie, statt, u, als, mal, sowie\n",
      "KOUI\tums, statt, um, ohne, anstatt\n",
      "KOUS\tseit, obwohl, ehe, indessen, so, wie, obgleich, zumal, während, daß\n",
      "NE\tjohannes, mainz, rom, rabins, iwf, kwasniewski, müller, italien, spanien, wien\n",
      "NN\tregierung, koalition, kosten, staat, prozent, hilfe, uhr, anteil, ausland, kollegen\n",
      "NNA\tschwarze, beschuldigten, studierende, beamte, parteivorsitzende, verantwortlichen, gute, beschuldigte, fremden, landesvorsitzender\n",
      "NNI\tzustandekommen, schweigen, zusammengehen, erleben, verschwindenlassen, aussterben, kehren, morden, wissen, wirtschaften\n",
      "PDAT\tjenes, demselben, dieselbe, dieses, di, derartigen, denselben, jenen, dasselbe, jenem\n",
      "PDS\tdie, diese, dessen, jene, d., jener, dies, derselben, jenen, dem\n",
      "PIAT\tanderem, solches, andere, keines, weniger, anderen, was, all, meiste, einige\n",
      "PIS\twelche, keinen, mehr, etliche, einer, öfter, jedes, alles, jemanden, aller\n",
      "PPER\tmir, ick, ich, es, 's, dir, ihm, s, dich, ihnen\n",
      "PPOSAT\tmeines, meinem, meiner, meinen, ihren, seinen, mein, unseren, unseres, seine\n",
      "PPOSS\tmeiner, seines, seinen, ihren, unseren, seinem, seine\n",
      "PRELAT\tdessen, deren\n",
      "PRELS\twas, der, denen, demzufolge, welcher, die, welchem, den, dem, welches\n",
      "PRF\teuch, mich, dich, mir, einander, sich, uns\n",
      "PROAV\tdanach, daraufhin, dahin, daran, zudem, demzufolge, darüber, hieran, woraufhin, miteinander\n",
      "PTKA\tam, allzu, zu\n",
      "PTKANT\tbitte, nein, gewiß, ja\n",
      "PTKNEG\tnicht\n",
      "PTKVZ\twider, vorbei, hervor, teil, pleite, raus, hinunter, ab, heraus, dazwischen\n",
      "PTKZU\tzu, zum, zur\n",
      "PWAT\twelcher, welche, welchen, welchem, welch, welches, wieviele, wieviel\n",
      "PWAV\twobei, wogegen, warum, wieviel, woran, was, wie, wodurch, wohin, wieso\n",
      "PWS\twer, wem, wieviel, was, welcher, welches, wieviele, wen, welche\n",
      "TRUNC\talters-, zwei-, klein-, forschungs-, eisen-, radio-, kirchen-, städte-, zukunfts-, volks-\n",
      "UNKNOWN\t\n",
      "VA(FIN)\thast, habe, sind, waren, seien, hatte, werde, wirt, bin, wird\n",
      "VA(IMP)\thabt, werdet, werde, sei, seid\n",
      "VA(INF)\twerden, haben, sein, worden\n",
      "VA(PP)\tgewesen, werdet, gehabt, worden, seid, geworden, habt\n",
      "VM(FIN)\tkonnte, magen, möchte, möchten, läßt, können, muße, könnten, dürfe, dürften\n",
      "VM(INF)\tkönnen, wollen, sollen, dürfen, müssen\n",
      "VM(PP)\tgewollt\n",
      "VV(FIN)\tverspricht, fand, stehe, scheinen, trifft, fordern, folgt, führt, besteht, kritisierte\n",
      "VV(IMP)\tkandidiere, faß, jubel, handle, kehrt, spart, schaut, lohn, erhöhe, bremse\n",
      "VV(INF)\tbeschäftigen, überwinden, warten, prüfen, liefern, wachsen, vermeiden, reduzieren, holen, drücken\n",
      "VV(IZU)\taufzunehmen, durchzuführen, ansetzen, einzurichten, auszubauen, auszurichten, einzuführen, umzugehen, zuzumuten, abzusehen\n",
      "VV(PP)\tgearbeitet, vermieden, gewarnt, ausgesprochen, gestellt, abgesehen, abgelehnt, veröffentlicht, gesetzt, ausgelöst\n",
      "XY\tq, +, rds, gra, h., jk, f., s, j., epd\n"
     ]
    }
   ],
   "source": [
    "tagger.list_postags()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ACR_NE             kws, d., anc, bgh, egb, iter, npd, raf, dws, hbv\n",
      "ACR_NN             mrd, dm, gmbh, kp, kwg, ngo, fckw, vatu, pvc, a320\n",
      "ADJ                westlich, wirtschaftlich, jung, nötig, ander, gesetzlich, deutsch, nigerianisch, kommunistisch, groß\n",
      "ADJ_COMP           er\n",
      "ADJ_INVAR          würzburger, bregenzer, kopenhagener, seeheimer, baseler, mecklenburger, petersburger, weise, haager, bochumer\n",
      "ADJ_IRR            größte, höchstverantwortlichen, letztes, klar, umgerechnet, größtem, größten, viertgrößte, politisch, erforderlich\n",
      "ADJ_SUP            st, est\n",
      "ADJ_VAR            veritabl, gröb, größt, größer, innern, edl, sinistr, profitabl, höch, millionenteur\n",
      "FUGE               nen, es, s, n, en, e, er\n",
      "HYPHEN             -\n",
      "NE_VAR             shakespear, n', vereinten, lersnerstr., l', tian'anmen-platz, özgür, thüringer, aserbaidschan, großbritannien\n",
      "NN_IRR             ``aldi''-brüder, -führer, ``kavadi''-träger, ``klassik''-mitarbeiter, -steuer, -desaster, -fenster, öfen, -melodien, ``inkatha''-kämpfer\n",
      "NN_VAR             fäch, erträg, gärten, häng, kräft, züg, auskünft, zöll, löhn, gründ\n",
      "NN_VAR_EL          münz, kron, erd, buß, kirch, grenz, end, eck, sprach, hochschul\n",
      "NN_VAR_ELIS        farb\n",
      "ORD_ABR            40., 4., 19., 27., 14., 22., 20., 13., 196., 23.\n",
      "PDAT_VAR           dieselbe, denjenige, di, derartig, desselbe, demselbe, diejenige, denselbe, dies, dasselbe\n",
      "PIS_IRR            ihresgleichen\n",
      "PIS_VAR            viel, wat, ein, wen'g\n",
      "PPOSAT_VAR         eur, ihr, eurer\n",
      "PREF_NEG           un\n",
      "PREF_PP            ge\n",
      "PRESPART           nd, end\n",
      "PTKVZ_DUBIUM       über, unter, um, durch, wieder\n",
      "PTKVZ_SEP          vorbei, heraus, zu, statt, feil, still, hervor, auseinander, herum, zurück\n",
      "SUF_ADJ            er, em, en, es, e\n",
      "SUF_ADJ_NN         en, er\n",
      "SUF_DERIV_VV_ADJ   bar\n",
      "SUF_DIM            chen\n",
      "SUF_FEM            in\n",
      "SUF_FIN            st, test, te, t, eten, est, en, tet, et, e\n",
      "SUF_IMP            et, t, e\n",
      "SUF_INF            ns, 'n, ens, en, te, n\n",
      "SUF_IZU            n, en\n",
      "SUF_NE             s, -, \", ist, en, e, is, n, 's, es\n",
      "SUF_NN             a, ien, s, sen, ten, p, nen, ste, \", en\n",
      "SUF_PDAT           n, es, en, e, em\n",
      "SUF_PIAT           l, en, e, em, es, r, er, n\n",
      "SUF_PIS            es, e, n, er, r, em, ie, en, s, m\n",
      "SUF_PP             te, en, n, ete, et, t, end\n",
      "SUF_PPOSAT         em, n, ers, m, es, er, en, e\n",
      "SYMB               k, m, %, b, e, d, a, §, q, z\n",
      "URL                http://virtual.design-exhibition.com, http://vs.sony.co.jp, http://www.blacksun.com, http://www.intervista.com, http://www.worlds.net, http://vrml.sgi.com, http://www.vdi.com\n",
      "VA                 dabeisei, word, werd, hab, sei\n",
      "VA_IRR             gewesen, wär's, ist, war's, sind, warst, wär, seien, wären, wart\n",
      "VA_VAR             word, wurd, ha, hat's, bü, hätt', würd, wir, wird, hätt\n",
      "VA_VAR_PP          word\n",
      "VM                 könn, soll, woll, dürf, möch, müss, möcht, möchte, mög\n",
      "VM_VAR             vermißt, mocht, muß, darf, ließ, konnt, musst, durft, mus, mußt\n",
      "VM_VAR_PP          moch, konn\n",
      "VV                 pfleg, frag, schütz, heb, plan, stamm, prüf, lös, richt, hol\n",
      "VV_IRR             ausgetimed\n",
      "VV_VAR             ri, nahm, dräng, böt, zieh, schri, wußt, schwand, miß, goss\n",
      "VV_VAR_PP          wies, durchzog, mitbeschloss, beworb, nann, nachvollzog, lad, troff, einbezog, zieh\n",
      "VVnp               verschärf, beauftrag, beschuldig, präsentier, behaupt, interessier, erziel, erfüll, verfüg, veränder\n",
      "VVnp_VAR           vergiß, erwies, geniess, genoss, verbrannt, geschieh, erlit, gestand, verlör, übertraf\n",
      "VVnp_VAR_PP        überwies, überschritt, borg, verfaß, befaß, verschoss, verstand, entgegengebrach, verwand, gedach\n"
     ]
    }
   ],
   "source": [
    "tagger.list_mtags()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dutch<a class=\"anchor\" id=\"sec-dutch\"></a>\n",
    "\n",
    "You can load trained morphology models for some other languages in the same way as shown above for German. Here a few examples for a Dutch model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_nl = ht.HanoverTagger('morphmodel_dutch.pgz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('staa+t', 'WW(pv,tgw,met-t)')\n",
      "('staat', 'N(soort,ev,basis,zijd,stan)')\n"
     ]
    }
   ],
   "source": [
    "print(tagger_nl.analyze('staat',taglevel=2,pos='WW(pv,tgw,met-t)'))\n",
    "print(tagger_nl.analyze('staat',taglevel=2,pos='N(soort,ev,basis,zijd,stan)'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('WW(pv,tgw,met-t)', -7.763586419721995),\n",
       " ('N(soort,ev,basis,zijd,stan)', -8.121654473590889)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger_nl.tag_word('staat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('huishoudhulp', 'N(soort,ev,dim,onz,stan)')\n",
      "N(soort,ev,dim,onz,stan)\n",
      "('huishoudhulp', 'N(soort,ev,dim,onz,stan)')\n",
      "('huis+houd+hulp+je', 'N(soort,ev,dim,onz,stan)')\n",
      "('huishoudhulp', [('huis', 'N(soort,onz)'), ('houd', 'WW'), ('hulp', 'N(soort,zijd)'), ('je', 'SUF_DIM')], 'N(soort,ev,dim,onz,stan)')\n"
     ]
    }
   ],
   "source": [
    "print(tagger_nl.analyze('huishoudhulpje'))\n",
    "print(tagger_nl.analyze('huishoudhulpje',taglevel=0))\n",
    "print(tagger_nl.analyze('huishoudhulpje',taglevel=1))\n",
    "print(tagger_nl.analyze('huishoudhulpje',taglevel=2))\n",
    "print(tagger_nl.analyze('huishoudhulpje',taglevel=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('N(soort,ev,basis,onz,stan)', -9.735243330616047),\n",
       " ('WW(inf,vrij,zonder)', -12.609773795398848),\n",
       " ('WW(pv,tgw,mv)', -13.014584292537286)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger_nl.tag_word('vertrouwen')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Elk',\n",
      "  'elk',\n",
      "  [('elk', 'VNW(onbep,det,stan,prenom,zonder,evon)')],\n",
      "  'VNW(onbep,det,stan,prenom,zonder,evon)'),\n",
      " ('jaar', 'jaar', [('jaar', 'N(soort,onz)')], 'N(soort,ev,basis,onz,stan)'),\n",
      " ('wisselen',\n",
      "  'wissel',\n",
      "  [('wissel', 'WW'), ('en', 'SUF_WW(mv)')],\n",
      "  'WW(pv,tgw,mv)'),\n",
      " ('ruim', 'ruim', [('ruim', 'ADJ')], 'ADJ(vrij,basis,zonder)'),\n",
      " ('1', '1', [('1', 'TW(hoofd,prenom,stan)')], 'TW(hoofd,prenom,stan)'),\n",
      " ('miljoen',\n",
      "  'miljoen',\n",
      "  [('miljoen', 'N(soort,onz)')],\n",
      "  'N(soort,ev,basis,onz,stan)'),\n",
      " ('Nederlanders',\n",
      "  'nederlander',\n",
      "  [('nederlander', 'N(eigen,zijd)'), ('s', 'SUF_N_S')],\n",
      "  'N(eigen,mv,basis)'),\n",
      " ('van', 'van', [('van', 'VZ(init)')], 'VZ(init)'),\n",
      " ('zorgverzekeraar',\n",
      "  'zorgverzekeraar',\n",
      "  [('zorg', 'N(soort,zijd)'), ('verzekeraar', 'N(soort,zijd)')],\n",
      "  'N(soort,ev,basis,zijd,stan)'),\n",
      " ('.', '.', [('.', 'LET()')], 'LET()')]\n"
     ]
    }
   ],
   "source": [
    "sent = \"Elk jaar wisselen ruim 1 miljoen Nederlanders van zorgverzekeraar. \"\n",
    "\n",
    "words = nltk.word_tokenize(sent)\n",
    "lemmata = tagger_nl.tag_sent(words,taglevel= 3)\n",
    "pprint(lemmata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## English<a class=\"anchor\" id=\"sec-english\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_en = ht.HanoverTagger('morphmodel_en.pgz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VBG\n",
      "('walk', 'VBG')\n",
      "('walk+ing', 'VBG')\n",
      "('walk', [('walk', 'VB'), ('ing', 'SUF_ING')], 'VBG')\n"
     ]
    }
   ],
   "source": [
    "print(tagger_en.analyze('walking',taglevel=0))\n",
    "print(tagger_en.analyze('walking',taglevel=1))\n",
    "print(tagger_en.analyze('walking',taglevel=2))\n",
    "print(tagger_en.analyze('walking',taglevel=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('VBZ', -12.031221308520081), ('NNS', -12.225015720529305)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger_en.tag_word('walks')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you analyze English sentences, make sure that the word tokenization is done properly. The model provided works with the default word tokenization from NLTK, which splits words like _cannot_ and _don't_ . If the wrong type of apostrope is used, tokenization might not gove the expected results, as is the case i the first variant of the following sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['VBG', 'AT', 'JJ', 'NN', 'MD', 'BE', 'AT', 'JJ', 'NN', ',', 'QL', 'RB', 'BEZ', 'AT', 'JJ', 'NN', 'IN', 'NNS', 'TO', 'VB', ',', 'NNS', 'IN', 'NN', ',', 'NN', 'TO', 'VB', 'CC', 'AP', '.']\n",
      "----\n",
      "[('Tackling', 'tackl', 'VBG'), ('the', 'the', 'AT'), ('entire', 'entire', 'JJ'), ('kitchen', 'kitchen', 'NN'), ('can', 'can', 'MD'), ('be', 'be', 'BE'), ('an', 'a', 'AT'), ('intimidating', 'intimidating', 'JJ'), ('task', 'task', 'NN'), (',', ',', ','), ('so', 'so', 'QL'), ('here', 'here', 'RB'), (\"'s\", \"'s\", 'BEZ'), ('a', 'a', 'AT'), ('manageable', 'manageable', 'JJ'), ('list', 'list', 'NN'), ('of', 'of', 'IN'), ('things', 'thing', 'NNS'), ('to', 'to', 'TO'), ('clean', 'clean', 'VB'), (',', ',', ','), ('ingredients', 'ingredient', 'NNS'), ('to', 'to', 'IN'), ('check', 'check', 'NN'), (',', ',', ','), ('equipment', 'equipment', 'NN'), ('to', 'to', 'TO'), ('organize', 'organize', 'VB'), ('and', 'and', 'CC'), ('more', 'more', 'AP'), ('.', '.', '.')]\n",
      "----\n",
      "[('Tackling', 'tackl+ing', 'VBG'), ('the', 'the', 'AT'), ('entire', 'entire', 'JJ'), ('kitchen', 'kitchen', 'NN'), ('can', 'can', 'MD'), ('be', 'be', 'BE'), ('an', 'a', 'AT'), ('intimidating', 'intimidating', 'JJ'), ('task', 'task', 'NN'), (',', ',', ','), ('so', 'so', 'QL'), ('here', 'here', 'RB'), (\"'s\", \"'s\", 'BEZ'), ('a', 'a', 'AT'), ('manageable', 'manage+able', 'JJ'), ('list', 'list', 'NN'), ('of', 'of', 'IN'), ('things', 'thing+s', 'NNS'), ('to', 'to', 'TO'), ('clean', 'clean', 'VB'), (',', ',', ','), ('ingredients', 'ingredient+s', 'NNS'), ('to', 'to', 'IN'), ('check', 'check', 'NN'), (',', ',', ','), ('equipment', 'equipment', 'NN'), ('to', 'to', 'TO'), ('organize', 'organize', 'VB'), ('and', 'and', 'CC'), ('more', 'more', 'AP'), ('.', '.', '.')]\n",
      "----\n",
      "[('Tackling', 'tackl', [('tackl', 'VB'), ('ing', 'SUF_ING')], 'VBG'), ('the', 'the', [('the', 'AT')], 'AT'), ('entire', 'entire', [('entire', 'JJ')], 'JJ'), ('kitchen', 'kitchen', [('kitchen', 'NN')], 'NN'), ('can', 'can', [('can', 'MD')], 'MD'), ('be', 'be', [('be', 'BE')], 'BE'), ('an', 'a', [('an', 'AT_VAR')], 'AT'), ('intimidating', 'intimidating', [('intimidating', 'JJ')], 'JJ'), ('task', 'task', [('task', 'NN')], 'NN'), (',', ',', [(',', ',')], ','), ('so', 'so', [('so', 'QL')], 'QL'), ('here', 'here', [('here', 'RB')], 'RB'), (\"'s\", \"'s\", [(\"'s\", 'BEZ')], 'BEZ'), ('a', 'a', [('a', 'AT')], 'AT'), ('manageable', 'manageable', [('manage', 'VB'), ('able', 'SUF_VBJJ')], 'JJ'), ('list', 'list', [('list', 'NN')], 'NN'), ('of', 'of', [('of', 'IN')], 'IN'), ('things', 'thing', [('thing', 'NN'), ('s', 'SUF_NN_S')], 'NNS'), ('to', 'to', [('to', 'TO')], 'TO'), ('clean', 'clean', [('clean', 'VB')], 'VB'), (',', ',', [(',', ',')], ','), ('ingredients', 'ingredient', [('ingredient', 'NN'), ('s', 'SUF_NN_S')], 'NNS'), ('to', 'to', [('to', 'IN')], 'IN'), ('check', 'check', [('check', 'NN')], 'NN'), (',', ',', [(',', ',')], ','), ('equipment', 'equipment', [('equipment', 'NN')], 'NN'), ('to', 'to', [('to', 'TO')], 'TO'), ('organize', 'organize', [('organize', 'VB')], 'VB'), ('and', 'and', [('and', 'CC')], 'CC'), ('more', 'more', [('more', 'AP')], 'AP'), ('.', '.', [('.', '.')], '.')]\n"
     ]
    }
   ],
   "source": [
    "#sent = \"Tackling the entire kitchen can be an intimidating task, so here’s a manageable list of things to clean, ingredients to check, equipment to organize and more.\"\n",
    "sent = \"Tackling the entire kitchen can be an intimidating task, so here's a manageable list of things to clean, ingredients to check, equipment to organize and more.\"\n",
    "\n",
    "words = nltk.word_tokenize(sent)\n",
    "\n",
    "print(tagger_en.tag_sent(words,taglevel = 0))\n",
    "print('----')\n",
    "print(tagger_en.tag_sent(words,taglevel = 1))\n",
    "print('----')\n",
    "print(tagger_en.tag_sent(words,taglevel = 2))\n",
    "print('----')\n",
    "print(tagger_en.tag_sent(words,taglevel = 3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}