CivicHonorsAdvancedKGv2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyM8P3GQNBluLX77R5DOHlIC",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nelslindahlx/KnowledgeReduce/blob/main/CivicHonorsAdvancedKGv2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Code Overview\n",
        "\n",
        "The code provided is a comprehensive Python script designed for execution in Google Colab, specifically tailored to analyze content from \"https://civichonors.com/\". It encompasses a range of functionalities from web scraping to advanced data visualization and natural language processing. Here's a summary of each step in the code:\n",
        "\n",
        "### 1. Environment Setup\n",
        "- **1.1 Installation of Libraries**: Essential libraries like `requests`, `beautifulsoup4` (bs4), `spacy`, `networkx`, `matplotlib`, `textblob`, `plotly`, `gensim`, and `nltk` are installed. These libraries are fundamental for web scraping, text processing, natural language processing, and visualization.\n",
        "- **1.2 Importing Libraries**: All the installed libraries are imported for use in the subsequent steps.\n",
        "\n",
        "### 2. Data Acquisition and Processing\n",
        "- **2.1 Fetch Webpage Content**: The script fetches the HTML content of \"https://civichonors.com/\" using the `requests` library.\n",
        "- **2.2 Extract Text from HTML**: The `BeautifulSoup` library is used to parse the HTML content and extract the relevant textual data, stripping away any HTML tags, scripts, or styles.\n",
        "\n",
        "### 3. Advanced Natural Language Processing\n",
        "- **3.1 Custom Entity Recognition and Relationship Extraction**: Utilizing `spacy` and its `Matcher` class, custom entity recognition patterns are defined and applied to the text. This step identifies specific entities like awards, civic terms, and organizations.\n",
        "- **3.2 Sentiment Analysis**: The `TextBlob` library is used to perform sentiment analysis on the extracted text, providing insights into the overall sentiment conveyed in the content.\n",
        "\n",
        "### 4. Knowledge Graph Creation and Management\n",
        "- **4.1 Create Advanced Knowledge Graph**: The script constructs a knowledge graph using `networkx`. This graph includes nodes representing the identified entities and edges denoting their relationships.\n",
        "\n",
        "### 5. Visualization and Analysis\n",
        "- **5.1 Interactive Knowledge Graph Visualization**: An interactive visualization of the knowledge graph is created using `plotly`, which allows for an engaging and insightful exploration of the graph.\n",
        "- **5.2 Topic Modeling**: The `gensim` library is employed for topic modeling. The script processes the text, creates a document-term matrix, and applies the LDA (Latent Dirichlet Allocation) model to uncover the main topics within the text.\n",
        "\n",
        "### 6. Execution in Google Colab\n",
        "- The workflow is structured to run sequentially in a Google Colab notebook, ensuring each step logically follows from the previous one. This structured approach facilitates a thorough analysis of the webpage content, from initial data acquisition to in-depth NLP analysis, knowledge graph creation, and sophisticated visualization techniques.\n",
        "\n",
        "This code is a robust example of how various Python libraries and tools can be combined to extract, process, analyze, and visualize web content in a meaningful and insightful way. It demonstrates the power of Python in handling and interpreting complex web data."
      ],
      "metadata": {
        "id": "o1pyYR8izVaZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 1: Environment Setup"
      ],
      "metadata": {
        "id": "4SL5w-lbvSZV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1.1 Install Necessary Libraries"
      ],
      "metadata": {
        "id": "sSYjnqnsvUQj"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AR8DJoUgvRxe",
        "outputId": "2125b4a5-c466-4d93-9006-18e41a070df9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.31.0)\n",
            "Collecting bs4\n",
            "  Downloading bs4-0.0.1.tar.gz (1.1 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (3.6.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (3.2.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.7.1)\n",
            "Requirement already satisfied: textblob in /usr/local/lib/python3.10/dist-packages (0.17.1)\n",
            "Requirement already satisfied: plotly in /usr/local/lib/python3.10/dist-packages (5.15.0)\n",
            "Requirement already satisfied: gensim in /usr/local/lib/python3.10/dist-packages (4.3.2)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.8.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests) (2023.11.17)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from bs4) (4.11.2)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.10)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.8)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy) (8.1.12)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.10)\n",
            "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.9.0)\n",
            "Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.10.3)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (6.4.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (4.66.1)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.23.5)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.10.13)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy) (67.7.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (23.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.3.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.46.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.5)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (9.4.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2)\n",
            "Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from plotly) (8.2.3)\n",
            "Requirement already satisfied: scipy>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from gensim) (1.11.4)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.3.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2023.6.3)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.5.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.1.4)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->bs4) (2.5)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy) (2.1.3)\n",
            "Building wheels for collected packages: bs4\n",
            "  Building wheel for bs4 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1256 sha256=b864bcbddb2c6fb4b3a78fedf6d69c9f236528e902e0451f5c24ebc5d43883ee\n",
            "  Stored in directory: /root/.cache/pip/wheels/25/42/45/b773edc52acb16cd2db4cf1a0b47117e2f69bb4eb300ed0e70\n",
            "Successfully built bs4\n",
            "Installing collected packages: bs4\n",
            "Successfully installed bs4-0.0.1\n",
            "2023-12-15 20:46:44.295927: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2023-12-15 20:46:44.296011: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2023-12-15 20:46:44.297768: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2023-12-15 20:46:44.308344: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-12-15 20:46:45.737508: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "Collecting en-core-web-sm==3.6.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.6.0/en_core_web_sm-3.6.0-py3-none-any.whl (12.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.7.0,>=3.6.0 in /usr/local/lib/python3.10/dist-packages (from en-core-web-sm==3.6.0) (3.6.1)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (1.0.10)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2.0.8)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (8.1.12)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (1.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2.0.10)\n",
            "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (0.9.0)\n",
            "Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (0.10.3)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (6.4.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (4.66.1)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (1.23.5)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2.31.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (1.10.13)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (3.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (67.7.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (23.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (4.5.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2023.11.17)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (0.1.4)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.10.0,>=0.3.0->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (8.1.7)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<3.7.0,>=3.6.0->en-core-web-sm==3.6.0) (2.1.3)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_sm')\n"
          ]
        }
      ],
      "source": [
        "!pip install requests bs4 spacy networkx matplotlib textblob plotly gensim nltk\n",
        "!python -m spacy download en_core_web_sm"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "1.2 Import Libraries"
      ],
      "metadata": {
        "id": "nVA1bVccvZfy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import spacy\n",
        "from spacy.matcher import Matcher\n",
        "import networkx as nx\n",
        "import matplotlib.pyplot as plt\n",
        "import plotly.graph_objects as go\n",
        "from textblob import TextBlob\n",
        "import gensim\n",
        "from gensim import corpora\n",
        "from gensim.models import LdaModel\n",
        "import nltk\n",
        "from nltk.tokenize import RegexpTokenizer\n",
        "from nltk.corpus import stopwords"
      ],
      "metadata": {
        "id": "maCAQj8Qva4W"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 2: Data Acquisition and Processing"
      ],
      "metadata": {
        "id": "Jfmh5UqYvcwl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "2.1 Fetch Webpage Content"
      ],
      "metadata": {
        "id": "UOEH6UqeveiK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def fetch_webpage_content(url):\n",
        "    response = requests.get(url)\n",
        "    if response.status_code == 200:\n",
        "        return response.text\n",
        "    else:\n",
        "        return None\n",
        "\n",
        "webpage_content = fetch_webpage_content('https://civichonors.com/')"
      ],
      "metadata": {
        "id": "NefWYPPovgdq"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "2.2 Extract Text from HTML"
      ],
      "metadata": {
        "id": "AqIfUMS-viMN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def extract_text_from_html(html_content):\n",
        "    soup = BeautifulSoup(html_content, 'html.parser')\n",
        "    for script_or_style in soup(['script', 'style']):\n",
        "        script_or_style.extract()\n",
        "    text = soup.get_text()\n",
        "    return text\n",
        "\n",
        "extracted_text = extract_text_from_html(webpage_content)"
      ],
      "metadata": {
        "id": "yPTiAXZpvjnV"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 3: Advanced Natural Language Processing"
      ],
      "metadata": {
        "id": "fvRLW2tevlQO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "3.1 Custom Entity Recognition and Relationship Extraction"
      ],
      "metadata": {
        "id": "yrJ2l24RvtIQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nlp = spacy.load(\"en_core_web_sm\")\n",
        "\n",
        "def define_custom_patterns():\n",
        "    matcher = Matcher(nlp.vocab)\n",
        "\n",
        "    award_pattern = [{\"LOWER\": \"award\"}]\n",
        "    civic_term_pattern = [{\"LOWER\": \"civic\"}, {\"IS_ALPHA\": True}]\n",
        "    organization_pattern = [{\"LOWER\": \"association\"}, {\"IS_ALPHA\": True, \"OP\": \"?\"}]\n",
        "\n",
        "    matcher.add(\"AWARD\", [award_pattern])\n",
        "    matcher.add(\"CIVIC_TERM\", [civic_term_pattern])\n",
        "    matcher.add(\"ORGANIZATION\", [organization_pattern])\n",
        "\n",
        "    return matcher\n",
        "\n",
        "custom_matcher = define_custom_patterns()\n",
        "\n",
        "def advanced_nlp_processing(text):\n",
        "    doc = nlp(text)\n",
        "    standard_entities = [(ent.text, ent.label_) for ent in doc.ents]\n",
        "    matches = custom_matcher(doc)\n",
        "    custom_entities = [(doc[start:end].text, nlp.vocab.strings[match_id]) for match_id, start, end in matches]\n",
        "    return standard_entities + custom_entities\n",
        "\n",
        "advanced_entities = advanced_nlp_processing(extracted_text)"
      ],
      "metadata": {
        "id": "eKqhT8IDvl0t"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "3.2 Sentiment Analysis"
      ],
      "metadata": {
        "id": "NzWUxnyovwDE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def analyze_sentiment(text):\n",
        "    blob = TextBlob(text)\n",
        "    return blob.sentiment.polarity\n",
        "\n",
        "# Example: Analyze the sentiment of the entire extracted text\n",
        "sentiment_score = analyze_sentiment(extracted_text)"
      ],
      "metadata": {
        "id": "CSB4Em1Yvx_3"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 4: Knowledge Graph Creation and Management"
      ],
      "metadata": {
        "id": "vnPwziIMv1hK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "4.1 Create Advanced Knowledge Graph"
      ],
      "metadata": {
        "id": "DrnRWXfYv4HL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def create_and_store_knowledge_graph(advanced_entities):\n",
        "    G = nx.Graph()\n",
        "    for entity, label in advanced_entities:\n",
        "        G.add_node(entity, label=label)\n",
        "        G.add_edge(entity, \"extracted_from\", label=label)\n",
        "    return G\n",
        "\n",
        "knowledge_graph = create_and_store_knowledge_graph(advanced_entities)"
      ],
      "metadata": {
        "id": "yb3Sy7q0v3OJ"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 5: Visualization and Analysis"
      ],
      "metadata": {
        "id": "7W628otfv6q9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "5.1 Interactive Knowledge Graph Visualization"
      ],
      "metadata": {
        "id": "EfJ0LcSMv9Lp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def interactive_visualize_knowledge_graph(knowledge_graph):\n",
        "    edge_x = []\n",
        "    edge_y = []\n",
        "    for edge in knowledge_graph.edges():\n",
        "        x0, y0 = knowledge_graph.nodes[edge[0]]['pos']\n",
        "        x1, y1 = knowledge_graph.nodes[edge[1]]['pos']\n",
        "        edge_x.extend([x0, x1, None])\n",
        "        edge_y.extend([y0, y1, None])\n",
        "\n",
        "    edge_trace = go.Scatter(\n",
        "        x=edge_x, y=edge_y,\n",
        "        line=dict(width=0.5, color='#888'),\n",
        "        hoverinfo='none',\n",
        "        mode='lines')\n",
        "\n",
        "    node_x = []\n",
        "    node_y = []\n",
        "    for node in knowledge_graph.nodes():\n",
        "        x, y = knowledge_graph.nodes[node]['pos']\n",
        "        node_x.append(x)\n",
        "        node_y.append(y)\n",
        "\n",
        "    node_trace = go.Scatter(\n",
        "        x=node_x, y=node_y,\n",
        "        mode='markers',\n",
        "        hoverinfo='text',\n",
        "        marker=dict(showscale=True, colorscale='YlGnBu', size=10))\n",
        "\n",
        "    fig = go.Figure(data=[edge_trace, node_trace],\n",
        "                    layout=go.Layout(\n",
        "                        showlegend=False,\n",
        "                        hovermode='closest',\n",
        "                        margin=dict(b=0,l=0,r=0,t=0)))\n",
        "\n",
        "    fig.show()\n",
        "\n",
        "# Example: Visualize the knowledge graph\n",
        "pos = nx.spring_layout(knowledge_graph)  # Set the positions of the nodes\n",
        "for node in knowledge_graph.nodes:\n",
        "    knowledge_graph.nodes[node]['pos'] = list(pos[node])\n",
        "interactive_visualize_knowledge_graph(knowledge_graph)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "IJQT2QmKv_aW",
        "outputId": "aa2ceb1d-cb9c-44d7-a7c1-295704edb346"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script charset=\"utf-8\" src=\"https://cdn.plot.ly/plotly-2.24.1.min.js\"></script>                <div id=\"1977dd2d-85ea-4901-8a4d-4b7347ed5187\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"1977dd2d-85ea-4901-8a4d-4b7347ed5187\")) {                    Plotly.newPlot(                        \"1977dd2d-85ea-4901-8a4d-4b7347ed5187\",                        [{\"hoverinfo\":\"none\",\"line\":{\"color\":\"#888\",\"width\":0.5},\"mode\":\"lines\",\"x\":[-0.9054884960786804,-0.003664383835386732,null,-0.003664383835386732,-0.7401968271409917,null,-0.003664383835386732,-0.36983011804257965,null,-0.003664383835386732,-0.9701703516034483,null,-0.003664383835386732,0.563887969425789,null,-0.003664383835386732,-0.3642914136525418,null,-0.003664383835386732,0.3593378681916143,null,-0.003664383835386732,-0.835349913105015,null,-0.003664383835386732,0.8256732245883652,null,-0.003664383835386732,0.3845634436844661,null,-0.003664383835386732,-0.2893204441329471,null,-0.003664383835386732,-0.7867884707531642,null,-0.003664383835386732,0.00924935638097553,null,-0.003664383835386732,0.17493022996336047,null,-0.003664383835386732,0.8892696377507492,null,-0.003664383835386732,-0.6020865859624809,null,-0.003664383835386732,-0.24501927259392964,null,-0.003664383835386732,-0.5499472690432284,null,-0.003664383835386732,-0.5127403744393718,null,-0.003664383835386732,0.44317248068235615,null,-0.003664383835386732,0.24052583751913437,null,-0.003664383835386732,-0.8958395788671599,null,-0.003664383835386732,0.1141599999642776,null,-0.003664383835386732,-0.6129409354948676,null,-0.003664383835386732,0.03972117585742548,null,-0.003664383835386732,-0.4468440331187648,null,-0.003664383835386732,-0.6754671090703906,null,-0.003664383835386732,-0.7567385810093328,null,-0.003664383835386732,0.8169484432448167,null,-0.003664383835386732,0.5610063392592085,null,-0.003664383835386732,-0.014227952717972482,null,-0.003664383835386732,0.7885836808067992,null,-0.003664383835386732,0.7384262954752746,null,-0.003664383835386732,-0.9969655344852878,null,-0.003664383835386732,0.17095379673929695,null,-0.003664383835386732,0.6072010828837164,null,-0.003664383835386732,0.4209798964695524,null,-0.003664383835386732,-0.5936289138761188,null,-0.003664383835386732,-0.6932825766693715,null,-0.003664383835386732,0.4725388484490127,null,-0.003664383835386732,0.25195252197313583,null,-0.003664383835386732,-0.16284954169902313,null,-0.003664383835386732,0.473893477028527,null,-0.003664383835386732,0.16986990320831924,null,-0.003664383835386732,-0.8787013239360658,null,-0.003664383835386732,0.1595925623051145,null,-0.003664383835386732,0.9928641630335686,null,-0.003664383835386732,0.7801118446122239,null,-0.003664383835386732,0.6875814623136376,null,-0.003664383835386732,-0.26782126649187643,null,-0.003664383835386732,-0.5216316824182243,null,-0.003664383835386732,-0.5077923653766317,null,-0.003664383835386732,-0.4561086798575854,null,-0.003664383835386732,0.059364972895942246,null,-0.003664383835386732,0.8594822641045448,null,-0.003664383835386732,-0.19402275123499518,null,-0.003664383835386732,-0.6214868601895712,null,-0.003664383835386732,-0.7546233928837849,null,-0.003664383835386732,-0.26866047899247786,null,-0.003664383835386732,-0.9083465689399717,null,-0.003664383835386732,0.25645845862012007,null,-0.003664383835386732,-0.3687163863482623,null,-0.003664383835386732,-0.7920129474743768,null,-0.003664383835386732,0.8605521177721938,null,-0.003664383835386732,0.5659338428481465,null,-0.003664383835386732,-0.9433527144595178,null,-0.003664383835386732,0.24451847466835847,null,-0.003664383835386732,0.7394671370057544,null,-0.003664383835386732,-0.43454812219488953,null,-0.003664383835386732,0.7893660950205991,null,-0.003664383835386732,0.6191084387186672,null,-0.003664383835386732,0.5716624047307709,null,-0.003664383835386732,0.28701305770959007,null,-0.003664383835386732,-0.6942778593379838,null,-0.003664383835386732,0.0019845969967339145,null,-0.003664383835386732,-0.1574461698712373,null,-0.003664383835386732,-0.0361539932584737,null,-0.003664383835386732,0.17529235037000912,null,-0.003664383835386732,-0.20088603201959598,null,-0.003664383835386732,-0.285508885475217,null,-0.003664383835386732,0.7411235739800511,null,-0.003664383835386732,0.9992974908698582,null,-0.003664383835386732,0.08654229271517984,null,-0.003664383835386732,0.3010768489777451,null,-0.003664383835386732,0.0011688429310772557,null,-0.003664383835386732,-0.6142482073662215,null,-0.003664383835386732,-0.3120579651850063,null,-0.003664383835386732,-0.1058500030547879,null,-0.003664383835386732,-0.38586238446756993,null,-0.003664383835386732,0.7991210721930332,null,-0.003664383835386732,0.6440829028512312,null,-0.003664383835386732,-0.29089876597196024,null,-0.003664383835386732,-0.5599129021894044,null,-0.003664383835386732,-0.8052152913228532,null,-0.003664383835386732,-0.9341686745252313,null,-0.003664383835386732,0.7121037891456319,null,-0.003664383835386732,-0.5366441285812398,null,-0.003664383835386732,0.8076712294038146,null,-0.003664383835386732,-0.35373266530188724,null,-0.003664383835386732,0.16624426348998608,null,-0.003664383835386732,-0.39635772568207384,null,-0.003664383835386732,0.6573324982997658,null,-0.003664383835386732,0.827994511954231,null,-0.003664383835386732,0.44827116433749853,null,-0.003664383835386732,-0.6792313505290369,null,-0.003664383835386732,0.90433782726259,null,-0.003664383835386732,0.4522335946761477,null,-0.003664383835386732,0.6157885467987155,null,-0.003664383835386732,-0.10686712917120958,null,-0.003664383835386732,0.4868527644620576,null,-0.003664383835386732,0.47169102019555853,null,-0.003664383835386732,-0.4890040829127544,null,-0.003664383835386732,-0.3169122275022687,null,-0.003664383835386732,0.3795806225749202,null,-0.003664383835386732,0.2952899936545623,null,-0.003664383835386732,0.5034763966689422,null,-0.003664383835386732,-0.5233855427541211,null,-0.003664383835386732,-0.13451114111788062,null,-0.003664383835386732,-0.1663440729378771,null,-0.003664383835386732,-0.4547587017836652,null,-0.003664383835386732,-0.692070545136543,null,-0.003664383835386732,-0.6779204290918334,null,-0.003664383835386732,-0.06257626842618508,null,-0.003664383835386732,0.9185997102262662,null,-0.003664383835386732,-0.4448219245213286,null,-0.003664383835386732,-0.724800280122838,null,-0.003664383835386732,0.5603993139392208,null,-0.003664383835386732,-0.8454845826007374,null,-0.003664383835386732,0.38398230912885073,null,-0.003664383835386732,-0.1392604443431891,null,-0.003664383835386732,-0.04625032587862596,null,-0.003664383835386732,-0.10263659155904947,null,-0.003664383835386732,-0.355814324386098,null,-0.003664383835386732,0.45838167309397193,null,-0.003664383835386732,0.7698901468922484,null,-0.003664383835386732,0.739376478171297,null,-0.003664383835386732,-0.22134344592563665,null,-0.003664383835386732,-0.39698084237243814,null,-0.003664383835386732,0.7573998611461182,null,-0.003664383835386732,0.8720245233970815,null,-0.003664383835386732,0.5876221514571672,null,-0.003664383835386732,0.9038007997893813,null,-0.003664383835386732,-0.40892973741721644,null,-0.003664383835386732,0.5661394038407844,null,-0.003664383835386732,0.0026241023073839075,null,-0.003664383835386732,-0.16500790829097134,null,-0.003664383835386732,0.26275738151265154,null,-0.003664383835386732,0.32625355608864354,null,-0.003664383835386732,0.9745495160842154,null,-0.003664383835386732,-0.5146406862775602,null,-0.003664383835386732,-0.0572064312630983,null,-0.003664383835386732,0.3902691324716401,null,-0.003664383835386732,0.23071107321565318,null,-0.003664383835386732,0.5646567733235863,null,-0.003664383835386732,0.32564290406099805,null,-0.003664383835386732,0.48617956418454766,null,-0.003664383835386732,-0.6190612308086072,null,-0.003664383835386732,-0.7004807737391863,null,-0.003664383835386732,0.03237105146664455,null,-0.003664383835386732,-0.29328327208513494,null,-0.003664383835386732,0.9301795487961159,null,-0.003664383835386732,0.6520661432121445,null,-0.003664383835386732,-0.6131765969700248,null,-0.003664383835386732,-0.19886398533555996,null,-0.003664383835386732,-0.6111537182392702,null,-0.003664383835386732,-0.31973637199627447,null,-0.003664383835386732,0.3199414163685537,null,-0.003664383835386732,0.8436023693605786,null,-0.003664383835386732,-0.7249145733224273,null,-0.003664383835386732,0.5480344780399082,null,-0.003664383835386732,0.16211319973580152,null,-0.003664383835386732,-0.8431715316642797,null,-0.003664383835386732,0.21292854190994276,null,-0.003664383835386732,-0.48718469335460907,null,-0.003664383835386732,0.06627888013928453,null,-0.003664383835386732,-0.9698899446546609,null,-0.003664383835386732,0.11722430608301358,null,-0.003664383835386732,-0.19214970216256153,null,-0.003664383835386732,0.9725668307567964,null,-0.003664383835386732,0.9999999999999999,null,-0.003664383835386732,0.8808821749087321,null,-0.003664383835386732,0.15229682364370095,null,-0.003664383835386732,-0.780470016879927,null,-0.003664383835386732,-0.4016739594934336,null,-0.003664383835386732,0.7016018524001503,null,-0.003664383835386732,-0.2297283155381515,null,-0.003664383835386732,0.6955996733343242,null,-0.003664383835386732,0.016354901729295393,null,-0.003664383835386732,-0.6779239683490896,null,-0.003664383835386732,0.45204006507888084,null,-0.003664383835386732,0.42991717349972824,null,-0.003664383835386732,-0.2916801318369135,null,-0.003664383835386732,0.1043336805581651,null,-0.003664383835386732,0.9469527333854435,null,-0.003664383835386732,-0.9764915563300627,null,-0.003664383835386732,-0.3955561638267931,null,-0.003664383835386732,0.05344482518779311,null,-0.003664383835386732,0.30540514685904807,null,-0.003664383835386732,0.47778721692582726,null,-0.003664383835386732,-0.9084381756355043,null,-0.003664383835386732,-0.8704666889946392,null,-0.003664383835386732,0.6069796977259527,null,-0.003664383835386732,-0.7735953014091923,null,-0.003664383835386732,-0.20903521665075803,null,-0.003664383835386732,0.06347130638952075,null,-0.003664383835386732,-0.7733334484490111,null,-0.003664383835386732,0.600783480622705,null,-0.003664383835386732,-0.6159095119715511,null,-0.003664383835386732,-0.7082501736717892,null,-0.003664383835386732,0.90256222988986,null,-0.003664383835386732,0.34342516566303616,null,-0.003664383835386732,-0.5297046342414053,null,-0.003664383835386732,0.47928192260495456,null,-0.003664383835386732,-0.23806849034014052,null,-0.003664383835386732,-0.1217870852956012,null,-0.003664383835386732,0.18236498977817767,null,-0.003664383835386732,-0.6359736421517379,null,-0.003664383835386732,0.5026857951656882,null,-0.003664383835386732,-0.4497377447828778,null,-0.003664383835386732,-0.8309186106457478,null,-0.003664383835386732,-0.49372617036238076,null,-0.003664383835386732,0.2897022081182285,null,-0.003664383835386732,0.6401494431769582,null,-0.003664383835386732,0.6353186126580715,null,-0.003664383835386732,0.46208483079974544,null,-0.003664383835386732,-0.23155661810756897,null,-0.003664383835386732,-0.03843875877228894,null,-0.003664383835386732,-0.5371069408966253,null,-0.003664383835386732,-0.4521418883119377,null,-0.003664383835386732,0.0878673135284625,null,-0.003664383835386732,0.4670328170726349,null,-0.003664383835386732,-0.7205380196586034,null,-0.003664383835386732,-0.11587124703980788,null,-0.003664383835386732,0.55782594123151,null,-0.003664383835386732,-0.4569599606040535,null,-0.003664383835386732,0.6730479939700299,null,-0.003664383835386732,0.34988629953688327,null,-0.003664383835386732,0.9683952169240138,null,-0.003664383835386732,-0.4178178290724908,null,-0.003664383835386732,0.8847032267324255,null,-0.003664383835386732,0.7731660685671351,null,-0.003664383835386732,-0.626058775235016,null,-0.003664383835386732,0.4373196166236433,null,-0.003664383835386732,-0.9469280048162192,null,-0.003664383835386732,0.24044956433427825,null,-0.003664383835386732,-0.10929274247468412,null,-0.003664383835386732,0.6865673892010911,null,-0.003664383835386732,-0.5782481050493005,null,-0.003664383835386732,-0.31991201029741667,null,-0.003664383835386732,-0.22617042786829486,null,-0.003664383835386732,0.35119821847890037,null,-0.003664383835386732,0.694486495496413,null,-0.003664383835386732,0.7944069212767444,null,-0.003664383835386732,-0.9332566625138736,null,-0.003664383835386732,0.6226525572585242,null,-0.003664383835386732,-0.7980843332270959,null,-0.003664383835386732,-0.022403795594077004,null,-0.003664383835386732,0.3707785804501813,null,-0.003664383835386732,-0.5846557042901017,null,-0.003664383835386732,0.16624152077974086,null,-0.003664383835386732,0.3015833505170526,null,-0.003664383835386732,0.6810306208524924,null,-0.003664383835386732,0.09101348237478321,null,-0.003664383835386732,0.2664907073825142,null,-0.003664383835386732,-0.7995967302938521,null,-0.003664383835386732,-0.6090518698900738,null,-0.003664383835386732,-0.5250130361206208,null,-0.003664383835386732,-0.8172623785402786,null,-0.003664383835386732,-0.7104180480347054,null,-0.003664383835386732,0.09928091035636737,null,-0.003664383835386732,-0.8958869486734503,null,-0.003664383835386732,-0.838785879550208,null,-0.003664383835386732,-0.08391420731186497,null,-0.003664383835386732,0.32107365381943676,null,-0.003664383835386732,0.2041309127358124,null,-0.003664383835386732,0.650698950087515,null,-0.003664383835386732,-0.04381015085797339,null,-0.003664383835386732,0.8078006754648502,null,-0.003664383835386732,-0.9965334983254667,null,-0.003664383835386732,-0.34046595055775725,null,-0.003664383835386732,-0.2824022847745272,null],\"y\":[0.44017592531378724,-0.004253037203168265,null,-0.004253037203168265,0.14103995788723522,null,-0.004253037203168265,-0.6072719118322496,null,-0.004253037203168265,0.21571283467333666,null,-0.004253037203168265,-0.4785448882154157,null,-0.004253037203168265,-0.1278894118076058,null,-0.004253037203168265,0.8894479212571685,null,-0.004253037203168265,0.5158035321608121,null,-0.004253037203168265,-0.554711632223827,null,-0.004253037203168265,0.7902574115662622,null,-0.004253037203168265,-0.5092713341047813,null,-0.004253037203168265,-0.1289732812200516,null,-0.004253037203168265,0.42560588514579506,null,-0.004253037203168265,-0.7916476933584443,null,-0.004253037203168265,-0.399753284494228,null,-0.004253037203168265,0.1250730119687902,null,-0.004253037203168265,-0.9794950426306624,null,-0.004253037203168265,0.43991555208852196,null,-0.004253037203168265,-0.5529995737733082,null,-0.004253037203168265,0.5065062927340611,null,-0.004253037203168265,-0.657961627182063,null,-0.004253037203168265,-0.46350404355670577,null,-0.004253037203168265,-0.7387416777904008,null,-0.004253037203168265,-0.7723302068805082,null,-0.004253037203168265,-0.49386395484957035,null,-0.004253037203168265,-0.4141982403785277,null,-0.004253037203168265,0.7231795770090511,null,-0.004253037203168265,-0.6511158073728922,null,-0.004253037203168265,0.3911909531070655,null,-0.004253037203168265,-0.704915117380226,null,-0.004253037203168265,-0.9950574163608339,null,-0.004253037203168265,0.3104140288976242,null,-0.004253037203168265,0.4446544843594136,null,-0.004253037203168265,0.0861826037494336,null,-0.004253037203168265,-0.9803151607611926,null,-0.004253037203168265,-0.0622089956425805,null,-0.004253037203168265,-0.6319958971183769,null,-0.004253037203168265,0.5223701160978959,null,-0.004253037203168265,-0.7288219334679119,null,-0.004253037203168265,-0.87655794716452,null,-0.004253037203168265,-0.9344673672021069,null,-0.004253037203168265,-0.684241059144838,null,-0.004253037203168265,-0.7138112868804342,null,-0.004253037203168265,-0.598267315208027,null,-0.004253037203168265,-0.0719010299050823,null,-0.004253037203168265,-0.36977537836543317,null,-0.004253037203168265,-0.06385563155818084,null,-0.004253037203168265,-0.4494175667441863,null,-0.004253037203168265,-0.12813684075324172,null,-0.004253037203168265,-0.25530884613487426,null,-0.004253037203168265,0.587179963091485,null,-0.004253037203168265,0.06787507776301374,null,-0.004253037203168265,-0.8009789242079971,null,-0.004253037203168265,-0.25574087495753256,null,-0.004253037203168265,-0.14943618683572013,null,-0.004253037203168265,-0.5440954981900135,null,-0.004253037203168265,-0.0028082926948777547,null,-0.004253037203168265,-0.2926281625283434,null,-0.004253037203168265,-0.6391899407787116,null,-0.004253037203168265,-0.2938716769067696,null,-0.004253037203168265,0.8502695363619943,null,-0.004253037203168265,-0.7661989446864664,null,-0.004253037203168265,0.20241726393795187,null,-0.004253037203168265,0.07661660232830936,null,-0.004253037203168265,-0.606638666958268,null,-0.004253037203168265,0.01764491815498567,null,-0.004253037203168265,-0.23906897154123385,null,-0.004253037203168265,-0.5278893037805807,null,-0.004253037203168265,-0.6894716336337485,null,-0.004253037203168265,-0.21960984489501612,null,-0.004253037203168265,-0.2854373856005446,null,-0.004253037203168265,0.3934499487824517,null,-0.004253037203168265,0.23809079416687284,null,-0.004253037203168265,-0.20525343373150093,null,-0.004253037203168265,-0.5942118497313473,null,-0.004253037203168265,0.5125552495530654,null,-0.004253037203168265,0.9466240015380261,null,-0.004253037203168265,0.8109852136790028,null,-0.004253037203168265,0.4018610327199681,null,-0.004253037203168265,0.14998982867779453,null,-0.004253037203168265,0.6790958668739511,null,-0.004253037203168265,0.023542656489954437,null,-0.004253037203168265,-0.9877797669912587,null,-0.004253037203168265,-0.3822644999127505,null,-0.004253037203168265,-0.8009657280590783,null,-0.004253037203168265,-0.574203261091437,null,-0.004253037203168265,-0.4018117620881211,null,-0.004253037203168265,-0.5920629443144043,null,-0.004253037203168265,0.05534624361739632,null,-0.004253037203168265,0.5153203409709772,null,-0.004253037203168265,-0.5462328467088005,null,-0.004253037203168265,0.5305171152933182,null,-0.004253037203168265,-0.42735366304299205,null,-0.004253037203168265,0.35450056370857846,null,-0.004253037203168265,-0.37893641816727763,null,-0.004253037203168265,0.5692567881968835,null,-0.004253037203168265,-0.7671662450340726,null,-0.004253037203168265,-0.009018359207401353,null,-0.004253037203168265,0.42923640677456154,null,-0.004253037203168265,0.36460298687663856,null,-0.004253037203168265,0.7722612560146255,null,-0.004253037203168265,-0.7017140684453718,null,-0.004253037203168265,-0.3498338068224227,null,-0.004253037203168265,0.327766808062835,null,-0.004253037203168265,0.47096695967974256,null,-0.004253037203168265,0.4096575062942861,null,-0.004253037203168265,-0.7923842470600377,null,-0.004253037203168265,0.06765870466908323,null,-0.004253037203168265,-0.9961845015125069,null,-0.004253037203168265,0.7190187513263339,null,-0.004253037203168265,0.23354619462824194,null,-0.004253037203168265,-0.06554391564969426,null,-0.004253037203168265,0.7836373294246917,null,-0.004253037203168265,-0.8910731908074851,null,-0.004253037203168265,-0.48912524588602146,null,-0.004253037203168265,-0.30887261626429136,null,-0.004253037203168265,-0.32014967530617916,null,-0.004253037203168265,-0.9086531895650864,null,-0.004253037203168265,-0.4014292633093885,null,-0.004253037203168265,0.3662648637194452,null,-0.004253037203168265,-0.6049892146665214,null,-0.004253037203168265,-0.08532794099616438,null,-0.004253037203168265,0.29938791536751697,null,-0.004253037203168265,0.3370428819559644,null,-0.004253037203168265,0.200744796689128,null,-0.004253037203168265,-0.4632268745817167,null,-0.004253037203168265,0.8095489554367788,null,-0.004253037203168265,0.03203242816429645,null,-0.004253037203168265,-0.23848756812418698,null,-0.004253037203168265,0.8397862548854086,null,-0.004253037203168265,0.5669502167579,null,-0.004253037203168265,0.9460587972004362,null,-0.004253037203168265,-0.867776149604529,null,-0.004253037203168265,0.6037396603185048,null,-0.004253037203168265,-0.648134502932243,null,-0.004253037203168265,-0.34487691075501464,null,-0.004253037203168265,0.918981778826403,null,-0.004253037203168265,-0.2980865549433062,null,-0.004253037203168265,0.18301504747038191,null,-0.004253037203168265,0.48981658539707335,null,-0.004253037203168265,-0.39464963539576337,null,-0.004253037203168265,0.18619649866007681,null,-0.004253037203168265,-0.5065110962845326,null,-0.004253037203168265,0.6920061546667308,null,-0.004253037203168265,0.7073727074178204,null,-0.004253037203168265,0.9776510978934575,null,-0.004253037203168265,-0.7662864524449224,null,-0.004253037203168265,-0.5867160941181575,null,-0.004253037203168265,-0.1456317279696508,null,-0.004253037203168265,-0.6620408736554434,null,-0.004253037203168265,-0.8913080981929014,null,-0.004253037203168265,0.6941563647042297,null,-0.004253037203168265,-0.07940340066292229,null,-0.004253037203168265,-0.1785287764130181,null,-0.004253037203168265,-0.9581248985040242,null,-0.004253037203168265,0.8291406628944853,null,-0.004253037203168265,0.3615110749390955,null,-0.004253037203168265,0.060955748567813085,null,-0.004253037203168265,-0.9164314423928351,null,-0.004253037203168265,0.9742884675241353,null,-0.004253037203168265,-0.21232687549916174,null,-0.004253037203168265,0.706050423584356,null,-0.004253037203168265,-0.268232055132807,null,-0.004253037203168265,0.2507077399057154,null,-0.004253037203168265,0.22227824240980262,null,-0.004253037203168265,0.6749805964517884,null,-0.004253037203168265,0.9592977721910839,null,-0.004253037203168265,0.23593734554286344,null,-0.004253037203168265,0.6631383231360035,null,-0.004253037203168265,-0.8142466522185376,null,-0.004253037203168265,-0.4945633435315994,null,-0.004253037203168265,0.25889891051159464,null,-0.004253037203168265,-0.8715809875375613,null,-0.004253037203168265,-0.2045976848032944,null,-0.004253037203168265,-0.6762825218418502,null,-0.004253037203168265,-0.23832139779465386,null,-0.004253037203168265,0.8853099064910768,null,-0.004253037203168265,0.7241914658704749,null,-0.004253037203168265,-0.30200222108137154,null,-0.004253037203168265,0.1528609717916121,null,-0.004253037203168265,-0.47315511849684816,null,-0.004253037203168265,0.5900535384182796,null,-0.004253037203168265,-0.02935183625897673,null,-0.004253037203168265,0.9116575376320986,null,-0.004253037203168265,0.01278636769048076,null,-0.004253037203168265,0.628275403489265,null,-0.004253037203168265,-0.24440637846171995,null,-0.004253037203168265,0.9793406028397859,null,-0.004253037203168265,-0.36315906075046156,null,-0.004253037203168265,0.9009763165101574,null,-0.004253037203168265,-0.3773245580638285,null,-0.004253037203168265,-0.8945202269500249,null,-0.004253037203168265,-0.8716549758806368,null,-0.004253037203168265,0.08401203994533538,null,-0.004253037203168265,-0.1312313067886055,null,-0.004253037203168265,0.5989242645467977,null,-0.004253037203168265,0.545502920792337,null,-0.004253037203168265,0.4784171384937585,null,-0.004253037203168265,0.1187868920883475,null,-0.004253037203168265,-0.18004537250511493,null,-0.004253037203168265,0.11199857540438019,null,-0.004253037203168265,0.2943893973752063,null,-0.004253037203168265,0.45650140291296637,null,-0.004253037203168265,-0.9039765599848838,null,-0.004253037203168265,0.8066378985052864,null,-0.004253037203168265,-0.5635968722915559,null,-0.004253037203168265,0.16959055055090794,null,-0.004253037203168265,0.7983577678629691,null,-0.004253037203168265,0.2675794797351386,null,-0.004253037203168265,-0.035612797457463506,null,-0.004253037203168265,-0.8100254250487124,null,-0.004253037203168265,0.2888474363365588,null,-0.004253037203168265,-0.5604674659109123,null,-0.004253037203168265,-0.04813711520406763,null,-0.004253037203168265,-0.7875992610165888,null,-0.004253037203168265,0.4849033499255618,null,-0.004253037203168265,-0.47866201920715773,null,-0.004253037203168265,0.002308603160777466,null,-0.004253037203168265,0.5051446029096937,null,-0.004253037203168265,-0.3406446156114938,null,-0.004253037203168265,0.7453536638962995,null,-0.004253037203168265,0.11237974546350327,null,-0.004253037203168265,-0.7760557134243149,null,-0.004253037203168265,0.7835139658952022,null,-0.004253037203168265,-0.12792363066070045,null,-0.004253037203168265,0.8111693672986239,null,-0.004253037203168265,-0.4083411329230964,null,-0.004253037203168265,0.8398452356343937,null,-0.004253037203168265,0.6602534265183435,null,-0.004253037203168265,0.9731146782394222,null,-0.004253037203168265,0.4215142037853376,null,-0.004253037203168265,0.5520644843727025,null,-0.004253037203168265,-0.24969562499195302,null,-0.004253037203168265,0.545556262792238,null,-0.004253037203168265,0.846476606615287,null,-0.004253037203168265,0.37417547224854464,null,-0.004253037203168265,0.587650083324279,null,-0.004253037203168265,0.26545094262409424,null,-0.004253037203168265,-0.8928413527645339,null,-0.004253037203168265,-0.2796889314861692,null,-0.004253037203168265,0.10158101149389201,null,-0.004253037203168265,0.6261786887330393,null,-0.004253037203168265,-0.4680508687225495,null,-0.004253037203168265,0.2896864949282061,null,-0.004253037203168265,0.9459554487507844,null,-0.004253037203168265,0.643636335198148,null,-0.004253037203168265,0.23878054888329056,null,-0.004253037203168265,0.707463730362309,null,-0.004253037203168265,0.88440659233174,null,-0.004253037203168265,-0.8057719452507762,null,-0.004253037203168265,-0.7095208901486955,null,-0.004253037203168265,-0.6223688126898995,null,-0.004253037203168265,-0.09661619386189078,null,-0.004253037203168265,0.15162582472344963,null,-0.004253037203168265,0.6092555253446416,null,-0.004253037203168265,-0.421279657702642,null,-0.004253037203168265,0.8433875078819613,null,-0.004253037203168265,-0.012464819994273197,null,-0.004253037203168265,-0.13853445521891164,null,-0.004253037203168265,0.9681320358786382,null,-0.004253037203168265,0.7713052345881783,null,-0.004253037203168265,-0.4295535180623853,null,-0.004253037203168265,0.22663738651066054,null,-0.004253037203168265,0.6291519554808505,null,-0.004253037203168265,0.6024657461104744,null,-0.004253037203168265,-0.6720549811757285,null,-0.004253037203168265,-0.8673934626742303,null,-0.004253037203168265,-0.21339823330579577,null,-0.004253037203168265,0.3804346775951959,null,-0.004253037203168265,0.695131331121295,null,-0.004253037203168265,0.3592077397923904,null,-0.004253037203168265,-0.5292291304976039,null,-0.004253037203168265,0.7613926261052704,null,-0.004253037203168265,0.36147036391930465,null,-0.004253037203168265,0.7049813250966607,null,-0.004253037203168265,0.5007135896190802,null,-0.004253037203168265,-0.7087168167727923,null,-0.004253037203168265,0.6100809734569316,null,-0.004253037203168265,-0.050958629210161685,null,-0.004253037203168265,0.30813198107206863,null,-0.004253037203168265,-0.7404355907515572,null],\"type\":\"scatter\"},{\"hoverinfo\":\"text\",\"marker\":{\"colorscale\":[[0.0,\"rgb(255,255,217)\"],[0.125,\"rgb(237,248,177)\"],[0.25,\"rgb(199,233,180)\"],[0.375,\"rgb(127,205,187)\"],[0.5,\"rgb(65,182,196)\"],[0.625,\"rgb(29,145,192)\"],[0.75,\"rgb(34,94,168)\"],[0.875,\"rgb(37,52,148)\"],[1.0,\"rgb(8,29,88)\"]],\"showscale\":true,\"size\":10},\"mode\":\"markers\",\"x\":[-0.9054884960786804,-0.003664383835386732,-0.7401968271409917,-0.36983011804257965,-0.9701703516034483,0.563887969425789,-0.3642914136525418,0.3593378681916143,-0.835349913105015,0.8256732245883652,0.3845634436844661,-0.2893204441329471,-0.7867884707531642,0.00924935638097553,0.17493022996336047,0.8892696377507492,-0.6020865859624809,-0.24501927259392964,-0.5499472690432284,-0.5127403744393718,0.44317248068235615,0.24052583751913437,-0.8958395788671599,0.1141599999642776,-0.6129409354948676,0.03972117585742548,-0.4468440331187648,-0.6754671090703906,-0.7567385810093328,0.8169484432448167,0.5610063392592085,-0.014227952717972482,0.7885836808067992,0.7384262954752746,-0.9969655344852878,0.17095379673929695,0.6072010828837164,0.4209798964695524,-0.5936289138761188,-0.6932825766693715,0.4725388484490127,0.25195252197313583,-0.16284954169902313,0.473893477028527,0.16986990320831924,-0.8787013239360658,0.1595925623051145,0.9928641630335686,0.7801118446122239,0.6875814623136376,-0.26782126649187643,-0.5216316824182243,-0.5077923653766317,-0.4561086798575854,0.059364972895942246,0.8594822641045448,-0.19402275123499518,-0.6214868601895712,-0.7546233928837849,-0.26866047899247786,-0.9083465689399717,0.25645845862012007,-0.3687163863482623,-0.7920129474743768,0.8605521177721938,0.5659338428481465,-0.9433527144595178,0.24451847466835847,0.7394671370057544,-0.43454812219488953,0.7893660950205991,0.6191084387186672,0.5716624047307709,0.28701305770959007,-0.6942778593379838,0.0019845969967339145,-0.1574461698712373,-0.0361539932584737,0.17529235037000912,-0.20088603201959598,-0.285508885475217,0.7411235739800511,0.9992974908698582,0.08654229271517984,0.3010768489777451,0.0011688429310772557,-0.6142482073662215,-0.3120579651850063,-0.1058500030547879,-0.38586238446756993,0.7991210721930332,0.6440829028512312,-0.29089876597196024,-0.5599129021894044,-0.8052152913228532,-0.9341686745252313,0.7121037891456319,-0.5366441285812398,0.8076712294038146,-0.35373266530188724,0.16624426348998608,-0.39635772568207384,0.6573324982997658,0.827994511954231,0.44827116433749853,-0.6792313505290369,0.90433782726259,0.4522335946761477,0.6157885467987155,-0.10686712917120958,0.4868527644620576,0.47169102019555853,-0.4890040829127544,-0.3169122275022687,0.3795806225749202,0.2952899936545623,0.5034763966689422,-0.5233855427541211,-0.13451114111788062,-0.1663440729378771,-0.4547587017836652,-0.692070545136543,-0.6779204290918334,-0.06257626842618508,0.9185997102262662,-0.4448219245213286,-0.724800280122838,0.5603993139392208,-0.8454845826007374,0.38398230912885073,-0.1392604443431891,-0.04625032587862596,-0.10263659155904947,-0.355814324386098,0.45838167309397193,0.7698901468922484,0.739376478171297,-0.22134344592563665,-0.39698084237243814,0.7573998611461182,0.8720245233970815,0.5876221514571672,0.9038007997893813,-0.40892973741721644,0.5661394038407844,0.0026241023073839075,-0.16500790829097134,0.26275738151265154,0.32625355608864354,0.9745495160842154,-0.5146406862775602,-0.0572064312630983,0.3902691324716401,0.23071107321565318,0.5646567733235863,0.32564290406099805,0.48617956418454766,-0.6190612308086072,-0.7004807737391863,0.03237105146664455,-0.29328327208513494,0.9301795487961159,0.6520661432121445,-0.6131765969700248,-0.19886398533555996,-0.6111537182392702,-0.31973637199627447,0.3199414163685537,0.8436023693605786,-0.7249145733224273,0.5480344780399082,0.16211319973580152,-0.8431715316642797,0.21292854190994276,-0.48718469335460907,0.06627888013928453,-0.9698899446546609,0.11722430608301358,-0.19214970216256153,0.9725668307567964,0.9999999999999999,0.8808821749087321,0.15229682364370095,-0.780470016879927,-0.4016739594934336,0.7016018524001503,-0.2297283155381515,0.6955996733343242,0.016354901729295393,-0.6779239683490896,0.45204006507888084,0.42991717349972824,-0.2916801318369135,0.1043336805581651,0.9469527333854435,-0.9764915563300627,-0.3955561638267931,0.05344482518779311,0.30540514685904807,0.47778721692582726,-0.9084381756355043,-0.8704666889946392,0.6069796977259527,-0.7735953014091923,-0.20903521665075803,0.06347130638952075,-0.7733334484490111,0.600783480622705,-0.6159095119715511,-0.7082501736717892,0.90256222988986,0.34342516566303616,-0.5297046342414053,0.47928192260495456,-0.23806849034014052,-0.1217870852956012,0.18236498977817767,-0.6359736421517379,0.5026857951656882,-0.4497377447828778,-0.8309186106457478,-0.49372617036238076,0.2897022081182285,0.6401494431769582,0.6353186126580715,0.46208483079974544,-0.23155661810756897,-0.03843875877228894,-0.5371069408966253,-0.4521418883119377,0.0878673135284625,0.4670328170726349,-0.7205380196586034,-0.11587124703980788,0.55782594123151,-0.4569599606040535,0.6730479939700299,0.34988629953688327,0.9683952169240138,-0.4178178290724908,0.8847032267324255,0.7731660685671351,-0.626058775235016,0.4373196166236433,-0.9469280048162192,0.24044956433427825,-0.10929274247468412,0.6865673892010911,-0.5782481050493005,-0.31991201029741667,-0.22617042786829486,0.35119821847890037,0.694486495496413,0.7944069212767444,-0.9332566625138736,0.6226525572585242,-0.7980843332270959,-0.022403795594077004,0.3707785804501813,-0.5846557042901017,0.16624152077974086,0.3015833505170526,0.6810306208524924,0.09101348237478321,0.2664907073825142,-0.7995967302938521,-0.6090518698900738,-0.5250130361206208,-0.8172623785402786,-0.7104180480347054,0.09928091035636737,-0.8958869486734503,-0.838785879550208,-0.08391420731186497,0.32107365381943676,0.2041309127358124,0.650698950087515,-0.04381015085797339,0.8078006754648502,-0.9965334983254667,-0.34046595055775725,-0.2824022847745272],\"y\":[0.44017592531378724,-0.004253037203168265,0.14103995788723522,-0.6072719118322496,0.21571283467333666,-0.4785448882154157,-0.1278894118076058,0.8894479212571685,0.5158035321608121,-0.554711632223827,0.7902574115662622,-0.5092713341047813,-0.1289732812200516,0.42560588514579506,-0.7916476933584443,-0.399753284494228,0.1250730119687902,-0.9794950426306624,0.43991555208852196,-0.5529995737733082,0.5065062927340611,-0.657961627182063,-0.46350404355670577,-0.7387416777904008,-0.7723302068805082,-0.49386395484957035,-0.4141982403785277,0.7231795770090511,-0.6511158073728922,0.3911909531070655,-0.704915117380226,-0.9950574163608339,0.3104140288976242,0.4446544843594136,0.0861826037494336,-0.9803151607611926,-0.0622089956425805,-0.6319958971183769,0.5223701160978959,-0.7288219334679119,-0.87655794716452,-0.9344673672021069,-0.684241059144838,-0.7138112868804342,-0.598267315208027,-0.0719010299050823,-0.36977537836543317,-0.06385563155818084,-0.4494175667441863,-0.12813684075324172,-0.25530884613487426,0.587179963091485,0.06787507776301374,-0.8009789242079971,-0.25574087495753256,-0.14943618683572013,-0.5440954981900135,-0.0028082926948777547,-0.2926281625283434,-0.6391899407787116,-0.2938716769067696,0.8502695363619943,-0.7661989446864664,0.20241726393795187,0.07661660232830936,-0.606638666958268,0.01764491815498567,-0.23906897154123385,-0.5278893037805807,-0.6894716336337485,-0.21960984489501612,-0.2854373856005446,0.3934499487824517,0.23809079416687284,-0.20525343373150093,-0.5942118497313473,0.5125552495530654,0.9466240015380261,0.8109852136790028,0.4018610327199681,0.14998982867779453,0.6790958668739511,0.023542656489954437,-0.9877797669912587,-0.3822644999127505,-0.8009657280590783,-0.574203261091437,-0.4018117620881211,-0.5920629443144043,0.05534624361739632,0.5153203409709772,-0.5462328467088005,0.5305171152933182,-0.42735366304299205,0.35450056370857846,-0.37893641816727763,0.5692567881968835,-0.7671662450340726,-0.009018359207401353,0.42923640677456154,0.36460298687663856,0.7722612560146255,-0.7017140684453718,-0.3498338068224227,0.327766808062835,0.47096695967974256,0.4096575062942861,-0.7923842470600377,0.06765870466908323,-0.9961845015125069,0.7190187513263339,0.23354619462824194,-0.06554391564969426,0.7836373294246917,-0.8910731908074851,-0.48912524588602146,-0.30887261626429136,-0.32014967530617916,-0.9086531895650864,-0.4014292633093885,0.3662648637194452,-0.6049892146665214,-0.08532794099616438,0.29938791536751697,0.3370428819559644,0.200744796689128,-0.4632268745817167,0.8095489554367788,0.03203242816429645,-0.23848756812418698,0.8397862548854086,0.5669502167579,0.9460587972004362,-0.867776149604529,0.6037396603185048,-0.648134502932243,-0.34487691075501464,0.918981778826403,-0.2980865549433062,0.18301504747038191,0.48981658539707335,-0.39464963539576337,0.18619649866007681,-0.5065110962845326,0.6920061546667308,0.7073727074178204,0.9776510978934575,-0.7662864524449224,-0.5867160941181575,-0.1456317279696508,-0.6620408736554434,-0.8913080981929014,0.6941563647042297,-0.07940340066292229,-0.1785287764130181,-0.9581248985040242,0.8291406628944853,0.3615110749390955,0.060955748567813085,-0.9164314423928351,0.9742884675241353,-0.21232687549916174,0.706050423584356,-0.268232055132807,0.2507077399057154,0.22227824240980262,0.6749805964517884,0.9592977721910839,0.23593734554286344,0.6631383231360035,-0.8142466522185376,-0.4945633435315994,0.25889891051159464,-0.8715809875375613,-0.2045976848032944,-0.6762825218418502,-0.23832139779465386,0.8853099064910768,0.7241914658704749,-0.30200222108137154,0.1528609717916121,-0.47315511849684816,0.5900535384182796,-0.02935183625897673,0.9116575376320986,0.01278636769048076,0.628275403489265,-0.24440637846171995,0.9793406028397859,-0.36315906075046156,0.9009763165101574,-0.3773245580638285,-0.8945202269500249,-0.8716549758806368,0.08401203994533538,-0.1312313067886055,0.5989242645467977,0.545502920792337,0.4784171384937585,0.1187868920883475,-0.18004537250511493,0.11199857540438019,0.2943893973752063,0.45650140291296637,-0.9039765599848838,0.8066378985052864,-0.5635968722915559,0.16959055055090794,0.7983577678629691,0.2675794797351386,-0.035612797457463506,-0.8100254250487124,0.2888474363365588,-0.5604674659109123,-0.04813711520406763,-0.7875992610165888,0.4849033499255618,-0.47866201920715773,0.002308603160777466,0.5051446029096937,-0.3406446156114938,0.7453536638962995,0.11237974546350327,-0.7760557134243149,0.7835139658952022,-0.12792363066070045,0.8111693672986239,-0.4083411329230964,0.8398452356343937,0.6602534265183435,0.9731146782394222,0.4215142037853376,0.5520644843727025,-0.24969562499195302,0.545556262792238,0.846476606615287,0.37417547224854464,0.587650083324279,0.26545094262409424,-0.8928413527645339,-0.2796889314861692,0.10158101149389201,0.6261786887330393,-0.4680508687225495,0.2896864949282061,0.9459554487507844,0.643636335198148,0.23878054888329056,0.707463730362309,0.88440659233174,-0.8057719452507762,-0.7095208901486955,-0.6223688126898995,-0.09661619386189078,0.15162582472344963,0.6092555253446416,-0.421279657702642,0.8433875078819613,-0.012464819994273197,-0.13853445521891164,0.9681320358786382,0.7713052345881783,-0.4295535180623853,0.22663738651066054,0.6291519554808505,0.6024657461104744,-0.6720549811757285,-0.8673934626742303,-0.21339823330579577,0.3804346775951959,0.695131331121295,0.3592077397923904,-0.5292291304976039,0.7613926261052704,0.36147036391930465,0.7049813250966607,0.5007135896190802,-0.7087168167727923,0.6100809734569316,-0.050958629210161685,0.30813198107206863,-0.7404355907515572],\"type\":\"scatter\"}],                        {\"hovermode\":\"closest\",\"margin\":{\"b\":0,\"l\":0,\"r\":0,\"t\":0},\"showlegend\":false,\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('1977dd2d-85ea-4901-8a4d-4b7347ed5187');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "5.2 Topic Modeling"
      ],
      "metadata": {
        "id": "jv4g9P7DwCTg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('stopwords')\n",
        "\n",
        "def preprocess_text(text):\n",
        "    tokenizer = RegexpTokenizer(r'\\w+')\n",
        "    tokens = tokenizer.tokenize(text.lower())\n",
        "    filtered_tokens = [token for token in tokens if token not in stopwords.words('english')]\n",
        "    return filtered_tokens\n",
        "\n",
        "def perform_topic_modeling(text, num_topics=5, passes=15):\n",
        "    documents = text.split('\\n')\n",
        "    processed_docs = [preprocess_text(doc) for doc in documents if doc.strip()]\n",
        "    dictionary = corpora.Dictionary(processed_docs)\n",
        "    corpus = [dictionary.doc2bow(doc) for doc in processed_docs]\n",
        "    lda_model = LdaModel(corpus, num_topics=num_topics, id2word=dictionary, passes=passes)\n",
        "    return lda_model.print_topics(num_words=4)\n",
        "\n",
        "topics = perform_topic_modeling(extracted_text)\n",
        "\n",
        "# Print each topic line by line\n",
        "for i, topic in enumerate(topics):\n",
        "    print(f\"Topic {i+1}:\")\n",
        "    words = topic[1].split('+')\n",
        "    for word in words:\n",
        "        print(f\" - {word.strip()}\")\n",
        "    print()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dI8nBGBPwC-X",
        "outputId": "09eb8798-cb4f-4c0c-d4b5-6d29c80d6a3c"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Topic 1:\n",
            " - 0.029*\"community\"\n",
            " - 0.014*\"civic\"\n",
            " - 0.013*\"honors\"\n",
            " - 0.012*\"leadership\"\n",
            "\n",
            "Topic 2:\n",
            " - 0.063*\"community\"\n",
            " - 0.038*\"civic\"\n",
            " - 0.032*\"program\"\n",
            " - 0.031*\"honors\"\n",
            "\n",
            "Topic 3:\n",
            " - 0.042*\"program\"\n",
            " - 0.040*\"civic\"\n",
            " - 0.035*\"community\"\n",
            " - 0.032*\"honors\"\n",
            "\n",
            "Topic 4:\n",
            " - 0.035*\"program\"\n",
            " - 0.029*\"civic\"\n",
            " - 0.026*\"honors\"\n",
            " - 0.017*\"individuals\"\n",
            "\n",
            "Topic 5:\n",
            " - 0.062*\"community\"\n",
            " - 0.025*\"individuals\"\n",
            " - 0.025*\"technology\"\n",
            " - 0.016*\"within\"\n",
            "\n"
          ]
        }
      ]
    }
  ]
}