CivicHonorsKGv10.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMVBycWJq5yTMGR1f6UzQ3B",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nelslindahlx/KnowledgeReduce/blob/main/CivicHonorsKGv10.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Code Summary\n",
        "\n",
        "The code developed for execution in Google Colab represents a comprehensive workflow for extracting, processing, and organizing data from a website into a knowledge graph. Here's a summary of each step in the code:\n",
        "\n",
        "### Step 1: Install Necessary Libraries\n",
        "- Installs the required Python libraries: `requests` for web scraping, `beautifulsoup4` for HTML parsing, `networkx` for graph operations, and `spacy` for natural language processing.\n",
        "\n",
        "### Step 2: Import Libraries and Define Classes\n",
        "- Imports essential Python modules and defines two classes:\n",
        "  - `ReliabilityRating`: An enumeration (Enum) for categorizing the reliability of facts.\n",
        "  - `KnowledgeGraph`: A class for creating and managing a knowledge graph, including methods for adding facts, calculating quality scores, updating these scores, and retrieving facts.\n",
        "\n",
        "### Step 3: Scrape Data from the Website\n",
        "- Fetches and parses the HTML content of a specified website, extracting text data from common HTML elements like paragraphs, headings, and lists.\n",
        "\n",
        "### Step 4: Store Extracted Data in the KnowledgeGraph\n",
        "- Initializes an instance of the `KnowledgeGraph`.\n",
        "- Stores each extracted fact as a node in the graph with attributes such as fact statement, category, tags, etc.\n",
        "\n",
        "### Step 5: Retrieve and Display 10 Facts\n",
        "- Displays the first 10 facts from the knowledge graph, giving an overview of the extracted data, and prints the total number of facts extracted.\n",
        "\n",
        "### Step 6: Store the KnowledgeGraph in GEXF and GraphML Formats\n",
        "- Saves the graph in GEXF and GraphML file formats, suitable for analysis and visualization in various graph analysis tools.\n",
        "\n",
        "### Step 7: Ensure Uniqueness of Facts in the Dataset\n",
        "- Removes duplicate facts from the knowledge graph based on their statements to maintain data uniqueness.\n",
        "\n",
        "### Step 8: Advanced Cleaning and Combining of Facts\n",
        "- Implements advanced cleaning techniques, such as removing very short facts and deduplicating semantically similar facts using basic string comparison.\n",
        "\n",
        "### Step 9: Super Aggressive Advanced Cleaning (Refined)\n",
        "- Employs the `en_core_web_md` spaCy model for NLP tasks.\n",
        "- Removes facts based on complex criteria, like semantic similarity, using NLP techniques to refine the dataset's quality further.\n",
        "\n",
        "This complete workflow effectively extracts information from a website, transforming it into a structured and clean dataset within a knowledge graph. It's particularly useful in scenarios where data quality, structure, and semantic richness are crucial. The inclusion of advanced NLP techniques in the later steps ensures a high-quality dataset, free from redundancies and rich in diverse information."
      ],
      "metadata": {
        "id": "cLVuQtowdT4j"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 1: Install Necessary Libraries"
      ],
      "metadata": {
        "id": "61hozs7HZsLB"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "clRygd5kZpFj",
        "outputId": "7ce240a2-b7c0-475f-c611-1a78c9642069"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.31.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (4.11.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (3.2.1)\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (3.6.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests) (2023.11.17)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4) (2.5)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.10)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.8)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy) (8.1.12)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.10)\n",
            "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.9.0)\n",
            "Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.10.3)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (6.4.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (4.66.1)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.23.5)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.10.13)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy) (67.7.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (23.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.5.0)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.1.4)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.10.0,>=0.3.0->spacy) (8.1.7)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy) (2.1.3)\n",
            "2023-12-20 13:34:17.347339: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2023-12-20 13:34:17.347400: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2023-12-20 13:34:17.349129: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2023-12-20 13:34:17.361504: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-12-20 13:34:19.264152: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "Collecting en-core-web-md==3.6.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.6.0/en_core_web_md-3.6.0-py3-none-any.whl (42.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.8/42.8 MB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.7.0,>=3.6.0 in /usr/local/lib/python3.10/dist-packages (from en-core-web-md==3.6.0) (3.6.1)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.0.10)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.0.8)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (8.1.12)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.1.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.0.10)\n",
            "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.9.0)\n",
            "Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.10.3)\n",
            "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (6.4.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (4.66.1)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.23.5)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.31.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.10.13)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (67.7.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (23.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.3.0)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (4.5.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2023.11.17)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.1.4)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.10.0,>=0.3.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (8.1.7)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.1.3)\n",
            "Installing collected packages: en-core-web-md\n",
            "Successfully installed en-core-web-md-3.6.0\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_md')\n"
          ]
        }
      ],
      "source": [
        "!pip install requests beautifulsoup4 networkx spacy\n",
        "!python -m spacy download en_core_web_md\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 2: Import Libraries and Define Classes"
      ],
      "metadata": {
        "id": "XD_VIFu-Zv2S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import networkx as nx\n",
        "from datetime import datetime\n",
        "from enum import Enum\n",
        "import difflib\n",
        "import spacy\n",
        "\n",
        "class ReliabilityRating(Enum):\n",
        "    UNVERIFIED = 1\n",
        "    POSSIBLY_TRUE = 2\n",
        "    LIKELY_TRUE = 3\n",
        "    VERIFIED = 4\n",
        "\n",
        "class KnowledgeGraph:\n",
        "    def __init__(self):\n",
        "        self.graph = nx.DiGraph()\n",
        "\n",
        "    def calculate_quality_score(self, reliability_rating, usage_count):\n",
        "        # Adjusted to handle string representation of Enum\n",
        "        rating_value = ReliabilityRating[reliability_rating].value if isinstance(reliability_rating, str) else reliability_rating.value\n",
        "        base_score = 10 * rating_value\n",
        "        usage_bonus = 2 * usage_count\n",
        "        return base_score + usage_bonus\n",
        "\n",
        "    def add_fact(self, fact_id, fact_statement, category, tags, date_recorded, last_updated,\n",
        "                 reliability_rating, source_id, source_title, author_creator,\n",
        "                 publication_date, url_reference, related_facts, contextual_notes,\n",
        "                 access_level, usage_count):\n",
        "        # Convert list and datetime objects to strings\n",
        "        tags_str = ', '.join(tags) if tags else ''\n",
        "        date_recorded_str = date_recorded.isoformat() if isinstance(date_recorded, datetime) else date_recorded\n",
        "        last_updated_str = last_updated.isoformat() if isinstance(last_updated, datetime) else last_updated\n",
        "        publication_date_str = publication_date.isoformat() if isinstance(publication_date, datetime) else publication_date\n",
        "\n",
        "        quality_score = self.calculate_quality_score(reliability_rating, usage_count)\n",
        "        self.graph.add_node(fact_id,\n",
        "                            fact_statement=fact_statement,\n",
        "                            category=category,\n",
        "                            tags=tags_str,\n",
        "                            date_recorded=date_recorded_str,\n",
        "                            last_updated=last_updated_str,\n",
        "                            reliability_rating=reliability_rating,\n",
        "                            quality_score=quality_score,\n",
        "                            source_id=source_id,\n",
        "                            source_title=source_title,\n",
        "                            author_creator=author_creator,\n",
        "                            publication_date=publication_date_str,\n",
        "                            url_reference=url_reference,\n",
        "                            contextual_notes=contextual_notes,\n",
        "                            access_level=access_level,\n",
        "                            usage_count=usage_count)\n",
        "\n",
        "        for related_fact_id in related_facts:\n",
        "            self.graph.add_edge(fact_id, related_fact_id)\n",
        "\n",
        "    def update_quality_score(self, fact_id):\n",
        "        if fact_id not in self.graph:\n",
        "            raise ValueError(\"Fact ID not found in the graph.\")\n",
        "        fact = self.graph.nodes[fact_id]\n",
        "        new_score = self.calculate_quality_score(fact['reliability_rating'], fact['usage_count'])\n",
        "        self.graph.nodes[fact_id]['quality_score'] = new_score\n",
        "\n",
        "    def get_fact(self, fact_id):\n",
        "        if fact_id not in self.graph:\n",
        "            raise ValueError(\"Fact ID not found in the graph.\")\n",
        "        return self.graph.nodes[fact_id]"
      ],
      "metadata": {
        "id": "mw1-i2ZzZ1JH"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 3: Scrape Data from the Website"
      ],
      "metadata": {
        "id": "Rn6RrXg_Z55m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Fetch data from the website\n",
        "url = \"https://civichonors.com/\"\n",
        "response = requests.get(url)\n",
        "soup = BeautifulSoup(response.text, 'html.parser')\n",
        "\n",
        "# Function to extract text from a soup object\n",
        "def extract_text(element):\n",
        "    return ' '.join(element.stripped_strings)\n",
        "\n",
        "# Generic function to find facts in common HTML structures\n",
        "def find_facts(soup):\n",
        "    facts = []\n",
        "\n",
        "    # Look for paragraphs\n",
        "    for p in soup.find_all('p'):\n",
        "        text = extract_text(p)\n",
        "        if text: facts.append(text)\n",
        "\n",
        "    # Look for headings\n",
        "    for header_tag in ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']:\n",
        "        for header in soup.find_all(header_tag):\n",
        "            text = extract_text(header)\n",
        "            if text: facts.append(text)\n",
        "\n",
        "    # Look for list items\n",
        "    for li in soup.find_all('li'):\n",
        "        text = extract_text(li)\n",
        "        if text: facts.append(text)\n",
        "\n",
        "    return facts\n",
        "\n",
        "# Extract facts using the generic function\n",
        "facts = find_facts(soup)"
      ],
      "metadata": {
        "id": "neHkbBnWZ_8K"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 4: Store Extracted Data in the KnowledgeGraph"
      ],
      "metadata": {
        "id": "uOUJY03SaBwF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize the KnowledgeGraph\n",
        "kg = KnowledgeGraph()\n",
        "\n",
        "# Store each fact in the KnowledgeGraph\n",
        "for i, fact in enumerate(facts):\n",
        "    kg.add_fact(\n",
        "        fact_id=str(i),\n",
        "        fact_statement=fact,\n",
        "        category=\"General\",\n",
        "        tags=[\"CivicHonors\", \"WebScraped\"],\n",
        "        date_recorded=datetime.now(),\n",
        "        last_updated=datetime.now(),\n",
        "        reliability_rating=ReliabilityRating.LIKELY_TRUE.name,\n",
        "        source_id=\"CivicHonors\",\n",
        "        source_title=\"Civic Honors Website\",\n",
        "        author_creator=\"Web Scraping\",\n",
        "        publication_date=datetime.now(),\n",
        "        url_reference=url,\n",
        "        related_facts=[],\n",
        "        contextual_notes=\"Extracted from Civic Honors website\",\n",
        "        access_level=\"Public\",\n",
        "        usage_count=0\n",
        "    )"
      ],
      "metadata": {
        "id": "GjxxQZSeaHeR"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 5: Retrieve and Display 10 Facts"
      ],
      "metadata": {
        "id": "S572InVIaIlk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Print the total number of facts extracted\n",
        "total_facts = len(facts)\n",
        "print(f\"Total facts extracted: {total_facts}\")\n",
        "\n",
        "# Retrieve and display up to 10 facts\n",
        "for i in range(min(10, total_facts)):\n",
        "    fact_id = str(i)\n",
        "    try:\n",
        "        fact = kg.get_fact(fact_id)\n",
        "        print(f\"Fact {i+1}: {fact}\")\n",
        "    except ValueError as e:\n",
        "        print(f\"Fact {i+1}: {str(e)}\")\n",
        "        break  # Stop the loop if a fact ID is not found"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8GhKDJF9aMjW",
        "outputId": "56a4f44b-5bb3-4e57-e508-47cdaf1e4dd3"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total facts extracted: 310\n",
            "Fact 1: {'fact_statement': 'Civic Honors', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321628', 'last_updated': '2023-12-20T13:34:52.321632', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321644', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 2: {'fact_statement': 'Graduation with Civic Honors: Unlock the Power of  Community Opportunity', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321700', 'last_updated': '2023-12-20T13:34:52.321701', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321703', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 3: {'fact_statement': 'This book was published in 2006. It has a formal copyright. You can buy a physical copy if you want or just read it online here. I was going to update this to be an online PDF, but it seemed like a better idea to just make it a very large single page of prose. Enjoy!', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321727', 'last_updated': '2023-12-20T13:34:52.321728', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321729', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 4: {'fact_statement': 'Dr. Nels Lindahl, June 14, 2020, Denver, Colorado', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321750', 'last_updated': '2023-12-20T13:34:52.321751', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321753', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 5: {'fact_statement': 'Believing in a dream like graduation with civic honors is only the first step in the process toward advocating the creation of a graduation with civic honors program.', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321771', 'last_updated': '2023-12-20T13:34:52.321772', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321774', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 6: {'fact_statement': 'The story behind graduation with civic honors\\xa0began in an academic setting during the spring semester of 2002 at the University of Kansas. I took a class from Dr. H. George Frederickson\\xa0entitled Concepts of Civil Society. During the Concepts of Civil Society class, my collegiate interests focused on civic engagement.\\xa0At one point during the class, Dr. Frederickson posed a question to the class about what colleges could do to get people involved within the community. Over the course of the next year thinking about the idea of civil society, I became interested in pursuing a degree in the field of public administration. Thanks to the faculty of the Public Administration Department at the University of Kansas, I decided to endeavor to enter graduate school. I would like to thank Dr. Raymond Davis for advice and guidance, Dr. Thomas Longoria\\xa0for defining the importance of collaboration, and Dr. H. George Frederickson\\xa0for a thoughtful introduction to the world of civil society.', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321792', 'last_updated': '2023-12-20T13:34:52.321793', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321795', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 7: {'fact_statement': 'Graduation with civic honors was only a dream until late in 2002. Dr. Charles J. Carlsen, the president\\xa0of Johnson County Community College,\\xa0learned of the idea from Susan Lindahl and, along with the board of trustees and civic honors steering committee, envisioned becoming the first community college in the state of Kansas to designate a graduation with civic honors. In 2004, after Johnson County Community College developed the initial graduation with civic honors\\xa0pilot program, the next step was to start spreading the graduation with civic honors\\xa0message. I was honored to submit an article with Dr. Charles J. Carlsen and Susan Lindahl entitled, “Civic Honors Program at Johnson County Community College,” to the Journal for Civic Commitment for publication. The resulting publication (Carlsen, Lindahl, & Lindahl, 2004) was the first step toward globally sharing the positive message of graduation with civic honors.', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321817', 'last_updated': '2023-12-20T13:34:52.321818', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321820', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 8: {'fact_statement': 'To realize the dream, students actually would have to graduate with civic honors. During the May 2005 graduation at Johnson County Community College, the dream became a reality when four students—including Deborah DeGrate, Carrie Donham, Chris Engle, and Jennifer Pittman-Leeper—were the first to graduate with civic honors. The story will be complete when students all over the world are graduating with civic honors.', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321840', 'last_updated': '2023-12-20T13:34:52.321841', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321843', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 9: {'fact_statement': 'Society has experienced a true revival of public interest in civic engagement.', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321863', 'last_updated': '2023-12-20T13:34:52.321863', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321865', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n",
            "Fact 10: {'fact_statement': 'Robert Putnam’s (2000) epic work Bowling Alone brought the idea of civic participation\\xa0to the forefront of the public mind. The value of civic participation\\xa0is essential to the process of value implementation in the form of civic engagement. Civic engagement is a value choice, and the implementation of that value choice is individual civic participation\\xa0in the community. One of the most basic definitions of civic engagement\\xa0involves thinking about how government, society, and citizens interact. In terms of how scholars discuss civic engagement,\\xa0the definition takes on a benevolent feel, referencing citizen activities that benefit civil society. Strengthening civic engagement, in practice, involves building civic skills, increasing active voter participation, and using public service announcements to encourage volunteering\\xa0to strengthen civil society. Watershed events like 9-11 have created a reflective sense of national interest and significantly increased the willingness of individuals to participate in civic engagement.', 'category': 'General', 'tags': 'CivicHonors, WebScraped', 'date_recorded': '2023-12-20T13:34:52.321884', 'last_updated': '2023-12-20T13:34:52.321885', 'reliability_rating': 'LIKELY_TRUE', 'quality_score': 30, 'source_id': 'CivicHonors', 'source_title': 'Civic Honors Website', 'author_creator': 'Web Scraping', 'publication_date': '2023-12-20T13:34:52.321887', 'url_reference': 'https://civichonors.com/', 'contextual_notes': 'Extracted from Civic Honors website', 'access_level': 'Public', 'usage_count': 0}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 6: Store the KnowledgeGraph in GEXF and GraphML Formats"
      ],
      "metadata": {
        "id": "tJHI4T28a_6d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Store the graph in GEXF format\n",
        "gexf_path = '/content/knowledge_graph.gexf'\n",
        "nx.write_gexf(kg.graph, gexf_path)\n",
        "print(f\"Graph stored in GEXF format at: {gexf_path}\")\n",
        "\n",
        "# Store the graph in GraphML format\n",
        "graphml_path = '/content/knowledge_graph.graphml'\n",
        "nx.write_graphml(kg.graph, graphml_path)\n",
        "print(f\"Graph stored in GraphML format at: {graphml_path}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dC6G0nFKbC33",
        "outputId": "60f3f35b-d57a-4f1a-bf38-8ab051e83d4a"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Graph stored in GEXF format at: /content/knowledge_graph.gexf\n",
            "Graph stored in GraphML format at: /content/knowledge_graph.graphml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 7: Ensure Uniqueness of Facts in the Dataset"
      ],
      "metadata": {
        "id": "ceqgxEGxUxdM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def remove_duplicate_facts(knowledge_graph):\n",
        "    unique_facts = set()\n",
        "    nodes_to_remove = []\n",
        "\n",
        "    for fact_id, fact_data in knowledge_graph.graph.nodes(data=True):\n",
        "        fact_statement = fact_data['fact_statement']\n",
        "        if fact_statement in unique_facts:\n",
        "            nodes_to_remove.append(fact_id)\n",
        "        else:\n",
        "            unique_facts.add(fact_statement)\n",
        "\n",
        "    # Remove duplicate nodes\n",
        "    for fact_id in nodes_to_remove:\n",
        "        knowledge_graph.graph.remove_node(fact_id)\n",
        "\n",
        "    print(f\"Removed {len(nodes_to_remove)} duplicate facts.\")\n",
        "\n",
        "# Call the function to remove duplicate facts\n",
        "remove_duplicate_facts(kg)\n",
        "\n",
        "# Optional: Print the total number of unique facts remaining\n",
        "print(f\"Total unique facts remaining: {kg.graph.number_of_nodes()}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z0ZB11K_U0Gc",
        "outputId": "df0d131c-76ec-437d-dc0d-effad864b208"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Removed 0 duplicate facts.\n",
            "Total unique facts remaining: 310\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 8: Advanced Cleaning and Combining of Facts"
      ],
      "metadata": {
        "id": "C0ZkAAb8VCBh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def advanced_cleaning(knowledge_graph, similarity_threshold=0.8, short_fact_threshold=20):\n",
        "    nodes_to_remove = []\n",
        "    facts_processed = set()\n",
        "\n",
        "    # Remove short facts\n",
        "    for fact_id, fact_data in knowledge_graph.graph.nodes(data=True):\n",
        "        if len(fact_data['fact_statement']) < short_fact_threshold:\n",
        "            nodes_to_remove.append(fact_id)\n",
        "            continue\n",
        "\n",
        "        # Check for similarity with already processed facts\n",
        "        for processed_fact in facts_processed:\n",
        "            similarity = difflib.SequenceMatcher(None, fact_data['fact_statement'], processed_fact).ratio()\n",
        "            if similarity > similarity_threshold:\n",
        "                nodes_to_remove.append(fact_id)\n",
        "                break\n",
        "\n",
        "        facts_processed.add(fact_data['fact_statement'])\n",
        "\n",
        "    # Further custom strategies can be added here\n",
        "\n",
        "    # Remove identified nodes\n",
        "    for fact_id in nodes_to_remove:\n",
        "        knowledge_graph.graph.remove_node(fact_id)\n",
        "\n",
        "    print(f\"Removed {len(nodes_to_remove)} facts based on advanced cleaning criteria.\")\n",
        "\n",
        "# Call the function for advanced cleaning\n",
        "advanced_cleaning(kg)\n",
        "\n",
        "# Optional: Print the total number of facts after cleaning\n",
        "print(f\"Total facts after advanced cleaning: {kg.graph.number_of_nodes()}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UoMDWAoSU06N",
        "outputId": "6e7be353-fa4c-4c6e-9ed0-26b99c78da70"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Removed 5 facts based on advanced cleaning criteria.\n",
            "Total facts after advanced cleaning: 305\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 9: Super Aggressive Advanced Cleaning (Refined)"
      ],
      "metadata": {
        "id": "_EJymWj3Yv75"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the medium-sized English language model with word vectors\n",
        "nlp = spacy.load(\"en_core_web_md\")\n",
        "\n",
        "def super_aggressive_cleaning(knowledge_graph, similarity_threshold=0.85):\n",
        "    nodes_to_remove = set()\n",
        "    facts_content = {}\n",
        "\n",
        "    # Pre-process and store spaCy Doc objects for each fact\n",
        "    for fact_id, fact_data in knowledge_graph.graph.nodes(data=True):\n",
        "        try:\n",
        "            doc = nlp(fact_data['fact_statement'])\n",
        "            facts_content[fact_id] = doc\n",
        "        except Exception as e:\n",
        "            print(f\"Error processing fact ID {fact_id}: {str(e)}\")\n",
        "\n",
        "    # Compare each fact with others for semantic similarity\n",
        "    for fact_id, doc in facts_content.items():\n",
        "        # Skip if already marked for removal\n",
        "        if fact_id in nodes_to_remove:\n",
        "            continue\n",
        "\n",
        "        for other_fact_id, other_doc in facts_content.items():\n",
        "            if fact_id != other_fact_id and other_fact_id not in nodes_to_remove:\n",
        "                similarity = doc.similarity(other_doc)\n",
        "                if similarity > similarity_threshold:\n",
        "                    nodes_to_remove.add(other_fact_id)\n",
        "\n",
        "    # Remove identified nodes\n",
        "    for fact_id in nodes_to_remove:\n",
        "        knowledge_graph.graph.remove_node(fact_id)\n",
        "\n",
        "    print(f\"Removed {len(nodes_to_remove)} facts based on super aggressive cleaning criteria.\")\n",
        "\n",
        "# Call the function for super aggressive cleaning\n",
        "super_aggressive_cleaning(kg)\n",
        "\n",
        "# Print the total number of facts after super aggressive cleaning\n",
        "print(f\"Total facts after super aggressive cleaning: {kg.graph.number_of_nodes()}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u4SxFiWpYyRt",
        "outputId": "ddbbf58c-a2db-424d-f729-f189f1bc227d"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Removed 268 facts based on super aggressive cleaning criteria.\n",
            "Total facts after super aggressive cleaning: 37\n"
          ]
        }
      ]
    }
  ]
}