USAGE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>First Step</h1>\n",
    "<p>The first step in using the scripts of this repository is to acquire the entities that contain the mashed dois. \n",
    "This can be done by utilising the Sparql endpoint of Opencitations Meta. This query allows the user to retrieve all of the journals inside of the endpoint which possess more than one Doi.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Placeholder\n",
    "# PREFIX datacite: <http://purl.org/spar/datacite/>\n",
    "# PREFIX dcterms: <http://purl.org/dc/terms/>\n",
    "# PREFIX fabio: <http://purl.org/spar/fabio/>\n",
    "\n",
    "# SELECT ?journal (COUNT(?identifier) AS ?doiCount)\n",
    "# WHERE {\n",
    "#   ?journal a fabio:Journal .\n",
    "#   ?journal datacite:hasIdentifier ?identifier .\n",
    "#   ?identifier datacite:usesIdentifierScheme datacite:doi .\n",
    "# }\n",
    "# GROUP BY ?journal\n",
    "# HAVING (COUNT(?identifier) > 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>After getting the results in a csv format we can start working on the entities we gathered from the query. The csv contains the url of the journals and their omid, and the number of dois assigned to that specific entity. By using the DoiFinder object we can retrieve the dois of the entities</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from DoiFinder import DOIFinder\n",
    "import pandas as pd\n",
    "import tqdm\n",
    "\n",
    "finder = DOIFinder()\n",
    "    \n",
    "# Read the CSV file containing journal URLs\n",
    "journal_doi_df = pd.read_csv(\"insert_file_path\", usecols=['journal'])\n",
    "\n",
    "# Iterate over each URL in the 'journal' column\n",
    "for idx, initial_url in tqdm(journal_doi_df['journal'].items(), desc=\"Processing journals\"):\n",
    "    initial_url = str(initial_url).strip()  # Clean the URL string\n",
    "        \n",
    "    # Check if it's a valid URL\n",
    "    if initial_url and (initial_url.startswith('http://') or initial_url.startswith('https://')):\n",
    "        # Find DOI links in the URL\n",
    "        doi_links = finder.find_doi_links(initial_url)\n",
    "            \n",
    "        # Save the found DOI links (if any) into the 'journal_doi' column\n",
    "        if doi_links:\n",
    "                # Save the first DOI link or concatenate multiple DOIs\n",
    "            journal_doi_df.at[idx, 'journal_doi'] = \", \".join(doi_links)\n",
    "    else:\n",
    "        print(f\"Invalid URL: {initial_url}\")\n",
    "\n",
    "# Save the updated DataFrame with the new 'journal_doi' column to a CSV file\n",
    "journal_doi_df.to_csv(\"insert_file_path\", index=False)\n",
    "print(\"Updated CSV file saved.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Now it's time to use the DOIOpener object to validate these dois and see which ones need to be deleted from the original entity, which ones point to other entities, and which ones belong to the entity they are assigned to. This process needs to be done manually since there isn't a way to automatically validate the Dois</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from doi_opener import DOIOpener\n",
    "# Insert the list of dois\n",
    "doi_list = []\n",
    "\n",
    "doi_opener = DOIOpener(doi_list)\n",
    "doi_opener.process_and_open_urls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Once the correct dois are gathered, we discard the ones that are either completely wrong or contain a 404 error, then we can proceed via the SPARQLCitationExtractor to retrieve all of the citations that point to the entities with mashed dois and the ones where the entities with the mashed dois point to other entities. After we can reuse the Doifinder object to find the dois of the citing and cited entities.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from citation_finder import SPARQLCitationExtractor\n",
    "# insert the csv containing the urls of the entities\n",
    "extractor = SPARQLCitationExtractor(\n",
    "    endpoint_url=\"https://opencitations.net/index/sparql\",\n",
    "    input_csv_path=\"insert_path\",\n",
    "    output_directory=\"insert_directory\"\n",
    ")\n",
    "    \n",
    "extractor.extract_citing_data()\n",
    "extractor.extract_cited_data()\n",
    "extractor.merge_citing_and_cited_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Now we have all of the citations regarding the mashed entities which need to be removed later. \n",
    "We can proceed to gather the metadata of the mashed entities with the MetadataGatherer object. This object needs to be used with a semiautomatic approach since it retrieves the html tags from the page which contains the metadata that we need for the upload to OpenCitations Meta, change the tags to be retrieved inside of the script and proceed for cases.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Before continuing we need to address the problem of the dois that point to an entity different then the one we assigned to, and to do this after opening the dois and verifying that they are indeed articles and not journals we can retrieve their citations and their metadata, this way we can recrete the connections to other entities and upload the new entities later on OpenCitations Meta. The DOIProcessor object can gather the metadata of these articles and their citations.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from metadata_researcher import MetadataGatherer\n",
    "    # Replace the list below with the DOIs you want to process, or with a file containing dois.\n",
    "example_dois = [\n",
    "    \"10.1234/example-doi-1\",\n",
    "    \"10.5678/example-doi-2\"\n",
    "    ]\n",
    "\n",
    "# Create a DOIProcessor instance\n",
    "processor = MetadataGatherer(example_dois)\n",
    "\n",
    "# Process DOIs and save metadata to a JSON file\n",
    "processor.process_dois(output_file=\"insert_file_path\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Now that we have the metadata regarding the entities that have mashed doi, we can proceed by gathering the metadata of the entities cited by these entities with the CrossRefProcessor object </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cited_articles_metadata_gatherer import CrossRefProcessor\n",
    "processor = CrossRefProcessor(\n",
    "    input_csv=\"insert_file_path\",\n",
    "    output_json=\"insert_file_path\",\n",
    "    output_csv=\"insert_file_path\",\n",
    "    filtered_json=\"insert_file_path\"\n",
    ")\n",
    "    \n",
    "# Process data and save results\n",
    "processor.process_data()\n",
    "    \n",
    "# Filter and save referenced DOIs\n",
    "processor.filter_referenced_dois()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Now the elements to reconnect all of the entities are all present we just need to order them and reconnect them. The DOIMatcher object will help us by connecting the Entities that cite the mashed entities to the mashed entities</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from citing_doi_matcher import DOIMatcher\n",
    "processor = DOIMatcher(\n",
    "    csv_file=\"insert_file_path\",\n",
    "    json_file=\"insert_file_path\",\n",
    "    output_file=\"insert_file_path\"\n",
    ")\n",
    "\n",
    "processor.process()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>The remaining operations regard the upload to Meta and the restablishment of the connections between the entities, which will be done in time.\n",
    "Remaining operations: upload the entities on the new version of meta; retrieve their omid, match the omid to other identifiers; recreate the connections between the entities and the citations.</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}