vignettes/03_extract_mammals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Step 4 - bringing the names together"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "At this point, we are ready to bring the different cleaned named together, in\n",
    "order to move on to the actual prediction. By the end of this step, we will\n",
    "have a cleaned name of mammals, corresponding to the tree, the Canadian\n",
    "species pool, and the European metaweb, all reconciled to the GBIF taxonomy\n",
    "backbone."
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "using Phylo\n",
    "using EcologicalNetworks\n",
    "using DelimitedFiles\n",
    "using CSV\n",
    "using DataFrames\n",
    "using ProgressMeter"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "We start by re-reading the tree from its nexus file - note that in the next\n",
    "steps, we will move away from `Phylo` to use `PhyloNetworks`, which handles\n",
    "the actual simulation of characters. But for name manipulation, `Phylo` is\n",
    "slightly easier to work with."
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "tree = open(parsenexus, joinpath(\"data\", \"mammals.nex\"))[\"*UNTITLED\"]"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can not read the file with the names of the European metaweb, their tree\n",
    "equivalent, and the corect GBIF name."
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "namelist = DataFrame(CSV.File(joinpath(\"artifacts\", \"names_metaweb_tree_gbif.csv\")))"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Prepare the metaweb from European data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will create a `UnipartiteNetwork` to store the European interactions. This\n",
    "is a much sparser version of the way the data are originally presented, and we\n",
    "will furthermore ensure that the species have the correct (*i.e.* GBIF) names.\n",
    "`EcologicalNetworks.jl` can handle having taxa objects from GBIF as nodes, but\n",
    "this is not something we will do here; strings are more than enough to do the\n",
    "matching, and we will not get back to the GBIF functions past this step."
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "mwspecies = unique(namelist.name)\n",
    "M = UnipartiteNetwork(zeros(Bool, length(mwspecies), length(mwspecies)), mwspecies)"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Filling the European metaweb"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We need to start by preparing a dictionary of values, linking species codes to\n",
    "species names (metaweb names, that is, not the correct GBIF names):"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "speciescodes = readdlm(joinpath(\"data\", \"Spp_Id.txt\"))[2:end, :]\n",
    "speciesdict = Dict([\n",
    "    speciescodes[i, 1] => speciescodes[i, 2] for i in 1:size(speciescodes, 1)\n",
    "])"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "The next step is to read the adjacency matrix for the European metaweb, which\n",
    "has species codes as identifiers, and `0` or `1` as values:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "mwlines = readlines(joinpath(\"data\", \"Metaweb_adults.csv\"))"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "This next line will read the elements of the first row, from columns 2 to the\n",
    "end, and replace the codes by the names - this is our sorted list of species\n",
    "we can use:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "mwhead = [speciesdict[sp] for sp in replace.(split(mwlines[1], \",\"), '\"' => \"\")[2:end]]"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "We then walk through the rows one by one, splitting them on the separator\n",
    "(`,`), and using the first element to identify the species. Everything that\n",
    "has a `1` is added as an interaction to the network object:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "for row in mwlines[2:end]\n",
    "    splitrow = replace.(split(row, \",\"), '\"' => \"\")\n",
    "    from = speciesdict[splitrow[1]]\n",
    "    realname = namelist[isequal(from).(namelist.metaweb), :name]\n",
    "    if length(realname) == 0\n",
    "        continue\n",
    "    else\n",
    "        sp_from = only(realname)\n",
    "        int = findall(isequal(\"1\"), splitrow[2:end])\n",
    "        if !isempty(int)\n",
    "            to = mwhead[int]\n",
    "            to_names = namelist[map(n -> n in to, namelist.metaweb), :name]\n",
    "            for t in to_names\n",
    "                M[sp_from, t] = true\n",
    "            end\n",
    "        end\n",
    "    end\n",
    "end"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, out of precaution, we drop the species without interactions (there\n",
    "are none), and drop the zeros from the sparse matrix in which interactions are\n",
    "stored:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "simplify!(M)"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "We then save this artifact as a much more readable CSV edgelist, which we will\n",
    "use for the rest of the analysis:"
   ],
   "metadata": {}
  },
  {
   "outputs": [],
   "cell_type": "code",
   "source": [
    "open(joinpath(\"artifacts\", \"europeanmetaweb.csv\"), \"w\") do euio\n",
    "    for int in sort(interactions(M); by=x -> x.from)\n",
    "        println(euio, \"$(int.from),$(int.to)\")\n",
    "    end\n",
    "end"
   ],
   "metadata": {},
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "---\n",
    "\n",
    "*This notebook was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).*"
   ],
   "metadata": {}
  }
 ],
 "nbformat_minor": 3,
 "metadata": {
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.2"
  },
  "kernelspec": {
   "name": "julia-1.6",
   "display_name": "Julia 1.6.2",
   "language": "julia"
  }
 },
 "nbformat": 4
}