ML3science CM110 contribution

README.md

@@ -28,16 +28,16 @@ We are using the surveys of the WorldBank as our true gold standart. You have to
 
 - [0_download_country_codes](src/0_lsms_processing/0_download_country_codes.ipynb): Download the country codes for all Sub Saharian African countries from the WorldBank API to use the same country codes.
 - [1_check_lsms_availability](src/0_lsms_processing/1_check_lsms_availability.ipynb): Checks the availability of the LSMS for the given countries.
-- [2_consent_lsms_form](src/0_lsms_processing/2_consent_lsms_form.ipynb): Poor mans approach to automate the download. The WorldBank requires to fill a consent form and this file does it for us and downloads the survey files for us. You can download our downloaded surveys from [here](https://drive.google.com/file/d/1IlF66tdPrty5OmGdWGd7iN39KZCV-iKD/view?usp=sharing).
+- [2_consent_lsms_form](src/0_lsms_processing/2_consent_lsms_form.ipynb): Poor mans approach to automate the download. The WorldBank requires to fill a consent form and this file does it for us and downloads the survey files for us. You can download our downloaded surveys from [here](https://drive.google.com/file/d/18W813JASB23pBPWm2S0890b1ZUX0HWVU/view?usp=sharing).
 - [3_process_surveys](src/0_lsms_processing/3_process_surveys.ipynb): Preprocesses the RAW survey data. Please find the processing steps in [lib/lsms.py](src/lib/lsms.py). 
 
 After running this code you should have processed survey files in [data/lsms/processed](data/lsms/processed).
 
 ### Satellite data and features
 
-To download the data please execute the [0_download_satellite.ipynb](src/1_feature_generation/0_download_satellite.ipynb) notebook. However we recommend you to execute it on Google Colab. For this we have a modified [Colab](src/1_feature_generation/0.1_download_satellite_colab.ipynb) of the notebook, which contains all necessary libs. Since you would need to install Earth Engine locally. You also need a [Google Earth Engine account](https://earthengine.google.com/) to execute the code. Researchers, NGO's and country get free access within a short time. You can download our extracted data from [here](https://drive.google.com/file/d/1HJ3Q6BhmcZsRxb-JjhSkL6zH7hoMj1HB/view?usp=sharing).
+To download the data please execute the [0_download_satellite.ipynb](src/1_feature_generation/0_download_satellite.ipynb) notebook. However we recommend you to execute it on Google Colab. For this we have a modified [Colab](src/1_feature_generation/0.1_download_satellite_colab.ipynb) of the notebook, which contains all necessary libs. Since you would need to install Earth Engine locally. You also need a [Google Earth Engine account](https://earthengine.google.com/) to execute the code. Researchers, NGO's and country get free access within a short time. You can download our extracted data from [here](https://drive.google.com/drive/folders/1wV-fis5BG8_zCtB1NU6mi43o_5UfdMfG?usp=share_link).
 
-After you download the data you can train the CNN using [1_cnn.ipynb](src/1_feature_generation/1_cnn.ipynb). Again we recommend to execute it on Colab for this you can use our [colab](src/1_feature_generation/1.1_cnn colab.ipynb) version. If you don't want to train the network from scratch, you can use our [weights](https://drive.google.com/file/d/1Vt6wC4d0qdbyzJlIILPCaf8zWoMbTzGB/view?usp=sharing).
+After you download the data you can train the CNN using [1_cnn.ipynb](src/1_feature_generation/1_cnn.ipynb). Again we recommend to execute it on Colab for this you can use our [colab](src/1_feature_generation/1.1_cnn colab.ipynb) version. If you don't want to train the network from scratch, you can use our [weights](https://drive.google.com/drive/folders/1ctwl-LYlprZutHvoB2zg2ELC-TgJOHBg?usp=share_link).
 
 ⚠ Caution: The tfrecords need a lot of RAM! 
 

colab_ssh.ipynb

@@ -42,10 +42,21 @@
   }
  ],
  "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.8.3",
+   "language": "python",
+   "name": "python3"
+  },
   "language_info": {
-   "name": "python"
+   "name": "python",
+   "version": "3.8.3"
   },
-  "orig_nbformat": 4
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "d02c496bff33d56b3c08f9b675298569973d3021c909db82586ed45629749a93"
+   }
+  }
  },
  "nbformat": 4,
  "nbformat_minor": 2

data/countries_meta/counties.csv

@@ -0,0 +1,55 @@
+country
+Algeria
+Angola
+Benin
+Botswana
+Burkina
+Burundi
+Cameroon
+Cape Verde
+Central African Republic
+Chad
+Comoros
+Congo
+Democratic Republic of the Congo
+Djibouti
+Egypt
+Equatorial Guinea
+Eritrea
+Ethiopia
+Gabon
+Gambia
+Ghana
+Guinea
+Guinea-Bissau
+Ivory Coast
+Kenya
+Lesotho
+Liberia
+Libya
+Madagascar
+Malawi
+Mali
+Mauritania
+Mauritius
+Morocco
+Mozambique
+Namibia
+Niger
+Nigeria
+Rwanda
+Sao Tome and Principe
+Senegal
+Seychelles
+Sierra Leone
+Somalia
+South Africa
+South Sudan
+Sudan
+Swaziland
+Tanzania
+Togo
+Tunisia
+Uganda
+Zambia
+Zimbabwe

data/countries_meta/counties_lsms_time_valid.csv

@@ -49,7 +49,7 @@ South Africa,ZAF,2015,https://microdata.worldbank.org/index.php/catalog/2882
 South Africa,ZAF,1999,https://microdata.worldbank.org/index.php/catalog/1576
 South Africa,ZAF,1993,https://microdata.worldbank.org/index.php/catalog/297
 South Africa,ZAF,1993,https://microdata.worldbank.org/index.php/catalog/902
-Uganda,UGA,2021,https://microdata.worldbank.org/index.php/catalog/3765
+Uganda,UGA,2022,https://microdata.worldbank.org/index.php/catalog/3765
 Uganda,UGA,2021,https://microdata.worldbank.org/index.php/catalog/4183
 Uganda,UGA,2020,https://microdata.worldbank.org/index.php/catalog/3902
 Uganda,UGA,2019,https://microdata.worldbank.org/index.php/catalog/3795
@@ -59,6 +59,7 @@ Uganda,UGA,2014,https://microdata.worldbank.org/index.php/catalog/2663
 Uganda,UGA,2012,https://microdata.worldbank.org/index.php/catalog/2059
 Uganda,UGA,2011,https://microdata.worldbank.org/index.php/catalog/2166
 Uganda,UGA,2010,https://microdata.worldbank.org/index.php/catalog/1001
+Tanzania,TZA,2021,https://microdata.worldbank.org/index.php/catalog/4542
 Tanzania,TZA,2020,https://microdata.worldbank.org/index.php/catalog/3885
 Tanzania,TZA,2016,https://microdata.worldbank.org/index.php/catalog/2584
 Tanzania,TZA,2016,https://microdata.worldbank.org/index.php/catalog/2863
@@ -70,6 +71,11 @@ Tanzania,TZA,2010,https://microdata.worldbank.org/index.php/catalog/2251
 Tanzania,TZA,2011,https://microdata.worldbank.org/index.php/catalog/1050
 Tanzania,TZA,2009,https://microdata.worldbank.org/index.php/catalog/76
 Tanzania,TZA,2015,https://microdata.worldbank.org/index.php/catalog/3814
-Tanzania,TZA,2004,https://microdata.worldbank.org/index.php/catalog/79
 Burkina Faso,BFA,2019,https://microdata.worldbank.org/index.php/catalog/4290
 Burkina Faso,BFA,2014,https://microdata.worldbank.org/index.php/catalog/2538
+Côte d'Ivoire,CIV,2019,https://microdata.worldbank.org/index.php/catalog/4292
+Côte d'Ivoire,CIV,2016,https://microdata.worldbank.org/index.php/catalog/2789
+Côte d'Ivoire,CIV,1989,https://microdata.worldbank.org/index.php/catalog/82
+Côte d'Ivoire,CIV,1988,https://microdata.worldbank.org/index.php/catalog/85
+Côte d'Ivoire,CIV,1987,https://microdata.worldbank.org/index.php/catalog/84
+Côte d'Ivoire,CIV,1986,https://microdata.worldbank.org/index.php/catalog/83

data/countries_meta/countries.csv

@@ -10,6 +10,7 @@ Equatorial Guinea
 Gabon
 Kenya
 Nigeria
+Burkina Faso
 Rwanda
 São Tomé and Príncipe
 Tanzania
@@ -26,6 +27,7 @@ Lesotho
 Madagascar
 Malawi
 Mauritius
+Morocco
 Mozambique
 Namibia
 Seychelles
@@ -37,7 +39,7 @@ Benin
 Mali
 Burkina Faso
 Cape Verde
-Ivory Coast
+Côte d'Ivoire
 Gambia
 Ghana
 Guinea

data/countries_meta/countries_code.csv

@@ -23,6 +23,7 @@ Malawi,MWI
 Mali,MLI
 Mauritania,MRT
 Mauritius,MUS
+Morocco,MAR
 Mozambique,MOZ
 Namibia,NAM
 Niger,NER
@@ -44,6 +45,7 @@ Tanzania,TZA
 Burkina Faso,BFA
 Zambia,ZMB
 South Sudan,SSD
+Côte d'Ivoire,CIV
 Republic of the Congo,COG
 Democratic Republic of the Congo,COD
 Ivory Coast,CI