examples/BOILERPLATE.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align = \"center\">Boilerplate/Template Design</h1>\n",
    "\n",
    "---\n",
    "\n",
    "**Objective:** The file provides a simple *boilerplate* to concentrate on what is necessary, and stop doing same tasks (`DRY` - Don't Repeat Yourself)! The boilerplate is also configured with certain [**nbextensions**](https://gitlab.com/ZenithClown/computer-configurations-and-setups) that I personally use. Install them, if required, else ignore them as they do not participate in any type of code-optimizations. For any new project *edit* this file or `File > Make a Copy` to get started with the project. Some settings and configurations are already provided, as mentioned below. In addition, some user defined modules are available to import. Check `CHANGELOG.md` for more details, however specific *user-defined* imports maybe documented/versioned seperately. Any dependent [**`submodule(s)`**](https://www.atlassian.com/git/tutorials/git-submodule) is available under `../utilities/submodules` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:51.958257Z",
     "start_time": "2024-08-29T11:55:51.936102Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current Code Version: v0.1.2\n"
     ]
    }
   ],
   "source": [
    "# show current code version using https://semver.org/ convention\n",
    "# version release information is also available under CHANGELOG.md\n",
    "__version__ = open(\"../VERSION\", 'rt').read() # bump codecov\n",
    "print(f\"Current Code Version: {__version__}\")\n",
    "\n",
    "# the author name is skipped, however copywright is provided as such\n",
    "# commit level author is available on git commits, and details can be setup\n",
    "# the template repository is designed to keep code simple, create or edit copyright\n",
    "__copyright__ = \"Copywright © 2023 Debmalya Pramanik\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Imports\n",
    "\n",
    "A code must be written such that it is always _production ready_. The conventional guidelines provided under [**PEP8**](https://peps.python.org/pep-0008/#imports) defines the conventional or syntactically useful ways of defining and/or manipulating functions. Necessar guidelines w.r.t. code imports are mentioned below, and basic libraries and import settings are defined.\n",
    "\n",
    " 1. Imports should be on separate lines,\n",
    " 2. Import order should be:\n",
    "    * standard library/modules,\n",
    "    * related third party imports,\n",
    "    * local application/user defined imports\n",
    " 3. Wildcard import (`*`) should be avoided, else specifically tagged with **`# noqa: F403`** as per `flake8` or **`# pylint: disable=unused-import`** as per `pylint`\n",
    " 4. Avoid using relative imports; use explicit imports instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:51.973432Z",
     "start_time": "2024-08-29T11:55:51.960257Z"
    }
   },
   "outputs": [],
   "source": [
    "import os   # miscellaneous os interfaces\n",
    "import sys  # configuring python runtime environment\n",
    "# import time # library for time manipulation, and logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:51.988623Z",
     "start_time": "2024-08-29T11:55:51.974330Z"
    }
   },
   "outputs": [],
   "source": [
    "# use `datetime` to control and preceive the environment\n",
    "# in addition `pandas` also provides date time functionalities\n",
    "import datetime as dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:52.003674Z",
     "start_time": "2024-08-29T11:55:51.990610Z"
    }
   },
   "outputs": [],
   "source": [
    "# from copy import deepcopy      # dataframe is mutable\n",
    "# from tqdm import tqdm as TQ    # progress bar for loops\n",
    "# from uuid import uuid4 as UUID # unique identifier for objs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:52.019705Z",
     "start_time": "2024-08-29T11:55:52.004675Z"
    }
   },
   "outputs": [],
   "source": [
    "# import warnings # module for warnings management"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Debugging & Logging\n",
    "\n",
    "[**`logging`**](https://docs.python.org/3/howto/logging.html) is a standard python module that is meant for tracking any events that happen during any software/code operations. This module is super powerful and helpful for code debugging and other purposes. The next section defines a `logging` configuration in **`../logs/`** directory. Modify the **`LOGS_DIR`** variable under *Global Arguments* to change the default directory. The module is configured with a simplistic approach, such that any `print())` statement can be update to `logging.LEVEL_NAME()` and the code will work. Use logging operations like:\n",
    "\n",
    "```python\n",
    " >> logging.debug(\"This is a Debug Message.\")\n",
    " >> logging.info(\"This is a Information Message.\")\n",
    " >> logging.warning(\"This is a Warning Message.\")\n",
    " >> logging.error(\"This is a ERROR Message.\")\n",
    " >> logging.critical(\"This is a CRITICAL Message.\")\n",
    "```\n",
    "\n",
    "Note: some directories related to logging is created by default. This can be updated/changed in the following configuration section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:52.034985Z",
     "start_time": "2024-08-29T11:55:52.020708Z"
    }
   },
   "outputs": [],
   "source": [
    "# import logging # configure logging on `global arguments` section, as file path is required"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Analysis and AI/ML Libraries\n",
    "\n",
    "Import of data analysis and AI/ML libraries required at different intersections. Check settings and configurations [here](https://gitlab.com/ZenithClown/computer-configurations-and-setups) and code snippets [here](https://gitlab.com/ZenithClown/computer-configurations-and-setups/-/tree/master/template/snippets/vscode) for understanding settings that is used in this notebook. The code uses `matplotlib.styles` which is a custom `.mplstyle` file recognised by the `matplotlib` downlodable from [this link](https://gitlab.com/ZenithClown/computer-configurations-and-setups/-/tree/master/settings/python/matplotlib)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:53.612283Z",
     "start_time": "2024-08-29T11:55:52.035971Z"
    }
   },
   "outputs": [],
   "source": [
    "# import swifter # https://github.com/jmcarpenter2/swifter\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "%precision 3\n",
    "pd.set_option('display.max_rows', 50) # max. rows to show\n",
    "pd.set_option('display.max_columns', 17) # max. cols to show\n",
    "np.set_printoptions(precision = 3, threshold = 15) # set np options\n",
    "pd.options.display.float_format = '{:,.3f}'.format # float precisions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:55.678849Z",
     "start_time": "2024-08-29T11:55:53.613296Z"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "sns.set_style('whitegrid');\n",
    "# plt.style.use('default-style'); # http://tinyurl.com/mpl-default-style"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:55:55.694228Z",
     "start_time": "2024-08-29T11:55:55.679871Z"
    }
   },
   "outputs": [],
   "source": [
    "# sklearn metrices for analysis can be imported as below\n",
    "# considering `regression` problem, rmse is imported metrics\n",
    "# for rmse, use `squared = False` : https://stackoverflow.com/a/18623635/\n",
    "# from sklearn.metrics import mean_squared_error as MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.051762Z",
     "start_time": "2024-08-29T11:55:55.696321Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow Version: 2.12.0\n",
      "GPU Computing Not Available. If `GPU` is present, check configuration. Detected Devices:\n",
      "  >  [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(f\"Tensorflow Version: {tf.__version__}\", end = \"\\n\") # required >= 2.8\n",
    "\n",
    "# check physical devices, and gpu compute capability (if available)\n",
    "if len(tf.config.list_physical_devices(device_type = \"GPU\")):\n",
    "    # https://stackoverflow.com/q/38009682/6623589\n",
    "    # https://stackoverflow.com/a/59179238/6623589\n",
    "    print(\"GPU Computing Available.\", end = \" \")\n",
    "    \n",
    "    # experimentally, get the gpu details and computation power\n",
    "    # https://www.tensorflow.org/api_docs/python/tf/config/experimental/get_device_details\n",
    "    devices = tf.config.list_physical_devices(device_type = \"GPU\")[0] # first\n",
    "    details = tf.config.experimental.get_device_details(devices) # only first\n",
    "    details.get('device_name', 'compute_capability')\n",
    "    print(f\"EXPERIMENTAL : {details}\")\n",
    "else:\n",
    "    print(\"GPU Computing Not Available. If `GPU` is present, check configuration. Detected Devices:\")\n",
    "    print(\"  > \", tf.config.list_physical_devices())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.067087Z",
     "start_time": "2024-08-29T11:56:02.054824Z"
    }
   },
   "outputs": [],
   "source": [
    "# import xlwings as xw # https://www.xlwings.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User Defined Function(s)\n",
    "\n",
    "It is recommended that any UDFs are defined outside the scope of the *jupyter notebook* such that development/editing of function can be done more practically. As per *programming guidelines* as [`src`](https://fileinfo.com/extension/src) file/directory is beneficial in code development and/or production release. However, *jupyter notebook* requires *kernel restart* if any imported code file is changed in disc, for this frequently changing functions can be defined in this section.\n",
    "\n",
    "**Getting Started** with **`PYTHONPATH`**\n",
    "\n",
    "One must know what are [Environment Variable](https://medium.com/chingu/an-introduction-to-environment-variables-and-how-to-use-them-f602f66d15fa) and how to call/use them in your choice of programming language. Note that an environment variable is *case sensitive* in all operating systems (except windows, since DOS is not case sensitive). Generally, we can access environment variables from terminal/shell/command prompt as:\n",
    "\n",
    "```shell\n",
    "# macOS/*nix\n",
    "echo $VARNAME\n",
    "\n",
    "# windows\n",
    "echo %VARNAME%\n",
    "```\n",
    "\n",
    "Once you've setup your system with [`PYTHONPATH`](https://bic-berkeley.github.io/psych-214-fall-2016/using_pythonpath.html) as per [*python documentation*](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) is an important directory where any `import` statements looks for based on their order of importance. If a source code/module is not available check necessary environment variables and/or ask the administrator for the source files. For testing purpose, the module boasts the use of `src`, `utils` and `config` directories. However, these directories are available at `ROOT` level, and thus using `sys.path.append()` to add directories while importing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.082534Z",
     "start_time": "2024-08-29T11:56:02.068672Z"
    }
   },
   "outputs": [],
   "source": [
    "# append `src` and sub-modules to call additional files these directory are\n",
    "# project specific and not to be added under environment or $PATH variable\n",
    "# sys.path.append(os.path.join(\"..\", \"src\", \"agents\")) # agents for reinforcement modelling\n",
    "# sys.path.append(os.path.join(\"..\", \"src\", \"engine\")) # derivative engines for model control\n",
    "# sys.path.append(os.path.join(\"..\", \"src\", \"models\")) # actual models for decision making tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.097776Z",
     "start_time": "2024-08-29T11:56:02.085321Z"
    }
   },
   "outputs": [],
   "source": [
    "# also append the `utilities` directory for additional helpful codes\n",
    "# sys.path.append(os.path.join(\"..\", \"utilities\"))\n",
    "\n",
    "# you may also want to append the `utilities/submodules` directory\n",
    "# sys.path.append(os.path.join(\"..\", \"utilities\", \"submodules\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>DISCLAIMER:</b> The following codes are designed and created by the <a href = \"https://github.com/ZenithClown\">author</a> of this repository.\n",
    "Please read the <a href = \"https://github.com/ZenithClown/.github/blob/master/.github/CODE_OF_CONDUCT.md\">CODE OF CONDUCT</a> and\n",
    "<a href = \"https://github.com/ZenithClown/.github/blob/master/.github/CONTRIBUTING.md\">CONTRIBUTING</a> guidelines for more information.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>NOTE:</b> More information on Alert Box is available <a href = \"https://gist.github.com/DanielKotik/4b81480c479a57e0dd13ac4d153e4451\">here</a> for Markdown/Jupyter Notebooks.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.113713Z",
     "start_time": "2024-08-29T11:56:02.099680Z"
    }
   },
   "outputs": [],
   "source": [
    "# libraries hosted in pypi\n",
    "# import nlpurify # natural language utility functions, https://pypi.org/project/nlpurify/\n",
    "# import pandaswizard as pdw # wrapper function for the pandas, https://pypi.org/project/pandas-wizard/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.159469Z",
     "start_time": "2024-08-29T11:56:02.115701Z"
    }
   },
   "outputs": [],
   "source": [
    "import sqlparser # https://gist.github.com/ZenithClown/3fc21f94cf9567003b153bcfca738f6d\n",
    "import datetime_ as dt_ # https://gist.github.com/ZenithClown/d2dd294c5f528459e16b139c04c0b182"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Argument(s)\n",
    "\n",
    "The global arguments are *notebook* specific, however they may also be extended to external libraries and functions on import. The *boilerplate* provides a basic ML directory structure which contains a directory for `data` and a separate directory for `output`. In addition, a separate directory (`data/processed`) is created to save processed dataset such that preprocessing can be avoided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.175470Z",
     "start_time": "2024-08-29T11:56:02.160457Z"
    }
   },
   "outputs": [],
   "source": [
    "ROOT = \"..\" # the document root is one level up, that contains all code structure\n",
    "DATA = os.path.join(ROOT, \"data\") # the directory contains all data files, subdirectory (if any) can also be used/defined\n",
    "\n",
    "# processed data directory can be used, such that preprocessing steps is not\n",
    "# required to run again-and-again each time on kernel restart\n",
    "PROCESSED_DATA = os.path.join(DATA, \"processed\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.205764Z",
     "start_time": "2024-08-29T11:56:02.176457Z"
    }
   },
   "outputs": [],
   "source": [
    "# long projects can be overwhelming, and keeping track of files, outputs and\n",
    "# saved models can be intriguing! to help this out, `today` can be used. for\n",
    "# instance output can be stored at `output/<today>/` etc.\n",
    "# `today` is so configured that it permits windows/*.nix file/directory names\n",
    "\n",
    "# also, if used, update the `OUTPUT_DIR` configuration as required\n",
    "\n",
    "# today = dt.datetime.strftime(dt.datetime.strptime(time.ctime(), \"%a %b %d %H:%M:%S %Y\"), \"%a, %b %d %Y\")\n",
    "# print(f\"Code Execution Started on: {today}\") # only date, name of the sub-directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.220921Z",
     "start_time": "2024-08-29T11:56:02.206750Z"
    }
   },
   "outputs": [],
   "source": [
    "OUTPUT_DIR = os.path.join(ROOT, \"output\")\n",
    "\n",
    "# OUTPUT_DIR = os.path.join(ROOT, \"output\", today)\n",
    "# os.makedirs(OUTPUT_DIR, exist_ok = True) # create dir if not exist\n",
    "\n",
    "# also create directory for `logs`\n",
    "# LOGS_DIR = os.path.join(ROOT, \"logs\", open(\"../VERSION\", 'rt').read())\n",
    "# os.makedirs(LOGS_DIR, exist_ok = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.236922Z",
     "start_time": "2024-08-29T11:56:02.221927Z"
    }
   },
   "outputs": [],
   "source": [
    "# logging.basicConfig(\n",
    "#     filename = os.path.join(LOGS_DIR, f\"{today}.log\"), # change `reports` file name\n",
    "#     filemode = \"a\", # append logs to existing file, if file exists\n",
    "#     format = \"%(asctime)s - %(name)s - CLASS:%(levelname)s:%(levelno)s:L#%(lineno)d - %(message)s\",\n",
    "#     level = logging.DEBUG\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Development & PoC Section\n",
    "\n",
    "A typical machine learning project revolves around six important stages (as available in [Amazon ML Life Cycle Documentation](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/well-architected-machine-learning-lifecycle.html)). This notebook boilerplate can be used to understand the data file, perform statitical tests and other EDA as required for any AI/ML application. Later, using the below study a *full-fledged* application can be generated using other sections of the boilerplate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reporting & End Note(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-08-29T11:56:02.252956Z",
     "start_time": "2024-08-29T11:56:02.238910Z"
    }
   },
   "outputs": [],
   "source": [
    "# wb = xw.Book(os.path.join(ROOT, \"template\", \"template.xlsx\"))\n",
    "\n",
    "# # populate the sheets with sheet selection, and defining output cell, like:\n",
    "# wb.sheets[\"sheet\"][\"cell\"].options(header = False, index = False).value = data\n",
    "\n",
    "# # finally, close and save the object like:\n",
    "# outfile = f\"[{dt.datetime.now().date()} #{str(UUID()).upper()[:3]}] Output File Name.xlsx\"\n",
    "# print(f\"Output File Generated as: {outfile}\")\n",
    "\n",
    "# wb.save(os.path.join(OUTPUT_DIR, outfile))\n",
    "# wb.close()"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}