Paperless is a tool that extends the capabilities of Papermill by providing the ability to run Papermill via Google Cloud Dataproc Serverless.
https://medium.com/@benm-23166/paperless-dataproc-serverless-meets-jupyter-3c44c6458789
Papermill is a powerful tool for parameterizing and executing Jupyter Notebooks. However, by default papermill dosn't support Jupyter Kernel Gateway - it was impossible to run spark notebook vs Google Cloud Dataproc Serverless environment with Papermill tool - this is where Paperless helps.
Paperless bridges the gap between Papermill and Google Cloud Dataproc Serverless interactive mode, allowing you to seamlessly integrate the two and harness the power of serverless execution for your Jupyter Notebooks.
-
Serverless Execution: Run Papermill on Google Cloud Dataproc without managing the underlying infrastructure.
-
Scalability: Leverage the scalability of Google Cloud Dataproc for processing multiple Notebooks concurrently.
-
Cost-Effective: Pay only for the resources you consume during the execution, optimizing costs for your notebook parameterization tasks.
Pricing model is much more cost-effective considering other platofrms and technologies
resource: https://cloud.google.com/dataproc-serverless/pricing
Before using Paperless, make sure you have the following:
- A Google Cloud Platform (GCP) project
- Access to Google Cloud Dataproc Enable the API
- Papermill installed locally or in your environment
To authenticate your application using Application Default Credentials (ADC) with gcloud
-
If you haven't already installed the Google Cloud SDK, you can download and install it from the Google Cloud SDK documentation.
Open a terminal and run the following command to authenticate your Google Cloud SDK with your Google Cloud Platform (GCP) account:
gcloud auth login
gcloud auth application-default login
gcloud config set project <project_id>
gcloud config set compute/region <region>
gcloud config set dataproc/region <region>
pip install paperless
Parameters and details can be found in GCP Docs.
The default template name is: paperless-interactive
gcloud compute instance-templates create paperless-interactive --<extra params...>
You can change parameters as you need based on the jobs needs - check the docs for that.
Paperless excepts && supports all list or arguments exists in original Papermill package - the minimum needed for testing:
paperless <input_path> <output_path> ...
An extra environment variable that is special for Paperless: TEMPLATE_NAME=paperless-interactive Example:
export TEMPLATE_NAME=paperless-interactive && \
paperless ./resources/spark.ipynb ./resources/spark-out.ipynb
You're all set, enjoy :)
# Create a new directory for your project
git clone https://github.com/Plarium-Repo/paperless.git && cd paperless
# Create a virtual environment
python3 -m venv .venv
# Activate the virtual environment
# On Windows
.venv\Scripts\activate
# On macOS and Linux
source venv/bin/activate
# Install requirements
pip install -r requirements.txt
# Install the command line
python setup.py install
# Execute example
export TEMPLATE_NAME=paperless-interactive && ./.venv/bin/paperless ./tests/resources/test.ipynb ./tests/resources/test-out.ipynb
Made With Love ❤️ from 🇮🇱 🇮🇱