Notebooks are artefacts too

[machielg]

In our DevOps process, we treat our notebooks as artefacts just like py files. Some Tasks are based on python files and others are based on notebooks. We want to deploy the notebooks and schedule the deployed notebook. This would have the benefit of having the whole 'release' as one package and a git fingerprint of what we're running.

DBX allows scheduling of notebooks, but only when they are in a different repository or at least have a lifecycle outside of the other files in the dbx project. This would mean a disjoint workflow that actually should be joined, we would change the .py files in cohort with the notebooks and release them as a whole.

In short, the feature request is: allow notebooks to be deployed as artefacts (currently possible) and scheduled within the deployment.yml in a task (not supported)

[renardeinside]

hi @machielg ,
Notebooks shall not be stored in dbx artifact storage. have you tried the git_source approach as described here?

[machielg]

@renardeinside thanks for the suggestion. I have 2 questions. Why shall they not be stored in dbx artefact storage? The git source approach seems like a brittle or cumbersome solution for us if we want to treat the notebook as an integral part of the delivery, the source and notebook should work as a (versioned) whole, so why would I want to make a second repository?

[gwaffen-mck]

I'd like an answer to why notebooks shall not be stored in artifact storage as well. @machielg 's use case seems straight forward.