Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate HTML output and send it to Slack, make output files downloadable in the web UI #3

Merged
merged 62 commits into from
Jan 11, 2024
Merged
Show file tree
Hide file tree
Changes from 57 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
7419b2d
Generate HTML output and send it to Slack
nkaretnikov Nov 12, 2023
5bbf836
Remove redundant format string
nkaretnikov Dec 5, 2023
5692f51
Use `parameters` instead of iterating over `envs`
nkaretnikov Dec 5, 2023
a1c8c59
Use `requests` in `send_to_slack`
nkaretnikov Dec 5, 2023
7a4e650
Use variables in the `papermill` command
nkaretnikov Dec 5, 2023
243c7ad
Fix linting issues
nkaretnikov Dec 5, 2023
d212f09
Use `os.path.basename` since `script` args are serialized
nkaretnikov Dec 6, 2023
eb57d45
Account for `parameters` being `None`
nkaretnikov Dec 6, 2023
3ff0866
Add docs on how to send to Slack
nkaretnikov Dec 6, 2023
27eadc9
Log Slack script output to a file
nkaretnikov Dec 6, 2023
d8a4624
Add missing import
nkaretnikov Dec 6, 2023
ebdff98
Try passing `logger` to the Slack script
nkaretnikov Dec 6, 2023
be83ac4
Try using a local logger since it cannot be an arg
nkaretnikov Dec 6, 2023
b5ac5f9
Fix the import
nkaretnikov Dec 6, 2023
060d837
Log `staging_paths`
nkaretnikov Dec 7, 2023
2c5dc9e
Try using filenames from `staging_paths`
nkaretnikov Dec 7, 2023
7a5d3d4
Log `job.create_time`
nkaretnikov Dec 7, 2023
f7c391e
Log to file in workflows
nkaretnikov Dec 7, 2023
ec75989
Append to log file since logger uses it before that
nkaretnikov Dec 7, 2023
d6b91fb
Use filenames with correct start time for cron jobs
nkaretnikov Dec 7, 2023
6325d4a
Move Container out of Steps
nkaretnikov Dec 7, 2023
322b264
Try moving main into a script
nkaretnikov Dec 7, 2023
fb51c6e
Add missing imports
nkaretnikov Dec 7, 2023
b95bc36
Revert "Add missing imports"
nkaretnikov Dec 7, 2023
2129c76
Revert "Try moving main into a script"
nkaretnikov Dec 7, 2023
f2827eb
Try moving main out of Steps
nkaretnikov Dec 7, 2023
3dc76d8
Revert "Try moving main out of Steps"
nkaretnikov Dec 7, 2023
bf98603
Try moving main out of Steps and use Parameters
nkaretnikov Dec 7, 2023
5c6aa59
Try calling the container directly
nkaretnikov Dec 7, 2023
3824941
Add missing inputs
nkaretnikov Dec 7, 2023
6dcbc71
Try generating start_time within a step
nkaretnikov Dec 7, 2023
0d31ba2
Try passing placeholder value as string
nkaretnikov Dec 7, 2023
adf5231
Try reading `start_time` from the DB
nkaretnikov Dec 7, 2023
673b8bb
Rename variable that clashes with another one
nkaretnikov Dec 7, 2023
52f160a
Order by start_time
nkaretnikov Dec 7, 2023
792b335
Add a rename step
nkaretnikov Dec 7, 2023
1f7a136
Add missing imports
nkaretnikov Dec 8, 2023
9f23570
Add missing imports
nkaretnikov Dec 8, 2023
5b6fce8
Add debug prints
nkaretnikov Dec 8, 2023
6110682
Create a symlink to make files downloadable
nkaretnikov Dec 8, 2023
2f88046
Handle symlinks when deleting scheduled jobs
nkaretnikov Dec 8, 2023
c02cc01
Remove debug prints
nkaretnikov Dec 8, 2023
5c96c3b
Read `start_time` from the DB in `send_to_slack`
nkaretnikov Dec 8, 2023
36cec7a
Always try to run the rename-files step
nkaretnikov Dec 8, 2023
ce33d45
Remove all symlinks related to the same job
nkaretnikov Dec 8, 2023
4ca14e6
Move common code outside of classes
nkaretnikov Dec 8, 2023
c24af7b
Add helpers for default filenames
nkaretnikov Dec 8, 2023
90060e2
Fix typo in function name
nkaretnikov Dec 8, 2023
8cfda20
Clean up `send_to_slack`
nkaretnikov Dec 8, 2023
d596dd5
Clean up `rename_files`
nkaretnikov Dec 8, 2023
93f9c83
Do not create intermediate variable
nkaretnikov Dec 8, 2023
4320226
Call `main` the way it used to be done before
nkaretnikov Dec 8, 2023
f71a435
Use full name when walking files
nkaretnikov Dec 8, 2023
7ec67e3
Silence the linter warning
nkaretnikov Dec 8, 2023
28d4792
Add new doc sections to ToC
nkaretnikov Dec 8, 2023
06f93c0
Add Slack screenshots to the doc
nkaretnikov Dec 8, 2023
82683a3
Resize images in the doc
nkaretnikov Dec 8, 2023
3305f76
Use `create_output_filename` from jupyter-scheduler
nkaretnikov Dec 8, 2023
3a42dd0
Update the comment
nkaretnikov Dec 9, 2023
a303e83
Move logging setup code out of try block
nkaretnikov Dec 17, 2023
503c7cf
Log exception trace
nkaretnikov Dec 17, 2023
195eb5c
Update README
nkaretnikov Dec 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
- [argo-jupyter-scheduler](#argo-jupyter-scheduler)
- [Installation](#installation)
- [What is it?](#what-is-it)
- [Optional features](#optional-features)
- [Sending to Slack](#sending-to-slack)
- [A deeper dive](#a-deeper-dive)
- [`Job`](#job)
- [`Job Definition`](#job-definition)
Expand Down Expand Up @@ -52,6 +54,29 @@ If you want to run your Jupyter Notebook on a schedule, you need to be assured t

The solution is Argo-Jupyter-Scheduler: Jupyter-Scheduler front-end with an Argo-Workflows back-end.

## Optional features

### Sending to Slack

Argo-Jupyter-Scheduler allows sending HTML output of an executed notebook to a
Slack channel:

- See the Slack API docs on how to create a bot token (starts with `xoxb`)
- Invite your bot to a Slack channel which will be used for sending output
- When scheduling a notebook (as described above):
- Select a conda environment that has `papermill` installed
- Add the following `Parameters`:
- name: `SLACK_TOKEN`, value: `xoxb-<Slack bot token>`
- name: `SLACK_CHANNEL`, value: `<Slack channel name>` (without leading `#`, like `scheduled-jobs`).

Create job:

<img src="./assets/create-job-slack.png" alt="Create job Slack" width="400"/>

Slack output:

<img src="./assets/slack-output.png" alt="Slack output" width="800"/>
nkaretnikov marked this conversation as resolved.
Show resolved Hide resolved

## A deeper dive

In the Jupyter-Scheduler lab extension, you can create two things, a `Job` and a `Job Definition`.
Expand Down
Loading