4.0.0

Backwards-incompatible changes

• Switch to Wobbly for job storage. All previous job history will be lost unless the vo-cutouts database is converted into Wobbly's storage format and inserted into Wobbly's database.

Bug fixes

• Catch errors from parsing the dataset ID or creating a Butler in the backend worker and report them as proper worker exceptions so that they don't produce uncaught exception errors.
• Append a colon after the error code when reporting UWS errors.

Other changes

• Render all UWS XML output except for error VOTables using vo-models rather than hand-written XML templates.

What's Changed

• Bump python from 3.12.6-slim-bookworm to 3.12.7-slim-bookworm by @dependabot in #229
• DM-47027: Change get param dependency to correctly parse list params by @stvoutsin in #231
• DM-47811: Fixes to address FAST001 and FAST002 ruff rules and update-deps by @stvoutsin in #232
• DM-47928: Adjust for new Safir UWS library by @rra in #233
• DM-47790: Convert to vo-models for XML output by @rra in #234
• DM-47986: Convert to Wobby and the new Safir UWS library by @rra in #236
• DM-47986: Catch errors from initializing Butler in worker by @rra in #237
• DM-47986: Prepare 4.0.0 release by @rra in #238

New Contributors

• @stvoutsin made their first contribution in #231

Full Changelog: 3.2.0...4.0.0

3.2.0

New features

• Use Alembic to manage the schema of the UWS database. When upgrading to this version, set config.updateSchema to true in the Helm configuration for the first deployment. This release contains no schema changes, but needs to perform a migration to add the Alembic version information. The vo-cutouts components will now refuse to start if the database schema has changed and the database has not yet been migrated.

Bug fixes

• Restore logging configuration during startup of the backend worker, which re-adds support for the logging profile and log level and optionally configures structlog to use a JSON log format. This does not yet extend to the log messages issued directly by arq.

What's Changed

• Bump python from 3.12.4-slim-bookworm to 3.12.5-slim-bookworm by @dependabot in #221
• DM-45824: Re-add logging configuration for the backend worker by @rra in #222
• DM-46034: Use universal frozen dependencies, update by @rra in #223
• DM-46034: Move runtime dependencies to pyproject.toml by @rra in #224
• DM-46310: Use uv for Docker build by @rra in #226
• DM-46034: Add Alembic support by @rra in #225
• Bump python from 3.12.5-slim-bookworm to 3.12.6-slim-bookworm by @dependabot in #227
• DM-46034: Prepare 3.2.0 release by @rra in #228

Full Changelog: 3.1.0...3.2.0

3.1.0

New features

• The database worker pod now deletes the records for all jobs that have passed their destruction times once per hour.
• Restore support for execution duration and change the default execution duration back to 10 minutes. Use a very ugly hack to enforce a timeout in the backend worker that will hopefully not be too fragile.
• Add support for aborting jobs.
• Re-add the CUTOUT_TIMEOUT configuration option to change the default and maximum execution duration for cutout jobs.
• Support pre-signed URLs returned by the backend worker. If the result URL is an http or https URL, pass it to the client unchanged.
• Abort jobs on deletion or expiration if they are pending, queued, or executing.
• Worker pods now wait for 30 seconds (UWS database workers) or 55 seconds (cutout workers) for jobs to finish on shutdown before cancelling them.

Bug fixes

• Allow time durations in the configuration to be given in number of seconds as a string, which was accidentally broken in 3.0.0.
• Restore support for automatically starting an async job by setting phase=RUN in the POST body. The equivalent query parameter was always supported, but POST body support was accidentally dropped in 3.0.0.
• Add a colon after the error code and before the error message in error replies.
• Stop setting isPost when returning UWS parameters. This undocumented field is supposed to only be set if the parameter contains a raw POST value rather than a regular parameter, which is never the case here.

Other changes

• Stop upgrading the operating system packages in the worker image because the base image is so old that the package repositories no longer exist. This will hopefully be fixed in a future release of the Science Pipelines base image based on AlmaLinux.
• Some XML output from UWS handlers is now handled by vo-models instead of hand-written XML templates. More responses will hopefully be converted in the future.

What's Changed

• DM-45088: Revert "Bump science pipelines stack for new Butler" by @dhirving in #183
• DM-45132: Clean up change log entries, fix worker build by @rra in #184
• DM-45132: Add change log anchor for 3.0.0 release by @rra in #185
• DM-45138: Use shared Ruff configuration, update dependencies by @rra in #186
• DM-45138: Stop installling libpq-dev in Docker image by @rra in #187
• DM-45138: Separate API model from worker domain model by @rra in #188
• DM-45138: Add an arq cron job to delete expired jobs by @rra in #189
• DM-45138: Add a colon after the error code by @rra in #190
• DM-45138: Finish the worker and API model separation by @rra in #192
• DM-45138: Add deserialization support for astropy types by @rra in #193
• DM-45138: Fix the names of the UWS worker functions by @rra in #194
• DM-45138: Temporarily pin numpy to allow unpickling to work by @rra in #195
• DM-45138: Serialize worker parameters before dispatching by @rra in #196
• DM-45138: Add timeout support by @rra in #197
• DM-45138: Restore parsing of seconds as a string by @rra in #198
• DM-45138: Fix parsing of string number of seconds for timeout by @rra in #199
• DM-45138: Properly add colon after error code by @rra in #200
• DM-44509: Upgrade science pipelines stack for new Butler by @dhirving in #191
• DM-45138: Flesh out all for all modules by @rra in #201
• DM-45138: Remove dependency on ResultStore from JobService by @rra in #202
• DM-45138: Add support for aborting jobs by @rra in #203
• DM-45138: Switch to jinja2 PackageLoader by @rra in #204
• DM-45138: Make InvalidCutoutParametersError generic by @rra in #205
• DM-45138: Add summary and description for added routes by @rra in #206
• DM-45138: Update GitHub Actions configuration by @rra in #207
• DM-45138: Switch to vo-models where possible for UWS by @rra in #208
• DM-45138: Support pre-signed URLs returned by the backend by @rra in #209
• DM-45138: Move availability handler into UWS library by @rra in #210
• DM-45138: Refactor phase support for async jobs by @rra in #211
• DM-45138: Abort jobs on deletion or expiration by @rra in #212
• DM-45138: Wait for jobs to finish before cancelling them by @rra in #213
• DM-45138: Move UWS queue name constant to worker module by @rra in #214
• DM-45138: Move timedelta handling into Pydantic types by @rra in #215
• DM-45138: Move PostgreSQL URL manipulation into type by @rra in #216
• DM-45138: Move Redis DSN validation into a type by @rra in #217
• DM-45281: Convert to the Safir UWS library by @rra in #218
• DM-45281: Remove unnecessary dependencies by @rra in #219
• DM-45281: Collect change log for 3.1.0 release by @rra in #220

Full Changelog: 3.0.0...3.1.0

3.0.0

Backwards-incompatible changes

• Cancelling or aborting jobs is not supported by the combination of arq and sync worker functions. Properly reflect this in job metadata by forcing execution duration to 0 to indicate that no limit is applied. Substantially increase the default arq job timeout since the timeout will be ineffective anyway.
• Drop the CUTOUT_TIMEOUT configuration option since we have no way of enforcing a timeout on jobs.
• Upgrade the base image for the backend worker to the latest weekly. This includes a new version of lsst.daf.butler, which targets a new version of the Butler server with a backwards-incompatible REST API.

New features

• Support human-readable 4h30m20s-style strings for CUTOUT_LIFETIME and CUTOUT_SYNC_TIMEOUT in addition to numbers of seconds.

Other changes

• Unknown failures in the worker backend are now recorded as fatal UWS errors rather than transient errors. This is the more conservative choice for unknown exceptions.

What's Changed

• DM-44763: Take advantage of class serialization by @rra in #170
• DM-44763: Refactor the UWS support library by @rra in #171
• DM-44763: Drop vo-cutouts run command by @rra in #172
• DM-44763: Move runid handling to a dependency by @rra in #173
• DM-44763: Reflect the lack of job cancellation support by @rra in #174
• DM-44763: Document constraints when adding handlers by @rra in #175
• DM-44763: Fix typing of retry_async_transaction decorator by @rra in #176
• DM-44763: Stop using db_session_dependency in UWS by @rra in #177
• DM-44763: Support human-readable timedelta strings by @rra in #178
• DM-44763: Move logging to the service layer by @rra in #179
• DM-44763: Refactor worker exception handling by @rra in #180
• DM-44503: Bump science pipelines stack for new Butler by @dhirving in #181
• Collect fragments for new release by @athornton in #182

New Contributors

• @athornton made their first contribution in #182

Full Changelog: 2.0.0...3.0.0

2.0.0

Backwards-incompatible changes

• Change the job queuing system from Dramatiq to arq. This change should be transparent to users when creating new jobs, but any in-progress jobs at the time of the upgrade will be orphaned.
• Use workload identity for all authentication when deployed on Google Cloud. Separate service account keys are no longer required or used. The vo-cutouts Google service account now requires the storage.legacyBucketWriter role in addition to storage.objectViewer.

New features

• Add support for gs storage URLs in addition to s3 storage URLs. When a gs storage URL is used, the image cutout backend will use the Google Cloud Storage Python API to store the results instead of boto, which will work correctly with workload identity.
• Catch the error thrown when the cutout has no overlap with the specified image and return a more specific error message to the user.
• Add support for sending Slack notifications for uncaught exceptions in route handlers.
• Add support for sending Slack notifications for unexpected errors when processing cutout jobs.
• If the backend image processing code fails with an exception, include a traceback of that exception in the detail portion of the job error.

Bug fixes

• Queuing a job for execution in the frontend is now async and will not block the event loop, which may help with performance under load.
• Report fatal (not transient) errors on backend failures. We have no way of knowing whether a failure will go away on retry, so make the conservative assumption that it won't.

What's Changed

• Bump python from 3.12.2-slim-bookworm to 3.12.3-slim-bookworm by @dependabot in #160
• DM-44606: Update dependencies and tox-docker by @rra in #161
• DM-44606: Change passwords to SecretStr by @rra in #162
• DM-44606: Convert to arq by @rra in #163
• DM-44606: Drop pydantic-settings from worker, add timeout by @rra in #164
• DM-44720: Add Slack reporting of uncaught exceptions by @rra in #165
• DM-44720: Improve serialization of task errors by @rra in #166
• DM-44720: Report unexpected errors from backend processing by @rra in #167
• Bump python from 3.12.3-slim-bookworm to 3.12.4-slim-bookworm by @dependabot in #168
• DM-44606: Prepare 2.0.0 release by @rra in #169

Full Changelog: 1.1.1...2.0.0

1.1.1

Other changes

• Update to the latest weekly as a base image for the cutout worker, which picks up new versions of lsst-resources and the Butler client.

What's Changed

• Bump actions/checkout from 3 to 4 by @dependabot in #156
• Install tox-docker for periodic CI checks by @rra in #157
• DM-43846: Update dependencies by @rra in #158
• DM-43846: Prepare 1.1.1 release by @rra in #159

Full Changelog: 1.1.0...1.1.1

1.1.0

New features

• Add support for querying the Butler server rather than instantiating local Butler instances. To support this, vo-cutouts now requires delegated tokens from Gafaelfawr so that it can make API calls on behalf of the user.
• Send uvicorn logs through structlog for consistent JSON formatting and add context expected by Google Cloud Logging to each log message.

Other changes

• Standardize the environment variables used for configuration. Rename SAFIR_ environment variables to CUTOUT_, remove SAFIR_LOG_NAME, and add CUTOUT_PATH_PREFIX to control the API path prefix. This is handled by the Phalanx chart, so should be invisible to users.
• Add a change log maintained using scriv.
• Use Ruff for linting and formatting instead of Black, flake8, and isort.

What's Changed

• DM-37516: Fix requirements/main.in Safir comment by @rra in #103
• [neophile] Update dependencies by @sqrbot in #104
• Fix package version by @rra in #105
• [neophile] Update dependencies by @sqrbot in #106
• [neophile] Update dependencies by @sqrbot in #107
• [neophile] Update dependencies by @sqrbot in #109
• Bump docker/build-push-action from 3 to 4 by @dependabot in #108
• Bump python from 3.11.1-slim-bullseye to 3.11.2-slim-bullseye by @dependabot in #110
• [neophile] Update dependencies by @sqrbot in #111
• [neophile] Update dependencies by @sqrbot in #112
• [neophile] Update dependencies by @sqrbot in #113
• [neophile] Update dependencies by @sqrbot in #114
• [neophile] Update dependencies by @sqrbot in #115
• [neophile] Update dependencies by @sqrbot in #116
• [neophile] Update dependencies by @sqrbot in #117
• [neophile] Update dependencies by @sqrbot in #119
• Bump python from 3.11.2-slim-bullseye to 3.11.3-slim-bullseye by @dependabot in #118
• [neophile] Update dependencies by @sqrbot in #120
• [neophile] Update dependencies by @sqrbot in #121
• [neophile] Update dependencies by @sqrbot in #122
• [neophile] Update dependencies by @sqrbot in #123
• [neophile] Update dependencies by @sqrbot in #124
• [neophile] Update dependencies by @sqrbot in #125
• [neophile] Update dependencies by @sqrbot in #126
• [neophile] Update dependencies by @sqrbot in #127
• [neophile] Update dependencies by @sqrbot in #129
• Bump python from 3.11.3-slim-bullseye to 3.11.4-slim-bullseye by @dependabot in #128
• [neophile] Update dependencies by @sqrbot in #130
• [neophile] Update dependencies by @sqrbot in #131
• Bump python from 3.11.4-slim-bullseye to 3.11.5-slim-bullseye by @dependabot in #132
• Bump actions/checkout from 3 to 4 by @dependabot in #133
• Bump docker/build-push-action from 4 to 5 by @dependabot in #134
• Bump docker/login-action from 2 to 3 by @dependabot in #135
• Bump docker/setup-buildx-action from 2 to 3 by @dependabot in #136
• DM-42191: Allow use of Butler client/server by @dhirving in #140
• Bump actions/setup-python from 4 to 5 by @dependabot in #138
• Bump python from 3.11.5-slim-bullseye to 3.12.2-slim-bullseye by @dependabot in #141
• DM-42937: Adopt scriv and uv and update to Python 3.12 by @rra in #142
• DM-42937: Update run target and mypy configuration by @rra in #143
• DM-42937: Convert to Ruff and fix most issues it uncovered by @rra in #144
• DM-42937: Switch to pydantic-settings for configuration by @rra in #145
• Bump actions/cache from 3 to 4 by @dependabot in #147
• Bump pre-commit/action from 3.0.0 to 3.0.1 by @dependabot in #146
• DM-42937: Update to current GitHub workflows by @rra in #148
• DM-42937: Switch to Self types where appropriate by @rra in #149
• DM-42937: Use Safir datetime utility functions by @rra in #150
• DM-42937: Add call to configure_uvicorn_logging by @rra in #151
• DM-42937: Remove deprecated FastAPI and Starlette constructs by @rra in #152
• DM-42937: Simplify the Docker configuration by @rra in #153
• DM-42937: Accept timedelta config values as strings by @rra in #154
• DM-42937: Prepare 1.1.0 release by @rra in #155

New Contributors

• @dhirving made their first contribution in #140

Full Changelog: 1.0.0...1.1.0

1.0.0

There are no major functionality changes in this release. It updates dependencies, packaging, and coding style, makes more use of Safir utility functions, and bumps the version to 1.0.0 since this is acceptable as a release candidate, even though we hope to add additional functionality later.

What's Changed

• [neophile] Update dependencies by @sqrbot in #68
• [neophile] Update dependencies by @sqrbot in #69
• [neophile] Update dependencies by @sqrbot in #70
• [neophile] Update dependencies by @sqrbot in #72
• Bump python from 3.10.5-slim-bullseye to 3.10.6-slim-bullseye by @dependabot in #71
• [neophile] Update dependencies by @sqrbot in #73
• [neophile] Update dependencies by @sqrbot in #74
• [neophile] Update dependencies by @sqrbot in #75
• [neophile] Update dependencies by @sqrbot in #76
• [neophile] Update dependencies by @sqrbot in #78
• Bump python from 3.10.6-slim-bullseye to 3.10.7-slim-bullseye by @dependabot in #77
• [neophile] Update dependencies by @sqrbot in #79
• [neophile] Update dependencies by @sqrbot in #80
• [neophile] Update dependencies by @sqrbot in #81
• [neophile] Update dependencies by @sqrbot in #82
• [neophile] Update dependencies by @sqrbot in #84
• [neophile] Update dependencies by @sqrbot in #85
• [neophile] Update dependencies by @sqrbot in #87
• [neophile] Update dependencies by @sqrbot in #88
• [neophile] Update dependencies by @sqrbot in #89
• [neophile] Update dependencies by @sqrbot in #90
• [neophile] Update dependencies by @sqrbot in #92
• [neophile] Update dependencies by @sqrbot in #93
• [neophile] Update dependencies by @sqrbot in #94
• [neophile] Update dependencies by @sqrbot in #95
• [neophile] Update dependencies by @sqrbot in #96
• Bump python from 3.10.7-slim-bullseye to 3.11.1-slim-bullseye by @dependabot in #91
• DM-37516: Update packaging by @rra in #97
• DM-37516: Use GCS support from Safir by @rra in #98
• DM-37516: Bump stack container version by @rra in #99
• DM-37516: Stop pushing containers to Docker Hub by @rra in #100
• DM-37516: Update typing to avoid deprecations by @rra in #101
• DM-37516: Update docstring syntax around types by @rra in #102

Full Changelog: 0.4.2...1.0.0

0.4.2

• Clip stencils at the edge of the image instead of raising an error in the backend. Practical experience with the Portal and deeper thought about possible scientific use cases have shown this to be a more practical and user-friendly approach.
• Drop support for Python 3.9.
• Update dependencies.

0.4.1

• Stop masking pixels outside the cutout stencil. The current performance of masking is unreasonably slow for CIRCLE cutouts, and masking isn't required by the SODA standard. We may revisit this later with a faster algorithm.
• Update dependencies.