Shutdown event sometimes does not trigger when reloading (watchfiles)

[Blue9]

Environment:

macOS 14.5 (23F79)
Python 3.12.4
uvicorn==0.30.6
watchfiles==0.24.0

When running uvicorn with reloading, the shutdown lifespan event will not run if there are multiple reloads in quick succession (for example if you save a file twice when using WatchFiles). I haven't observed this behavior with StatReload.

Below is a minimal reproducible example:

import asyncio


async def app(scope, receive, send):
    if scope["type"] == "lifespan":
        while True:
            message = await receive()
            if message["type"] == "lifespan.startup":
                print("Server is starting up")
                await asyncio.sleep(0.5)
                await send({"type": "lifespan.startup.complete"})
            elif message["type"] == "lifespan.shutdown":
                print("Server is shutting down")
                await asyncio.sleep(0.5)
                await send({"type": "lifespan.shutdown.complete"})
                return
    elif scope["type"] == "http":
        await send({"type": "http.response.start", "status": 200, "headers": [[b"content-type", b"text/plain"]]})
        await send({"type": "http.response.body", "body": b"Hello, World!"})

Save as main.py and run uvicorn --reload main:app (also make sure uvicorn is using watchfiles).
Then, in your code editor, save the file two times quickly (you can increase the asyncio.sleep() duration to make this easier). You should see that the shutdown event is not run when reloading the second time. For example, these are the logs I see when saving twice in quick succession:

INFO:     Will watch for changes in these directories: ['/Users/ggggg/uvicorn-test']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [79287] using WatchFiles
INFO:     Started server process [79289]
INFO:     Waiting for application startup.
Server is starting up
INFO:     Application startup complete.
WARNING:  WatchFiles detected changes in 'server.py'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
Server is shutting down
INFO:     Application shutdown complete.
INFO:     Finished server process [79289]
INFO:     Started server process [79295]
INFO:     Waiting for application startup.
Server is starting up
WARNING:  WatchFiles detected changes in 'server.py'. Reloading...
INFO:     Application startup complete.
INFO:     Started server process [79296]
INFO:     Waiting for application startup.
Server is starting up
INFO:     Application startup complete.

Server is shutting down is not printed a second time. I believe this is because in Server._serve(), we eagerly return if self.should_exit is already set to True here. If I change this to the following:

await self.startup(sockets=sockets)
if self.should_exit:
    await self.shutdown(sockets=sockets)  # Added
    return
await self.main_loop()
await self.shutdown(sockets=sockets)

The shutdown event always runs when reloading and resources are cleaned up.

Happy to open a PR for the above if this is a welcome change. If not, is there another change we could make to fix this issue? This is important because I need to clean up resources in my shutdown event.

Another edge case is that any time two signals arrive in quick succession, if the second one is a SIGINT (so for example, save the above file and press Ctrl+C in your terminal immediately after), uvicorn interprets that as a force-exit and will not communicate the shutdown event. It should ideally check that both signals are SIGINTs (so a double Ctrl+C) to avoid falsely setting force_exit. Another observation in this example is that uvicorn will capture the SIGTERM from the reloader (essentially converting a SIGINT to a SIGTERM+SIGINT when re-raising the signals). Is that intended?

This is part of a broader question of whether the shutdown event should run in the case of a force-exit. I understand if that's not desired, but there doesn't seem to be a good workaround to clean up resources in that case (e.g., in the case of using multiprocessing, this will lead to orphan processes).

[Blue9]

If anyone else is running into these issues, I have a fork here with the changes, adding await self.shutdown(sockets=sockets) and always running the shutdown event, even in the case of a force-exit: https://github.com/Blue9/uvicorn. I can confirm I'm not running into the same issues with this change.

[ramipellumbi]

I am really interested in a solution to this. The current implementation seems like it did not consider the shutdown event being used to clean up resources. Is there a way the uvicorn recommends cleaning up resources prior to shutdown instead?

[Blue9]

Also, a custom signal handler will not work because uvicorn overrides them before re-raising any signals: https://github.com/encode/uvicorn/blob/0.32.0/uvicorn/server.py#L327. Maybe this block of code should only reset the signal handlers if they are self.handle_exit and otherwise leave them as they are. That said, I would prefer to keep all cleanup code in the lifespan event if possible and not have to deal with manual signal handling. Any thoughts here @Kludex?