Requests distribution

[SlavaSkvortsov]

Hey 👋

Recently we noticed that request distribution between different processes on a single machine looks like this:

Some of the processes get most of the requests, while others barely pick up any.

We use uvicorn together with supervisor (guide).

Here is an example of the `supervisor` config:

[fcgi-program:uvicorn]
numprocs=4
command=uvicorn app:app --fd 0
socket=tcp://127.0.0.1:8000
process_name=test_%(process_num)02d

and here is a simple app for testing

import asyncio
import os

from fastapi import FastAPI

app = FastAPI()

# Global counter and lock
request_count = 0
request_count_lock = asyncio.Lock()


# Function to safely increment and print request count
async def increment_request_count():
    global request_count
    async with request_count_lock:
        request_count += 1


@app.get("/")
async def read_root():
    await increment_request_count()
    return {"pid": os.getpid(), 'count': request_count}

and a simple script to test it

import asyncio

import httpx
from httpx import Limits


async def call(client):
    response = await client.get('http://127.0.0.1:8000')
    print(response.text)
    return response.json()


async def main():
    results = {}

    async with httpx.AsyncClient(limits=Limits(max_connections=10, max_keepalive_connections=10)) as client:
        for result in (await asyncio.gather(*[call(client) for _ in range(1000)])):
            results[result['pid']] = result['count']

    print(results)


if __name__ == '__main__':
    asyncio.run(main())

This setup provides poor requests distribution. I tried using uvicorn as a process manager, but the result is the same.

Previously, there was a similar discussion, but about gunicorn. Right now, gunicorn struggles with the same problem, but there is a PR that fixes it by utilizing SO_REUSEPORT socket option that

For TCP sockets, this option allows accept(2) load distribution in a multi-threaded server to be improved by using a distinct listener socket for each thread. This provides improved load distribution as compared to traditional techniques such using a single accept(2)ing thread that distributes connections, or having multiple threads that compete to accept(2) from the same socket

Sorry for the long intro 😅 Here is my question:
Is there a way to achieve more or less even request distribution across the workers on the same machine using uvicorn?

If it's not possible to do that using just uvicorn, which approach could you recommend to improve the balancing?

Thank you!!

[Kludex]

Hi @SlavaSkvortsov , right... I don't know where the issue relies on. We do set the SO_REUSEPORT, but it doesn't seem to be enough:

uvicorn/uvicorn/config.py

Line 512 in a507532

sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

Any ideas @graingert @abersheeran ?

My recommendation would be to rely on an external load balancer e.g. nginx.

[graingert]

Also there's an additional footgun here, I think you should not set REUSEPORT when running the dev server (reload mode) otherwise if a user runs the dev server on two different terminals Linux will load balance between them, and in reality the user will expect the second server to fail to bind to an already in use port

[Kludex]

man.freebsd.org/cgi/man.cgi?setsockopt(2) bsd is different and I think needs SO_REUSEPORT_LB which isn't supported in python

I've found this implementation around:

https://github.com/cherrypy/cheroot/blob/1ff20b18e98e424e0616654c28c92524deef348a/cheroot/server.py#L1983-L2007

[graingert]

But I don't think python socket defines that flag

[graingert]

see how SO_REUSEPORT behaves differently on macos and linux:

import threading
import sys
import socket
import concurrent.futures

def main():
    serving = threading.Event()
    accepted = set()

    def serve(name):
        with socket.create_server(("localhost", 8080), reuse_port=True) as sock:
            serving.set()
            while True:
                client_sock, address = sock.accept()
                accepted.add(name)
                if len(accepted) > 1:
                    print("done!")
                    return
                print(f"sock accepted on server {name}")
                with client_sock:
                    pass

    def client():
        serving.wait()
        while True:
            with socket.create_connection(("localhost", 8080)):
                pass

    with concurrent.futures.ThreadPoolExecutor(3) as tpe:
        f1 = tpe.submit(serve, 1)
        f2 = tpe.submit(serve, 2)
        f3 = tpe.submit(client)
        for v in concurrent.futures.as_completed((f1, f2, f3)):
            v.result()



if __name__ == "__main__":
    sys.exit(main())

on macos this runs forever printing sock accepted on server 1 forever, on linux it terminates after both servers accept a connection

[graingert]

one neat thing you can do is for the first socket bind you can set SO_REUSEPORT 0, then after it's bound set SO_REUSEPORT 1, this way if another server is bound to the port you get an OSError 98

[abersheeran]

Please make sure you use the new multi-process manager and increase the concurrency (the test script you gave establishes at most 10 connections, so the test is too accidental).

[minmax]

We have the same problem with gunicorn, our application structure is an asynchronous gunicorn-worker with processing of most requests in a synchronous thread (there is only one thread). gunicorn -> uvicorn-worker -> sync thread

Perhaps if you repeat this load profile, the balancing problem will be visible.

[graingert]

@SlavaSkvortsov does https://github.com/encode/uvicorn/pull/2472/files fix it for you?

[graingert]

I also just noticed I got the logic inverted of the supported modes and have pushed a fix and test

[SlavaSkvortsov]

Oh wow, it's working!

On Gentoo:
Patched, with supervisor:

(venv-patched) ~/slava.skvortsov $ python test-copy.py 
{80879: 324, 80881: 341, 80880: 75, 80882: 260}
48.70701523736938

Patched, with uvicorm's process manager

(venv-patched) ~/slava.skvortsov $ python test-copy.py 
{87512: 279, 87511: 251, 87510: 226, 87509: 244}
8.80605852051113

Amazing!

I don't understand, though, why it's not working with supervisor? Is there a way to make them compatible?

[graingert]

supervisor binds the socket and passes it to the child process. The fix should be in supervisor to bind a new socket for each process with SO_REUSEPORT

[SlavaSkvortsov]

I see, thanks!

Do you plan to release the PR? Would like to avoid using a fork in production 😅

[graingert]

currently I'm fighting coverage issues, and don't have much time to work on it - I'll need someone to finish it off for me.

I raised an issue with supervisor to add the feature there: Supervisor/supervisor#1661