Extensions that support Flask & Quart

[pgjones]

I've been thinking about recommending patterns for extensions so that they can support both Flask and Quart. I'd like to discuss these patterns here, then publish them in the docs and adopt them for the extensions I maintain.

The problem

The problem is that IO is either async/await based or not, and if it is the former the full call stack must be async/await based. I think we should rule out Flask-Patch, greenback, or other patching solutions as I think they are unpleasant and will not be adopted wide-scale.

How extensions interact

I think we can simplify extensions to do two things, the first is alter the app instance and the second is to decorate route handlers i.e.

class Extension:
    def __init__(self, app):
        app.method = self.method

    async def method(self):  # Should this be async?
        pass

def extension(func):
    async def wrapper(*args, **kwargs):  # Should this be async?
        await func(*args, **kwargs)  # should this await?
    return wrapper

To summarise the problem: The extension needs to know if it is extending Flask or Quart at import time and adjust between async def and def based on which framework it extends.

The solution

I think we should advise that extensions can be one of three types:

1. The extension does its own IO, e.g. Quart-DB.
2. The extension does no IO e.g. Quart-CORS, Flask-CORS.
3. The extension uses Flask/Quart inbuilt IO e.g. Quart-Schema, Flask-WTF (as either request.get_json() or request.form)

The first type

Extensions that do their own IO should not try to support Flask and Quart, as the extension must choose how it does the IO i.e. is it async/await based or not.

The second and third types

These are both need to figure out if they are extending Quart or Flask, and I propose this pattern,

try:
    from quart import current_app
except ImportError:
    from flask import current_app
    _using_quart = False
else:
    _using_quart = True

class Extension:
    def __init__(self, app):
        if _using_quart:
            app.method = self.async_method
        else:
            app.method = self.method

    async def async_method(self):
        pass
        
    def sync_method(self):
        pass

def extension(func):
    if _using_quart:
        async def wrapper(*args, **kwargs):
            await current_app.ensure_async(func)(*args, **kwargs)
    else:
        def wrapper(*args, **kwargs):
            current_app.ensure_sync(func)(*args, **kwargs)
    return wrapper        

Note on type 3

In the above pattern request can be imported alongside current_app and await request.get_json() or request.get_json can be added as appropriate. For example,

def extension(func):
    if _using_quart:
        async def wrapper(*args, **kwargs):
            data = await request.get_json()
            validate_data(data)
            await current_app.ensure_async(func)(*args, data=data, **kwargs)
    else:
        def wrapper(*args, **kwargs):
            data = request.get_json()
            validate_data(data)
            current_app.ensure_sync(func)(*args, data=data, **kwargs)
    return wrapper 

Downsides to this solution

Whilst it is unlikely that Flask and Quart will be installed in the same environment, it now becomes vital that they are not.

[davidism]

I think we can avoid the restriction on installing both. As we discussed yesterday, we might want to install both anyway for Quart to share code with Flask.

Instead, the app objects can gain an attribute like async, async_first, or prefers_async. Flask can hard-code this to False, and Quart to True. I like the pattern of defining sync_ and async_ prefixed versions of methods, then assigning the non-prefixed name to one or the other.

def apply_async(app, names):
    prefix = "async" if app.prefers_async else "sync"

    for name in names:
        setattr(self, name, getattr(self, f"{prefix}_{name}"))

To support older versions, since they would already need to import both in your original example, they can use isinstance instead.

if hasattr(app, "prefers_async"):
    prefers_async = app.prefers_async
else:
    from quart import Quart

    prefers_async = isinstance(app, Quart)

[pgjones]

I'm not sure how this would work with decorators - the app is not available at the time a decision needs to be made e.g. when decorating blueprint handlers.

[davidism]

I don't think blueprint decorators have a problem, as they defer registration until the app is available. But I can see how your point would apply to extension decorators, although I can't think of an example right now.

[pgjones]

For clarity I was thinking about,

@blueprint.get("/")
@extension
async def route():
    ...

[pgjones]

An alternative idea I've been considering is to require the usage of ensure_async and ensure_sync so that the decorator could just inspect the func to decide if it is async or not,

def extension(func):
    if iscoroutinefunction(func):
        async def wrapper(*args, **kwargs):
    else:
        def wrapper(*args, **kwargs):
    return wrapper
    
# E.g. usage in a Quart codebase
@extension
@ensure_async
def index():
   ...

With the class extension simply determining if the app is a Quart or Flask instance, i.e.:

class Extension:
    def __init__(self, app):
        if isinstance(app, Quart):
            app.method = self.async_method
        else:
            app.method = self.method

    async def async_method(self):
        pass
        
    def sync_method(self):
        pass

[davidism]

I don't see why @ensure_async before @extension_decorator is needed. Some extensions are already attempting to support both def and async def by calling ensure_sync inside the wrapper, for example Flask-Login's @login_required does this. In a theoretical extension that supports both, wouldn't the following be ok?

def login_required(f):
    if iscouroutinefunction(f):
        return async_login_required(f)
    else:
        return sync_login_required(f)

I think this might solve the decorator issue with my idea too. Since we're going to support both sync and async user defined functions, we can switch on what the user defined.

[davidism]

So basically, a view decorator that wants to support async and sync has to return async or sync, it shouldn't change the color of the function. Then it works exactly as it does now, where app.add_url_rule (and other app methods) is what decides to call ensure_sync or not.

[pgjones]

This works for type 2 extensions, of which Flask-Login is. It doesn't work for type 3 extensions. For example consider an extension that needs to do something with the request JSON, which as a pure Quart extension would be,

def extension(func):
    async def wrapper(*args, **kwargs):
        data = await request.get_json()
        ...
        return await ensure_async(func)(*args, **kwargs)
    return wrapper

To make this extension support Flask and Quart, we need to know somehow if the request.get_json needs to be awaited at import time as the wrapper must be either sync or async depending on it. By enforcing the ensure_(a)sync we can use the func to figure it out, i.e.

def extension(func):
    if iscoroutinefunc(func):
        async def wrapper(*args, **kwargs):
            data = await request.get_json()
            ...
            return await func(*args, **kwargs)
    else:
        def wrapper(*args, **kwargs):
            data = request.get_json()
            ...
            return func(*args, **kwargs)
    return wrapper

As with the enforced usage of ensure_(a)sync all Quart functions will be coroutine functions and all Flask sync-functions.

[davidism]

Hmm, I think I see what you're getting at. We know whether the user's function is sync or async, but we don't know whether the app they'll register it with is sync or async so we don't know if it's safe to await app.get_json().

At the expense of a bit more complexity in these types of decorators, we could combine this with my app.prefers_async idea to do this:

if iscoroutinefunction(f):
    async def wrapper(...):
        if current_app.prefers_async:
            data = await request.get_json()
        else:
            data = request.get_json()
        
        return await f(...)

[pgjones]

This works for the async case, but it can't work for the sync case i.e. I don't know how this can work,

if not iscoroutinefunction(f):
    def wrapper(...):
        if current_app.prefers_async:
            data = await request.get_json()  # Issues
        else:
            data = request.get_json()
        
        return f(...)

[pgjones]

@patkennedy79 I wondered if you had a view?

[patkennedy79]

The idea of extensions supporting both Flask and Quart is excellent!

It seems like there are four combinations (Sync/Async Function, Flask/Quart App) that need to be handled with the Type 3 (The extension uses Flask/Quart inbuilt IO) situation:

from quart.utils import run_sync


def extension(func):
    if iscoroutinefunction(f):                              # Async Function
        async def wrapper(…):
            if current_app.prefers_async:                   # Quart
                data = await request.get_json()
            else:                                           # Flask
                data = request.get_json()
            return await f(…)
     else:                                                  # Sync Function
         def wrapper(…):
            if current_app.prefers_async:                   # Quart
                data = ensure_sync(request.get_json())
            else:
                data = request.get_json().                  # Flask
            return f(…)

Is ensure_sync(request.get_json()) valid in Quart to run data = await request.get_json() in a synchronous environment?

[jace]

Since the prefers_async/prefers_sync flags will be new, it will be getattr(current_app, 'prefers_async', False) for extensions that support older versions of Flask. This will get tedious (and slow with a full MRO search), so it should be done once per function entry.

[pgjones]

I've had a further thought about the type 3 extensions. The issue as I see it can be solved if raising a RuntimeError like so, is acceptable:

def extension(func):
    if iscoroutinefunction(func):
        async def wrapper(*args, **kwargs):
            if isinstance(request, QuartRequest):
                data = await request.get_json()
            else:
                data = request.get_json()
            validate_data(data)
            return await func(*args, data=data, **kwargs)
    else:
        def wrapper(*args, **kwargs):
            if isinstance(request, QuartRequest):
                raise RuntimeError("func must be a coroutine")
            else:
                data = request.get_json()
            validate_data(data)
            return func(*args, data=data, **kwargs)
    return wrapper 

I think this is acceptable as Quart already disallows the usage of Request coroutine-methods in sync route handlers. Therefore if the extension also raises an error then there is no additional issues.

This roughly matches @patkennedy79's suggestion above, only without the usage of ensure_sync. I think using ensure_sync would be problematic as it is highly likely to nest.

[pgjones]

There is an additional question of whether to use isinstance(request, QuartRequest) or current_app.prefers_async and what request itself points at (no guarantee only Quart or Flask is installed). Maybe,

try:
    from flask import request as flask_request 
except ImportError:
    flask_request = None

try:
    from quart import request as quart_request 
except ImportError:
    quart_request = None

...

if current_app.IS_QUART:
    data = await quart_request.get_json()
else:
    data = flask_request.get_json()