Sphinx typehints plugin invalidates caches

[jl-wynen]

When using the typehints sphinx extension, the sphinx caches seem to be invalidated every time tox -e docs is called. This means that building the docs takes too long. Sphinx prints this

[config changed ('typehints_formatter')] 101 added, 0 changed, 0 removed

Can we do something about this?