Add named volume for mounting host files via OLYMPIA_MOUNT

[KevinMind]

Rleates to: mozilla/addons#15066

Description

• adds OLYMPIA_MOUNT/OLYMPIA_UID setting to inform if host files are mounted
• merge docker-compose.ci.yml as we now control host mounting via the argument above
• add test checking host file presence in the container

Context

Follow up to #22912

By introducing a specific argument for the desired mounting strategy we can unify our docker compose files and have a simpler flow for mounting files correctly.

We must use bind mounts when mounting files from the host, named volumes (even with a host mounted device) do not send appropriate inotify events in a timely manner. This sucks and should be fixed in docker but for now it seems safer to use bind mounts

We map arguments OLYMPIA_* to HOST_* in the .env file. This is because the setup.py script does not always accept the user input and we don't want to let an internal command (like docker compose <whatever>) to rely on a potentially invalid value lingering on the environment but instead only read the explicitly set values in the .env. By using different names in that transaction we can guarantee this.

It is possible that we should raise errors in setup.py when invalid values are passed for OLYMPIA_MOUNT but this patch introduces enough changes and we can see if this is an issue and raise it then.

Testing

Prod Mode

To verify expected mount behavior, run each:

make up DOCKER_TARGET=production OLYMPIA_MOUNT=<production|development>

• expect the .env to have a value HOST_MOUNT equal to the specified value

• expect the web/worker containers to have an environment variable set to the specified value

• expect settings.OLYMPIA_MOUNT to equal the specified value

• If developmemnt is provided

  • Expect the .env to have a value HOST_MOUNT_SOURCE equal to "./"
  • expect docker compose to use a bind mount for the /data/olympia volume

• if production is provided

  • Expect the .env to have a value HOST_MOUNT_SOURCE equal to "data_olympia_"
  • expect docker compose to use a named volume data_olympia that is not mounted on the host. (changing files on the host should not impact files in the container, no uwsgi reload)

To test this, attach a shell in the containers and inspect the files. Additionally, you can run the check command to verify expectations.

Additionally, verify that volumes are correctly mounted to nginx

• /data/olympia -> the same files in web/worker
• /data/olympia/storage -> files in the shared storage bind mount
• /data/olympia/site-static -> files in the web container

And to worker

• /data/olympia -> the same files in web/worker
• /data/olympia/storage -> files in the shared storage bind mount

Important

Worker gets shared storage but not static files. these should be accessed via URL in the worker container if at all

Dev Mode

When running the project in dev mode (DOCKER_TARGET=dev) then the OLYMPIA_MOUNT will always be set to dev, even if specified otherwise.

make up DOCKER_TARGET=production OLYMPIA_MOUNT=<production|development>

• Run the assertions in Prod mode, but always assuming that OLYMPIA_MOUNT is "development" regardless of docker target

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[diox]

Could you resolve the few conflicts in that PR ?

[KevinMind]

@diox this is now done, and:

Could you resolve the few conflicts in that PR ?

Turns out this PR was too primitive to land after some of the other PRs landed. The goal was to strip as much out as possible to make it smaller. That doesn't work anymore so I added as little back until it is now working.

• merged the CI compose file
• use mapping of OLYMPIA_MOUNT to make the container behave correctly.

Edit: eating my words. we need proper validation on the setup script values.

[eviljeff]

🤔

[eviljeff]

should we have a default for HOST_MOUNT_SOURCE here, like the development value. (I guess something like ${HOST_MOUNT_SOURCE:?-./ but I'm not super familiar with the docker-compose syntax).

(same for the next usage of HOST_MOUNT_SOURCE

[KevinMind]

We should not. This syntax :? is actually requiring this value to be set. This value must be defined and a valid value and there are no circumstances where it should be undefined.