Covalent Migrations via CLI

[AlejandroEsquivel]

One question regarding the migrate cli command: I’m wondering if we want just to incorporate this functionality every time covalent starts instead of having a separate covalent cli command.

The reasoning for this would be as covalent would break whenever say a feature branch is merged into develop, one would need to run always run the migrate command otherwise the application would error out when trying to access columns which may not exist (due to newly introduced fields as part of feature branch).

What I'm proposing is maybe a --no-migrate flag (or simply the -d (development) flag) on covalent start if we want to avoid running migrations.

Moreover I think that the alembic cli tool should be used to manage migrations for development purposes and we shouldn't try to achieve feature parity with our own cli.

[venkatBala]

I see the value for automating the migrate command with covalent start, but there might be some caveats that we may want to be watchful of. In cases when migrate is not successful, covalent start would be unsuccessful and left in a dirty state. If support for recovering from such situations can be looked into. Migrations might take time if there are too many of them (as the DB schema evolves) ... which can potentially slow down covalent start.

Automating the migrations with covalent start will help definitely, but exposing it through a CLI can allow us to add more functionality, such as

• covalent migrate rollback
• covalent migrate revert <N> (revert from HEAD back N migration steps)
• ...

@AlejandroEsquivel thoughts?

[wjcunningham7]

I think that covalent start should notify the user if a migration is needed, and tell them the command to run to perform it. This avoids possibly long or unsuccessful migrations frustrating users.

[venkatBala]

@santoshkumarradha just a small suggestion... that we add a db subcommand to covalent CLI this way all db related actions can be grouped nicely and this would also allow us to implement more db specific actions. For instance one can consider the following CLI options

• covalent db migrate
• covalent db reset
• covalent db export
• covalent db snapshot
• covalent db replicate/backup

[AlejandroEsquivel]

These are some good points @venkatBala @wjcunningham7

For the following cases:

1. Unsuccessful migrations - this should be a rare circumstance if we are also to include data migrations in addition to schema migrations however in such a case the user would be left at a certain migration revision version that isn't the latest (head). I would be wary of allowing a user to start covalent without the migrations anyways as this may cause unexpected behaviour and maybe cause the database to be in an even more indeterministic state, I would think the only option in failed migrations would be to do covalent purge instead of starting covalent without the required migrations - which is may be a flow they are already accustomed to if they experience issues.

2. Long migrations - these migrations would ultimately still be required to run prior to covalent start, as starting covalent without migrations would likely lead to errors regardless - separating the commands therefore won't really influence the time to be able to successfully start covalent.

As for migration commands living in covalent cli I'm not opposed to exposing a few commonly used commands however there are more commands needed to achieve parity with alembic https://alembic.sqlalchemy.org/en/latest/tutorial.html or(or see the following https://github.com/AgnostiqHQ/covalent/blob/0c612f30df64cea0011bdc131067a3e9a02d6767/alembic/README.md)

As for the path forward do you all agree with the following for an MVP of migration features:

1. Expose covalent migrate which will run all the migrations which have not run
2. If covalent requires a migration to be run it will not start covalent but advise the user the covalent migrate command will need to be run (as suggested by @wjcunningham7)
3. If covalent migrate fails it will print a RTD link where it has more information about how to proceed.

Down the road we can assess if it makes sense to incorporate migrate into covalent start.

[wjcunningham7]

In the case that migration isn't performed, the user may instead wish to revert to the previous version of Covalent rather than perform the migration. That's why I prefer the message.