Better Schema Registry Integration

[iMacTia]

We use a remote schema registry (Confluent Cloud) in our application, and it's really handy to have an integration with it in the first place. However, the process for importing new schemas into the application currently presents some friction points:

1. Schemas need to be downloaded locally and registered with a producer in order to be used for generating Ruby classes. It would be really cool if deimos was able to get them directly from the schema registry instead! Is there any reason why that's not possible at the moment?
2. The Schema Registry is not considered the source of truth. Deimos currently not only reads but also writes against the schema registry, based on the files found locally. This can be particularly dangerous in case someone with an outdated schema attempts to produce a message because it will effectively roll back the schema version in the schema registry to an older one. We would like to limit deimos to only read from the schema registry and use that during the validation of a message (e.g. compare the auto-generated ruby class and warn when the application is producing using an outdated schema) without updating the schema registry in the process
3. Auto-generated schema files don't work out-of-the-box. We might be doing something wrong here, but basically we have our schemas under a certain namespace (e.g. org.company.my_schemas). This means that schema files need to go into a nested folder structure (e.g. org/company/my_schemas). The auto-generated Ruby class gets placed alongside the schema file, but the only module in the file is module Schemas. This breaks Rails auto-loading, which won't be able to find these classes. So we manually move them to a different location, but that means every time the rake deimos:generate_schema_classes is executed, it will re-create all the schemas that were previously moved. The solution here would be to either include the schema namespace in the ruby file modules, or create the ruby file in the root (specified in the Deimos config).

Based on the feedback on these points, I'd be happy to work on some improvements PRs 🙌

[dorner]

Hey @iMacTia ! These are all great questions and I have responses, but would you mind breaking these up into multiple topics? I think the discussion might get too unwieldy otherwise.

[iMacTia]

Makes sense! I didn't know which approach you'd prefer so I went for grouping similar things, but I agree it might be easier to go through them individually!