Add support for generating multi-version CRDs and CRD conversion webhooks

[ulucinar]

Description of your changes

Depends on: crossplane/upjet#321, crossplane/upjet#326, crossplane/upjet#331

This PR introduces the multi-version CRD & API conversion function generation capabilities to the provider. More details can be found in crossplane/upjet#321. It also incorporates the resolver transformer from crossplane/upjet#331 to prevent the import cycles that frequently occur when a managed resource's API version is bumped leaving some other managed resources in its group at their previous versions. The transformer is run as part of the code generation pipeline defined in apis/generate.go as follows:

go run github.com/crossplane/upjet/cmd/resolver -g aws.upbound.io -a github.com/upbound/provider-aws/internal/apis -s

The dynamic reference source initialization is implemented using a type registry in internal/apis/scheme.go, which we can move to upjet in the future releases.

This PR also introduces a new optional command-line flag for the provider binary, --certs-dir. The default value for this flag is /tls/server, the mount path for the Crossplane-generated TLS certificate and the key, when the provider package is installed by Crossplane. The provider process looks for the tls.crt and tls.key files under the folder specified with --certs-dir while starting the conversion Webhook server. This flag can also be used to disable the Webhook server for development purposes, as discussed below.

Apart from the discussion in crossplane/upjet#321, this PR takes care of generating CRDs with the conversion strategy set to Webhook. However, the generated CRDs containing the conversion strategy are not stored under package/crds under the repo root. That path still contains the generated CRDs but without the conversion stanzas and they can still be used during development of the managed resources, and for running the provider process locally without the need to configure any TLS certificates. While developing the implementation of a managed resource, there's generally no need to involve the conversion webhooks and the conversion function chains they invoke, unless you're specifically interested in the conversion functions (or the upjet generated spoke implementations). Although the generated CRDs at package/crds do not contain the conversion stanzas, if there's at least one resource in the provider with multiple API versions, the provider, by default, attempts to start the conversion Webhook server, which will fail if the TLS certificates are not properly configured. In such cases, if you'd like to run the provider with the Webhook server disabled, then you can just set the --certs-dir command line argument to an empty string to disable the conversion webhook server, i.e., by specifying --certs-dir="".

The PR uses kustomize together with the yq to generate the CRDs with the proper conversion stanzas specifying the conversion strategy of Webhook for converting between the hub & spoke API versions. As discussed above, the CRDs specifying the conversion strategies are not stored at the path package/crds and instead, they can be found under the path _output/package/crds in the repo root. yq is used to split the multi-doc YAML output from kustomize as the up xpkg batch command relies on the generated CRD filenames while preparing the provider family (resource) packages. Various alternatives have been considered instead of incorporating yq in the code generation pipeline as detailed in the description of upbound/build#249. Notably a shell script implementation of the split function was considerably slower than the yq --split-exp command and we've chosen to incorporate yq into our code generation pipeline.

yq has been incorporated as a kustomize plugin with the name SplitTransformer. Another alternative was to just pipe the output from kustomize (via a shell pipe) to the yq --split-exp command. However, the plugin approach is better for bundling the kustomization transformations as a reusable module. We will consider moving the kustomization configuration for generating CRDs with the proper conversion stanzas as a reusable component to uptest (where there are already some reusable components related with the official provider repos' build pipelines) or to upjet.

With this PR, we also need to bump the golangci-lint version to v1.55.2. The golangci-lint version in the corresponding reusable workflow is not parameterized and is currently hardcoded. upbound/official-providers-ci#178 makes it parameterized and we will retrigger the lint job after it's merged.

I have:

• Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

@sergenyalcin has implemented a variety of CustomConversions and FieldRenameConversions (please see crossplane/upjet#321 for further context) on top of this PR and tested the underlying machinery successfully. We are planning to introduce those Conversions with a separate PR on top of this PR.

[ulucinar]

/test-examples="examples/iam/v1beta1/role.yaml"

[ulucinar]

/test-examples="examples/iam/v1beta1/role.yaml"

[ulucinar]

/test-examples="examples/ec2/v1beta1/vpc.yaml"

[ulucinar]

/test-examples="examples/ec2/v1beta1/vpc.yaml"

[ulucinar]

/test-examples="examples/ec2/v1beta1/vpc.yaml"

[ulucinar]

/test-examples="examples/rds/v1beta1/clusterroleassociation.yaml"

[ulucinar]

/test-examples="examples/docdb/v1beta1/cluster.yaml"

[ulucinar]

/test-examples="examples/rds/v1beta1/clusterinstance.yaml"

[ulucinar]

check-diff failure is currently expected because we are not running the code transformer from crossplane/upjet#331 as part of the generate make target yet.

[turkenf]

/test-examples="examples/rds/dbinstanceautomatedbackupsreplication.yaml"
Wrong file path

[turkenf]

/test-examples="examples/rds/v1beta1/dbinstanceautomatedbackupsreplication.yaml"

Uptest run: https://github.com/upbound/provider-aws/actions/runs/7629167136

[turkenf]

/test-examples="examples/rds/v1beta1/clustersnapshot.yaml"

Uptest run: https://github.com/upbound/provider-aws/actions/runs/7629183379

[turkenf]

/test-examples="examples/rds/v1beta1/clusteractivitystream.yaml"

Uptest run: https://github.com/upbound/provider-aws/actions/runs/7629188626

[turkenf]

/test-examples="examples/rds/v1beta1/clusterendpoint.yaml"

Uptest run: https://github.com/upbound/provider-aws/actions/runs/7629205986

[turkenf]

/test-examples="examples/rds/v1beta1/clusterparametergroup.yaml"

Uptest run: https://github.com/upbound/provider-aws/actions/runs/7629208677

[turkenf]

/test-examples="examples/rds/v1beta1/dbsnapshotcopy.yaml"

Uptest run: https://github.com/upbound/provider-aws/actions/runs/7629211590

[ulucinar]

/test-examples="examples/docdb/v1beta1/cluster.yaml"

[ulucinar]

/test-examples="examples/rds/v1beta1/clusterroleassociation.yaml"

[ulucinar]

The linter failure is expected as discussed in the description of the PR. After we merge upbound/official-providers-ci#178, we will need to retrigger the CI job.

[ulucinar]

/test-examples="examples/docdb/v1beta1/cluster.yaml"

[ulucinar]

/test-examples="examples/docdb/v1beta1/cluster.yaml"

[sergenyalcin]

Thanks @ulucinar @turkenf LGTM!

[ulucinar]

/test-examples="examples/rds/v1beta1/clusterroleassociation.yaml"

[ulucinar]

Hi @sergenyalcin,
Did some cleanup in the commit history. Thank you!