Terraform deployment not compatible with CID dashboard deployment

[nickcasscs]

When deploying either the CUDOS, Cost Intelligence or KPI dashboards the CID Dashboard deployment via terraform wrapper uses a cloudformation template that relies on a stack output cid-DataExport-Database from the cloudformation destination deployment. When deploying the solution using the cur-setup-destination terraform module this stack doesn't exist and as a result, the stack output also does not exist; this causes the stack deploy to fail.

By passing CurVersion = "1.0" to the cid deployment stack parameters this issue looks like it is resolved.
There isn't any reference in the documentation that this stack parameter is required currently in either the Terraform deployment instructions or the README for the cid-dashboards terraform module.

It would save implementers a lot of time spent troubleshooting if this is noted in the documentation.

[iakov-aws]

@sean-nixon we probably need to provide terrarform wrappers for Data Exports modules as well

[christrotter]

Also running into something similar. We are being advised by our TAM to go forward with CUDOS v5 and CUR 2.0, but the 'source' Terraform module provided only creates the aws_cur_report_definition resource: https://github.com/aws-samples/aws-cudos-framework-deployment/blob/main/terraform-modules/cur-setup-source/main.tf#L253

That resource is legacy and not CUR 2.0.

Rather, the resource it should be using is this:aws_bcmdataexports_export
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/bcmdataexports_export

For us to move forward, we will have to copy the source module and modify it so that it uses the correct resource. However - I think that once that issue is addressed, the rest of the provided bits and pieces should work, as all of the complicated downstream infrastructure has already been coded to support CUR 2.0.

I will report back once I have been successful!

[christrotter]

Some notes on the changes I had to make.

• Adding logic via a variable to allow for choosing cur 1.0 or cur 2.0; which creates one or the other resources (alternatively, could be re-worked pretty easily to do both)
• Fixing some deprecated inline policy statements (moving to policy attachments)
• Added bcm permissions
• Adding an output with failure handling

So far, working for me!

I wrote the code/test/readme stuff but didn't get to the steps of actually making a PR. Let me know if you'd like me to do so.

[iakov-aws]

Great job. Please submit a pr. Much appreciate your contribution

[christrotter]

Hopefully this is helpful. I followed the contributions guide as best I could.

#1040

[christrotter]

Amusingly, I was already using that CurVersion parameter when I wrote my PR. However, due to switching to v2 and removing that parameter, I'm now getting the same cid-DataExport-Database error that @nickcasscs gets. Will look at addressing this next.
edit: I wrote a TF wrapper module for the data-exports-aggregation.yaml stack and that gets me farther, but encountering new issues w. bucket namings. Will keep poking!

[christrotter]

Ok, am now encountering many issues that stem from the bucket name being wrong. I believe this is due to a shift in naming conventions in the move from CUR1 to CUR2.

CUR1 = s3://${prefix}-${accountID}-shared/cur
CUR2 = s3://${prefix}-${accountID}-data-exports/cur2/

The existing TF instructions create the CUR1 naming pattern, but the data-exports-aggregation stack expects CUR2.

aws-cudos-framework-deployment/cfn-templates/data-exports-aggregation.yaml

Line 173 in 6706725

BucketName: !Sub "${ResourcePrefix}-${AWS::AccountId}-data-exports"

BucketName: !Sub "${ResourcePrefix}-${AWS::AccountId}-data-exports"

This sheds some light:

aws-cudos-framework-deployment/cfn-templates/cid-cfn.yml

Lines 136 to 140 in 6706725

	CURVersion:
	Type: String
	Default: '2.0'
	AllowedValues: ["1.0", "2.0"]
	Description: "We support 2 options: CUR 2.0 from DataExports stack managed by CID or CUR 1(Legacy). When you switch to CUR2 consider KeepLegacyCURTable parameter to yes for keeping Legacy CUR table for migration period."

i.e. "We support 2 options: CUR 2.0 from DataExports stack managed by CID or CUR 1(Legacy).

However, just a few lines down from that...

aws-cudos-framework-deployment/cfn-templates/cid-cfn.yml

Line 144 in 6706725

Default: 's3://cid-{account_id}-shared/cur/'

Default: 's3://cid-{account_id}-shared/cur/'

If CUR version2 is a default, and this is a default, then the data-exports stack can't work?

And then down here:

aws-cudos-framework-deployment/cfn-templates/cid-cfn.yml

Line 944 in 6706725

raise Exception(f'CUR BucketPath={parts[0]} format is not recognized. It must be s3://(bucket)/cur or s3://(bucket)/(curprefix)/(curname)/(curname) ')

CUR BucketPath={parts[0]} format is not recognized. It must be s3://(bucket)/cur or s3://(bucket)/(curprefix)/(curname)/(curname)

Still digging - there is something fundamental I'm missing maybe...?

[iakov-aws]

if we use CUR2, the stack with CUR2 comes with athena table and crawler. So we do not manage it here in this stack as we do for CUR2

[christrotter]

Ok, so I wrote some simple Terraform wrappers for the three Cloudformation links here: https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi/deploy
(they basically work like this: https://github.com/aws-samples/aws-cudos-framework-deployment/blob/main/terraform-modules/cid-dashboards/main.tf)

I broke apart the URLs from that page to get the needed config, plus looked at the Cloudformation YAML files for more details.

After applying them in order, everything is working wonderfully. So I think the lesson is, for CUR2/CudosV5, do not use the existing TF modules. If it would help, I can submit a PR with my example code, but honestly I whipped it off in an hour or so - not exactly a shining example of Terraform work.