Terraform module that makes Athena query history querable via Iceberg tables
The main idea of this module is to make Athena query history querable via Iceberg tables.
- Terraform 1.8.0+
- Docker 24.0.0+
- Lambda function to create the Iceberg table
- Lambda function to dispatch Event Bridge notifications to Firehose, enriching the event with the query execution context
- Event Bridge to trigger the Lambda function that dispatches the event to Firehose
- Firehose to write to S3 with Iceberg format, and Glue Catalog as Iceberg Catalog
To use properly this module you need to have a Glue Database, an Athena Workgroup and an S3 bucket to store Iceberg table.
module "athena_query_observability_from_github" {
source = "git::https://github.com/nicor88/terraform-aws-athena-query-observability?ref=v0.0.1"
glue_database_name = "athena_observability"
glue_table_name = "query_state_change_example"
s3_bucket_table_location = "s3-data-bucket"
s3_table_location_prefix = "athena_observability/query_state_change_example"
firehose_s3_error_output_prefix = "athena_observability/query_state_change_example_errors"
create_table_athena_workgroup_name = "primary"
create_table_athena_s3_output_bucket_name = "athena-query-results-bucket"
}
It might be necessary to force table recreation in some cases, for example when changing the table schema because of a change of the table schema of the module.
Use this settings:
glue_table_name = "table_name_v2"
force_table_creation_trigger = "force"
In glue_table_name set a table name different than the previous one e.g. "table_name_v2". In force_table_creation_trigger put a value diffrent from the default value, that is an empty string.
force_table_creation = "true"
force_table_creation_trigger = "force_recreating_table"
firehose_name_suffix = "-v2"
When forcing table recreation, it can be that some events gets losts, therefore be careful when using any of the proposed options.
- It can happen that the module doesn't delete Cloudwatch logs, you might need to do it manually. Anyhow, this can be problematic if you remove the module and add it again.
- When forcing table recreation, it can happen that Firehose doesn't pick up the new table. The solution is to force recreation of the Firehose delivery stream by setting the
firehose_name_suffix
variable, or read the force table recreation section.