Idea: Modules

[teej]

Like Terraform, Titan would benefit from a way to group resources into a template that can be instantiated many times.

The primary motivation for this is to allow Titan to provide better abstractions for some resource types. Top of mind for me are access roles, and other RBAC patterns, where you have a role with a standardized set of grants attached to it. In the future, Titan should support more complex resource structures, like task DAGs or data shares.

Custom user modules should also be possible. Today, this is straightforward for users to accomplish in Python. But for Yaml there isn't a great way to approximate a module. It should work like Terraform, where users can have a module.yml side-by-side with a titan.yml, or have the module in a standalone repo that users can import with a git reference.

I don't have an intuition for how/if modules would work with the export cli, so I will defer that for now.

Modules:

• Should declare inputs and outputs. They should be typed and should coerce and type check at runtime.
• Should allow arbitrary collections of resources
• Should allow lightly scoped or unscoped name references. For example, a module author should be able to specify a set of schema-scoped resources without specifying a schema name.
• Should be able to specify a module name similar to a resource name.

[teej]

Here's what a module might look like for Access Roles in Snowflake.

---
# titan.yml
modules:
  - name: raw_db_access_roles
    source: "./db_access_role.module.yml"
    role_name: RAW_DB_READ
    database: RAW
    schemas: 
      - fivetran
      - quickbooks
      - salesforce
      - rippling

---
# db_access_role.module.yml
variables:
  - name: role_name
    type: string
    description: The name of the role

  - name: database
    type: string

  - name: schemas
    type: list[string]
    default:
      - PUBLIC

output:
  - name: roles
    type: list[string]
    description: Names of the access roles created
    value: ...tbd...


resources:
  # Read-only role
  - resource_type: role
    for_each: var.schemas
    name: "db_{{ var.database }}_sch_{{ each.value }}_read"

  # # Database usage grant
  - resource_type: grant
    priv: USAGE
    on_database: "{{ var.database }}"
    for_each: var.schemas
    to_role: "db_{{ var.database }}_sch_{{ each.value }}_read"

  # # Schema usage grants
  - resource_type: grant
    for_each: var.schemas
    to_role: "db_{{ var.database }}_sch_{{ each.value }}_read"
    priv: USAGE
    on_schema: "{{ var.database }}.{{ each.value }}"

  # # Table grants
  - resource_type: grant
    for_each: var.schemas
    to_role: "db_{{ var.database }}_sch_{{ each.value }}_read"
    priv: SELECT
    on_all_tables_in_schema: "{{ each.value }}"

  - resource_type: future_grant
    for_each: var.schemas
    to_role: "db_{{ var.database }}_sch_{{ each.value }}_read"
    priv: SELECT
    on_future_tables_in_schema: "{{ each.value }}"

  # # View grants
  - resource_type: grant
    for_each: var.schemas
    to_role: "db_{{ var.database }}_sch_{{ each.value }}_read"
    priv: SELECT
    on_all_views_in_schema: "{{ each.value }}"

  - resource_type: future_grant
    for_each: var.schemas
    to_role: "db_{{ var.database }}_sch_{{ each.value }}_read"
    priv: SELECT
    on_future_views_in_schema: "{{ each.value }}"

[teej]

Using a built-in module:

# titan.yml
modules:
  - name: raw_db_access_roles
    source: "titan.modules.database_access_roles"
    role_name: RAW_DB_READ
    database: RAW
    schemas: 
      - fivetran
      - quickbooks
      - salesforce
      - rippling