This document describes the architecture of the til interpreter, and explains the design choices behind it.
The architecture of til borrows a few ideas from HashiCorp Terraform. For instance, we reason about a
Bridge description as a directed graph where each component of a messaging system is a vertex ("node")
connected to one or more other components by an edge ("link") that represents the flow of events in that system
(Terraform calls it the "Resource Graph").
A graph is also a good fit for HCL's evaluation model, where the dependencies of each expression should be
analyzed in order to infer the evaluation context. For example, the evaluation of a router block containing a
reference to a transformer block can only be performed after the transformer block itself is evaluated. The HCL
guide refers to this pattern as Interdependent Blocks.
However, this is where the comparison with Terraform ends due to fundamental differences in what both of these tools try to achieve.
Like Terraform, til...
- Interprets configuration files written in a language based on the HCL syntax (the TriggerMesh Integration Language).
- Supports cross-references between HCL configuration blocks using traversal expressions.
- Represents configuration blocks and the relationships between them as a directed graph.
- Uses internal schemas (
hcldec.Spec) for decoding the contents of HCL blocks into concrete types/values.
Unlike Terraform, til...
- Is not concerned about the deployment aspects of the components it interprets. The interpreter generates deployment manifests which users are free to deploy using the tool of their choice (kubectl, Helm, Terraform, ...).
- Does not depend on side-effects during runtime to evaluate certain parts of a configuration (such as creating a certain resource in order to be able to determine the value of a variable). A Bridge Description File is interpreted statically.
- Leverages graphs to represent data flows, not dependencies.
Below is a high level representation of the internal request flow within the interpreter when a user generates deployment manifests from a Bridge description. Each subsystem is described in more details in the rest of this document.
The role of the core.GraphBuilder is to place each component of a Bridge onto a graph and connect it
to other components, based on the information contained in its HCL block.
This is achieved by successive transformations performed by implementers of the
core.GraphTransformer interface. In the process, the graph builder attaches to each graph vertex any
information susceptible to be useful to translators (e.g. a schema to decode the contents of the corresponding HCL
block, looked up in a map of supported component implementations). The inspiration for this iterative approach comes
from Terraform's design, and aims at making the process of building the graph more testable, even though the complexity
of til is not comparable to Terraform's.
In order to avoid leaking the "graph" domain into the "configuration" domain, and having to implement large interfaces
for each type in the config package, we leverage Go's behavioral polymorphism by wrapping config types
(such as config.Channel, config.Source, etc.) into structs that represent graph vertices. Each vertex type (e.g.
core.ChannelVertex) implements methods that are only relevant to the core package, and
generally to a single core.GraphTransformer. For instance, if a type of component can be the destination of events
within the Bridge, and by definition exposes a Knative Addressable type, its vertex type will implement
core.AddressableVertex.
For a component type to be able to appear in a Bridge description, it needs to have a corresponding implementation that allows the interpreter to decode its HCL configuration and translate it into one or more Kubernetes API objects.
Such implementation combines three Go interfaces defined in the translation package: Decodable,
Translatable and Addressable. The semantics of those interfaces maps directly into the types of operations performed
by the interpreter:
Decodableprovides thehcldec.Specthat is used by HCL APIs (such ashcldec.Decode) for decoding component-type-specific segments of HCL configuration into data that can be interpreted in the Go language. As long as a component requires some kind of non-generic configuration, it must implement this interface.Translatablebridges the gap between a HCL configuration block and its representation in the Kubernetes space by generating values that represent Kubernetes objects, from configurations decoded from the Bridge description. All components implement this interface.Addressableallows determining the address at which a component accepts events. Components that can ingest events implement this interface so that HCL reference expressions in other components can be evaluated to actual addresses.
As a means of comparisons, Terraform has a concept of "provider" which also interacts with custom "resource" types by using decode schemas and calling CRUD functions on the decoded configuration data.
Currently, all components implementations reside in-tree, inside sub-packages of internal/components.
The core.BridgeTranslator is responsible for evaluating the simple connected graph built by
core.GraphBuilder, and translating it into a list of Kubernetes manifests which can be deployed to a target
environment (cluster) to materialize the Bridge.
It does so by creating a topological ordering of the graph which ensures, whenever possible, that the address (event destination) of a given Bridge component is readily available when components which depend on that address are evaluated/translated. This ordering allows the translation to be accurate without having to visit a graph vertex multiple times (e.g. once to store its event address in the evaluation context, once to translate it after all addresses within the Bridge have been determined).
Event addresses are stored as variables inside a core.Evaluator, which accumulates them as the graph
is being walked.
