1BRC in Tinybird

[sdairs]

Aloha!

Tinybird is a cloud analytics platform built on top of ClickHouse - there's been some discussions solving 1brc with ClickHouse already, so I took a slightly different approach in thinking about it as a streaming problem and contasting it with the batch side.

I built a little web app to make the example interactive, as there is nothing you can run locally for this example: https://1brc.alasdairb.com/

You can run the query in batch (aggregate all 1bn rows) which takes 3-4 seconds, or, if we re-imagine the problem as a streaming one (i.e. we are constantly being fed new measurement events) we can solve the problem in realtime using event-driven incremental aggregation at ingestion time. The web app has a button to push new fake measurements into Tinybird, which are aggregated in real time using a materialized view. You can run the realtime aggregation, which is really just querying the latest pre-aggregates from the MV - thus taking <30ms even with >1bn rows ingested through it.

The source is here:
https://github.com/sdairs/1brc

As a data engineer, it took about 2 minutes to solve the data side, and a full day to write the crappy frontend which is not my area of expertise 😂