Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Build Enriched Traces & Transactions + Aggregation Tables #1161

Draft
wants to merge 47 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
c4353f0
draft
MSilb7 Dec 12, 2024
5ff10a9
adds
MSilb7 Dec 12, 2024
a2eade6
fix func names
MSilb7 Dec 12, 2024
ee2d62d
init push
MSilb7 Dec 12, 2024
ebeb470
push example aggregation
MSilb7 Dec 12, 2024
cc12c72
mod
MSilb7 Dec 13, 2024
443e6c4
mods
MSilb7 Dec 13, 2024
b33ee03
juice it up
MSilb7 Dec 13, 2024
dd03a2a
resolve sources
MSilb7 Dec 13, 2024
ccc8ed9
migrate to existing transaction_fees and rename
MSilb7 Dec 13, 2024
563ad68
update names
MSilb7 Dec 13, 2024
365ee57
checkpoint
MSilb7 Dec 13, 2024
ae2f95c
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 13, 2024
6503ab7
check
MSilb7 Dec 13, 2024
e4a2c77
check
MSilb7 Dec 13, 2024
a785060
undo
MSilb7 Dec 13, 2024
7905ab0
fix eet
MSilb7 Dec 13, 2024
6592682
simplify views
MSilb7 Dec 13, 2024
1617cba
mod nb
MSilb7 Dec 13, 2024
09cfe79
push latest - through refined transactions
MSilb7 Dec 13, 2024
012bed3
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 13, 2024
379100c
clean
MSilb7 Dec 13, 2024
9d09853
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 13, 2024
a185a85
push updates to models
MSilb7 Dec 13, 2024
0fc755f
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 13, 2024
e705cac
checkpoint, base models DONE
MSilb7 Dec 13, 2024
28d7b5f
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 14, 2024
61cab69
separate l1 fee scalar versions
MSilb7 Dec 14, 2024
ae7b7db
fixes and create aggregate trace models
MSilb7 Dec 14, 2024
dd00b26
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 14, 2024
b11df41
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 15, 2024
7112e82
split txs and traces - push tx aggregates
MSilb7 Dec 15, 2024
8164f9a
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 15, 2024
6e03596
Merge branch 'main' into traces-txs-models-v0
MSilb7 Dec 16, 2024
45ed343
rename traces & include all types
MSilb7 Dec 16, 2024
9f147e2
push dev notebook
MSilb7 Dec 16, 2024
afeb39d
Merge branch 'main' into traces-txs-models-v0
lithium323 Dec 16, 2024
f6da4d4
Merge branch 'main' into traces-txs-models-v0
lithium323 Dec 16, 2024
0633750
Undo changes to example_daily_data.ipynb
lithium323 Dec 16, 2024
e2e57ae
Apply sqlfluff to refined_transactions_fees.sql.j2
lithium323 Dec 16, 2024
d634da1
Merge branch 'main' into traces-txs-models-v0
lithium323 Dec 16, 2024
70b34f5
Make diff wih main smaller
lithium323 Dec 16, 2024
2190036
Merge branch 'main' into traces-txs-models-v0
lithium323 Dec 17, 2024
a49845e
Make diff wih main smaller
lithium323 Dec 17, 2024
0489465
Merge branch 'main' into traces-txs-models-v0
lithium323 Dec 17, 2024
9226dde
Format sql
lithium323 Dec 18, 2024
cbea965
Merge branch 'main' into traces-txs-models-v0
lithium323 Dec 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions src/op_analytics/datapipeline/models/code/enriched_traces.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import duckdb

from op_analytics.datapipeline.models.compute.querybuilder import TemplatedSQLQuery
from op_analytics.datapipeline.models.compute.registry import register_model
from op_analytics.datapipeline.models.compute.types import NamedRelations


@register_model(
input_datasets=["ingestion/traces_v1", "ingestion/transactions_v1", "ingestion/blocks_v1"],
expected_outputs=["enriched_traces_v1"],
auxiliary_views=[
TemplatedSQLQuery(
template_name="transaction_fees",
context={},
),
TemplatedSQLQuery(
template_name="enriched_transactions",
context={},
),
TemplatedSQLQuery(
template_name="enriched_traces",
context={},
),
],
)
def daily_address_summary(duckdb_client: duckdb.DuckDBPyConnection) -> NamedRelations:
MSilb7 marked this conversation as resolved.
Show resolved Hide resolved
return {
"enriched_traces_v1": duckdb_client.view("enriched_traces"),
}
25 changes: 25 additions & 0 deletions src/op_analytics/datapipeline/models/code/enriched_transaction.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import duckdb

from op_analytics.datapipeline.models.compute.querybuilder import TemplatedSQLQuery
from op_analytics.datapipeline.models.compute.registry import register_model
from op_analytics.datapipeline.models.compute.types import NamedRelations


@register_model(
input_datasets=["ingestion/transactions_v1", "ingestion/blocks_v1"],
expected_outputs=["enriched_transactions_v1"],
auxiliary_views=[
TemplatedSQLQuery(
template_name="transaction_fees",
context={},
),
TemplatedSQLQuery(
template_name="enriched_transactions",
context={},
),
],
)
def daily_address_summary(duckdb_client: duckdb.DuckDBPyConnection) -> NamedRelations:
return {
"enriched_transactions_v1": duckdb_client.view("enriched_transactions"),
}
Loading