[DRAFT] WIP QML Load Snapshot Signet (160,000 height) #424

D33r-Gee · 2024-10-11T17:00:10Z

Based on #421. For evaluation purposes only!

GUI Integration for UTXO Snapshot Loading

Overview

This PR adds initial GUI support for loading a signet UTXO snapshot at height 160,000, building on Bitcoin Core's assumeutxo infrastructure.

What This PR Does

Adds a basic GUI interface for loading UTXO snapshots via the Connection Settings panel
Implements non-blocking snapshot loading via QT Thread
Provides visual confirmation of snapshot activation

Implementation Details

Core Components Modified

Node Interface (src/node/interfaces.cpp)

Implements snapshotLoad() based on the loadtxoutset RPC
Uses background QT thread for non-blocking operation

QML Integration (src/qml/models/nodemodel.cpp)

Manages snapshot loading state and progress
Uses Qt's thread system to prevent UI blocking
Provides progress updates via Qt signals

UI Components (src/qml/components/SnapshotSettings.qml)

File selection dialog
Snapshot verification status

Key Design Decisions

Non-blocking Implementation

Snapshot loading runs in separate QT thread
UI remains responsive during load

Extensibility

Interface designed to accommodate future assumeutxo changes
Error handling framework in place

Testing Instructions

Build
Launch Bitcoin Core QML GUI
Navigate to Connection Settings
Test with sample snapshot:

magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat

Verify (see screenshots below):
- "Snapshot Loaded" confirmation appears
- Block clock shows ~50% progress
- For errors, check debug.log for [snapshot] or [loadsnapshot] messages

Ubuntu 22.04 Screenshots

Expected Behavior

File selection dialog works
Success/failure state properly displayed
Node becomes usable while background validation continues

Future Work

Handler Implementation

Add proper Handler interface for snapshot events
Implement callback-based progress updates
Remove polling-based progress updates

Error Handling

Add user-facing error messages
Implement recovery paths
Add snapshot validation UI once background validation has completed

Integration

Coordinate with upstream assumeutxo changes
Add support for network-specific snapshots (testnet and mainnet)
Integrate with initial setup wizard

Notes for Reviewers

Snapshot Compatibility

Currently only works with signet UTXO snapshot
Limited to specific 160,000 height
Validates against chain parameters m_assumeutxo_data
(the changes there can be discarded once synced since m_assumeutxo_data is already hardcoded)

This is a work in progress - feedback welcome on the approach and implementation details.

D33r-Gee · 2024-10-15T17:46:32Z

With cb409e3 rebased and got rid off the extra options_mode settings which are not relevant for this evaluation.

Also deleted the extra loadsnapshot.cpp file and instead added the load snapshot function in src/node/interfaces.cpp

yashrajd · 2024-10-16T14:05:33Z

Should I be getting a binary specific for this issue? I tried loading a signet snapshot into the one for #421 but was unable to...

D33r-Gee · 2024-10-16T16:28:05Z

Should I be getting a binary specific for this issue?

Yes please download the artifact for your system here

I tried loading a signet snapshot into the one for #421 but was unable to...

That's correct behaviour, 421 is UI only and doesn't not have backend wiring

D33r-Gee · 2024-10-16T19:59:27Z

With b77f7c1 minor tweaks to the src/node/interfaces.cpp snapshotload() function.

It mimics the rpc call loadtxoutset from bitcoin/bitcoin repository

Next will connect the loading to the progress bar

D33r-Gee · 2024-10-22T22:12:36Z

with c59e557 Integrated the UI with the C++ backend, implementing a progress callback function to dynamically update the loading progress based on the number of coins processed.

Touched src/validation.h and src/validation.cpp to extract the coin loading progress.

Next: will update the "Loaded Snapshot" stack with the real info from the snapshot

D33r-Gee · 2024-10-25T20:46:36Z

With e3f8c70 moved the load snapshot functionality from nodeModel to optionsModel since it make it easier to store the snapshot info and reduces redundant variable declarations.

jarolrod

With e3f8c70 moved the load snapshot functionality from nodeModel to optionsModel since it make it easier to store the snapshot info and reduces redundant variable declarations.

e3f8c70 is a regression because optionsmodel is not an appropriate place for this, and we are now violating the role of settings.json to store the state of a process.

NodeModel and AppMode are more appropriate places for this information. But the entire approach here is still a concept NACK as we are performing unecessary changes to validation and chainparams.

src/qml/models/options_model.cpp

jarolrod · 2024-10-26T22:34:28Z

src/qml/models/options_model.cpp

+    if (new_snapshot_load_completed != m_snapshot_load_completed) {
+        m_snapshot_load_completed = new_snapshot_load_completed;
+        if (m_onboarded) {
+            m_node.updateRwSetting("snapshotloadcompleted", new_snapshot_load_completed);


This would be polluting settings.json that stores read/write configuration options for core -> with a state of a process. We cannot do this.

Makes sense, although I have a question for when we would want to allow the user to load a snapshot during onboarding, could the snapshot file path be stored in the settings.json?

yashrajd · 2024-10-29T10:27:35Z

Ok I tested artefact#235 and functionality & UI seems to work.

Didn't see block clock so couldn't verify easily. Adding some UI feedback below:

fix list item width

fix list item width
fix icon size

blockHeight is accurate
the date in the copy above it seems right
didn't check hash accuracy
hash value is overflowing as you can see, I'd probably wrap it over 2-3 lines, perhaps add copy functionality

D33r-Gee · 2024-10-31T22:22:43Z

with 90dc0dd rebased over main and addressed @jarolrod concerns by reverting back to just a minimal integration of the loading snapshot functionality.

@yashrajd I will address the UI issues next

As a follow up will investigate how to access the snapshot verification progress

Also looking into how to make the snapshot data persist and be available for display (i.e. snapshot block height, date, and merkle hash)

D33r-Gee · 2024-10-31T22:30:40Z

But the entire approach here is still a concept NACK as we are performing unecessary changes to validation and chainparams.

For the chainparams the changes there are only temporary for evaluating and testing within this repo. I checked upstream and the signet snapshot has already been hardcoded so once our repo is synced the changes in chainparams.cpp can be discarded.

D33r-Gee · 2024-11-01T18:46:48Z

with 85592f5 made the snapshot activation persist on node restart by making use of the interface with chain->hasAssumedValidChain

Also changed the green-check icon size to 30 pix

Next looking into how to extract the progress from the ActivateSnapshot process

D33r-Gee · 2024-11-04T23:38:00Z

with 955c65d modified the snapshotLoad() function in src/node/interfaces.cpp and the initializeSnapshot() function in nodemodel.cpp to extract progress information using regular expressions.

This design decision aimed to keep snapshot loading mechanisms self-contained.

However, it looks like the use of std::stod is triggering some lint failures... will look into how to address that.

Perhaps the way to go is to use the handler in src/node/interfaces_ui ... will explore that path then update

D33r-Gee · 2024-11-08T21:48:51Z

with 0eba513 reverted back to minimal snapshot load functionality.

Snapshot loading progress is still not were I want it to be so in the meantime replaced the progress bar with placeholder text.

Also updated the snapshot info display. once successfully loaded, and I'm using the hardcoded info in chainparams since if the snapshot has loaded successfully it means the info there has been validated.
This is make the info persistent and not rely on the snapshot.dat file (in case it gets deleted)

D33r-Gee · 2024-11-12T21:33:40Z

With cd2ec69 added a section to keep track of snapshot loading, it is not ideal so will look for better ways to achieve it.

Since the core functionality is there will ask for "pre-reviews" before un-drafting

src/qml/components/SnapshotSettings.qml

D33r-Gee · 2024-11-13T21:24:50Z

with 5d0756c updated SnapshotSettings.qml with comments documenting the loading process.

with 262bb03 updated nodemodel by adding the m_snapshot_loading bool so that the "Snapshot Loading" page persist if the user navigates away from it.

Next looking into how to finesse the snapshot loading progress...

D33r-Gee · 2024-11-13T23:03:09Z

with a6bede5 removed comments from SnapshotSettings.qml

D33r-Gee · 2024-11-19T17:09:17Z

with 799b4cd updated SnapshotSettings.qml by deleting comments and adding more properties to filedialog.

Also wanted to see if the Win64 CI completed successfully, which it did.

Next will ask for feedback before un-drafting

D33r-Gee · 2024-11-22T18:25:26Z

with 5d2ffb0 updated snapshotLoad() in src/node/interfaces.cpp. Left the polling loop for now as I investigate how to handle loading the snapshot if Pre-syncing and Syncing headers is still happening.

Also removed JS functions and moved the logic to the nodemodel

This introduce the UI flow to load a AssumeUTXO snapshot into the Bitcoin Core App. It modifies the connection seetings and adds a SnapshotSettings file, Icon, and modified profress bar. It has been rebased

Adds minimal wiring to connect QML GUI to loading a signet UTXO snapshot via the connection settings. Uses SnapshotSettings.qml to allow user interaction. Modifies src/interfaces/node.h, src/node/interfaces.cpp and chainparams.cpp (temporarily for signet snapshot testing) to expose snapshot loading functionality through the node model. Current limitations: - Not integrated with onboarding process - Requires manual navigation to connection settings after initial startup - Snapshot verification progress is not implemented yet Testing: 1. Start the node 2. Complete onboarding 3. Navigate to connection settings 4. Load snapshot from provided interface

D33r-Gee · 2024-11-25T17:04:07Z

8beb50a rebased over main

Sjors · 2024-12-09T04:00:57Z

src/kernel/chainparams.cpp

@@ -371,6 +371,13 @@ class SigNetParams : public CChainParams {

        vFixedSeeds.clear();

+        m_assumeutxo_data = MapAssumeutxo{
+            {
+                160000,


8beb50a: the signet snapshot was merged into Bitcoin Core more than a year ago in bitcoin/bitcoin@edbed31. This also seems to be using the older serialization format from before bitcoin/bitcoin#29612.

I guess that's all fine for a proof of concept, but it would be better if the main branch of this repo was more up to date.

Hi Sjors! Thanks for taking a look and yes completely agree with:

it would be better if the main branch of this repo was more up to date

We are currently taking steps to sync with upstream, meanwhile I wanted to get the ball rolling, although I will pull in the update versions of the assume utxo functionality so that when the sync happens this PR can be merged with minimal refactoring.

Another thing you could do is look into making a PR to the Bitcoin Core repo with your proposed interface changes. One way to go about that is to somehow use them in existing or a new RPC method. E.g. snapshot load progress is probably useful to have available.

Another thing you could do is look into making a PR to the Bitcoin Core repo with your proposed interface changes.

That's the current medium term plan, once we have an implementation that works and makes sense here we plan to go upstream (bitcoin/bitcoin) and open a PR there.

johnny9

A handler similar to handleNotifyBlockTip will be needed to so the UI interface can be event driven instead of polling the interfaces module.

johnny9 · 2024-12-09T19:54:44Z

src/node/interfaces.cpp

+        // Wait for the block to appear in the block index
+        //TODO: remove this once another method is implemented
+        constexpr int max_wait_seconds = 600; // 10 minutes
+        for (int i = 0; i < max_wait_seconds; ++i) {


We shouldn't have a blocking component here. If there are any issues we should just fail.

johnny9 · 2024-12-09T19:55:13Z

src/node/interfaces.cpp

    ArgsManager& args() { return *Assert(Assert(m_context)->args); }
    ChainstateManager& chainman() { return *Assert(m_context->chainman); }
    NodeContext* m_context{nullptr};
+    std::atomic<double> m_snapshot_progress{0.0};


I imagine the actual progress should be owned by the object in charge of doing the processing. ChainStateManager?

Yes! Will set it up so that the handler notifies the snapshot progress and that ownership belongs to ChainstateManager.

Sjors · 2024-12-10T02:47:37Z

src/qml/components/SnapshotSettings.qml

+                descriptionSize: 17
+                descriptionLineHeight: 1.1
+                description: snapshotInfo && snapshotInfo["date"] ?
+                    qsTr("It contains transactions up to %1. Newer transactions still need to be downloaded." +


It's not really "transactions up to", it's all coins (utxo set) that exist at the snapshot height.

Maybe say something like:

It contains a snapshot as of block %1. Once loaded, all blocks from %2 to the present are download. After that Bitcoin Core is ready to use, while all historical blocks from genesis to %2 are downloaded and validated in the background.

(intentionally leaves it vague what's in the snapshot)

Great suggestion! I will run by the designers...

@yashrajd @GBKS what are your thoughts on the suggestion? Does it make sense?

In the sequence, this is the fourth snapshot-related screen (state) people see:

Snapshot option in settings where people learn about the benefit of this feature

"Load snapshot" screen that gives more detail about how it works

"Loading snapshot" screen state with a loader that tells people they can do other stuff in the mean time

"Snapshot loaded" screen state that informs people what was imported, what to keep in mind and what will happen next (this is the screen that we are talking about in this thread)

I think the point that not all historical transactions are in the snapshot is worth clarifying. The first time people are exposed to what this feature does is in 2, so I'd probably start tweaking at that point. We call it a "recent transaction snapshot" there, which is more precise than just "transactions" ("unspent transaction snapshot" would be a little closer).

We don't need to be 100% technically accurate and I'm not sure users are well-served if we're completely vague about this either. They should be able to form a good enough mental model for making decisions and understanding their impact.

So how about this?

It contains recent transactions up to %1. Newer and historical transactions will be automatically downloaded and verified.

Or this?

It contains unspent transactions up to %1. Newer and historical transactions will be automatically downloaded and verified.

Either way, we should ensure to tweak the text across the sequence if we make changes here.

We show the block height in a "View details" option below (did not check if it's implemented in this PR already), so not sure whether we need to include that in the text as well.

As for what happens next, my thinking was that we just say that the data will be filled in and verified here (keep it short), and then have indicators contextually in the UI as to what is happening (balance, send and transaction list have respective load states). So less absorption of logic here, and more contextual info in the appropriate places.

Sorry, this has gotten quite long. Hope it's a helpful response.

We don't need to be 100% technically accurate and I'm not sure users are well-served if we're completely vague about this either.

I think it's better to be incomplete than incorrect. The nice thing about being vague is that it's clear to the reader there's more to learn.

One reason the phrasing "recent / historical transactions" is potentially confusing is that people may think they can recover historical wallet transactions with it. But it has nothing to do with that.

The concept that's useful to explain is that the snapshot allows us to start syncing at block %1, saving time by skipping historical blocks. This way the user can get started (with their wallet) earlier. Meanwhile it's safe, because we do check those historical blocks in the background later.

TL;DR I reviewed the onboarding text again, and it seems like the use of transactions terminology in the snapshot section provides context continuity with what's said before...see screenshot below.

Unless we come up with something better, I'd be fine with retaining what we have currently though I agree it's inaccurate. Suggestions below:

I think it's better to be incomplete than incorrect. The nice thing about being vague is that it's clear to the reader there's more to learn.

It does seem like we can strike a better balance between technical accuracy & usefulness to user here, and use vagueness where required.

We show the block height in a "View details" option below (did not check if it's implemented in this PR already), so not sure whether we need to include that in the text as well.

Agree, this was in the initial mockups I did. edit: removed something here. read on...

The first time people are exposed to what this feature does is in 2, so I'd probably start tweaking at that point.

Agree.

"Load snapshot" screen that gives more detail about how it works.

This screen can omit mention of transactions "The snapshot allows you to start using the application more quickly. Once loaded, it will be automatically verified in the background. Learn more" instead of "You can start using the application more quickly by loading a recent transaction snapshot. It will be automatically verified in the background."

"Snapshot loaded" screen state that informs people what was imported, what to keep in mind and what will happen next (this is the screen that we are talking about in this thread)

I'd suggest something like "The loaded snapshot contains data up to [some date]. The initial download will continue now. The snapshot is verified in the background." instead of "It contains transactions up to September 10, 2023. Newer transactions still need to be downloaded. The data will be verified in the background."

edit: rephrased my own suggestion slightly.

Should we move this conversation into a separate issue? It's just about text details, and maybe we don't want to weigh down the overall technical development of this feature? Text is easy to replace separately.

@GBKS that makes sense.

Issue created. Let's continue there.

D33r-Gee marked this pull request as draft October 11, 2024 17:11

D33r-Gee force-pushed the qml-load-snapshot-signet branch from cb409e3 to 670ca05 Compare October 15, 2024 17:43

D33r-Gee mentioned this pull request Oct 15, 2024

cpp: exposing AssumeUTXO functionality to the GUI (QML) via the Node interface bitcoin/bitcoin#31094

Closed

1 task

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 670ca05 to b77f7c1 Compare October 16, 2024 19:53

D33r-Gee force-pushed the qml-load-snapshot-signet branch from b77f7c1 to c59e557 Compare October 22, 2024 21:57

D33r-Gee force-pushed the qml-load-snapshot-signet branch from c59e557 to e3f8c70 Compare October 25, 2024 20:42

jarolrod reviewed Oct 26, 2024

View reviewed changes

D33r-Gee force-pushed the qml-load-snapshot-signet branch from e3f8c70 to 90dc0dd Compare October 31, 2024 22:09

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 90dc0dd to 85592f5 Compare November 1, 2024 18:41

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 85592f5 to 955c65d Compare November 4, 2024 22:39

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 955c65d to 0eba513 Compare November 8, 2024 18:35

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 0eba513 to cd2ec69 Compare November 12, 2024 21:30

jarolrod reviewed Nov 12, 2024

View reviewed changes

src/qml/components/SnapshotSettings.qml Outdated Show resolved Hide resolved

jarolrod reviewed Nov 12, 2024

View reviewed changes

src/qml/components/SnapshotSettings.qml Outdated Show resolved Hide resolved

D33r-Gee force-pushed the qml-load-snapshot-signet branch 2 times, most recently from 5d0756c to 262bb03 Compare November 13, 2024 21:23

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 262bb03 to a6bede5 Compare November 13, 2024 22:50

D33r-Gee force-pushed the qml-load-snapshot-signet branch from a6bede5 to 799b4cd Compare November 19, 2024 16:09

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 799b4cd to f23ffb8 Compare November 21, 2024 18:38

D33r-Gee changed the title ~~[DRAFT] WIP QML Load Snapshot Signet~~ [DRAFT] WIP QML Load Snapshot Signet (160,000 height) Nov 21, 2024

D33r-Gee force-pushed the qml-load-snapshot-signet branch from f23ffb8 to 5d2ffb0 Compare November 22, 2024 18:01

D33r-Gee added 2 commits November 25, 2024 08:19

qml: Introduce UI Flow for Loading Snapshot

bc72124

This introduce the UI flow to load a AssumeUTXO snapshot into the Bitcoin Core App. It modifies the connection seetings and adds a SnapshotSettings file, Icon, and modified profress bar. It has been rebased

D33r-Gee force-pushed the qml-load-snapshot-signet branch from 5d2ffb0 to 8beb50a Compare November 25, 2024 17:00

Sjors reviewed Dec 9, 2024

View reviewed changes

johnny9 reviewed Dec 9, 2024

View reviewed changes

Sjors reviewed Dec 10, 2024

View reviewed changes

GBKS mentioned this pull request Dec 16, 2024

Revise snapshot feature copy BitcoinDesign/Bitcoin-Core-App#132

Open

[DRAFT] WIP QML Load Snapshot Signet (160,000 height) #424

Are you sure you want to change the base?

[DRAFT] WIP QML Load Snapshot Signet (160,000 height) #424

Conversation

D33r-Gee commented Oct 11, 2024 • edited Loading

GUI Integration for UTXO Snapshot Loading

Overview

What This PR Does

Implementation Details

Core Components Modified

Key Design Decisions

Testing Instructions

Expected Behavior

Future Work

Notes for Reviewers

D33r-Gee commented Oct 15, 2024

yashrajd commented Oct 16, 2024

D33r-Gee commented Oct 16, 2024

D33r-Gee commented Oct 16, 2024 • edited Loading

D33r-Gee commented Oct 22, 2024 • edited Loading

D33r-Gee commented Oct 25, 2024

jarolrod left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashrajd commented Oct 29, 2024 • edited Loading

D33r-Gee commented Oct 31, 2024 • edited Loading

D33r-Gee commented Oct 31, 2024

D33r-Gee commented Nov 1, 2024

D33r-Gee commented Nov 4, 2024

D33r-Gee commented Nov 8, 2024

D33r-Gee commented Nov 12, 2024

D33r-Gee commented Nov 13, 2024

D33r-Gee commented Nov 13, 2024

D33r-Gee commented Nov 19, 2024

D33r-Gee commented Nov 22, 2024

D33r-Gee commented Nov 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnny9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sjors Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

D33r-Gee Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashrajd Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

D33r-Gee commented Oct 11, 2024 •

edited

Loading

D33r-Gee commented Oct 16, 2024 •

edited

Loading

D33r-Gee commented Oct 22, 2024 •

edited

Loading

yashrajd commented Oct 29, 2024 •

edited

Loading

D33r-Gee commented Oct 31, 2024 •

edited

Loading

Sjors Dec 10, 2024 •

edited

Loading

D33r-Gee Dec 10, 2024 •

edited

Loading

yashrajd Dec 13, 2024 •

edited

Loading