Batch minting #113

StanChe · 2024-09-20T02:48:31Z

A finalized version of the Bubblegum update for batch minting

Batch operations

As was observed, minting of assets constitutes over 90% of all operations pertaining to digital assets. In order to reduce the number of transactions and optimize the time and cost it takes to put your tree on-chain during the Solana heavy load events Metaplex has introduced the batch-mint operations. The batch extension to the Bubblegum program introduces offline Merkle tree creation, enabling users to prepare and manage Merkle trees offline before finalizing them on-chain. The resulting trees are fully compatible with the regular trees. The snapshot of the tree created is required to be stored off-chain so the replay of the tree creation is possible on any indexer.

Batch-Mint Operations

Introduction

The latest extension to the Bubblegum contract introduces batch-mint operations, allowing users to mint multiple cNFTs in just several transactions, which significantly reduces on-chain overhead and optimizes minting processes for large collections.

How It Works

With the batch-mint operations, users can prepare an entire set of NFTs off-chain, populate them within a Merkle tree structure, and then mint them to the tree in a small number of transactions. This process is designed to handle large-scale NFT collections more efficiently.

Steps to Perform a Batch-Mint

In order to simplify the Merkle tree creation and interactions we recommend using the SDK.

To understand the batch-mint flow, let's recall the structure of a tree data account:

+--------+----------------------+-----------------+
| Header |     Tree body        |     Canopy      |
+--------+----------------------+-----------------+
   56      depends on the tree    (2ⁿ⁺¹ - 2) * 32
  bytes    depth and buffer size       bytes

where n is the depth of the canopy.

Prepare Tree

Invoke the prepare_tree method to initialize an account with a tree header and an empty tree body and empty canopy buffer.
The tree is set up with initial parameters (tree size and canopy if required) but remains empty, allowing offline filling.

Fill Tree Offline

User populates the tree offline, ensuring all necessary leaves are in place.
If the canopy is required, then canopy leaf nodes are populated in the process of adding asset leaves to tree.
The final root, proof, and last leaf are prepared for submission.

Serialize and Upload the Tree

The offline tree is serialized and uploaded to an IPFS-like service, such as Arweave, ensuring public access.

Adding canopy (for a tree with a canopy)

To transfer canopy leaf nodes from offline tree to the solana tree data account the add_canopy method is invoked, tree body at this stage stays empty

Finalize Tree

To finalize the tree the methods finalize_tree_with_root for a tree without verified collections or finalize_tree_with_root_and_collection for a tree with one verified collection are used. Signatures from both the tree owner and a designated staker are required.
The staker is responsible for ensuring the data's availability and consistency, verifying it before signing off.

Manage the Tree
- Once the batch minting process is complete, you can manage the tree and its NFTs using all the standard Bubblegum operations.
- You can also mint additional assets into a batch-minted tree as if it's a regular tree.

📄 `prepare_tree`

Prepare a tree structure that will be used to hold multiple NFTs in a batch-mint operation. This step initializes the tree and allocates the necessary resources for subsequent operations.

Accounts

Name	Writable	Signer	Description
`tree_authority`	✅		The `TreeConfig` PDA account that is initialized by this instruction.
`merkle_tree`	✅		Unchecked account representing the Merkle tree, must be zero-initialized.
`payer`	✅	✅	The account responsible for paying the transaction fees.
`tree_creator`		✅	The creator of the tree, who must sign the transaction.
`log_wrapper`			The Solana Program Library Wrapper (`spl-noop`) program ID for logging.
`compression_program`			The Solana Program Library `spl-account-compression` program ID.
`system_program`			The Solana System Program ID.

Arguments

Argument	Offset	Size	Description
`max_depth`	0	4	The maximum depth of the Merkle tree.
`max_buffer_size`	4	4	The maximum buffer size for the Merkle tree.
`public`	8	1	An optional boolean indicating if the tree is public.

📄 `add_canopy`

Add an optional canopy to the tree structure. A canopy is used to optimize the verification process for the tree, making it easier to validate NFT ownership.

Accounts

Name	Writable	Signer	Description
`tree_authority`			The `TreeConfig` PDA account previously initialized by `prepare_tree`.
`merkle_tree`	✅		The account representing the Merkle tree to which the canopy is being added.
`tree_delegate`		✅	The delegate authorized to modify the tree.
`log_wrapper`			The Solana Program Library Wrapper (`spl-noop`) program ID for logging.
`compression_program`			The Solana Program Library `spl-account-compression` program ID.
`system_program`			The Solana System Program ID.

Arguments

Argument	Offset	Size	Description
`start_index`	0	4	The starting index for the canopy nodes being added.
`canopy_nodes`	4	???	A vector of canopy nodes (32-byte arrays) to append to the Merkle tree.

📄 `finalize_tree_with_root`

Finalize the tree structure by setting the Merkle root, which represents the entire batch of NFTs. This operation completes the preparation phase and makes the tree ready for usage.

Accounts

Name	Writable	Signer	Optional	Description
`tree_authority`	✅			The `TreeConfig` PDA account previously initialized by `prepare_tree`.
`merkle_tree`	✅			The account containing the Merkle tree structure.
`payer`	✅	✅		The account responsible for paying the transaction fees.
`tree_delegate`		✅		The delegate of the tree, responsible for finalizing it.
`staker`		✅		The account of the staker, required to have the minimal required stake to allow batch-minting.
`registrar`				The account representing the registrar for managing stake accounts.
`voter`				The account representing the voting account.
`mining`				The account representing the mining account on rewards contract.
`fee_receiver`	✅			The account designated to receive protocol fees.
`log_wrapper`				The Solana Program Library Wrapper (`spl-noop`) program ID for logging.
`compression_program`				The Solana Program Library `spl-account-compression` program ID.
`system_program`				The Solana System Program ID.
remaining accounts			✅	`Pubkeys`(s) that are 32-byte Keccak256 hash values that represent the nodes for this cNFT's Merkle proof.

Arguments

Argument	Offset	Size	Description
`root`	0	32	The Merkle root hash for the tree.
`rightmost_leaf`	32	32	The hash of the rightmost leaf node in the tree.
`rightmost_index`	64	4	The index of the rightmost leaf node in the tree.
`_metadata_url`	68	???	A string - URL for the uploaded batch-mint json, required by indexers to fetch the tree for initialization.
`_metadata_hash`	???	???	A string representing a hex-encoded xxh3_128 hash of the uploaded batch-mint json-file.

📄 `finalize_tree_with_root_and_collection`

Finalize the tree structure by setting the Merkle root and associating it with a specific NFT collection. This operation allows having a verified collection for NFTs in the batch.

Accounts

Name	Writable	Signer	Optional	Description
`tree_authority`	✅			The `TreeConfig` PDA account previously initialized by `prepare_tree`.
`merkle_tree`	✅			The account containing the Merkle tree structure.
`payer`	✅	✅		The account responsible for paying the transaction fees.
`tree_delegate`		✅		The delegate of the tree, responsible for finalizing it.
`staker`		✅		The account of the staker, required to have the minimal required stake to allow batch-minting.
`collection_authority`		✅		Either the collection update authority or a delegate.
`registrar`				The account representing the registrar for managing stake accounts.
`voter`				The account representing the voting account.
`mining`				The account representing the mining authority.
`fee_receiver`	✅			The account designated to receive protocol fees.
`collection_authority_record_pda`			✅	Either a metadata delegate record PDA for a collection delegate, or a legacy collection authority record PDA.
`collection_mint`				The account representing the collection mint.
`collection_metadata`	✅			Metadata account of the collection. Modified in the case of a sized collection.
`edition_account`				The account representing the Master Edition account of the collection.
`log_wrapper`				The Solana Program Library Wrapper (`spl-noop`) program ID for logging.
`compression_program`				The Solana Program Library `spl-account-compression` program ID.
`system_program`				The Solana System Program ID.
remaining accounts			✅	`Pubkeys`(s) that are 32-byte Keccak256 hash values that represent the nodes for this cNFT's Merkle proof.

Arguments

Argument	Offset	Size	Description
`root`	0	32	The Merkle root hash for the tree.
`rightmost_leaf`	32	32	The hash of the rightmost leaf node in the tree.
`rightmost_index`	64	4	The index of the rightmost leaf node in the tree.
`_metadata_url`	68	???	A string - URL for the uploaded batch-mint json, required by indexers to fetch the tree for initialization.
`_metadata_hash`	???	???	A string representing a hex-encoded xxh3_128 hash of the uploaded batch-mint json-file.

# Conflicts: # programs/bubblegum/program/src/processor/create_tree_with_root.rs

# Conflicts: # programs/bubblegum/program/src/processor/create_tree_with_root.rs # programs/bubblegum/program/tests/rollup.rs # programs/bubblegum/program/tests/utils/tree.rs

# Conflicts: # clients/js/src/generated/instructions/prepareTree.ts # clients/rust/src/generated/instructions/finalize_tree_with_root.rs # idls/bubblegum.json # programs/bubblegum/program/src/processor/finalize_tree_with_root.rs # programs/bubblegum/program/tests/rollup.rs # programs/bubblegum/program/tests/utils/tree.rs

Add canopy size check

# Conflicts: # programs/bubblegum/program/tests/rollup.rs

The reason for adding prep_tree are still unknown, but keeping it for now. As the next steps I'd suggest to get rid of it to keep the styling consistent across bubblegum methods

next steps: - test are really shitty, we need to create test helpers to build this - when the SDK is ready - use it in tests

[MTG-545] feat: update client

…cleanup # Conflicts: # clients/rust/Cargo.lock # clients/rust/Cargo.toml # programs/bubblegum/Cargo.lock # programs/bubblegum/program/Cargo.toml # programs/bubblegum/program/src/processor/create_tree.rs

…nment

…cleanup

vercel · 2024-09-20T02:48:34Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
mpl-bubblegum-js-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 30, 2024 11:28am

…on is not available on mainnet yet

danenbm

Nice clean code change, looks good. Left a few comments. Still need to read through the tests though, will review those tomorrow morning.

danenbm · 2024-09-26T00:01:19Z

clients/js/src/generated/errors/mplBubblegum.ts

+codeToErrorMap.set(0x179c, StakingVoterMismatchError);
+nameToErrorMap.set('StakingVoterMismatch', StakingVoterMismatchError);
+
+/** FeeReceiverMismatch: Fee receiver mismatch */


Nit: It would be nice if some of these new Bubblegum errors were a bit more descriptive. For example, if this one said something more like who the fee receiver is mismatched with.

Like some of the spl-account compression ones have a bit of wording that really helps with the clarity, i.e.:
BatchNotInitializedError: Tree header was not initialized for batch processing
CanopyRootMismatchError: Canopy root does not match the root of the tree

danenbm · 2024-09-26T07:27:59Z

programs/bubblegum/program/src/state/mod.rs

@@ -12,6 +12,21 @@ pub const VOUCHER_PREFIX: &str = "voucher";
 pub const ASSET_PREFIX: &str = "asset";
 pub const COLLECTION_CPI_PREFIX: &str = "collection_cpi";

+pub const MAX_ACC_PROOFS_SIZE: u32 = 17;
+
+// TODO: set real keys before mainnet deploy


Ok I see this looks like it is fixed in #114

danenbm · 2024-09-26T07:50:19Z

programs/bubblegum/program/Cargo.toml

-version = "0.12.0"
+version = "0.13.0"


we actually normally don't version the program package now given that we don't publish it ever, so you could leave this at 12 unless there's another reason.

danenbm · 2024-09-26T07:52:45Z

programs/bubblegum/program/src/processor/add_canopy.rs

+    )]
+    pub tree_authority: Account<'info, TreeConfig>,
+    #[account(mut)]
+    /// CHECK:


Nit: no comment detail

danenbm · 2024-09-26T07:54:50Z

programs/bubblegum/program/src/processor/add_canopy.rs

+    // incoming_tree_delegate is the tree owner as set in prepare tree, it's required to do any modificaitons to the tree,
+    // including the canopy setup
+    require!(
+        incoming_tree_delegate == authority.tree_delegate,
+        BubblegumError::TreeAuthorityIncorrect,
+    );


why doesn't this allow authority.tree_creator in addition to authority.tree_delegate?

danenbm · 2024-09-26T09:19:11Z

clients/js/src/generated/instructions/index.ts

+export * from './addCanopy';
+export * from './appendCanopyNodes';
 export * from './burn';
 export * from './cancelRedeem';
 export * from './createTreeConfig';
 export * from './decompressV1';
 export * from './delegate';
+export * from './finalizeTreeWithRoot';
+export * from './finalizeTreeWithRootAndCollection';
+export * from './initPreparedTreeWithRoot';
 export * from './mintToCollectionV1';
 export * from './mintV1';
+export * from './prepareBatchMerkleTree';
+export * from './prepareTree';


I think these spl-account-compression instructions such as appendCanopyNodes do not need to be included in our JS and Rust clients. They are added by default and can be removed in kinobi. See kinobi.cjs where other spl-account-compression instructions such as append and closeEmptyTree are marked for deletion.

danenbm · 2024-09-26T09:24:02Z

clients/rust/Cargo.toml

-version = "1.4.0"
+version = "1.5.0"


Note: you don't need to update this version as cargo release does this as part of our client publish CI.

danenbm · 2024-09-26T09:25:24Z

clients/rust/Cargo.toml

-solana-program = "^1.14"
-thiserror = "^1.0"
+solana-program = "~1.18.11"
+thiserror = "1.0.63"


The Rust client is designed to not require regular Solana program dependency updates. Can it be left at ^1.14 in the client?

danenbm · 2024-09-26T09:29:22Z

configs/scripts/program/dump.sh

-    RPC="https://api.mainnet-beta.solana.com"
+    RPC="https://api.devnet.solana.com"


hmm I think in general we want to be testing with mainnet-beta programs. But I think spl-account-compression you must use the devnet version for this. We did this in mpl-core with the oracle-test-program on devnet, I think we just cloned the script. Can you do that here as well?

danenbm · 2024-09-26T09:31:56Z

programs/bubblegum/program/tests/simple.rs

@@ -276,6 +273,26 @@ async fn test_cannot_create_tree_needing_too_many_proofs_with_too_small_canopy()
 async fn test_cannot_create_tree_needing_too_many_proofs_with_no_canopy() {
    let context = BubblegumTestContext::new().await.unwrap();

+    let tree_create_result = context.create_tree_with_canopy::<19, 64>(1, true).await;


I think this should be called with 0 correct?

danenbm

A couple last comments. There's also some CI failures that need to be fixed. Overall code change looks great. As mentioned before, this is a clean code change. Good documentation and good testing as well.

danenbm · 2024-09-26T17:15:17Z

programs/bubblegum/README.md

@@ -72,6 +77,52 @@ Compressed NFTs support transferring ownership, delegating authority, and burnin

 Redeeming a cNFT removes the leaf from the Merkle tree and creates a voucher PDA account.  The voucher account can be sent to the `decompress_v1` instruction to decompress the cNFT into a Token Metadata NFT.  As mentioned above this will cost rent for the Metadata and Master Edition `token-metadata` accounts that are created during the decompression process.  Note that after a cNFT is redeemed but before it is decompressed, the process can be reversed using `cancel_redeem`.  This puts the cNFT back into the Merkle tree.

+## Batch-Mint Operations


Great README additions!

danenbm · 2024-09-26T21:46:12Z

programs/bubblegum/program/tests/batch-mint.rs

+    // TODO: good luck trying to make it work with those allignment requirements of the WrappedMining struct,
+    // let account_type:u8 = mplx_rewards::state::AccountType::Mining.into();
+    // mining_acc_data[0] = account_type;
+    // let mining_acc = mplx_rewards::state::WrappedMining::from_bytes_mut(&mut mining_acc_data)
+    //     .expect("Failed to create mining account");
+    // mining_acc.mining.owner = voter_authority;
+    // mining_acc.mining.stake_from_others = 0;
+    // so here is a hacky way to set the owner of the mining account directly
+    mining_acc_data[32..64].copy_from_slice(&voter_authority.to_bytes());


Nit: I think this should be resolved or cleaned up with however we want to solve it for the medium/long term, before going into main.

danenbm · 2024-09-27T23:47:13Z

programs/bubblegum/program/Cargo.toml

+mplx-staking-states = { git = "https://github.com/metaplex-foundation/aura-staking.git" }
+mplx-rewards = { git = "https://github.com/metaplex-foundation/aura-rewards.git", features = ["no-entrypoint"] }


I think these should be published now right?

* feat: use constants from external crate * feat: change after change in imported crate * chore: drop useless error and use into() for type convertion * chore: move keys cast out of runtime * chore: drop unused import

n00m4d and others added 30 commits May 16, 2024 17:29

feat: add create tree with root instruction

b25553a

feat: update solana versions

0339d56

feat: client update

403765d

feat: add new instr to instruction name enum

1e185ed

feat: SOL protocol fee for rollup operation

4bef5d4

test: update

67fd4a3

feat: add temp script to prepare env to run tests

883f43b

Merge branch 'feat/rollup' into feat/rollup-sol-fee

060bd11

# Conflicts: # programs/bubblegum/program/src/processor/create_tree_with_root.rs

tests

771d115

feat: change submodule for spl

06577c5

comments

9c9f43f

feat: add new instructions and cover it with tests

323edc9

Merge branch 'feat/rollup' into feat/rollup-sol-fee

b04e6a1

# Conflicts: # programs/bubblegum/program/src/processor/create_tree_with_root.rs # programs/bubblegum/program/tests/rollup.rs # programs/bubblegum/program/tests/utils/tree.rs

client: generate new sdk

88a955a

feat: change prep tree and finalize tree instructions a little bit

db4a868

merge + tests

250c363

feat: add canopy size check

1565158

rebase + git submodule update

7aaf57d

merge + tests

c464ba3

fmt

e039c04

Merge pull request #3 from n00m4d/feat/canopy-size-check

20478c6

Add canopy size check

Merge branch 'feat/rollup' into feat/rollup-sol-fee

c24672f

# Conflicts: # programs/bubblegum/program/tests/rollup.rs

merge

dfe9784

[MTG-234] calculate weighted stake

b2d39bb

prepare tree cleaned up

bd1b4b3

The reason for adding prep_tree are still unknown, but keeping it for now. As the next steps I'd suggest to get rid of it to keep the styling consistent across bubblegum methods

extending a comment and reusing the existing variable

f9ccebb

Merge branch 'feature/weighted-stake' into feature/cleanup

2c4b1ea

finalize tree with root finalized

08e8238

next steps: - test are really shitty, we need to create test helpers to build this - when the SDK is ready - use it in tests

client lib manual update

a3f2290

RequescoS and others added 8 commits August 29, 2024 14:36

thiserror update

5340099

thiserror update

702e5cd

Merge pull request #16 from adm-metaex/feat/update-client

2213f22

[MTG-545] feat: update client

Merge commit 'cdd8fc3cde9f9f30518bb265d8fbf4be117df128' into feature/…

0607f92

…cleanup # Conflicts: # clients/rust/Cargo.lock # clients/rust/Cargo.toml # programs/bubblegum/Cargo.lock # programs/bubblegum/program/Cargo.toml # programs/bubblegum/program/src/processor/create_tree.rs

using spl crate

b3c48f0

account compression idl update, deps update and parameter names allig…

aa01de9

…nment

fix formatting in toml

c187cba

Merge commit '42ffed35da6b2a673efacd63030a360eac3ae64e' into feature/…

dc50cf2

…cleanup

using devnet of solana to dump contracts as the spl account compressi…

1906a77

…on is not available on mainnet yet

vercel bot deployed to Preview September 24, 2024 16:34 View deployment

regenerated clients

d352e05

vercel bot deployed to Preview September 24, 2024 18:26 View deployment

StanChe requested review from danenbm and blockiosaurus September 25, 2024 09:32

bump rust client version to 1.5.0

6f9e36b

vercel bot deployed to Preview September 25, 2024 09:57 View deployment

using a public crate for spl-account-compression

d87ede8

vercel bot deployed to Preview September 25, 2024 10:00 View deployment

fixed the noop version and client lock file update

9a7ee42

vercel bot deployed to Preview September 25, 2024 10:02 View deployment

updated refs to staking and rewards contracts

cd5704e

vercel bot deployed to Preview September 25, 2024 10:07 View deployment

fix formating in cargo

c3010c1

vercel bot deployed to Preview September 25, 2024 10:31 View deployment

danenbm requested changes Sep 26, 2024

View reviewed changes

danenbm reviewed Sep 27, 2024

View reviewed changes

Use constants from external crate (#114)

07678e6

* feat: use constants from external crate * feat: change after change in imported crate * chore: drop useless error and use into() for type convertion * chore: move keys cast out of runtime * chore: drop unused import

vercel bot deployed to Preview September 30, 2024 11:28 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch minting #113

Batch minting #113

StanChe commented Sep 20, 2024

vercel bot commented Sep 20, 2024 •

edited

Loading

danenbm left a comment

danenbm Sep 26, 2024

danenbm Sep 26, 2024 •

edited

Loading

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm left a comment

danenbm Sep 26, 2024

danenbm Sep 26, 2024

danenbm Sep 27, 2024

		RPC="https://api.mainnet-beta.solana.com"
		RPC="https://api.devnet.solana.com"

		@@ -72,6 +77,52 @@ Compressed NFTs support transferring ownership, delegating authority, and burnin

		Redeeming a cNFT removes the leaf from the Merkle tree and creates a voucher PDA account. The voucher account can be sent to the `decompress_v1` instruction to decompress the cNFT into a Token Metadata NFT. As mentioned above this will cost rent for the Metadata and Master Edition `token-metadata` accounts that are created during the decompression process. Note that after a cNFT is redeemed but before it is decompressed, the process can be reversed using `cancel_redeem`. This puts the cNFT back into the Merkle tree.

		## Batch-Mint Operations

		mplx-staking-states = { git = "https://github.com/metaplex-foundation/aura-staking.git" }
		mplx-rewards = { git = "https://github.com/metaplex-foundation/aura-rewards.git", features = ["no-entrypoint"] }

Batch minting #113

Are you sure you want to change the base?

Batch minting #113

Conversation

StanChe commented Sep 20, 2024

Batch operations

Batch-Mint Operations

Introduction

How It Works

Steps to Perform a Batch-Mint

📄 prepare_tree

📄 add_canopy

📄 finalize_tree_with_root

📄 finalize_tree_with_root_and_collection

vercel bot commented Sep 20, 2024 • edited Loading

danenbm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danenbm Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danenbm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

📄 `prepare_tree`

📄 `add_canopy`

📄 `finalize_tree_with_root`

📄 `finalize_tree_with_root_and_collection`

vercel bot commented Sep 20, 2024 •

edited

Loading

danenbm Sep 26, 2024 •

edited

Loading