Mainnet Stable Release 1.2.0

[lk86]

As many of you are already aware, we have been working to stablize version 1.2.0 for quite some time. he O(1) Labs Team is proud to announce that 1.2.0 Stable is ready for use in production on Mainnet! 1.2.0 contains a variety of Quality of Life Improvements for node operators, including some new tools, graphql queries, cli commands, metrics, not to mention the countless efforts that went on behind the scenes to keep Mainnet and our team at O(1) Labs running smoothly.

In addition, 1.2.0 includes the client for the proposed update to the Mina Foundation Delegation Program (some further server work is required before the updated program launches, stay tuned for more announcements from the Mina Foundation on this topic). You can test the client implementation and configuration yourself as described below in the New Features section.

New Features:

GraphQL:

• Match GraphQL sync status with the daemonStatus GraphQL, use sync status from status #8255
• Allows the pooledUserCommands query to select transactions by hash or ID, to make it more consistent with transactionStatus Accept transactions by ID in pooledUserCommands #8293
• Adds new GraphQL endpoint stagedLedgerProofEmitted to indicate that a new staged ledger proof was formed Feature: Add stagedLedgerProofEmitted to GraphQL #8658

CLI Commands:

• Improved handling of locally-generated transactions to allow re-sending transactions and give the user more control over rebroadcasting transactions. No more already saw this errors! Allow local re-sending of txns, reset re-broadcast timer when doing so #8973
• mina advanced vrf is a set of new advanced commands for generating and evaluating VRF witnesses to prove that your key has won a slot Vrf evaluation commands #8814
• mina advanced hash-transaction is a new command for hashing transactions Add hash-transaction advanced command #8294
• mina ledger export snarked-ledger is a new command for exporting snarked ledgers with proven balances Export snarked ledger #8844
• Show the coinbase-receiver in mina client status if it is set, and adds a new command mina advanced set-coinbase-reciever to change the coinbase receiver at runtime Add coinbase receiver to status, allow it to be changed by an advanced command #8127
• mina advanced chain-id-inputs is a new command for displaying the inputs to the ChainID Print inputs to chain id #8264
• mina advanced send-rosetta-transaction now immediately sends the transaction instead of waiting for the command to exit Dispatch transactions as they come in to send-rosetta-transaction #8186
• mina accounts commands now support retrieving/uploading keys and account information over a remote graphQL interface instead of just local files Support local filesystem and rest-server modes for accounts import, accounts list #7282
• Return exit code 1 when mina ledger export commands fail Exit with code 1 if there is an error in ledger export #8720
• New flag --minimum-block-reward to mina daemon that allows a block producer to set the minimum net reward from producing a block. minimum block reward #8605
  • Addresses concerns from Prevent large snark works sneaking in on coinbases #8576 and block producers who have overpaid for snark work. If paying the fees for a given snark would drop the resulting profit below the given value, an empty block with no transactions is produced instead.

Other:

• More coda->mina changes Docstrings, Coda to Mina #8238
• Allow passing peers-list-url parameter as part of the daemon.json or other config file Allow passing a peer-list-url in the runtime config #8295
• Quieter logging and improved messaging during daemon startup [TRIVIAL] Don't print a large error in the pretty logs for missing config files #8261 [Trivial] Improve error message from missing --generate-genesis-proof #8203 Fix misspelling of cache_exceptions #8146 Filter out invalid peers from peer files #8260 more metadata in snark-pool error logs #8488 Use formatted exception printing for prover/verifier sub-monitors #8503
• Improved Mina Foundation Uptime Tracking! Allow uptime submitter keyfile #9499
  • This feature no longer requires importing your keypair, now supports a new flag --uptime-submitter-key which takes the path to your private key, just like --block-producer-key.
  • To test the new system, pass in the following information to the daemon:
    • The path to the private key with the flag --uptime-submitter-key </path/to/private/key>
    • The URL of our testing backend server with the flag --uptime-url http://34.134.227.208/v1/submit
    • The password for the keypair associated to the given public key with the environment variable UPTIME_PRIVKEY_PASS=My_V3ry_S3cure_Password. If you are using a .mina-env file on Debian then this value should be on its own line, not included in EXTRA_FLAGS=.
• New environment variables in .mina-env for use with the mina systemd service Lk86/fix mina service wantedby #9375 lk86/updated-mina-systemd-proposal #9533
  • Passes in the --peer-list-url automatically to correspond to the mina package (mina-mainnet for mainnet peers list, and mina-devnet for devnet peers).
  • This value can still be overridden with PEERS_LIST_URL variable in .mina-env
  • Block production is disabled by default, use EXTRA_FLAGS as shown below to configure your block producer keys
  • LOG_LEVEL variable to set the daemon log level, with the default value Info
  • FILE_LOG_LEVEL variable to set the daemong file log level, with the default value Debug
  • Remove --generate-genesis-proof true as it no longer does anything in this mina version
  • Changed WantedBy to default.target instead of multi-user.target as the latter does not work in user services (as many users have reported)

EXTRA_FLAGS="--block-producer-key /home/mina/my-wallet --uptime-submitter-key /home/mina/my-wallet --uptime-url http://34.134.227.208/v1/submit"
UPTIME_PRIVKEY_PASS=My_V3ry_S3cure_Password
MINA_PRIVKEY_PASS=My_V3ry_S3cure_Password
CODA_PRIVKEY_PASS=My_V3ry_S3cure_Password
LOG_LEVEL=Info
FILE_LOG_LEVEL=Debug

Note: MINA_PRIVKEY_PASS will not be supported until the 1.3.0+ Mina Daemon releases but it won't hurt to start providing it now.

Stability Improvements and Bug Fixes:

• Guard against underflow when calculating balances via GraphQL Guard against underflow when computing liquid balance #8096
• More thorough fix for nodes getting stuck at old block heights [master-qa] Add much more aggressive handling of waits, with logging upon errors #8352 Move code to wait and log errors to library code #8358 [master-qa] Add aggressive exception handling around precomputed block uploads #8419 Use a try_with to catch any exceptions from verifier's Rpc_parallel worker #8506
• Don't add "extra" snark work even if there is space and budget discard extra snark work even if there is space and budget #8098
• Check that keys are valid before adding transaction to the pool [HOTFIX] Check public keys are valid before placing transactions in the pool #8442
• Fix edge case where transactions were dropped from the mempool [Compatible] fix handling committed transactions #8473
• Check the validity of keys when they are provided as arguments to the CLI check cli public keys #8447
• Catch errors when the chain runs out of available accounts in the ledger Catch errors when the ledger is full #7258
• More verbose logging around long async cycles to help debug their source Log monitor chain on long async cycles #8809
• Rebroadcast snark work until it is included in a block Rebroadcast snark work until it is included in a block #8793
• Prevent crashes caused by short forks off of the root of the transition frontier devnet catchup failure fix #8952
• Rebroadcast transactions in chunks to better support reliable transaction broadcasts Rebroadcast in chunks #8965
• Make calls to the verifier async to avoid lockups and bottlenecks in verifying transactions Make verifier async #8961
• Improves logic for communicating between rust and ocaml processes to prevent some memory corruption and segfaults Upgrade ocaml-rs, tweak some bindings to be safer #8998
• Limit snark work broadcasts to a safe rate below the libp2p rate limit to avoid getting throttled by peers limit local snark work broadcast #9011
• Revert the changes to staged ledger diff application [hotfix] Revert staged ledger diff changes #9068
• Fix super-catchup cache invalidation to allow re-downloading failed state hashes Fix/super catchup stuck at the tip #9132
• Fix remaining WNOHANG crashes caused by child process management Wrap a couple of Process.run #9171
• Prevent mina ledger export-ledger staged-ledger from blocking RPC heartbeats from being processed get_ledger in sys thread for release #9168
• Fix file descriptor leak by ensuring stdin is closed for all child processes after they have exited Close stdin for some processes #9178
• Mitigation for the offline issue (crash when node has been running for >34 hours) Crash if offline for 30min #9301 stop long running daemon #9387 add --stop-time flag to configure this behavior update stoppage time #9506
• Hardening for the transaction mempool and mitigations to prevent broadcasting bad transactions. See our Retrospective for more information about our approach with these fixes and why they were important.

Improved Packaging and CI:

• Use the mina-mainnet package in docker release/mainnet-v1.1.1 Fix dockerfiles for mainnet package #8239
• Terraform for mainnet and devnet2 Mainnet Infra #8225 Devnet -> Devnet2 to fix errors with manually deleted namespace #8179 Update patch archive node script #8250 update mainnet seeds #8248
• Attempted fix of clientSDK publish CI job Added NPM token env variable to running buildkite container #8325
• Optional Genesis Proof, Prover/Verifier Keys, and other network-specific artifacts at build-time and at startup Make compile-time genesis proof creation optional #8497 Stop using hard-coded compile-time verification keys #8584 Stop generating/downloading snark keys during build #8672 Don't generate genesis proof at startup #8764 [Release] Don't look for genesis proof file at startup #8947
• New tool for managing generated/downloaded keys, snark_keys/gen_keys.exe, which takes in a config-file and downloads or generates the appropriate artifacts Add support for choosing constraint constants in gen_keys, remove as dep #8554
• Refactor docker entrypoint script to be more flexible and allow for arbitrary scripts to execute in /entrypoints.d/ Refactor Mina dockerfiles entrypoint setup #8263
• Enable Alpha Builds enable-alpha-builds  #8876
  • coda->mina across CI/CD, scripts, and docker/debian tags
  • consolidate mainnet, devnet, and archive-node builds into a single job
  • migrate Ubuntu 18.04 containers to debian:stretch-slim for lightness
  • lay groundwork for Ubuntu 20.04 and Debian Buster support
  • remove legacy code including circle-ci
  • adds dockerfiles/Dockerfile-deb-builder which can build complete .deb packages for all mina packages
  • remove debugging symbols from dune builds
  • remove unused dependencies from debian packages
• Dockerfile now uses UserID 50000 instead of 192920 lk86/use-an-even-safer-user-id  #9435 lk86/fix-docker-permissions Use a smaller UserID for the mina user in docker #9326
• Debian stretch archive image no longer requires man to use postgres package lk86/fix-archive-docker-image Fix archive node docker installation fo… #9410
• Fix $KEY_NAME quoting Lk86/fix systemd quoting #9446

Improvements to ancillary tools (rosetta, archive-node, sidecar, etc.)

Rosetta:

• Add minimum transaction fee to Rosetta Add minimum tx fee to rosetta construction/metadata #8170
• New functions in the ClientSDK for supporting Rosetta-formatted keypairs Client SDK, add Rosetta signed txn to signed cmd method #8324
• Add locked_ and total_ balance metadata to Rosetta account and balance responses Add {locked,total}_balance to Rosetta account/balance response #8141
• Add support for network=mainnet to Rosetta Makes network=mainnet work properly in Rosetta #8877

Libraries:

• New version of ppx_version Bump ppx_version #8435 Update ppx_version #8734
• New version of snarky snarky submodule updated #8908
• Additional utility library for managing Pickles_types Add Pickles_types.Higher_kinded_poly #8075

Archive-node and related tools:

• Canonical representation for "extensional blocks" for backing up and restoring archive node data Canonicalize extensional blocks #8312
• mina-extract-blocks tool for restoring archive node data from an existing archive node missing_subchain reborn as extract_blocks #8343
• mina-replayer improvements Replayer, test db from mainnet #8329 Update epoch ledger handling in replayer #8717 Replayer, don't create epoch ledger if exists #8739 Replayer and delegation compliance, undo bad optimization #8751
• Tools for patching bad balances from before the 1.1.5 archive node release App to swap balances for combined fee transfers #8583 Scripts for patching swapped balances #8611 improve-bad-balance-scripting Add fix-all-balances.sh script #8732 improve-bad-balance-scripting Add conditional for the ----- lines #8815 See complete instructions for a walkthrough on how to patch your archive.

Delegation Compliance App:

• New Delegation Compliance App for calculating payments back to the foundation from the top 120 block producers Delegation compliance app #8735 Audit trail, remove duplicate data in delegation_compliance #8760 Delegation compliance, simplified #8792 CSV file output for compliance #8873 Delegation compliance, count all blocks in epoch #8884 Delegation compliance, CSV stitching #8923

Documentation:

• Feature/8187 spec blockchain proof verification #8374 Blockchain proof verification
• Docs for child processes #8486 Child Processes
• Some docs for bin_io #8428 bin_io Serialization Format
• [README] fix link to contributing doc #8670 Fix link to the CONTRIBUTING doc
• Slightly better macOS dev setup experience #8678 Improved README-dev.md

Monitoring, Testing, Alerting:

And 62 separate PRs to improve our monitoring, testing, and alerting infrastructure:
#8077 #8050 #8099 #8241 #8249 #8214 #8216 #8182 #8259 #8300 #8313 #8302 #8328 #8342 #8360 #7842 #8258 #8240 #8237 #8332 #8414 #8427 #8361 #8341 #8458 #8233 #8455 #8530 #8596 #8606 #8535 #8637 #8631 #8645 #8622 #8582 #8691 #8676 #8689 #8640 #8664 #8707 #8688 #8683 #8768 #8752 #8740 #8783 #7884 #8758 #8810 #8861 #8871 #8869 #8850 #8909 #8926 #9222 #9390 #9436 #9408 #9377

Complete Changelog:

Linked below is the full git history between 1.1.5 and 1.1.6alpha2. Split into 4 chunks as github will only display 250 commits at a time:

• February 22-March 4: 1.1.5...16d41f4
• March 4-April 6: 16d41f4...0e14abf
• April 6-May 3: 0e14abf...19a5324
• May 3-June 2: 19a5324...1.1.6alpha4
• June 2-June 28: 1.1.6alpha4...1.2.0beta1

It may also be useful to read the changes from each alpha and beta release individually to understand how we got to this result:
1.1.6alpha4
1.1.6alpha5
1.2.0alpha1
1.2.0beta1
1.2.0beta2
1.2.0beta4
1.2.0beta5
1.2.0beta8

Upgrading & Connecting

Debian Packages:

From now on builds will be packaged and released to the debian repo based on debian release codenames (stretch for Debian 9/Ubuntu 18.04 and soon buster for Debian 10 / Ubuntu 20.04), and a release channel (alpha, beta, stable, unstable).

Repository Setup:

To allow for installation of mina stretch stable builds:

echo "deb [trusted=yes] http://packages.o1test.net stretch stable" | sudo tee /etc/apt/sources.list.d/mina.list
sudo apt-get update

Installation:

Mina Daemon:
sudo apt-get install -y mina-mainnet=1.2.0-fe51f1e
Mina Archive:
sudo apt-get install -y mina-archive=1.2.0-fe51f1e
Mina Generate Keypair:
sudo apt-get install -y mina-generate-keypair=1.2.0-fe51f1e
Archive Node Schema for initializing postgres:
create_schema.sql

Docker Images:

Mina Daemon:
minaprotocol/mina-daemon:1.2.0-fe51f1e-mainnet
Mina Archive:
minaprotocol/mina-archive:1.2.0-fe51f1e
Mina Generate Keypair:
minaprotocol/mina-generate-keypair:1.2.0-fe51f1e
Archive Node Schema for initializing postgres:
create_schema.sql

Sandbox Node: For testing in an isolated, single-node network without snarks

As of 1.2.0, the demo functionality is built into the standard daemon container, via the environment variable RUN_DEMO=true.
Just docker run -it --name mina-demo -e RUN_DEMO=true minaprotocol/mina-daemon:1.2.0-fe51f1e-devnet and go!

NEW Devnet:

This release is the first to support the newly relaunched Devnet. To use any of the above packages or docker images with devnet instead of mainnet, you can simply replace mainnet with devnet in either the name of the package or the docker image tag.

Block Producer Stats Sidecar:

The block producer sidecar will be replaced with the foundation delegation uptime tracker described above, once the backend verification system is ready. In the meantime any version of the sidecar since ~1.1.6 should work the same way, and the data produced by the sidecar still counts towards foundation delegation elegibility. Stay tuned on Discord for further announcements from Mina Foundation in regards to the Delegation Program.

Questions and Feedback

If you discover any issues or regressions from 1.1.8, please report them so we can look into it. Please include the mina version output or 1.2.0 in the issue title/description so we can more easily sort through the issues.
For questions and feedback, please go to the #mainnet-block-producers channel on our Discord or comment on the Github Discussion for this release, we're excited to hear about your experience with this new build!

Step by Step Guide:

Check out our documentation for complete instructions on using this version to connect to Mainnet.

If you are running the correct version on the correct network, mina client status will show:

Chain id:     5f704cc0c82e0ed70e873f0893d7e06f148524e3f0bdae2afb02e7819a0c24d1
Git SHA-1:    fe51f1ef5cfbcb4d5f559af39080963cf28ff1af

---

This discussion was created from the release Mainnet Stable Release 1.2.0.