2.2.1

CODE_COLOR: CODE_RED_MAINNET
RELEASE_VERSION: 2.2.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: TRUE

This release patches a bug found in the 2.2.0 release

Non-protocol changes

There was a bug in the integration between ethereum implicit accounts and the compiled contract cache which sometimes caused the nodes to get stuck. This would most often happen during state sync, but could also happen by itself. Please update your nodes to avoid getting stuck.

A node that hits this bug will print an error about an InvalidStateRoot in the logs and then it'll be unable to sync.
It's possible to recover a stalled node by clearing the compiled contract cache and rolling back one block:

1. Stop the neard process
2. Download the new version of neard
3. Clear the compiled contract cache: rm -rf ~/.near/data/contracts
4. Undo the last block: ./neard undo-block
5. Start neard

After that the node should be able to recover and sync with the rest of the network.

Protocol upgrade voting

Voting for protocol 71 has been postponed to 2024-09-23 10:00 UTC. Validators were asked to roll back to 2.1.1, so protocol upgrade proposed in the 2.2.0 release won't go through. 2.2.1 changes the voting date to 2024-09-23 to give everyone time to update to 2.2.1 before the upgrade to 71 takes place.

2.2.1-rc.1

CODE_COLOR: CODE_RED_TESTNET
RELEASE_VERSION: 2.2.1-rc.1
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: TRUE

This release patches a bug found in the 2.2.0 release

Non-protocol changes

There was a bug in the integration between ethereum implicit accounts and the compiled contract cache which sometimes caused the nodes to get stuck. This would most often happen during state sync, but could also happen by itself. Please update your nodes to avoid getting stuck.

A node that hits this bug will print an error about an InvalidStateRoot in the logs and then it'll be unable to sync.
It's possible to recover a stalled node by clearing the compiled contract cache and rolling back one block:

1. Stop the neard process
2. Download the new version of neard
3. Clear the compiled contract cache: rm -rf ~/.near/data/contracts
4. Undo the last block: ./neard undo-block
5. Start neard

After that the node should be able to recover and sync with the rest of the network.

2.2.0

CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 2.2.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Protocol Changes

• The minimum validator stake has been set to a lower value. The small-stake validators that were kicked out during the shift to stateless validation will be able to rejoin the network.
• Better algorithm for validator kickouts

Non-protocol Changes

• Fix spammy messages about calculating gas for PromiseYield receipts.
• Don't crash when the CPU doesn't have SHA-NI instructions. It's still a hardware requirement, there is no guarantee that nodes without this instruction will be able to keep up with the network, but neard will now be able to run (slowly) on CPUs without this instruction.

Protocol upgrade voting

Voting for protocol version 71 will start on Monday 2024-09-16 19:00 UTC

2.2.0-rc.2

CODE_COLOR: CODE_RED_TESTNET
RELEASE_VERSION: 2.2.0-rc.2
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Changes

This is an emergency testnet release.
In 2.2.0-rc.1 the protocol upgrade was supposed to update the eth wallet contract on testnet to a new version,
but there was a bug which caused some of the nodes to stay with the old version of the contract. Because of that
some of the nodes had one version and some had the other and the consensus diverged.
Some nodes didn't accept the chunks and they got stuck.

This release fixes the bug - all nodes will continue using the old eth-wallet contract, as this is what the
canonical chain does.

Nodes that got stuck need manual intervention. The steps to recover a stalled node are:

1. Stop the neard process
2. Download the new version of neard
3. Clear the compiled contract cache: rm -rf ~/.near/data/contracts
4. Undo the last block: ./neard undo-block
5. Start neard

After that the node should be able to sync with the rest of the network.

2.2.0-rc.1

CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 2.2.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Protocol Changes

• The minimum validator stake has been set to a lower value. The small-stake validators that were kicked out during the shift to stateless validation will be able to rejoin the network.
• Better algorithm for validator kickouts
• (Testnet only) update the eth-implicit accounts contract on testnet to match the one on mainnet.

Non-protocol Changes

• Fix spammy messages about calculating gas for PromiseYield receipts.
• Don't crash when the CPU doesn't have SHA-NI instructions. It's still a hardware requirement, there is no guarantee that nodes without this instruction will be able to keep up with the network, but neard will now be able to run (slowly) on CPUs without this instruction.

Protocol upgrade voting

Voting for protocol version 71 will start on Tuesday 2024-09-03 10:00 UTC

2.1.1

CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 2.1.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Protocol Changes

The protocol upgrade introduces two features:

• Eth-Implicit Accounts NEP-0518
• Host Functions for BLS12-381 Curve Operations NEP-0488

Non-protocol changes

#11711 enforces the SHA-NI extensions. Without this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Protocol Upgrade Voting

Voting for upgrading to protocol version 70 will start on Monday 2024-08-26 10:00:00 AM UTC.

2.1.0-rc.3

CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 2.1.0-rc.3
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes

This release addresses the graceful shutdown of the awc::Client.

Non-protocol changes

#11711 enforces the SHA-NI extentions. Whithout this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Full Changelog: 2.1.0-rc.2...2.1.0-rc.3

2.0.0

CODE_COLOR: CODE_YELLOW_MAINNET
RELEASE_VERSION: 2.0.0
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Release note revision history

August 12th

The release note was revised on August 12th to update the "Failover node" section.
If you haven't reviewed it yet, please revisit the changes mentioned below for the latest updates.

• Failover node
  • Old failover procedure (Option 1)
    • Removed the "copy node_key.json" step.
  • Validator key hot swap (Option 2)
    • Added "Required Changes After Protocol Version Upgrade to 69".
    • Before protocol version upgrade to 69, the backup node's config.json should just match the primary node's config.json.

August 9th

The release note was revised on August 9th to include additional information. If you haven't reviewed it yet, please revisit the sections mentioned below for the latest updates.

• Config changes - Important notes
  • What recommended changes to config.json imply
  • Notes on memtries
  • State sync config
• Failover node

Protocol Upgrade Voting

This release contains two protocol upgrades. The first protocol upgrade introduces the congestion control feature, and the second upgrade introduces a set of protocol changes for stateless validation. There are therefore two relevant protocol version voting dates. This doesn't require any action on the part of node operators, as your node will follow the current protocol version's rules as long as you upgrade. Validator nodes will vote first for protocol version 68, and then for version 69 after 68 is adopted.

Voting for upgrading to protocol version 68 will start on Monday 2024-08-12 14:00:00 UTC and voting for upgrading to protocol version 69 will start on Wednesday 2024-08-14 04:00:00 UTC

Congestion Control

Relevant NEP: near/NEPs#539
RPC API change: #11419

Congestion control introduces a feature that's meant to limit the transactions and receipts included in chunks when a shard is under load. Before this protocol change, when a particular shard is under heavy load, users can often experience frustrating experiences where transactions take a long time to be included because there is a very long queue of delayed receipts that take up chunk space. When this delayed receipt queue grows to an unreasonable size, it can be difficult for users to understand when things will settle down and their transactions will begin to be included again.

The congestion control feature adds fields to the chunk header that indicate the load on the corresponding shard in terms of delayed and outgoing receipt gas, and the protocol rules limit transactions and receipts destined for congested shards, mitigating the problem of delayed receipt queues growing unbounded to unreasonable levels.

Things to note

Transactions can be rejected with a “ShardCongested” error if the chain is congested heavily

Stateless Validation

Relevant NEP: near/NEPs#509

This protocol upgrade is the main reason for designating this release as version 2.0.0. It will alter the roles of validators within the network and fundamentally change the process of validating state transitions. Currently when a node (validator or not) receives a new block and downloads all its chunks, it applies the transactions and receipts contained in those chunks by looking up and modifying values in the state of each shard. This means nodes need to keep the full state of all shards on disk. The stateless validation protocol upgrade will introduce a change that allows nodes to check the validity of chunks without needing to keep a local copy of the full state. There will be two roles after the upgrade: chunk producers and chunk validators. Chunk producers hold the state of the shard they are assigned to in memory and produce chunks for that shard. Chunk validators do not maintain any state locally and rotate on every single block to verify the state transition of a shard. A specific node may function both as a chunk producer and a chunk validator in the network. Whether a node is a chunk producer is based on stake: top 100 nodes by stake are chunk producers.

Changes to validator roles

Here is a high level summary of changes to validator roles

TL;DR;

• All validators
  • Serve as chunk validators
  • Additional network usage is expected
  • Less disk usage is expected
• Top 100 validators by stake:
  • In addition to chunk validator role, serve as chunk/block producers and block validators
  • Do not have to track all shards
  • State of tracked shard is loaded into memory
• All other validators
  • Don’t need to track any shard

Overview of different validator roles

• [NO CHANGES] Block producers (top 100 validators):
  • (Same as today) Produce blocks, (new) including waiting for chunk endorsements
  • (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding)
  • (Same as today) Should have a high barrier of entry for security reasons, to make block double signing harder.
  • (New) No longer require tracking any shard
• Chunk producers (top 100 validators):
  • (Same as today) Produce chunks
  • (Same as today) Must track the shard it produces the chunk for
  • (New) Produces and distributes state witnesses to chunk validators
• [New] Chunk validators (all validators):
  • Validate state witnesses and sends chunk endorsements to block producers
  • Do not require tracking any shard
  • Must collectively have a majority of all the validator stake, to ensure the security of chunk validation.

Reward calculation

Reward calculation will remain the same and be based on online ratio.

Changes to contract requirements

• New size limits were introduced to limit the size of state witness:
  • Transaction size must not exceed 1.5MB (used to be 4MB).
  • Receipt size must not exceed 4MB (possibly it may be reduced further down to 1.5MB).
  • A receipt is not allowed to read more than 4MB of state during a single function call.
  • Total size of transactions included in two consecutive chunks is limited to 4MB
  • Cross shard bandwidth is more limited than before
• Increased cost of sending receipts
  • Sending a receipt to a shard (cross contract call) will now cost 50 TGas / MiB. Previously the cost varied between 2-18 TGas / MiB

Config changes

This section outlines the configuration changes that node operators must apply to their nodes.

Important notes (PLEASE READ BEFORE PROCEEDING)

What recommended changes to config.json imply

Please note that the existing behavior of the tracked_shards config option referred to below is a bit counter-intuitive. For historical reasons, there are only two meaningful kinds of values, empty and nonempty lists. Any non-empty list (e.g. [0] or [0,1,2,3]) will have the node track all shards. An empty list will indicate that we don't need to track all shards, which will be the case for validators after the network has reached protocol version 69. The store.load_mem_tries_for_tracked_shards config option referred to below is a boolean that indicates whether the full state should be loaded into memory. This should only be necessary for validators. RPC node operators should keep this option set to false. If you set this to true for an RPC node, it may fail to respond to certain RPC requests.

Notes on memtries

As noted in the previous section, there may be issues if you set store.load_mem_tries_for_tracked_shards to true for RPC nodes. Until these issues are fixed, RPC node operators should refrain from setting this config option, and validators should refrain from using their validator nodes as RPC nodes. Please note that if you set store.load_mem_tries_for_tracked_shards=true on your RPC node, it may reach an inconsistent state that will cause issues even after reverting back to store.load_mem_tries_for_tracked_shards=false.

State sync config

In a previous release, we discouraged enabling state sync, as there were issues with state sync being enabled during a resharding. For this release, those issues are no longer present, and validators should re-enable state sync by setting state_sync_enabled=true in the config, as well as setting the state_sync config option to an appropriate value. The value of the state_sync key in the current default config at https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/config.json is the following:

{
  "sync": {
    "ExternalStorage": {
      "location": {
        "GCS": {
          "bucket": "state-parts"
        }
      },
      "num_concurrent_requests": 25,
      "num_concurrent_requests_during_catchup": 5
    }
  }
}

Estimated protocol upgrade timeline

• Current Protocol version: 67
• Voting on protocol version 68 upgrade: Monday, August 12, 2024, at 14:00 UTC
• Protocol version 68 upgrade: approximately Tuesday, August 13, 2024, at 17:00 UTC (2 epochs after voting)
• Voting on protocol version 69 upgrade: Wednesday, August 14, 2024, at 04:00 UTC
• Protocol version 69 upgrade: approximately Thursday, August 15, 2024, at 07:00 UTC (2 epochs after voting)

About transition period:
The transition period is the time between the initial voting on Monday, August 12, 2024, at 14:00 UTC and the upgrade to Protocol Version 69, which occurs 2 epochs after the second voting. Each epoch is approximately 13.5 hours, but may vary slightly. We anticipate the upgrade will happen around Thursday, August 15, 2024, at 07:00 UTC. To verify if the transition is complete, simply check the protocol version; it should read 69.

When to make configuration change

• After upgrading to neard binary 2.0.0: Make your co...

2.1.0-rc.2

CODE_COLOR: CODE_GREEN_TESTNET
RELEASE_VERSION: 2.1.0-rc.2
PROTOCOL_UPGRADE: FALSE
DATABASE_UPGRADE: FALSE
SECURITY_UPGRADE: FALSE

Fixes

This release addresses the RPC endpoint issue where all handler errors are misclassified as 500.

Non-protocol changes

#11711 enforces the SHA-NI extentions. Whithout this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Full Changelog: 2.1.0-rc.1...2.1.0-rc.2

2.1.0-rc.1

CODE_COLOR: CODE_YELLOW_TESTNET
RELEASE_VERSION: 2.1.0-rc.1
PROTOCOL_UPGRADE: TRUE
DATABASE_UPGRADE: TRUE
SECURITY_UPGRADE: FALSE

Protocol Changes

The protocol upgrade introduces two features:

• Eth-Implicit Accounts NEP-0518
• Host Functions for BLS12-381 Curve Operations NEP-0488

Non-protocol changes

#11711 enforces the SHA-NI extentions. Whithout this, nodes are more likely to fall out of sync.
Make sure that the CPU has support for SHA-NI : lscpu | grep sha.

Protocol Upgrade Voting

Voting for upgrading to protocol version 70 will start on Tuesday 2024-08-06 10:00:00 UTC.