Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Reduce network bandwidth, improve parablock times: optimize approval-distribution #5164

Merged
merged 80 commits into from
Apr 19, 2022
Merged
Show file tree
Hide file tree
Changes from 74 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
5315731
gossip-support: be explicit about dimensions
rphmeier Mar 21, 2022
e56a1d2
some guide updates
rphmeier Mar 21, 2022
a7f1bc0
update network-bridge to distinguish x and y dimensions
rphmeier Mar 21, 2022
8c6b4a5
get everything to compile
rphmeier Mar 21, 2022
38e998b
beginnings
rphmeier Mar 21, 2022
92925da
some TODOs
rphmeier Mar 22, 2022
d888b4b
polkadot runtime: use relevant_authorities
rphmeier Mar 23, 2022
167468c
make gossip topologies per-session
rphmeier Mar 23, 2022
221ffb0
better formatting
rphmeier Mar 23, 2022
66e79bf
gossip support: use current session validators
rphmeier Mar 23, 2022
dd8a300
expand in comment
rphmeier Mar 23, 2022
cfa3532
adjust tests and fix index bug
rphmeier Mar 23, 2022
c934eef
add past/present/future connection test and clean up code
rphmeier Mar 23, 2022
6531322
fmt
rphmeier Mar 23, 2022
a3214b9
network bridge: updated types
rphmeier Mar 23, 2022
2b5c787
update protocols to new gossip topology message
rphmeier Mar 23, 2022
8a4d0ac
guide updates
rphmeier Mar 23, 2022
6171803
Merge branch 'master' into rh-approval-grid
rphmeier Mar 23, 2022
3d32758
add session to BlockApprovalMeta
rphmeier Mar 23, 2022
a645fb3
add session to block info
rphmeier Mar 23, 2022
1ec4957
refactor knowledge and remove most unify logic
rphmeier Mar 23, 2022
7d790cc
start replacing gossip_peers with new SessionTopologies
rphmeier Mar 23, 2022
6d8717a
add routing information to message state
rphmeier Mar 23, 2022
7da64de
add some utilities to SessionTopology
rphmeier Mar 24, 2022
0a3ffa7
implement new gossip topology logic
rphmeier Mar 24, 2022
476c0b0
re-implement unify_with_peer
rphmeier Mar 24, 2022
6263ab1
distribute assignments according to topology
rphmeier Mar 24, 2022
2e7d047
finish grid topology implementation
rphmeier Mar 24, 2022
8e01613
refactor network bridge slightly
rphmeier Mar 25, 2022
d37f2f2
issue connection requests on all past/present/future
rphmeier Mar 25, 2022
1c39537
fmt
rphmeier Mar 25, 2022
3f5b6c3
address grumbles
rphmeier Mar 25, 2022
481f72c
tighten invariants in unify_with_peer
rphmeier Mar 25, 2022
fc43f43
implement random propagation
rphmeier Mar 25, 2022
023ed7a
refactor: extract required routing adjustment logic
rphmeier Mar 25, 2022
38f1e82
some block-age logic
rphmeier Mar 25, 2022
d216a2c
aggressively propagate messages when finality is slow
rphmeier Mar 25, 2022
a4b47f1
Merge branch 'master' into rh-approval-grid
rphmeier Mar 25, 2022
831192b
overhaul aggression system to have 3 levels
rphmeier Mar 25, 2022
93b5132
add aggression metrics
rphmeier Mar 25, 2022
85a568f
Merge branch 'master' into rh-approval-grid
rphmeier Mar 29, 2022
ba86d16
remove aggression L3
rphmeier Mar 30, 2022
f4448ba
reduce random circulation
rphmeier Mar 30, 2022
e0bef50
remove PeerData
rphmeier Mar 30, 2022
bf33489
get approval tests compiling
rphmeier Mar 30, 2022
3a3a4cc
use btree_map in known_by to make deterministic
rphmeier Mar 30, 2022
a56e5ef
Revert "use btree_map in known_by to make deterministic"
rphmeier Mar 30, 2022
3ef1002
test XY grid propagation
rphmeier Mar 30, 2022
4387333
remove stray println
rphmeier Mar 30, 2022
fc92560
test unshared dimension propagation
rphmeier Mar 30, 2022
852205a
add random gossip check
rphmeier Mar 30, 2022
b88cd62
test unify_with_peer better
rphmeier Mar 30, 2022
f793a67
test sending after getting gossip topology
rphmeier Mar 30, 2022
99dfb11
test L1 aggression on originator
rphmeier Mar 30, 2022
1853485
test L1 aggression for non-originators
rphmeier Mar 30, 2022
4b9059b
test non-originator aggression L2
rphmeier Mar 30, 2022
481694c
fnt
rphmeier Mar 30, 2022
32a0d1e
~spellcheck
rphmeier Mar 30, 2022
031a17a
fix statement-distribution tests
rphmeier Mar 30, 2022
c168e31
fix flaky test
rphmeier Mar 31, 2022
945d5a5
Merge branch 'master' into rh-approval-grid
rphmeier Mar 31, 2022
dac4eef
fix metrics typo
rphmeier Mar 31, 2022
8edad52
re-send periodically
rphmeier Mar 31, 2022
3d8f73a
test resending
rphmeier Mar 31, 2022
f4b6edf
Merge branch 'master' into rh-approval-grid
rphmeier Mar 31, 2022
28ff301
typo
rphmeier Apr 1, 2022
649e56d
add more metrics about apd messages
rphmeier Apr 1, 2022
0e5b376
Merge branch 'master' into rh-approval-grid
rphmeier Apr 10, 2022
4381479
Merge branch 'rh-approval-grid' of https://github.com/paritytech/polk…
rphmeier Apr 10, 2022
1522b79
add back unify_with_peer logs
rphmeier Apr 10, 2022
9fc7980
make Resend an enum
rphmeier Apr 10, 2022
14c3ea7
be more explicit when resending
rphmeier Apr 10, 2022
d0dc36d
fmt
rphmeier Apr 10, 2022
d406a2c
fix error
rphmeier Apr 10, 2022
1657d86
Merge branch 'master' into rh-approval-grid
rphmeier Apr 12, 2022
43a4123
add a TODO for refactoring
rphmeier Apr 12, 2022
383e7a0
remove debug metrics
rphmeier Apr 19, 2022
c8782c6
add some guide stuff
rphmeier Apr 19, 2022
a8b7bf8
fmt
rphmeier Apr 19, 2022
2f790ef
update runtime API in test-runtim
rphmeier Apr 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions node/core/approval-voting/src/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,7 @@ pub(crate) async fn handle_new_head(
parent_hash: block_header.parent_hash,
candidates: included_candidates.iter().map(|(hash, _, _, _)| *hash).collect(),
slot,
session: session_index,
});

imported_candidates.push(BlockImportedCandidates {
Expand Down
1 change: 1 addition & 0 deletions node/core/approval-voting/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1006,6 +1006,7 @@ fn distribution_messages_for_activation(
parent_hash: block_entry.parent_hash(),
candidates: block_entry.candidates().iter().map(|(_, c_hash)| *c_hash).collect(),
slot: block_entry.slot(),
session: block_entry.session(),
});

for (i, (_, candidate_hash)) in block_entry.candidates().iter().enumerate() {
Expand Down
3 changes: 3 additions & 0 deletions node/network/approval-distribution/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,13 @@ polkadot-node-network-protocol = { path = "../protocol" }
polkadot-node-subsystem = { path = "../../subsystem" }
polkadot-node-subsystem-util = { path = "../../subsystem-util" }
polkadot-primitives = { path = "../../../primitives" }
rand = "0.8"

futures = "0.3.21"
gum = { package = "tracing-gum", path = "../../gum" }

[dev-dependencies]
sp-authority-discovery = { git = "https://github.com/paritytech/substrate", branch = "master" }
sp-core = { git = "https://github.com/paritytech/substrate", branch = "master", features = ["std"] }

polkadot-node-subsystem-util = { path = "../../subsystem-util" }
Expand All @@ -23,5 +25,6 @@ polkadot-node-subsystem-test-helpers = { path = "../../subsystem-test-helpers" }
assert_matches = "1.4.0"
schnorrkel = { version = "0.9.1", default-features = false }
rand_core = "0.5.1" # should match schnorrkel
rand_chacha = "0.3.1"
env_logger = "0.9.0"
log = "0.4.16"
Loading