---
news 2018-Dec-16:
## refactored!
Everything is in a different place now, sorry. But it urgently needed cleanup.
Please open an issue (or fork & pull request) if you find a broken link, thanks.
--> https://github.com/drandreaskrueger/chainhammer/issues
---
TPS measurements of parity aura, geth clique, quorum, tobalaba, etc. It should work with any Ethereum type chain; we focused on PoA consensus.
hammer/- submits many transactions to blockchain - see next chapter 'chronology'reader/- reads in the whole chain, and visualizes TPS, blocktime, gas, bytes - see reader/README.mdresults/- for each client one markdown filedocs/- see esp. reproduce.mdenv/- Python virtualenv, created via script, see belowtests/- all started via./pytest.sh
See the results/ folder:
- log.md: initial steps; also tried Quorum's private transactions
- quorum.md: raft consensus, quorum is a geth fork
- tobalaba.md: parity fork of EnergyWebFoundation
- quorum-IBFT.md: IstanbulBFT, 2nd consensus algo in quorum
- geth.md: geth clique PoA algorithm
- parity.md: parity aura PoA algorithm, many attempts to accelerate
- eos.md: not begun yet
- substrate.md: not begun yet
| hardware | node type | #nodes | config | peak TPS_av | final TPS_av |
|---|---|---|---|---|---|
| t2.micro | parity aura | 4 | (D) | 45.5 | 44.3 |
| t2.large | parity aura | 4 | (D) | 53.5 | 52.9 |
| t2.xlarge | parity aura | 4 | (J) | 57.1 | 56.4 |
| t2.2xlarge | parity aura | 4 | (D) | 57.6 | 57.6 |
| t2.micro | parity instantseal | 1 | (G) | 42.3 | 42.3 |
| t2.xlarge | parity instantseal | 1 | (J) | 48.1 | 48.1 |
| t2.2xlarge | geth clique | 3+1 +2 | (B) | 421.6 | 400.0 |
| t2.xlarge | geth clique | 3+1 +2 | (B) | 386.1 | 321.5 |
| t2.xlarge | geth clique | 3+1 | (K) | 372.6 | 325.3 |
| t2.large | geth clique | 3+1 +2 | (B) | 170.7 | 169.4 |
| t2.small | geth clique | 3+1 +2 | (B) | 96.8 | 96.5 |
| t2.micro | geth clique | 3+1 | (H) | 124.3 | 122.4 |
| t2.micro SWAP | quorum crux IBFT | 4 | (I) SWAP! | 98.1 | 98.1 |
| t2.micro | quorum crux IBFT | 4 | (F) | lack of RAM | |
| t2.large | quorum crux IBFT | 4 | (F) | 207.7 | 199.9 |
| t2.xlarge | quorum crux IBFT | 4 | (F) | 439.5 | 395.7 |
| t2.xlarge | quorum crux IBFT | 4 | (L) | 389.1 | 338.9 |
| t2.2xlarge | quorum crux IBFT | 4 | (F) | 435.4 | 423.1 |
| c5.4xlarge | quorum crux IBFT | 4 | (F) | 536.4 | 524.3 |
Reproduce these results easily; for the config column also see there.
Quickest reproduction with my Amazon AMI readymade image.
And see that bottom of parity.md and geth.md
and quorum-IBFT.md for the latest runs, and additional details.
- how I initially got this faster, on Quorum, step by step, please read the 1st logbook log.md
- then I improved per client, see each in #chronology
- what to try next: TODO.md = e.g. vary transaction size, automate more, etc.
See other-projects.md using chainhammer, or projects which are similar to this.
Please report back when you have done other / new measurements.
- parity PE#9393 60 TPS ? (parity aura v1.11.11)
- parity SE#58521 parity TPS optimization - please help - stackexchange.com
- geth GE#17447 Sudden drop in TPS after total 14k transactions.
- quorum Q#479 Sudden drop in TPS around 14k transactions (Quorum IBFT)
New in v44: Installer for EVERYTHING that this repo needs!
cd chainhammer # you must be in main folder
scripts/install.sh
but be CAREFUL: Better only use on a disposable/cloud/virtualbox machine, and NOT on your main work machine!!
or
only create the virtualenv for the chainhammer Python programs, then look into scripts/install-virtualenv.sh
(For more details see reproduce.md).
All python scripts & jupyer notebooks must be run within that virtualenv, e.g.:
cd hammer; source ../env/bin/activate
Now start your ethereum node(s), or simply: source env/bin/activate; testrpc-py
Before first ever run of chainhammer:
touch account-passphrase.txt
./deploy.py andtests
It tests whether communication with the ethereum node is working,
and initially creates local files about the compiled and deployed contract.
If there are connection problems, check the ports in config.py -->
RPCaddress, RPCaddress2.
Remember, in each new terminal virtualenv: cd hammer; source ../env/bin/activate
first terminal:
./tps.py
second terminal:
./deploy.py; ./send.py
Then, after all (e.g. 20,001) transactions have been seen,
extract the whole chain into parity-run17.db (example);
and create the diagrams
cd ../reader
rm -f parity-run17.db*
./blocksDB_create.py parity-run17.db
./blocksDB_diagramming.py parity-run17.db Parity-run-17
./pytest.sh
enables the virtualenv,
then starts a testrpc-py Ethereum simulator on http://localhost:8545 in the background,
logging into tests/logs/;
then runs ./deploy.py andtests;
and finally runs all the unittests, also logging into tests/logs/.
(Instead of testrpc-py) if you want to run tests with another node,
just start that; and run pytest manually:
source env/bin/activate
cd hammer; ./deploy.py andtests; cd ..
py.test -v --cov
There were 64 tests on December 4th, all 64 PASSED
(see this logfile -->
cat tests/logs/*.ansi because colors) on these different Ethereum providers:
- testrpc instantseal (
testrpc-py) 9 seconds - geth Clique (
geth-dev) 54 seconds - quorum IBFT (
blk-io/crux) 31 seconds - parity instantseal (
parity-deploy) 8 seconds - parity aura (
parity-deploy) 74 seconds
Please credit this as:
benchmarking scripts "chainhammer"
beginning developed at Electron.org.uk 2018
current maintainer: Dr Andreas Krueger 2018
https://github.com/drandreaskrueger/chainhammer
Consider to submit your improvements & usage as pull request. Thanks.
The open source tools 'chainhammer' submits a high load of smart contract transactions to an Ethereum based blockchain, then 'chainreader' reads the whole chain, and produces diagrams of TPS, blocktime, gasUsed and gasLimit, and the blocksize. https://github.com/drandreaskrueger/chainhammer
examples:
geth.md = geth (go ethereum client), "Clique" consensus.
50,000 transactions to an Amazon t2.xlarge machine.
Interesting artifact that after ~14k transactions, the speed drops considerably - but recovers again. Reported.
reader/img/geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png
quorum-IBFT.md = Quorum (geth fork), IBFT consensus, 20 millions gasLimit, 1 second istanbul.blockperiod; 20000 transactions multi-threaded with 23 workers. Initial average >400 TPS then drops to below 300 TPS, see quorum issue)
OLD RUN on a desktop machine.
quorum.md = Quorum (geth fork), raft consensus, 1000 transactions multi-threaded with 23 workers, average TPS around 160 TPS, and 20 raft blocks per second)
OLD RUN on a desktop machine.
tobalaba.md = Public "Tobalaba" chain of the EnergyWebFoundation (parity fork), PoA; 20k transactions; > 150 TPS if client is well-connected.
parity.md#run-18 = using parity-deploy.sh dockerized network of 4 local nodes with increased gasLimit, and 5 seconds blocktime; 20k transactions; ~ 60 TPS on an Amazon t2.xlarge machine.
N.B.: Could not work with parity v2 yet because of bugs PD#76 and PE#9582 --> everything still on parity v1.11.11
parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png
Calling all parity experts: How to improve these too slow TPS results?
See issue PE#9393, and the detailed log of what I've tried already, and the 2 shortest routes to reproducing the results: reproduce.md.
Thanks.