[PoA] Network stop at epoch block when run more than 1 sealer using clef signer

[0xcuonghx]

System information

Geth version: v1.10.17
OS & Version: Docker

Expected behaviour

Mining block & Synchronizing the block on the network as normal

Actual behaviour

Stop at epoch block

Steps to reproduce the behaviour

[https://github.com/cuonghx-gutech/docker-geth-private-network]

cd clef-geth-docker
docker-compose up

Backtrace

[node1_1  | INFO [05-20|06:49:35.957] Commit new sealing work                  number=4 sealhash=2761af..a328af uncles=0 txs=0 gas=0 fees=0 elapsed=984.527ms
node1_1  | INFO [05-20|06:49:36.985] Successfully sealed new block            number=4 sealhash=2761af..a328af hash=8ce753..d70491 elapsed=2.012s
node1_1  | INFO [05-20|06:49:36.986] 🔨 mined potential block                  number=4 hash=8ce753..d70491
node1_1  | INFO [05-20|06:49:36.987] Commit new sealing work                  number=5 sealhash=25522e..c87bb2 uncles=0 txs=0 gas=0 fees=0 elapsed=1.040ms
node1_1  | WARN [05-20|06:49:36.987] Block sealing failed                     err="signed recently, must wait for others"
node1_1  | INFO [05-20|06:49:36.987] Commit new sealing work                  number=5 sealhash=25522e..c87bb2 uncles=0 txs=0 gas=0 fees=0 elapsed=1.873ms
node2_1  | INFO [05-20|06:49:36.989] Imported new chain segment               blocks=1 txs=0 mgas=0.000 elapsed="873.1µs" mgasps=0.000 number=4 hash=8ce753..d70491 dirty=0.00B
node2_1  | INFO [05-20|06:49:36.990] Commit new sealing work                  number=5 sealhash=6344e3..c571bb uncles=0 txs=0 gas=0 fees=0 elapsed="548.8µs"
clef2_1  | INFO [05-20|06:49:36.999] Op approved 
node2_1  | INFO [05-20|06:49:37.055] New local node record                    seq=1,653,029,370,332 id=e8c992cc7e365a3b ip=8.21.11.140 udp=57331 tcp=30303
node1_1  | INFO [05-20|06:49:37.381] New local node record                    seq=1,653,029,370,694 id=839e7c3ae225f02a ip=8.21.11.140 udp=63701 tcp=30303
clef2_1  | DEBUG[05-20|06:49:37.971] Served account_signData                  reqid=5 duration=979.6799ms
node2_1  | INFO [05-20|06:49:37.972] Commit new sealing work                  number=5 sealhash=6344e3..c571bb uncles=0 txs=0 gas=0 fees=0 elapsed=982.279ms
node2_1  | ERROR[05-20|06:49:38.948] Failed to prepare header for sealing     err="unauthorized signer"
node2_1  | INFO [05-20|06:49:38.948] Successfully sealed new block            number=5 sealhash=6344e3..c571bb hash=bc88d0..7d4608 elapsed=1.992s
node2_1  | INFO [05-20|06:49:38.948] 🔨 mined potential block                  number=5 hash=bc88d0..7d4608
node2_1  | INFO [05-20|06:49:39.000] New local node record                    seq=1,653,029,370,333 id=e8c992cc7e365a3b ip=8.21.11.140 udp=57331 tcp=30303
node1_1  | INFO [05-20|06:49:40.770] Looking for peers                        peercount=1 tried=47 static=2
node2_1  | INFO [05-20|06:49:41.173] Looking for peers                        peercount=0 tried=66 static=2
node1_1  | INFO [05-20|06:49:44.812] Block synchronisation started 
node1_1  | INFO [05-20|06:49:44.812] Mining aborted due to sync 
node1_1  | INFO [05-20|06:49:44.818] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=68.90B throttle=8192
node1_1  | WARN [05-20|06:49:44.819] Failed to load old bad blocks            error="leveldb: not found"
node1_1  | ERROR[05-20|06:49:44.820] 
node1_1  | ########## BAD BLOCK #########
node1_1  | Chain config: {ChainID: 1337 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: 5 Petersburg: 6 Istanbul: 7, Muir Glacier: <nil>, Berlin: <nil>, London: <nil>, Arrow Glacier: <nil>, MergeFork: <nil>, Terminal TD: <nil>, Engine: clique}
node1_1  | 
node1_1  | Number: 5
node1_1  | Hash: 0xbc88d02a11ff3e866e87e4758f59aa79d60c73a4e63bdf18f31e7fc7387d4608
node1_1  | 
node1_1  | 
node1_1  | Error: unauthorized signer
node1_1  | ##############################
node1_1  |  
node1_1  | WARN [05-20|06:49:44.820] Synchronisation failed, dropping peer    peer=e8c992cc7e365a3b1fd5ff1fd69ff1e9e1929afab183e056eebb8c4695209475 err="retrieved hash chain is invalid: unauthorized signer"
node1_1  | INFO [05-20|06:49:44.821] Commit new sealing work                  number=5 sealhash=feb13c..e63076 uncles=0 txs=0 gas=0 fees=0 elapsed="492.6µs"
node1_1  | WARN [05-20|06:49:44.821] Block sealing failed                     err="signed recently, must wait for others"
node1_1  | INFO [05-20|06:49:44.821] Commit new sealing work                  number=5 sealhash=feb13c..e63076 uncles=0 txs=0 gas=0 fees=0 elapsed="754.3µs"
node1_1  | INFO [05-20|06:49:50.784] Looking for peers                        peercount=0 tried=50 static=2
node2_1  | INFO [05-20|06:49:51.322] Looking for peers                        peercount=0 tried=54 static=2
node2_1  | ERROR[05-20|06:49:56.835] Snapshot extension registration failed   peer=e6ad1738 err="peer connected on snap without compatible eth support"
node2_1  | INFO [05-20|06:50:01.503] Looking for peers                        peercount=0 tried=25 static=2
node1_1  | INFO [05-20|06:50:02.148] Looking for peers                        peercount=0 tried=41 static=2
node1_1  | INFO [05-20|06:50:05.333] Mining aborted due to sync 
node1_1  | INFO [05-20|06:50:05.339] Skip duplicated bad block                number=5 hash=bc88d0..7d4608
node1_1  | ERROR[05-20|06:50:05.339] 
node1_1  | ########## BAD BLOCK #########
node1_1  | Chain config: {ChainID: 1337 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: 5 Petersburg: 6 Istanbul: 7, Muir Glacier: <nil>, Berlin: <nil>, London: <nil>, Arrow Glacier: <nil>, MergeFork: <nil>, Terminal TD: <nil>, Engine: clique}
node1_1  | 
node1_1  | Number: 5
node1_1  | Hash: 0xbc88d02a11ff3e866e87e4758f59aa79d60c73a4e63bdf18f31e7fc7387d4608
node1_1  | 
node1_1  | 
node1_1  | Error: unauthorized signer
node1_1  | ##############################
node1_1  |  
node1_1  | WARN [05-20|06:50:05.339] Synchronisation failed, dropping peer    peer=e8c992cc7e365a3b1fd5ff1fd69ff1e9e1929afab183e056eebb8c4695209475 err="retrieved hash chain is invalid: unauthorized signer"
node1_1  | INFO [05-20|06:50:05.339] Commit new sealing work                  number=5 sealhash=f54821..980c10 uncles=0 txs=0 gas=0 fees=0 elapsed="161.2µs"
node1_1  | WARN [05-20|06:50:05.340] Block sealing failed                     err="signed recently, must wait for others"
node1_1  | INFO [05-20|06:50:05.340] Commit new sealing work                  number=5 sealhash=f54821..980c10 uncles=0 txs=0 gas=0 fees=0 elapsed="456.7µs"
node2_1  | INFO [05-20|06:50:11.469] Looking for peers                        peercount=1 tried=22 static=2
node1_1  | INFO [05-20|06:50:15.306] Mining aborted due to sync 
node1_1  | INFO [05-20|06:50:15.317] Skip duplicated bad block                number=5 hash=bc88d0..7d4608
node1_1  | ERROR[05-20|06:50:15.317] 
node1_1  | ########## BAD BLOCK #########
node1_1  | Chain config: {ChainID: 1337 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: 5 Petersburg: 6 Istanbul: 7, Muir Glacier: <nil>, Berlin: <nil>, London: <nil>, Arrow Glacier: <nil>, MergeFork: <nil>, Terminal TD: <nil>, Engine: clique}
node1_1  | 
node1_1  | Number: 5
node1_1  | Hash: 0xbc88d02a11ff3e866e87e4758f59aa79d60c73a4e63bdf18f31e7fc7387d4608
node1_1  | 
node1_1  | 
node1_1  | Error: unauthorized signer
node1_1  | ##############################
node1_1  |  
node1_1  | WARN [05-20|06:50:15.318] Synchronisation failed, dropping peer    peer=e8c992cc7e365a3b1fd5ff1fd69ff1e9e1929afab183e056eebb8c4695209475 err="retrieved hash chain is invalid: unauthorized signer"
node1_1  | INFO [05-20|06:50:15.319] Commit new sealing work                  number=5 sealhash=f38b61..67287b uncles=0 txs=0 gas=0 fees=0 elapsed="397.4µs"
node1_1  | WARN [05-20|06:50:15.321] Block sealing failed                     err="signed recently, must wait for others"
node1_1  | INFO [05-20|06:50:15.321] Commit new sealing work                  number=5 sealhash=f38b61..67287b uncles=0 txs=0 gas=0 fees=0 elapsed=2.155ms
node2_1  | INFO [05-20|06:50:21.483] Looking for peers                        peercount=0 tried=87 static=2
node1_1  | INFO [05-20|06:50:22.494] Looking for peers                        peercount=0 tried=21 static=2]

When submitting logs: please submit them as text and not screenshots.

[jwasinger]

I am looking into this.