[BUG] - Mainnet Node stops syncing altogether after encountering an invalid block

[mesudip]

External

Our mainnet node stopped syncing after this same error message twice

[ Invalid block xxx at slot xxx: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (WrappedShelleyEraFailure (UtxoFailure (FeeTooSmallUTxO (Coin 1006053) (Coin 1001829)))))))])}))))))

When it first appeared we were on node version 1.31.0. Then we upgraded to 1.34.1 and the issue was seen again after 4 days.

Current Solution

Restarting the node solves the sync issue and it quickly catch ups to the latest tip.

Node Log

[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:29:28.46 UTC] Chain extended, new tip: fb24144096b303148f1b60cd974997950f8f99c9961b7bec41d86fc494479673 at slot 67265076
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:29:31.21 UTC] Chain extended, new tip: c411cb8f6a1a969edbe783126acd0e620b7d8ced2a8226da3d47c775ab237c3a at slot 67265080
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:29:33.16 UTC] Chain extended, new tip: b49dbbdeb7912f585479860979cca0e181451dbbbaa35531998cb54f5ed95e16 at slot 67265082
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:29:35.11 UTC] Chain extended, new tip: 82a3b2ce366fcb7a3ccfb04744c2fd7680b7025dd8be73af4abfa91a91a4196a at slot 67265084
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:29:41.36 UTC] Chain extended, new tip: 72aa49917a34f592fae7e950fd8b67fcc7f4c3d42fa4c12d0a1c6fa20e6ac83b at slot 67265090
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:30:25.25 UTC] IP LocalAddress "/ipc/node.socket@184420" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:30:56.99 UTC] Chain extended, new tip: 66d4d6a4df23df9c789a9c10fac578059aceb8073bc30c5213129a6e7f3ca955 at slot 67265165
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:31:34.69 UTC] Chain extended, new tip: 76078b5c84e837eb9fdc8fc3f9fc6f6098e6e5ef0353e04b17411da2fc8e3a85 at slot 67265203
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:31:47.86 UTC] Chain extended, new tip: c418d8828f521f3ad904dcb5a2fc753733dbc21c1d864fcdd062a4e38c3605d6 at slot 67265216
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:32:40.96 UTC] Chain extended, new tip: ca0a6d36c182fb45b4a9b4201aec4de5bbcb244516c269b31f7f221dc43c80ae at slot 67265269
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:32:43.71 UTC] Chain extended, new tip: cbe47b2ca0e623b021812cd2cfa2eaaffb8146284a2b3af115d027f56fbba97b at slot 67265272
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:32:52.77 UTC] IP LocalAddress "/ipc/node.socket@184475" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 23> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:33:21.80 UTC] Chain extended, new tip: 2ec7e4e60a2d9b7439e102be7603819f8227931b220485a9da08b3adfd6aa60a at slot 67265310
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:33:30.59 UTC] Chain extended, new tip: c2526c1d869d63f66742b2020ef8a1112d09921f51ab2be95c62b4aa9e12b96d at slot 67265319
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:33:40.76 UTC] Chain extended, new tip: 4a9e215e1b94897fbd2c7289f5145ab159edc3c0d572f8448a26fc4e03f7fff4 at slot 67265329
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:34:03.80 UTC] Chain extended, new tip: cd4031f67c0a0d9f095af44b46d54603575a1fd4b9de8097d5e9fc2639489537 at slot 67265352
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:34:52.62 UTC] IP LocalAddress "/ipc/node.socket@184528" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 27> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.ChainDB:Notice:30156] [2022-07-26 10:34:56.04 UTC] Chain extended, new tip: 8c8a9728a3ba4f9ad0f27a73e2a4595a89ee0d84d7639862922241dd1a21c28f at slot 67265404
[989a3ceb:cardano.node.ChainDB:Error:30156] [2022-07-26 10:35:33.62 UTC] Invalid block 4948f4835f6857ad246759845e5b7133d0393dca6ee691d32b35e44c9e0cddf5 at slot 67265442: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (WrappedShelleyEraFailure (UtxoFailure (FeeTooSmallUTxO (Coin 1006053) (Coin 1001829)))))))])}))))))
[989a3ceb:cardano.node.ChainDB:Info:30156] [2022-07-26 10:35:33.62 UTC] Valid candidate 8c8a9728a3ba4f9ad0f27a73e2a4595a89ee0d84d7639862922241dd1a21c28f at slot 67265404
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:37:20.40 UTC] IP LocalAddress "/ipc/node.socket@184591" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:39:20.76 UTC] IP LocalAddress "/ipc/node.socket@184628" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 34> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:41:48.30 UTC] IP LocalAddress "/ipc/node.socket@184689" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:43:48.34 UTC] IP LocalAddress "/ipc/node.socket@184732" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:46:16.54 UTC] IP LocalAddress "/ipc/node.socket@184793" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:48:16.81 UTC] IP LocalAddress "/ipc/node.socket@184835" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:50:43.92 UTC] IP LocalAddress "/ipc/node.socket@184892" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:52:43.96 UTC] IP LocalAddress "/ipc/node.socket@184936" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 23> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:55:11.48 UTC] IP LocalAddress "/ipc/node.socket@184977" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 39> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:57:11.24 UTC] IP LocalAddress "/ipc/node.socket@185021" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 10:59:39.22 UTC] IP LocalAddress "/ipc/node.socket@185068" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 23> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:01:39.66 UTC] IP LocalAddress "/ipc/node.socket@185105" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 43> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:04:06.63 UTC] IP LocalAddress "/ipc/node.socket@185156" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 23> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:06:06.48 UTC] IP LocalAddress "/ipc/node.socket@185187" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 48> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:08:34.32 UTC] IP LocalAddress "/ipc/node.socket@185238" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:10:34.74 UTC] IP LocalAddress "/ipc/node.socket@185281" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 43> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:13:01.69 UTC] IP LocalAddress "/ipc/node.socket@185320" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:15:01.91 UTC] IP LocalAddress "/ipc/node.socket@185361" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:17:30.62 UTC] IP LocalAddress "/ipc/node.socket@185408" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:19:30.46 UTC] IP LocalAddress "/ipc/node.socket@185443" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:21:58.17 UTC] IP LocalAddress "/ipc/node.socket@185492" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:23:58.21 UTC] IP LocalAddress "/ipc/node.socket@185532" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:26:25.86 UTC] IP LocalAddress "/ipc/node.socket@185581" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:28:25.89 UTC] IP LocalAddress "/ipc/node.socket@185617" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")
[989a3ceb:cardano.node.LocalErrorPolicy:Error:30187] [2022-07-26 11:30:53.70 UTC] IP LocalAddress "/ipc/node.socket@185664" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 44> closed when reading data, waiting on next header True")

[mesudip]

It is getting more frequent

[806b9cfc:cardano.node.ChainDB:Info:256] [2022-07-28 02:49:49.37 UTC] Block fits onto some fork: 4160d6880040107c64dd74e3b143176afae587d676f53700ebaaf08eef7977e7 at slot 67410297
[806b9cfc:cardano.node.ChainDB:Notice:256] [2022-07-28 02:49:49.45 UTC] Switched to a fork, new tip: 4160d6880040107c64dd74e3b143176afae587d676f53700ebaaf08eef7977e7 at slot 67410297
[806b9cfc:cardano.node.ChainDB:Notice:256] [2022-07-28 02:50:42.21 UTC] Chain extended, new tip: 17bdb8552bd3e4b14e72968b376525fdc9cb394b394995deeeeca84b42017673 at slot 67410350
[806b9cfc:cardano.node.ChainDB:Notice:256] [2022-07-28 02:50:56.34 UTC] Chain extended, new tip: da991c49ca72d21e3fe6af054baf990c83c98a8c557d06735f612ba92acf2481 at slot 67410364
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 02:51:35.19 UTC] IP LocalAddress "/ipc/node.socket@46703" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 23> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.ChainDB:Notice:256] [2022-07-28 02:51:47.24 UTC] Chain extended, new tip: 27ca0727f1d4a4bd6cc959ae80c6b7ced9c6c2858cd56112542e4d41aef00620 at slot 67410415
[806b9cfc:cardano.node.Mempool:Info:509631] [2022-07-28 02:52:35.43 UTC] fromList [("tx",Object (fromList [("txid",String "481f84e4")])),("kind",String "TraceMempoolAddedTx"),("mempoolSize",Object (fromList [("bytes",Number 1115.0),("numTxs",Number 1.0)]))]
[806b9cfc:cardano.node.DnsSubscription:Error:508335] [2022-07-28 02:53:17.38 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Application Exception: 18.159.46.171:3001 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (ServerAgency TokNext TokMustReply)
[806b9cfc:cardano.node.ErrorPolicy:Notice:275] [2022-07-28 02:53:17.40 UTC] IP 18.159.46.171:3001 ErrorPolicySuspendConsumer (Just (ApplicationExceptionTrace ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (ServerAgency TokNext TokMustReply))) 20s
[806b9cfc:cardano.node.DnsSubscription:Notice:509784] [2022-07-28 02:53:18.41 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 54.254.117.140:3001
[806b9cfc:cardano.node.DnsSubscription:Notice:509785] [2022-07-28 02:53:18.44 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 18.158.147.46:3001
[806b9cfc:cardano.node.DnsSubscription:Notice:509785] [2022-07-28 02:53:18.44 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt End, destination 18.158.147.46:3001 outcome: ConnectSuccessLast
[806b9cfc:cardano.node.ErrorPolicy:Notice:275] [2022-07-28 02:53:18.44 UTC] IP 54.254.117.140:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[806b9cfc:cardano.node.ChainDB:Error:256] [2022-07-28 02:53:18.65 UTC] Invalid block 9dece185434fb9a1582f459be012637059945c0f6c3c4bc74fde332f25f8f4a9 at slot 67410475: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (WrappedShelleyEraFailure (UtxoFailure (FeeTooSmallUTxO (Coin 1006053) (Coin 1001829))))))),ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (WrappedShelleyEraFailure (UtxoFailure (FeeTooSmallUTxO (Coin 1011641) (Coin 1007417)))))))])}))))))
[806b9cfc:cardano.node.ChainDB:Info:256] [2022-07-28 02:53:18.65 UTC] Valid candidate 27ca0727f1d4a4bd6cc959ae80c6b7ced9c6c2858cd56112542e4d41aef00620 at slot 67410415
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 02:54:01.92 UTC] IP LocalAddress "/ipc/node.socket@46765" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 30> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 02:56:01.70 UTC] IP LocalAddress "/ipc/node.socket@46817" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 02:58:28.78 UTC] IP LocalAddress "/ipc/node.socket@46870" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:00:29.19 UTC] IP LocalAddress "/ipc/node.socket@46921" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 30> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:02:57.42 UTC] IP LocalAddress "/ipc/node.socket@46982" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 30> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:04:57.23 UTC] IP LocalAddress "/ipc/node.socket@47022" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:07:24.56 UTC] IP LocalAddress "/ipc/node.socket@47083" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:09:24.98 UTC] IP LocalAddress "/ipc/node.socket@47131" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 31> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:11:52.30 UTC] IP LocalAddress "/ipc/node.socket@47187" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 32> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:13:52.10 UTC] IP LocalAddress "/ipc/node.socket@47235" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:16:20.61 UTC] IP LocalAddress "/ipc/node.socket@47292" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:18:21.06 UTC] IP LocalAddress "/ipc/node.socket@47339" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 35> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:20:47.66 UTC] IP LocalAddress "/ipc/node.socket@47389" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 35> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:22:47.61 UTC] IP LocalAddress "/ipc/node.socket@47434" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:25:15.04 UTC] IP LocalAddress "/ipc/node.socket@47489" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")
[806b9cfc:cardano.node.LocalErrorPolicy:Error:287] [2022-07-28 03:27:15.50 UTC] IP LocalAddress "/ipc/node.socket@47527" ErrorPolicyUnhandledApplicationException (MuxError MuxBearerClosed "<socket: 28> closed when reading data, waiting on next header True")

[JaredCorduan]

This looks to be the same issue as #4228, which is fixed by IntersectMBO/cardano-ledger#2936

[dcoutts]

Restarting the node solves the sync issue and it quickly catch ups to the latest tip.

Is your node configured to talk to many peers or just one? My initial suspicion about a node appearing "stuck" is that it now has no peers to talk to, because it temporarily banned the one it received a bad block from. Restarting the node would have the effect that it forgets the temporary bans. If it's only configured to talk to one other peer then after banning that one peer, it'd have none. Public relays on mainnain should of course be configured to talk to many other public relays.

[dorin100]

@mesudip based on the above feedback, can we close this issue?

[mesudip]

I've not yet encountered this issue after updating the node to new version. I am closing this.