Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key

[dolphintwo]

Server and Service

• 8C / 16G / ubuntu20.04.5
• sync from zero without snapshots.
• binary versions: 4.44.0, 4.39.1, 4.38.0
• service unit file:

[Unit]
Description=Astar
After=network.target

[Service]
Type=simple
User=ubuntu
Group=ubuntu
ExecStart=/usr/local/bin/astar-collator \
	--chain astar \
	--validator --name "*****" \
	--rpc-cors all --telemetry-url 'wss://telemetry.polkadot.io/submit/ 0' \
	--out-peers 35 --in-peers 35 \
	--execution wasm --wasm-execution=compiled \
	-d /data/astar/ \
	--trie-cache-size 1 \
	--no-prometheus
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

error_log

2023-01-30 10:52:24 [Relaychain] ✨ Imported #14026789 (0xc2a5…3a20)
2023-01-30 10:52:24 [Parachain] Starting collation. relay_parent=0xc2a579d079f28c7a33415fb00c766786e8574fecc235d578707e0c359b583a20 at=0x48d9c10c4f8962b2199080932fc2018165e37fdbc9f391f36db54260786e887e
2023-01-30 10:52:25 [Parachain] ✨ Imported #2834782 (0x6c4d…d7ae)
2023-01-30 10:52:28 [Parachain] 💤 Idle (53 peers), best: #2834781 (0x48d9…887e), finalized #2834779 (0x9d92…54f1), ⬇ 129.2kiB/s ⬆ 212.1kiB/s
2023-01-30 10:52:28 [Relaychain] 💤 Idle (11 peers), best: #14026789 (0xc2a5…3a20), finalized #14026786 (0x00ce…4e34), ⬇ 52.5kiB/s ⬆ 134.6kiB/s
2023-01-30 10:52:30 [Relaychain] ✨ Imported #14026790 (0xd43d…d57f)
====================
Version: 4.39.1-58039d060c1
   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/alloc/src/boxed.rs:1952:9
      std::panicking::rust_panic_with_hook
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/std/src/panicking.rs:673:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/std/src/panicking.rs:560:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/std/src/panicking.rs:556:5
   5: core::panicking::panic_fmt
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/core/src/panicking.rs:142:14
   6: core::result::unwrap_failed
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/core/src/result.rs:1791:5
   7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
   8: sp_io::storage::get_version_1
   9: sp_io::storage::ExtStorageGetVersion1::call
  10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: wasmtime_runtime::traphandlers::catch_traps::call_closure
  18: wasmtime_setjmp
  19: sc_executor_wasmtime::runtime::perform_call
  20: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
  21: sc_executor_common::wasm_runtime::WasmInstance::call_export
  22: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  23: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  24: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  25: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
  26: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  27: <astar_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::__runtime_api_internal_call_api_at
  28: sc_transaction_pool::api::validate_transaction_blocking
  29: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  30: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  31: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  32: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  33: tokio::runtime::task::raw::poll
  34: std::sys_common::backtrace::__rust_begin_short_backtrace
  35: core::ops::function::FnOnce::call_once{{vtable.shim}}
  36: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/alloc/src/boxed.rs:1938:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/alloc/src/boxed.rs:1938:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/81f391930301afbc121b7c468138069daa354bf8/library/std/src/sys/unix/thread.rs:108:17
  37: start_thread
  38: clone
Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x457f2abf0f75072884b3a62fe18739926170cac1e0ff25d8e9dbdb34f01248e7"', /home/astar-admin/.cargo/git/checkouts/substrate-7e08433d4c370a21/2dff067/primitives/state-machine/src/ext.rs:189
This is a bug. Please report it at:
        https://github.com/AstarNetwork/Astar/issues/new
astar.service: Main process exited, code=exited, status=1/FAILURE
astar.service: Failed with result 'exit-code'.
astar.service: Scheduled restart job, restart counter is at 1.
Stopped Astar.
Started Astar.
2023-01-30 10:52:35 Astar Collator
2023-01-30 10:52:35 ✌️  version 4.39.1-58039d060c1
2023-01-30 10:52:35 ❤️  by Stake Technologies <devops@stake.co.jp>, 2019-2023
2023-01-30 10:52:35 📋 Chain specification: Astar
2023-01-30 10:52:35 🏷  Node name: ******
2023-01-30 10:52:35 👤 Role: AUTHORITY
2023-01-30 10:52:35 💾 Database: RocksDb at /data/astar/chains/astar/db/full
2023-01-30 10:52:35 ⛓  Native runtime: astar-47 (astar-0.tx2.au1)
2023-01-30 10:52:35 Parachain id: Id(2006)

[shunsukew]

Thank you for reporting, This issue comes from upstream Substrate.
Related discussion: paritytech/polkadot#6377 same issue was there on Polkadot.

Patch paritytech/substrate#12927 was made and will be included in Astar by updating Substrate dependency version and it should fix this issue.

As a workaround, using --pruning archive mode can avoid this issue. you may need to cleanup paradb ({base path}/chains/astar/db) and sync from the beginning.

[dolphintwo]

Thanks for your reply, I will use four different versions to sync, and hope that the sync will be completed normally.

[dolphintwo]

Sync with archive mode does avoid this problem