Regression: indexing bitcoin mainnet from scratch makes electrs panic

[edouardklein]

On the latest commit (currently 503b740), running

cargo run --release --bin electrs -- -vvvv --daemon-dir=/data/blockchains/BTC/ --db-dir=/data/blockchains/BTC/electrs/ --network mainnet --address-search --index-unspendables

Yields the following backtrace:

> thread 'main' (2618) panicked at src/new_index/schema.rs:520:76:called Result::unwrap() on an Err value: Error(Msg("missing txo 3c04284d2ba5f65d15122953eba8329adb778c5b3881ecd2830926cb57161400:0"), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(  
 0: error_chain::backtrace::imp::InternalBacktrace::new   
1: electrs::new_index::schema::lookup_txos  
 2: electrs::new_index::schema::Indexer::_index  
 3: electrs::new_index::fetch::Fetcher<T>::map   
4: electrs::new_index::schema::Indexer::update   
5: electrs::main   
6: std::sys::backtrace::__rust_begin_short_backtrace  
 7: main  
 8: __libc_start_call_main   
9: __libc_start_main_impl  
10: _start) } })stack backtrace:  
 0:     0x565389ea9f95 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h0bf65c5ade08ad27 
  1:     0x565389d1abe7 - core::fmt::write::he00acd824749ac8b 
  2:     0x565389ea7491 - std::io::Write::write_fmt::h90fe4513e04f986c 
  3:     0x565389ea9e2f - std::sys::backtrace::BacktraceLock::print::h3775b21ceaf4ad1d 
  4:     0x565389e9cc66 - std::panicking::panic_with_hook::h0fc085f8b6efb327 
  5:     0x565389eaa3f7 - std::panicking::panic_handler::{{closure}}::h9d26b9b04a44e043  
 6:     0x565389eaa3a9 - std::sys::backtrace::__rust_end_short_backtrace::hce0c26d73f44f74b  
 7:     0x565389e9adac - __rustc[881eff9798cad629]::rust_begin_unwind   8:     0x565389b843eb - core::panicking::panic_fmt::h7509f5fc1f52ade1   
9:     0x565389b846f1 - core::result::unwrap_failed::h4811fcb6dcdb23ec  
10:     0x565389e2431c - electrs::new_index::schema::Indexer::_index::h704b5e9859f0197d  
11:     0x565389e26f8f - electrs::new_index::fetch::Fetcher<T>::map::h57fb5b5bf9fd1422  
12:     0x565389e1edc1 - electrs::new_index::schema::Indexer::update::had48397a4a9f640b  
13:     0x565389ca69ec - electrs::main::h323dd1c456f6e711  
14:     0x565389c9e063 - std::sys::backtrace::__rust_begin_short_backtrace::h918d2ba5502615b1 
 15:     0x565389cab2e6 - main  
16:     0x7f57fe14ebd7 - __libc_start_call_main  
17:     0x7f57fe14ec95 - __libc_start_main_impl  
18:     0x565389c9d941 - _start 
 19:                0x0 - <unknown>fish: Job 1, 'RUST_BACKTRACE=full cargo run -…' terminated by signal SIGABRT (Abort)

Reverting back to 2c33745 solves the problem.

I checked the state of /data/blockchains/BTC/ for the incriminated txo:

 bitcoin-cli -datadir=/data/blockchains/BTC/ getblockhash 119041
0000000000001f30f8e6e01b410f5b2001d6aee69401f5a97a28aceb33e99352

bitcoin-cli -datadir=/data/blockchains/BTC/ getrawtransaction \
                                         3c04284d2ba5f65d15122953eba8329adb778c5b3881ecd2830926cb57161400
\
                                         true 0000000000001f30f8e6e01b410f5b2001d6aee69401f5a97a28aceb33e99352 | jq '.vout[0]'
{
  "value": 9.80000000,
  "n": 0,
  "scriptPubKey": {
    "asm": "OP_DUP OP_HASH160 b010ba17bb034873991301660a6a73654f744eb3 OP_EQUALVERIFY OP_CHECKSIG",
    "desc": "addr(1H3wxyDZH5MNgz9Au7tQmYfVgyXhk3qsjd)#4ueqfdcd",
    "hex": "76a914b010ba17bb034873991301660a6a73654f744eb388ac",
    "address": "1H3wxyDZH5MNgz9Au7tQmYfVgyXhk3qsjd",
    "type": "pubkeyhash"
  }
}

which looks correct to me.

I'm in a pickle here and can't investigate further right now. I'll stay on 2c33745 for the time being, in the meantime I hope this report is useful.

[EddieHouston]

Thanks for reporting this and sorry it is causing you troubles. It's possible that
something is happening with the blk*.dat fetcher in that it may not send blocks
in chain order as the new single pass indexing seems to require.
...more investigation needed to fully root cause and resolve.

Workaround: add --jsonrpc-import to your invocation:

  cargo run --release --bin electrs -- -vvvv \                                                                                                                                                                                                  
    --daemon-dir=/data/blockchains/BTC/ \
    --db-dir=/data/blockchains/BTC/electrs/ \                                                                                                                                                                                                   
    --network mainnet --address-search --index-unspendables \
    --jsonrpc-import                                                                                                                                                                                                                            

That routes block fetching through bitcoind's JSON-RPC, which delivers blocks in
chain order and avoids the ordering assumption that f66e7fb relies on. It's
slightly slower than reading blk files directly, but should let you sync from scratch on
the latest commit without reverting.

[EddieHouston]

btw... the blk file fetcher is likely going away in the future and --jsonrpc-import will become the default.