This speeds up wallet refresh by directly retrieving a tx's amount output indices.
It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.
"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.
Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:
tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]
Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>
The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.
See also d045dfa7ce
This is a list of existing output amounts along with the number
of outputs of that amount in the blockchain.
The daemon command takes:
- no parameters: all outputs with at least 3 instances
- one parameter: all outputs with at least that many instances
- two parameters: all outputs within that many instances
The default starts at 3 to avoid massive spamming of all dust
outputs in the blockchain, and is the current minimum mixin
requirement.
An optional vector of amounts may be passed, to request
histogram only for those outputs.
bdec7cb BlockchainLMDB: Use DB error helper consistently (warptangent)
c5932eb BlockchainLMDB: Add DB error to exception (warptangent)
a49c355 Blockchain: Omit verbose time stats messages by default (warptangent)
When keys are contiguous and monotonically increasing, this gets
denser page utilization (doesn't leave padding in page splits).
Can't be used for keys that are inserted in random order (e.g. hashes)
In total this only saves around 1.5% of space compared to original
DB code. The previous patch accounted for 0.8% savings on its own;
the blocks tables just aren't that big.
1995923 BlockchainLMDB: Deal with DB exceptions at block level with particularity (warptangent)
c16cc20 BlockchainLMDB: Add sanity check for inconsistent state (warptangent)
9118d0a BlockchainLMDB: Call destructor on allocated txn if setup fails (warptangent)
f5581c3 BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr (warptangent)
Add another DB error exception type to distinguish failed txn setup from
general use of txn.
This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" db. This is most noticeable for
HDDs.
See ffcf6bdb95
Data should be removed in the reverse order it was added.
This matches the order of removal in
blockchain_storage::pop_transaction_from_global_index.
See f11def012f
b39aae7 Tweak 45800a25e9 (hyc)
4a5a5ff blockchain: always stop the ioservice before returning (moneromooo-monero)
78b65cf db_lmdb: safety close db at exit (moneromooo-monero)
45800a2 db_lmdb: fix a strdup/delete[] mistmatch (moneromooo-monero)
79beed2 tests: fix various tests by using parameters better suited to monero (moneromooo-monero)
d0a8362 tests: fix some double spending tests (moneromooo-monero)
2358d0d tests: use 255 as a "too high" block version (moneromooo-monero)
f33a88c blockchain: fix a few block addition bugs (moneromooo-monero)
a9ff11c blockchain: fix an off by one error in unlocked time check (moneromooo-monero)
f294be3 blockchain: reinstate double spending checks in check_tx_inputs (moneromooo-monero)
737b6d6 blockchain: make some flag twiddling code closer to the original (moneromooo-monero)
81cb0fc blockchain: fix bitflipping test with quantized block rewards (moneromooo-monero)
22ddf09 blockchain: add missing m_tx_pool.on_blockchain_dec (moneromooo-monero)
d837c0c blockchain: fix switch to alternative blockchain for more than one block (moneromooo-monero)
5cec076 blockchain: add a missing validity check to rollback_blockchain_switching (moneromooo-monero)
3cabdb5 core: catch exceptions from get_output_key (moneromooo-monero)
5eef645 db: throw when given a non txout_to_key output to add (moneromooo-monero)
The check was explicit in the original version, so it seems
safer to make it explicit here, especially as it is now done
implicitely in a different place, away from the original check.
Data should be removed in the reverse order it was added. Not doing so
breaks assumptions and can cause problems in other DB implementations.
This matches the order of tx removal in
blockchain_storage::purge_block_data_from_blockchain.
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" subdb. This is most noticeable for
HDDs.
As before, the dup list is still walked if necessary (but in reverse),
and the global output index still confirmed to be the one looked for.
But under proper use, the result will be found at the end of the dup
list, so we start there.
Removing an amount output index is always done in the context of popping
a block, so the global output index being looked for should be the last
one in that amount key's dup list. Even if the txs themselves aren't
removed in reverse order (supposed to be according to original
implementation), the specified amount output index will still be near
the end, because the txs are in the same block.
TEST:
Pop blocks with blockchain_import.
Blocks should be successfully removed with no errors shown.
bitmonerod should be able to start syncing from the reduced blockchain
height.
cbded43 core_tests: fix ring_signature_1 tests (moneromooo-monero)
c3d208f core_tests: bump default test fee to 0.02 monero (moneromooo-monero)
10da0a0 add a --fakechain argument for tests (moneromooo-monero)
eee44e6 unit_tests: fix block reward test using post hard fork settings (moneromooo-monero)
595893f blockchain: log block (not chain) height in "BLOCK SUCCESFULLY ADDED" (moneromooo-monero)
2369968 blockchain: fix off by one in get_blocks (moneromooo-monero)
8af913a db_lmdb: implement BlockchainLMDB::reset (moneromooo-monero)
4833f4f db_bdb: implement BlockchainBDB::reset (moneromooo-monero)
18bf06e tx_pool: fix "minumim" typo in message (moneromooo-monero)
44f1267 tests: fix a typo in test name (moneromooo-monero)
1494557 db_lmdb: create all needed directories, not just the leaf one (moneromooo-monero)
015b68a db_bdb: create all needed directories, not just the leaf one (moneromooo-monero)
f141869 tests: remove data-dir argument registration (moneromooo-monero)
When throwing an exception from being unable to begin an LMDB
transaction, include the reason.
It's often been due to a write transaction attempted within a write
transaction (batch mode), but there can be other reasons such as write
transaction attempted while database was opened read only, or
environment's map needs to be resized.
Data is only guaranteed to be valid within the lifetime of a txn.
You cannot use data returned from LMDB after the txn ends.
Also, fixed a missing txn.commit BlockchainLMDB::get_tx_unlock_time()
This is a precaution for older Berkeley DB versions.
- smooth reports an issue running with 4.7:
DB_ENV->log_set_config: DB_LOG_IN_MEMORY: method not permitted
after handle's open method
- this works just fine with 5.3
- we do not use DB_LOG_IN_MEMORY, but we use DB_LOG_AUTO_REMOVE
- libdb docs say some flags must be set before open, and some
may be set at any time, but never say some must be set after
open
- moving the call to log_set_config before open works with 5.3
Therefore, it seems best to move the call before open.
Early DB versions did not store key images for inputs if the
transaction spending them had no outputs (ie, all fee). This
is not correct, as this would allow these outputs to be double
spent. This was fixed in 533acc30ed
a few months ago, but databases having synced blocks 2021612 and
685498 with a faulty version will be missing those key images
in the spent keys database. This code checks for this, and adds
those key images if they are missing.
It looks like some of the indices passed to the DB access functions
are already bumped by 1. Moreover, the existing code was not
throwing DB errors with 0 keys, and this is unlikely if it really
was using 0 keys. Last, this patch broke sync from scratch in at
least one case. So I'm calling it bad and reverting it.
This reverts commit bfc97401ae81bb30278a318de7f048c653bf6582.
