This speeds up wallet refresh by directly retrieving a tx's amount output indices.
It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.
"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.
Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:
tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]
Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>
The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.
See also d045dfa7ce
b4eada9 wallet: make load_keys check types when loading JSON (moneromooo-monero)
3e55725 wallet: make the JSON reading type safe (moneromooo-monero)
f8d05f3 common: new json_util.h (moneromooo-monero)
This can generate non decomposed outputs for very large block
rewards (or not so large ones if a miner decides to not quantize
the block rewards). Out of an abundance of caution, we refuse
to generate those. They are still accepted by the consensus code,
however.
d5d46e6 tests: obligatory hardfork unit build fix after interface change (moneromooo-monero)
25672d3 wallet: pass std::function by const ref, not value (moneromooo-monero)
0be6e08 wallet: do not leak owned amounts to the daemon unless --trusted-daemon (moneromooo-monero)
12146da wallet: change sweep_dust to sweep_unmixable (moneromooo-monero)
600a3cf New RPC and daemon command to get output histogram (moneromooo-monero)
f9a2fd2 wallet: handle rare case where fee adjustment can bump to the next kB (moneromooo-monero)
f26651a wallet: factor fee calculation (moneromooo-monero)
This allows appropriate action to be taken, like displaying
the reason to the user.
Do just that in simplewallet, which should help a lot in
determining why users fail to send.
Also make it so a tx which is accepted but not relayed is
seen as a success rather than a failure.
With the change in mixin rules for v2, the "annoying" outputs are
slightly changed. There is high correlation between dust and
unmixable, but no equivalence.
This is a list of existing output amounts along with the number
of outputs of that amount in the blockchain.
The daemon command takes:
- no parameters: all outputs with at least 3 instances
- one parameter: all outputs with at least that many instances
- two parameters: all outputs within that many instances
The default starts at 3 to avoid massive spamming of all dust
outputs in the blockchain, and is the current minimum mixin
requirement.
An optional vector of amounts may be passed, to request
histogram only for those outputs.
This was meant to go in v2, but the miner tx slipped through
the cracks as it doesn't go through the main tx verification
since it doesn't get added to the pool.
It takes a filename containing JSON data to generate a wallet.
The following fields are valid:
version: integer, should be 1
filename: string, path/filename for the newly created wallet
scan_from_height: 64 bit unsigned integer, optional
password: string, optional
viewkey: string, hex representation
spendkey: string, hex representation
seed: string, optional, list of words separated by spaces
Either seed or private keys should be given. If using private
keys, the spend key may be omitted (the wallet will not be
able to spend, but will see incoming transactions).
If scan_from_height is given, blocks below this height will not
be checked for transactions as an optimization.
tx_pool.h doxygen documentation completed.
Many notes made on areas for improvement, be that functionality or
code clarity.
Commented code and unused code removed.
The functions in src/cryptonote_core/checkpoints_create.{h,cpp} should
be member functions of the checkpoints class, if nothing else for the
sake of keeping their documentation together.
This commit covers moving those functions to be member functions of the
checkpoints class as well as documenting those functions.
All functions in src/cryptonote_core/checkpoints.h are now documented in
doxygen style.
checkpoints.cpp has been reviewed, one function has been marked for
discussion on correctness.
All functions are now documented in doxygen format. Comments have been
updated to reflect the current state of the code. Many areas for
improvement in clarity and design have been noted, as well as cruft to
be removed. These changes are not reflected in this commit both to
allow time for comment and to keep commits organized by purpose.
Example of current return for `print_block 912345`:
timestamp: 1452793716
previous hash:
b61c58b2e0be53fad5ef9d9731a55e8a81d972b8d90ed07c04fd37ca6403ff78
nonce: 1646
is orphan: 0
height: 912345
depth: 85434
hash:
e22cf75f39ae720e8b71b3d120a5ac03f0db50bba6379e2850975b4859190bc6difficul
ty: 815625611
reward: 7388968946286
{
"major_version": 1,
"minor_version": 2,
…
Without `std::endl`, the difficulty gets smashed on the end of the hash.
It would try to join the auto refresh thread, which would
only happen after it was done, which would take a long time
when doing so on a newly created wallet.
8bc1bd6 wallet: use minimum mixin when RPC asks for too low mixin (moneromooo-monero)
31d2e0f wallet_rpc_server: make use_fork_rules public (moneromooo-monero)
After the fork, normal transfer functions called via RPC
use the minimum mixin 2 if 0 or 1 is requested. While the
incoming transaction may be valid (eg, it has an unmixable
and at most a mixable input), it is a simple way to make
sure RPC users can't get a seemingly random accept/reject
behavior if they don't update their requested mixin.
bdec7cb BlockchainLMDB: Use DB error helper consistently (warptangent)
c5932eb BlockchainLMDB: Add DB error to exception (warptangent)
a49c355 Blockchain: Omit verbose time stats messages by default (warptangent)
TEST:
blockchain_export -h
This should show "berkeley" as an available option to --database.
With an existing BerkeleyDB database, run:
blockchain_export --database berkeley
This is already the default for the daemon, but by checking a command
line argument and calling a Blockchain member function setter.
Initialize the variable to false so it's not dependent on an external
command-line argument check. This allows utilities like
blockchain_import to have a reasonable default without code changes.
c2a1fee simplewallet: prompt for private keys when generating wallets (moneromooo-monero)
4513b4c simplewallet: add a new --restore-from-keys option (moneromooo-monero)
11d555c Fix crash in std::map for connections_map (Howard Chu)
014f886 std::condvar is broken on Win32 with gcc/g++ 4.8 too (Howard Chu)
7c86c59 Use boost::thread instead of std::thread (Howard Chu)
57e75fa BlockchainBDB: Check if hard fork subdbs need reset (warptangent)
47f6cf8 BlockchainBDB: Support blockchain_import --drop-hard-fork command (warptangent)
27d4e50 core: check whether an update is needed straight away (moneromooo-monero)
8892173 core: print "update needed" hard fork notifications in red (moneromooo-monero)
Pass the CMake bit width setting to compile flags for blockchain_import
and blockchain_converter.
For LMDB on 32-bit, hyc has found that batch size of 100 appears to be a
good default.
3b13a74 Shutup about VERSION 0 (Howard Chu)
1537477 Use cursor in get_output_key (Howard Chu)
f2faf8c Use MDB_APPEND mode where possible (Howard Chu)
090b548 Use cursors in write txns (Howard Chu)
ed08d21 Keep a running blocksize count (Howard Chu)
0fc9334 Win32 import batchsize tweaks (Howard Chu)
27f76e2 blockchain_import: Build string for db type list (warptangent)
1aa8a9d blockchain_import: Add mode argument representing multiple DB flags (warptangent)
cffc411 blockchain_import: Support BerkeleyDB (warptangent)
19c1aaa blockchain_import: Add database type argument (warptangent)
0fedce0 blockchain_import: Make LMDB-specific names general (warptangent)
When keys are contiguous and monotonically increasing, this gets
denser page utilization (doesn't leave padding in page splits).
Can't be used for keys that are inserted in random order (e.g. hashes)
In total this only saves around 1.5% of space compared to original
DB code. The previous patch accounted for 0.8% savings on its own;
the blocks tables just aren't that big.
These modes match those optionally provided as part of the daemon's
--db-type argument.
Argument after the # is interpreted as a composite mode if there's only
one (no comma separated arguments).
Sample usage:
blockchain_import --database lmdb#fastest
blockchain_import --database berkeley#fastest
Multiple specific DB flags are still supported, e.g.
blockchain_import --database lmdb#nosync,nordahead
blockchain_import --database berkeley#txn_nosync
1995923 BlockchainLMDB: Deal with DB exceptions at block level with particularity (warptangent)
c16cc20 BlockchainLMDB: Add sanity check for inconsistent state (warptangent)
9118d0a BlockchainLMDB: Call destructor on allocated txn if setup fails (warptangent)
f5581c3 BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr (warptangent)
c7e6b77 crypto: only check MONERO_USE_SOFTWARE_AES once (moneromooo-monero)
74aef21 crypto: use software AES based on the MONERO_USE_SOFTWARE_AES env var (moneromooo-monero)
Add another DB error exception type to distinguish failed txn setup from
general use of txn.
This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
Remove LINK_SEARCH_START_STATIC and LINK_SEARCH_END_STATIC. This is more
appropriate when the compiler flag -static is used.
This had been causing CMake to omit the linker flags necesssary to
distinguish static and dynamic library linking. CMake had assumed static
linking for the target, causing it to omit explicit static link flags.
That is problematic without the -static compile flag being set.
With a library located in system directories, like libboost_date_time,
the full static path (.a), though found correctly by CMake, was treated
by the linker as a dynamic library. This is because
target_link_libraries() transforms the full path to -l<libname> if it's
in a system directory. Without -static or explicit linker flags, the
dynamic library (.so) is linked.
Removing the above two properties removes the assumption of static. So
-Wl;-Bstatic is inserted where needed. This causes -l<libname> to
properly refer to the static library instead of dynamic.
Setting to no or 0 also works. If set, any other value enables it.
Useful for running with valgrind in cases where it fails at
properly implementing AES-NI.
We also replace the --fakechain option with an optional structure
containing details about configuration for the core/blockchain,
for test purposes. This seems more future friendly.
7658ac0 blockchain: revert handle_get_objects adding block id on tx not found (moneromooo-monero)
3a0f4d8 berkeleydb: fix delete/free mismatch (moneromooo-monero)
1642be2 minor bugfixes and refactoring (Thomas Winget)
098dcf2 unit_tests: fix mnemonics unit test testing invalid seeds (moneromooo-monero)
If it is, it points to reuse of a tx key, which isn't meant to happen.
If it does, a key image collision means that only one of those
outputs is spendable, so the wallet selects the larger amount,
unless that output was spent already.
This causes a discrepancy betewen reported received inputs and
payment total.
Since tx keys are 256 bits, this should never happen except if
done on purpose, or if a sender uses a bad PRNG.
Locking just one db turns out to not have been a good idea, since
the pool and p2p state fdles have to be used anyway.
Also ensure the directory exists before tring to lock.
- Blockchain should store if it's running on testnet or not
- moved loading compiled-in block hashes to its own function for clarity
- on handle_get_objects, should now correctly return false if a block's
transactions are missing
- replace instances of BOOST_FOREACH with C++11 for loops in Blockchain.
7fc6fa3 wallet: forbid dust altogether in output selection where appropriate (moneromooo-monero)
5e1a739 blockchain: log number of outputs available for a new tx (moneromooo-monero)
bcac101 daemon: fix a few issues reported by valgrind (moneromooo-monero)
a7e8174 tx_pool: fix serialization of new relayed data (moneromooo-monero)
601ad76 hardfork: fix mixup in indexing variable in get_voting_info (moneromooo-monero)
444e22f blockchain: remove unused timer (moneromooo-monero)
7edfdd8 blockchain: fix m_sync_counter uninitialized variable use (moneromooo-monero)
d97582c epee: use generate_random_bytes for new random uuids (moneromooo-monero)
17c7c9c epee: remove dodgy random code that nobody uses (moneromooo-monero)
A boost lock is used to determine whether more than one process
wants to access the database. The boost file_lock doesn't seem
to like locking directories, so we use an arbitrary file in it.
This allows to still run two daemons if they have different
database directories (ie, LMDB/BDB, different data directories).
In particular, ensure we check the status of RPC response structures,
as some functions will return success, but with a BUSY status, when
the daemon is not yet synced, and the response will not filled.
94b98fb tx_pool: do not accept txes not in a block if they timed out before (moneromooo-monero)
eadbdf3 tx_pool: fix use of invalidated iterator (moneromooo-monero)
3b1d7e0 Fix V1/V2 use of hard fork related parameters (moneromooo-monero)
This is intended to avoid cases where a timed out tx will be
re-relayed by another peer for which it has not timed out yet,
which would cause the tx to stay in the network's pool for a
long time (until all peers time it out before another one
tries to relay it again).
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" db. This is most noticeable for
HDDs.
See ffcf6bdb95
Data should be removed in the reverse order it was added.
This matches the order of removal in
blockchain_storage::pop_transaction_from_global_index.
See f11def012f
c5baf30 use load_file_to_string and exit with error on file read errors (Fabian Hirschmann)
7b2d27f remove new lines at the end of the password only (Fabian Hirschmann)
302cc9c add --password-file option (Fabian Hirschmann)
The value will be different depending on whether we've reached
the first hard fork, which allows a larger size, or not.
This fixes transactions being rejected by the daemon on mainnet
where the first hard fork is not yet active.
This ensures this will be done without fail, as the error prone
matching of every return with a call to KILL_IOSERVICE leads to
hard to debug corruption when one is missing.
- only try to stop if actually started
- print number of threads before zeroing it
This fixes the suspiciously doubled "Mining has been stopped"
message on exit.
Blockchain hashes and key images are flushed, and blocks are
pulled anew from the daemon.
The console command is shortened to match bc_height.
This should make it a lot easier on users who are currently
told to remove this particular cache file but keep the keys
one, etc, etc.
61ce8d6 wallet_rpc_server: exit async, so we reply to stop_wallet RPC (moneromooo-monero)
9847db6 wallet: do not return error if incoming_transfers finds none (moneromooo-monero)
b39aae7 Tweak 45800a25e9 (hyc)
4a5a5ff blockchain: always stop the ioservice before returning (moneromooo-monero)
78b65cf db_lmdb: safety close db at exit (moneromooo-monero)
45800a2 db_lmdb: fix a strdup/delete[] mistmatch (moneromooo-monero)
79beed2 tests: fix various tests by using parameters better suited to monero (moneromooo-monero)
d0a8362 tests: fix some double spending tests (moneromooo-monero)
2358d0d tests: use 255 as a "too high" block version (moneromooo-monero)
f33a88c blockchain: fix a few block addition bugs (moneromooo-monero)
a9ff11c blockchain: fix an off by one error in unlocked time check (moneromooo-monero)
f294be3 blockchain: reinstate double spending checks in check_tx_inputs (moneromooo-monero)
737b6d6 blockchain: make some flag twiddling code closer to the original (moneromooo-monero)
81cb0fc blockchain: fix bitflipping test with quantized block rewards (moneromooo-monero)
22ddf09 blockchain: add missing m_tx_pool.on_blockchain_dec (moneromooo-monero)
d837c0c blockchain: fix switch to alternative blockchain for more than one block (moneromooo-monero)
5cec076 blockchain: add a missing validity check to rollback_blockchain_switching (moneromooo-monero)
3cabdb5 core: catch exceptions from get_output_key (moneromooo-monero)
5eef645 db: throw when given a non txout_to_key output to add (moneromooo-monero)
If the block reward was too high, the verification failed flag
was set, but the function continued. The code which was supposed
to trap this flag and return failure failed to trap it, and,
while the block was not added to the chain, the function would
return success.
The reason for avoiding returning when the block reward problem
was detected was to be able to return any transactions to the
pool if needed. This is now mooted by moving the transaction
return code to a separate function, which is now called at all
appropriate points, making the logic much simpler, and hopefully
correct now.
We also move the hard fork version check after the prev_id check,
as block which does not go on the top of the chain might not
have the expected version there, without being invalid just for
this reason.
Last, we trap the case where a block fails to be added due to
using already spent key images, to set the verification failed
flag.
This fixes some double spending tests.
This may or may not be unneeded in normal (non test) circumstances,
to be determined later. Keeping these for now may be slower, but safer.
Block reward may now be less than the full amount allowed.
This was breaking the bitflipping test.
We now keep track of whether a block which was accepted by the core
has a lower than allowed block reward, and allow this in the test.
The check was explicit in the original version, so it seems
safer to make it explicit here, especially as it is now done
implicitely in a different place, away from the original check.
ffcf6bd BlockchainLMDB: When removing, find amount output index fast by starting at end (warptangent)
f11def0 BlockchainLMDB: Remove tx outputs in reverse order (warptangent)
Data should be removed in the reverse order it was added. Not doing so
breaks assumptions and can cause problems in other DB implementations.
This matches the order of tx removal in
blockchain_storage::purge_block_data_from_blockchain.
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" subdb. This is most noticeable for
HDDs.
As before, the dup list is still walked if necessary (but in reverse),
and the global output index still confirmed to be the one looked for.
But under proper use, the result will be found at the end of the dup
list, so we start there.
Removing an amount output index is always done in the context of popping
a block, so the global output index being looked for should be the last
one in that amount key's dup list. Even if the txs themselves aren't
removed in reverse order (supposed to be according to original
implementation), the specified amount output index will still be near
the end, because the txs are in the same block.
TEST:
Pop blocks with blockchain_import.
Blocks should be successfully removed with no errors shown.
bitmonerod should be able to start syncing from the reduced blockchain
height.
8ea7af1 Allow the wallet to access hard fork information (moneromooo-monero)
760331b epee: make log macros behave like statements (moneromooo-monero)
3f2970f Add missing semicolons after log statements (moneromooo-monero)
In particular, <boost/program_options.hpp> blows up daemon.cpp.obj,
making it too big to compile in debug mode on Win32. Even on a
release build it drops daemon.cpp.o on Linux from 31MB to 20MB.
This has no effect on the final linked binary size.
The version number passed to those data's serialize function
was always 0, not the wallet's version as I had expected.
A version number now exists for these structures so they're
versioned correctly.
9079a32 Fix typo (warptangent)
3796941 blockchain.cpp: Change indentation from 4 to 2 spaces (warptangent)
725acc7 Replace tabs with two spaces for consistency with rest of codebase (warptangent)
Sample use:
DNS_PUBLIC=tcp torsocks bin/bitmonerod --p2p-bind-ip 127.0.0.1
Test:
Run above with --log-level 4 with and without DNS_PUBLIC environment
variable set.
DNS debugging info should show successful DNS lookups only when
DNS_PUBLIC is set to "tcp":
DNS lookup for seeds.moneroseeds.se: 17 results
DNS lookup for seeds.moneroseeds.ae.org: 17 results
DNS lookup for seeds.moneroseeds.ch: 12 results
DNS lookup for seeds.moneroseeds.li: 12 results
cbded43 core_tests: fix ring_signature_1 tests (moneromooo-monero)
c3d208f core_tests: bump default test fee to 0.02 monero (moneromooo-monero)
10da0a0 add a --fakechain argument for tests (moneromooo-monero)
eee44e6 unit_tests: fix block reward test using post hard fork settings (moneromooo-monero)
595893f blockchain: log block (not chain) height in "BLOCK SUCCESFULLY ADDED" (moneromooo-monero)
2369968 blockchain: fix off by one in get_blocks (moneromooo-monero)
8af913a db_lmdb: implement BlockchainLMDB::reset (moneromooo-monero)
4833f4f db_bdb: implement BlockchainBDB::reset (moneromooo-monero)
18bf06e tx_pool: fix "minumim" typo in message (moneromooo-monero)
44f1267 tests: fix a typo in test name (moneromooo-monero)
1494557 db_lmdb: create all needed directories, not just the leaf one (moneromooo-monero)
015b68a db_bdb: create all needed directories, not just the leaf one (moneromooo-monero)
f141869 tests: remove data-dir argument registration (moneromooo-monero)
This prevents the intermediate thread from exiting properly, as
fork creates a child process with only one thread, so any existing
data_logger thread will not be in the child. Since this thread
sets a flag the data_logger dtor blocks on, all children threads
will hang on exit.
When throwing an exception from being unable to begin an LMDB
transaction, include the reason.
It's often been due to a write transaction attempted within a write
transaction (batch mode), but there can be other reasons such as write
transaction attempted while database was opened read only, or
environment's map needs to be resized.
The core tests use the blockchain, and reset it to be able
to add test data to it. This does not play nice with the
databases, since those will save that data without an explicit
save call.
We add a fakechain flag that the tests will set, which tells
the core and blockchain code to use a separate database, as
well as skip a few things like checkpoints and fixup, which
only make sense for real data.
Data is only guaranteed to be valid within the lifetime of a txn.
You cannot use data returned from LMDB after the txn ends.
Also, fixed a missing txn.commit BlockchainLMDB::get_tx_unlock_time()
This fixes coretests, which does not register daemon specific arguments,
but uses core, which uses those arguments. Also gets rid of an unwanted
dependency on daemon code from core.
This is a precaution for older Berkeley DB versions.
- smooth reports an issue running with 4.7:
DB_ENV->log_set_config: DB_LOG_IN_MEMORY: method not permitted
after handle's open method
- this works just fine with 5.3
- we do not use DB_LOG_IN_MEMORY, but we use DB_LOG_AUTO_REMOVE
- libdb docs say some flags must be set before open, and some
may be set at any time, but never say some must be set after
open
- moving the call to log_set_config before open works with 5.3
Therefore, it seems best to move the call before open.
Early DB versions did not store key images for inputs if the
transaction spending them had no outputs (ie, all fee). This
is not correct, as this would allow these outputs to be double
spent. This was fixed in 533acc30ed
a few months ago, but databases having synced blocks 2021612 and
685498 with a faulty version will be missing those key images
in the spent keys database. This code checks for this, and adds
those key images if they are missing.
Berkeley DB uses 1 based indices for RECNO databases, and the
implementation of BlockchainDB for Berkeley DB assumes 1 based
indices are passed to the API, whereas the LMDB one assumes
0 based indices. This is all internally consisteny, but since
the BDB code stores 1 based indices in the database, external
users have to be aware of this, as the indices will be off by
one depending on which DB is used.
^C while in manual refresh will cancel the refresh, since that's
often an annoying thing to have to wait for. Also, a manual refresh
command will interrupt any running background refresh and take
over, rather than wait for the background refresh to be done, and
look to be hanging.
9ee48e9 wallet2: speed up wallet refresh for large miners (moneromooo-monero)
4905903 wallet2: parallelize pulling blocks and processing them on refresh (moneromooo-monero)
d0eaf1d wallet2: maintain the short chain manually when refreshing (moneromooo-monero)
a4e9506 wallet2: split pull blocks between pulling and processing (moneromooo-monero)
The daemon will be polled every 90 seconds for new blocks.
It is enabled by default, and can be turned on/off with
set auto-refresh 1 and set auto-refresh 0 in the wallet.
This allows them to be saved as a fixed (one byte) chunk whatever
the value. Using a varint will use two bytes as the high bit gets
set.
This is backward compatible with current usage (0-2 values).
d887c18 hardfork: fix more major/minor issues (moneromooo-monero)
3b47ca2 hardfork: fix rescan on load (moneromooo-monero)
4cea2b1 Add IP blocking for misbehaving nodes (adapted from Boolberry) (Javier Smooth)
9c64b12 quiet down p2p logging a bit (Javier Smooth)
53c75ab blockchain: log versions as numbers, not characters (moneromooo-monero)
edade8d hardfork: fix actual/voting confusion (moneromooo-monero)
9b945f5 wallet: make the refresh optimizations selectable via command line (moneromooo-monero)
d2c0313 wallet: speedup refresh from daemon (moneromooo-monero)
ea707c7 wallet2: minor cleanup (moneromooo-monero)
55a2da7 wallet2: speedup refresh a bit (moneromooo-monero)
Also add some more tests, and rename some instances of
"version" and "add" for clarity.
NOTE: the starting height values are sometimes wrong.
I suspect this is due to the hard fork reorg code being
buggy, since they're good when syncing after the fact.
However, they're not actually used by the consensus code,
so I'm ignoring this for now, but this needs debugging.
