Commit Graph

346 Commits

Author SHA1 Message Date
luigi1111
46bcd91db4
Merge pull request #4256
d6fc870 db_lmdb: resize blockchain database when 90% filled (moneromooo-monero)
2018-09-10 15:04:50 -05:00
luigi1111
77ed11e627
Merge pull request #4270
29dea03 epee: resize vectors where possible in serialization (moneromooo-monero)
76affd9 epee: some speedup in parsing (moneromooo-monero)
dc6c069 db_lmdb: speedup the get_output_distribution common case (moneromooo-monero)
76ac5a8 wallet2: ask for a binary output distribution, for speed (moneromooo-monero)
2018-09-04 13:19:58 -05:00
p8p
9d6539923e
is_hdd update 2018-08-25 04:31:22 -07:00
luigi1111
4a4ea78ecd
Merge pull request #4204
b278b83 core: sync database based on bytes added, not blocks added (moneromooo-monero)
2018-08-22 20:43:51 -05:00
luigi1111
7e4507a048
Merge pull request #4200
43f7110 blockchain_db: remove unused get_output_key variant (moneromooo-monero)
2018-08-22 20:40:37 -05:00
moneromooo-monero
dc6c0696fd
db_lmdb: speedup the get_output_distribution common case 2018-08-19 15:58:21 +00:00
luigi1111
2530dc6710
Merge pull request #4161
be02eb9 db_lmdb: demote the 'batch already enabled' log, it's harmless (moneromooo-monero)
2018-08-15 17:27:48 -05:00
moneromooo-monero
d6fc870d04
db_lmdb: resize blockchain database when 90% filled
instead of a random ratio from 60% to 90%.
2018-08-14 08:55:49 +00:00
moneromooo-monero
b278b83860
core: sync database based on bytes added, not blocks added
Blocks have a very wide range, whereas actual size is the relevant
quantity to consider when syncing
2018-08-12 16:13:46 +00:00
moneromooo-monero
43f71100bc
blockchain_db: remove unused get_output_key variant
It was actually incorrect, as it would not return commitment
2018-07-30 21:57:47 +00:00
luigi1111
0dddfeacc9
Merge pull request #4129
ee31383 db_lmdb: don't sync a read only DB (moneromooo-monero)
2018-07-27 14:30:57 -05:00
moneromooo-monero
be02eb9389
db_lmdb: demote the "batch already enabled" log, it's harmless 2018-07-20 23:41:40 +01:00
luigi1111
3e026ff6ed
Merge pull request #4013
e5592c4 rpc: add blockchain disk size to getinfo (moneromooo-monero)
2018-07-19 13:40:42 -05:00
luigi1111
21afa71ba6
Merge pull request #3981
45e419b db: store cumulative rct output distribution in the db for speed (moneromooo-monero)
2018-07-19 13:39:24 -05:00
luigi1111
025187e6c9
Merge pull request #3854
149da42 db_lmdb: enable batch transactions by default (stoffu)
34cb6b4 add --regtest and --fixed-difficulty for regression testing (vicsn)
9e1403e update get_info RPC and bump RPC version (vicsn)
207b66e first new functional tests (vicsn)
2018-07-19 13:35:25 -05:00
moneromooo-monero
45e419bd5c
db: store cumulative rct output distribution in the db for speed
This gets rid of the temporary precalc cache.

Also make the RPC able to send data back in binary or JSON,
since there can be a lot of data

This bumps the LMDB database format to v3, with migration.
2018-07-13 11:37:04 +01:00
moneromooo-monero
ee31383a52
db_lmdb: don't sync a read only DB
This would only throw
2018-07-11 18:14:22 +01:00
victorsintnicolaas
34cb6b4b70 add --regtest and --fixed-difficulty for regression testing
on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest.
Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain
Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'.
Queries hard fork heights info of other network types
2018-06-29 10:06:11 +02:00
luigi1111
cbfa0a2a7e
Merge pull request #3987
2d10830 db_ldmb: print db version when complaining about incompatibility (moneromooo-monero)
2018-06-27 16:23:17 -05:00
luigi1111
c3ec5373b3
Merge pull request #3936
d81e042 tx_pool: initialize bitflags padding since it gets written to storage (moneromooo-monero)
2018-06-25 14:53:27 -05:00
moneromooo-monero
e5592c4bab
rpc: add blockchain disk size to getinfo
This should help new nodes predict how much disk space will be
needed for a full sync
2018-06-20 23:48:37 +01:00
stoffu
149da420e9 db_lmdb: enable batch transactions by default 2018-06-14 20:56:11 +02:00
moneromooo-monero
2d10830cfb
db_ldmb: print db version when complaining about incompatibility 2018-06-11 17:57:02 +01:00
moneromooo-monero
284fe6438d
db_lmdb: warn about slowness when running off a spinning disk 2018-06-08 20:59:02 +01:00
moneromooo-monero
d81e042306
tx_pool: initialize bitflags padding since it gets written to storage
Avoids valgrind reporting uninitialized data usage
2018-06-05 20:26:44 +01:00
luigi1111
a7ea14dc6a
Merge pull request #3876
740da1b core: fix automatic safe db sync mode switching (moneromooo-monero)
e942d34 protocol: do not switch to unsafe sync mode for just a few blocks (moneromooo-monero)
2018-05-30 15:46:13 -05:00
moneromooo-monero
740da1ba95
core: fix automatic safe db sync mode switching 2018-05-28 12:14:04 +01:00
moneromooo-monero
b9389e582e
db_lmdb: save pruned and prunable tx data separately
This bumps DB version to 2, migration code will run for v1 DBs
2018-05-23 22:48:12 +01:00
moneromooo-monero
5f8ffca34d
speed up get_output_distribution (and precalc common case) 2018-05-21 19:51:58 +01:00
moneromooo-monero
ffeeefde60
speedup get_output_histogram for all amounts when min_count > 0
This skips the vast majority of "dust" output amounts with just
one instance on the chain. Clocks in at 0.15% of the original
time on testnet.
2018-04-23 10:54:28 +01:00
moneromooo-monero
41f727ce42
add RPC to get a histogram of outputs of a given amount 2018-03-16 10:32:26 +00:00
Riccardo Spagni
0e5c006207
Merge pull request #3307
223d7d0c db_lmdb: fix free space reporting (moneromooo-monero)
2018-03-05 19:11:52 +02:00
Edward Betts
fbcc91c2a4 Correct spelling mistakes. 2018-03-05 17:00:40 +00:00
moneromooo-monero
223d7d0c7a
db_lmdb: fix free space reporting
reported by Brad Richards
2018-02-23 09:06:10 +00:00
moneromooo-monero
6514f0ed7b
db_lmdb: fix return code mixup in for_all_* 2018-02-19 14:55:58 +00:00
Riccardo Spagni
f4a6bc79d9
Merge pull request #3226
e4646379 keccak: fix mdlen bounds sanity checking (moneromooo-monero)
2e3e90ac pass large parameters by const ref, not value (moneromooo-monero)
61defd89 blockchain: sanity check number of precomputed hash of hash blocks (moneromooo-monero)
9af6b2d1 ringct: fix infinite loop in unused h2b function (moneromooo-monero)
8cea8d0c simplewallet: double check a new multisig wallet is multisig (moneromooo-monero)
9b98a6ac threadpool: catch exceptions in dtor, to avoid terminate (moneromooo-monero)
24803ed9 blockchain_export: fix buffer overflow in exporter (moneromooo-monero)
f3f7da62 perf_timer: rewrite to make it clear there is no division by zero (moneromooo-monero)
c6ea3df0 performance_tests: remove add_arg call stray extra param (moneromooo-monero)
fa6b4566 fuzz_tests: fix an uninitialized var in setup (moneromooo-monero)
03887f11 keccak: fix sanity check bounds test (moneromooo-monero)
ad11db91 blockchain_db: initialize m_open in base class ctor (moneromooo-monero)
bece67f9 miner: restore std::cout precision after modification (moneromooo-monero)
1aabd14c db_lmdb: check hard fork info drop succeeded (moneromooo-monero)
2018-02-16 14:26:58 +01:00
Riccardo Spagni
260d666cf4
Merge pull request #3231
84a8b2da Don't create readtxn until after txn_safe gate check (Howard Chu)
2018-02-16 14:24:01 +01:00
moneromooo-monero
ad11db9144
blockchain_db: initialize m_open in base class ctor
It's cleaner this way, since it's a base class field

Coverity 136568
2018-02-02 15:21:39 +00:00
moneromooo-monero
1aabd14c21
db_lmdb: check hard fork info drop succeeded
Coverity 136364
2018-02-02 15:08:22 +00:00
Howard Chu
84a8b2da14
Don't create readtxn until after txn_safe gate check 2018-01-31 18:18:01 +00:00
xmr-eric
18216f19dd Update 2018 copyright 2018-01-26 10:03:20 -05:00
Riccardo Spagni
0a110aefbb
Merge pull request #2971
ae860230 Fix exceptions not finding txpool txes when relaying (moneromooo-monero)
2018-01-10 11:45:15 +01:00
Howard Chu
294adc8341
Additional fix for core_tests
Reset thread-local info if it doesn't match the current env.
Only happens when a process opens/closes env multiple times in the
same process, doesn't affect monerod.
2017-12-28 20:24:08 +00:00
Howard Chu
d52b732efb
Fix stale readcursor flags
Reset thread-specific flags when a write txn is started.
Also remove some redundant start-readtxn code.
2017-12-28 00:59:04 +00:00
moneromooo-monero
ae8602303a
Fix exceptions not finding txpool txes when relaying 2017-12-20 13:02:53 +00:00
moneromooo-monero
a4240d9ffc
catch const exceptions 2017-12-18 15:15:43 +00:00
moneromooo-monero
d753d716a6
fix a few leaks by throwing objects, not newed pointers to objects 2017-12-18 15:15:13 +00:00
moneromooo-monero
09ce03d612
move includes around to lessen overall load 2017-12-16 22:46:38 +00:00
flozilla
f9fad18608
blockchain_db: sanity check on tx/hash vector sizes
It could trip on a corrupt/crafted file if the user has disabled
input verification.
2017-11-15 10:02:23 +00:00
Riccardo Spagni
5de492be40
Merge pull request #2629
383ff4f6 remove "using namespace std" from headers (moneromooo-monero)
2017-11-14 21:29:49 +02:00
moneromooo-monero
383ff4f689
remove "using namespace std" from headers
It's nasty, and actually breaks on Solaris, where if.h fails to
build due to:

  struct map *if_memmap;
2017-11-14 16:56:10 +00:00
Riccardo Spagni
0d0ce8c21e
Merge pull request #2742
7c7d3672 Increase LMDB maxreaders if large number of threads in use (Howard Chu)
6738753b Use max_concurrency as-is (Howard Chu)
2017-11-14 15:30:03 +02:00
Riccardo Spagni
fd0740e5e7
Merge pull request #2615
10013e94 Protect node privacy by proper filtering in restricted-mode RPC answers (binaryFate)
2017-11-14 14:53:10 +02:00
binaryFate
10013e9434 Protect node privacy by proper filtering in restricted-mode RPC answers
This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode.
This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected.
In practice, when running with `--restricted-rpc`:
* get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero.
* get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions
* get_transaction_pool_hashes.bin will not list such transaction
* get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed

The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality.
Fixes #2590.
2017-11-08 13:07:07 +01:00
moneromooo-monero
ccf53a566c
track double spending in the txpool
Transactions in the txpool are marked when another transaction
is seen double spending one or more of its inputs.
This is then exposed wherever appropriate.

Note that being marked with this "double spend seen" flag does
NOT mean this transaction IS a double spend and will never be
mined: it just means that the network has seen at least another
transaction spending at least one of the same inputs, so care
should be taken to wait for a few confirmations before acting
upon that transaction (ie, mostly of use for merchants wanting
to accept unconfirmed transactions).
2017-11-06 00:05:44 +00:00
Howard Chu
7c7d36725a
Increase LMDB maxreaders if large number of threads in use 2017-11-02 16:33:48 +00:00
ston1th
f946bb7be6 fixed an LMDB issue on OpenBSD (#2699) which leads to a crash in monerod 2017-10-26 17:16:06 +02:00
Riccardo Spagni
b293ad9a1a
Merge pull request #2640
22b51e06 db_lmdb: include chain height when failing to find an output key (moneromooo-monero)
5db433b3 blockchain: avoid exceptions in output verification (moneromooo-monero)
2017-10-15 18:59:45 +02:00
moneromooo-monero
22b51e0672
db_lmdb: include chain height when failing to find an output key
To help debugging logs.
2017-10-12 13:28:28 +01:00
moneromooo-monero
bc1b93333e
frob level 1 logs a bit for consistency
Level 1 logs map to INFO, so setting log level to 1 should
show these. Demote some stuff to DEBUG to avoid spam, though.
2017-10-07 20:03:42 +01:00
Riccardo Spagni
e5a2cdbfad
Merge pull request #2457
d2d8a41c Use actual batch size for resize estimates (Howard Chu)
2017-09-25 16:57:07 +02:00
moneromooo-monero
9aeae7a224
db_lmdb: fix use of uninitialized key in for_blocks_range 2017-09-19 14:11:24 +01:00
Howard Chu
d2d8a41c47
Use actual batch size for resize estimates
And optimize import startup:

Remember start_height position during initial count_blocks pass
to avoid having to reread entire file again to arrive at start_height
2017-09-17 22:08:40 +01:00
Riccardo Spagni
1e57e48342
Merge pull request #2372
c867357a cryptonote_protocol: error handling on cleanup_handle_incoming_blocks (moneromooo-monero)
ce901fcb Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks (moneromooo-monero)
84fa015e core: guard against exceptions in handle_incoming_{block,tx} (moneromooo-monero)
2017-09-02 11:30:57 +02:00
moneromooo-monero
ce901fcb31
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks 2017-08-29 15:43:32 +01:00
moneromooo-monero
a95e460c71
move db specific options to BlockchainDB
Avoids common depending on blockchain_db, which can cause
link errors.
2017-08-29 11:43:34 +01:00
Riccardo Spagni
36b978f50e
Merge pull request #2314
c22d22e2 Cleanup test impact of adding safesyncmode() method (Howard Chu)
9a859844 Toggle SAFE syncmode on and off automatically (Howard Chu)
2017-08-26 23:43:22 +02:00
Howard Chu
9a859844f4
Toggle SAFE syncmode on and off automatically
If monerod is started with default sync mode, set it to SAFE after
synchronization completes. Set it back to FAST if synchronization
restarts (e.g. because another peer has a longer blockchain).

If monerod is started with an explicit sync mode, none of this
automation takes effect.
2017-08-20 16:30:28 +01:00
Howard Chu
80344740bd
More DB support cleanup
Hide DB types from db_types.h - no reason to recompile dependencies
when DB types change.

Also remove lingering in-memory DB references, they've been
obsolete since 9e82b694da
2017-08-20 13:57:36 +01:00
Howard Chu
4c7f8ac04f
DB cleanup
Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp
should ever be including DB-specific headers any more.
2017-08-19 18:11:38 +01:00
moneromooo-monero
c6e200a8ab
core: add --db-salvage command line flag
Use to load the database when the primary meta page is corrupted
2017-08-12 19:04:44 +01:00
moneromooo-monero
214fd81e93
some include cleanup 2017-07-31 16:36:52 +01:00
moneromooo-monero
235df7f484
blockchain_db: add a txpool tx getter which returns existence
Avoids exception spam for the "nope, not found" case
2017-06-11 15:36:48 +01:00
Howard Chu
d17c0fc2d0
Don't copy blockchain for coinbase_tx_sum
Changed Blockchain::for_all_blocks() to for_blocks_range()
Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:14:24 +01:00
Riccardo Spagni
a4c13ea092
Merge pull request #2039
4b932ff3 changed crypto to cncrypto so it generated libcncrypto (Gentian)
2017-05-30 21:24:11 +02:00
moneromooo-monero
b52abd1370
Move txpool to the database
Integration could go further (ie, return_tx_to_pool calls should
not be needed anymore, possibly other things).

poolstate.bin is now obsolete.
2017-05-25 22:23:37 +01:00
Gentian
4b932ff314 changed crypto to cncrypto so it generated libcncrypto
fix a cmakelist
2017-05-23 07:45:40 -04:00
Howard Chu
d3aaf74080
Partial fix #1876, check thread in block_txn_start()
Don't allow use of existing batch txn if it's from the wrong thread
2017-03-19 12:27:03 +00:00
kenshi84
7d07c64fe5
fix dependency: put HardFork back to cryptonote_basic, made some BlockchainDB functions virtual again to avoid missing symbols error 2017-03-10 11:22:39 +09:00
Howard Chu
eb62dcc871
Remove redundant num_txs() method
Cleanup of bf1348b7e2
2017-03-03 16:52:45 +00:00
moneromooo-monero
12adb4a3f3
core: move hardfork back to cryptonote_core
should fix a cross dependency betewen cryptonote_basic and
blockchain_db
2017-02-25 16:41:35 +00:00
Riccardo Spagni
c3599fa7b9
update copyright year, fix occasional lack of newline at line end 2017-02-21 19:38:18 +02:00
Riccardo Spagni
49efd3add9
Merge pull request #1727
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
2017-02-21 11:27:15 +02:00
Riccardo Spagni
59d96eedd8
Merge pull request #1724
cca95c1c blockchain_db: do not throw on expected partial results getting keys (moneromooo-monero)
2017-02-21 11:23:03 +02:00
Howard Chu
2e9136767e
Handle map resizes from other processes 2017-02-18 22:22:24 +00:00
Howard Chu
bf1348b7e2
Can't cache num_txs or num_outputs either
Same reason as 3ff54bdd7a
2017-02-18 21:01:24 +00:00
Howard Chu
dc53e9eef2
Add a few read txns to streamline
Slight perf gain, but mainly to reduce spam at loglevel 3
2017-02-18 21:01:24 +00:00
moneromooo-monero
0288310e3b
blockchain_db: add "raw" blobdata getters for block and transaction
This speeds up operations such as serving blocks to syncing peers
2017-02-13 21:11:37 +00:00
moneromooo-monero
cca95c1c7a
blockchain_db: do not throw on expected partial results getting keys
When scanning for outputs used in a set of incoming blocks,
we expect that some of the inputs in their transactions will
not be found in the blockchain, as they could be in previous
blocks in that set. Those outputs will be scanned there at
a later point. In this case, we add a flag to control wehther
an output not being found is expected or not.
2017-02-13 19:05:30 +00:00
moneromooo-monero
f2986ccfc1
db_lmdb: fix bad height saved in tx data
The recent change to not keep separate track of the blockchain
height caused the reported height to jump early in the lmdb
transaction (when the block data is added to the blocks table),
rather than at the end, after everything succeeded. Since the
block data is added before the transaction data, this caused
the transaction data to be saved with a height one more than
its expected value.

Fix this by saving the block data last. This should have no
side effects.
2017-02-11 10:16:18 +00:00
kenshi84
8027ce0c75 extract some basic code from libcryptonote_core into libcryptonote_basic 2017-02-08 22:45:15 +09:00
Miguel Herranz
95f3e193f0 Update and use blockchain data files defines 2017-01-28 15:55:14 +01:00
Howard Chu
843769f88f
Tweak some msg loglevels 2017-01-16 20:42:59 +00:00
Howard Chu
990e08f090
Fix PR#1506, off by one in chain height 2017-01-16 10:00:06 +00:00
moneromooo-monero
5833d66f65
Change logging to easylogging++
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.

To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:

This one is (mostly) silent, only outputting fatal errors:

MONERO_LOGS=*:FATAL

This one is very verbose:

MONERO_LOGS=*:TRACE

This one is totally silent (logwise):

MONERO_LOGS=""

This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):

MONERO_LOGS=*:WARNING,verify:FATAL

Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE

Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:

MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE

Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.

Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.

The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
2017-01-16 00:25:46 +00:00
Riccardo Spagni
65e33b1bc5
Merge pull request #1506
3ff54bdd Check for correct thread before ending batch transaction (Howard Chu)
eaf8470b Must wait for previous batch to finish before starting new one (Howard Chu)
c903c554 Don't cache block height, always get from DB (Howard Chu)
eb1fb601 Tweak default db-sync-mode to fast:async:1 (Howard Chu)
0693cff9 Use batch transactions when syncing (Howard Chu)
2017-01-15 14:43:12 -05:00
Howard Chu
3ff54bdd7a
Check for correct thread before ending batch transaction 2017-01-15 15:51:57 +00:00
Howard Chu
c903c5541e
Don't cache block height, always get from DB 2017-01-14 22:43:06 +00:00
Howard Chu
0693cff925
Use batch transactions when syncing
Faster throughput while avoiding corruption. I.e., makes
running with --db-sync-mode safe more tolerable.
2017-01-14 22:43:06 +00:00
Riccardo Spagni
006bb0e957
Merge pull request #1538
aff28178 Remove db-auto-remove-logs (Miguel Herranz)
1229c685 Remove berkeley from db_type initialization (Miguel Herranz)
e3090558 Show available types for db-type command (Miguel Herranz)
046ab33d Remove berkeley from blockchain_db_types (Miguel Herranz)
2017-01-13 14:33:59 -05:00