Commit Graph

42 Commits

Author SHA1 Message Date
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
5d304cabfd Fix loop bug when calling core::get_block_template, causing calling thread to lock up. 2015-07-10 22:09:21 -07:00
Thomas Winget
b1d92bcc37
Fixes changes to sort tx by fee per kb 2015-05-13 20:27:06 -04:00
Riccardo Spagni
012164fff8
resolved merge conflict in tx_pool.cpp 2015-05-13 11:18:22 +02:00
Thomas Winget
385d7c0495
Sort txs by per-kb-fee for miners 2015-04-30 01:02:12 -04:00
Thomas Winget
1b2614ba83
When removing 'stuck' transactions, don't ignore the first tx in the pool 2015-04-30 00:23:00 -04:00
warptangent
641d824f37
Keep memory pool consistent when stuck tx removed
When a stuck tx is removed from memory pool, first remove the associated
spent key images.
2015-04-23 07:04:36 -07:00
warptangent
b76857f9d9
Add mempool output to daemon via command and RPC
This is for the "print_pool" command and "get_transaction_pool" RPC
method.

Add mempool's spent key images to the results.
2015-04-23 07:04:36 -07:00
warptangent
84fe5fbd65
Add compile-time support for both db implementations: in-memory and LMDB
Usage:

default is lmdb for blockchain branch:
$ make release

same as:
$ DATABASE=lmdb make release

for original in-memory implementation:
$ DATABASE=memory make release
2015-02-02 11:53:09 -08:00
Thomas Winget
74a1a89e27
Integrate BlockchainDB into cryptonote_core
Probably needs more looking at -- lot of things were done...in a rushed
sort of way.  That said, it all builds and *should* be at least
testable.

update for rebase (warptangent 2015-01-04)
  fix conflicts with upstream CMakeLists.txt files

  src/CMakeLists.txt (remove edits from original commit)
  tests/CMakeLists.txt (remove edits from original commit)
  src/cryptonote_core/CMakeLists.txt (edit)
  - use blockchain db .cpp and .h files
  - add LMDB_LIBRARIES
2015-01-04 19:29:51 -08:00
Riccardo Spagni
f4b69d553a
year updated in license 2015-01-02 18:52:46 +02:00
iamsmooth
dde7897fd4 Disable legacy fees for now 2014-11-11 02:03:00 +00:00
iamsmooth
cc74b43651 Remove DEFAULT_FEE, add temporary acceptance of too-small per-kb fee >= 0.1, denominations based on DEFAULT_DUST_THRESHOLD, document fee arg to create_transactions as unused, se DEFAULT_DUST_THRESHOLD for wallet dust collection instead of calcualted tx fee 2014-11-09 10:53:29 +00:00
Thomas Winget
557e27fd21 per kb fees 2014-11-06 22:09:07 -07:00
Riccardo Spagni
59a8366bb1 Revert "low risk, potentially varint overflow bug patched thanks to BBR"
This reverts commit 4e2b2b942d.
2014-09-25 08:24:42 +02:00
Riccardo Spagni
4e2b2b942d low risk, potentially varint overflow bug patched thanks to BBR 2014-09-24 22:17:33 -04:00
fluffypony
4ba680f294 a few more error messages moved to log level 1 2014-09-09 12:28:16 +02:00
fluffypony
cf5a8b1d6c moved non-critical warnings and errors to log level 1 2014-09-09 11:32:00 +02:00
Riccardo Spagni
d1b0ee7ecd Merge pull request #64 from mikezackles/bytecoin_tx_pool_tmp
tx pool fixes, courtesy of Bytecoin
2014-08-01 17:31:15 +02:00
fluffypony
6fc995fe5d License updated to BSD 3-clause 2014-07-23 15:03:52 +02:00
Zachary Michaels
61e447f7c8 Remove dead code 2014-07-17 19:18:24 -04:00
Zachary Michaels
e48cf2a3a9 Remove second tx size check 2014-07-17 18:47:18 -04:00
Zachary Michaels
23c914ccfe Don't check max tx size when adding to block
This is now done when adding transactions to the pool.
2014-07-17 17:11:21 -04:00
Zachary Michaels
e3b8c58496 Don't check for min fee when adding tx to block
This is now done when transactions enter the pool.
2014-07-17 17:10:35 -04:00
Zachary Michaels
232e23e90f Decrease max block size from 200% median to 130% 2014-07-17 16:56:31 -04:00
Zachary Michaels
c017bb0656 Use print_money in log 2014-07-17 16:56:28 -04:00
Zachary Michaels
d9f3421ce7 Minimum tx fee for entering pool 2014-07-17 16:55:40 -04:00
Zachary Michaels
8e99cee062 Start fresh if tx_pool deserialize fails 2014-07-17 16:53:48 -04:00
Zachary Michaels
a3f5764869 Add transaction size limit
Transactions larger than the limit will not enter the pool
2014-07-17 16:53:48 -04:00
Zachary Michaels
4d25350a82 Use const where appropriate in tx_pool 2014-07-17 16:31:31 -04:00
mydesktop
eba62452c0 set max block size for gbt 2014-06-26 16:42:24 -04:00
monero-project
2a2b36d425 fix function typo 2014-06-24 13:30:41 -04:00
monero-project
c050ff43bf Fix copy/paste bug, thx Boolberry 2014-06-24 13:20:15 -04:00
fluffypony
3bc16dc0e6 proper tx_pool handling from CryptoZoidberg / BBR 2014-06-15 09:48:13 +02:00
monero-project
2475ec877f typo in tx_pool.cpp 2014-05-28 13:19:24 -04:00
mydesktop
9761529941 fix typing mismatch 2014-05-27 16:45:50 -04:00
mydesktop
328a52a543 dynamic tx size scaling for tx mempool 2014-05-26 20:57:50 -04:00
mydesktop
f545fd8ff0 maximum block size 130% of median 2014-05-26 19:51:22 -04:00
mydesktop
31a59785b0 temporary fix for block reward dos 2014-05-25 20:25:37 -04:00
thankful_for_today
0fd82c910b mining bug fixed 2014-04-20 00:53:40 +04:00
Antonio Juarez
29c2859a3e json rpc for wallet and bugfix 2014-04-02 17:00:17 +01:00
Antonio Juarez
296ae46ed8 moved all stuff to github 2014-03-03 22:07:58 +00:00