Commit Graph

171 Commits

Author SHA1 Message Date
Riccardo Spagni
e6ab2dfdf3
updated in-source lmdb 2015-07-16 11:33:10 +02:00
Thomas Winget
ea08c768e2 open() flag O_DSYNC isn't on BSD, use O_SYNC
If the detected OS is FreeBSD, tell LMDB to compile with
MDB_DSYNC=O_SYNC instead of the default O_DSYNC, as BSD does not
implement this flag.
2015-07-16 11:22:39 +02:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
Riccardo Spagni
0f4a036437
re-add Windows DNS bug fix, per c0de96f8bd 2015-06-02 11:33:04 +02:00
Riccardo Spagni
6a1190792b
update libunbound 2015-05-31 16:36:48 +02:00
Thomas Winget
bed2331904
Require BerkeleyDB to be installed (for now) if building non-static 2015-04-22 01:51:14 -04:00
Riccardo Spagni
41f0a8fe4d
update lmdb64 2015-04-14 21:07:39 +02:00
Thomas Winget
a8bc7182ea
Merge BlockchainDB into upstream 2015-04-07 17:56:18 -04:00
Thomas Winget
9519526224
Only compile BerkeleyDB as an option in non-static 2015-04-07 15:02:20 -04:00
Riccardo Spagni
cbddb5f690
updated unbound cmake for static builds 2015-04-06 19:39:59 +02:00
Riccardo Spagni
7172bdab60
minor CMakeLists fix for Unbound, ldns -> sldns 2015-04-05 00:09:06 +02:00
Riccardo Spagni
8d4772fa43
reinstate accidentally removed CMakeLists 2015-04-04 23:56:12 +02:00
Riccardo Spagni
ea32a84e45
Merge pull request #251
c0de96f Fixed DNS resolution bug in Windows (Thomas Winget)
2015-04-02 16:31:13 +02:00
Thomas Winget
c0de96f8bd
Fixed DNS resolution bug in Windows
Due to a bug in unbound, we were passing a string containing a null
character to ub_ctx_resolvconf and ub_ctx_hosts rather than a NULL
pointer.  On *nix this wasn't causing headache, but on Windows this was
causing unbound to not correctly load DNS settings from the OS.

Note on the bug: in a Windows-specific code branch in the function
ub_ctx_hosts(), if the hosts file specified was a NULL pointer, a call
to getenv() was stored in a local char* and later freed.  This is
incorrect, as we do not own that data, and caused the program to crash.
2015-04-02 09:22:31 -04:00
Riccardo Spagni
1f49833d4f
update unbound from upstream 2015-04-02 11:16:18 +02:00
Thomas Winget
94cb295db4
Merge upstream into blockchain 2015-03-29 09:58:18 -04:00
meshpoint
c9424c1316 fix cmake miniupnpc typo
which matters on case-sensitive filesystems
2015-03-26 15:28:31 +04:00
Thomas Winget
8e3347f310
Pull blockchain changes into berkeleydb branch 2015-03-17 19:52:53 -04:00
Thomas Winget
6e9e8ab620
Move db_drivers/ to external/
Also change LMDB Cmake variables to CACHE rather than upgrading them
through several parent scopes.
2015-03-17 18:55:24 -04:00
Riccardo Spagni
3a3c07c330
fixed msys2 / mingw folders based on architecture, added license to unbound CMakeList as that is not part of standard Unbound 2015-03-17 18:26:38 -04:00
Thomas Winget
11e815761f
Revert "Moved db_drivers/ into external/ for consistency"
This reverts commit b21335642e.
2015-03-17 18:03:11 -04:00
Thomas Winget
daf2a8f6cb
Revert "Build fixed, goofed up some CMake"
This reverts commit 8b82f3c57f.
2015-03-17 18:01:25 -04:00
Thomas Winget
43477b7dac
BerkeleyDB Blockchain building, not working yet
Everything except actually *using* BlockchainBDB is wired up, but the db
itself is not yet working.  Some error about user mem not large enough.
I think I know what this error means, but I can't determine the cause.

Notes: BerkeleyDB does not allow 0-indexing in its recno type databases,
  so block numbers *in the database* will be 1-indexed.  Modifications
  to indexing have been made as needed.
2015-03-16 09:14:51 -04:00
Thomas Winget
cade0da8f1
CMake wiring, minor cleanup, minor test addition
Make Cmake things aware of BerkeleyDB and BlockchainBDB

Make the BlockchainDB unit tests aware of BlockchainBDB
2015-03-16 04:17:53 -04:00
Thomas Winget
8b82f3c57f
Build fixed, goofed up some CMake
Forgot that CMake vars set to PARENT_SCOPE will still vanish if that
parent scope goes...out of scope.  LMDB vars elevated one more scope to
compensate for moving db_drivers/ into external/
2015-03-10 12:51:23 -04:00
Thomas Winget
b21335642e
Moved db_drivers/ into external/ for consistency 2015-03-09 15:48:46 -04:00
Thomas Winget
767aac274b Remove unused dependency 2015-01-04 19:31:20 -08:00
Thomas Winget
90d6f8bf62 Adding libglim as an external library
libglim is an Apache-licensed C++ wrapper for lmdb, and rather than
rolling our own it seems prudent to use it.

Note: lmdb is not included in it, and unless something happens as did
with libunbound, should be installed via each OS' package manager or
equivalent.
2015-01-04 18:41:44 -08:00
Riccardo Spagni
f4b69d553a
year updated in license 2015-01-02 18:52:46 +02:00
Riccardo Spagni
831933425b
update unbound from upstream 2014-12-04 23:10:49 +02:00
Riccardo Spagni
affde29754
moved rapidjson to external folder, fixed CMake 2014-12-01 21:15:50 +02:00
Ben Boeckel
de4fc40576 mingw: copy required libraries to the build tree
These are found as shared libraries and need to be copied so that PATH
manipulation isn't necessary outside of an msys shell.
2014-11-18 17:03:14 -05:00
Ben Boeckel
d855fe4e89 miniupnpc: bump the _POSIX_C_SOURCE feature macro
FreeBSD doesn't expose IPv6 structures without this.
2014-11-10 11:13:15 -05:00
Ben Boeckel
c696492549 unbound: fix getaddrinfo detection for 32-bit windows
On Windows, getaddrinfo is part of the Windows API and as such is
__stdcall, not __cdecl, so check_function_exists fails because the
declaration doesn't match the mangling __stdcall has. Instead, use a
header to include the symbol as declared on the system and use
check_symbol_exists instead.

Tested-By: greatwolf on IRC
2014-10-24 17:40:52 -04:00
Ben Boeckel
4b6515cfb5 unbound: fix type checking 2014-10-24 15:30:44 -04:00
Ben Boeckel
d43a20f8f4 unbound: plumb the libdir up
This is necessary for static builds where the linking is passed around
to dependent targets, but the library is lost.
2014-10-24 15:29:56 -04:00
Ben Boeckel
7d708e4223 cmake: support 2.8.7
Older versions of CMake support LINK_{PUBLIC,PRIVATE} while newer
versions prefer PUBLIC and PRIVATE instead, but still support the LINK_
prefix.
2014-10-24 15:29:51 -04:00
Ben Boeckel
031e3da724 cmake: remove scream-make (all-caps functions) 2014-10-23 16:42:35 -04:00
Ben Boeckel
abbd5c0bdb unbound: import cmake build system 2014-10-23 16:42:35 -04:00
Ben Boeckel
e59b5b7078 miniupnpc: clean up build system
Changes:

  - remove amiga and solaris conditionals
  - remove -fPIC hackery (use POSITION_INDEPENDENT_CODE instead)
  - remove tests
  - retab
2014-10-23 16:42:35 -04:00
Ben Boeckel
475fe209a8 cmake: minor cleanups (indentation and typos) 2014-10-23 16:42:35 -04:00
Ben Boeckel
a43f1a88d8 cmake: remove configuration variables
CMAKE_BUILD_TYPE is meant for single-config build tools (e.g., make and
ninja) while CMAKE_CONFIGURATION_TYPES is meant for multi-config build
tools (e.g., Xcode and Visual Studio). They should not be mixed or
manually set.
2014-10-23 16:42:34 -04:00
Ben Boeckel
a87ce09461 cmake: factor out error messages
Instead of using BoldRed and ColourReset everywhere, wrap it up in a
function.
2014-10-23 16:42:34 -04:00
Ben Boeckel
9aa48b6c50 miniupnpc: clear out else/endfoo command arguments 2014-10-23 16:42:34 -04:00
Riccardo Spagni
7c1d3b5090
fixed miniupnpc dynamic target 2014-10-06 23:54:48 +02:00
Riccardo Spagni
2cf94c1321
fix for mingw not playing nicely with libunbound configure, fix for correctly finding static libs on various operating systems 2014-10-06 22:29:07 +02:00
Riccardo Spagni
e55982a892
set the winsock variable differently on Windows 2014-10-06 19:40:53 +02:00
Riccardo Spagni
0586a27a13
fixed configure_command for libunbound under mingw 2014-10-06 19:19:47 +02:00
Riccardo Spagni
f8fd03602c
turns out mingw does actually produce .a libs and not .dll.a 2014-10-06 19:17:40 +02:00
Riccardo Spagni
1afba7311e
quieten CMake when it can't find packages 2014-10-06 18:56:59 +02:00
Riccardo Spagni
735a017bb3
removed required flags from miniupnp and unbound 2014-10-06 18:51:03 +02:00
Riccardo Spagni
5829b35147 include openssl in linbunbound static linking, added errors for openssl and expat 2014-10-06 18:08:33 +02:00
Riccardo Spagni
c70fa8689f reference the correct unbound static lib 2014-10-06 18:08:33 +02:00
Riccardo Spagni
954a244afe added libtool's install command 2014-10-06 18:08:33 +02:00
Riccardo Spagni
fd52015a2d use the correct CMake variable for static builds 2014-10-06 18:08:33 +02:00
Riccardo Spagni
cc0f6a6080 build libunbound correctly 2014-10-06 18:08:32 +02:00
Riccardo Spagni
8c00098c75 fix missing parentheses 2014-10-06 18:08:32 +02:00
Riccardo Spagni
cee87473ad build libunbound from external if no local libunbound or for static builds 2014-10-06 18:08:32 +02:00
Riccardo Spagni
9ef094b356
added unbound to external deps 2014-10-05 23:44:31 +02:00
Riccardo Spagni
fce9dea1b9
fix for miniupnpc static compile under Windows 2014-10-05 20:22:14 +02:00
Riccardo Spagni
18281f4d20 remove dangling upnp port mappings, updated miniupnpc 2014-10-02 18:54:55 +02:00
Riccardo Spagni
dda5a03146
fixed FreeBSD miniupnpc niggly 2014-09-24 21:23:23 +02:00
Riccardo Spagni
524eae58cc
never use the baked-in miniupnpc on FreeBSD because explosions 2014-09-24 20:57:09 +02:00
Riccardo Spagni
d34f681ed6
fixed miniupnpc static building 2014-09-21 02:23:28 +02:00
fluffypony
b35c1e2491 fixed UPNP_LIBRARIES scope 2014-09-11 13:28:49 +02:00
fluffypony
d21cf293e1 added license to CMake 2014-09-11 08:25:07 +02:00
fluffypony
36c7ea9f77 more CMake tweaks to allow detection on OS X 2014-09-11 08:22:49 +02:00
fluffypony
8d8b47e69f more dynamic miniupnp fixes 2014-09-10 20:01:30 +02:00
fluffypony
7d01dad8f4 use external miniupnpc if available 2014-09-10 18:14:57 +02:00
fluffypony
5a65991480 exclude local miniupnpc for FreeBSD, install from ports instead 2014-09-09 14:03:42 +02:00
Antonio Juarez
9682a15400 Port mapping with UPnP 2014-04-09 13:14:35 +01:00