Commit Graph

38 Commits

Author SHA1 Message Date
Nick Mathewson
afef059795 Merge remote-tracking branch 'public/prop140_aftermath_cfg' 2017-05-15 17:26:47 -04:00
Nick Mathewson
7591518d16 Copy valid/fresh-until and signatories values into diffs. 2017-05-15 17:21:55 -04:00
Nick Mathewson
fd1190581d Store fresh/valid-until and signatories values on all consensus objects. 2017-05-15 17:21:55 -04:00
Nick Mathewson
dcc533fb13 Implement functions to expose valid/fresh-until and voters
These still won't do anything till I get the values to be filled in.

Also, I changed the API a little (with corresponding changes in
directory.c) to match things that it's easier to store.
2017-05-15 17:21:55 -04:00
Nick Mathewson
2f06345db3 Move stub accessor functions a level higher, to consdiffmgr 2017-05-15 17:21:55 -04:00
Alexander Færøy
64116ab97f
Fix tautological constant out-of-range comparison warnings. 2017-05-12 17:59:29 +02:00
Alexander Færøy
a1e8ef0076
Fix DoubleNL warning from make check-spaces. 2017-05-12 17:57:11 +02:00
Nick Mathewson
db370bb8a8
Test fix: expect old consensuses to be deleted if not deflate-compressed 2017-05-12 17:45:44 +02:00
Nick Mathewson
30dfb36148
consdiffmgr: Reload latest consensus entries on start. 2017-05-12 17:45:33 +02:00
Nick Mathewson
8100305e71
consdiffmgr: expose cached consensuses 2017-05-12 17:45:24 +02:00
Nick Mathewson
7b0dcf5c4a
Cleanup logic: only retain zlib-compressed consensuses
Now that we're making a bunch of these with consdiffmgr, we should
throw out all but one when we get a newer consensus.
2017-05-12 17:45:15 +02:00
Nick Mathewson
151cd121a2
consdiffmgr: compress incoming consensuses in the background
Also, compress them in several ways.

This breaks the unit tests; subsequent commits will make them pass
again.
2017-05-12 17:45:07 +02:00
Nick Mathewson
6da31ec484
consdiffmgr: Extract the code for compressing and storing
We're going to use this for consensuses too.
2017-05-12 17:44:55 +02:00
Nick Mathewson
c276c10d3b Turn consdiffmgr.cache_max_age_hours into a parameter 2017-05-05 09:11:06 -04:00
Nick Mathewson
6beb7028d8 Do not BUG on missing sha3-as-signed field
This can happen if you've been running an earlier alpha on your
relay.  Instead, just ignore the entry.
2017-05-04 08:50:21 -04:00
Nick Mathewson
3af9704e45 bug#22143/prop#140: in consdiffmgr, store and use digest-as-signed
We need to index diffs by the digest-as-signed of their source
consensus, so that we can find them even from consensuses whose
signatures are encoded differently.
2017-05-04 08:36:50 -04:00
Nick Mathewson
fd437f2a02 Remove special-casing for NO_METHOD in consdiffmgr.c 2017-05-02 08:37:57 -04:00
Nick Mathewson
b8f7488e94 Fix a brazen memleak in consdiffmgr_add_consensus() 2017-04-28 15:41:52 -04:00
Nick Mathewson
00a12337ff Merge branches 'consdiffmgr_orig_squashed' and 'actually_compute_diffs_squashed' 2017-04-27 21:43:06 -04:00
Nick Mathewson
1e1581a24e Pre-compress consensus diffs with supported consensus methods. 2017-04-27 21:40:46 -04:00
Nick Mathewson
a1172b6774 Store archived consensuses compressed on disk.
I'm just using gzip compression now, for performance.
2017-04-27 21:40:46 -04:00
Nick Mathewson
7a0964279f Functionality to ensure there is space to add files to cache. 2017-04-27 21:40:13 -04:00
Nick Mathewson
920475f293 New force-delete option on consensus_cache_delete_pending()
If we're out of file space in the storage directory, we'll need to
get rid of old files fast.
2017-04-27 21:40:13 -04:00
Nick Mathewson
466e914088 Lower the file limit in consdiffmgr, to support seccomp2 2017-04-27 21:40:13 -04:00
Nick Mathewson
fba8d7b222 Initialize consdiffmgr when running as (or becoming) a server. 2017-04-27 21:40:12 -04:00
Sebastian Hahn
ee478bdf38 Fix coverity cid 1405510
This is a false positive, but let's appease coverity.
2017-04-26 08:45:38 +02:00
Nick Mathewson
24f7059704 Configure sandbox using consdiffmgr; free cdm on exit. 2017-04-25 19:52:34 -04:00
Nick Mathewson
9e081a44a9 Teach consdiffmgr to remember two digests: one compressed, one not. 2017-04-25 16:49:47 -04:00
Nick Mathewson
6cc21aa89c consdiffmgr: add tests for cdm_entry_get_sha3_value 2017-04-24 11:01:40 -04:00
Nick Mathewson
43d683e0ad Fix reference leak & handle leak in consensus_diff_worker_replyfn
Found by previous test.
2017-04-24 11:01:40 -04:00
Nick Mathewson
7ca86b9cd6 Add a hashtable to consdiffmgr to keep track of diff status
In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
   * Remembering which diffs were in-progress, so that we didn't
     re-launch them.
   * Remembering which diffs had failed, so that we didn't try to
     recompute them over and over.
   * Having a fast way to look up the diff from a given consensus to
     the latest consensus of a given flavor.

This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem.  It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.
2017-04-24 11:01:40 -04:00
Nick Mathewson
69a212ff3d Consdiffmgr: extract "get a sha3 digest" function.
I'll be using this a lot in the hashtable tweaks here.
2017-04-24 11:01:40 -04:00
Nick Mathewson
655f1c8e01 consdiffmgr: function to re-validate stored sha3 digests at startup 2017-04-24 11:00:28 -04:00
Nick Mathewson
6c86e63029 Consdiffmgr: use aggressive-release flag on consensuses
This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
35f6b678ab Test the easiest cases of consdiffmgr_cleanup.
One more to go: deleting the old diffs.
2017-04-24 10:59:25 -04:00
Nick Mathewson
1fade37287 consdiffmgr non-test: check for initialization failure
Unfortunately, this test doesn't work, so I've left it
defined-out. There is currently no way in our unit tests to catch a
fatal assertion failure.
2017-04-24 10:59:25 -04:00
Nick Mathewson
7fc37d41b4 Unit tests for consdiffmgr module
Initial tests. These just try adding a few consensuses, looking
them up, and making sure that consensus diffs are generated in a
more or less reasonable-looking way.  It's enough for 87% coverage,
but it leaves out a lot of functionality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
bc91808c47 Add a "Consensus diff manager" module.
This module's job is to remember old consensus documents, to
calculate their diffs on demand, and to .

There are some incomplete points in this code; I've marked them with
"XXXX". I intend to fix them in separate commits, since I believe
doing it in separate commits will make the branch easier to review.
2017-04-24 10:59:24 -04:00