Nick Mathewson
de5f0d8ba7
Replace crash on missing handle in consdiffmgr with nonfatal assert
...
Attempts to mitigate 22752.
2017-06-28 14:21:21 -04:00
Alexander Færøy
07f2940b45
Set published_out for consensus cache entries in spooled_resource_estimate_size().
...
This patch ensures that the published_out output parameter is set to the
current consensus cache entry's "valid after" field.
See: https://bugs.torproject.org/22702
2017-06-27 18:25:48 -04:00
Nick Mathewson
bbeba2412e
Fix resource leak in parse_consensus_request()
...
We were allocating diff_hash_in_url on some URLs, but not freeing it.
Fixes CID 1409669. Bug not in any released Tor.
2017-05-16 10:47:41 -04:00
Nick Mathewson
2ca827104d
Report deleted consensuses as NOT_FOUND rather than AVAILABLE
...
This bug happened because of a bogus pointer check in
consdiffmgr_find_consensus(), not in any released Tor.
Fixes CID 1409670. Good catch, Coverity!
2017-05-16 10:44:24 -04:00
Nick Mathewson
460b923026
fix a memory leak
2017-05-15 18:13:20 -04:00
Nick Mathewson
afef059795
Merge remote-tracking branch 'public/prop140_aftermath_cfg'
2017-05-15 17:26:47 -04:00
Nick Mathewson
7591518d16
Copy valid/fresh-until and signatories values into diffs.
2017-05-15 17:21:55 -04:00
Nick Mathewson
fd1190581d
Store fresh/valid-until and signatories values on all consensus objects.
2017-05-15 17:21:55 -04:00
Nick Mathewson
dcc533fb13
Implement functions to expose valid/fresh-until and voters
...
These still won't do anything till I get the values to be filled in.
Also, I changed the API a little (with corresponding changes in
directory.c) to match things that it's easier to store.
2017-05-15 17:21:55 -04:00
Nick Mathewson
2f06345db3
Move stub accessor functions a level higher, to consdiffmgr
2017-05-15 17:21:55 -04:00
Alexander Færøy
64116ab97f
Fix tautological constant out-of-range comparison warnings.
2017-05-12 17:59:29 +02:00
Alexander Færøy
a1e8ef0076
Fix DoubleNL warning from make check-spaces.
2017-05-12 17:57:11 +02:00
Nick Mathewson
db370bb8a8
Test fix: expect old consensuses to be deleted if not deflate-compressed
2017-05-12 17:45:44 +02:00
Nick Mathewson
30dfb36148
consdiffmgr: Reload latest consensus entries on start.
2017-05-12 17:45:33 +02:00
Nick Mathewson
8100305e71
consdiffmgr: expose cached consensuses
2017-05-12 17:45:24 +02:00
Nick Mathewson
7b0dcf5c4a
Cleanup logic: only retain zlib-compressed consensuses
...
Now that we're making a bunch of these with consdiffmgr, we should
throw out all but one when we get a newer consensus.
2017-05-12 17:45:15 +02:00
Nick Mathewson
151cd121a2
consdiffmgr: compress incoming consensuses in the background
...
Also, compress them in several ways.
This breaks the unit tests; subsequent commits will make them pass
again.
2017-05-12 17:45:07 +02:00
Nick Mathewson
6da31ec484
consdiffmgr: Extract the code for compressing and storing
...
We're going to use this for consensuses too.
2017-05-12 17:44:55 +02:00
Nick Mathewson
c276c10d3b
Turn consdiffmgr.cache_max_age_hours into a parameter
2017-05-05 09:11:06 -04:00
Nick Mathewson
6beb7028d8
Do not BUG on missing sha3-as-signed field
...
This can happen if you've been running an earlier alpha on your
relay. Instead, just ignore the entry.
2017-05-04 08:50:21 -04:00
Nick Mathewson
3af9704e45
bug#22143/prop#140: in consdiffmgr, store and use digest-as-signed
...
We need to index diffs by the digest-as-signed of their source
consensus, so that we can find them even from consensuses whose
signatures are encoded differently.
2017-05-04 08:36:50 -04:00
Nick Mathewson
fd437f2a02
Remove special-casing for NO_METHOD in consdiffmgr.c
2017-05-02 08:37:57 -04:00
Nick Mathewson
b8f7488e94
Fix a brazen memleak in consdiffmgr_add_consensus()
2017-04-28 15:41:52 -04:00
Nick Mathewson
00a12337ff
Merge branches 'consdiffmgr_orig_squashed' and 'actually_compute_diffs_squashed'
2017-04-27 21:43:06 -04:00
Nick Mathewson
1e1581a24e
Pre-compress consensus diffs with supported consensus methods.
2017-04-27 21:40:46 -04:00
Nick Mathewson
a1172b6774
Store archived consensuses compressed on disk.
...
I'm just using gzip compression now, for performance.
2017-04-27 21:40:46 -04:00
Nick Mathewson
7a0964279f
Functionality to ensure there is space to add files to cache.
2017-04-27 21:40:13 -04:00
Nick Mathewson
920475f293
New force-delete option on consensus_cache_delete_pending()
...
If we're out of file space in the storage directory, we'll need to
get rid of old files fast.
2017-04-27 21:40:13 -04:00
Nick Mathewson
466e914088
Lower the file limit in consdiffmgr, to support seccomp2
2017-04-27 21:40:13 -04:00
Nick Mathewson
fba8d7b222
Initialize consdiffmgr when running as (or becoming) a server.
2017-04-27 21:40:12 -04:00
Sebastian Hahn
ee478bdf38
Fix coverity cid 1405510
...
This is a false positive, but let's appease coverity.
2017-04-26 08:45:38 +02:00
Nick Mathewson
24f7059704
Configure sandbox using consdiffmgr; free cdm on exit.
2017-04-25 19:52:34 -04:00
Nick Mathewson
9e081a44a9
Teach consdiffmgr to remember two digests: one compressed, one not.
2017-04-25 16:49:47 -04:00
Nick Mathewson
6cc21aa89c
consdiffmgr: add tests for cdm_entry_get_sha3_value
2017-04-24 11:01:40 -04:00
Nick Mathewson
43d683e0ad
Fix reference leak & handle leak in consensus_diff_worker_replyfn
...
Found by previous test.
2017-04-24 11:01:40 -04:00
Nick Mathewson
7ca86b9cd6
Add a hashtable to consdiffmgr to keep track of diff status
...
In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
* Remembering which diffs were in-progress, so that we didn't
re-launch them.
* Remembering which diffs had failed, so that we didn't try to
recompute them over and over.
* Having a fast way to look up the diff from a given consensus to
the latest consensus of a given flavor.
This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem. It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.
2017-04-24 11:01:40 -04:00
Nick Mathewson
69a212ff3d
Consdiffmgr: extract "get a sha3 digest" function.
...
I'll be using this a lot in the hashtable tweaks here.
2017-04-24 11:01:40 -04:00
Nick Mathewson
655f1c8e01
consdiffmgr: function to re-validate stored sha3 digests at startup
2017-04-24 11:00:28 -04:00
Nick Mathewson
6c86e63029
Consdiffmgr: use aggressive-release flag on consensuses
...
This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
35f6b678ab
Test the easiest cases of consdiffmgr_cleanup.
...
One more to go: deleting the old diffs.
2017-04-24 10:59:25 -04:00
Nick Mathewson
1fade37287
consdiffmgr non-test: check for initialization failure
...
Unfortunately, this test doesn't work, so I've left it
defined-out. There is currently no way in our unit tests to catch a
fatal assertion failure.
2017-04-24 10:59:25 -04:00
Nick Mathewson
7fc37d41b4
Unit tests for consdiffmgr module
...
Initial tests. These just try adding a few consensuses, looking
them up, and making sure that consensus diffs are generated in a
more or less reasonable-looking way. It's enough for 87% coverage,
but it leaves out a lot of functionality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
bc91808c47
Add a "Consensus diff manager" module.
...
This module's job is to remember old consensus documents, to
calculate their diffs on demand, and to .
There are some incomplete points in this code; I've marked them with
"XXXX". I intend to fix them in separate commits, since I believe
doing it in separate commits will make the branch easier to review.
2017-04-24 10:59:24 -04:00
