Commit Graph

13295 Commits

Author SHA1 Message Date
Nick Mathewson
b8f7488e94 Fix a brazen memleak in consdiffmgr_add_consensus() 2017-04-28 15:41:52 -04:00
Nick Mathewson
00a12337ff Merge branches 'consdiffmgr_orig_squashed' and 'actually_compute_diffs_squashed' 2017-04-27 21:43:06 -04:00
Nick Mathewson
1e1581a24e Pre-compress consensus diffs with supported consensus methods. 2017-04-27 21:40:46 -04:00
Nick Mathewson
a1172b6774 Store archived consensuses compressed on disk.
I'm just using gzip compression now, for performance.
2017-04-27 21:40:46 -04:00
Nick Mathewson
7a0964279f Functionality to ensure there is space to add files to cache. 2017-04-27 21:40:13 -04:00
Nick Mathewson
920475f293 New force-delete option on consensus_cache_delete_pending()
If we're out of file space in the storage directory, we'll need to
get rid of old files fast.
2017-04-27 21:40:13 -04:00
Nick Mathewson
466e914088 Lower the file limit in consdiffmgr, to support seccomp2 2017-04-27 21:40:13 -04:00
Nick Mathewson
ab73bda060 Pass incoming consensus documents to the consdiffmgr code 2017-04-27 21:40:12 -04:00
Nick Mathewson
7b8d48a6cb Clean the consdiffmgr cache and launch new diffs as needed. 2017-04-27 21:40:12 -04:00
Nick Mathewson
fba8d7b222 Initialize consdiffmgr when running as (or becoming) a server. 2017-04-27 21:40:12 -04:00
Nick Mathewson
480dab4f2f Use a cast to try to avoid a tautalogical comparison warning 2017-04-27 11:58:26 -04:00
Nick Mathewson
10a4f9cd07 Merge branch 'parse_accept_encoding' 2017-04-27 11:31:31 -04:00
Nick Mathewson
2903c329aa Move the "supported compression bitmask" into compress.[ch] 2017-04-27 11:30:51 -04:00
Nick Mathewson
33a2fd065d Merge branch 'dirreq' 2017-04-27 10:08:32 -04:00
Nick Mathewson
b4fe0a6a03 Improve control flow in authority_certs_fetch_resource_impl 2017-04-27 09:30:08 -04:00
Nick Mathewson
52316f9969 Include UPLOAD_RENDDESC_V2 in PURPOSE_IS_UPLOAD
This was only used in one place before, and it's safe to update it.
2017-04-27 09:27:00 -04:00
Sebastian Hahn
ee478bdf38 Fix coverity cid 1405510
This is a false positive, but let's appease coverity.
2017-04-26 08:45:38 +02:00
Nick Mathewson
24f7059704 Configure sandbox using consdiffmgr; free cdm on exit. 2017-04-25 19:52:34 -04:00
Nick Mathewson
65ff0f8267 Bitmask out the compression methods that we do not support 2017-04-25 19:07:17 -04:00
Nick Mathewson
fec3050ea9 Tests for parse_accept_encoding 2017-04-25 19:01:05 -04:00
Nick Mathewson
fd48b757d3 Parse recognized entries from the Accept-Encoding header. 2017-04-25 19:01:05 -04:00
Nick Mathewson
9e081a44a9 Teach consdiffmgr to remember two digests: one compressed, one not. 2017-04-25 16:49:47 -04:00
Nick Mathewson
43db91bd87 Teach cov-exclude to detect runaway LCOV_EXCL_START lines
Also, fix two instances of runaway LCOV_EXCL_START lines.
2017-04-25 10:59:19 -04:00
Nick Mathewson
232c9e14a8 Merge branch 'atomic_counters' 2017-04-25 10:46:23 -04:00
Nick Mathewson
ec7c512d27 Merge remote-tracking branch 'dgoulet/bug21293_031_01' 2017-04-25 10:43:53 -04:00
David Goulet
cb8ac1f331 trace: Add a basic event-tracing infrastructure.
This commit adds the src/trace directory containing the basics for our tracing
subsystem. It is not used in the code base. The "src/trace/debug.h" file
contains an example on how we can map our tor trace events to log_debug().

The tracing subsystem can only be enabled by tracing framework at compile
time. This commit introduces the "--enable-tracing-debug" option that will
make all "tor_trace()" function be maped to "log_debug()".

Closes #13802

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-25 10:37:31 -04:00
Nick Mathewson
4266ec766a Use atomic counters for compressor allocation. 2017-04-25 10:29:07 -04:00
David Goulet
368b091329 relay: Change LD_BUG log to LD_PROTOCOL_WARN
That log statement can be triggered if somebody on the Internet behaves badly
which is possible with buggy implementation for instance.

Fixes #21293

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-25 09:27:34 -04:00
Nick Mathewson
ba405f86bf Merge branch 'ahf_prop278_21662_squashed' 2017-04-25 08:12:59 -04:00
Alexander Færøy
c2d1d949de Use tor_compress_supports_method() before printing library versions.
This patch ensures that Tor checks if a given compression method is
supported before printing the version string when calling `tor
--library-versions`.

Additionally, we use the `tor_compress_supports_method()` to check if a
given version is supported for Tor's start-up version string, but here
we print "N/A" if a given compression method is unavailable.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:10 -04:00
Alexander Færøy
04682d302a Add tor_compress_get_total_allocation() function.
This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
be4dc54634 Display LZMA and Zstandard versions when starting Tor.
See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
2fa7b722ce Show liblzma and libzstd versions in tor --library-versions.
See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
ce1feae9d9 Add --enable-zstd to our configure script.
This patch adds support for enabling support for Zstandard to our configure
script. By default, the --enable-zstd option is set to "auto" which means if
libzstd is available we'll build Tor with Zstandard support.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
157af1d26e Add --enable-lzma to our configure script.
This patch adds support for enabling support for LZMA to our configure
script. By default, the --enable-lzma option is set to "auto" which
means if liblzma is available we'll build Tor with LZMA support.

See: https://bugs.torproject.org/21662
2017-04-25 08:06:02 -04:00
Alexander Færøy
04583df452 Rename the torgzip module to compress.
See https://bugs.torproject.org/21663
2017-04-25 08:06:01 -04:00
Alexander Færøy
9d5bc1a935 Move zlib compression code into its own module.
This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.

Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.

See https://bugs.torproject.org/21664
2017-04-25 08:06:01 -04:00
Nick Mathewson
4cefda85e4 Merge remote-tracking branch 'dgoulet/ticket21980_031_01' 2017-04-24 12:42:11 -04:00
Nick Mathewson
e55c1412c1 Fix a signed/unsigned comparison warning on 32-bit 2017-04-24 11:41:11 -04:00
Nick Mathewson
b7567a6282 Merge branch 'consdiffmgr_squashed' 2017-04-24 11:02:22 -04:00
Nick Mathewson
6cc21aa89c consdiffmgr: add tests for cdm_entry_get_sha3_value 2017-04-24 11:01:40 -04:00
Nick Mathewson
43d683e0ad Fix reference leak & handle leak in consensus_diff_worker_replyfn
Found by previous test.
2017-04-24 11:01:40 -04:00
Nick Mathewson
7ca86b9cd6 Add a hashtable to consdiffmgr to keep track of diff status
In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
   * Remembering which diffs were in-progress, so that we didn't
     re-launch them.
   * Remembering which diffs had failed, so that we didn't try to
     recompute them over and over.
   * Having a fast way to look up the diff from a given consensus to
     the latest consensus of a given flavor.

This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem.  It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.
2017-04-24 11:01:40 -04:00
Nick Mathewson
69a212ff3d Consdiffmgr: extract "get a sha3 digest" function.
I'll be using this a lot in the hashtable tweaks here.
2017-04-24 11:01:40 -04:00
Nick Mathewson
655f1c8e01 consdiffmgr: function to re-validate stored sha3 digests at startup 2017-04-24 11:00:28 -04:00
Nick Mathewson
6c86e63029 Consdiffmgr: use aggressive-release flag on consensuses
This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
82bb8afb60 Add handle support to consensus_cache_entry_t
This will allow us to have weak references to cache entries.
2017-04-24 10:59:25 -04:00
Nick Mathewson
35f6b678ab Test the easiest cases of consdiffmgr_cleanup.
One more to go: deleting the old diffs.
2017-04-24 10:59:25 -04:00
Nick Mathewson
1fade37287 consdiffmgr non-test: check for initialization failure
Unfortunately, this test doesn't work, so I've left it
defined-out. There is currently no way in our unit tests to catch a
fatal assertion failure.
2017-04-24 10:59:25 -04:00
Nick Mathewson
7fc37d41b4 Unit tests for consdiffmgr module
Initial tests. These just try adding a few consensuses, looking
them up, and making sure that consensus diffs are generated in a
more or less reasonable-looking way.  It's enough for 87% coverage,
but it leaves out a lot of functionality.
2017-04-24 10:59:25 -04:00