Commit Graph

25676 Commits

Author SHA1 Message Date
Georg Koppen
d6dd05f6d8 Bug 22114: Fix wrong values in torrc_format.txt comments 2017-05-01 13:53:07 -04:00
Taylor Yu
5a2f1a836f Fix help message for --enable-lzma
Fixes #22111.
2017-05-01 11:16:25 -04:00
Sebastian Hahn
915fa39d0f Add --enable-rust configure switch
Introduce a way to optionally enable Rust integration for our builds. No
actual Rust code is added yet and specifying the flag has no effect
other than failing the build if rustc and cargo are unavailable.
2017-04-29 08:55:57 +02:00
Nick Mathewson
b8f7488e94 Fix a brazen memleak in consdiffmgr_add_consensus() 2017-04-28 15:41:52 -04:00
Nick Mathewson
dcc0a49a84 Merge branch 'ticket21648' 2017-04-28 11:04:49 -04:00
Nick Mathewson
531835f561 Increase MALLOC_MP_LIM to 16MB
Increase the maximum allowed size passed to mprotect(PROT_WRITE)
from 1MB to 16MB. This was necessary with the glibc allocator
in order to allow worker threads to allocate more memory --
which in turn is necessary because of our new use of worker
threads for compression.

Closes ticket #22096. Found while working on #21648.
2017-04-28 10:55:10 -04:00
Nick Mathewson
00a12337ff Merge branches 'consdiffmgr_orig_squashed' and 'actually_compute_diffs_squashed' 2017-04-27 21:43:06 -04:00
Nick Mathewson
1e1581a24e Pre-compress consensus diffs with supported consensus methods. 2017-04-27 21:40:46 -04:00
Nick Mathewson
a1172b6774 Store archived consensuses compressed on disk.
I'm just using gzip compression now, for performance.
2017-04-27 21:40:46 -04:00
Nick Mathewson
7a0964279f Functionality to ensure there is space to add files to cache. 2017-04-27 21:40:13 -04:00
Nick Mathewson
920475f293 New force-delete option on consensus_cache_delete_pending()
If we're out of file space in the storage directory, we'll need to
get rid of old files fast.
2017-04-27 21:40:13 -04:00
Nick Mathewson
466e914088 Lower the file limit in consdiffmgr, to support seccomp2 2017-04-27 21:40:13 -04:00
Nick Mathewson
ab73bda060 Pass incoming consensus documents to the consdiffmgr code 2017-04-27 21:40:12 -04:00
Nick Mathewson
16d6ab6640 Fix use-after-free bug in storage_dir sandbox code. 2017-04-27 21:40:12 -04:00
Nick Mathewson
7b8d48a6cb Clean the consdiffmgr cache and launch new diffs as needed. 2017-04-27 21:40:12 -04:00
Nick Mathewson
fba8d7b222 Initialize consdiffmgr when running as (or becoming) a server. 2017-04-27 21:40:12 -04:00
Alexander Færøy
0672b33f1e
Enforce 16 MB upper bound of memory usage in LZMA decoder.
This patch changes two things in our LZMA compression backend:

- We lower the preset values for all `compression_level_t` values to
  ensure that we can run the LZMA decoder with less than 65 MB of memory
  available. This seems to have a small impact on the real world usage
  and fits well with our needs.

- We set the upper bound of memory usage for the LZMA decoder to 16 MB.

See: https://bugs.torproject.org/21665
2017-04-27 20:09:20 +02:00
Alexander Færøy
e5122b91a9
Only compare the first 3 bytes when trying to detect LZMA compression.
This patch changes the logic in `detect_compression_method()` to only
use the 3 first bytes when checking if a given input is LZMA encoded.
2017-04-27 20:07:08 +02:00
Nick Mathewson
480dab4f2f Use a cast to try to avoid a tautalogical comparison warning 2017-04-27 11:58:26 -04:00
Nick Mathewson
10a4f9cd07 Merge branch 'parse_accept_encoding' 2017-04-27 11:31:31 -04:00
Nick Mathewson
2903c329aa Move the "supported compression bitmask" into compress.[ch] 2017-04-27 11:30:51 -04:00
Nick Mathewson
49deb1e1b8 Document and test nul-terminating behavior of tor_uncompress()
We added this as a safety feature, but there are a few places in the
code that actually depend on it.
2017-04-27 10:59:48 -04:00
Nick Mathewson
199e61feb5 whitespace fix 2017-04-27 10:55:39 -04:00
Nick Mathewson
ba9f235e17 Re-enable the partial-input test for zstd.
There were two issues here: first, zstd didn't exhibit the right
behavior unless it got a very large input.  That's fine.

The second issue was a genuine bug, fixed by 39cfaba9e2.
2017-04-27 10:43:38 -04:00
Nick Mathewson
39cfaba9e2 Fix handling of "final" flag in zstd decompression
We were returning "DONE" on truncated input streams, which was not
what we wanted.
2017-04-27 10:42:05 -04:00
Nick Mathewson
166aa8d741 Have a separate entry point for each compresion test 2017-04-27 10:25:52 -04:00
Nick Mathewson
36e62ae7e2 Merge remote-tracking branch 'ahf/bugs/22085' 2017-04-27 10:12:48 -04:00
Nick Mathewson
33a2fd065d Merge branch 'dirreq' 2017-04-27 10:08:32 -04:00
Alexander Færøy
e0a3819dcc
Test support for multiple compression backends in the buffer code.
This patch refactors the buffer compression tests to support multiple
comprssion backends.

See: https://bugs.torproject.org/22085
2017-04-27 15:51:14 +02:00
Alexander Færøy
22e6ad6f26
Clean up mentions of 'zlib' and rename the mentions to 'compressed'.
This patch cleans up in various places where 'zlib' is mentioned.
2017-04-27 15:51:14 +02:00
Alexander Færøy
7bececbd69
Refactor compression tests into a single test.
This patch refactors our compression tests such that deflate, gzip,
lzma, and zstd are all tested using the same code.

Additionally we use run-time checks to see if the given compression
method is supported instead of using HAVE_LZMA and HAVE_ZSTD.

See: https://bugs.torproject.org/22085
2017-04-27 15:51:14 +02:00
Nick Mathewson
b4fe0a6a03 Improve control flow in authority_certs_fetch_resource_impl 2017-04-27 09:30:08 -04:00
Nick Mathewson
52316f9969 Include UPLOAD_RENDDESC_V2 in PURPOSE_IS_UPLOAD
This was only used in one place before, and it's safe to update it.
2017-04-27 09:27:00 -04:00
Taylor Yu
224259a929 Document Bridge line transport arguments
Bridge lines in torrc can contain key=value settings as per-connection
arguments to a pluggable transport.  tor.1.txt hadn't been updated to
reflect this.
2017-04-26 16:39:36 -04:00
Nick Mathewson
e5eb260b7a forward-port changelog and release notes 2017-04-26 15:39:10 -04:00
Nick Mathewson
d95892c5d2 Merge branch 'maint-0.3.0' 2017-04-26 15:37:52 -04:00
Nick Mathewson
7ae7e3c2d8 bump to 0.3.0.6-dev 2017-04-26 15:37:27 -04:00
Nick Mathewson
7fb9586953 Fix compilation when lzma or zstd is absent 2017-04-26 15:00:40 -04:00
Nick Mathewson
4038202f89 Avoid a warning from the use of floating-point in zstd
Replace "(preset - 0.5) * 1mb" with "preset * 1mb - 0.5 mb", to
avoid warning about converting double to size_t.
2017-04-26 14:21:45 -04:00
Nick Mathewson
be0557f759 Merge remote-tracking branch 'ahf/bugs/22066' 2017-04-26 14:20:01 -04:00
Alexander Færøy
e42c204f67
Approximate memory usage needed for the Zstandard backend.
This patch adds support for measuring the approximated memory usage by
the individual `tor_zstd_compress_state_t` object instances.

See: https://bugs.torproject.org/22066
2017-04-26 19:54:18 +02:00
Alexander Færøy
2aa28e7cb7
Better documentation for tor_uncompress().
This patch fixes the documentation string for `tor_uncompress()` to
ensure that it does not explicitly mention zlib or gzip since we now
support multiple compression backends.
2017-04-26 19:54:18 +02:00
Alexander Færøy
341824687a
Approximate memory usage needed for the LZMA backend.
This patch adds support for measuring the approximated memory usage by
the individual `tor_lzma_compress_state_t` object instances.

The LZMA library provides the functions `lzma_easy_encoder_memusage()`
and `lzma_easy_decoder_memusage()` which is used to find the estimated
usage in bytes.

See: https://bugs.torproject.org/22066
2017-04-26 19:54:18 +02:00
Nick Mathewson
d92a56c70b Merge branch 'maint-0.3.0' 2017-04-26 13:31:19 -04:00
Nick Mathewson
31b79f02ab Bump version to 0.3.0.6 2017-04-26 13:30:39 -04:00
Nick Mathewson
99e943998d Add getpid() to the seccomp2 sandbox.
We hadn't needed this before, because most getpid() callers on Linux
were looking at the vDSO version of getpid().  I don't know why at
least one version of OpenSSL seems to be ignoring the vDSO, but this
change should fix it.

Fixes bug 21943; bugfix on 0.2.5.1-alpha when the sandbox was
introduced.
2017-04-26 12:56:06 -04:00
Sebastian Hahn
71c8974af0 Fix coverity cid 1405509
Locking in the init function is not necessary, but coverity gets
confused about it. So let's trick it.
2017-04-26 08:48:24 +02:00
Sebastian Hahn
ee478bdf38 Fix coverity cid 1405510
This is a false positive, but let's appease coverity.
2017-04-26 08:45:38 +02:00
Alexander Færøy
08d86e8408
Store compression overhead from tor_compress_state_t.
The `tor_compress_state_t` data-type is used as a wrapper around the
more specialized state-types used by the various compression backends.
This patch ensures that the overhead of this "thin" wrapper type is
included in the value returned by `tor_compress_get_total_allocation()`.

See: https://bugs.torproject.org/22066
2017-04-26 02:56:21 +02:00
Alexander Færøy
fac8ac0e4a
Remove unused header from the Zstandard compression backend.
Since we stopped looking at Zstandard error codes there is no need to
include the zstd_errors.h header file anymore.
2017-04-26 02:54:34 +02:00