Nick Mathewson
00a12337ff
Merge branches 'consdiffmgr_orig_squashed' and 'actually_compute_diffs_squashed'
2017-04-27 21:43:06 -04:00
Nick Mathewson
1e1581a24e
Pre-compress consensus diffs with supported consensus methods.
2017-04-27 21:40:46 -04:00
Nick Mathewson
a1172b6774
Store archived consensuses compressed on disk.
...
I'm just using gzip compression now, for performance.
2017-04-27 21:40:46 -04:00
Nick Mathewson
7a0964279f
Functionality to ensure there is space to add files to cache.
2017-04-27 21:40:13 -04:00
Nick Mathewson
920475f293
New force-delete option on consensus_cache_delete_pending()
...
If we're out of file space in the storage directory, we'll need to
get rid of old files fast.
2017-04-27 21:40:13 -04:00
Nick Mathewson
466e914088
Lower the file limit in consdiffmgr, to support seccomp2
2017-04-27 21:40:13 -04:00
Nick Mathewson
ab73bda060
Pass incoming consensus documents to the consdiffmgr code
2017-04-27 21:40:12 -04:00
Nick Mathewson
16d6ab6640
Fix use-after-free bug in storage_dir sandbox code.
2017-04-27 21:40:12 -04:00
Nick Mathewson
7b8d48a6cb
Clean the consdiffmgr cache and launch new diffs as needed.
2017-04-27 21:40:12 -04:00
Nick Mathewson
fba8d7b222
Initialize consdiffmgr when running as (or becoming) a server.
2017-04-27 21:40:12 -04:00
Alexander Færøy
0672b33f1e
Enforce 16 MB upper bound of memory usage in LZMA decoder.
...
This patch changes two things in our LZMA compression backend:
- We lower the preset values for all `compression_level_t` values to
ensure that we can run the LZMA decoder with less than 65 MB of memory
available. This seems to have a small impact on the real world usage
and fits well with our needs.
- We set the upper bound of memory usage for the LZMA decoder to 16 MB.
See: https://bugs.torproject.org/21665
2017-04-27 20:09:20 +02:00
Alexander Færøy
e5122b91a9
Only compare the first 3 bytes when trying to detect LZMA compression.
...
This patch changes the logic in `detect_compression_method()` to only
use the 3 first bytes when checking if a given input is LZMA encoded.
2017-04-27 20:07:08 +02:00
Nick Mathewson
480dab4f2f
Use a cast to try to avoid a tautalogical comparison warning
2017-04-27 11:58:26 -04:00
Nick Mathewson
10a4f9cd07
Merge branch 'parse_accept_encoding'
2017-04-27 11:31:31 -04:00
Nick Mathewson
2903c329aa
Move the "supported compression bitmask" into compress.[ch]
2017-04-27 11:30:51 -04:00
Nick Mathewson
49deb1e1b8
Document and test nul-terminating behavior of tor_uncompress()
...
We added this as a safety feature, but there are a few places in the
code that actually depend on it.
2017-04-27 10:59:48 -04:00
Nick Mathewson
199e61feb5
whitespace fix
2017-04-27 10:55:39 -04:00
Nick Mathewson
ba9f235e17
Re-enable the partial-input test for zstd.
...
There were two issues here: first, zstd didn't exhibit the right
behavior unless it got a very large input. That's fine.
The second issue was a genuine bug, fixed by 39cfaba9e2
2017-04-27 10:43:38 -04:00
Nick Mathewson
39cfaba9e2
Fix handling of "final" flag in zstd decompression
...
We were returning "DONE" on truncated input streams, which was not
what we wanted.
2017-04-27 10:42:05 -04:00
Nick Mathewson
166aa8d741
Have a separate entry point for each compresion test
2017-04-27 10:25:52 -04:00
Nick Mathewson
36e62ae7e2
Merge remote-tracking branch 'ahf/bugs/22085'
2017-04-27 10:12:48 -04:00
Nick Mathewson
33a2fd065d
Merge branch 'dirreq'
2017-04-27 10:08:32 -04:00
Alexander Færøy
e0a3819dcc
Test support for multiple compression backends in the buffer code.
...
This patch refactors the buffer compression tests to support multiple
comprssion backends.
See: https://bugs.torproject.org/22085
2017-04-27 15:51:14 +02:00
Alexander Færøy
22e6ad6f26
Clean up mentions of 'zlib' and rename the mentions to 'compressed'.
...
This patch cleans up in various places where 'zlib' is mentioned.
2017-04-27 15:51:14 +02:00
Alexander Færøy
7bececbd69
Refactor compression tests into a single test.
...
This patch refactors our compression tests such that deflate, gzip,
lzma, and zstd are all tested using the same code.
Additionally we use run-time checks to see if the given compression
method is supported instead of using HAVE_LZMA and HAVE_ZSTD.
See: https://bugs.torproject.org/22085
2017-04-27 15:51:14 +02:00
Nick Mathewson
b4fe0a6a03
Improve control flow in authority_certs_fetch_resource_impl
2017-04-27 09:30:08 -04:00
Nick Mathewson
52316f9969
Include UPLOAD_RENDDESC_V2 in PURPOSE_IS_UPLOAD
...
This was only used in one place before, and it's safe to update it.
2017-04-27 09:27:00 -04:00
Taylor Yu
224259a929
Document Bridge line transport arguments
...
Bridge lines in torrc can contain key=value settings as per-connection
arguments to a pluggable transport. tor.1.txt hadn't been updated to
reflect this.
2017-04-26 16:39:36 -04:00
Nick Mathewson
e5eb260b7a
forward-port changelog and release notes
2017-04-26 15:39:10 -04:00
Nick Mathewson
d95892c5d2
Merge branch 'maint-0.3.0'
2017-04-26 15:37:52 -04:00
Nick Mathewson
7ae7e3c2d8
bump to 0.3.0.6-dev
2017-04-26 15:37:27 -04:00
Nick Mathewson
7fb9586953
Fix compilation when lzma or zstd is absent
2017-04-26 15:00:40 -04:00
Nick Mathewson
4038202f89
Avoid a warning from the use of floating-point in zstd
...
Replace "(preset - 0.5) * 1mb" with "preset * 1mb - 0.5 mb", to
avoid warning about converting double to size_t.
2017-04-26 14:21:45 -04:00
Nick Mathewson
be0557f759
Merge remote-tracking branch 'ahf/bugs/22066'
2017-04-26 14:20:01 -04:00
Alexander Færøy
e42c204f67
Approximate memory usage needed for the Zstandard backend.
...
This patch adds support for measuring the approximated memory usage by
the individual `tor_zstd_compress_state_t` object instances.
See: https://bugs.torproject.org/22066
2017-04-26 19:54:18 +02:00
Alexander Færøy
2aa28e7cb7
Better documentation for tor_uncompress().
...
This patch fixes the documentation string for `tor_uncompress()` to
ensure that it does not explicitly mention zlib or gzip since we now
support multiple compression backends.
2017-04-26 19:54:18 +02:00
Alexander Færøy
341824687a
Approximate memory usage needed for the LZMA backend.
...
This patch adds support for measuring the approximated memory usage by
the individual `tor_lzma_compress_state_t` object instances.
The LZMA library provides the functions `lzma_easy_encoder_memusage()`
and `lzma_easy_decoder_memusage()` which is used to find the estimated
usage in bytes.
See: https://bugs.torproject.org/22066
2017-04-26 19:54:18 +02:00
Nick Mathewson
d92a56c70b
Merge branch 'maint-0.3.0'
2017-04-26 13:31:19 -04:00
Nick Mathewson
31b79f02ab
Bump version to 0.3.0.6
2017-04-26 13:30:39 -04:00
Sebastian Hahn
71c8974af0
Fix coverity cid 1405509
...
Locking in the init function is not necessary, but coverity gets
confused about it. So let's trick it.
2017-04-26 08:48:24 +02:00
Sebastian Hahn
ee478bdf38
Fix coverity cid 1405510
...
This is a false positive, but let's appease coverity.
2017-04-26 08:45:38 +02:00
Alexander Færøy
08d86e8408
Store compression overhead from tor_compress_state_t.
...
The `tor_compress_state_t` data-type is used as a wrapper around the
more specialized state-types used by the various compression backends.
This patch ensures that the overhead of this "thin" wrapper type is
included in the value returned by `tor_compress_get_total_allocation()`.
See: https://bugs.torproject.org/22066
2017-04-26 02:56:21 +02:00
Alexander Færøy
fac8ac0e4a
Remove unused header from the Zstandard compression backend.
...
Since we stopped looking at Zstandard error codes there is no need to
include the zstd_errors.h header file anymore.
2017-04-26 02:54:34 +02:00
Nick Mathewson
24f7059704
Configure sandbox using consdiffmgr; free cdm on exit.
2017-04-25 19:52:34 -04:00
Nick Mathewson
65ff0f8267
Bitmask out the compression methods that we do not support
2017-04-25 19:07:17 -04:00
Nick Mathewson
fec3050ea9
Tests for parse_accept_encoding
2017-04-25 19:01:05 -04:00
Nick Mathewson
fd48b757d3
Parse recognized entries from the Accept-Encoding header.
2017-04-25 19:01:05 -04:00
Nick Mathewson
2655a72d89
Use x-lzma, not x-lzma2, as the identifier
2017-04-25 19:00:52 -04:00
Nick Mathewson
9e081a44a9
Teach consdiffmgr to remember two digests: one compressed, one not.
2017-04-25 16:49:47 -04:00
Nick Mathewson
0274ea749a
Function to convert compression methods to/from strings.
2017-04-25 16:47:46 -04:00
