Increase the maximum allowed size passed to mprotect(PROT_WRITE)
from 1MB to 16MB. This was necessary with the glibc allocator
in order to allow worker threads to allocate more memory --
which in turn is necessary because of our new use of worker
threads for compression.
Closes ticket #22096. Found while working on #21648.
This patch changes two things in our LZMA compression backend:
- We lower the preset values for all `compression_level_t` values to
ensure that we can run the LZMA decoder with less than 65 MB of memory
available. This seems to have a small impact on the real world usage
and fits well with our needs.
- We set the upper bound of memory usage for the LZMA decoder to 16 MB.
See: https://bugs.torproject.org/21665
There were two issues here: first, zstd didn't exhibit the right
behavior unless it got a very large input. That's fine.
The second issue was a genuine bug, fixed by 39cfaba9e2.
This patch refactors our compression tests such that deflate, gzip,
lzma, and zstd are all tested using the same code.
Additionally we use run-time checks to see if the given compression
method is supported instead of using HAVE_LZMA and HAVE_ZSTD.
See: https://bugs.torproject.org/22085
This patch adds support for measuring the approximated memory usage by
the individual `tor_zstd_compress_state_t` object instances.
See: https://bugs.torproject.org/22066
This patch fixes the documentation string for `tor_uncompress()` to
ensure that it does not explicitly mention zlib or gzip since we now
support multiple compression backends.
This patch adds support for measuring the approximated memory usage by
the individual `tor_lzma_compress_state_t` object instances.
The LZMA library provides the functions `lzma_easy_encoder_memusage()`
and `lzma_easy_decoder_memusage()` which is used to find the estimated
usage in bytes.
See: https://bugs.torproject.org/22066
We hadn't needed this before, because most getpid() callers on Linux
were looking at the vDSO version of getpid(). I don't know why at
least one version of OpenSSL seems to be ignoring the vDSO, but this
change should fix it.
Fixes bug 21943; bugfix on 0.2.5.1-alpha when the sandbox was
introduced.
The `tor_compress_state_t` data-type is used as a wrapper around the
more specialized state-types used by the various compression backends.
This patch ensures that the overhead of this "thin" wrapper type is
included in the value returned by `tor_compress_get_total_allocation()`.
See: https://bugs.torproject.org/22066
OS X's ar(1) doesn't allow us to create an archive with no object files.
This patch adds a stub file with a stub function in it to make OS X
happy again.
Since we have a streaming API for each compression backend, we don't
need a non-streaming API for each: we can build a common
non-streaming API at the front-end.
This commit adds the src/trace directory containing the basics for our tracing
subsystem. It is not used in the code base. The "src/trace/debug.h" file
contains an example on how we can map our tor trace events to log_debug().
The tracing subsystem can only be enabled by tracing framework at compile
time. This commit introduces the "--enable-tracing-debug" option that will
make all "tor_trace()" function be maped to "log_debug()".
Closes#13802
Signed-off-by: David Goulet <dgoulet@torproject.org>
That log statement can be triggered if somebody on the Internet behaves badly
which is possible with buggy implementation for instance.
Fixes#21293
Signed-off-by: David Goulet <dgoulet@torproject.org>
This patch renames the `compress` parameter of the
`tor_zlib_compress_new()` function to `_compress` to avoid shadowing the
`compress()` function in zlib.h.
This patch splits up `tor_compress_memory_level()` into static functions
in the individual compression backends, which allows us to tune the
values per compression backend rather than globally.
See: https://bugs.torproject.org/21662
Use a switch-statement in `tor_compress()` and `tor_uncompress()` for
the given `compress_method_t` parameter. This allows us to have the
compiler detect if we forgot add a handler in these functions for a
newly added enumeration value.
See: https://bugs.torproject.org/21662
This patch ensures that Tor checks if a given compression method is
supported before printing the version string when calling `tor
--library-versions`.
Additionally, we use the `tor_compress_supports_method()` to check if a
given version is supported for Tor's start-up version string, but here
we print "N/A" if a given compression method is unavailable.
See: https://bugs.torproject.org/21662
This patch adds `tor_compress_version_str()` and
`tor_compress_header_version_str()` to get the version strings of the
different compression schema providers. Both functions returns `NULL` in
case a given `compress_method_t` is unknown or unsupported.
See: https://bugs.torproject.org/21662
This patch adds support for checking if a given `compress_method_t` is
supported by the currently running Tor instance using
`tor_compress_supports_method()`.
See: https://bugs.torproject.org/21662
This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.
See: https://bugs.torproject.org/21662
This patch adds support for enabling support for Zstandard to our configure
script. By default, the --enable-zstd option is set to "auto" which means if
libzstd is available we'll build Tor with Zstandard support.
See: https://bugs.torproject.org/21662
This patch adds support for enabling support for LZMA to our configure
script. By default, the --enable-lzma option is set to "auto" which
means if liblzma is available we'll build Tor with LZMA support.
See: https://bugs.torproject.org/21662
This patch changes the way `tor_compress_new()`,
`tor_compress_process()`, and `tor_compress_free()` handles different
compression methods. This should give us compiler warnings in case an
additional compression method is added, but the developer forgets to add
handlers in the three aforementioned functions.
See https://bugs.torproject.org/21663
This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.
Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.
See https://bugs.torproject.org/21664
Remove duplicate code that validates a service object which is now in
rend_validate_service().
Add some comments on why we nullify a service in the code path of
rend_config_services().
Signed-off-by: David Goulet <dgoulet@torproject.org>
This new function validates a service object and is used everytime a service
is successfully loaded from the configuration file.
It is currently copying the validation that rend_add_service() also does which
means both functions validate. It will be decoupled in the next commit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The item referred to the cdm_ht_set_status() case where the item was
not already in the hashtable. But that already happens naturally
when we scan the directory on startup... and we already have a test
for that.
This commit adds some helper functions to look up the diff from one
consensus and to make sure that applying it leads to another. Then
we add them throughout the existing test cases. Doing this turned
up a reference-leaking bug in consensus_diff_worker_replyfn.
In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
* Remembering which diffs were in-progress, so that we didn't
re-launch them.
* Remembering which diffs had failed, so that we didn't try to
recompute them over and over.
* Having a fast way to look up the diff from a given consensus to
the latest consensus of a given flavor.
This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem. It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.
This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.
Initial tests. These just try adding a few consensuses, looking
them up, and making sure that consensus diffs are generated in a
more or less reasonable-looking way. It's enough for 87% coverage,
but it leaves out a lot of functionality.
This module's job is to remember old consensus documents, to
calculate their diffs on demand, and to .
There are some incomplete points in this code; I've marked them with
"XXXX". I intend to fix them in separate commits, since I believe
doing it in separate commits will make the branch easier to review.
The GETINFO extra-info/digest/<digest> broke in commit 568dc27a19 that
refactored the base16_decode() API to return the decoded length.
Unfortunately, that if() condition should have checked for the correct length
instead of an error which broke the command in tor-0.2.9.1-alpha.
Fixes#22034
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit mainly moves the responsibility for directory request
construction one level higher. It also allows a directory request
to contain a pointer to a routerstatus, which will get turned into
the correct contact information at the last minute.
We do dump HS stats now at log info everytime the intro circuit creation retry
period limit has been reached. However, the log was upgraded to warning if we
actually were over the elapsed time (plus an extra slop).
It is actually something that will happen in tor in normal case. For instance,
if the network goes down for 10 minutes then back up again making
have_completed_a_circuit() return false which results in never updating that
retry period marker for a service.
Fixes#22032
Signed-off-by: David Goulet <dgoulet@torproject.org>
This patch makes the internal `get_memlevel()` a part of the public
compression API as `tor_compress_memory_level()`.
See https://bugs.torproject.org/21663
This patch refactors our streaming compression code to allow us to
extend it with non-zlib/non-gzip based compression schemas.
See https://bugs.torproject.org/21663
To allow us to use the API name `tor_compress` and `tor_uncompress` as
the main entry-point for all compression/uncompression and not just gzip
and zlib.
See https://bugs.torproject.org/21663
This patch removes the unused `is_gzip_supported()` and changes the
documentation string around the `compress_method_t` enumeration to
explicitly state that both `ZLIB_METHOD` and `GZIP_METHOD` are both
always supported.
Zlib version 1.2.0 was released on the 9'th of March, 2003 according to
their ChangeLog.
See https://bugs.torproject.org/21663
This patch changes some of the tt_assert() usage in test_util_gzip() to
use tt_int_op() to get better error messages upon failure.
Additionally we move to use explicit NULL checks.
See https://bugs.torproject.org/21663
The consdiff generation logic would skip over lines added at the start of the
second file, and generate a diff that it would the immediately refuse because
it couldn't be used to reproduce the second file from the first. Fixes#21996.
The reason for making the temporary list public is to keep it encapsulated in
the rendservice subsystem so the prop224 code does not have direct access to
it and can only affect it through the rendservice pruning function.
It also has been modified to not take list as arguments but rather use the
global lists (main and temporary ones) because prop224 code will call it to
actually prune the rendservice's lists. The function does the needed rotation
of pointers between those lists and then prune if needed.
In order to make the unit test work and not completely horrible, there is a
"impl_" version of the function that doesn't free memory, it simply moves
pointers around. It is directly used in the unit test and two setter functions
for those lists' pointer have been added only for unit test.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Now we have separate getters and setters for service-side and relay-side. I
took this approach over adding arguments to the already existing methods to
have more explicit type-checking, and also because some functions would grow
too large and dirty.
This commit also fixes every callsite to use the new function names which
modifies the legacy HS (v2) and the prop224 (v3) code.
Signed-off-by: David Goulet <dgoulet@torproject.org>
One of the goals of this change is to have trunnel API/ABI being more explicit
so we namespace them with "trn_*". Furthermore, we can now create
hs_cells.[ch] without having to confuse it with trunnel which used to be
"hs_cell_*" before that change.
Here are the perl line that were used for this rename:
perl -i -pe 's/cell_extension/trn_cell_extension/g;' src/*/*.[ch]
perl -i -pe 's/cell_extension/trn_cell_extension/g;' src/trunnel/hs/*.trunnel
perl -i -pe 's/hs_cell_/trn_cell_/g;' src/*/*.[ch]
perl -i -pe 's/hs_cell_/trn_cell_/g;' src/trunnel/hs/*.trunnel
And then "./scripts/codegen/run_trunnel.sh" with trunnel commit id
613fb1b98e58504e2b84ef56b1602b6380629043.
Fixes#21919
Signed-off-by: David Goulet <dgoulet@torproject.org>
Pinning EntryNodes along with hidden services can be possibly harmful (for
instance #14917 and #21155) so at the very least warn the operator if this is
the case.
Fixes#21155
Signed-off-by: David Goulet <dgoulet@torproject.org>
These tests tried to use ridiculously large buffer sizes to check
the sanity-checking in the code; but since the sanity-checking
changed, these need to change too.
Now that base64_decode() checks the destination buffer length against
the actual number of bytes as they're produced, shared_random.c no
longer needs the "SR_COMMIT_LEN+2" workaround.
Remove base64_decode_nopad() because it is redundant now that
base64_decode() correctly handles both padded and unpadded base64
encodings with "right-sized" output buffers.
Test base64_decode() with odd sized decoded lengths, including
unpadded encodings and padded encodings with "right-sized" output
buffers. Convert calls to base64_decode_nopad() to base64_decode()
because base64_decode_nopad() is redundant.
base64_decode() was applying an overly conservative check on the
output buffer length that could incorrectly produce an error if the
input encoding contained padding or newlines. Fix this by checking
the output buffer length against the actual decoded length produced
during decoding.
When we "fixed" #18280 in 4e4a7d2b0c
in 0291 it appears that we introduced a bug: The base32_encode
function can read off the end of the input buffer, if the input
buffer size modulo 5 is not equal to 0 or 3.
This is not completely horrible, for two reasons:
* The extra bits that are read are never actually used: so this
is only a crash when asan is enabled, in the worst case. Not a
data leak.
* The input sizes passed to base32_encode are only ever multiples
of 5. They are all either DIGEST_LEN (20), REND_SERVICE_ID_LEN
(10), sizeof(rand_bytes) in addressmap.c (10), or an input in
crypto.c that is forced to a multiple of 5.
So this bug can't actually trigger in today's Tor.
Closes bug 21894; bugfix on 0.2.9.1-alpha.
It looks like 32_encoded_size/64_encode_size APIs are inconsistent
not only in the number of "d"s they have, but also in whether they
count the terminating NUL. Taylor noted this in 86477f4e3f,
but I think we should note the inconsistently more loudly in order
to avoid trouble.
(I ran into trouble with this when writing 30b13fd82e243713c6a0d.)
Note down in the routerstatus_t of a node if the router supports the HSIntro=4
version for the ed25519 authentication key and HSDir=2 version for the v3
descriptor supports.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Some of those defines will be used by the v3 HS protocol so move them to a
common header out of rendservice.c. This is also ground work for prop224
service implementation.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Another building blocks for prop224 service work. This also makes the function
takes specific argument instead of the or_option_t object.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When a client tried to connect to an invalid port of an hidden service, a
warning was printed:
[warn] connection_edge_process_relay_cell (at origin) failed.
This is because the connection subsystem wants to close the circuit because
the port can't be found and then returns a negative reason to achieve that.
However, that specific situation triggered a warning. This commit prevents it
for the specific case of an invalid hidden service port.
Fixes#16706
Signed-off-by: David Goulet <dgoulet@torproject.org>
In order to avoid src/or/hs_service.o to contain no symbols and thus making
clang throw a warning, the functions are now exposed not just to unit tests.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Check that route_len_for_purpose() (helper for new_route_len())
correctly fails a non-fatal bug assertion if it encounters an
unhandled circuit purpose when it is called with exit node info.
Add a new helper function route_len_for_purpose(), which explicitly
lists all of the known circuit purposes for a circuit with a chosen
exit node (unlike previously, where the default route length for a
chosen exit was DEFAULT_ROUTE_LEN + 1 except for two purposes). Add a
non-fatal assertion for unhandled purposes that conservatively returns
DEFAULT_ROUTE_LEN + 1.
Add copious comments documenting which circuits need an extra hop and
why.
Thanks to nickm and dgoulet for providing background information.
This change makes it so those those APIs will not require prior
inclusion of openssl headers. I've left some APIs alone-- those
will change to be extra-private.
The old implementation had duplicated code in a bunch of places, and
it interspersed spool-management with resource management. The new
implementation should make it easier to add new resource types and
maintain the spooling code.
Closing ticket 21651.
When calculating max sampled size, Tor would only count the number of
bridges in torrc, without considering that our state file might already
have sampled bridges in it. This caused problems when people swap
bridges, since the following error would trigger:
[warn] Not expanding the guard sample any further; just hit the
maximum sample threshold of 1
