Sometimes when we call exit(), it's because the process is
completely hopeless: openssl has a broken AES-CTR implementation, or
the clock is in the 1960s, or something like that.
But sometimes, we should return cleanly from tor_main() instead, so
that embedders can keep embedding us and start another Tor process.
I've gone through all the exit() and _exit() calls to annotate them
with "exit ok" or "XXXX bad exit" -- the next step will be to fix
the bad exit()s.
First step towards 23848.
Our socket accounting functions assumed that we'd never be asked to
close a socket that we didn't open ourselves. But now we want to
support taking control sockets that we inherit -- so we need a way
of taking ownership of them, so we don't freak out later on when we
close them.
This patch ensures that we return TOR_COMPRESS_BUFFER_FULL in case we
have a input bytes left to process, but are out of output buffer or in
case we need to finish where the compression implementation might need
to write an epilogue.
See: https://bugs.torproject.org/23551
This patch ensures that we return TOR_COMPRESS_BUFFER_FULL in case we
have a input bytes left to process, but are out of output buffer or in
case we need to finish where the compression implementation might need
to write an epilogue.
See: https://bugs.torproject.org/23551
Use this value instead of hardcoded values of 32 everywhere. This also
addresses the use of REND_DESC_ID_V2_LEN_BASE32 in
hs_lookup_last_hid_serv_request() for the HSDir encoded identity digest length
which is accurate but semantically wrong.
Fixes#23305.
Signed-off-by: David Goulet <dgoulet@torproject.org>
There are two reasons this is likeliest to happen -- no kernel
support, and some bug in Tor. We'll ask people to check the former
before they report. Closes 23090.
By convention, a function that frobs a foo_t should be called
foo_frob, and it should have a foo_t * as its first argument. But
for many of the buf_t functions, the buf_t was the final argument,
which is silly.
Our convention is that functions which manipulate a type T should be
named T_foo. But the buffer functions were super old, and followed
all kinds of conventions. Now they're uniform.
Here's the perl I used to do this:
\#!/usr/bin/perl -w -i -p
s/read_to_buf\(/buf_read_from_socket\(/;
s/flush_buf\(/buf_flush_to_socket\(/;
s/read_to_buf_tls\(/buf_read_from_tls\(/;
s/flush_buf_tls\(/buf_flush_to_tls\(/;
s/write_to_buf\(/buf_add\(/;
s/write_to_buf_compress\(/buf_add_compress\(/;
s/move_buf_to_buf\(/buf_move_to_buf\(/;
s/peek_from_buf\(/buf_peek\(/;
s/fetch_from_buf\(/buf_get_bytes\(/;
s/fetch_from_buf_line\(/buf_get_line\(/;
s/fetch_from_buf_line\(/buf_get_line\(/;
s/buf_remove_from_front\(/buf_drain\(/;
s/peek_buf_startswith\(/buf_peek_startswith\(/;
s/assert_buf_ok\(/buf_assert_ok\(/;
The GNU C Library (glibc) offers an function which allocates the
necessary memory automatically [0]. When it is available, we use that.
Otherwise we depend upon the `getcwd` function which requires a
preallocated buffer (and its size). This function was used incorrectly
by depending on the initial buffer size being big enough and otherwise
failing to return the current working directory. The proper way of
getting the current working directory requires a loop which doubles the
buffer size if `getcwd` requires it. This code was copied from [1] with
modifications to fit the context.
[0] https://www.gnu.org/software/hurd/hurd/porting/guidelines.html
[1] http://pubs.opengroup.org/onlinepubs/9699919799/functions/getcwd.html
We don't actually want Coverity to complain when a BUG() check can
never fail, since such checks can prevent us from introducing bugs
later on.
Closes ticket 23054. Closes CID 1415720, 1415724.
Instead of choosing a lower-priority job with a 1/37 chance, have
the chance be 1/37 for half the threads, and 1/2147483647 for the
other half. This way if there are very slow jobs of low priority,
they shouldn't be able to grab all the threads when there is better
work to do.
Each piece of queued work now has an associated priority value; each
priority goes on a separate queue.
With probability (N-1)/N, the workers will take work from the highest
priority nonempty queue. Otherwise, they'll look for work in a
queue of lower priority. This behavior is meant to prevent
starvation for lower-priority tasks.
In the Linux kernel, the BUG() macro causes an instant panic. Our
BUG() macro is different, however: it generates a nonfatal assertion
failure, and is usable as an expression.
Additionally, this patch tells util_bug.h to make all assertion
failures into fatal conditions when we're building with a static
analysis tool, so that the analysis tool can look for instances
where they're reachable.
Fixes bug 23030.
Wow, it sure seems like some compilers can't implement isnan() and
friends in a way that pleases themselves!
Fixes bug 22915. Bug trigged by 0.2.8.1-alpha and later; caused by
clang 4.
We need to keep these around for TAP and old-style hidden services,
but they're obsolete, and we shouldn't encourage anyone to use them.
So I've added "obsolete" to their names, and a comment explaining
what the problem is.
Closes ticket 23026.
Closes bug 22964. Based on Teor's replacement there, but tries
to put the comment in a more logical place, and explain why we're
actually disabling compression in the first place.
In zstd 1.3.0, once you have called ZSTD_endStream and been told
that your putput buffer is full, it really doesn't want you to call
ZSTD_compressStream again. ZSTD 1.2.0 didn't seem to mind about
this.
This patch fixes the issue by making sure never to call
ZSTD_endStream if there's any more data on the input buffer to
process, by flushing even when we're about to call "endStream", and
by never calling "compress" or "flush" after "endStream".
This change prevents us from generating corrupt messages when we
are confused about codepage settings, and makes Windows errors
consistent with the rest of our logs.
Fixes bug 22520; bugfix on 0.1.2.8-alpha. Patch from "Vort".
- Move some crypto structures so that they are visible by tests.
- Introduce a func to count number of hops in cpath which will be used
by the tests.
- Mark a function as mockable.
The 22081 fix disabled -Wfloat-conversion, but -Wfloat-conversion
didn't exist in every relevant mingw; it was added in GCC 4.9.x some
time, if the documentation can be trusted.
Bug not in any released version of tor.
When setting the maximum number of connections allowed by the OS,
always allow some extra file descriptors for other files.
Fixes bug 22797; bugfix on 0.2.0.10-alpha.
We just have to suppress these warnings: Mingw's math.h uses gcc's
__builtin_choose_expr() facility to declare isnan, isfinite, and
signbit. But as implemented in at least some versions of gcc,
__builtin_choose_expr() can generate type warnings even from
branches that are not taken.
Fixes bug 22801; bugfix on 0.2.8.1-alpha.
This patch fixes a crash in our LZMA module where liblzma will allocate
slightly more data than it is allowed to by its limit, which leads to a
crash.
See: https://bugs.torproject.org/22751
If COMPRESS_OK occurs but data is neither consumed nor generated,
treat it as a BUG and a COMPRESS_ERROR.
This change is meant to prevent infinite loops in the case where
we've made a mistake in one of our compression backends.
Closes ticket 22672.
(This approach can lose accuracy, but it's only in debug-level messages.)
Fixes windows compilation. Bugfix on recent compress.c changes; bug
not in any released Tor.
A fair number of our mock_impl declarations were messed up so that
even our special AM_ETAGSFLAGS couldn't find them.
This should be a whitespace-only patch.
parse_config_line_from_str_verbose() already looks for strings
that are surrounded by quotes, and processes them with
unescape_string(). So things were getting decoded twice, which was
(in turn) playing havoc with backslashes on Windows.
This gives an indication in the log that Tor was built with Rust
support, as well as laying some groundwork for further string-returning
APIs to be converted to Rust
config_get_lines is now split into two functions:
- config_get_lines which is the same as before we had %include
- config_get_lines_include which actually processes %include
This will allow us to treat NO_METHOD as a real compression method,
and to simplify code that currently does
if (compressing) {
compress
} else {
copy
}
Increase the maximum allowed size passed to mprotect(PROT_WRITE)
from 1MB to 16MB. This was necessary with the glibc allocator
in order to allow worker threads to allocate more memory --
which in turn is necessary because of our new use of worker
threads for compression.
Closes ticket #22096. Found while working on #21648.
This patch changes two things in our LZMA compression backend:
- We lower the preset values for all `compression_level_t` values to
ensure that we can run the LZMA decoder with less than 65 MB of memory
available. This seems to have a small impact on the real world usage
and fits well with our needs.
- We set the upper bound of memory usage for the LZMA decoder to 16 MB.
See: https://bugs.torproject.org/21665
This patch adds support for measuring the approximated memory usage by
the individual `tor_zstd_compress_state_t` object instances.
See: https://bugs.torproject.org/22066
This patch fixes the documentation string for `tor_uncompress()` to
ensure that it does not explicitly mention zlib or gzip since we now
support multiple compression backends.
This patch adds support for measuring the approximated memory usage by
the individual `tor_lzma_compress_state_t` object instances.
The LZMA library provides the functions `lzma_easy_encoder_memusage()`
and `lzma_easy_decoder_memusage()` which is used to find the estimated
usage in bytes.
See: https://bugs.torproject.org/22066
We hadn't needed this before, because most getpid() callers on Linux
were looking at the vDSO version of getpid(). I don't know why at
least one version of OpenSSL seems to be ignoring the vDSO, but this
change should fix it.
Fixes bug 21943; bugfix on 0.2.5.1-alpha when the sandbox was
introduced.
The `tor_compress_state_t` data-type is used as a wrapper around the
more specialized state-types used by the various compression backends.
This patch ensures that the overhead of this "thin" wrapper type is
included in the value returned by `tor_compress_get_total_allocation()`.
See: https://bugs.torproject.org/22066
Since we have a streaming API for each compression backend, we don't
need a non-streaming API for each: we can build a common
non-streaming API at the front-end.
This patch renames the `compress` parameter of the
`tor_zlib_compress_new()` function to `_compress` to avoid shadowing the
`compress()` function in zlib.h.
This patch splits up `tor_compress_memory_level()` into static functions
in the individual compression backends, which allows us to tune the
values per compression backend rather than globally.
See: https://bugs.torproject.org/21662
Use a switch-statement in `tor_compress()` and `tor_uncompress()` for
the given `compress_method_t` parameter. This allows us to have the
compiler detect if we forgot add a handler in these functions for a
newly added enumeration value.
See: https://bugs.torproject.org/21662
This patch adds `tor_compress_version_str()` and
`tor_compress_header_version_str()` to get the version strings of the
different compression schema providers. Both functions returns `NULL` in
case a given `compress_method_t` is unknown or unsupported.
See: https://bugs.torproject.org/21662
This patch adds support for checking if a given `compress_method_t` is
supported by the currently running Tor instance using
`tor_compress_supports_method()`.
See: https://bugs.torproject.org/21662
This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.
See: https://bugs.torproject.org/21662
This patch changes the way `tor_compress_new()`,
`tor_compress_process()`, and `tor_compress_free()` handles different
compression methods. This should give us compiler warnings in case an
additional compression method is added, but the developer forgets to add
handlers in the three aforementioned functions.
See https://bugs.torproject.org/21663
This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.
Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.
See https://bugs.torproject.org/21664
This patch makes the internal `get_memlevel()` a part of the public
compression API as `tor_compress_memory_level()`.
See https://bugs.torproject.org/21663
This patch refactors our streaming compression code to allow us to
extend it with non-zlib/non-gzip based compression schemas.
See https://bugs.torproject.org/21663
To allow us to use the API name `tor_compress` and `tor_uncompress` as
the main entry-point for all compression/uncompression and not just gzip
and zlib.
See https://bugs.torproject.org/21663
This patch removes the unused `is_gzip_supported()` and changes the
documentation string around the `compress_method_t` enumeration to
explicitly state that both `ZLIB_METHOD` and `GZIP_METHOD` are both
always supported.
Zlib version 1.2.0 was released on the 9'th of March, 2003 according to
their ChangeLog.
See https://bugs.torproject.org/21663
Remove base64_decode_nopad() because it is redundant now that
base64_decode() correctly handles both padded and unpadded base64
encodings with "right-sized" output buffers.
base64_decode() was applying an overly conservative check on the
output buffer length that could incorrectly produce an error if the
input encoding contained padding or newlines. Fix this by checking
the output buffer length against the actual decoded length produced
during decoding.
When we "fixed" #18280 in 4e4a7d2b0c
in 0291 it appears that we introduced a bug: The base32_encode
function can read off the end of the input buffer, if the input
buffer size modulo 5 is not equal to 0 or 3.
This is not completely horrible, for two reasons:
* The extra bits that are read are never actually used: so this
is only a crash when asan is enabled, in the worst case. Not a
data leak.
* The input sizes passed to base32_encode are only ever multiples
of 5. They are all either DIGEST_LEN (20), REND_SERVICE_ID_LEN
(10), sizeof(rand_bytes) in addressmap.c (10), or an input in
crypto.c that is forced to a multiple of 5.
So this bug can't actually trigger in today's Tor.
Closes bug 21894; bugfix on 0.2.9.1-alpha.
It looks like 32_encoded_size/64_encode_size APIs are inconsistent
not only in the number of "d"s they have, but also in whether they
count the terminating NUL. Taylor noted this in 86477f4e3f,
but I think we should note the inconsistently more loudly in order
to avoid trouble.
(I ran into trouble with this when writing 30b13fd82e243713c6a0d.)
This change makes it so those those APIs will not require prior
inclusion of openssl headers. I've left some APIs alone-- those
will change to be extra-private.
This patch fixes a regression described in bug #21757 that first
appeared after commit 6e78ede73f which was an attempt to fix bug #21654.
When switching from buffered I/O to direct file descriptor I/O our
output strings from get_string_from_pipe() might contain newline
characters (\n). In this patch we modify tor_get_lines_from_handle() to
ensure that the function splits the newly read string at the newline
character and thus might return multiple lines from a single call to
get_string_from_pipe().
Additionally, we add a test case to test_util_string_from_pipe() to
ensure that get_string_from_pipe() correctly returns multiple lines in a
single call.
See: https://bugs.torproject.org/21757
See: https://bugs.torproject.org/21654
We could use one of these for holding "junk" descriptors and
unparseable things -- but we'll _need_ it for having cached
consensuses and diffs between them.
(This commit was extracted by nickm based on the final outcome of
the project, taking only the changes in the files touched by this
commit from the consdiff_rebased branch. The directory-system
changes are going to get worked on separately.)
Windows doesn't let you check the socket error for a socket with
WSAGetLastError() and getsockopt(SO_ERROR). But
getsockopt(SO_ERROR) clears the error on the socket, so you can't
call it more than once per error.
When we introduced recv_ni to help drain alert sockets, back in
0.2.6.3-alpha, we had the failure path for recv_ni call getsockopt()
twice, though: once to check for EINTR and one to check for EAGAIN.
Of course, we never got the eagain, so we treated it as an error,
and warned about: "No error".
The fix here is to have these functions return -errno on failure.
Fixes bug 21540; bugfix on 0.2.6.3-alpha.
So we require that SMARTLIST_FOREACH_END() have the name of the loop
variable in it. But right now the only enforcement for that is to
clear the variable at the end of the loop, which is really not
sufficient: I spent 45 minutes earlier today debugging an issue
where I had said:
SMARTLIST_FOREACH_BEGIN(spool, spooled_resource_t *, spooled) {
...
} SMARTLIST_FOREACH_END(spool);
This patch makes it so that ONLY loop variables can be used, by
referring to the _sl_idx variable.
This patch removes the `tor_fgets()` wrapper around `fgets(3)` since it
is no longer needed. The function was created due to inconsistency
between the returned values of `fgets(3)` on different versions of Unix
when using `fgets(3)` on non-blocking file descriptors, but with the
recent changes in bug #21654 we switch from unbuffered to direct I/O on
non-blocking file descriptors in our utility module.
We continue to use `fgets(3)` directly in the geoip and dirserv module
since this usage is considered safe.
This patch also removes the test-case that was created to detect
differences in the implementation of `fgets(3)` as well as the changes
file since these changes was not included in any releases yet.
See: https://bugs.torproject.org/21654
This patch changes a number of read loops in the util module to use
less-than comparison instead of not-equal-to comparison. We do this in
the case that we have a bug elsewhere that might cause `numread` to
become larger than `count` and thus become an infinite loop.
This patch removes the buffered I/O stream usage in process_handle_t and
its related utility functions. This simplifies the code and avoids racy
code where we used buffered I/O on non-blocking file descriptors.
See: https://bugs.torproject.org/21654
This patch modifies `tor_read_all_handle()` to use read(2) instead of
fgets(3) when reading the stdout from the child process. This should
eliminate the race condition that can be triggered in the 'slow/util/*'
tests on slower machines running OpenBSD, FreeBSD and HardenedBSD.
See: https://bugs.torproject.org/21654
This patch adds the `tor_fgets()` function to our compatibility layer.
`tor_fgets()` adds an additional check for whether the error-bit have
been enabled for the given file stream, if that is the case and `errno`
is set to `EAGAIN` we make sure that we always return NULL.
Unfortunately `fgets(3)` behaves differently on different versions of
the C library.
See: https://bugs.torproject.org/21416
See: https://bugs.torproject.org/20988
This feature makes it possible to turn off memory sentinels (like
those used for safety in buffers.c and memarea.c) when fuzzing, so
that we can catch bugs that they would otherwise prevent.
Since 0.2.4.11-alpha (in 0196647970) we've tried to randomize
the start time to up to some time in the past. But unfortunately we
allowed the start time to be in the future as well, which isn't
really legit.
The new behavior lets the start time be be up to
MAX(cert_lifetime-2days, 0) in the past, but never in the future.
Fixes bug 21420; bugfix on 0.2.4.11-alpha.
Fixes bug 20894; bugfix on 0.2.0.16-alpha.
We already applied a workaround for this as 20834, so no need to
freak out (unless you didn't apply 20384 yet).
