There are two reasons this is likeliest to happen -- no kernel
support, and some bug in Tor. We'll ask people to check the former
before they report. Closes 23090.
By convention, a function that frobs a foo_t should be called
foo_frob, and it should have a foo_t * as its first argument. But
for many of the buf_t functions, the buf_t was the final argument,
which is silly.
Our convention is that functions which manipulate a type T should be
named T_foo. But the buffer functions were super old, and followed
all kinds of conventions. Now they're uniform.
Here's the perl I used to do this:
\#!/usr/bin/perl -w -i -p
s/read_to_buf\(/buf_read_from_socket\(/;
s/flush_buf\(/buf_flush_to_socket\(/;
s/read_to_buf_tls\(/buf_read_from_tls\(/;
s/flush_buf_tls\(/buf_flush_to_tls\(/;
s/write_to_buf\(/buf_add\(/;
s/write_to_buf_compress\(/buf_add_compress\(/;
s/move_buf_to_buf\(/buf_move_to_buf\(/;
s/peek_from_buf\(/buf_peek\(/;
s/fetch_from_buf\(/buf_get_bytes\(/;
s/fetch_from_buf_line\(/buf_get_line\(/;
s/fetch_from_buf_line\(/buf_get_line\(/;
s/buf_remove_from_front\(/buf_drain\(/;
s/peek_buf_startswith\(/buf_peek_startswith\(/;
s/assert_buf_ok\(/buf_assert_ok\(/;
The GNU C Library (glibc) offers an function which allocates the
necessary memory automatically [0]. When it is available, we use that.
Otherwise we depend upon the `getcwd` function which requires a
preallocated buffer (and its size). This function was used incorrectly
by depending on the initial buffer size being big enough and otherwise
failing to return the current working directory. The proper way of
getting the current working directory requires a loop which doubles the
buffer size if `getcwd` requires it. This code was copied from [1] with
modifications to fit the context.
[0] https://www.gnu.org/software/hurd/hurd/porting/guidelines.html
[1] http://pubs.opengroup.org/onlinepubs/9699919799/functions/getcwd.html
We don't actually want Coverity to complain when a BUG() check can
never fail, since such checks can prevent us from introducing bugs
later on.
Closes ticket 23054. Closes CID 1415720, 1415724.
Instead of choosing a lower-priority job with a 1/37 chance, have
the chance be 1/37 for half the threads, and 1/2147483647 for the
other half. This way if there are very slow jobs of low priority,
they shouldn't be able to grab all the threads when there is better
work to do.
Each piece of queued work now has an associated priority value; each
priority goes on a separate queue.
With probability (N-1)/N, the workers will take work from the highest
priority nonempty queue. Otherwise, they'll look for work in a
queue of lower priority. This behavior is meant to prevent
starvation for lower-priority tasks.
In the Linux kernel, the BUG() macro causes an instant panic. Our
BUG() macro is different, however: it generates a nonfatal assertion
failure, and is usable as an expression.
Additionally, this patch tells util_bug.h to make all assertion
failures into fatal conditions when we're building with a static
analysis tool, so that the analysis tool can look for instances
where they're reachable.
Fixes bug 23030.
Wow, it sure seems like some compilers can't implement isnan() and
friends in a way that pleases themselves!
Fixes bug 22915. Bug trigged by 0.2.8.1-alpha and later; caused by
clang 4.
We need to keep these around for TAP and old-style hidden services,
but they're obsolete, and we shouldn't encourage anyone to use them.
So I've added "obsolete" to their names, and a comment explaining
what the problem is.
Closes ticket 23026.
Closes bug 22964. Based on Teor's replacement there, but tries
to put the comment in a more logical place, and explain why we're
actually disabling compression in the first place.
In zstd 1.3.0, once you have called ZSTD_endStream and been told
that your putput buffer is full, it really doesn't want you to call
ZSTD_compressStream again. ZSTD 1.2.0 didn't seem to mind about
this.
This patch fixes the issue by making sure never to call
ZSTD_endStream if there's any more data on the input buffer to
process, by flushing even when we're about to call "endStream", and
by never calling "compress" or "flush" after "endStream".
This change prevents us from generating corrupt messages when we
are confused about codepage settings, and makes Windows errors
consistent with the rest of our logs.
Fixes bug 22520; bugfix on 0.1.2.8-alpha. Patch from "Vort".
- Move some crypto structures so that they are visible by tests.
- Introduce a func to count number of hops in cpath which will be used
by the tests.
- Mark a function as mockable.
The 22081 fix disabled -Wfloat-conversion, but -Wfloat-conversion
didn't exist in every relevant mingw; it was added in GCC 4.9.x some
time, if the documentation can be trusted.
Bug not in any released version of tor.
When setting the maximum number of connections allowed by the OS,
always allow some extra file descriptors for other files.
Fixes bug 22797; bugfix on 0.2.0.10-alpha.
We just have to suppress these warnings: Mingw's math.h uses gcc's
__builtin_choose_expr() facility to declare isnan, isfinite, and
signbit. But as implemented in at least some versions of gcc,
__builtin_choose_expr() can generate type warnings even from
branches that are not taken.
Fixes bug 22801; bugfix on 0.2.8.1-alpha.
This patch fixes a crash in our LZMA module where liblzma will allocate
slightly more data than it is allowed to by its limit, which leads to a
crash.
See: https://bugs.torproject.org/22751
If COMPRESS_OK occurs but data is neither consumed nor generated,
treat it as a BUG and a COMPRESS_ERROR.
This change is meant to prevent infinite loops in the case where
we've made a mistake in one of our compression backends.
Closes ticket 22672.
(This approach can lose accuracy, but it's only in debug-level messages.)
Fixes windows compilation. Bugfix on recent compress.c changes; bug
not in any released Tor.
A fair number of our mock_impl declarations were messed up so that
even our special AM_ETAGSFLAGS couldn't find them.
This should be a whitespace-only patch.
parse_config_line_from_str_verbose() already looks for strings
that are surrounded by quotes, and processes them with
unescape_string(). So things were getting decoded twice, which was
(in turn) playing havoc with backslashes on Windows.
This gives an indication in the log that Tor was built with Rust
support, as well as laying some groundwork for further string-returning
APIs to be converted to Rust
config_get_lines is now split into two functions:
- config_get_lines which is the same as before we had %include
- config_get_lines_include which actually processes %include
This will allow us to treat NO_METHOD as a real compression method,
and to simplify code that currently does
if (compressing) {
compress
} else {
copy
}
Increase the maximum allowed size passed to mprotect(PROT_WRITE)
from 1MB to 16MB. This was necessary with the glibc allocator
in order to allow worker threads to allocate more memory --
which in turn is necessary because of our new use of worker
threads for compression.
Closes ticket #22096. Found while working on #21648.
This patch changes two things in our LZMA compression backend:
- We lower the preset values for all `compression_level_t` values to
ensure that we can run the LZMA decoder with less than 65 MB of memory
available. This seems to have a small impact on the real world usage
and fits well with our needs.
- We set the upper bound of memory usage for the LZMA decoder to 16 MB.
See: https://bugs.torproject.org/21665
This patch adds support for measuring the approximated memory usage by
the individual `tor_zstd_compress_state_t` object instances.
See: https://bugs.torproject.org/22066
This patch fixes the documentation string for `tor_uncompress()` to
ensure that it does not explicitly mention zlib or gzip since we now
support multiple compression backends.
This patch adds support for measuring the approximated memory usage by
the individual `tor_lzma_compress_state_t` object instances.
The LZMA library provides the functions `lzma_easy_encoder_memusage()`
and `lzma_easy_decoder_memusage()` which is used to find the estimated
usage in bytes.
See: https://bugs.torproject.org/22066
We hadn't needed this before, because most getpid() callers on Linux
were looking at the vDSO version of getpid(). I don't know why at
least one version of OpenSSL seems to be ignoring the vDSO, but this
change should fix it.
Fixes bug 21943; bugfix on 0.2.5.1-alpha when the sandbox was
introduced.
The `tor_compress_state_t` data-type is used as a wrapper around the
more specialized state-types used by the various compression backends.
This patch ensures that the overhead of this "thin" wrapper type is
included in the value returned by `tor_compress_get_total_allocation()`.
See: https://bugs.torproject.org/22066
Since we have a streaming API for each compression backend, we don't
need a non-streaming API for each: we can build a common
non-streaming API at the front-end.
This patch renames the `compress` parameter of the
`tor_zlib_compress_new()` function to `_compress` to avoid shadowing the
`compress()` function in zlib.h.
This patch splits up `tor_compress_memory_level()` into static functions
in the individual compression backends, which allows us to tune the
values per compression backend rather than globally.
See: https://bugs.torproject.org/21662
Use a switch-statement in `tor_compress()` and `tor_uncompress()` for
the given `compress_method_t` parameter. This allows us to have the
compiler detect if we forgot add a handler in these functions for a
newly added enumeration value.
See: https://bugs.torproject.org/21662
This patch adds `tor_compress_version_str()` and
`tor_compress_header_version_str()` to get the version strings of the
different compression schema providers. Both functions returns `NULL` in
case a given `compress_method_t` is unknown or unsupported.
See: https://bugs.torproject.org/21662
This patch adds support for checking if a given `compress_method_t` is
supported by the currently running Tor instance using
`tor_compress_supports_method()`.
See: https://bugs.torproject.org/21662
This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.
See: https://bugs.torproject.org/21662
This patch changes the way `tor_compress_new()`,
`tor_compress_process()`, and `tor_compress_free()` handles different
compression methods. This should give us compiler warnings in case an
additional compression method is added, but the developer forgets to add
handlers in the three aforementioned functions.
See https://bugs.torproject.org/21663
This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.
Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.
See https://bugs.torproject.org/21664
This patch makes the internal `get_memlevel()` a part of the public
compression API as `tor_compress_memory_level()`.
See https://bugs.torproject.org/21663
This patch refactors our streaming compression code to allow us to
extend it with non-zlib/non-gzip based compression schemas.
See https://bugs.torproject.org/21663
To allow us to use the API name `tor_compress` and `tor_uncompress` as
the main entry-point for all compression/uncompression and not just gzip
and zlib.
See https://bugs.torproject.org/21663
This patch removes the unused `is_gzip_supported()` and changes the
documentation string around the `compress_method_t` enumeration to
explicitly state that both `ZLIB_METHOD` and `GZIP_METHOD` are both
always supported.
Zlib version 1.2.0 was released on the 9'th of March, 2003 according to
their ChangeLog.
See https://bugs.torproject.org/21663
Remove base64_decode_nopad() because it is redundant now that
base64_decode() correctly handles both padded and unpadded base64
encodings with "right-sized" output buffers.
base64_decode() was applying an overly conservative check on the
output buffer length that could incorrectly produce an error if the
input encoding contained padding or newlines. Fix this by checking
the output buffer length against the actual decoded length produced
during decoding.
When we "fixed" #18280 in 4e4a7d2b0c
in 0291 it appears that we introduced a bug: The base32_encode
function can read off the end of the input buffer, if the input
buffer size modulo 5 is not equal to 0 or 3.
This is not completely horrible, for two reasons:
* The extra bits that are read are never actually used: so this
is only a crash when asan is enabled, in the worst case. Not a
data leak.
* The input sizes passed to base32_encode are only ever multiples
of 5. They are all either DIGEST_LEN (20), REND_SERVICE_ID_LEN
(10), sizeof(rand_bytes) in addressmap.c (10), or an input in
crypto.c that is forced to a multiple of 5.
So this bug can't actually trigger in today's Tor.
Closes bug 21894; bugfix on 0.2.9.1-alpha.
It looks like 32_encoded_size/64_encode_size APIs are inconsistent
not only in the number of "d"s they have, but also in whether they
count the terminating NUL. Taylor noted this in 86477f4e3f,
but I think we should note the inconsistently more loudly in order
to avoid trouble.
(I ran into trouble with this when writing 30b13fd82e243713c6a0d.)
This change makes it so those those APIs will not require prior
inclusion of openssl headers. I've left some APIs alone-- those
will change to be extra-private.
This patch fixes a regression described in bug #21757 that first
appeared after commit 6e78ede73f which was an attempt to fix bug #21654.
When switching from buffered I/O to direct file descriptor I/O our
output strings from get_string_from_pipe() might contain newline
characters (\n). In this patch we modify tor_get_lines_from_handle() to
ensure that the function splits the newly read string at the newline
character and thus might return multiple lines from a single call to
get_string_from_pipe().
Additionally, we add a test case to test_util_string_from_pipe() to
ensure that get_string_from_pipe() correctly returns multiple lines in a
single call.
See: https://bugs.torproject.org/21757
See: https://bugs.torproject.org/21654
We could use one of these for holding "junk" descriptors and
unparseable things -- but we'll _need_ it for having cached
consensuses and diffs between them.
(This commit was extracted by nickm based on the final outcome of
the project, taking only the changes in the files touched by this
commit from the consdiff_rebased branch. The directory-system
changes are going to get worked on separately.)
Windows doesn't let you check the socket error for a socket with
WSAGetLastError() and getsockopt(SO_ERROR). But
getsockopt(SO_ERROR) clears the error on the socket, so you can't
call it more than once per error.
When we introduced recv_ni to help drain alert sockets, back in
0.2.6.3-alpha, we had the failure path for recv_ni call getsockopt()
twice, though: once to check for EINTR and one to check for EAGAIN.
Of course, we never got the eagain, so we treated it as an error,
and warned about: "No error".
The fix here is to have these functions return -errno on failure.
Fixes bug 21540; bugfix on 0.2.6.3-alpha.
So we require that SMARTLIST_FOREACH_END() have the name of the loop
variable in it. But right now the only enforcement for that is to
clear the variable at the end of the loop, which is really not
sufficient: I spent 45 minutes earlier today debugging an issue
where I had said:
SMARTLIST_FOREACH_BEGIN(spool, spooled_resource_t *, spooled) {
...
} SMARTLIST_FOREACH_END(spool);
This patch makes it so that ONLY loop variables can be used, by
referring to the _sl_idx variable.
This patch removes the `tor_fgets()` wrapper around `fgets(3)` since it
is no longer needed. The function was created due to inconsistency
between the returned values of `fgets(3)` on different versions of Unix
when using `fgets(3)` on non-blocking file descriptors, but with the
recent changes in bug #21654 we switch from unbuffered to direct I/O on
non-blocking file descriptors in our utility module.
We continue to use `fgets(3)` directly in the geoip and dirserv module
since this usage is considered safe.
This patch also removes the test-case that was created to detect
differences in the implementation of `fgets(3)` as well as the changes
file since these changes was not included in any releases yet.
See: https://bugs.torproject.org/21654
This patch changes a number of read loops in the util module to use
less-than comparison instead of not-equal-to comparison. We do this in
the case that we have a bug elsewhere that might cause `numread` to
become larger than `count` and thus become an infinite loop.
This patch removes the buffered I/O stream usage in process_handle_t and
its related utility functions. This simplifies the code and avoids racy
code where we used buffered I/O on non-blocking file descriptors.
See: https://bugs.torproject.org/21654
This patch modifies `tor_read_all_handle()` to use read(2) instead of
fgets(3) when reading the stdout from the child process. This should
eliminate the race condition that can be triggered in the 'slow/util/*'
tests on slower machines running OpenBSD, FreeBSD and HardenedBSD.
See: https://bugs.torproject.org/21654
This patch adds the `tor_fgets()` function to our compatibility layer.
`tor_fgets()` adds an additional check for whether the error-bit have
been enabled for the given file stream, if that is the case and `errno`
is set to `EAGAIN` we make sure that we always return NULL.
Unfortunately `fgets(3)` behaves differently on different versions of
the C library.
See: https://bugs.torproject.org/21416
See: https://bugs.torproject.org/20988
This feature makes it possible to turn off memory sentinels (like
those used for safety in buffers.c and memarea.c) when fuzzing, so
that we can catch bugs that they would otherwise prevent.
Since 0.2.4.11-alpha (in 0196647970) we've tried to randomize
the start time to up to some time in the past. But unfortunately we
allowed the start time to be in the future as well, which isn't
really legit.
The new behavior lets the start time be be up to
MAX(cert_lifetime-2days, 0) in the past, but never in the future.
Fixes bug 21420; bugfix on 0.2.4.11-alpha.
Fixes bug 20894; bugfix on 0.2.0.16-alpha.
We already applied a workaround for this as 20834, so no need to
freak out (unless you didn't apply 20384 yet).
maint-0.2.7-redux is an attempt to try to re-create a plausible
maint-0.2.7 branch. I've started from the tor-0.2.7.6, and then I
merged maint-0.2.6 into the branch.
This has produced 2 conflicts: one related to the
rendcommon->rendcache move, and one to the authority refactoring.
Check size argument to memwipe() for underflow.
Closes bug #18089. Reported by "gk", patch by "teor".
Bugfix on 0.2.3.25 and 0.2.4.6-alpha (#7352),
commit 49dd5ef3 on 7 Nov 2012.
If tor_mmap_file is called with a file which is larger than SIZE_MAX,
only a small part of the file will be memory-mapped due to integer
truncation.
This can only realistically happen on 32 bit architectures with large
file support.
Because <unset> makes more sense than AAAAAAAAAAAAAAAAAAA...
(I have indeed verified that ed25519_fmt() is only used for
logging. This patch also clarifies the intention that ed25519_fmt()
is only for logging.
Closes ticket 21037.
This came up on #21035, where somebody tried to build on a linux
system with kernel headers including CLOCK_MONOTONIC_COARSE, then
run on a kernel that didn't support it.
I've adopted a belt-and-suspenders approach here: we detect failures
at initialization time, and we also detect (loudly) failures later on.
Fixes bug 21035; bugfix on 0.2.9.1-alpha when we started using
monotonic time.
I got confused when I saw my Tor saying it was opening a file
that doesn't exist. It turns out it isn't opening it, it's just
calling open() on it and then moving on when it's not there.
This includes:
* making bridge_info_t exposed but opaque
* allowing guards where we don't know an identity
* making it possible to learn the identity of a guard
* creating a guard that lacks a node_t
* remembering a guard's address and port.
* Looking up a guard by address and port.
* Only enforcing the rule that we need a live consensus to update
the "listed" status for guards when we are not using bridges.
They added clock_gettime(), but with tv_nsec as a long, whereas
tv_usec is a __darwin_suseconds_t (a.k.a. 'int'). Now, why would
they do that? Are they preparing for a world where there are more
than 2 billion nanoseconds per second? Are they planning for having
int be less than 32 bits again? Or are they just not paying
attention to the Darwin API?
Also, they forgot to mark clock_gettime() as Sierra-only, so even
if we fixed the issue here, we'd still be stick with portability
breakage like we were for 0.2.9.
So, just disable clock_gettime() on apple.
Tor 0.2.9 has a broader range of fixes and workarounds here, but for
0.2.8, we're just going to maintain the existing behavior.
(The alternative would be to backport both
1eba088054 and
16fcbd21c9 , but the latter is kind of
a subtle kludge in the configure.ac script, and I'm not a fan of
backporting that kind of thing.)
(OpenSSL 1.1 makes EVP_CIPHER_CTX opaque, _and_ adds acceleration
for counter mode on more architectures. So it won't work if we try
the older approach, and it might help if we try the newer one.)
Fixes bug 20588.
It's not okay to use the same varargs list twice, and apparently
some windows build environments produce code here that would leave
tor_asprintf() broken. Fix for bug 20560; bugfix on 0.2.2.11-alpha
when tor_asprintf() was introduced.
(OpenSSL 1.1 makes EVP_CIPHER_CTX opaque, _and_ adds acceleration
for counter mode on more architectures. So it won't work if we try
the older approach, and it might help if we try the newer one.)
Fixes bug 20588.
In our code to write public keys to a string, for some unfathomable
reason since 253f0f160e, we would allocate a memory BIO, then
set the NOCLOSE flag on it, extract its memory buffer, and free it.
Then a little while later we'd free the memory buffer with
BUF_MEM_free().
As of openssl 1.1 this doesn't work any more, since there is now a
BIO_BUF_MEM structure that wraps the BUF_MEM structure. This
BIO_BUF_MEM doesn't get freed in our code.
So, we had a memory leak!
Is this an openssl bug? Maybe. But our code was already pretty
silly. Why mess around with the NOCLOSE flag here when we can just
keep the BIO object around until we don't need the buffer any more?
Fixes bug 20553; bugfix on 0.0.2pre8
This function is allowed to return NULL if the certified key isn't
RSA. But in a couple of places we were treating this as a bug or
internal error, and in one other place we weren't checking for it at
all!
Caught by Isis during code review for #15055. The serious bug was
only on the 15055 branch, thank goodness.
All supported Tors (0.2.4+) require versions of openssl that can
handle this.
Now that our link certificates are RSA2048, this might actually help
vs fingerprinting a little.
See proposal 244. This feature lets us stop looking at the internals
of SSL objects, *and* should let us port better to more SSL libraries,
if they have RFC5705 support.
Preparatory for #19156
Use the following coccinelle script to change uses of
smartlist_add(sl, tor_strdup(str)) to
smartlist_add_strdup(sl, string) (coccinelle script from nickm
via bug 20048):
@@
expression a;
expression b;
@@
- smartlist_add
+ smartlist_add_strdup
(a,
- tor_strdup(
b
- )
)
This commit adds or improves the module-level documenation for:
buffers.c circuitstats.c command.c connection_edge.c control.c
cpuworker.c crypto_curve25519.c crypto_curve25519.h
crypto_ed25519.c crypto_format.c dircollate.c dirserv.c dns.c
dns_structs.h fp_pair.c geoip.c hibernate.c keypin.c ntmain.c
onion.c onion_fast.c onion_ntor.c onion_tap.c periodic.c
protover.c protover.h reasons.c rephist.c replaycache.c
routerlist.c routerparse.c routerset.c statefile.c status.c
tor_main.c workqueue.c
In particular, I've tried to explain (for each documented module)
what each module does, what's in it, what the big idea is, why it
belongs in Tor, and who calls it. In a few cases, I've added TODO
notes about refactoring opportunities.
I've also renamed an argument, and fixed a few DOCDOC comments.
(Specifically, carriage return after a quoted value in a config
line. Fixes bug 19167; bugfix on 0.2.0.16-alpha when we introduced
support for quoted values. Unit tests, changes file, and this
parenthetical by nickm.)
This is a kludge to deal with the fact that `tor_addr_t` doesn't contain
`sun_path`. This currently ONLY happens when circuit isolation is being
checked, for an isolation mode that is force disabled anyway, so the
kludge is "ugly but adequate", but realistically, making `tor_addr_t`
and the AF_UNIX SocksPort code do the right thing is probably the better
option.
