Commit Graph

18846 Commits

Author SHA1 Message Date
Nick Mathewson
ec84fc1d8e Improve documentation on get_{peer,own}_certificate()
Make it clear that we're returning a newly allocated copy.
2017-06-05 15:27:33 -04:00
Nick Mathewson
8e9392c267 Repair link_handshake unit tests to mock tor_tls_get_own_cert()
The tests previously assumed that the link handshake code would be
calling get_my_certs() -- when I changed it to call get_own_cert()
instead for the (case 2) 22460 fix, the tests failed, since the tls
connection wasn't really there.

This change makes us start mocking out the tor_tls_get_own_cert()
function too.

It also corrects the behavior of the mock_get_peer_cert() function
-- it should have been returning a newly allocated copy.
2017-06-05 15:27:33 -04:00
Nick Mathewson
39b7e89c28 Test prerequisites: function to dup a cert, make get_own_cert mockable. 2017-06-05 15:27:33 -04:00
Nick Mathewson
50facb40bb On v3 link handshake, send the correct link certificate
Previously we'd send the _current_ link certificate, which would
cause a handshaking failure when the TLS context rotated.
2017-06-05 15:27:33 -04:00
Nick Mathewson
2e5220cb8b Merge branch 'maint-0.2.4' into maint-0.2.5 2017-06-05 14:38:54 -04:00
Nick Mathewson
4ee48cb434 Fix C89 warning (since Tor 0.2.4-5 still care about that.) 2017-06-05 14:38:38 -04:00
Nick Mathewson
db2f18b1f9 Merge branch 'maint-0.3.0' 2017-06-05 12:02:47 -04:00
Nick Mathewson
578a4392e9 Merge branch 'maint-0.2.9' into maint-0.3.0 2017-06-05 12:02:26 -04:00
Nick Mathewson
d75be189df Merge branch 'maint-0.2.8' into maint-0.2.9 2017-06-05 12:02:15 -04:00
Nick Mathewson
33fcc0f61d Merge branch 'maint-0.2.7-redux' into maint-0.2.8 2017-06-05 12:01:17 -04:00
Nick Mathewson
3f2d1f7f07 Merge branch 'maint-0.2.6' into maint-0.2.7-redux 2017-06-05 12:00:41 -04:00
Nick Mathewson
9ea3d0877a Merge branch 'maint-0.2.5' into maint-0.2.6 2017-06-05 12:00:27 -04:00
Nick Mathewson
1a540b5792 Merge branch 'maint-0.2.4' into maint-0.2.5 2017-06-05 12:00:08 -04:00
Nick Mathewson
e3ebae4804 Fix undefined behavior in geoip_parse_entry().
Fixes bug 22490; bugfix on 6a241ff3ff in 0.2.4.6-alpha.

Found by teor using clang-5.0's AddressSanitizer stack-use-after-scope.
2017-06-05 10:09:39 -04:00
Nick Mathewson
26d9fffae4 Merge branch 'bug22466_diagnostic_030' 2017-06-05 09:52:09 -04:00
Nick Mathewson
be741d7e63 Merge branch 'maint-0.3.0' 2017-06-05 09:51:57 -04:00
Nick Mathewson
e5bdfd66cf Make code more clear about own_link_cert safety
It's okay to call add_ed25519_cert with a NULL argument: so,
document that.  Also, add a tor_assert_nonfatal() to catch any case
where we have failed to set own_link_cert when conn_in_server_mode.
2017-06-05 09:35:55 -04:00
rl1987
f8c98759e5 Use string_is_valid_hostname in SOCKS4 request parsing codepath 2017-06-04 13:22:45 +02:00
rl1987
7f05f89663 Don't reject SOCKS5 requests that contain IP strings 2017-06-04 13:14:55 +02:00
rl1987
9e2f780923 Refrain from needless SOCKS5 warning 2017-06-03 18:04:47 +02:00
Nick Mathewson
41ed9e978b Regenerate RSA->ed25519 identity crosscertificate as needed 2017-06-01 10:04:52 -04:00
Nick Mathewson
f2068ef862 Use tor_assert_nonfatal() to try to detect #22466 2017-06-01 09:42:32 -04:00
Nick Mathewson
34a6755b94 Fix ed25519 link certificate race on tls context rotation
Whenever we rotate our TLS context, we change our Ed25519
Signing->Link certificate.  But if we've already started a TLS
connection, then we've already sent the old X509 link certificate,
so the new Ed25519 Signing->Link certificate won't match it.

To fix this, we now store a copy of the Signing->Link certificate
when we initialize the handshake state, and send that certificate
as part of our CERTS cell.

Fixes one case of bug22460; bugfix on 0.3.0.1-alpha.
2017-06-01 09:26:24 -04:00
Nick Mathewson
a9be768959 Bugfix: Regenerate more certificates when appropriate
Previously we could sometimes change our signing key, but not
regenerate the certificates (signing->link and signing->auth) that
were signed with it.  Also, we would regularly replace our TLS x.509
link certificate (by rotating our TLS context) but not replace our
signing->link ed25519 certificate.  In both cases, the resulting
inconsistency would make other relays reject our link handshakes.

Fixes two cases of bug 22460; bugfix on 0.3.0.1-alpha.
2017-05-31 18:45:35 -04:00
Andreas Stieger
1763aa058b Fix GCC 7 -Wimplicit-fallthrough warnings (32 bit)
Add magic comments recognized by default -Wimplicit-fallthrough=3
Follow-up to e5f464, fixes Ticket 22446 for 32 bit.
2017-05-31 09:30:35 -04:00
Nick Mathewson
9d59769db7 Improve error message when all permitted Exits are down
The old "No specified non-excluded exit routers seem to be running"
message was somewhat confusing.

Fix for 7890.
2017-05-30 10:59:04 -04:00
David Goulet
5b33d95a3d hs: Correctly validate v3 descriptor encrypted length
The encrypted_data_length_is_valid() function wasn't validating correctly the
length of the encrypted data of a v3 descriptor. The side effect of this is
that an HSDir was rejecting the descriptor and ultimately not storing it.

Fixes #22447

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-30 10:27:42 -04:00
Nick Mathewson
671c5dcde3 Merge remote-tracking branch 'public/bug6298' 2017-05-30 10:27:37 -04:00
Nick Mathewson
0fbe1a2c6f Merge remote-tracking branch 'teor/bug22424' 2017-05-30 08:50:45 -04:00
Nick Mathewson
184d889f8a Merge remote-tracking branch 'jigsaw/fix-22417-without-3-star' 2017-05-30 08:47:14 -04:00
Nick Mathewson
f9615f9d77 Merge remote-tracking branch 'teor/bug22421' 2017-05-30 08:42:20 -04:00
Andreas Stieger
e5f4642db3 Fix GCC 7 -Wimplicit-fallthrough warnings
Add magic comments recognized by default -Wimplicit-fallthrough=3
or break, as required.
2017-05-30 08:33:27 -04:00
Roger Dingledine
d1580ad49b remove obsolete comment
we should have taken out this comment with commit aadff6274
during ticket 16480.
2017-05-30 02:42:32 -04:00
Daniel Pinto
94d321120e Replace 3-star pointer with 2-star pointer 2017-05-28 20:24:48 +01:00
teor
79725289e1
If we do underflow the know usage of a storage, recalculate it
Fixes bug #22424 on 0.3.1.1-alpha.
2017-05-28 22:34:43 +10:00
teor
69b234a0a8
Refactor storage usage reductions into a static function
No behaviour change.

Part of #22424.
2017-05-28 22:28:43 +10:00
teor
334fe6bb6b
Don't underflow usage when it is unknown and a file is removed
Part of #22424.
2017-05-28 22:16:00 +10:00
teor
9e36b0beb9
Always check for usage underflow when removing a file in storage.c
Part of #22424.
2017-05-28 22:12:09 +10:00
teor
f6841ae263
Fix comment typos in storage.c 2017-05-28 22:11:22 +10:00
teor
25ea8be9de
Update the client bootstrap comment in config.c for exponential backoff
This brings the description up to date with the exponential backoff
code introduced in 0.2.9.1-alpha.

Fixes bug #22421.
2017-05-28 21:01:08 +10:00
Daniel Pinto
f8ccf8d9a9 Fix crash with %include
Fixes crash when including a folder that contains a non-empty file
without any values followed by any other non-empty file.
2017-05-28 09:53:14 +01:00
Roger Dingledine
084b64ba2e simplify because relay_crypt_one_payload can't fail 2017-05-28 01:51:22 -04:00
Nick Mathewson
6fcaf83c98 Cleanup MOCK_IMPL (etc) to be findable with etags
A fair number of our mock_impl declarations were messed up so that
even our special AM_ETAGSFLAGS couldn't find them.

This should be a whitespace-only patch.
2017-05-26 14:07:06 -04:00
Nick Mathewson
159a8061cf Mock tor_addr_lookup() during part of addr/basic test.
If this function isn't mocked, then our unit tests break on stupid
networks where localhost is broken or absent. Fixes bug 6298; bugfix
on 0.0.9pre2.
2017-05-26 13:53:32 -04:00
Nick Mathewson
5742e4fd8e bump version to 0.3.1.2-alpha-dev 2017-05-26 10:23:53 -04:00
Nick Mathewson
2550ac7bb1 update version to 0.3.1.2-alpha 2017-05-26 08:58:08 -04:00
Nick Mathewson
ab9976b724 Merge remote-tracking branch 'arma/bug22368' 2017-05-25 08:54:51 -04:00
Roger Dingledine
657297a9f8 Merge branch 'maint-0.3.0' 2017-05-25 00:28:11 -04:00
Roger Dingledine
83439e78cc Merge branch 'maint-0.2.9' into maint-0.3.0 2017-05-25 00:27:27 -04:00
teor
ec61ae59a5 Stop leaking keypin-rejected routerinfos on directory authorities
When directory authorities reject a router descriptor due to keypinning,
free the router descriptor rather than leaking the memory.

Fixes bug 22370; bugfix on 0.2.7.2-alpha.
2017-05-25 00:09:40 -04:00
Roger Dingledine
5f74749fba get rid of some dead code (leftover from commit fa04fe1) 2017-05-24 23:37:00 -04:00
Roger Dingledine
d22d565331 add copy of MyFamily element to the descriptor, not the element itself
If we add the element itself, we will later free it when we free the
descriptor, and the next time we go to look at MyFamily, things will
go badly.

Fixes the rest of bug 22368; bugfix on 0.3.1.1-alpha.
2017-05-24 23:37:00 -04:00
Roger Dingledine
a7e75ff796 don't free the values in options->MyFamily when we make a descriptor
If we free them here, we will still attempt to access the freed memory
later on, and also we will double-free when we are freeing the config.

Fixes part of bug 22368.
2017-05-24 23:32:32 -04:00
Nick Mathewson
511c900686 Merge branch 'master' of git-rw.torproject.org:/tor 2017-05-24 10:25:00 -04:00
Nick Mathewson
b80a35e683 Improve the message we log on unexpected dirauth status code
It's still not great, but should be less confusing what's wrong
here.

Closes ticket 1121.
2017-05-24 09:08:59 -04:00
Roger Dingledine
cabad0b6c7 remove unused node_get_published_on function
in retrospect, we should have removed this with commit 1289474d
as part of #11742, which was the last caller of it.
2017-05-24 00:43:44 -04:00
Nick Mathewson
994595ae5d Don't say "downloading 1 descriptor, 4 at a time"
Fixes bug 19648, bugfix on 0.1.1.8-alpha.
2017-05-23 09:47:11 -04:00
teor
af98b862a5
Fix comments of functions that return tor_snprintf
No code changes needed: in the places where we actually check the
return value of these functions, we handle it correctly.
2017-05-23 18:44:45 +10:00
Roger Dingledine
4e3ea6d5c8 fix minor grammar error in comment 2017-05-22 20:06:38 -04:00
Nick Mathewson
d9b1471303 Merge remote-tracking branch 'ahf/bugs/22305' 2017-05-22 11:57:03 -04:00
Alexander Færøy
9604980733 Log a warning if we receive a disallowed compression method for an anonymous connection.
See: https://bugs.torproject.org/22305
2017-05-22 15:52:41 +00:00
Nick Mathewson
281f06a094 bump to 0.3.1.1-alpha 2017-05-22 11:52:19 -04:00
Nick Mathewson
9a50c73104 Merge remote-tracking branch 'ahf/bugs/22305' 2017-05-22 10:57:26 -04:00
Alexander Færøy
5a0eab68e1 Ensure that only GZip and Zlib compression is handled for anonymous connections.
See: https://bugs.torproject.org/22305
2017-05-22 14:45:12 +00:00
Alexander Færøy
2b26ac1390 Refactor error path handling in connection_dir_client_reached_eof().
This patch lifts the return value, rv, variable to the beginning of the
function, adds a 'done' label for clean-up and function exit and makes
the rest of the function use the rv value + goto done; instead of
cleaning up in multiple places.

See: https://bugs.torproject.org/22305
2017-05-22 14:42:18 +00:00
Nick Mathewson
cfe0a45750 Fix a compilation warning about duplicate typedef 2017-05-22 10:39:43 -04:00
Nick Mathewson
e5a929fef8 Raise common code for creating circuit_guard_state_t
This will help if we ever need to add more fields or change the
semantics of existing fields.
2017-05-22 09:13:18 -04:00
Nick Mathewson
b2e9a107b7 Merge remote-tracking branch 'asn/bug21969_bridges' 2017-05-22 09:09:16 -04:00
George Kadianakis
52498b8183 Set guard state on bridge descriptor fetches.
We used to not set the guard state in launch_direct_bridge_descriptor_fetch().
So when a bridge descriptor fetch failed, the guard subsystem would never
learn about the fail (and hence the guard's reachability state would not
be updated).
2017-05-22 15:57:33 +03:00
George Kadianakis
6009c89165 Set guard state on bridge descriptor fetches.
We used to not set the guard state in launch_direct_bridge_descriptor_fetch().
So when a bridge descriptor fetch failed, the guard subsystem would never
learn about the fail (and hence the guard's reachability state would not
be updated).
2017-05-22 15:56:32 +03:00
George Kadianakis
e102ad60d0 Refactor directory_initiate_command() so that it accepts guard_state. 2017-05-22 15:45:46 +03:00
Alexander Færøy
26795da900 Don't add "Accept-Encoding" header if directory connection is anonymous.
See: https://bugs.torproject.org/22305
2017-05-22 12:36:27 +00:00
Nick Mathewson
90894c87a5 Merge branch 'maint-0.3.0' 2017-05-22 08:32:18 -04:00
Nick Mathewson
5c52d3c2c0 Merge branch 'maint-0.2.9' into maint-0.3.0 2017-05-22 08:32:07 -04:00
Roger Dingledine
6e5486b11a dir auths reject 0.2.9.x for x<5, due to bug 20499
Directory authorities now reject relays running versions
0.2.9.1-alpha through 0.2.9.4-alpha, because those relays
suffer from bug 20499 and don't keep their consensus cache
up-to-date.

Resolves ticket 20509.
2017-05-22 08:31:39 -04:00
Roger Dingledine
0698a0beca fix two typos in comments
closes ticket 22322
2017-05-22 01:43:52 -04:00
Roger Dingledine
a18b41cc77 fix typo in comment 2017-05-19 22:14:56 -04:00
Nick Mathewson
09b3cb0d72 Whoops; I broke check-spaces. 2017-05-19 16:02:58 -04:00
Nick Mathewson
13034e1574 Try another approach to fixing the bug in the #1922 test.
This isn't elegant, but it seems to be the best way around all of
the issues involved in escaping and quoting that we've gotten into
over the years.
2017-05-19 15:50:29 -04:00
Nick Mathewson
19615bce64 mingw fix: avoid "unused var" warning. 2017-05-19 15:50:12 -04:00
Nick Mathewson
8cb08f716d fix a coverity resource leak warning 2017-05-19 15:20:57 -04:00
Nick Mathewson
4e6b13a38a Fix uninitialized-variable warning on options_init_from_string(). 2017-05-19 15:18:27 -04:00
Nick Mathewson
d950ad0dfd Remove call to get_unquoted_path in config_process_include()
parse_config_line_from_str_verbose() already looks for strings
that are surrounded by quotes, and processes them with
unescape_string().  So things were getting decoded twice, which was
(in turn) playing havoc with backslashes on Windows.
2017-05-19 14:09:51 -04:00
Nick Mathewson
3628efe29c Try to fix windows config/include_path_syntax test
It was trying to do %include "foo\", which won't work.  It has to be
%include "foo\\".
2017-05-19 09:55:22 -04:00
Nick Mathewson
dcfed8c1c8 tt_* macros can "goto done;" so define any freeable things before them. 2017-05-19 08:55:14 -04:00
Nick Mathewson
ff1af5550a strlen() returns size_t 2017-05-19 08:54:56 -04:00
Nick Mathewson
69ef94820b Merge branch 'add_rust_squashed' 2017-05-19 08:47:18 -04:00
Sebastian Hahn
70c067102b Allow Rust build using locally supplied crates or crates.io
This adds a couple of configure commands to control whether we're
requiring all dependencies to be available locally (default) or not
(--enable-cargo-online-mode). When building from a tarball, we require
the RUST_DEPENDENCIES variable to point to the local repository of
crates. This also adds src/ext/rust as a git submodule that contains
such a local repository for easy setup.
2017-05-19 08:47:11 -04:00
Sebastian Hahn
d6f9a4f11a cargo-online-mode configure argument
Passing --enable-cargo-online-mode during configure allows cargo to make
network requests while building Tor or running tests. If this flag is
not supplied, the dependencies need to be available in the form of a
local mirror.
2017-05-19 08:47:11 -04:00
Sebastian Hahn
ed15511ac3 Add rustfmt.toml 2017-05-19 08:47:11 -04:00
Sebastian Hahn
f0516ed348 Run cargo test during make check 2017-05-19 08:47:11 -04:00
Sebastian Hahn
f8ef7c65d1 Add some Rust utility functions and print support
This gives an indication in the log that Tor was built with Rust
support, as well as laying some groundwork for further string-returning
APIs to be converted to Rust
2017-05-19 08:47:10 -04:00
Nick Mathewson
92d335b3dc Merge remote-tracking branch 'jigsaw/torrc-dir-fix-1922_squashed2' 2017-05-19 08:46:13 -04:00
Nick Mathewson
d34fa32ece Merge branch 'ticket21953_029' 2017-05-19 06:49:04 -04:00
Nick Mathewson
15cc41e664 Define HeapEnableTerminationOnCorruption if the headers don't.
MSDN says that it's always going to be 1, and they're usually pretty
accurate about that.

Fixes a bug in 21953.
2017-05-19 06:46:49 -04:00
Roger Dingledine
553cd7f0c5 fix typo 2017-05-19 02:06:44 -04:00
Daniel Pinto
ba3a5f82f1 Add support for %include funcionality on torrc #1922
config_get_lines is now split into two functions:
 - config_get_lines which is the same as before we had %include
 - config_get_lines_include which actually processes %include
2017-05-18 23:44:16 +01:00
Nick Mathewson
d1340bd5ac bump to 0.3.0.7-dev 2017-05-18 12:28:28 -04:00
Nick Mathewson
b214c2c095 Merge branch 'maint-0.3.0' 2017-05-18 10:06:27 -04:00
Nick Mathewson
935cd77f62 Merge branch 'bug22252_029' into maint-0.3.0 2017-05-18 10:06:06 -04:00
Daniel Pinto
e04da2828d Fixed error on BASEXX_NOPAD LEN and BUFSIZE macros #21872 2017-05-17 00:22:11 +01:00
Nick Mathewson
2ba58f275c Fix crash when starting with LearnCircuitBuildTimeout 0.
Before we've set our options, we can neither call get_options() nor
networkstatus_get_latest_consensus().

Fixes bug 22252; bugfix on 4d9d2553ba
in 0.2.9.3-alpha.
2017-05-16 11:20:12 -04:00
Nick Mathewson
bbeba2412e Fix resource leak in parse_consensus_request()
We were allocating diff_hash_in_url on some URLs, but not freeing it.

Fixes CID 1409669.  Bug not in any released Tor.
2017-05-16 10:47:41 -04:00
Nick Mathewson
2ca827104d Report deleted consensuses as NOT_FOUND rather than AVAILABLE
This bug happened because of a bogus pointer check in
consdiffmgr_find_consensus(), not in any released Tor.

Fixes CID 1409670.  Good catch, Coverity!
2017-05-16 10:44:24 -04:00
Nick Mathewson
d29f494ec2 Merge branch 'maint-0.3.0' 2017-05-16 08:39:36 -04:00
Nick Mathewson
492f8a7c44 Merge branch 'maint-0.2.9' into maint-0.3.0 2017-05-16 08:39:22 -04:00
Nick Mathewson
a7bcab2639 Merge branch 'maint-0.2.8' into maint-0.2.9 2017-05-16 08:38:59 -04:00
teor
5b45d73293
Update fallback directory mirrors in May 2017
Replace the 177 fallbacks originally introduced in Tor 0.2.9.8 in
December 2016 (of which ~126 were still functional), with a list of
151 fallbacks (32 new, 119 existing, 58 removed) generated in May 2017.

Resolves ticket 21564.
2017-05-16 19:02:42 +10:00
Nick Mathewson
8324631dd9 bump version to 0.3.0.7 2017-05-15 18:25:07 -04:00
Nick Mathewson
a546487287 Merge branch 'maint-0.3.0' 2017-05-15 18:24:38 -04:00
Nick Mathewson
294d80044d remove a variable I missed 2017-05-15 18:16:58 -04:00
Nick Mathewson
d3279d4304 Do not try to uncompress an empty spool 2017-05-15 18:13:38 -04:00
Nick Mathewson
460b923026 fix a memory leak 2017-05-15 18:13:20 -04:00
Nick Mathewson
e1b3c1bbec Test fix: always set address in new_dir_conn() 2017-05-15 18:00:38 -04:00
Nick Mathewson
f9d8ade912 Dircache protocol version 2 adds support for diffs 2017-05-15 17:53:36 -04:00
Nick Mathewson
da6b00443c Try not to mess up caches with the X-Or-Diff-From-Consensus header 2017-05-15 17:53:15 -04:00
Nick Mathewson
4531fdbbff Split consensus-request parsing into a separate function
This ought to make the control flow a tiny bit more readable.
2017-05-15 17:51:53 -04:00
Nick Mathewson
eb3c8d376d Prop140, continued: accept "diff/<HASH>" in URLs, per proposal. 2017-05-15 17:42:17 -04:00
Nick Mathewson
afef059795 Merge remote-tracking branch 'public/prop140_aftermath_cfg' 2017-05-15 17:26:47 -04:00
Nick Mathewson
dae1242532 Merge branch 'ahf_bugs_21667_2_squashed' 2017-05-15 17:22:12 -04:00
Alexander Færøy
9e3f304113 Fix dir_handle_get/... test-cases for prop#278 support.
See: https://bugs.torproject.org/21667
2017-05-15 17:21:55 -04:00
Nick Mathewson
65d9408448 dir_handle_get: repair two test cases, note the fixes for 3 others 2017-05-15 17:21:55 -04:00
Alexander Færøy
008194035f Handle non-compressed requests gracefully.
This patch makes us use FALLBACK_COMPRESS_METHOD to try to fetch an
object from the consensus diff manager in case no mutually supported
result was found. This object, if found, is then decompressed using the
spooling system to the client.

See: https://bugs.torproject.org/21667
2017-05-15 17:21:55 -04:00
Alexander Færøy
8d730af0f7 Remove old consensus fetching code from handle_get_current_consensus().
This patch removes the calls to spooled_resource_new() when trying to
download the consensus. All calls should now be going through the
consdiff manager.

See: https://bugs.torproject.org/21667
2017-05-15 17:21:55 -04:00
Alexander Færøy
fade313ba3 Fix too wide line from make check-spaces.
See: https://bugs.torproject.org/21667
2017-05-15 17:21:55 -04:00
Alexander Færøy
ae33deb91d Check for best consensus when no consensusdiff was found.
This patch ensures that we use the current consensus in the case where
no consensus diff was found or a consensus diff wasn't requested.

See: https://bugs.torproject.org/21667
2017-05-15 17:21:55 -04:00
Nick Mathewson
7591518d16 Copy valid/fresh-until and signatories values into diffs. 2017-05-15 17:21:55 -04:00
Nick Mathewson
fd1190581d Store fresh/valid-until and signatories values on all consensus objects. 2017-05-15 17:21:55 -04:00
Nick Mathewson
dcc533fb13 Implement functions to expose valid/fresh-until and voters
These still won't do anything till I get the values to be filled in.

Also, I changed the API a little (with corresponding changes in
directory.c) to match things that it's easier to store.
2017-05-15 17:21:55 -04:00
Nick Mathewson
2f06345db3 Move stub accessor functions a level higher, to consdiffmgr 2017-05-15 17:21:55 -04:00
Alexander Færøy
ef2a62b2ff Fetch the current consensus from the conscache subsystem.
This patch changes handle_get_current_consensus() to make it read the
current consensus document from the consensus caching subsystem.

See: https://bugs.torproject.org/21667
2017-05-15 17:21:55 -04:00
Nick Mathewson
3b8888c544 Initialize the HS cache at startup
Failure to do this caused an assertion failure with #22246 . This
assertion failure can be triggered remotely, so we're tracking it as
medium-severity TROVE-2017-002.
2017-05-15 13:49:29 -04:00
Nick Mathewson
71a21256b0 Merge branch 'bug22245_024' 2017-05-15 11:46:55 -04:00
Nick Mathewson
85f75d617c Add hs_test_helpers.h to noinst_HEADERS 2017-05-15 11:45:20 -04:00
Nick Mathewson
a6514b8a20 Fix a logic error in hibernate.c
Closes bug 22245; bugfix on 0.0.9rc1, when bandwidth accounting was
first introduced.

Found by Andrey Karpov and reported at https://www.viva64.com/en/b/0507/
2017-05-15 11:43:18 -04:00
Nick Mathewson
0e348720fc Fix assertion to actually have a chance of failing
This assertion can only fail if we mess up some of our other code,
but let's try to get it right.

Closes 22244.
2017-05-15 11:27:12 -04:00
Nick Mathewson
1ec45bb546 Merge branch 'bug18100_029' 2017-05-15 11:19:44 -04:00
Nick Mathewson
4473271c66 Fix the TRPOXY typo in connection_edge.c
Also add a get_options() call so it compiles.

Fixes bug 18100; bugfix on 0.2.6.3-alpha. Patch from "d4fq0fQAgoJ".
2017-05-15 11:16:50 -04:00
Nick Mathewson
d3a39cf8d0 Merge remote-tracking branch 'public/bug20270_029' 2017-05-15 10:45:20 -04:00
Alexander Færøy
363f4b8db5
Add stub functions for querying metadata about the consensus. 2017-05-13 01:05:15 +02:00
Alexander Færøy
64116ab97f
Fix tautological constant out-of-range comparison warnings. 2017-05-12 17:59:29 +02:00
Alexander Færøy
a1e8ef0076
Fix DoubleNL warning from make check-spaces. 2017-05-12 17:57:11 +02:00
Nick Mathewson
077d3085ec
actually enable background compresion for consensuses 2017-05-12 17:45:55 +02:00
Nick Mathewson
db370bb8a8
Test fix: expect old consensuses to be deleted if not deflate-compressed 2017-05-12 17:45:44 +02:00
Nick Mathewson
30dfb36148
consdiffmgr: Reload latest consensus entries on start. 2017-05-12 17:45:33 +02:00
Nick Mathewson
8100305e71
consdiffmgr: expose cached consensuses 2017-05-12 17:45:24 +02:00
Nick Mathewson
7b0dcf5c4a
Cleanup logic: only retain zlib-compressed consensuses
Now that we're making a bunch of these with consdiffmgr, we should
throw out all but one when we get a newer consensus.
2017-05-12 17:45:15 +02:00
Nick Mathewson
151cd121a2
consdiffmgr: compress incoming consensuses in the background
Also, compress them in several ways.

This breaks the unit tests; subsequent commits will make them pass
again.
2017-05-12 17:45:07 +02:00
Nick Mathewson
6da31ec484
consdiffmgr: Extract the code for compressing and storing
We're going to use this for consensuses too.
2017-05-12 17:44:55 +02:00
Alexander Færøy
7a3efe25d9
Use different preferences for compression methods when streaming.
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
141f6e3211
Add client_meth_pref array to define client compression preference.
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
59d17ca2bb
Fix indentation when using the ternary operator in handle_get_status_vote().
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
ef187bc280
Use compression_method_get_human_name() in connection_dir_client_reached_eof()
This patch refactors connection_dir_client_reached_eof() to use
compression_method_get_human_name() to set description1 and
description2 variables.

See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
3a05687c6d
Add API for getting human readable descriptions of a compress_method_t
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
f8218b5ada
Use compression_method_get_by_name() instead of explicit checks.
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
cf2f7a1bea
Decide compression method in the various handle_* functions().
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
a3a31fa120
Send "Accept-Encoding" to directory servers.
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
6305637197
Use tor_compress_supports_method() instead of constants.
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
61b6de5906
Handle Zstandard and LZMA in our check for correct guessed compression.
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
fbef257c43
Handle x-zstd and x-tor-lzma in parse_http_response().
See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Alexander Færøy
ca632144e5
Use dir_compressed(_len) instead of dir_z(_len).
This patch renames `dir_z` to `dir_compressed` and `dir_z_len` to
`dir_compressed_len`.

See: https://bugs.torproject.org/21667
2017-05-12 17:18:45 +02:00
Nick Mathewson
4410271446 Merge branch 'ticket21953_029' 2017-05-12 08:40:30 -04:00
Nick Mathewson
503f101d2b Enable some windows hardening features
One (HeapEnableTerminationOnCorruption) is on-by-default since win8;
the other (PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION) supposedly only
affects ATL, which (we think) we don't use.  Still, these are good
hygiene. Closes ticket 21953.
2017-05-11 16:39:02 -04:00
Taylor Yu
61a367cadd Untangle cleanup logic in test_hs_intropoint.c
Cleanup logic in test_intro_point_registration() invoked tt_assert()
in a way that could cause it to jump backward into the cleanup code if
the assertion failed, causing Coverity to see a double free (CID
1397192).  Move the tt_assert() calls into a helper function having
the well-defined task of testing hs_circuitmap_free_all().
Fixes #22231.
2017-05-11 13:06:45 -04:00
Nick Mathewson
9905659573 Merge remote-tracking branch 'arma/cleanup22213' 2017-05-11 09:20:12 -04:00
Nick Mathewson
6390a0c3b6 Merge branch 'ticket21871_031_03_squashed' 2017-05-11 08:33:32 -04:00
David Goulet
ae1d4cfdad prop224: Change encryption keys descriptor encoding
A descriptor only contains the curve25519 public key in the enc-key field so
the private key should not be in that data structure. The service data
structures will have access to the full keypair (#20657).

Furthermore, ticket #21871 has highlighted an issue in the proposal 224 about
the encryption key and legacy key being mutually exclusive. This is very wrong
and this commit fixes the code to follow the change to the proposal of that
ticket.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-11 08:33:26 -04:00
Nick Mathewson
dc9274dec3 Merge remote-tracking branch 'argonblue/bug22177' 2017-05-11 07:41:21 -04:00
Nick Mathewson
38003f4350 Merge remote-tracking branch 'argonblue/bug22221' 2017-05-11 07:25:22 -04:00
Roger Dingledine
8f1ddf0cd2 Remove unused "ROUTER_ADDED_NOTIFY_GENERATOR" internal value
We already have a way to return a 400 response code along with a
personalized message response for the uploader.

Resolves ticket 22213.
2017-05-10 20:03:07 -04:00
Roger Dingledine
466e27feae simplify functions now that they don't use options param 2017-05-10 17:57:35 -04:00
Roger Dingledine
2330a3713d Merge branch 'maint-0.3.0' 2017-05-10 17:36:34 -04:00
Roger Dingledine
716d48581d resolve now-unused parameter from #21642 fix 2017-05-10 17:35:36 -04:00
Taylor Yu
3156392e9b Test config line parsing errors
Add tests for lower-level syntax errors in config file lines.
Fixes #22177.
2017-05-10 16:51:40 -04:00
Nick Mathewson
1a497dcd1e Merge branch 'maint-0.3.0' 2017-05-10 16:28:07 -04:00
Nick Mathewson
a868b84599 Merge branch 'maint-0.2.9' into maint-0.3.0 2017-05-10 16:27:15 -04:00
Nick Mathewson
8f5da804da Merge branch 'prop275_minimal_029' into maint-0.2.9 2017-05-10 16:26:45 -04:00
Taylor Yu
2bf4263800 Resurrect dead code in test_channelpadding.c
A for-loop in test_channelpadding_timers() would never run because it
was trying to increment a counter up to CHANNELS_TO_TEST/3 after an
earlier block already incremented it to CHANNELS_TO_TEST/2.

Fixes #22221, CID 1405983.
2017-05-10 13:57:18 -04:00
Nick Mathewson
ee3ccd2fac #22211 Fix a comment in routerparse.c 2017-05-10 11:16:07 -04:00
Nick Mathewson
d76cffda60 Merge remote-tracking branch 'public/my-family-list-fix-4498' 2017-05-10 11:12:24 -04:00
Nick Mathewson
8266d193a6 Restore wget behavior when fetching compressed objects
We do this by treating the presence of .z as meaning ZLIB_METHOD,
even if Accept-Encoding does not include deflate.

This fixes bug 22206; bug not in any released tor.
2017-05-10 11:09:52 -04:00
Nick Mathewson
5dab99d6a8 Fix compilation on libevent2-only systems
Patch from rubiate; fixes bug 22219.  Remember, we don't support
libevent1 any more.
2017-05-10 11:08:49 -04:00
Nick Mathewson
95fa7d1cf8 In channelpadding tests that touch libevent, call event_reinit().
This is necessary to avoid crashes and test failures on kevent-based
systems.

Fixes bug 22209; bug not in any released Tor.
2017-05-10 11:01:13 -04:00
Roger Dingledine
0266c4ac81 add an XXX with a minor bug in dirserv_add_multiple_descriptors 2017-05-10 03:11:29 -04:00
Nick Mathewson
9f5b71a7ca Add a check and a cast in rephist.c to fix a warning 2017-05-09 11:13:22 -04:00
Nick Mathewson
b2cb3c33ac Tidy or_options_t by removing obsolete options.
Nothing was setting or inspecting these fields, and they were marked
as OBSOLETE() in config.c -- but somehow we still had them in the
or_options_t structure.  Ouch.
2017-05-09 10:40:24 -04:00
Nick Mathewson
2a1013948d Merge branch 'dgoulet_ticket22060_031_01_squashed' 2017-05-09 10:32:21 -04:00
David Goulet
b867295ffd config: Parse ports at the start of options_validate()
There was a bug that got exposed with the removal of ORListenAddress. Within
server_mode(), we now only check ORPort_set which is set in parse_ports().

However, options_validate() is using server_mode() at the start to check if we
need to look at the uname but then the ORPort_set is unset at that point
because the port parsing was done just after. This commit fixes that.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
7f95ef6e66 config: Remove {Control,DNS,Dir,Socks,Trans,NATD,OR}ListenAddress option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
039e2a24da config: Remove TLSECGroup option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
8aedc589ed config: Remove WarnUnsafeSocks option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
60cf5ac297 config: Remove CloseHSServiceRendCircuitsImmediatelyOnTimeout option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
87e9dc48d1 config: Remove CloseHSClientCircuitsImmediatelyOnTimeout option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
bc34654ba2 config: Remove FastFirstHopPK option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
09bc858dd5 config: Remove ExcludeSingleHopRelays option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
d52a1e2faa config: Remove AllowSingleHopExits option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
fea72571df config: Remove AllowSingleHopCircuits option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:52 -04:00
David Goulet
2b9823b310 config: Remove AllowInvalidNodes option
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.

Partially fixes #22060

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:51 -04:00
David Goulet
0958e3b208 test: Remove HS decode valid intro point test
The descriptor fields can't be validated properly during encoding because they
are signed by a descriptor signing key that we don't have in the unit test.

Removing the test case for now but ultimately we need an independent
implementation that can encode descriptor and test our decoding functions with
that.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:37 -04:00
David Goulet
0cc18ef64c test: Move duplicate HS test code and unify it
Create the hs_test_helpers.{c|h} files that contains helper functions to
create introduction point, descriptor and compare descriptor.

Used by both the hs cache and hs descriptor tests. Unify them to avoid code
duplication.

Also, this commit fixes the usage of the signing key that was wrongly used
when creating a cross signed certificate.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-09 10:30:37 -04:00
Nick Mathewson
ff7e8531ec Report common reasons for rejecting a relay before uncommon ones
"You're running 0.2.2; upgrade!" is more sensible than "You have no
ntor key!"

Closes ticket 20270; bugfix on 0.2.9.3-alpha.
2017-05-09 10:28:51 -04:00
Nick Mathewson
2f714d60a4 Minor cleanups in test_config.
Use a more standard indentation style for the end block; remove
redundant if() around tor_free.
2017-05-09 08:20:14 -04:00
Nick Mathewson
b8abedfeee Split MyFamily into user-specified version and normalized version
This change prevents a no-longer-supported behavior where we change
options that would later be written back to torrc with a SAVECONF.

Also, use the "Pointer to final pointer" trick to build the
normalized list, to avoid special-casing the first element.
2017-05-09 08:19:26 -04:00
Nick Mathewson
674c1d2594 Fix an implicit conversion warning 2017-05-09 07:26:14 -04:00
Nick Mathewson
7751df61ca Fix a warning about a shadowed global 2017-05-09 07:25:52 -04:00
Nick Mathewson
ef2248d09b Fix a warning about an extraneous semicolon 2017-05-09 07:25:34 -04:00
Nick Mathewson
2e4f3b36bd clang-i386: use house style for public-when-testing variables
This fixes a warning from jenkins.
2017-05-08 15:38:05 -04:00
Nick Mathewson
4d6c79d1de Fix some clang-i386 warnings in master. 2017-05-08 15:34:37 -04:00
Nick Mathewson
fb97f76e71 whitespace fixes 2017-05-08 13:57:08 -04:00
Nick Mathewson
4d30dde156 Merge branch 'netflow_padding-v6-rebased2-squashed' 2017-05-08 13:54:59 -04:00
Mike Perry
9f8e462c89 Fix some channelpadding test issues.
asan was finding an alignment issue with a cast, so set the field in the
trunnel struct and then encode it instead. Also, enable log capture and
verification.
2017-05-08 13:49:23 -04:00
Mike Perry
02a5835c27 Fix issues from dgoulet's code review.
https://gitlab.com/dgoulet/tor/merge_requests/24
2017-05-08 13:49:23 -04:00
Mike Perry
687a85950a Cache netflow-related consensus parameters.
Checking all of these parameter lists for every single connection every second
seems like it could be an expensive waste.

Updating globally cached versions when there is a new consensus will still
allow us to apply consensus parameter updates to all existing connections
immediately.
2017-05-08 13:49:23 -04:00
Mike Perry
ae4d8c9c74 Fix a breakage in test_options.c.
IMO, these tests should be calling options_init() to properly set everything
to default values, but when that is done, about a dozen tests fail. Setting
the one default value that broke the tests for my branch. Sorry for being
lame.
2017-05-08 13:49:22 -04:00
Mike Perry
20a3d4efac Remove a PredictedPortsRelevantTime test.
The option was deprecated by bug #17592.
2017-05-08 13:49:22 -04:00
Mike Perry
76c9330f9d Bug 17604: Converge on only one long-lived TLS conn between relays.
Accomplished via the following:

1. Use NETINFO cells to determine if both peers will agree on canonical
   status. Prefer connections where they agree to those where they do not.
2. Alter channel_is_better() to prefer older orconns in the case of multiple
   canonical connections, and use the orconn with more circuits on it in case
   of age ties.

Also perform some hourly accounting on how many of these types of connections
there are and log it at info or notice level.
2017-05-08 13:49:22 -04:00
Mike Perry
d5a151a067 Bug 17592: Clean up connection timeout logic.
This unifies CircuitIdleTimeout and PredictedCircsRelevanceTime into a single
option, and randomizes it.

It also gives us control over the default value as well as relay-to-relay
connection lifespan through the consensus.

Conflicts:
	src/or/circuituse.c
	src/or/config.c
	src/or/main.c
	src/test/testing_common.c
2017-05-08 13:49:22 -04:00
Mike Perry
b0e92634d8 Netflow record collapsing defense.
This defense will cause Cisco, Juniper, Fortinet, and other routers operating
in the default configuration to collapse netflow records that would normally
be split due to the 15 second flow idle timeout.

Collapsing these records should greatly reduce the utility of default netflow
data for correlation attacks, since all client-side records should become 30
minute chunks of total bytes sent/received, rather than creating multiple
separate records for every webpage load/ssh command interaction/XMPP chat/whatever
else happens to be inactive for more than 15 seconds.

The defense adds consensus parameters to govern the range of timeout values
for sending padding packets, as well as for keeping connections open.

The defense only sends padding when connections are otherwise inactive, and it
does not pad connections used solely for directory traffic at all. By default
it also doesn't pad inter-relay connections.

Statistics on the total padding in the last 24 hours are exported to the
extra-info descriptors.
2017-05-08 13:49:21 -04:00
Nick Mathewson
35025ee51f Merge branch 'maint-0.3.0' 2017-05-08 13:40:41 -04:00
Nick Mathewson
d792d2a14d Merge branch 'maint-0.2.9' into maint-0.3.0 2017-05-08 13:40:26 -04:00
Nick Mathewson
0df22e8f5f Merge branch 'bug21943_029' into maint-0.2.9 2017-05-08 13:40:14 -04:00
Nick Mathewson
03fdf232bf Merge remote-tracking branch 'ahf/coverity/1405875' 2017-05-08 08:19:13 -04:00
Nick Mathewson
300f8e6f0a Merge branch 'maint-0.3.0' 2017-05-08 08:07:49 -04:00
Nick Mathewson
f5876917aa Merge branch 'maint-0.2.9' into maint-0.3.0 2017-05-08 08:07:38 -04:00
Nick Mathewson
5ff0b519ec Merge branch 'maint-0.2.8' into maint-0.2.9 2017-05-08 08:07:32 -04:00
Nick Mathewson
dc34cd7a04 Merge branch 'maint-0.2.7-redux' into maint-0.2.8 2017-05-08 08:07:19 -04:00
Nick Mathewson
71dd1d716a Merge branch 'maint-0.2.6' into maint-0.2.7-redux 2017-05-08 08:07:11 -04:00
Nick Mathewson
159ddf6dde Merge branch 'maint-0.2.5' into maint-0.2.6 2017-05-08 08:07:04 -04:00
Nick Mathewson
68d6720452 Merge branch 'maint-0.2.4' into maint-0.2.5 2017-05-08 08:06:59 -04:00
Karsten Loesing
5207e41ffe Update geoip and geoip6 to the May 2 2017 database. 2017-05-08 10:09:42 +02:00
Nick Mathewson
9decf86711 Merge remote-tracking branch 'dgoulet/ticket21978_031_02' 2017-05-05 16:32:25 -04:00
Alexander Færøy
7344a4fdc6
Remove dead code in test_options_validate_impl().
Dead branch found by Coverity in CID #1405875.
2017-05-05 17:07:18 +02:00
Nick Mathewson
c276c10d3b Turn consdiffmgr.cache_max_age_hours into a parameter 2017-05-05 09:11:06 -04:00
Nick Mathewson
c985592874 prop140 clients now only try to get diffs from recent consensuses
Rationale: If it's a year old, the relay won't have a diff to it.

This is as specified in prop140
2017-05-05 09:11:06 -04:00
Alexander Færøy
60e97953ef
Fix memory leak found in CID #1405876. 2017-05-05 11:35:12 +02:00
Nick Mathewson
c6fe65fcaf Grammar fix in a log message 2017-05-04 08:58:06 -04:00
Nick Mathewson
baf489fc08 Fix: our directory.c code expects header constants to end with a : 2017-05-04 08:57:34 -04:00
Nick Mathewson
6beb7028d8 Do not BUG on missing sha3-as-signed field
This can happen if you've been running an earlier alpha on your
relay.  Instead, just ignore the entry.
2017-05-04 08:50:21 -04:00
Nick Mathewson
e1d31f2a2f Update the consdiff directory code based on #22143 fixes
These are mostly just identifier renames, except for one place in
routerparse.c where we switch to using a correct hash.
2017-05-04 08:49:02 -04:00
Nick Mathewson
a8eccb6363 Turn DEFAULT_IF_MODIFIED_SINCE_DELAY into a const 2017-05-04 08:37:41 -04:00
Nick Mathewson
c12d2cb2dc Request (and try to use) consensus diffs. 2017-05-04 08:37:41 -04:00
Nick Mathewson
912b0641e9 Generate X-Or-Diff-From-Consensus headers correctly. 2017-05-04 08:37:41 -04:00
Nick Mathewson
94ae99067f Remove excess indentation from previous commit.
Review this with 'diff -b' to confirm
2017-05-04 08:37:41 -04:00
Nick Mathewson
afa39cef6c Extract the consensus-only part of directory_get_from_dirserver
Right now it just sets an if-modified-since header, but it's about
to get even bigger.

This patch avoids changing indentation; the next patch will be
whitespace fixes.
2017-05-04 08:37:41 -04:00
Nick Mathewson
57710c1587 New function to add additional headers to a directory request 2017-05-04 08:37:41 -04:00
Nick Mathewson
112286338b Store the sha3 of a networkstatus as part of the networkstatus_t
Also store it in the cached_dir_t.
2017-05-04 08:37:41 -04:00
Nick Mathewson
0418357ffd Serve consensus diffs on request. 2017-05-04 08:37:41 -04:00
Nick Mathewson
e5f82969ca Support writing Content-Encoding headers other than deflate
Right now this only sends "deflate" or "identity", but there's more
to come.
2017-05-04 08:37:41 -04:00
Nick Mathewson
e051c47e98 Remove old unused indentation from handle_get_current_consensus
This commit removes a pair of meaningless braces, and changes
whitespace only.
2017-05-04 08:37:41 -04:00
Nick Mathewson
a32083bd03 Add consensus_cache_entry spooling support to spooled_resource_t 2017-05-04 08:37:41 -04:00
Nick Mathewson
24ba1864d8 Merge branch 'ticket22143_squashed' 2017-05-04 08:36:59 -04:00
Nick Mathewson
df2bcaeb4a Add a test for $ with non-delete commands. 2017-05-04 08:36:50 -04:00
Nick Mathewson
87f6979a03 Remove some unused digests from test_consdiffmgr.c
These were unused before I started working on #22143 -- I just found
them while I was lookinging for digests to update.
2017-05-04 08:36:50 -04:00
Nick Mathewson
3af9704e45 bug#22143/prop#140: in consdiffmgr, store and use digest-as-signed
We need to index diffs by the digest-as-signed of their source
consensus, so that we can find them even from consensuses whose
signatures are encoded differently.
2017-05-04 08:36:50 -04:00
Nick Mathewson
c8baa9b783 bug#22143/prop#140: Use <n>,$d commands in diffs to remove signatures
In this patch I add support for "delete through end of file" in our
ed diff handler, and generate our diffs so that they remove
everything after in the consensus after the signatures begin.
2017-05-04 08:36:50 -04:00
Taylor Yu
38a13b91a8 Fix additional leaks in #22103 tests
test_options_validate_impl() incorrectly executed subsequent phases of
config parsing and validation after an expected error.  This caused
msg to leak when those later phases (which would likely produce errors
as well) overwrote it.
2017-05-03 13:47:06 -04:00
Nick Mathewson
5acddbbbf7 bug#22143/prop#140: identify input diffs by their digest-as-signed
See may 3 changes to prop140 for more background.
2017-05-03 13:09:08 -04:00
Nick Mathewson
77d9dd39c3 Merge remote-tracking branch 'argonblue/bug22103_fixtest' 2017-05-03 09:33:37 -04:00
David Goulet
90b840af60 control: Fix NULL pointer access in HS desc event
This was introduced 90562fc23a adding a code
path where we pass a NULL pointer for the HSDir fingerprint to the control
event subsystem. The HS desc failed function wasn't handling properly that
pointer for a NULL value.

Two unit tests are also added in this commit to make sure we handle properly
the case of a NULL hsdir fingerprint and a NULL content as well.

Fixes #22138

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-03 09:26:17 -04:00
Taylor Yu
7b64f1773d Fix memory management for #22103 tests
Code movement in the commit introducings tests for #22103 uncovered a
latent memory management bug.

Refactor the log message checking from test_options_checkmsgs() into a
helper test_options_checklog().  This avoids a memory leak (and
possible double-free) in a test failure condition.

Don't reuse variables (especially pointers to allocated memory!) for
multiple unrelated purposes.

Fixes CID 1405778.
2017-05-03 08:59:29 -04:00
Nick Mathewson
e0b1fd4d3d Merge branch 'refactor_dir_client_handler' 2017-05-02 20:42:00 -04:00
Nick Mathewson
7bc9f93abb Merge branch 'teor_connection-with-client-v2_squashed' 2017-05-02 19:18:04 -04:00
teor
f9af7e8bd0 Accurately identify client connections by their lack of peer authentication
This means that we bail out earlier if asked to extend to a client.

Follow-up to 21407.
Fixes bug 21406; bugfix on 0.2.4.23.
2017-05-02 19:17:56 -04:00
Taylor Yu
a91f948cca Add tests for #22103
Also factor out the error message comparisions from
test_options_validate_impl() into a separate function so it can check
for error messages in different phases of config parsing.
2017-05-02 16:13:14 -04:00
Taylor Yu
5494087ed7 Delete useless checks in confparse.c
config_parse_interval() and config_parse_msec_interval() were checking
whether the variable "ok" (a pointer to an int) was null, rather than
derefencing it.  Both functions are static, and all existing callers
pass a valid pointer to those static functions.  The callers do check
the variables (also confusingly named "ok") whose addresses they pass
as the "ok" arguments, so even if the pointer check were corrected to
be a dereference, it would be redundant.

Fixes #22103.
2017-05-02 16:12:30 -04:00
Nick Mathewson
e0c937f316 Reindent the just-extracted directory response handler functions 2017-05-02 13:11:44 -04:00
Nick Mathewson
db86b9194d Break connection_dir_client_reached_eof() into smaller functions
This was a >630-line function, which doesn't make anybody happy.  It
was also mostly composed of a bunch of if-statements that handled
different directory responses differently depending on the original
purpose of the directory connection.  The logical refactoring here
is to move the body of each switch statement into a separate handler
function, and to invoke those functions from a separate switch
statement.

This commit leaves whitespace mostly untouched, for ease of review.
I'll reindent in the next commit.
2017-05-02 13:06:25 -04:00
Nick Mathewson
fd437f2a02 Remove special-casing for NO_METHOD in consdiffmgr.c 2017-05-02 08:37:57 -04:00
Nick Mathewson
584ab1f29b Merge branch 'compress_none_v2_squashed' 2017-05-02 08:33:35 -04:00
Nick Mathewson
3836d9481f Add unit tests for the NO_METHOD compressor
These required some special-casing, since some of the assumption
about real compression algorithms don't actually hold for the
identity transform.  Specifically, we had assumed:

  - compression functions typically change the lengths of their
    inputs.
  - decompression functions can detect truncated inputs
  - compression functions have detectable headers

None of those is true for the identity transformation.
2017-05-02 08:31:32 -04:00
Nick Mathewson
1bc21111d8 Treat the identity transformation as another kind of compression.
This will allow us to treat NO_METHOD as a real compression method,
and to simplify code that currently does

   if (compressing) {
      compress
   } else {
      copy
   }
2017-05-02 08:31:32 -04:00
Nick Mathewson
465448e659 Merge branch 'dgoulet_bug22042_031_01_squashed' 2017-05-01 16:08:23 -04:00
David Goulet
aadb99e5f9 control: Fix comment of control_event_hs_descriptor_content
Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-01 15:50:38 -04:00
David Goulet
90562fc23a hs: Trigger control event when client can't pick HSDir
Inform the control port with an HS_DESC failed event when the client is unable
to pick an HSDir. It's followed by an empty HS_DESC_CONTENT event. In order to
achieve that, some control port code had to be modified to accept a NULL HSDir
identity digest.

This commit also adds a trigger of a failed event when we are unable to
base64-decode the descriptor cookie.

Fixes #22042

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-05-01 15:50:38 -04:00
Nick Mathewson
c486ef57a3 Rename x-lzma to x-tor-lzma
We shouldn't call it lzma, because we are imposing a limit on the
memory needed for decoding.
2017-05-01 15:31:28 -04:00
Nick Mathewson
4837421d7c Merge remote-tracking branch 'ahf/bugs/21665' 2017-05-01 14:22:49 -04:00
Sebastian Hahn
915fa39d0f Add --enable-rust configure switch
Introduce a way to optionally enable Rust integration for our builds. No
actual Rust code is added yet and specifying the flag has no effect
other than failing the build if rustc and cargo are unavailable.
2017-04-29 08:55:57 +02:00
Nick Mathewson
b8f7488e94 Fix a brazen memleak in consdiffmgr_add_consensus() 2017-04-28 15:41:52 -04:00
Nick Mathewson
531835f561 Increase MALLOC_MP_LIM to 16MB
Increase the maximum allowed size passed to mprotect(PROT_WRITE)
from 1MB to 16MB. This was necessary with the glibc allocator
in order to allow worker threads to allocate more memory --
which in turn is necessary because of our new use of worker
threads for compression.

Closes ticket #22096. Found while working on #21648.
2017-04-28 10:55:10 -04:00
Nick Mathewson
00a12337ff Merge branches 'consdiffmgr_orig_squashed' and 'actually_compute_diffs_squashed' 2017-04-27 21:43:06 -04:00
Nick Mathewson
1e1581a24e Pre-compress consensus diffs with supported consensus methods. 2017-04-27 21:40:46 -04:00
Nick Mathewson
a1172b6774 Store archived consensuses compressed on disk.
I'm just using gzip compression now, for performance.
2017-04-27 21:40:46 -04:00
Nick Mathewson
7a0964279f Functionality to ensure there is space to add files to cache. 2017-04-27 21:40:13 -04:00
Nick Mathewson
920475f293 New force-delete option on consensus_cache_delete_pending()
If we're out of file space in the storage directory, we'll need to
get rid of old files fast.
2017-04-27 21:40:13 -04:00
Nick Mathewson
466e914088 Lower the file limit in consdiffmgr, to support seccomp2 2017-04-27 21:40:13 -04:00
Nick Mathewson
ab73bda060 Pass incoming consensus documents to the consdiffmgr code 2017-04-27 21:40:12 -04:00
Nick Mathewson
16d6ab6640 Fix use-after-free bug in storage_dir sandbox code. 2017-04-27 21:40:12 -04:00
Nick Mathewson
7b8d48a6cb Clean the consdiffmgr cache and launch new diffs as needed. 2017-04-27 21:40:12 -04:00
Nick Mathewson
fba8d7b222 Initialize consdiffmgr when running as (or becoming) a server. 2017-04-27 21:40:12 -04:00
Alexander Færøy
0672b33f1e
Enforce 16 MB upper bound of memory usage in LZMA decoder.
This patch changes two things in our LZMA compression backend:

- We lower the preset values for all `compression_level_t` values to
  ensure that we can run the LZMA decoder with less than 65 MB of memory
  available. This seems to have a small impact on the real world usage
  and fits well with our needs.

- We set the upper bound of memory usage for the LZMA decoder to 16 MB.

See: https://bugs.torproject.org/21665
2017-04-27 20:09:20 +02:00
Alexander Færøy
e5122b91a9
Only compare the first 3 bytes when trying to detect LZMA compression.
This patch changes the logic in `detect_compression_method()` to only
use the 3 first bytes when checking if a given input is LZMA encoded.
2017-04-27 20:07:08 +02:00
Nick Mathewson
480dab4f2f Use a cast to try to avoid a tautalogical comparison warning 2017-04-27 11:58:26 -04:00
Nick Mathewson
10a4f9cd07 Merge branch 'parse_accept_encoding' 2017-04-27 11:31:31 -04:00
Nick Mathewson
2903c329aa Move the "supported compression bitmask" into compress.[ch] 2017-04-27 11:30:51 -04:00
Nick Mathewson
49deb1e1b8 Document and test nul-terminating behavior of tor_uncompress()
We added this as a safety feature, but there are a few places in the
code that actually depend on it.
2017-04-27 10:59:48 -04:00
Nick Mathewson
199e61feb5 whitespace fix 2017-04-27 10:55:39 -04:00
Nick Mathewson
ba9f235e17 Re-enable the partial-input test for zstd.
There were two issues here: first, zstd didn't exhibit the right
behavior unless it got a very large input.  That's fine.

The second issue was a genuine bug, fixed by 39cfaba9e2.
2017-04-27 10:43:38 -04:00
Nick Mathewson
39cfaba9e2 Fix handling of "final" flag in zstd decompression
We were returning "DONE" on truncated input streams, which was not
what we wanted.
2017-04-27 10:42:05 -04:00
Nick Mathewson
166aa8d741 Have a separate entry point for each compresion test 2017-04-27 10:25:52 -04:00
Nick Mathewson
36e62ae7e2 Merge remote-tracking branch 'ahf/bugs/22085' 2017-04-27 10:12:48 -04:00
Nick Mathewson
33a2fd065d Merge branch 'dirreq' 2017-04-27 10:08:32 -04:00
Alexander Færøy
e0a3819dcc
Test support for multiple compression backends in the buffer code.
This patch refactors the buffer compression tests to support multiple
comprssion backends.

See: https://bugs.torproject.org/22085
2017-04-27 15:51:14 +02:00
Alexander Færøy
22e6ad6f26
Clean up mentions of 'zlib' and rename the mentions to 'compressed'.
This patch cleans up in various places where 'zlib' is mentioned.
2017-04-27 15:51:14 +02:00
Alexander Færøy
7bececbd69
Refactor compression tests into a single test.
This patch refactors our compression tests such that deflate, gzip,
lzma, and zstd are all tested using the same code.

Additionally we use run-time checks to see if the given compression
method is supported instead of using HAVE_LZMA and HAVE_ZSTD.

See: https://bugs.torproject.org/22085
2017-04-27 15:51:14 +02:00
Nick Mathewson
b4fe0a6a03 Improve control flow in authority_certs_fetch_resource_impl 2017-04-27 09:30:08 -04:00
Nick Mathewson
52316f9969 Include UPLOAD_RENDDESC_V2 in PURPOSE_IS_UPLOAD
This was only used in one place before, and it's safe to update it.
2017-04-27 09:27:00 -04:00
Nick Mathewson
7ae7e3c2d8 bump to 0.3.0.6-dev 2017-04-26 15:37:27 -04:00
Nick Mathewson
7fb9586953 Fix compilation when lzma or zstd is absent 2017-04-26 15:00:40 -04:00
Nick Mathewson
4038202f89 Avoid a warning from the use of floating-point in zstd
Replace "(preset - 0.5) * 1mb" with "preset * 1mb - 0.5 mb", to
avoid warning about converting double to size_t.
2017-04-26 14:21:45 -04:00
Nick Mathewson
be0557f759 Merge remote-tracking branch 'ahf/bugs/22066' 2017-04-26 14:20:01 -04:00
Alexander Færøy
e42c204f67
Approximate memory usage needed for the Zstandard backend.
This patch adds support for measuring the approximated memory usage by
the individual `tor_zstd_compress_state_t` object instances.

See: https://bugs.torproject.org/22066
2017-04-26 19:54:18 +02:00
Alexander Færøy
2aa28e7cb7
Better documentation for tor_uncompress().
This patch fixes the documentation string for `tor_uncompress()` to
ensure that it does not explicitly mention zlib or gzip since we now
support multiple compression backends.
2017-04-26 19:54:18 +02:00
Alexander Færøy
341824687a
Approximate memory usage needed for the LZMA backend.
This patch adds support for measuring the approximated memory usage by
the individual `tor_lzma_compress_state_t` object instances.

The LZMA library provides the functions `lzma_easy_encoder_memusage()`
and `lzma_easy_decoder_memusage()` which is used to find the estimated
usage in bytes.

See: https://bugs.torproject.org/22066
2017-04-26 19:54:18 +02:00
Nick Mathewson
31b79f02ab Bump version to 0.3.0.6 2017-04-26 13:30:39 -04:00
Nick Mathewson
99e943998d Add getpid() to the seccomp2 sandbox.
We hadn't needed this before, because most getpid() callers on Linux
were looking at the vDSO version of getpid().  I don't know why at
least one version of OpenSSL seems to be ignoring the vDSO, but this
change should fix it.

Fixes bug 21943; bugfix on 0.2.5.1-alpha when the sandbox was
introduced.
2017-04-26 12:56:06 -04:00
Sebastian Hahn
71c8974af0 Fix coverity cid 1405509
Locking in the init function is not necessary, but coverity gets
confused about it. So let's trick it.
2017-04-26 08:48:24 +02:00
Sebastian Hahn
ee478bdf38 Fix coverity cid 1405510
This is a false positive, but let's appease coverity.
2017-04-26 08:45:38 +02:00
Alexander Færøy
08d86e8408
Store compression overhead from tor_compress_state_t.
The `tor_compress_state_t` data-type is used as a wrapper around the
more specialized state-types used by the various compression backends.
This patch ensures that the overhead of this "thin" wrapper type is
included in the value returned by `tor_compress_get_total_allocation()`.

See: https://bugs.torproject.org/22066
2017-04-26 02:56:21 +02:00
Alexander Færøy
fac8ac0e4a
Remove unused header from the Zstandard compression backend.
Since we stopped looking at Zstandard error codes there is no need to
include the zstd_errors.h header file anymore.
2017-04-26 02:54:34 +02:00
Nick Mathewson
24f7059704 Configure sandbox using consdiffmgr; free cdm on exit. 2017-04-25 19:52:34 -04:00
Nick Mathewson
65ff0f8267 Bitmask out the compression methods that we do not support 2017-04-25 19:07:17 -04:00
Nick Mathewson
fec3050ea9 Tests for parse_accept_encoding 2017-04-25 19:01:05 -04:00
Nick Mathewson
fd48b757d3 Parse recognized entries from the Accept-Encoding header. 2017-04-25 19:01:05 -04:00
Nick Mathewson
2655a72d89 Use x-lzma, not x-lzma2, as the identifier 2017-04-25 19:00:52 -04:00
Nick Mathewson
9e081a44a9 Teach consdiffmgr to remember two digests: one compressed, one not. 2017-04-25 16:49:47 -04:00
Nick Mathewson
0274ea749a Function to convert compression methods to/from strings. 2017-04-25 16:47:46 -04:00
Alexander Færøy
236cf49670
Add stub function for libor-trace.
OS X's ar(1) doesn't allow us to create an archive with no object files.
This patch adds a stub file with a stub function in it to make OS X
happy again.
2017-04-25 18:02:19 +02:00
Nick Mathewson
43db91bd87 Teach cov-exclude to detect runaway LCOV_EXCL_START lines
Also, fix two instances of runaway LCOV_EXCL_START lines.
2017-04-25 10:59:19 -04:00
Nick Mathewson
7a37cbbea3 Whitespace fixes. 2017-04-25 10:54:34 -04:00
Nick Mathewson
49868340f7 Merge branch 'unified_compress_squashed' 2017-04-25 10:51:13 -04:00
Nick Mathewson
49a5b50b31 zlib: Turn UINT_MAX overrun into an error, not an assert. 2017-04-25 10:50:50 -04:00
Nick Mathewson
4b01b45ec1 Add a "best compression" flag. 2017-04-25 10:50:50 -04:00
Nick Mathewson
880fb3e3a9 Combine all *compress/*uncompress backend function into one
Since we have a streaming API for each compression backend, we don't
need a non-streaming API for each: we can build a common
non-streaming API at the front-end.
2017-04-25 10:50:50 -04:00
Nick Mathewson
232c9e14a8 Merge branch 'atomic_counters' 2017-04-25 10:46:23 -04:00
Nick Mathewson
ec7c512d27 Merge remote-tracking branch 'dgoulet/bug21293_031_01' 2017-04-25 10:43:53 -04:00
David Goulet
cb8ac1f331 trace: Add a basic event-tracing infrastructure.
This commit adds the src/trace directory containing the basics for our tracing
subsystem. It is not used in the code base. The "src/trace/debug.h" file
contains an example on how we can map our tor trace events to log_debug().

The tracing subsystem can only be enabled by tracing framework at compile
time. This commit introduces the "--enable-tracing-debug" option that will
make all "tor_trace()" function be maped to "log_debug()".

Closes #13802

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-25 10:37:31 -04:00
Nick Mathewson
4266ec766a Use atomic counters for compressor allocation. 2017-04-25 10:29:07 -04:00
David Goulet
368b091329 relay: Change LD_BUG log to LD_PROTOCOL_WARN
That log statement can be triggered if somebody on the Internet behaves badly
which is possible with buggy implementation for instance.

Fixes #21293

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-25 09:27:34 -04:00
Alexander Færøy
91dd4a00f7
Rename shadowing variable in compress_zlib.c.
This patch renames the `compress` parameter of the
`tor_zlib_compress_new()` function to `_compress` to avoid shadowing the
`compress()` function in zlib.h.
2017-04-25 14:56:55 +02:00
Nick Mathewson
ba405f86bf Merge branch 'ahf_prop278_21662_squashed' 2017-04-25 08:12:59 -04:00
Alexander Færøy
2210b330e7 Fix newlines in compression headers.
See: https://bugs.torproject.org/21662
2017-04-25 08:11:32 -04:00
Alexander Færøy
cf912259ba Remove tor_compress_memory_level().
This patch splits up `tor_compress_memory_level()` into static functions
in the individual compression backends, which allows us to tune the
values per compression backend rather than globally.

See: https://bugs.torproject.org/21662
2017-04-25 08:11:32 -04:00
Alexander Færøy
69a41e8bc6 Use switch-statement in tor_{compress,uncompress}.
Use a switch-statement in `tor_compress()` and `tor_uncompress()` for
the given `compress_method_t` parameter. This allows us to have the
compiler detect if we forgot add a handler in these functions for a
newly added enumeration value.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:10 -04:00
Alexander Færøy
c2d1d949de Use tor_compress_supports_method() before printing library versions.
This patch ensures that Tor checks if a given compression method is
supported before printing the version string when calling `tor
--library-versions`.

Additionally, we use the `tor_compress_supports_method()` to check if a
given version is supported for Tor's start-up version string, but here
we print "N/A" if a given compression method is unavailable.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:10 -04:00
Alexander Færøy
6b905b38bb Add API entry-point for getting compression method version numbers.
This patch adds `tor_compress_version_str()` and
`tor_compress_header_version_str()` to get the version strings of the
different compression schema providers. Both functions returns `NULL` in
case a given `compress_method_t` is unknown or unsupported.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:10 -04:00
Alexander Færøy
1c77d8690c Add function to check if a given compression method is supported.
This patch adds support for checking if a given `compress_method_t` is
supported by the currently running Tor instance using
`tor_compress_supports_method()`.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
04682d302a Add tor_compress_get_total_allocation() function.
This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
be4dc54634 Display LZMA and Zstandard versions when starting Tor.
See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
2fa7b722ce Show liblzma and libzstd versions in tor --library-versions.
See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
380736d045 Add Zstandard support.
See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
ce1feae9d9 Add --enable-zstd to our configure script.
This patch adds support for enabling support for Zstandard to our configure
script. By default, the --enable-zstd option is set to "auto" which means if
libzstd is available we'll build Tor with Zstandard support.

See: https://bugs.torproject.org/21662
2017-04-25 08:10:09 -04:00
Alexander Færøy
bf1c07cb07 Add LZMA support.
See: https://bugs.torproject.org/21662
2017-04-25 08:10:06 -04:00
Alexander Færøy
157af1d26e Add --enable-lzma to our configure script.
This patch adds support for enabling support for LZMA to our configure
script. By default, the --enable-lzma option is set to "auto" which
means if liblzma is available we'll build Tor with LZMA support.

See: https://bugs.torproject.org/21662
2017-04-25 08:06:02 -04:00
Alexander Færøy
c171af0487 Use a switch-statement when checking for compression method.
This patch changes the way `tor_compress_new()`,
`tor_compress_process()`, and `tor_compress_free()` handles different
compression methods. This should give us compiler warnings in case an
additional compression method is added, but the developer forgets to add
handlers in the three aforementioned functions.

See https://bugs.torproject.org/21663
2017-04-25 08:06:01 -04:00
Alexander Færøy
300ac49685 Add compress_zlib.obj to src/common/Makefile.nmake.
See https://bugs.torproject.org/21663
2017-04-25 08:06:01 -04:00
Alexander Færøy
04583df452 Rename the torgzip module to compress.
See https://bugs.torproject.org/21663
2017-04-25 08:06:01 -04:00
Alexander Færøy
9d5bc1a935 Move zlib compression code into its own module.
This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.

Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.

See https://bugs.torproject.org/21664
2017-04-25 08:06:01 -04:00
Nick Mathewson
db95a6e171 Fix implicit conversion warnings in the period_num tests 2017-04-24 15:38:16 -04:00
David Goulet
6f27843d57 hs: Refactor rend_add_service()
Remove duplicate code that validates a service object which is now in
rend_validate_service().

Add some comments on why we nullify a service in the code path of
rend_config_services().

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-24 14:51:34 -04:00
David Goulet
ed7c0170c4 hs: Add rend_validate_service() function
This new function validates a service object and is used everytime a service
is successfully loaded from the configuration file.

It is currently copying the validation that rend_add_service() also does which
means both functions validate. It will be decoupled in the next commit.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-24 14:39:54 -04:00
Nick Mathewson
4cefda85e4 Merge remote-tracking branch 'dgoulet/ticket21980_031_01' 2017-04-24 12:42:11 -04:00
Nick Mathewson
d949589c09 Fix some leaks in the consdiffmgr tests 2017-04-24 11:45:13 -04:00
Nick Mathewson
e55c1412c1 Fix a signed/unsigned comparison warning on 32-bit 2017-04-24 11:41:11 -04:00
Nick Mathewson
b7567a6282 Merge branch 'consdiffmgr_squashed' 2017-04-24 11:02:22 -04:00
Nick Mathewson
eb14faa0c1 Remove a checklist item that was already tested
The item referred to the cdm_ht_set_status() case where the item was
not already in the hashtable.  But that already happens naturally
when we scan the directory on startup... and we already have a test
for that.
2017-04-24 11:01:40 -04:00
Nick Mathewson
af86895581 consdiffmgr test: do not launch a diff task that is already pending 2017-04-24 11:01:40 -04:00
Nick Mathewson
2e9e2d023b consdiffmgr: tests for consdiffmgr_validate() 2017-04-24 11:01:40 -04:00
Nick Mathewson
6cc21aa89c consdiffmgr: add tests for cdm_entry_get_sha3_value 2017-04-24 11:01:40 -04:00
Nick Mathewson
bb38657b77 consdiffmgr test: add a test for updating ht on clean/rescan.
This brings us back up to ~94% coverage
2017-04-24 11:01:40 -04:00
Nick Mathewson
de0142cd9d Expand diff-management test to cover reloading items from disk 2017-04-24 11:01:40 -04:00
Nick Mathewson
43d683e0ad Fix reference leak & handle leak in consensus_diff_worker_replyfn
Found by previous test.
2017-04-24 11:01:40 -04:00
Nick Mathewson
831e656baa consdiffmgr tests: add tests to validate diff lookup/application
This commit adds some helper functions to look up the diff from one
consensus and to make sure that applying it leads to another.  Then
we add them throughout the existing test cases.  Doing this turned
up a reference-leaking bug in consensus_diff_worker_replyfn.
2017-04-24 11:01:40 -04:00
Nick Mathewson
605bcfbf71 consdiffmgr: Enable in-progress test that was not previously working
Also, add a list of additional tests to write.
2017-04-24 11:01:40 -04:00
Nick Mathewson
7ca86b9cd6 Add a hashtable to consdiffmgr to keep track of diff status
In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
   * Remembering which diffs were in-progress, so that we didn't
     re-launch them.
   * Remembering which diffs had failed, so that we didn't try to
     recompute them over and over.
   * Having a fast way to look up the diff from a given consensus to
     the latest consensus of a given flavor.

This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem.  It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.
2017-04-24 11:01:40 -04:00
Nick Mathewson
69a212ff3d Consdiffmgr: extract "get a sha3 digest" function.
I'll be using this a lot in the hashtable tweaks here.
2017-04-24 11:01:40 -04:00
Nick Mathewson
655f1c8e01 consdiffmgr: function to re-validate stored sha3 digests at startup 2017-04-24 11:00:28 -04:00
Nick Mathewson
6c86e63029 Consdiffmgr: use aggressive-release flag on consensuses
This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
82bb8afb60 Add handle support to consensus_cache_entry_t
This will allow us to have weak references to cache entries.
2017-04-24 10:59:25 -04:00
Nick Mathewson
5726fec9c2 Consdiffmgr test: Make sure that diffs are removable
A diff is removable as soon as it no longer takes you to the most
recent consensus of the appropriate flavor.
2017-04-24 10:59:25 -04:00
Nick Mathewson
35f6b678ab Test the easiest cases of consdiffmgr_cleanup.
One more to go: deleting the old diffs.
2017-04-24 10:59:25 -04:00
Nick Mathewson
1fade37287 consdiffmgr non-test: check for initialization failure
Unfortunately, this test doesn't work, so I've left it
defined-out. There is currently no way in our unit tests to catch a
fatal assertion failure.
2017-04-24 10:59:25 -04:00
Nick Mathewson
d418f28cb5 consdiffmgr test: survive failures to compute a diff. 2017-04-24 10:59:25 -04:00
Nick Mathewson
b9c2f135bd Another consdiffmgr test: only generate the diffs that are needed
This test makes sure that we only generate the diffs we actually
want, rather than regenerating all the diffs every time anything
changes.
2017-04-24 10:59:25 -04:00
Nick Mathewson
7fc37d41b4 Unit tests for consdiffmgr module
Initial tests. These just try adding a few consensuses, looking
them up, and making sure that consensus diffs are generated in a
more or less reasonable-looking way.  It's enough for 87% coverage,
but it leaves out a lot of functionality.
2017-04-24 10:59:25 -04:00
Nick Mathewson
fe584f4012 Make cpuworker_queue_work function mockable.
I'll be using this in the unit tests for consdiffmgr.
2017-04-24 10:59:25 -04:00
Nick Mathewson
bc91808c47 Add a "Consensus diff manager" module.
This module's job is to remember old consensus documents, to
calculate their diffs on demand, and to .

There are some incomplete points in this code; I've marked them with
"XXXX". I intend to fix them in separate commits, since I believe
doing it in separate commits will make the branch easier to review.
2017-04-24 10:59:24 -04:00
George Kadianakis
39b5dca720 ed25519: Add python code to test our ed25519 validation.
See
https://lists.torproject.org/pipermail/tor-dev/2017-April/012213.html .
2017-04-24 16:34:53 +03:00
Nick Mathewson
9ba10d714a Merge branch 'consdiff_numeric_squashed' 2017-04-24 09:33:55 -04:00
Sebastian Hahn
a16de7a7cf consdiff: Reject ranges with non-numeric chars
Fixes bug #21964
2017-04-24 09:33:48 -04:00
Nick Mathewson
8ff2a8c0e2 Merge remote-tracking branch 'dgoulet/bug22032_031_01' 2017-04-24 09:30:30 -04:00
Taylor Yu
7bc636fdc9 Add regression test for #22304 2017-04-24 09:20:59 -04:00
Nick Mathewson
8b89faf424 Merge branch 'maint-0.3.0' 2017-04-24 09:20:43 -04:00
Nick Mathewson
ae374e0a56 Merge branch 'maint-0.2.9' into maint-0.3.0 2017-04-24 09:20:26 -04:00
David Goulet
4a9f689430 control: Wrong check on base16_decode return value
The GETINFO extra-info/digest/<digest> broke in commit 568dc27a19 that
refactored the base16_decode() API to return the decoded length.
Unfortunately, that if() condition should have checked for the correct length
instead of an error which broke the command in tor-0.2.9.1-alpha.

Fixes #22034

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-24 09:20:13 -04:00
Nick Mathewson
91553417c7 Fix a comment that held the remaining reference to the old api. 2017-04-21 16:39:05 -04:00
Nick Mathewson
d8ff01b2b4 Refactor directory_command_should_use_begindir to use directory_request_t 2017-04-21 15:55:23 -04:00
Nick Mathewson
2a99bf008d move the definition of directory_command_should_use_begindir 2017-04-21 15:54:50 -04:00
Nick Mathewson
cc677e88d4 Refactor directory_send_command to take a directory_request_t 2017-04-21 15:49:10 -04:00
Nick Mathewson
900d36e3b1 Rename, revise, and document directory_request_is_dir_specified. 2017-04-21 15:42:09 -04:00
Nick Mathewson
d7fab36037 Make directory_request_set_guard_state static. 2017-04-21 15:42:09 -04:00
Nick Mathewson
81045e19eb Document the new directory_request_t API and add some assertions 2017-04-21 15:42:03 -04:00
Nick Mathewson
3327e675fd Remove old directory_initiate_command_*() functions. 2017-04-21 15:14:52 -04:00
Nick Mathewson
8aadd60eac Replace remaining directory_initiate_command_* instances 2017-04-21 15:08:28 -04:00
Nick Mathewson
4e393f5318 Make launch_direct_bridge_descriptor_fetch use dirreqs. 2017-04-21 14:47:14 -04:00
Nick Mathewson
4ae65bb5d3 Refactor directory_get_from_dirserver to use request API. 2017-04-21 14:40:48 -04:00
Nick Mathewson
eee733be33 Refactor directory_post_to_dirservers to use requests 2017-04-21 14:35:42 -04:00
Nick Mathewson
c300fd7e65 Have directory_get_from_all_authorities use requests. 2017-04-21 14:31:35 -04:00
Nick Mathewson
a55bd00b0f continue refactoring directory API: remove initiate_command_rend
This commit mainly moves the responsibility for directory request
construction one level higher.  It also allows a directory request
to contain a pointer to a routerstatus, which will get turned into
the correct contact information at the last minute.
2017-04-21 14:18:06 -04:00
David Goulet
74f7657e56 hs: Fix a spurious warn in rend_log_intro_limit
We do dump HS stats now at log info everytime the intro circuit creation retry
period limit has been reached. However, the log was upgraded to warning if we
actually were over the elapsed time (plus an extra slop).

It is actually something that will happen in tor in normal case. For instance,
if the network goes down for 10 minutes then back up again making
have_completed_a_circuit() return false which results in never updating that
retry period marker for a service.

Fixes #22032

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-21 11:09:34 -04:00
Nick Mathewson
b7dd3518b9 missing smartlist_free in new consdiff test 2017-04-18 22:56:34 -04:00
Nick Mathewson
afe1af7de6 Merge remote-tracking branch 'asn/bug21971' 2017-04-18 21:38:28 -04:00
Nick Mathewson
eeb266c139 Merge remote-tracking branches 'sebastian/consdiff_add' and 'sebastian/consdiff_newline' 2017-04-18 21:31:53 -04:00
George Kadianakis
e1a59ade95 prop224: Add time period functions and unittests
This will be used by the build blinded key functions.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-18 11:03:15 -04:00
Alexander Færøy
e6c6606a17
Expose tor_compress_is_compression_bomb() as part of the public compression API.
This patch exposes the old `is_compression_bomb()` function as a public
API as part of the compression module.

See https://bugs.torproject.org/21663
2017-04-18 03:12:19 +02:00
Alexander Færøy
6b5172bcf2
Expose tor_compress_memory_level() as a public function in the compression module.
This patch makes the internal `get_memlevel()` a part of the public
compression API as `tor_compress_memory_level()`.

See https://bugs.torproject.org/21663
2017-04-18 02:23:25 +02:00
Alexander Færøy
b8c9f229d7
Rename write_to_buf_zlib() to write_to_buf_compress().
See https://bugs.torproject.org/21663
2017-04-18 01:23:39 +02:00
Alexander Færøy
40ed68290e
Rename zlib_state to compress_state in the test_buffers.
See https://bugs.torproject.org/21663
2017-04-18 01:12:01 +02:00
Alexander Færøy
4b9349192d
Rename zlib_state to compress_state in dir_connection_t.
This patch renames the `zlib_state` field in `dir_connection_t` to
`compress_state`.

See https://bugs.torproject.org/21663
2017-04-18 01:10:25 +02:00
George Kadianakis
0c39cefc52 prop224 tests: Remove useless NULL check before circuit_free().
Addresses coverity issue CID 1405130.
2017-04-17 16:36:55 +03:00
George Kadianakis
a3ce303432 prop224 tests: Don't use tt_size_op to compare ssize_t.
Addresses coverity CID 1405129.
2017-04-17 16:23:58 +03:00
Alexander Færøy
3c4459bcbf
Refactor the streaming compression code.
This patch refactors our streaming compression code to allow us to
extend it with non-zlib/non-gzip based compression schemas.

See https://bugs.torproject.org/21663
2017-04-17 14:57:37 +02:00
Alexander Færøy
44cb86adbe
Rename tor_gzip_{compress,uncompress} to tor_{compress,uncompress}.
To allow us to use the API name `tor_compress` and `tor_uncompress` as
the main entry-point for all compression/uncompression and not just gzip
and zlib.

See https://bugs.torproject.org/21663
2017-04-17 14:29:10 +02:00
Alexander Færøy
e8b025dfc3
Rename zlib_compression_level_t to compression_level_t.
See https://bugs.torproject.org/21663
2017-04-17 14:22:13 +02:00
Alexander Færøy
7460b9755a
Remove unused function is_gzip_supported().
This patch removes the unused `is_gzip_supported()` and changes the
documentation string around the `compress_method_t` enumeration to
explicitly state that both `ZLIB_METHOD` and `GZIP_METHOD` are both
always supported.

Zlib version 1.2.0 was released on the 9'th of March, 2003 according to
their ChangeLog.

See https://bugs.torproject.org/21663
2017-04-17 14:11:35 +02:00
Alexander Færøy
4b834e0d5e
Fix whitespace in test_util_gzip() around OP_*`.
See https://bugs.torproject.org/21663
2017-04-17 14:07:23 +02:00
Alexander Færøy
a8821d8366
Use tt_int_op() over tt_assert() and do explicit NULL checks in test_util_gzip().
This patch changes some of the tt_assert() usage in test_util_gzip() to
use tt_int_op() to get better error messages upon failure.

Additionally we move to use explicit NULL checks.

See https://bugs.torproject.org/21663
2017-04-17 14:02:16 +02:00
Sebastian Hahn
5a7e39c0cb Fix diff generation with line added at start
The consdiff generation logic would skip over lines added at the start of the
second file, and generate a diff that it would the immediately refuse because
it couldn't be used to reproduce the second file from the first. Fixes #21996.
2017-04-17 10:10:46 +02:00
Sebastian Hahn
459643502b Don't accept ranges for add commands in consdiff
Fixes ticket #21963
2017-04-17 06:49:27 +02:00
Nick Mathewson
91a6fbcf77 Begin work to turn "directory request" into an object
This patch adds the object type, and makes all of the functions
that used to feed into directory_initiate_command_rend() feed into
that type instead.
2017-04-16 18:45:48 -04:00
Nick Mathewson
b022ea32a6 Expand cpuworker API to allow other work types 2017-04-15 11:21:33 -04:00
Nick Mathewson
222122450c Add a config_line_prepend() function 2017-04-15 11:21:32 -04:00
Nick Mathewson
06ecb9432f conscache.c: do not match entries that are slated for removal. 2017-04-15 11:21:32 -04:00
Daniel Pinto
fa04fe1674 MyFamily config string is now a list. #4998 2017-04-14 13:04:37 +01:00
Nick Mathewson
b081a7ed21 Merge branch 'ticket21891_031_01_squashed' 2017-04-13 16:43:13 -04:00
David Goulet
0565f5a3bb hs: Make the service list pruning function public
The reason for making the temporary list public is to keep it encapsulated in
the rendservice subsystem so the prop224 code does not have direct access to
it and can only affect it through the rendservice pruning function.

It also has been modified to not take list as arguments but rather use the
global lists (main and temporary ones) because prop224 code will call it to
actually prune the rendservice's lists. The function does the needed rotation
of pointers between those lists and then prune if needed.

In order to make the unit test work and not completely horrible, there is a
"impl_" version of the function that doesn't free memory, it simply moves
pointers around. It is directly used in the unit test and two setter functions
for those lists' pointer have been added only for unit test.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-13 16:25:49 -04:00
Nick Mathewson
ba89520593 fix wide lines 2017-04-13 14:36:57 -04:00
Nick Mathewson
902672eac6 Merge branch 'ticket21889_031_01_squashed' 2017-04-13 14:23:59 -04:00
George Kadianakis
f02868bb53 hs: Add service-side circuitmap API.
Now we have separate getters and setters for service-side and relay-side. I
took this approach over adding arguments to the already existing methods to
have more explicit type-checking, and also because some functions would grow
too large and dirty.

This commit also fixes every callsite to use the new function names which
modifies the legacy HS (v2) and the prop224 (v3) code.

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-13 14:23:51 -04:00
Nick Mathewson
63e39b291e Merge remote-tracking branch 'dgoulet/bug21155_031_02' 2017-04-13 09:56:14 -04:00
Nick Mathewson
28ec2d9c2c Merge remote-tracking branch 'dgoulet/ticket21919_031_01' 2017-04-13 09:38:59 -04:00
Nick Mathewson
755c88a474 Merge branch 'asn/prop224-ntor-v2-squashed' 2017-04-13 09:22:34 -04:00
George Kadianakis
ea5901bf1c prop224: Add Python integration tests for HS ntor.
This test is identical to the ./src/test/test_ntor.sh integration test.
2017-04-13 09:22:19 -04:00
George Kadianakis
18ee145cda prop224: Add basic HS ntor unittest.
The test checks that introduce1/rendezvous1 key material is generated
correctly both for client-side and service-side.
2017-04-13 09:22:19 -04:00
George Kadianakis
50b0bc5bfe prop224: Add module that performs the HS ntor handshake.
and also does the key expansion.
2017-04-13 09:22:19 -04:00
David Goulet
bf68c78e94 hs: Make check-spaces happy after rename
Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-12 07:53:05 -04:00
David Goulet
6bacc3c7a8 hs: Change trunnel prop224 cell's namespace
One of the goals of this change is to have trunnel API/ABI being more explicit
so we namespace them with "trn_*". Furthermore, we can now create
hs_cells.[ch] without having to confuse it with trunnel which used to be
"hs_cell_*" before that change.

Here are the perl line that were used for this rename:

  perl -i -pe 's/cell_extension/trn_cell_extension/g;' src/*/*.[ch]
  perl -i -pe 's/cell_extension/trn_cell_extension/g;' src/trunnel/hs/*.trunnel
  perl -i -pe 's/hs_cell_/trn_cell_/g;' src/*/*.[ch]
  perl -i -pe 's/hs_cell_/trn_cell_/g;' src/trunnel/hs/*.trunnel

  And then "./scripts/codegen/run_trunnel.sh" with trunnel commit id
  613fb1b98e58504e2b84ef56b1602b6380629043.

Fixes #21919

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-11 13:46:41 -04:00
David Goulet
e5fc02c81c config: Warn if EntryNodes and HiddenService are used together
Pinning EntryNodes along with hidden services can be possibly harmful (for
instance #14917 and #21155) so at the very least warn the operator if this is
the case.

Fixes #21155

Signed-off-by: David Goulet <dgoulet@torproject.org>
2017-04-10 13:22:36 -04:00
Nick Mathewson
01fc93ffef Merge remote-tracking branch 'dgoulet/bug16706_031_01' 2017-04-10 12:12:14 -04:00
Nick Mathewson
f5258045c4 Fix some no-longer-reasonable unit tests for base64_decode()
These tests tried to use ridiculously large buffer sizes to check
the sanity-checking in the code; but since the sanity-checking
changed, these need to change too.
2017-04-10 12:08:31 -04:00
Taylor Yu
26dd4d92cf Remove SR_COMMIT_LEN workaround
Now that base64_decode() checks the destination buffer length against
the actual number of bytes as they're produced, shared_random.c no
longer needs the "SR_COMMIT_LEN+2" workaround.
2017-04-07 18:19:15 -04:00