The pruning process and the deleting ephemeral service function iterates over
all circuits and were asserting on rend_data for a matching circuit. This is
not good because now we have v3 circuits without a rend_data.
Fixes#23429
Signed-off-by: David Goulet <dgoulet@torproject.org>
Use the valid_after time from the consensus to get the time period number else
we might get out of sync with the overlap period that uses valid_after.
Make it an optional feature since some functions require passing a
specific time (like hs_get_start_time_of_next_time_period()).
Signed-off-by: David Goulet <dgoulet@torproject.org>
Undeprecate it;
rename it to TestingClientDNSRejectInternalAddresses;
add the old name as an alias;
reject configurations where it is set but TestingTorNetwork is not;
change the documentation accordingly.
Closes tickets 21031 and 21522.
Version 3 hidden service needs rendezvous point that have the protocol version
HSRend >= 2 else the rendezvous cells are rejected.
Fixes#23361
Signed-off-by: David Goulet <dgoulet@torproject.org>
The chdir() call in RunAsDaemon makes the behavior here surprising,
and either way of trying to resolve the surprise seems sure to
startle a significant fraction of users. Instead, let's refuse to
guess, and refuse these configurations.
Closes ticket 22731.
I'm doing this using the Proxy-Authorization: header to support
clients that understand it, and with a new tor-specific header that
makes more sense for our use.
By convention, a function that frobs a foo_t should be called
foo_frob, and it should have a foo_t * as its first argument. But
for many of the buf_t functions, the buf_t was the final argument,
which is silly.
Our convention is that functions which manipulate a type T should be
named T_foo. But the buffer functions were super old, and followed
all kinds of conventions. Now they're uniform.
Here's the perl I used to do this:
\#!/usr/bin/perl -w -i -p
s/read_to_buf\(/buf_read_from_socket\(/;
s/flush_buf\(/buf_flush_to_socket\(/;
s/read_to_buf_tls\(/buf_read_from_tls\(/;
s/flush_buf_tls\(/buf_flush_to_tls\(/;
s/write_to_buf\(/buf_add\(/;
s/write_to_buf_compress\(/buf_add_compress\(/;
s/move_buf_to_buf\(/buf_move_to_buf\(/;
s/peek_from_buf\(/buf_peek\(/;
s/fetch_from_buf\(/buf_get_bytes\(/;
s/fetch_from_buf_line\(/buf_get_line\(/;
s/fetch_from_buf_line\(/buf_get_line\(/;
s/buf_remove_from_front\(/buf_drain\(/;
s/peek_buf_startswith\(/buf_peek_startswith\(/;
s/assert_buf_ok\(/buf_assert_ok\(/;
This lets us drop the testing-only function buf_get_first_chunk_data(),
and lets us implement proto_http and proto_socks without looking at
buf_t internals.
The service needs the latest SRV and set of relays for the best accurate
hashring to upload its descriptor to so it needs a live consensus thus don't
do anything until we have it.
Fixes#23331
Signed-off-by: David Goulet <dgoulet@torproject.org>
When merging #20657, somehow hs_service_dir_info_changed() became unused
leading to not use the re-upload to HSDir when we were missing information
feature.
Turns out that it is not possible to pick an HSDir with a missing descriptor
because in order to compute the HSDir index, the descriptor is mandatory to
have so we can know its position on the hashring.
This commit removes that dead feature and fix the
hs_service_dir_info_changed() not being used.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Based on questions and comments from dgoulet, I've tried to fill
in the reasoning about why these functions work in the way that they
do, so that it will be easier for future programmers to understand
why this code exists and works the way it does.
We used to check if it was set to 0 which is what unused circuit have but when
the rendezvous circuit was cannibalized, the timestamp_dirty is not 0 but we
still need to reset it so we can actually use it without having the chance of
expiring the next second (or very soon).
Fixes#23123
Signed-off-by: David Goulet <dgoulet@torproject.org>
The function was never returning an error code on failure to parse the
OutboundAddress* options.
In the process, it was making our test_options_validate__outbound_addresses()
not test the right thing.
Fixes#23366
Signed-off-by: David Goulet <dgoulet@torproject.org>
This fixes a serious bug in our hsdir set change logic:
We used to add nodes in the list of previous hsdirs everytime we
uploaded to a new hsdir and we only cleared the list when we built a new
descriptor. This means that our prev_hsdirs list could end up with 7
hsdirs, if for some reason we ended up uploading our desc to 7 hsdirs
before rebuilding our descriptor (e.g. this can happen if the set of
hsdirs changed).
After our previous hdsir set had 7 nodes, then our old algorithm would
always think that the set has changed since it was comparing a smartlist
with 7 elements against a smartlist with 6 elements.
This commit fixes this bug, by clearning the prev_hsdirs list before we
upload to all hsdirs. This makes sure that our prev_hsdirs list always
contains the latest hsdirs!
Our logic for detecting hsdir set changes was needlessly compicated: we
had to sort smartlists and compare them.
Instead, we can simplify things by employing the following logic:
"We should reupload our descriptor if the latest HSDir set contains
nodes that were not previously there"
Since we can't be sure that we can unlink enough files on windows
here, let's let the number of permitted entries grow huge if it
really must.
We do this by letting the storagedir hold lots of entries, but still
trying to keep the number of entries under the configured limit. We
also have to tell consdiffmgr not to freak out if it can't actually
remove enough entries.
Part of a fix for bug 22752
Some parentheses were missing making the rend_max_intro_circs_per_period()
return a lower value than it was suppose to.
The calculation is that a service at most will open a number of intro points
that it wants which is 3 by default or HiddenServiceNumIntroductionPoints. Two
extra are launched for performance reason. Finally, this can happen twice for
two descriptors for the current and next time period.
From:
2 * n_intro_wanted + 2
...which resulted in 8 for 3 intro points, this commit fixes it to:
(n_intro_wanted + 2) * 2
... resulting in 12 possible intro point circuit which is the correct maximum
intro circuit allowed per period.
Last, this commit rate limits the the log message if we ever go above that
limit else over a INTRO_CIRC_RETRY_PERIOD, we can print it often!
Fixes#22159
Signed-off-by: David Goulet <dgoulet@torproject.org>
That check was wrong:
a) We should be making sure that the size of `key` is big enough before
proceeding, since that's the buffer that we would overread with the
tor_memeq() below.
The old check used to check that `req_key_str` is big enough which is
not right, since we won't read deep into that buffer.
The new check makes sure that `key` has enough size to survive the
tor_memeq(), and if not it moves to the next element of the strmap.
b) That check shouldn't be a BUG since that strmap contains
variable-sized elements and we should not be bugging out if we happen
to compare a small sized element (v2) to a bigger one (v3).
This way, we can clear off the directory requests from our cache and thus
allow the next client to query those HSDir again at the next SOCKS connection.
Signed-off-by: David Goulet <dgoulet@torproject.org>
v3 client now cleans up the HSDir request cache when a connection to a service
was successful.
Closes#23308
Signed-off-by: David Goulet <dgoulet@torproject.org>
We used to not copy the state which means that after HUP we would forget
if we are in overlap mode or not. That caused bugs where the service
would enter overlap mode twice, and rotate its descs twice, causing all
sorts of bugs.
Apart from the fact that a newly allocated service doesn't have descriptors
thus the move condition can never be true, the service needs the descriptor
signing key to cross-certify the authentication key of each intro point so we
need to move the descriptors between services and not only the intro points.
Fixes#23056
Signed-off-by: David Goulet <dgoulet@torproject.org>
We refactor the descriptor reupload logic to be similar to the v2 logic
where we update a global 'consider_republishing_rend_descriptors' flag
and then we use that to check for hash ring changes during the global
hidden service callbacks.
This fixes bugs where we would inspect the hash ring immediately as we
receive new dirinfo (e.g. consensus) but before running the hidden
service housekeeping events. That was leaving us in an inconsistent
state wrt hsdir indices and causing bugs all around.
The problem was that when we went from overlap mode to non-overlap mode,
we were not wiping the 'desc_next' descriptor and instead we left it on
the service. This meant that all functions that iterated service
descriptors were also inspecting the useless 'desc_next' descriptor that
should have been deleted.
This commit refactors rotate_all_descriptors() so that it rotates
descriptor both when entering overlap mode and also when leaving it.
A client can re-extend up to 3 intro points on the same circuit. This happens
when we get NACKed by the intro point for which we choose a new intro and
re-extend the circuit to it.
That process can be arbitrarly long so reset the dirty timestamp of the
circuit everytime we choose to re-extend so we get a bit more time to actually
do our introduction.
This is a client circuit so it is short live once opened thus giving us a bit
more time to complete the introduction is ok.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When looking for an introduction circuit in circuit_get_best(), we log an info
message if we are about to launch a new intro circuit in parallel. However,
the condition was considering marked for close circuit leading to the function
triggering the log info even though there is actually no valid intro circuit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Only register the RP circuit when it opens and not when we send the INTRODUCE1
cell else, when re-extending to a new IP, we would register the same RP
circuit with the same cookie twice leading to the circuit being closed.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Changed the assert_intro_circ_ok() to an almost non fatal function so tor can
recover properly. We keep the anonymity assert because if that is not right,
we have much deeper problems and client should stop sending bytes to the
network immediately.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This function has been replaced by hs_client_receive_rendezvous_acked(() doing
the same exact thing for both v2 and v3 service.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The client needs to find the right intro point object from the circuit
identity digest it is opened to. This new function does that.
Signed-off-by: David Goulet <dgoulet@torproject.org>
New function named hs_cell_introduce1_data_clear() is introduced to clear off
an hs_cell_introduce1_data_t object.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When a client decodes a descriptor, make sure it matches the expected blinded
key which is derived from the hidden service identity key.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Needed by the client when fetching a descriptor. This function checks the
directory purpose and hard assert if it is not for fetching.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit makes the client use the intro point state cache. It notes down
when we get a NACK from the intro point and then uses that cache to decide if
it should either close the circuits or re-extend to a new intro point.
This also introduces a very useful function that checks if an intro point is
usable that is query the state cache and checks a series of requirement.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This cache keeps track of the state of intro points which is needed when we
have failures when using them. It is similar to the failure cache of the
legacy system.
At this commit, it is unused but initialized, cleanup and freed.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This moves it to hs_client.c so it can be used by both system (legacy and
prop224). For now, only the legacy system uses it.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Don't assert() on rend_data when closing circuits to report an IP failure. The
prop224 code doesn't have yet the support for this.
Signed-off-by: David Goulet <dgoulet@torproject.org>
For now, prop224 doesn't have a mechanism to note down connection attempts so
we only do it for legacy system using rend_data.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The client can now handle RENDEZVOUS2 cell when they arrive. This consolidate
both hidden service version in one function.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The client is now able to handle an INTRODUCE_ACK cell and do the appropriate
actions.
An intro point failure cache is missing and a way to close all intro point
that were launched in parallel. Some notes are in the comment for that.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a function to parse an INTRODUCE_ACK cell in hs_cell.c. Furthermore, add
an enum that lists all possible expected status code.
Signed-off-by: David Goulet <dgoulet@torproject.org>
From an edge connection object, add a function that randomly pick an
introduction point for the requested service.
This follows the code design of rend_client_get_random_intro() and returns an
extend_info_t object ready to be used to extend to.
At this commit, it is not used yet.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When a descriptor fetch has completed and it has been successfully stored in
the client cache, this callback will take appropriate actions to attach
streams and/or launch neede circuits to connect to the service.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Client now handles a RENDEZVOUS_ESTABLISHED cell when it arrives on the
rendezvous circuit. This new function applies for both the legacy system and
prop224.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a function to build the cell.
Add a the logic to send the cell when the rendezvous circuit opens.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Make a single entry point for the entire HS subsystem when a client circuit
opens (every HS version).
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a function in hs_cell.{c|h} for a client to build an INTRODUCE1 cell using
an object that contains all the needed keys to do so.
Add an entry point in hs_client.c that allows a tor client to send an
INTRODUCE1 cell on a given introduction circuit.
It includes the building of the cell, sending it and the setup of the
rendezvous circuit with the circuit identifier.
The entry point function is still unused at this commit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The hs circuit file had this function that takes a list of link specifiers and
return a newly allocated extend info object. Make it public so the client side
can also use it to be able to extend to introduction point.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Put all the possible assert() we can do on a client introduction circuit in
one helper function to make sure it is valid and usable.
It is disabled for now so gcc doesn't complain that we have a unused function.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit only moves code into a function. The client code will need a way
to take a bunch of descriptor link specifier object and encode them into link
specifiers objects.
Make this a public function so it can be used outside of hs_descriptor.c.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This will be useful to the hidden service subsystem that needs to go over all
connections of a certain state to attach them to a hidden service circuit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
- Add tests that ensure that SOCKS requests for v2/v3 addresses get
intercepted and handled.
- Add test that stores and lookups an HS descriptor in the client-side cache.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Once a descriptor has been successfully downloaded from an HSDir, we flag the
directory connection to "has fetched descriptor" so the connection subsystem
doesn't trigger a new fetch on success.
Same has DIR_PURPOSE_HAS_FETCHED_RENDDESC_V2 but for prop224.
Signed-off-by: David Goulet <dgoulet@torproject.org>
- Also add tests for the hidserv_req subsystem.
- Introduce purge_v2_hidserv_req() wrapper to simplify v2 code.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Specifically move the pick_hsdir() function and all the HSDir request tracking
code. We plan to use all that code both for v2 and v3.
This commit only moves code.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Right now there's a single warn_if_unnamed flag for
router_get_consensus_status_by_nickname() and
node_get_by_nickname(), that is nearly always 1. I've turned it
into an 'unsigned' bitfield, and inverted its sense. I've added the
flags argument to node_get_by_hex_id() too, though it does nothing
there right now.
I've removed the router_get_consensus_status_by_nickname() function,
since it was only used in once place.
This patch changes the warning behavior of GETINFO ns/name/<name>,
since all other name lookups from the controller currently warn.
Later I'm going to add more flags, for ed25519 support.
We will need to edit this function, and it's already pretty huge. Let's make
it a bit smaller.
This commit moves code, fixes a 80 char line and add two lines at the start to
make it compile. Trivial change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We need this func so that we recognize SOCKS conns to v3 addresses.
- Also rename rend_valid_service_id() to rend_valid_v2_service_id()
- Also move parse_extended_hostname() tests to their own unittest, and
add a v3 address to the test as well.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When we enter overlap mode we start using the next hsdir index of
relays. However, we only compute the next hsdir index of relays when we
receive a consensus or their descriptor. This means that there is a
window of time between entering the overlap period and fetching the
consensus where relays have their next hsdir index uninitialized. This
patch fixes this by recomputing all hsdir indices when we first enter
the overlap period.
We want to reupload our descriptor if its set of responsible HSDirs
changed to minimize reachability issues.
This patch adds a callback everytime we get new dirinfo which checks if
the hash ring changed and reuploads descriptor if needed.
Because the HS subsystem calls it every second, change the log level to debug
so it doesn't spam the info log.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Our Windows compiler treats "time_t" as long long int but Linux likes it
long int so cast those to make Windows happy.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This change should improve overhead for downloading small numbers of
descriptors and microdescriptors by improving compression
performance and lowering directory request overhead.
Closes ticket 23220.
The old implementation would fail with super-long inputs. We never
gave it any, but still, it's nicer to dtrt here.
Reported by Guido Vranken. Fixes bug 19281.
- Fix various ssize_t/size_t confusions in the tests.
- Fix a weird memset argument:
"bad_memset: Argument -16 in memset loses precision in
memset(&desc_two->blinded_kp.pubkey.pubkey, -16, 32UL)."
- Fix check_after_deref instance in check_state_line_for_service_rev_counter():
"check_after_deref: Null-checking items suggests that it may be null,
but it has already been dereferenced on all paths leading to the
check."
Add a common function for both legacy and prop224 hidden service to increment
and decrement the rendezvous stream counter on an origin circuit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We used to have a small HS desc cert lifetime but those certs can stick
around for 36 hours if they get initialized in the beginning of overlap
period.
[warn] Bug: Non-fatal assertion !(hs_desc_encode_descriptor(desc->desc, &desc->signing_kp, &encoded_desc) < 0) failed in
upload_descriptor_to_hsdir at src/or/hs_service.c:1886. Stack trace: (on Tor 0.3.2.0-alpha-dev b4a14555597fb9b3)
I repurposed the old directory_request_set_hs_ident() into a new
directory_request_upload_set_hs_ident() which is only used for the
upload purpose and so it can assert on the dir_purpose.
When coding the client-side we can make a second function for fetch.
We used to do:
h = H(BLIND_STRING | H(A | s | B | N )
when we should be doing:
h = H(BLIND_STRING | A | s | B | N)
Change the logic so that hs_common.c does the hashing, and our ed25519
libraries just receive the hashed parameter ready-made. That's easier
than doing the hashing on the ed25519 libraries, since that means we
would have to pass them a variable-length param (depending on whether
's' is set or not).
Also fix the ed25519 test vectors since they were also double hashing.
We also had to alter the SRV functions to take a consensus as optional
input, since we might be setting our HSDir index using a consensus that
is currently being processed and won't be returned by the
networkstatus_get_live_consensus() function.
This change has two results:
a) It makes sure we are using a fresh consensus with the right SRV value
when we are calculating the HSDir hash ring.
b) It ensures that we will not use the sr_get_current/previous()
functions when we don't have a consensus which would have falsely
triggered the disaster SRV logic.
In Nick's words:
"We want to always return false if the platform is a Tor version, and it
is not as new as 0.3.0.8 -- but if the platform is not a Tor version, or
if the version is as new as 0.3.0.8, then we want to obey the protocol
list.
That way, other implementations of our protocol won't have to claim any
particular Tor version, and future versions of Tor will have the freedom
to drop this protocol in the distant future."
- Fix log message format string.
- Do extra circuit purpose check.
- wipe memory in a clear function
- Make sure we don't double add intro points in our list
- Make sure we don't double close intro circuits.
- s/tt_u64_op/tt_i64_op/
Turns out that introduction points don't care about the INTRODUCE2 cell
format as long as the top field is LEGACY_KEY_ID as expected. So let's
use a single INTRODUCE format regardless of the introduction point being
legacy or not.
This also removes the polymorphic void* situation.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We consider to be in overlap mode when we are in the period of time between a
fresh SRV and the beginning of the new time period (in the normal network this
is between 00:00 and 12:00 UTC). This commit edits that function to use the
above semantic logic instead of absolute times.
Signed-off-by: David Goulet <dgoulet@torproject.org>
It used to be that time periods were 24 hours long even on chutney,
which made testing harder. With this commit, time periods have the same
length as a full SRV protocol run, which means that they will change
every 4 minutes in a 10-second voting interval chutney network!
Make this function public so we can use it both in hs_circuit.c and
hs_service.c to avoid code duplication.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This introduces a callback to relaunch a service rendezvous circuit when a
previous one failed to build or expired.
It unifies the legacy function rend_service_relaunch_rendezvous() with one for
specific to prop224. There is now only one entry point for that which is
hs_circ_retry_service_rendezvous_point() supporting both legacy and prop224
circuits.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Change the timing for intro point's lifetime and maximum amount of circuit we
are allowed to launch in a TestingNetwork. This is particurlarly useful for
chutney testing to test intro point rotation.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When the circuit is about to be freed which has been marked close before, for
introduction circuit we now call this has_closed() callback so we can cleanup
any introduction point that have retried to many times or at least flag them
that their circuit is not established anymore.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We used to use NULL subcredential which is a terrible terrible idea. Refactor
HS unittests to use subcredentials.
Also add some non-fatal asserts to make sure that we always use subcredentials
when decoding/encoding descs.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit refactors the handle_hs_exit_conn() function introduced at a prior
commit that connects the rendezvous circuit to the edge connection used to
connect to the service virtual port requested in a BEGIN cell.
The refactor adds the support for prop224 adding the
hs_service_set_conn_addr_port() function that has the same purpose has
rend_service_set_connection_addr_port() from the legacy code.
The rend_service_set_connection_addr_port() has also been a bit refactored so
the common code can be shared between the two HS subsystems (legacy and
prop224).
In terms of functionallity, nothing has changed, we still close the circuits
in case of failure for the same reasons as the legacy system currently does.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit simply moves the code from the if condition of a rendezvous
circuit to a function to handle such a connection. No code was modified
_except_ the use or rh.stream_id changed to n_stream->stream_id so we don't
have to pass the cell header to the function.
This is groundwork for prop224 support which will break down the
handle_hs_exit_conn() depending on the version of hidden service the circuit
and edge connection is for.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Introduction point are rotated either if we get X amounts of INTRODUCE2 cells
on it or a time based expiration. This commit adds two consensus parameters
which are the min and max value bounding the random value X.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Imagine a Tor network where you have only 8 nodes available due to some
reasons. And your hidden service wants 8 introduction points. Everything is
fine but then a node goes down bringing the network to 7. The service will
retry 3 times that node and then give up but keep it in a failure cache for 5
minutes (INTRO_CIRC_RETRY_PERIOD) so it doesn't retry it non stop and exhaust
the maximum number of circuit retry.
In the real public network today, this is unlikely to happen unless the
ExcludeNodes list is extremely restrictive.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit adds a directory command function to make an upload directory
request for a service descriptor.
It is not used yet, just the groundwork.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This hsdir index value is used to give an index value to all node_t (relays)
that supports HSDir v3. An index value is then computed using the blinded key
to know where to fetch/upload the service descriptor from/to.
To avoid computing that index value everytime the client/service needs it, we
do that everytime we get a new consensus which then doesn't change until the
next one. The downside is that we need to sort them once we need to compute
the set of responsible HSDir.
Finally, the "hs_index" function is also added but not used. It will be used
in later commits to compute which node_t is a responsible HSDir for the
service we want to fetch/upload the descriptor.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add this helper function that can lookup and return all the needed object from
a circuit identifier. It is a pattern we do often so make it nicer and avoid
duplicating it everywhere.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add the entry point from the circuit subsystem of "circuit has opened" which
is for all type of hidden service circuits. For the introduction point, this
commit actually adds the support for handling those circuits when opened and
sending ESTABLISH_INTRO on a circuit.
Rendevzou point circuit aren't supported yet at this commit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit adds the functionality for a service to build its descriptor.
Also, a global call to build all descriptors for all services is added to the
service scheduled events.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add the main loop entry point to the HS service subsystem. It is run every
second and make sure that all services are in their quiescent state after that
which means valid descriptors, all needed circuits opened and latest
descriptors have been uploaded.
For now, only v2 is supported and placeholders for v3 actions for that main
loop callback.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a function for both the client and service side that is building a blinded
key from a keypair (service) and from a public key (client). Those two
functions uses the current time period information to build the key.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a new and free function for hs_desc_intro_point_t so the service can use
them to setup those objects properly.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Fixes bug 23106; bugfix on 0.2.4.8-alpha.
Fortunately, we only support big-endian and little-endian platforms,
and on both of those, hton*() and ntoh*() behave the same. And if
we did start to support middle endian systems (haha, no), most of
_those_ have hton*(x) == ntoh*(x) too.
Since we start with desc_clean_since = 0, we should have been
starting with non-null desc_dirty_reason.
Fixes bug 22884; bugfix on 0.2.3.4-alpha when X-Desc-Gen-Reason was
added.
Patch from Vort; fixes bug 23081; bugfix on fd992deeea in
0.2.1.16-rc when set_main_thread() was introduced.
See the changes file for a list of all the symptoms this bug has
been causing when running Tor as a Windows Service.
The condition was always true meaning that we would reconsider updating our
directory information every 2 minutes.
If valid_until is 6am today, then now - 24h == 1pm yesterday which means that
"valid_until < (now - 24h)" is false. But at 6:01am tomorrow, "valid_until <
(now - 24h)" becomes true which is that point that we shouldn't trust the
consensus anymore.
Fixes#23091
Signed-off-by: David Goulet <dgoulet@torproject.org>
One log statement was a warning and has been forgotten. It is triggered for a
successful attempt at introducting from a client.
It has been reported here:
https://lists.torproject.org/pipermail/tor-relays/2017-August/012689.html
Three other log_warn() statement changed to protocol warning because they are
errors that basically can come from the network and thus triggered by anyone.
Fixes#23078.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Now that half the threads are permissive and half are strict, we
need to make sure we have at least two threads, so that we'll
have at least one of each kind.
A prop224 descriptor was missing the onion key for an introduction point which
is needed to extend to it by the client.
Closes#22979
Signed-off-by: David Goulet <dgoulet@torproject.org>
Remove the legacy intro point key because both service and client only uses
the ed25519 key even though the intro point chosen is a legacy one.
This also adds the CLIENT_PK key that is needed for the ntor handshake.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We need to keep these around for TAP and old-style hidden services,
but they're obsolete, and we shouldn't encourage anyone to use them.
So I've added "obsolete" to their names, and a comment explaining
what the problem is.
Closes ticket 23026.
It makes more sense to have the version in the configuration object of the
service because it is afterall a torrc option (HiddenServiceVersion).
Signed-off-by: David Goulet <dgoulet@torproject.org>
The added function frees any allocated pointers in a service configuration
object and reset all values to 0.
Signed-off-by: David Goulet <dgoulet@torproject.org>
As per nickm suggestion, an array of config handlers will not play well with
our callgraph tool.
Instead, we'll go with a switch case on the version which has a good side
effect of allowing us to control what we pass to the function intead of a fix
set of parameters.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a helper function to parse uint64_t and also does logging so we can reduce
the amount of duplicate code.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This tests our hs_config.c API to properly load v3 services and register them
to the global map. It does NOT test the service object validity, that will be
the hs service unit test later on.
At this commit, we have 100% code coverage of hs_config.c.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Every hidden service option don't apply to every version so this new function
makes sure we don't have for instance an option that is only for v2 in a v3
configured service.
This works using an exclude lists for a specific version. Right now, there is
only one option that is not allowed in v3. The rest is common.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Try to load or/and generate service keys for v3. This write both the public
and private key file to disk along with the hostname file containing the onion
address.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This also adds unit test and a small python script generating a deterministic
test vector that a unit test tries to match.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit adds the support in the HS subsystem for loading a service from a
set of or_options_t and put them in a staging list.
To achieve this, service accessors have been created and a global hash map
containing service object indexed by master public key. However, this is not
used for now. It's ground work for registration process.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Introduces hs_init() located in hs_common.c which initialize the entire HS v3
subsystem. This is done _prior_ to the options being loaded because we need to
allocate global data structure before we load the configuration.
The hs_free_all() is added to release everything from tor_free_all().
Note that both functions do NOT handle v2 service subsystem but does handle
the common interface that both v2 and v3 needs such as the cache and
circuitmap.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add the hs_config.{c|h} files contains everything that the HS subsystem needs
to load and configure services. Ultimately, it should also contain client
functions such as client authorization.
This comes with a big refactoring of rend_config_services() which has now
changed to only configure a single service and it is stripped down of the
common directives which are now part of the generic handler.
This is ground work for prop224 of course but only touches version 2 services
and add XXX note for version 3.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This object is the foundation of proposal 224 service work. It will change
and be adapted as it's being used more and more in the codebase. So, this
version is just a basic skeleton one that *will* change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
These statistics were largely ununsed, and kept track of statistical information
on things like how many time we had done TLS or how many signatures we had
verified. This information is largely not useful, and would only be logged
after receiving a SIGUSR1 signal (but only if the logging severity level was
less than LOG_INFO).
* FIXES#19871.
* REMOVES note_crypto_pk_op(), dump_pk_op(), and pk_op_counts from
src/or/rephist.c.
* REMOVES every external call to these functions.
Relay operators (especially bridge operators) can use this to lower
or raise the number of consensuses that they're willing to hold for
diff generation purposes.
This enables a workaround for bug 22883.
Groundwork for more prop224 service and client code. This object contains
common data that both client and service uses.
Signed-off-by: David Goulet <dgoulet@torproject.org>
- Move some crypto structures so that they are visible by tests.
- Introduce a func to count number of hops in cpath which will be used
by the tests.
- Mark a function as mockable.
This commit paves the way for the e2e circuit unittests.
Add a stub for the prop224 equivalent of rend_client_note_connection_attempt_ended().
That function was needed for tests, since the legacy function would get
called when we attach streams and our client-side tests would crash with
assert failures on rend_data.
This also introduces hs_client.[ch] to the codebase.
This commit adds most of the work of #21859. It introduces hs_circuit.c
functions that can handle the setup of e2e circuits for prop224 hidden
services, and also for legacy hidden service clients. Entry points are:
prop224 circuits: hs_circuit_setup_e2e_rend_circ()
legacy client-side circuits: hs_circuit_setup_e2e_rend_circ_legacy_client()
This commit swaps the old rendclient code to use the new API.
I didn't try to accomodate the legacy service-side code in this API, since
that's too tangled up and it would mess up the new API considerably IMO (all
this service_pending_final_cpath_ref stuff is complicated and I didn't want to
change it).
Signed-off-by: David Goulet <dgoulet@torproject.org>
The legacy HS circuit code uses rend_data to match between circuits and
streams. We refactor some of that code so that it understands hs_ident
as well which is used for prop224.
circuit_init_cpath_crypto() is responsible for creating the cpath of legacy
SHA1/AES128 circuits currently. We want to use it for prop224 circuits, so we
refactor it to create circuits with SHA3-256 and AES256 as well.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We want to use the circuit_init_cpath_crypto() function to setup our
cpath, and that function accepts a key array as input. So let's make our
HS ntor key expansion function also return a key array as output,
instead of a struct.
Also, we actually don't need KH from the key expansion, so the key
expansion output can be one DIGEST256_LEN shorter. See here for more
info: https://trac.torproject.org/projects/tor/ticket/22052#comment:3
When the new path selection logic went into place, I accidentally
dropped the code that considered the _family_ of the exit node when
deciding if the guard was usable, and we didn't catch that during
code review.
This patch makes the guard_restriction_t code consider the exit
family as well, and adds some (hopefully redundant) checks for the
case where we lack a node_t for a guard but we have a bridge_info_t
for it.
Fixes bug 22753; bugfix on 0.3.0.1-alpha. Tracked as TROVE-2016-006
and CVE-2017-0377.
This makes our directory code check if a client is trying to fetch a
document that matches a digest from our latest consensus document.
See: https://bugs.torproject.org/22702
This patch ensures that the published_out output parameter is set to the
current consensus cache entry's "valid after" field.
See: https://bugs.torproject.org/22702
As of ac2f6b608a in 0.2.1.19-alpha,
Sebastian fixed bug 888 by marking descriptors as "impossible" by
digest if they got rejected during the
router_load_routers_from_string() phase. This fix stopped clients
and relays from downloading the same thing over and over.
But we never made the same change for descriptors rejected during
dirserv_add_{descriptor,extrainfo}. Instead, we tried to notice in
advance that we'd reject them with dirserv_would_reject().
This notice-in-advance check stopped working once we added
key-pinning and didn't make a corresponding key-pinning change to
dirserv_would_reject() [since a routerstatus_t doesn't include an
ed25519 key].
So as a fix, let's make the dirserv_add_*() functions mark digests
as undownloadable when they are rejected.
Fixes bug 22349; I am calling this a fix on 0.2.1.19-alpha, though
you could also argue for it being a fix on 0.2.7.2-alpha.
This mistake causes two possible bugs. I believe they are both
harmless IRL.
BUG 1: memory stomping
When we call the memset, we are overwriting two 0 bytes past the end
of packed_cell_t.body. But I think that's harmless in practice,
because the definition of packed_cell_t is:
// ...
typedef struct packed_cell_t {
TOR_SIMPLEQ_ENTRY(packed_cell_t) next;
char body[CELL_MAX_NETWORK_SIZE];
uint32_t inserted_time;
} packed_cell_t;
So we will overwrite either two bytes of inserted_time, or two bytes
of padding, depending on how the platform handles alignment.
If we're overwriting padding, that's safe.
If we are overwriting the inserted_time field, that's also safe: In
every case where we call cell_pack() from connection_or.c, we ignore
the inserted_time field. When we call cell_pack() from relay.c, we
don't set or use inserted_time until right after we have called
cell_pack(). SO I believe we're safe in that case too.
BUG 2: memory exposure
The original reason for this memset was to avoid the possibility of
accidentally leaking uninitialized ram to the network. Now
remember, if wide_circ_ids is false on a connection, we shouldn't
actually be sending more than 512 bytes of packed_cell_t.body, so
these two bytes can only leak to the network if there is another bug
somewhere else in the code that sends more data than is correct.
Fortunately, in relay.c, where we allocate packed_cell_t in
packed_cell_new() , we allocate it with tor_malloc_zero(), which
clears the RAM, right before we call cell_pack. So those
packed_cell_t.body bytes can't leak any information.
That leaves the two calls to cell_pack() in connection_or.c, which
use stack-alocated packed_cell_t instances.
In or_handshake_state_record_cell(), we pass the cell's contents to
crypto_digest_add_bytes(). When we do so, we get the number of
bytes to pass using the same setting of wide_circ_ids as we passed
to cell_pack(). So I believe that's safe.
In connection_or_write_cell_to_buf(), we also use the same setting
of wide_circ_ids in both calls. So I believe that's safe too.
I introduced this bug with 1c0e87f6d8
back in 0.2.4.11-alpha; it is bug 22737 and CID 1401591
This introduces node_supports_v3_hsdir() and node_supports_ed25519_hs_intro()
that checks the routerstatus_t of a node and if not present, checks the
routerinfo_t.
This is groundwork for proposal 224 service implementation in #20657.
Signed-off-by: David Goulet <dgoulet@torproject.org>
It is possible that at some point in time a client will encounter unknown or
new fields for an introduction point in a descriptor so let them ignore it for
forward compatibility.
Signed-off-by: David Goulet <dgoulet@torproject.org>
If `validate_only` is true, then just validate the configuration without warning
about it. This way, we only emit warnings when the listener is actually opened.
(Otherwise, every time we parse the config we will might re-warn and we would
need to keep state; whereas the listeners are only opened once.)
* FIXES#4019.
-authdir_mode_handles_descs(options, ROUTER_PURPOSE_BRIDGE) to authdir_mode_bridge(options).
- authdir_mode_handles_descs(options, ROUTER_PURPOSE_GENERAL) to authdir_mode_v3(options).
This prevents us from calling
allowed_anonymous_connection_compression_method() on the unused
guessed method (if any), and rejecting something that was already
safe to use.
Rationale: When use a guessed compression method, we already gave a
PROTOCOL_WARN when our guess differed from the declared method,
AND we gave a PROTOCOL_WARN when the declared method failed. It is
not a protocol problem that the guessed method failed too; it's just
a recovery attempt that failed.
A cached_dir_t object (for now) is always compressed with
DEFLATE_METHOD, but in handle_get_status_vote() to we were using the
general compression-negotiation code decide what compression to
claim we were using.
This was one of the reasons behind 22502.
Fixes bug 22669; bugfix on 0.3.1.1-alpha
Move the HTTPProxy option to the deprecated list so for now it will only warn
users but feature is still in the code which will be removed in a future
stable version.
Fixes#20575
Signed-off-by: David Goulet <dgoulet@torproject.org>
On an hidden service rendezvous circuit, a BEGIN_DIR could be sent
(maliciously) which would trigger a tor_assert() because
connection_edge_process_relay_cell() thought that the circuit is an
or_circuit_t but is an origin circuit in reality.
Fixes#22494
Reported-by: Roger Dingledine <arma@torproject.org>
Signed-off-by: David Goulet <dgoulet@torproject.org>
This fixes an assertion failure in relay_send_end_cell_from_edge_() when an
origin circuit and a cpath_layer = NULL were passed.
A service rendezvous circuit could do such a thing when a malformed BEGIN cell
is received but shouldn't in the first place because the service needs to send
an END cell on the circuit for which it can not do without a cpath_layer.
Fixes#22493
Reported-by: Roger Dingledine <arma@torproject.org>
Signed-off-by: David Goulet <dgoulet@torproject.org>
Apparently, the unit tests relied on being able to make ed->x509
link certs even when they hadn't set any server flags in the
options. So instead of making "client" mean "never generate an
ed->x509 cert", we'll have it mean "it's okay not to generate an
ed->x509 cert".
(Going with a minimal fix here, since this is supposed to be a
stable version.)
It's okay to call add_ed25519_cert with a NULL argument: so,
document that. Also, add a tor_assert_nonfatal() to catch any case
where we have failed to set own_link_cert when conn_in_server_mode.
Whenever we rotate our TLS context, we change our Ed25519
Signing->Link certificate. But if we've already started a TLS
connection, then we've already sent the old X509 link certificate,
so the new Ed25519 Signing->Link certificate won't match it.
To fix this, we now store a copy of the Signing->Link certificate
when we initialize the handshake state, and send that certificate
as part of our CERTS cell.
Fixes one case of bug22460; bugfix on 0.3.0.1-alpha.
Previously we could sometimes change our signing key, but not
regenerate the certificates (signing->link and signing->auth) that
were signed with it. Also, we would regularly replace our TLS x.509
link certificate (by rotating our TLS context) but not replace our
signing->link ed25519 certificate. In both cases, the resulting
inconsistency would make other relays reject our link handshakes.
Fixes two cases of bug 22460; bugfix on 0.3.0.1-alpha.
The encrypted_data_length_is_valid() function wasn't validating correctly the
length of the encrypted data of a v3 descriptor. The side effect of this is
that an HSDir was rejecting the descriptor and ultimately not storing it.
Fixes#22447
Signed-off-by: David Goulet <dgoulet@torproject.org>
A fair number of our mock_impl declarations were messed up so that
even our special AM_ETAGSFLAGS couldn't find them.
This should be a whitespace-only patch.
When directory authorities reject a router descriptor due to keypinning,
free the router descriptor rather than leaking the memory.
Fixes bug 22370; bugfix on 0.2.7.2-alpha.
If we add the element itself, we will later free it when we free the
descriptor, and the next time we go to look at MyFamily, things will
go badly.
Fixes the rest of bug 22368; bugfix on 0.3.1.1-alpha.
If we free them here, we will still attempt to access the freed memory
later on, and also we will double-free when we are freeing the config.
Fixes part of bug 22368.
This patch lifts the return value, rv, variable to the beginning of the
function, adds a 'done' label for clean-up and function exit and makes
the rest of the function use the rv value + goto done; instead of
cleaning up in multiple places.
See: https://bugs.torproject.org/22305
We used to not set the guard state in launch_direct_bridge_descriptor_fetch().
So when a bridge descriptor fetch failed, the guard subsystem would never
learn about the fail (and hence the guard's reachability state would not
be updated).
We used to not set the guard state in launch_direct_bridge_descriptor_fetch().
So when a bridge descriptor fetch failed, the guard subsystem would never
learn about the fail (and hence the guard's reachability state would not
be updated).
Directory authorities now reject relays running versions
0.2.9.1-alpha through 0.2.9.4-alpha, because those relays
suffer from bug 20499 and don't keep their consensus cache
up-to-date.
Resolves ticket 20509.
This gives an indication in the log that Tor was built with Rust
support, as well as laying some groundwork for further string-returning
APIs to be converted to Rust
config_get_lines is now split into two functions:
- config_get_lines which is the same as before we had %include
- config_get_lines_include which actually processes %include
Before we've set our options, we can neither call get_options() nor
networkstatus_get_latest_consensus().
Fixes bug 22252; bugfix on 4d9d2553ba
in 0.2.9.3-alpha.
Replace the 177 fallbacks originally introduced in Tor 0.2.9.8 in
December 2016 (of which ~126 were still functional), with a list of
151 fallbacks (32 new, 119 existing, 58 removed) generated in May 2017.
Resolves ticket 21564.
This patch makes us use FALLBACK_COMPRESS_METHOD to try to fetch an
object from the consensus diff manager in case no mutually supported
result was found. This object, if found, is then decompressed using the
spooling system to the client.
See: https://bugs.torproject.org/21667
This patch removes the calls to spooled_resource_new() when trying to
download the consensus. All calls should now be going through the
consdiff manager.
See: https://bugs.torproject.org/21667
This patch ensures that we use the current consensus in the case where
no consensus diff was found or a consensus diff wasn't requested.
See: https://bugs.torproject.org/21667
These still won't do anything till I get the values to be filled in.
Also, I changed the API a little (with corresponding changes in
directory.c) to match things that it's easier to store.
This patch changes handle_get_current_consensus() to make it read the
current consensus document from the consensus caching subsystem.
See: https://bugs.torproject.org/21667
Failure to do this caused an assertion failure with #22246 . This
assertion failure can be triggered remotely, so we're tracking it as
medium-severity TROVE-2017-002.
Closes bug 22245; bugfix on 0.0.9rc1, when bandwidth accounting was
first introduced.
Found by Andrey Karpov and reported at https://www.viva64.com/en/b/0507/
This patch refactors connection_dir_client_reached_eof() to use
compression_method_get_human_name() to set description1 and
description2 variables.
See: https://bugs.torproject.org/21667
One (HeapEnableTerminationOnCorruption) is on-by-default since win8;
the other (PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION) supposedly only
affects ATL, which (we think) we don't use. Still, these are good
hygiene. Closes ticket 21953.
A descriptor only contains the curve25519 public key in the enc-key field so
the private key should not be in that data structure. The service data
structures will have access to the full keypair (#20657).
Furthermore, ticket #21871 has highlighted an issue in the proposal 224 about
the encryption key and legacy key being mutually exclusive. This is very wrong
and this commit fixes the code to follow the change to the proposal of that
ticket.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We do this by treating the presence of .z as meaning ZLIB_METHOD,
even if Accept-Encoding does not include deflate.
This fixes bug 22206; bug not in any released tor.
Nothing was setting or inspecting these fields, and they were marked
as OBSOLETE() in config.c -- but somehow we still had them in the
or_options_t structure. Ouch.
There was a bug that got exposed with the removal of ORListenAddress. Within
server_mode(), we now only check ORPort_set which is set in parse_ports().
However, options_validate() is using server_mode() at the start to check if we
need to look at the uname but then the ORPort_set is unset at that point
because the port parsing was done just after. This commit fixes that.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
Deprecated in 0.2.9.2-alpha, this commits changes it as OBSOLETE() and cleans
up the code associated with it.
Partially fixes#22060
Signed-off-by: David Goulet <dgoulet@torproject.org>
This change prevents a no-longer-supported behavior where we change
options that would later be written back to torrc with a SAVECONF.
Also, use the "Pointer to final pointer" trick to build the
normalized list, to avoid special-casing the first element.
Checking all of these parameter lists for every single connection every second
seems like it could be an expensive waste.
Updating globally cached versions when there is a new consensus will still
allow us to apply consensus parameter updates to all existing connections
immediately.
Accomplished via the following:
1. Use NETINFO cells to determine if both peers will agree on canonical
status. Prefer connections where they agree to those where they do not.
2. Alter channel_is_better() to prefer older orconns in the case of multiple
canonical connections, and use the orconn with more circuits on it in case
of age ties.
Also perform some hourly accounting on how many of these types of connections
there are and log it at info or notice level.
This unifies CircuitIdleTimeout and PredictedCircsRelevanceTime into a single
option, and randomizes it.
It also gives us control over the default value as well as relay-to-relay
connection lifespan through the consensus.
Conflicts:
src/or/circuituse.c
src/or/config.c
src/or/main.c
src/test/testing_common.c
This defense will cause Cisco, Juniper, Fortinet, and other routers operating
in the default configuration to collapse netflow records that would normally
be split due to the 15 second flow idle timeout.
Collapsing these records should greatly reduce the utility of default netflow
data for correlation attacks, since all client-side records should become 30
minute chunks of total bytes sent/received, rather than creating multiple
separate records for every webpage load/ssh command interaction/XMPP chat/whatever
else happens to be inactive for more than 15 seconds.
The defense adds consensus parameters to govern the range of timeout values
for sending padding packets, as well as for keeping connections open.
The defense only sends padding when connections are otherwise inactive, and it
does not pad connections used solely for directory traffic at all. By default
it also doesn't pad inter-relay connections.
Statistics on the total padding in the last 24 hours are exported to the
extra-info descriptors.
Right now it just sets an if-modified-since header, but it's about
to get even bigger.
This patch avoids changing indentation; the next patch will be
whitespace fixes.
We need to index diffs by the digest-as-signed of their source
consensus, so that we can find them even from consensuses whose
signatures are encoded differently.
In this patch I add support for "delete through end of file" in our
ed diff handler, and generate our diffs so that they remove
everything after in the consensus after the signatures begin.
This was introduced 90562fc23a adding a code
path where we pass a NULL pointer for the HSDir fingerprint to the control
event subsystem. The HS desc failed function wasn't handling properly that
pointer for a NULL value.
Two unit tests are also added in this commit to make sure we handle properly
the case of a NULL hsdir fingerprint and a NULL content as well.
Fixes#22138
Signed-off-by: David Goulet <dgoulet@torproject.org>
config_parse_interval() and config_parse_msec_interval() were checking
whether the variable "ok" (a pointer to an int) was null, rather than
derefencing it. Both functions are static, and all existing callers
pass a valid pointer to those static functions. The callers do check
the variables (also confusingly named "ok") whose addresses they pass
as the "ok" arguments, so even if the pointer check were corrected to
be a dereference, it would be redundant.
Fixes#22103.
This was a >630-line function, which doesn't make anybody happy. It
was also mostly composed of a bunch of if-statements that handled
different directory responses differently depending on the original
purpose of the directory connection. The logical refactoring here
is to move the body of each switch statement into a separate handler
function, and to invoke those functions from a separate switch
statement.
This commit leaves whitespace mostly untouched, for ease of review.
I'll reindent in the next commit.
Inform the control port with an HS_DESC failed event when the client is unable
to pick an HSDir. It's followed by an empty HS_DESC_CONTENT event. In order to
achieve that, some control port code had to be modified to accept a NULL HSDir
identity digest.
This commit also adds a trigger of a failed event when we are unable to
base64-decode the descriptor cookie.
Fixes#22042
Signed-off-by: David Goulet <dgoulet@torproject.org>
Introduce a way to optionally enable Rust integration for our builds. No
actual Rust code is added yet and specifying the flag has no effect
other than failing the build if rustc and cargo are unavailable.
This commit adds the src/trace directory containing the basics for our tracing
subsystem. It is not used in the code base. The "src/trace/debug.h" file
contains an example on how we can map our tor trace events to log_debug().
The tracing subsystem can only be enabled by tracing framework at compile
time. This commit introduces the "--enable-tracing-debug" option that will
make all "tor_trace()" function be maped to "log_debug()".
Closes#13802
Signed-off-by: David Goulet <dgoulet@torproject.org>
That log statement can be triggered if somebody on the Internet behaves badly
which is possible with buggy implementation for instance.
Fixes#21293
Signed-off-by: David Goulet <dgoulet@torproject.org>
This patch ensures that Tor checks if a given compression method is
supported before printing the version string when calling `tor
--library-versions`.
Additionally, we use the `tor_compress_supports_method()` to check if a
given version is supported for Tor's start-up version string, but here
we print "N/A" if a given compression method is unavailable.
See: https://bugs.torproject.org/21662
This patch adds the `tor_compress_get_total_allocation()` which returns
an approximate number of bytes currently in use by all the different
compression backends.
See: https://bugs.torproject.org/21662
This patch adds support for enabling support for Zstandard to our configure
script. By default, the --enable-zstd option is set to "auto" which means if
libzstd is available we'll build Tor with Zstandard support.
See: https://bugs.torproject.org/21662
This patch adds support for enabling support for LZMA to our configure
script. By default, the --enable-lzma option is set to "auto" which
means if liblzma is available we'll build Tor with LZMA support.
See: https://bugs.torproject.org/21662
This patch refactors the `torgzip` module to allow us to extend a common
compression API to support multiple compression backends.
Additionally we move the gzip/zlib code into its own module under the
name `compress_zlib`.
See https://bugs.torproject.org/21664
Remove duplicate code that validates a service object which is now in
rend_validate_service().
Add some comments on why we nullify a service in the code path of
rend_config_services().
Signed-off-by: David Goulet <dgoulet@torproject.org>
This new function validates a service object and is used everytime a service
is successfully loaded from the configuration file.
It is currently copying the validation that rend_add_service() also does which
means both functions validate. It will be decoupled in the next commit.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In several places in the old code, we had problems that only an
in-memory index of diff status could solve, including:
* Remembering which diffs were in-progress, so that we didn't
re-launch them.
* Remembering which diffs had failed, so that we didn't try to
recompute them over and over.
* Having a fast way to look up the diff from a given consensus to
the latest consensus of a given flavor.
This patch adds a hashtable mapping from (flavor, source diff), to
solve the problem. It maps to a cache entry handle, rather than to
a cache entry directly, so that it doesn't affect the reference
counts of the cache entries, and so that we don't otherwise need to
worry about lifetime management.
This conscache flag tells conscache that it should munmap the
document as soon as reasonably possible, since its usage pattern is
expected to not have a lot of time-locality.
Initial tests. These just try adding a few consensuses, looking
them up, and making sure that consensus diffs are generated in a
more or less reasonable-looking way. It's enough for 87% coverage,
but it leaves out a lot of functionality.
This module's job is to remember old consensus documents, to
calculate their diffs on demand, and to .
There are some incomplete points in this code; I've marked them with
"XXXX". I intend to fix them in separate commits, since I believe
doing it in separate commits will make the branch easier to review.
The GETINFO extra-info/digest/<digest> broke in commit 568dc27a19 that
refactored the base16_decode() API to return the decoded length.
Unfortunately, that if() condition should have checked for the correct length
instead of an error which broke the command in tor-0.2.9.1-alpha.
Fixes#22034
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit mainly moves the responsibility for directory request
construction one level higher. It also allows a directory request
to contain a pointer to a routerstatus, which will get turned into
the correct contact information at the last minute.
We do dump HS stats now at log info everytime the intro circuit creation retry
period limit has been reached. However, the log was upgraded to warning if we
actually were over the elapsed time (plus an extra slop).
It is actually something that will happen in tor in normal case. For instance,
if the network goes down for 10 minutes then back up again making
have_completed_a_circuit() return false which results in never updating that
retry period marker for a service.
Fixes#22032
Signed-off-by: David Goulet <dgoulet@torproject.org>
This patch refactors our streaming compression code to allow us to
extend it with non-zlib/non-gzip based compression schemas.
See https://bugs.torproject.org/21663
To allow us to use the API name `tor_compress` and `tor_uncompress` as
the main entry-point for all compression/uncompression and not just gzip
and zlib.
See https://bugs.torproject.org/21663
The consdiff generation logic would skip over lines added at the start of the
second file, and generate a diff that it would the immediately refuse because
it couldn't be used to reproduce the second file from the first. Fixes#21996.
The reason for making the temporary list public is to keep it encapsulated in
the rendservice subsystem so the prop224 code does not have direct access to
it and can only affect it through the rendservice pruning function.
It also has been modified to not take list as arguments but rather use the
global lists (main and temporary ones) because prop224 code will call it to
actually prune the rendservice's lists. The function does the needed rotation
of pointers between those lists and then prune if needed.
In order to make the unit test work and not completely horrible, there is a
"impl_" version of the function that doesn't free memory, it simply moves
pointers around. It is directly used in the unit test and two setter functions
for those lists' pointer have been added only for unit test.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Now we have separate getters and setters for service-side and relay-side. I
took this approach over adding arguments to the already existing methods to
have more explicit type-checking, and also because some functions would grow
too large and dirty.
This commit also fixes every callsite to use the new function names which
modifies the legacy HS (v2) and the prop224 (v3) code.
Signed-off-by: David Goulet <dgoulet@torproject.org>
One of the goals of this change is to have trunnel API/ABI being more explicit
so we namespace them with "trn_*". Furthermore, we can now create
hs_cells.[ch] without having to confuse it with trunnel which used to be
"hs_cell_*" before that change.
Here are the perl line that were used for this rename:
perl -i -pe 's/cell_extension/trn_cell_extension/g;' src/*/*.[ch]
perl -i -pe 's/cell_extension/trn_cell_extension/g;' src/trunnel/hs/*.trunnel
perl -i -pe 's/hs_cell_/trn_cell_/g;' src/*/*.[ch]
perl -i -pe 's/hs_cell_/trn_cell_/g;' src/trunnel/hs/*.trunnel
And then "./scripts/codegen/run_trunnel.sh" with trunnel commit id
613fb1b98e58504e2b84ef56b1602b6380629043.
Fixes#21919
Signed-off-by: David Goulet <dgoulet@torproject.org>
Pinning EntryNodes along with hidden services can be possibly harmful (for
instance #14917 and #21155) so at the very least warn the operator if this is
the case.
Fixes#21155
Signed-off-by: David Goulet <dgoulet@torproject.org>
Now that base64_decode() checks the destination buffer length against
the actual number of bytes as they're produced, shared_random.c no
longer needs the "SR_COMMIT_LEN+2" workaround.
Note down in the routerstatus_t of a node if the router supports the HSIntro=4
version for the ed25519 authentication key and HSDir=2 version for the v3
descriptor supports.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Some of those defines will be used by the v3 HS protocol so move them to a
common header out of rendservice.c. This is also ground work for prop224
service implementation.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Another building blocks for prop224 service work. This also makes the function
takes specific argument instead of the or_option_t object.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When a client tried to connect to an invalid port of an hidden service, a
warning was printed:
[warn] connection_edge_process_relay_cell (at origin) failed.
This is because the connection subsystem wants to close the circuit because
the port can't be found and then returns a negative reason to achieve that.
However, that specific situation triggered a warning. This commit prevents it
for the specific case of an invalid hidden service port.
Fixes#16706
Signed-off-by: David Goulet <dgoulet@torproject.org>
In order to avoid src/or/hs_service.o to contain no symbols and thus making
clang throw a warning, the functions are now exposed not just to unit tests.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add a new helper function route_len_for_purpose(), which explicitly
lists all of the known circuit purposes for a circuit with a chosen
exit node (unlike previously, where the default route length for a
chosen exit was DEFAULT_ROUTE_LEN + 1 except for two purposes). Add a
non-fatal assertion for unhandled purposes that conservatively returns
DEFAULT_ROUTE_LEN + 1.
Add copious comments documenting which circuits need an extra hop and
why.
Thanks to nickm and dgoulet for providing background information.
The old implementation had duplicated code in a bunch of places, and
it interspersed spool-management with resource management. The new
implementation should make it easier to add new resource types and
maintain the spooling code.
Closing ticket 21651.
When calculating max sampled size, Tor would only count the number of
bridges in torrc, without considering that our state file might already
have sampled bridges in it. This caused problems when people swap
bridges, since the following error would trigger:
[warn] Not expanding the guard sample any further; just hit the
maximum sample threshold of 1
This patch changes the way we decide when to check for whether it's time
to rotate and/or expiry our onion keys. Due to proposal #274 we can now
have the keys rotate at different frequencies than before and we thus
do the check once an hour when our Tor daemon is running in server mode.
This should allow us to quickly notice if the network consensus
parameter have changed while we are running instead of having to wait
until the current parameters timeout value have passed.
See: See: https://bugs.torproject.org/21641
This patch adds a new timer that is executed when it is time to expire
our current set of old onion keys. Because of proposal #274 this can no
longer be assumed to be at the same time we rotate our onion keys since
they will be updated less frequently.
See: https://bugs.torproject.org/21641
This patch adds an API to get the current grace period, in days, defined
as the consensus parameter "onion-key-grace-period-days".
As per proposal #274 the values for "onion-key-grace-period-days" is a
default value of 7 days, a minimum value of 1 day, and a maximum value
defined by other consensus parameter "onion-key-rotation-days" also
defined in days.
See: https://bugs.torproject.org/21641
This patch turns `MIN_ONION_KEY_LIFETIME` into a new function
`get_onion_key_lifetime()` which gets its value from a network consensus
parameter named "onion-key-rotation-days". This allows us to tune the
value at a later point in time with no code modifications.
We also bump the default onion key lifetime from 7 to 28 days as per
proposal #274.
See: https://bugs.torproject.org/21641
There was a frequent block of code that did "find the next router
line, see if we've hit the end of the list, get the ID hash from the
line, and enforce well-ordering." Per Ahf's review, I'm extracting
it to its own function.
Previously, we operated on smartlists of NUL-terminated strings,
which required us to copy both inputs to produce the NUL-terminated
strings. Then we copied parts of _those_ inputs to produce an
output smartlist of NUL-terminated strings. And finally, we
concatenated everything into a final resulting string.
This implementation, instead, uses a pointer-and-extent pattern to
represent each line as a pointer into the original inputs and a
length. These line objects are then added by reference into the
output. No actual bytes are copied from the original strings until
we finally concatenate the final result together.
Bookkeeping structures and newly allocated strings (like ed
commands) are allocated inside a memarea, to avoid needless mallocs
or complicated should-I-free-this-or-not bookkeeping.
In my measurements, this improves CPU performance by something like
18%. The memory savings should be much, much higher.
Also, add very strict split/join functions, and totally forbid
nonempty files that end with somethig besides a newline. This
change is necessary to ensure that diff/apply are actually reliable
inverse operations.
The 2-line diff changs is needed to make the unit tests actually
test the cases that they thought they were testing.
The bogus free was found while testing those cases
(This commit was extracted by nickm based on the final outcome of
the project, taking only the changes in the files touched by this
commit from the consdiff_rebased branch. The directory-system
changes are going to get worked on separately.)
Also, relaxed the checks of encrypted_data_length_is_valid() since now
only one encrypted section has padding requirements and we don't
actually care to check that all the padding is there.
Consider starting code review from function encode_superencrypted_data().
- Refactor our HS desc crypto funcs to be able to differentiate between
the superencrypted layer and the encrypted layer so that different
crypto constants and padding is used in each layer.
- Introduce some string constants.
- Add some comments.
As part of the work for proposal #274 we are going to remove the need
for MIN_ONION_KEY_LIFETIME and turn it into a dynamic value defined by a
consensus parameter.
See: https://bugs.torproject.org/21641
The bridges+ipv6-min integration test has a client with bridges:
Bridge 127.0.0.1:5003
Bridge [::1]:5003
which got stuck in guard_selection_have_enough_dir_info_to_build_circuits()
because it couldn't find the descriptor of both bridges.
Specifically, the guard_has_descriptor() function could not find the
node_t of the [::1] bridge, because the [::1] bridge had no identity
digest assigned to it.
After further examination, it seems that during fetching the descriptor
for our bridges, we used the CERTS cell to fill the identity digest of
127.0.0.1:5003 properly. However, when we received a CERTS cell from
[::1]:5003 we actually ignored its identity digest because the
learned_router_identity() function was using
get_configured_bridge_by_addr_port_digest() which was returning the
127.0.0.1 bridge instead of the [::1] bridge (because it prioritizes
digest matching over addrport matching).
The fix replaces get_configured_bridge_by_addr_port_digest() with the
recent get_configured_bridge_by_exact_addr_port_digest() function. It
also relaxes the constraints of the
get_configured_bridge_by_exact_addr_port_digest() function by making it
return bridges whose identity digest is not yet known.
By using the _exact_() function, learned_router_identity() actually
fills in the identity digest of the [::1] bridge, which then allows
guard_has_descriptor() to find the right node_t and verify that the
descriptor is there.
FWIW, in the bridges+ipv6-min test both 127.0.0.1 and [::1] bridges
correspond to the same node_t, which I guess makes sense given that it's
actually the same underlying bridge.
This patch removes the `tor_fgets()` wrapper around `fgets(3)` since it
is no longer needed. The function was created due to inconsistency
between the returned values of `fgets(3)` on different versions of Unix
when using `fgets(3)` on non-blocking file descriptors, but with the
recent changes in bug #21654 we switch from unbuffered to direct I/O on
non-blocking file descriptors in our utility module.
We continue to use `fgets(3)` directly in the geoip and dirserv module
since this usage is considered safe.
This patch also removes the test-case that was created to detect
differences in the implementation of `fgets(3)` as well as the changes
file since these changes was not included in any releases yet.
See: https://bugs.torproject.org/21654
Make hidden services with 8 to 10 introduction points check for failed
circuits immediately after startup. Previously, they would wait for 5
minutes before performing their first checks.
Fixes bug 21594; bugfix on commit 190aac0eab in Tor 0.2.3.9-alpha.
Reported by alecmuffett.
This change is the only one necessary to allow future versions of
the microdescriptor consensus to replace every 'published' date with
e.g. 2038-01-01 00:00:00; this will save 50-75% in compressed
microdescriptor diff size, which is quite significant.
This commit is a minimal change for 0.2.9; future series will
reduce the use of the 'published' date even more.
Implements part of ticket 21642; implements part of proposal 275.
Previously, they would stop checking when they exceeded their intro point
creation limit.
Fixes bug 21596; bugfix on commit d67bf8b2f2 in Tor 0.2.7.2-alpha.
Reported by alecmuffett.
Previously, they would stop checking when they exceeded their intro point
creation limit.
Fixes bug 21596; bugfix on commit d67bf8b2f2 in Tor 0.2.7.2-alpha.
Reported by alecmuffett.
In that chutney test, the bridge client is configured to connect to
the same bridge at 127.0.0.1:5003 _and_ at [::1]:5003, with no
change in transports.
That meant, I think, that the descriptor is only assigned to the
first bridge when it arrives, and never the second.
- Make sure we check at least two guards for descriptor before making
circuits. We typically use the first primary guard for circuits, but
it can also happen that we use the second primary guard (e.g. if we
pick our first primary guard as an exit), so we should make sure we
have descriptors for both of them.
- Remove BUG() from the guard_has_descriptor() check since we now know
that this can happen in rare but legitimate situations as well, and we
should just move to the next guard in that case.
Previously I'd made a bad assumption in the implementation of
prop271 in 0.3.0.1-alpha: I'd assumed that there couldn't be two
guards with the same identity. That's true for non-bridges, but in
the bridge case, we allow two bridges to have the same ID if they
have different addr:port combinations -- in order to have the same
bridge ID running multiple PTs.
Fortunately, this assumption wasn't deeply ingrained: we stop
enforcing the "one guard per ID" rule in the bridge case, and
instead enforce "one guard per <id,addr,port>".
We also needed to tweak our implementation of
get_bridge_info_for_guard, since it made the same incorrect
assumption.
Fixes bug 21027; bugfix on 0.3.0.1-alpha.
This feature makes it possible to turn off memory sentinels (like
those used for safety in buffers.c and memarea.c) when fuzzing, so
that we can catch bugs that they would otherwise prevent.
Teor thinks that this connection_dirserv_add_dir_bytes_to_outbuf()
might be the problem, if the "remaining" calculation underflows. So
I'm adding a couple of checks there, and improving the casts.
When encoding a legacy ESTABLISH_INTRO cell, we were using the sizeof() on a
pointer instead of using the real size of the destination buffer leading to an
overflow passing an enormous value to the signing digest function.
Fortunately, that value was only used to make sure the destination buffer
length was big enough for the key size and in this case it always was because
of the overflow.
Fixes#21553
Signed-off-by: David Goulet <dgoulet@torproject.org>
strto* and _atoi64 accept +, -, and various whitespace before numeric
characters. And permitted whitespace is different between POSIX and Windows.
Fixes bug 21507 and part of 21508; bugfix on 0.0.8pre1.
This patch makes us store the number of sent and received RELAY_DATA
cells used for directory connections. We log the numbers after we have
received an EOF in connection_dir_client_reached_eof() from the
directory server.
Instead of returning 404 error code, this led to a NULL pointer being used and
thus a crash of tor.
Fixes#21471
Signed-off-by: David Goulet <dgoulet@torproject.org>
Fixes bug 20894; bugfix on 0.2.0.16-alpha.
We already applied a workaround for this as 20834, so no need to
freak out (unless you didn't apply 20384 yet).
This should be "impossible" without making a SHA1 collision, but
let's not keep the assumption that SHA1 collisions are super-hard.
This prevents another case related to 21278. There should be no
behavioral change unless -ftrapv is on.
I think this one probably can't underflow, since the input ranges
are small. But let's not tempt fate.
This patch also replaces the "cmp" functions here with just "eq"
functions, since nothing actually checked for anything besides 0 and
nonzero.
Related to 21278.
