Also, when we log about a failure from base32_decode(), we now
say that the length is wrong or that the characters were invalid:
previously we would just say that there were invalid characters.
Follow-up on 28913 work.
Hope is this will make it easier to test on the live tor network.
Does not need to be merged if we don't want to, but will come in handy
for researchers.
Co-authored-by: George Kadianakis <desnacked@riseup.net>
Redefine the set of bootstrap phases to allow display of finer-grained
progress in the early connection stages of connecting to a relay.
This includes adding intermediate phases for proxy and PT connections.
Also add a separate new phase to indicate obtaining enough directory
info to build circuits so we can report that independently of actually
initiating an ORCONN to build the first application circuit.
Previously, we would claim to be connecting to a relay when we had
merely finished obtaining directory info.
Part of ticket 27167.
Replace a few invocations of control_event_bootstrap() with calls from
the bootstrap tracker subsystem. This mostly leaves behavior
unchanged. The actual behavior changes come in the next commit.
Part of ticket 27167.
Add a tracker for bootstrap progress, tracking events related to
origin circuit and ORCONN states. This uses the ocirc_event and
orconn_event publish-subscribe subsystems.
Part of ticket 27167.
Add a publish-subscribe subsystem to publish messages about changes to
origin circuits.
Functions in circuitbuild.c and circuitlist.c publish messages to this
subsystem.
Move circuit event constants out of control.h so that subscribers
don't have to include all of control.h to take actions based on
messages they receive.
Part of ticket 27167.
Add a publish-subscribe subsystem to publish messages about changes to
OR connections.
connection_or_change_state() in connection_or.c and
control_event_or_conn_event() in control.c publish messages to this
subsystem via helper functions.
Move state constants from connection_or.h to orconn_state.h so that
subscribers don't have to include all of connection_or.h to take
actions based on changes in OR connection state. Move event constants
from control.h for similar reasons.
Part of ticket 27167.
This patch adds support for the new STATUS message that PT's can emit
from their standard out. The STATUS message uses the `config_line_t` K/V
format that was recently added in Tor.
See: https://bugs.torproject.org/28846
This patch changes the LOG pluggable transport message to use the recent
K/V parser that landed in Tor. This allows PT's to specify the log
severity level as well as the message. A mapping between the PT log
severity levels and Tor's log serverity level is provided.
See: https://bugs.torproject.org/28846
Previously we had decoded the asn.1 to get a public key, and then
discarded the asn.1 so that we had to re-encode the key to store it
in the onion_pkey field of a microdesc_t or routerinfo_t.
Now we can just do a tor_memdup() instead, which should be loads
faster.
Previously, we would decode the PEM wrapper for keys twice: once in
get_next_token, and once later in PEM decode. Now we just do all of
the wrapper and base64 stuff in get_next_token, and store the
base64-decoded part in the token object for keys and non-keys alike.
This change should speed up parsing slightly by letting us skip a
bunch of stuff in crypto_pk_read_*from_string(), including the tag
detection parts of pem_decode(), and an extra key allocation and
deallocation pair.
Retaining the base64-decoded part in the token object will allow us
to speed up our microdesc parsing, since it is the asn1 portion that
we actually want to retain.
This patch changes our EVENT_TRANSPORT_LOG event to be EVENT_PT_LOG. The
new message includes the path to the PT executable instead of the
transport name, since one PT binary can include multiple transport they
sometimes might need to log messages that are not specific to a given
transport.
See: https://bugs.torproject.org/28179
This patch changes our process_t's exit_callback to return a boolean
value. If the returned value is true, the process subsystem will call
process_free() on the given process_t.
See: https://bugs.torproject.org/28179
This patch moves the remaining code from subprocess.{h,c} to more
appropriate places in the process.c and process_win32.c module.
We also delete the now empty subprocess module files.
See: https://bugs.torproject.org/28179
This patch adds support for the "LOG" protocol message from a pluggable
transport. This allows pluggable transport developers to relay log
messages from their binary to Tor, which will both emit them as log
messages from the Tor process itself, but also pass them on via the
control port.
See: https://bugs.torproject.org/28180
See: https://bugs.torproject.org/28181
See: https://bugs.torproject.org/28182
This patch makes the managed proxy subsystem use the process_t data
structure such that we can get events from the PT process while Tor is
running and not just when the PT process is being configured.
See: https://bugs.torproject.org/28179
The strcmp_len() function was somewhat misconceived, since we're
only using it to test whether a length+extent string is equal to a
NUL-terminated string or not. By simplifying it and making it
inlined, we should be able to make it a little faster.
(It *does* show up in profiles.)
Closes ticket 28856.
I believe we originally added this for "just in case" safety, but it
isn't actually needed -- we never copy uninitialized stack here.
What's more, this one memset is showing up on our startup profiles,
so we ought to remove it.
Closes ticket 28852.
Add the bootstrap tag name to the log messages, so people
troubleshooting connection problems can look up a symbol instead of a
number. Closes ticket 28731.
When retrying all SOCKS connection because new directory information just
arrived, do not BUG() if a connection in state AP_CONN_STATE_RENDDESC_WAIT is
found to have a usable descriptor.
There is a rare case when this can happen as detailed in #28669 so the right
thing to do is put that connection back in circuit wait state so the
descriptor can be retried.
Fixes#28669
Signed-off-by: David Goulet <dgoulet@torproject.org>
This helper function marks an entry connection as pending for a circuit and
changes its state to AP_CONN_STATE_CIRCUIT_WAIT. The timestamps are set to
now() so it can be considered as new.
No behaviour change, this helper function will be used in next commit.
Part of #28669
Signed-off-by: David Goulet <dgoulet@torproject.org>
Use the helper function connection_ap_mark_as_waiting_for_renddesc()
introduced in previous commit everywhere in the code where an AP connection
state is transitionned to AP_CONN_STATE_RENDDESC_WAIT.
Part of #28669
Signed-off-by: David Goulet <dgoulet@torproject.org>
Specifically, if the consensus is older than the (estimted or
measured) release date for this version of tor, we assume that the
required versions may have changed in between that consensus and
this release.
Implements ticket 27735 and proposal 297.
This patch has routers use the same canonicalization logic as
authorities when encoding their family lists. Additionally, they
now warn if any router in their list is given by nickname, since
that's error-prone.
This patch also adds some long-overdue tests for family formatting.
Prop298 says that family entries should be formatted with
$hexids in uppercase, nicknames in lower case, $hexid~names
truncated, and everything sorted lexically. These changes implement
that ordering for nodefamily.c.
We don't _strictly speaking_ need to nodefamily.c formatting use
this for prop298 microdesc generation, but it seems silly to have
two separate canonicalization algorithms.
This representation is meant to save memory in microdescriptors --
we can't use it in routerinfo_t yet, since those families need to be
encoded losslessly for directory voting to work.
This representation saves memory in three ways:
1. It uses only one allocation per family. (The old way used a
smartlist (2 allocs) plus one strdup per entry.)
2. It stores identity digests in binary, not hex.
3. It keeps families in a canonical format, memoizes, and
reference-counts them.
Part of #27359.
This is part of 28422, so we don't have to call
consider_hibernation() once per second when we're dormant.
This commit does not remove delayed shutdown from hibernate.c: it
uses it as a backup shutdown mechanism, in case the regular shutdown
timer mechanism fails for some reason.
With the new refresh_service_descriptor() function we had both
refresh_service_descriptor() and update_service_descriptor() which is basically
the same thing.
This commit renames update_service_descriptor() to
update_service_descriptor_intro_points() to make it clear it's not a generic
refresh and it's only about intro points.
Commit changes no code.
Before this commit, we would create the descriptor signing key certificate
when first building the descriptor.
In some extreme cases, it lead to the expiry of the certificate which triggers
a BUG() when encoding the descriptor before uploading.
Ticket #27838 details a possible scenario in which this can happen. It is an
edge case where tor losts internet connectivity, notices it and closes all
circuits. When it came back up, the HS subsystem noticed that it had no
introduction circuits, created them and tried to upload the descriptor.
However, in the meantime, if tor did lack a live consensus because it is
currently seeking to download one, we would consider that we don't need to
rotate the descriptors leading to using the expired signing key certificate.
That being said, this commit does a bit more to make this process cleaner.
There are a series of things that we need to "refresh" before uploading a
descriptor: signing key cert, intro points and revision counter.
A refresh function is added to deal with all mutable descriptor fields. It in
turn simplified a bit the code surrounding the creation of the plaintext data.
We keep creating the cert when building the descriptor in order to accomodate
the unit tests. However, it is replaced every single time the descriptor is
uploaded.
Fixes#27838
Signed-off-by: David Goulet <dgoulet@torproject.org>
When storing a descriptor in the client cache, if we are about to replace an
existing descriptor, make sure to close every introduction circuits of the old
descriptor so we don't have leftovers lying around.
Ticket 27471 describes a situation where tor is sending an INTRODUCE1 cell on
an introduction circuit for which it doesn't have a matching intro point
object (taken from the descriptor).
The main theory is that, after a new descriptor showed up, the introduction
points changed which led to selecting an introduction circuit not used by the
service anymore thus for which we are unable to find the corresponding
introduction point within the descriptor we just fetched.
Closes#27471.
Signed-off-by: David Goulet <dgoulet@torproject.org>
It won't be used if there are no authorized client configured. We do that so
we can easily support the addition of a client with a HUP signal which allow
us to avoid more complex code path to generate that cookie if we have at least
one client auth and we had none before.
Fixes#27995
Signed-off-by: David Goulet <dgoulet@torproject.org>
Both client and service had their own code for this. Consolidate into one
place so we avoid duplication.
Closes#27549
Signed-off-by: David Goulet <dgoulet@torproject.org>
When freeing a configuration object from confparse.c in
dump_config(), we need to call the appropriate higher-level free
function (like or_options_free()) and not just config_free().
This only happens with options (since they're the one where
options_validate allocates extra stuff) and only when running
--dump-config with something other than minimal (since
OPTIONS_DUMP_MINIMAL doesn't hit this code).
Fixes bug 27893; bugfix on 0.3.2.1-alpha.
It differs from the rest of the rephist code in that it's actually
necessary for Tor to operate, so it should probably go somewhere
else. I'm not sure where yet, so I'll leave it in the same
directory, but give it its own file.
Since this is completely core functionality, I'm putting it in
core/mainloop, even though it depends on feature/hibernate. We'll
have to sort that out in the future.
The main.c code is responsible for initialization and shutdown;
the mainloop.c code is responsible for running the main loop of Tor.
Splitting the "generic event loop" part of mainloop.c from the
event-loop-specific part is not done as part of this patch.
The parts for handling cell formats should be in src/core/or.
The parts for handling onionskin queues should be in src/core/or.
Only the crypto wrapper belongs in src/core/crypto.
When sending the INTRODUCE1 cell, we acquire the needed data for the cell but
if the RP node_t has invalid data, we'll fail the send and completely kill the
SOCKS connection.
Instead, close the rendezvous circuit and return a transient error meaning
that Tor can recover by selecting a new rendezvous point. We'll also do the
same when we are unable to encode the INTRODUCE1 cell for which at that point,
we'll simply take another shot at a new rendezvous point.
Fixes#27774
Signed-off-by: David Goulet <dgoulet@torproject.org>
In dirauth:
* bwauth.c reads and uses bandwidth files
* guardfraction.c reads and uses the guardfraction file
* reachability.c tests relay reachability
* recommend_pkg.c handles the recommended-packages lines.
* recv_descs.c handles fingerprint files and processing incoming
routerinfos that relays upload to us
* voteflag.c computes flag thresholds and sets those thresholds on
routerstatuses when computing votes
In control:
* fmt_serverstatus.c generates the ancient "v1 server status"
format that controllers expect.
In nodelist:
* routerstatus_fmt.c formats routerstatus entries for a consensus,
a vote, or for the controller.
It is not enough to look at protover for v3 rendezvous support but also we
need to make sure that the curve25519 onion key is present or in other words
that the descriptor has been fetched and does contain it.
Fixes#27797.
Signed-off-by: David Goulet <dgoulet@torproject.org>
There are now separate modules for:
* the list of router descriptors
* the list of authorities and fallbacks
* managing authority certificates
* selecting random nodes
That unit test makes sure we don't have pending SOCK request if the descriptor
turns out to be unusable.
Part of #27410.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Client side, when a descriptor is finally fetched and stored in the cache, we
then go over all pending SOCKS request for that descriptor. If it turns out
that the intro points are unusable, we close the first SOCKS request but not
the others for the same .onion.
This commit makes it that we'll close all SOCKS requests so we don't let
hanging the other ones.
It also fixes another bug which is having a SOCKS connection in RENDDESC_WAIT
state but with a descriptor in the cache. At some point, tor will expire the
intro failure cache which will make that descriptor usable again. When
retrying all SOCKS connection (retry_all_socks_conn_waiting_for_desc()), we
won't end up in the code path where we have already the descriptor for a
pending request causing a BUG().
Bottom line is that we should never have pending requests (waiting for a
descriptor) with that descriptor in the cache (even if unusable).
Fixees #27410.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The trunnel functions are written under the assumption that their
allocators can fail, so GCC LTO thinks they might return NULL. In
point of fact, they're using tor_malloc() and friends, which can't
fail, but GCC won't necessarily figure that out.
Fixes part of #27772.
This patch changes HiddenServiceExportCircuitID so instead of being a
boolean it takes a string, which is the protocol. Currently only the
'haproxy' protocol is defined.
See: https://bugs.torproject.org/4700
All node_get_all_orports() does is allocate and return a smartlist
with at most two tor_addr_port_t members that match ORPort's of
node configuration. This is harmful for memory efficiency, as it
allocates the same stuff every time it is called. However,
node_is_a_configured_bridge() does not need to call it, as it
already has all the information to check if there is configured
bridge for a given node.
The new code is arranged in a way that hopefully makes each succeeding
linear search through bridge_list less likely.
This commit makes it that the authorized clients in the descriptor are in
random order instead of ordered by how they were read on disk.
Fixes#27545
Signed-off-by: David Goulet <dgoulet@torproject.org>
Existing cached directory information can cause misleadingly high
bootstrap percentages. To improve user experience, defer reporting of
directory information progress until at least one connection has
succeeded to a relay or bridge.
Closes ticket 27169.
If a tor client gets a descriptor that it can't decrypt, chances are that the
onion requires client authorization.
If a tor client is configured with client authorization for an onion but
decryption fails, it means that the configured keys aren't working anymore.
In both cases, we'll log notice the former and log warn the latter and the
rest of the decryption errors are now at info level.
Two logs statement have been removed because it was redundant and printing the
fetched descriptor in the logs when 80% of it is encrypted wat not helping.
Fixes#27550
Signed-off-by: David Goulet <dgoulet@torproject.org>
Track bootstrap phase (enumerated by bootstrap_status_t) independently
from the bootstrap progress (which can represent intermediate
progress). This allows control_event_bootstrap_problem() to avoid
doing a linear search through the bootstrap progress space to find the
current bootstrap phase.
Move the mostly-invariant part of control_event_boostrap() into a
helper control_event_bootstrap_core(). The helper doesn't modify any
state beyond doing logging and control port notifications.
Simplify control_event_bootstrap() by making it return void again. It
is currently a fairly complicated function, and it's made more
complicated by returning an int to signal whether it logged at NOTICE
or INFO.
The callers conditionally log messages at level NOTICE based on this
return value. Change the callers to unconditionally log their verbose
human-readable messages at level INFO to keep NOTICE logs less
cluttered.
This partially reverts the changes of #14950.
>>>> CID 1439133: Null pointer dereferences (REVERSE_INULL)
>>>> Null-checking "fields" suggests that it may be null, but it
>>>> has already been dereferenced on all paths leading to the check.
>>>> CID 1439132: Null pointer dereferences (REVERSE_INULL)
>>>> Null-checking "fields" suggests that it may be null, but it
>>>> has already been dereferenced on all paths leading to the check.
There are three reasons we use a cached_dir_t to hold a consensus:
1. to serve that consensus to a client
2. to apply a consensus diff to an existing consensus
3. to send the consensus to a controller.
But case 1 is dircache-only. Case 2 and case 3 both fall back to
networkstatus_read_cached_consensus(). So there's no reason for us
to store this as a client. Avoiding this saves about 23% of our RAM
usage, according to our experiments last month.
This is, semantically, a partial revert of e5c608e535.
Fixes bug 27247; bugfix on 0.3.0.1-alpha.
We already had fallback code for "dir/status-vote/current/consensus"
to read from disk if we didn't have a cached_dir_t available. But
there's a function in networkstatus_t that does it for us, so let's
do that.
Return a newly allocated fake client authorization object instead of taking
the object as a parameter.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When reloading tor, check if our the configured client authorization have
changed from what we previously had. If so, republish the updated descriptor.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Previously, the validation by decoding a created descriptor was disabled
because the interface had to be entirely changed and not implemented at the
time.
This commit re-enabled it because it is now implemented.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Parse the client authorization section from the descriptor, use the client
private key to decrypt the auth clients, and then use the descriptor cookie to
decrypt the descriptor.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This commit refactors the existing decryption code to make it compatible with
a new logic for when the client authorization is enabled.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Because this secret data building logic is not only used by the descriptor
encoding process but also by the descriptor decoding, refactor the function to
take both steps into account.
Signed-off-by: David Goulet <dgoulet@torproject.org>
The new ClientOnionAuthDir option is introduced which is where tor looks to
find the HS v3 client authorization files containing the client private key
material.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Previously, we encrypted the descriptor without the descriptor cookie. This
commit, when the client auth is enabled, the descriptor cookie is always used.
I also removed the code that is used to generate fake auth clients because it
will not be used anymore.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We need to generate all the related keys when building the descriptor, so that
we can encrypt the descriptor.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This function was a wrapper around RSA_check_key() in openssl, which
checks for invalid RSA private keys (like those where p or q are
composite, or where d is not the inverse of e, or where n != p*q).
We don't need a function like this in NSS, since unlike OpenSSL, NSS
won't let you import a bogus private key.
I've renamed the function and changed its return type to make it
more reasonable, and added a unit test for trying to read a key
where n != p*q.
The OpenSSL "RSA" object is currently 408 bytes compares to the ASN.1 encoding
which is 140 for a 1024 RSA key.
We save 268 bytes per descriptor (routerinfo_t) *and* microdescriptor
(microdesc_t). Scaling this to 6000 relays, and considering client usually
only have microdescriptors, we save 1.608 MB of RAM which is considerable for
mobile client.
This commit makes it that we keep the RSA onion public key (used for TAP
handshake) in ASN.1 format instead of an OpenSSL RSA object.
Changes is done in both routerinfo_t and microdesc_t.
Closes#27246
Signed-off-by: David Goulet <dgoulet@torproject.org>
By removing Tor2Web, there is no way a client can be non anonymous so we
remove that function and the callsites.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In order to switch the default HS version from 2 to 3, we need tor to be smart
and be able to decide on the version by trying to load the service keys during
configuration validation.
Part of #27215
Signed-off-by: David Goulet <dgoulet@torproject.org>
Part of #27215, we need to call the ed_key_init_from_file function during
option_validate() which is before the global_options variable is set.
This commit make ed_key_init_from_file() stop using get_options() and instead
now has a or_options_t parameter.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Bug description: For each descriptor, its revision counter is the OPE
ciphertext of the number of seconds since the start time of its SRV value.
This bug caused us to confuse the SRV start time in the middle of the lifetime
of a descriptor in some edge-cases, which caused descriptor rejects.
Bug cause: The bug occurs when we fetch a 23:00 consensus after
midnight (e.g. at 00:08 when not all dirauths have fetched the latest 00:00
consensus). In that case, the voting schedule (which was used for SRV start
time calculation) would return a valid-after past-midnight, whereas our
consensus would be pre-midnight, and that would confuse the SRV start time
computation which is used by HS revision counters (because we would reset the
start time of SRV, without rotating descriptors).
Bug fix: We now use our local consensus time to calculate the SRV start time,
instead of the voting schedule. The voting schedule does not work as originally
envisioned in this case, because it was created for voting by dirauths and not
for scheduling stuff on clients.
At the same time, sternly warn any person thinking about relying on
any particular format too strictly. If you do this, and your
program breaks, it is your bug, not mine.
We need this so that the tor_api user can specify some arguments,
while the tor_api implementation adds others.
This implementation detail should not be visible to tor_api users.
This is another attempt to fix 1437668. The assertion here should
be safe, since the rules of networkstatus_get_param() keep the value
it returns in range.
If an authority is not configured with a V3BandwidthsFile, this line
SHOULD NOT appear in its vote.
If an authority is configured with a V3BandwidthsFile, but parsing
fails, this line SHOULD appear in its vote, but without any headers.
Part of 3723, implements the spec in 26799.
If bandwidth file terminator is found, set end of headers flag
and do not store the line.
If it is not, parse a relay line and check whether it is a header
line.
* add bwlist_headers argument to dirserv_read_measured_bandwidth
in order to store all the headers found when parsing the file
* add bwlist_headers to networkstatus_t in order to store the
the headers found by the previous function
* include the bandwidth headers as string in vote documents
* add test to check that dirserv_read_measured_bandwidth generates
the bwlist_headers
This patch adds two assertions in get_net_param_from_list() to ensure
that the `res` value is correctly within the range of the output domain.
Hopefully fixes Coverity CID #1415721, #1415722, and #1415723.
See: https://bugs.torproject.org/26780
These are now part of crypto_init.c. The openssl-only parts now
live in crypto_openssl_mgt.c.
I recommend reviewing this patch with -b and --color-moved.
Fun fact: these files used to be called log.[ch] until we ran into
conflicts with systems having a log.h file. But now that we always
include "lib/log/log.h", we should be fine.
This commit won't build yet -- it just puts everything in a slightly
more logical place.
The reasoning here is that "src/core" will hold the stuff that every (or
nearly every) tor instance will need in order to do onion routing.
Other features (including some necessary ones) will live in
"src/feature". The "src/app" directory will hold the stuff needed
to have Tor be an application you can actually run.
This commit DOES NOT refactor the former contents of src/or into a
logical set of acyclic libraries, or change any code at all. That
will have to come in the future.
We will continue to move things around and split them in the future,
but I hope this lays a reasonable groundwork for doing so.
