This test runs practracker with a set of 0 thresholds, to make sure
that it enumerates all its values right. It tries running with an
empty exceptions file, and with an exceptions file that covers
_some_ of the data, and it makes sure that the outputs are as expected.
Now that there is only one toplevel place where we print problems,
we can redirect just that one print to a file when we are
regenerating the exceptions.txt file. Previously we redirected
sys.stdout, which is naughty, and forced us to send warnings (and
warnings alone) to stderr.
These flags let you suppress the message about the number of
problems and warnings, and let you control the thresholds above
which something counts as a problem.
I need this for testing.
Instead of having "consider" functions that have to call a global
ProblemVault, we can now generate all the metrics for the code
separately from the decision about what to do for them.
I'm about to refactor the code into a set of iterators that yield
*all* the metrics for the code, and then add a filter on top of that
to return the problems.
Previously, when TestingTorNetwork was set, we would manually adjust
the initvalue members of a bunch of other config_var_t, and then
re-run the early parts or parsing the options.
Now we treat the initvalue fields as immutable, but instead assign
to them in options_init(), as early as possible. Rather than
re-running the early parts of options, we just re-call the
options_init_from_string() function.
This patch de-kludges some of our code pretty handily. I think it
could later handle authorities and fallbacks, but for now I think we
should leave those alone.
When an exception is present, we can now violate the limit by a little
bit and only produce a warning. The strict flag overrides this
behavior.
I've given file sizes a 2% tolerances and function sizes/include
counts a 10% tolerance.
Part of 30752
Instead of excluding directories at the last minute if they happen
to appear in our filenames, we exclude them early, before recursing
into all their subdirectories.
Part of 29746.
The practracker_tests.py unit test file called a function by its old
name.
Also, practracker counted functions as starting one line after the
function name, and ending with the closing brace. Now they start
with the open brace and end with the closing brace.
Coverity doesn't understand that if begin_cell_parse() returns 0 and
sets is_begindir to 0, its address field will always be set.
Fixes bug 30126; bugfix on 0.2.4.7-alpha; Fixes CID 1447296.
Right now, this has been done at a high level by confparse.c, but it
makes more sense to lower it.
This API is radically un-typesafe as it stands; we'll be wrapping it
in a safer API as we do #30914 and lower the struct manipulation
code as well.
Closes ticket 30864.
Set the env vars:
* TOR_MASTER_NAME to override the tor master branch name, and
* TOR_WKT_NAME to override the worktree path
Fixes bug 30841; bugfix on 0.4.0.1-alpha.
Proposal 289 prevents SENDME-flooding by requiring the other side to
authenticate the data it has received. But this data won't actually
be random if they are downloading a known resource. "No problem",
we said, "let's fell the empty parts of our cells with some
randomness!" and we did that in #26871.
Unfortunately, if the relay data payloads are all completely full,
there won't be any empty parts for us to randomize.
Therefore, we now pick random "randomness windows" between
CIRCWINDOW_INCREMENT/2 and CIRCWINDOW_INCREMENT. We remember whether we have
sent a cell containing at least 16 bytes of randomness in that window. If we
haven't, then when the window is exhausted, we send one. (This window approach
is designed to lower the number of rng checks we have to do. The number 16 is
pulled out of a hat to change the attacker's guessing difficulty to
"impossible".)
Implements 28646.
We added this facility so that we could get deterministic PRNG
behavior for coverage testing on tests that use a replaced PRNG.
We need to have our coverage determinism tool test for this as well.
Turns out that we were only recording the "b_digest" but to have
bidirectionnal authenticated SENDMEs, we need to use the "f_digest" in the
forward cell situation.
Because of the cpath refactoring, this commit plays with the crypt_path_ and
relay_crypto_t API a little bit in order to respect the abstractions.
Previously, we would record the cell digest as the SENDME digest in the
decrypt cell function but to avoid code duplication (both directions needs to
record), we now do that right after iff the cell is recognized (at the edge).
It is now done in circuit_receive_relay_cell() instead.
We now also record the cell digest as the SENDME digest in both relay cell
encryption functions since they are split depending on the direction.
relay_encrypt_cell_outbound() and relay_encrypt_cell_inbound() need to
consider recording the cell digest depending on their direction (f vs b
digest).
Fixes#30428
Signed-off-by: David Goulet <dgoulet@torproject.org>
It turns out that only the exit side is validating the authenticated SENDME v1
logic and never the client side. Which means that if a client ever uploaded
data towards an exit, the authenticated SENDME logic wouldn't apply.
For this to work, we have to record the cell digest client side as well which
introduced a new function that supports both type of edges.
This also removes a test that is not valid anymore which was that we didn't
allow cell recording on an origin circuit (client).
Part of #30428
Signed-off-by: David Goulet <dgoulet@torproject.org>
Because github PRs choose the most recent origin/master at the time of the PR
(and for any fixups pushed to a PR later to send to CI), there are tons of
conflicts and unexpected practracker issues.
This means CI can suddenly fail after fixups to a branch that pass locally.
Then CI fails and we have to close and re-open the PR.
Split the core reply formatting code out of control_fmt.c into
control_proto.c. The remaining code in control_format.c deals with
specific subsystems and will eventually move to join those subsystems.
(This should be all of the command that work nicely with positional
arguments only.)
Some of these commands should probably treat extra arguments as
incorrect, but for now I'm trying to be careful not to break
any existing users.
The first line break in particular was mishandled: it was discarded
if no arguments came before it, which made it impossible to
distinguish arguments from the first line of the body.
To solve this, we need to allocate a copy of the command rather than
using NUL to separate it, since we might have "COMMAND\n" as our input.
Fixes ticket 29984.
Previously we excluded any line containing one of these terms from
consideration as the start or end of a function. Now we're more
careful, and we only ignore these terms when they appear to be
starting a function definition.
This was causing issues because the exceptions file is written using Posix
paths, whereas practracker in Windows was trying to match Windows paths ("\"
instead of "/").
- Introduce 'make check-best-practices'.
- Fix up Tor topdir etc to work with the way 'make check-local' gets called.
- Make practracker less likely to print useless stuff.
- every .h file needs an #ifndef/#define pair.
- They must refer to the same macro.
- The guard macros that they refer to must be unique across all headers.
Replace the "git pull" by one single "git fetch origin" and then "git merge"
into each defined branches.
This speeds up the process considerably.
Closes#29616
Signed-off-by: David Goulet <dgoulet@torproject.org>
No longer messes up with MOCK_IMPL. Also update the tests to show that. We are
still being innacurate on the line count in some cases, but that doesnt matter
so much.
It was looking for object files made with the old automake
directorations, but those changed when we split up our libraries.
Fixes bug 29435; bugfix on 0.3.5.1-alpha.
encoding and decoding.
There are bunches of places where we don't want to invest in a full
fuzzer, but we would like to make sure that some string operation
can handle all its possible inputs. This fuzzer uses the first byte
of its input to decide what to do with the rest of the input. Right
now, all the possibilities are decoding a string, and seeing whether
it is decodeable. If it is, we try to re-encode it and do the whole
thing again, to make sure we get the same result.
This turned up a lot of bugs in the key-value parser, and I think it
will help in other cases too.
Closes ticket 28808.
If a relay matches at least one fingerprint, IPv4 address, or IPv6
address in the fallback whitelist, it can become a fallback. This
reduces the work required to keep the list up to date.
Closes ticket 28768.
Tor clients on 0.3.5.6-rc? and later will use a consensus that will become
valid up to 24 hours in the future.
Clients on 0.3.5.5-alpha? and earlier won't accept future consensuses.
Update the fallback expiry tolerance to match tor's checks.
Part of 28768, follow-up on 28591.
Tor clients will use a consensus that expired up to 24 hours ago.
Clients on 0.3.5.5-alpha? and earlier won't select guards from an expired
consensus, but they can still bootstrap if they have existing guards.
Update the fallback expiry tolerance to match tor's checks.
Part of 28768, follow-up on 24661.
We can safely silence SC2086 warning on $CHECKERS, as contents of that
is hardcoded into script, and we don't want to require Bash to use Bash
array here. Double-quote $OUTPUTARG, as it's value depends on environment
variable.
This updateVersions.pl script was one of the only essential perl
scripts left in out maint system, and was the only one that used
autoconf to fill in the script.
This script adds a feature to define an APPROX_RELEASE_DATE macro
that is updated when the version changes. We'll use this to
implement prop297, so that we have an accurate view of when a
release date happens.
This is a very gentle commit that just lays the groundwork in the
build system: it puts the include files to build libtor-app.a into
src/core, and to build the tor executable into src/app. The
executable is now "src/app/tor".
Includes configuration files to enforce these rules on lib and
common. Of course, "common" *is* a modularity violation right now,
so these rules aren't as strict as I would like them to be.
The new gcov sometimes emits *s if there is a line containing
multiple basic blocks, and some are not executed. (The gcov
documentation says something weird about this point, so I'm trying
to interpret it into the compilerese that I'm familiar with.)
That's bug 26101.
Also, when we're looking for unique variations in our coverage, we
would like cov-diff to suppress timestamps on the diffs. That's bug
26102.
Both of these are bugfixes on 0.2.5.1-alpha when the cov-diff script
was introduced.
My apologies for the perl.
Avoid selecting fallbacks that change their IP addresses too often.
Select more fallbacks by ignoring the Guard flag, and allowing lower
cutoffs for the Running and V2Dir flags. Also allow a lower bandwidth,
and a higher number of fallbacks per operator (5% of the list).
Implements ticket 24785.
Add the generateFallbackDirLine.py script for automatically generating
fallback directory mirror lines from relay fingerprints. No more typos!
Add the lookupFallbackDirContact.py script for automatically looking up
operator contact info from relay fingerprints.
Implements ticket 24706.
This removes some redundant repeated lines.
Ticket 24681 will maintain the current fallback weights by changing
Tor's default fallback weight to 10.
Implements ticket 24679.
We only occasionally checked for fallbacks on the same machine.
And I'm not convinced it makes much of a difference with ~150 fallbacks.
Part of #22321.
To do this, we had to make sure it passes when the changes directory
is empty. I also tried to improve the quality of the output, and
fix some false-positive cases. Let's see how this goes!
Closes ticket 23564.
This is a large and important unit test for the hidden service version
3! It tests the service reachability for a client using different
consensus timings and makes sure that the computed hashring is the same
on both side so it is actually reachable.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This patch adds a script written by Nick for bug #13172 to clean up the
usage of ==, !=, <, >, <=, and >= by replacing them with their symbolic
OP_* counterpart. This will ensure that a tool like Coccinelle doesn't
get confused and silently ignore large blocks of code.
Increase the fallback stability requirement to 30 days.
When this was at 7 days, we chose far too many unstable fallbacks.
Decrease the guard flag requirement to 0.8.
When this was at 0.9, we lost too many fallbacks due to version upgrades.
(The running and v2dir flags ensure DirPorts are available to clients.)
Partial fixes to #20913.
Found using flake8 which gave the following output;
lintChanges.py:10:13: E225 missing whitespace around operator
lintChanges.py:25:1: E302 expected 2 blank lines, found 1
lintChanges.py:46:5: F841 local variable 'lines' is assigned to but never used
lintChanges.py:52:32: E228 missing whitespace around modulo operator
lintChanges.py:53:11: E201 whitespace after '('
lintChanges.py:56:41: E228 missing whitespace around modulo operator
Part of ticket 21096.
Sometimes, the fallback generation script doesn't add attributes to the
fallbacks in the list. If this happens, log an error, and avoid selecting
that fallback.
This is a rare issue: it should not change selection behaviour.
Fixes issue #20945.
Exclude relays that have been down for 1 or more days from the fallback
candidate list.
When a relay operator has multiple relays, this prioritises relays that are
up over relays that are down.
Fixes issue #20926.
7 days is a tradeoff between the expected time between major Tor releases,
which is 6 months, and the number of relays with enough stability.
Relays whose OnionOO stability timer is reset on restart by bug #18050
should upgrade to Tor 0.2.8.7 or later, which has a fix for this issue.
Closes ticket #20880; maintains short-term fix in e220214 in tor-0.2.8.2-alpha.
The functions it warns about are:
assert, memcmp, strcat, strcpy, sprintf, malloc, free, realloc,
strdup, strndup, calloc.
Also, fix a few lingering instances of these in the code. Use other
conventions to indicate _intended_ use of assert and
malloc/realloc/etc.
Only some very ancient distributions don't ship with Libevent 2 anymore,
even the oldest supported Ubuntu LTS version has it. This allows us to
get rid of a lot of compat code.
If we manually remove fallbacks in C by adding '/*' and '*/' on separate
lines, stem still parses them as being present, because it only looks at
the start of a line.
Add a comment to this effect in the generated source code.
Remove a fallback that changed its fingerprint after it was listed
This happened after to a software update:
https://lists.torproject.org/pipermail/tor-relays/2016-June/009473.html
Remove a fallback that changed IPv4 address
Remove two fallbacks that were slow to deliver consensuses,
we can't guarantee they'll be fast in future.
Blacklist all these fallbacks until operators confirm they're stable.
Operators have sent emails asking to have their relays added or
removed from the fallback list. Since none of the blacklisted
relays are in the hard-coded falback list, it does not need to
be changed.
Log a notice just before the script is about to perform a
potentially time-consuming operation
Clarify the warning when py2-ipaddress isn't found
Make log levels more consistent
No behavioural change (just logging)
As well as the existing reports of IPv6 address additions or removals,
the script now warns when keys change but IPv4:ORPort or
IPv6:IPv6ORPort remain the same.
Existing checks for other whitelist detail changes have also
been re-worded and upgraded to warnings.
This makes it easier for changes to be identified so operators can
be contacted to confirm whether the change is stable.
Also add misbehaving relays to updateFallbackDirs.py blacklist,
but leave them commented out in case it's a transient issue,
or it's been resolved by the download check fixes. (These
relays hang stem's downloader. It's unlikely they'll ever help
clients.)
Use IP address, effective family, and contact info to
discover and limit fallbacks to one per operator.
Also analyse netblock, ports, IP version, and Exit flag,
and print the results. Don't exclude any fallbacks from
the list because of netblocks, ports, IP version, or
Exit flag.
But as advertised bandwidth is controlled by relays,
use consensus weight and median weight to bandwidth ratio
to approximate measured bandwidth.
Includes minor comment changes and parameter reordering.
Previously, we would cut the list down to 100 fallbacks,
then check if they could serve a consensus, and comment
them out if they couldn't.
This would leave us with fewer than 100 active fallbacks.
Now, we stop when there are 100 active fallbacks.
Also count fallbacks with identical contact info.
Also fix minor logging issues.
Give each fallback a set weight of 10.0 for client selection.
Fallbacks must have at least 3000 consensus weight.
This is (nominally) 100 times the expected extra load of
20 kilobytes per second (50 GB per month).
Fixes issue #17905.
Improve the download test:
* Allow IPv4 DirPort checks to be turned off.
* Add a timeout to stem's consensus download.
* Actually check for download errors, rather than ignoring them.
* Simplify the timeout and download error checking logic.
Tweak whitelist/blacklist checks to be more robust.
Improve logging, make it warn by default.
Cleanse fallback comments more thoroughly:
* non-printables (yes, ContactInfo can have these)
* // comments (don't rely on newlines to prevent // */ escapes)
Document this convention.
Add a script to post-process .gcov files in order to stop nagging us
about excluded lines.
Teach cov-diff to handle these post-processed files.
Closes ticket 16792
Allow fallback directories which have been stable for 7 days
to work around #18050, which causes relays to submit descriptors
with 0 DirPorts when restarted. (Particularly during Tor version
upgrades.)
Ignore low fallback directory count in alpha builds.
Set the target count to 50.
Allow fallback directories which have been stable for 30 days
to work around #18050, which causes relays to submit descriptors
with 0 DirPorts when restarted. (Particularly during Tor version
upgrades.)
Ignore low fallback directory count in alpha builds.
* support maximum history age in _avg_generic_history()
* fix division-by-zero trap in _avg_generic_history()
* skip missing (i.e. null/None) intervals in _avg_generic_history()
* Python timedelta.total_seconds() function not available in 2.6;
replace with equivalent expression
* set DEBUG logging level to make relay exclusion reasons visible
* move CUTOFF_GUARD test to end in order to expose more exclusion
reasons
Patch by "starlight", merge modifications by "teor".
Allow cached or outdated Onionoo data to be used to choose
fallback directories, as long as it's less than a day old.
Modify last modified date checks in preparation for Onionoo change
"Tor has included a feature to fetch the initial consensus from nodes
other than the authorities for a while now. We just haven't shipped a
list of alternate locations for clients to go to yet.
Reasons why we might want to ship tor with a list of additional places
where clients can find the consensus is that it makes authority
reachability and BW less important.
We want them to have been around and using their current key, address,
and port for a while now (120 days), and have been running, a guard,
and a v2 directory mirror for most of that time."
Features:
* whitelist and blacklist for an opt-in/opt-out trial.
* excludes BadExits, tor versions that aren't recommended, and low
consensus weight directory mirrors.
* reduces the weighting of Exits to avoid overloading them.
* places limits on the weight of any one fallback.
* includes an IPv6 address and orport for each FallbackDir, as
implemented in #17327. (Tor won't bootstrap using IPv6 fallbacks
until #17840 is merged.)
* generated output includes timestamps & Onionoo URL for traceability.
* unit test ensures that we successfully load all included default
fallback directories.
Closes ticket #15775. Patch by "teor".
OnionOO script by "weasel", "teor", "gsathya", and "karsten".
These scripts are now a little more bulletproof, cache data a little
better, and generate more information. Notably, they search for the
vectors or edges to cut that would lower the size of the largest
SCC.
Extrainfo documents are now ed-signed just as are router
descriptors, according to proposal 220. This patch also includes
some more tests for successful/failing parsing, and fixes a crash
bug in ed25519 descriptor parsing.
This includes the link handshake variations for proposal220.
We'll use this for testing first, and then use it to extend our
current code to support prop220.
Additional fixes to make the change work;
- fix Python 2 vs 3 issues
- fix some PEP 8 warnings
- handle paths with numbers correctly
- mention the make rule in doc/HACKING.
