Commit Graph

59 Commits

Author SHA1 Message Date
teor
9864fcd4e3
Add a generated timestamp to the fallback directory header
This can act as a revision counter, as requested by atagar in 24742.

Part of 24725.
2018-01-05 13:27:47 +11:00
teor
d8ab1b803a
Remove an incorrect C comment from updateFallbackDirs.py's list generation code
Part of #24725.
2018-01-05 13:27:39 +11:00
teor
b6033f5734
Add a delimiter to the end of each fallback entry
This helps external parsers.

Also, add comments indicating where to add new fields in the fallback
format.

Part of 24725.
2018-01-05 13:27:32 +11:00
teor
beedf5fd81
Remove commas and equals signs from external string inputs to the fallback list
This makes sure that list parsers only see one comma per fallback entry,
and only see one equals sign per field.

Implements ticket 24726.
2018-01-05 13:27:22 +11:00
teor
6a27fc6245
Add an extrainfo cache flag for each fallback in a C comment
This allows stem to use fallbacks to fetch extra-info documents,
rather than using authorities.

Implements ticket 22759.
2018-01-05 13:27:14 +11:00
teor
561f18e724
Add a nickname to each fallback in a C comment
This makes it easier for operators to find their relays, and allows stem to
use nicknames to identify fallbacks.

Implements ticket 24600.
2018-01-05 13:27:07 +11:00
teor
ac5058ac8a
Remove the "weight=10" line from fallback directory mirror entries
This removes some redundant repeated lines.

Ticket 24681 will maintain the current fallback weights by changing
Tor's default fallback weight to 10.

Implements ticket 24679.
2018-01-05 13:26:53 +11:00
teor
8d226a2c7b
Add a type and version header to the fallback directory mirror file
This helps external parsers like stem and Relay Search.

Implements ticket 24725.
2018-01-05 13:26:44 +11:00
teor
9a0cf2376d
Update comment and add changes file for 20942 2018-01-05 13:26:14 +11:00
Dominique Ingoglia
23979d7682
Set CONSENSUS_EXPIRY_TOLERANCE to 0 2018-01-05 13:26:10 +11:00
Nick Mathewson
8e5450b752 Remove merge-conflict detritus 2017-12-20 10:15:00 -05:00
teor
1a29ad611c
Make fallback script usage instructions create a log file
Comment-only change.

Closes 24682.
2017-12-20 12:36:31 +11:00
teor
cd687a0a2c
Update fallback stability requirements
Decrease the minimum number of fallbacks to 100.

Final fix to 20913.
2017-05-16 18:12:10 +10:00
teor
0e81e2114a
Merge branch 'fallbacks-201702-v3' into fallbacks-2017-05 2017-05-16 18:03:00 +10:00
teor
3f28824931
Make the usage example in updateFallbackDirs.py actually work
And explain what it does.

Fixes bug 22270 in 0.3.0.3-alpha.
2017-05-16 17:35:47 +10:00
teor
71b79d6bf3
Modify fallback stability requirements
Increase the fallback stability requirement to 30 days.
When this was at 7 days, we chose far too many unstable fallbacks.

Decrease the guard flag requirement to 0.8.
When this was at 0.9, we lost too many fallbacks due to version upgrades.
(The running and v2dir flags ensure DirPorts are available to clients.)

Partial fixes to #20913.
2017-02-27 13:13:17 +11:00
Suphanat Chunhapanya
1e26d6dbe8 Add the output file to the usage 2017-01-25 13:11:57 -05:00
Suphanat Chunhapanya
e6eaf14350 Add usage for check_existing 2017-01-25 13:11:57 -05:00
Suphanat Chunhapanya
d97792d596 Logging existing fallbacks at warning level 2017-01-25 13:11:57 -05:00
Suphanat Chunhapanya
3fd58aa98d Automate checking existing fallbacks 2017-01-25 13:11:56 -05:00
teor
53ec087450
Avoid an error in the fallback script when a fallback doesn't have any uptime
Sometimes, the fallback generation script doesn't add attributes to the
fallbacks in the list. If this happens, log an error, and avoid selecting
that fallback.

This is a rare issue: it should not change selection behaviour.

Fixes issue #20945.
2016-12-19 15:06:04 +11:00
teor
654367f026
Allow fallbacks serving consensuses that expired less than 24 hours ago
This works around #20909, where relays serve stale consensuses for a short
time, and then recover.

Update to the fix for #20539.
2016-12-19 15:06:00 +11:00
teor
2d2bbaf259
Avoid checking fallback candidates' DirPorts if they are down in OnionOO
Exclude relays that have been down for 1 or more days from the fallback
candidate list.

When a relay operator has multiple relays, this prioritises relays that are
up over relays that are down.

Fixes issue #20926.
2016-12-19 15:05:56 +11:00
teor
4c832bcfac
Be more tolerant of a decreased number of fallbacks 2016-12-19 15:05:53 +11:00
teor
124c342364
Reduce fallback bandwidth requirement to 1 MByte/s 2016-12-19 15:05:49 +11:00
teor
35da99a712
Allow 3 fallbacks per operator
This is safe now we are choosing 200 fallbacks.

Closes ticket 20912.
2016-12-19 15:05:45 +11:00
teor
ee3e8fc3e9
Require fallbacks to have 90% Running, V2Dir, and Guard flags
This allows 73% of clients to bootstrap in the first 5 seconds without
contacting an authority.

Part of #18828.
2016-12-19 15:05:40 +11:00
teor
396bddaa4c
Require fallback directories to have the same address and port for 7 days
7 days is a tradeoff between the expected time between major Tor releases,
which is 6 months, and the number of relays with enough stability.

Relays whose OnionOO stability timer is reset on restart by bug #18050
should upgrade to Tor 0.2.8.7 or later, which has a fix for this issue.

Closes ticket #20880; maintains short-term fix in e220214 in tor-0.2.8.2-alpha.
2016-12-19 15:05:36 +11:00
teor
9629a25d10
Display the fingerprint when downloading consensuses from fallbacks 2016-12-19 15:05:33 +11:00
teor
8381d928cf
Exclude relays that deliver an expired consensus from the fallback list
Part of #20539, based on #20501.
2016-12-19 15:05:21 +11:00
teor
243d6fa0c7
Exclude relay versions affected by #20499 from the fallback list
Part of #20539, based on #20509.
2016-12-19 15:05:18 +11:00
teor
864a8eb283
Make fallback sort order configurable
Closes issue #20882.
2016-12-19 15:05:06 +11:00
teor
31e1439642
Select 200 fallback directories by default for each release
Closes ticket #20881.
2016-12-19 15:05:02 +11:00
teor
4eba30ca59
Provide bandwidth and consensus weight for each candidate fallback
And make it clear that updateFallbackDirs.py outputs bandwidth in
megabytes per second.

Closes #20878.
2016-12-19 15:04:58 +11:00
teor
6ed8e3764b
Add a missed return to fallbackdir_comment() in updateFallbackDirs.py
Closes ticket #20877.
2016-12-19 15:04:54 +11:00
teor (Tim Wilson-Brown)
26146dbe9e Comment-out fallbacks in a way the stem fallback parser understands
If we manually remove fallbacks in C by adding '/*' and '*/' on separate
lines, stem still parses them as being present, because it only looks at
the start of a line.

Add a comment to this effect in the generated source code.
2016-06-28 19:15:08 -04:00
teor (Tim Wilson-Brown)
4090612246
Avoid errors in updateFallbackDirs.py when there are no fallbacks 2016-06-22 11:56:31 +10:00
teor (Tim Wilson-Brown)
6ce53668f4
Document how to test the hard-coded fallback list 2016-06-22 11:56:27 +10:00
teor (Tim Wilson-Brown)
ab06b79cba Simplify string cleansing in fallback update script
No behavioural change
2016-04-29 11:59:46 +10:00
teor (Tim Wilson-Brown)
d41f92b006 Improve logging in fallback update script
Log a notice just before the script is about to perform a
potentially time-consuming operation

Clarify the warning when py2-ipaddress isn't found

Make log levels more consistent

No behavioural change (just logging)
2016-04-29 11:59:46 +10:00
teor (Tim Wilson-Brown)
46d813922a Improve comments in fallback update script 2016-04-29 11:43:55 +10:00
teor (Tim Wilson-Brown)
ba7691071e Report fallback directory detail changes when rebuilding list
As well as the existing reports of IPv6 address additions or removals,
the script now warns when keys change but IPv4:ORPort or
IPv6:IPv6ORPort remain the same.

Existing checks for other whitelist detail changes have also
been re-worded and upgraded to warnings.

This makes it easier for changes to be identified so operators can
be contacted to confirm whether the change is stable.
2016-04-26 19:26:22 -04:00
teor (Tim Wilson-Brown)
c157a31ee8 Limit fallbacks from the same operator
Use IP address, effective family, and contact info to
discover and limit fallbacks to one per operator.

Also analyse netblock, ports, IP version, and Exit flag,
and print the results. Don't exclude any fallbacks from
the list because of netblocks, ports, IP version, or
Exit flag.
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
999834324b Simplify fallback list output
When creating the list of fallbacks for a release:
* don't output fallback name and contact
* sort fallbacks by fingerprint
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
78ec782f76 Select fallbacks by bandwidth rather than consensus weight
But as advertised bandwidth is controlled by relays,
use consensus weight and median weight to bandwidth ratio
to approximate measured bandwidth.

Includes minor comment changes and parameter reordering.
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
e115523c98 Only count active fallbacks in updateFallbackDirs.py
Previously, we would cut the list down to 100 fallbacks,
then check if they could serve a consensus, and comment
them out if they couldn't.
This would leave us with fewer than 100 active fallbacks.

Now, we stop when there are 100 active fallbacks.

Also count fallbacks with identical contact info.
Also fix minor logging issues.
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
205a641e1d Remove fallback directory weights
Give each fallback a set weight of 10.0 for client selection.

Fallbacks must have at least 3000 consensus weight.
This is (nominally) 100 times the expected extra load of
20 kilobytes per second (50 GB per month).

Fixes issue #17905.
2016-04-15 12:21:32 +10:00
teor (Tim Wilson-Brown)
7e1b8ae79c Improve fallback selection and output
Improve the download test:
* Allow IPv4 DirPort checks to be turned off.
* Add a timeout to stem's consensus download.
* Actually check for download errors, rather than ignoring them.
* Simplify the timeout and download error checking logic.

Tweak whitelist/blacklist checks to be more robust.
Improve logging, make it warn by default.

Cleanse fallback comments more thoroughly:
* non-printables (yes, ContactInfo can have these)
* // comments (don't rely on newlines to prevent // */ escapes)
2016-04-15 12:21:32 +10:00
teor (Tim Wilson-Brown)
e2202146d1 Update default fallback directories for 0.2.8.2-alpha (Feb 2016)
Allow fallback directories which have been stable for 7 days
to work around #18050, which causes relays to submit descriptors
with 0 DirPorts when restarted. (Particularly during Tor version
upgrades.)

Ignore low fallback directory count in alpha builds.
Set the target count to 50.
2016-02-27 10:04:00 +01:00
teor (Tim Wilson-Brown)
a6017d61e1 Fix minor fallback directory script issues with time zones 2016-02-27 10:03:59 +01:00