Commit Graph

99 Commits

Author SHA1 Message Date
Nick Mathewson
3934e78bb9 Make format_changelog.py add links to bugs 2016-05-27 09:26:49 -04:00
Nick Mathewson
5dbb0bdae9 Merge branch 'maint-0.2.8' 2016-05-16 08:33:53 -04:00
teor (Tim Wilson-Brown)
ab06b79cba Simplify string cleansing in fallback update script
No behavioural change
2016-04-29 11:59:46 +10:00
teor (Tim Wilson-Brown)
d41f92b006 Improve logging in fallback update script
Log a notice just before the script is about to perform a
potentially time-consuming operation

Clarify the warning when py2-ipaddress isn't found

Make log levels more consistent

No behavioural change (just logging)
2016-04-29 11:59:46 +10:00
teor (Tim Wilson-Brown)
46d813922a Improve comments in fallback update script 2016-04-29 11:43:55 +10:00
Nick Mathewson
fb9c9e04f0 Merge branch 'maint-0.2.8' 2016-04-26 19:27:39 -04:00
teor (Tim Wilson-Brown)
de08f2d96b Whitelist additional fallbacks after creating April 2016 list 2016-04-26 19:26:22 -04:00
teor (Tim Wilson-Brown)
ba7691071e Report fallback directory detail changes when rebuilding list
As well as the existing reports of IPv6 address additions or removals,
the script now warns when keys change but IPv4:ORPort or
IPv6:IPv6ORPort remain the same.

Existing checks for other whitelist detail changes have also
been re-worded and upgraded to warnings.

This makes it easier for changes to be identified so operators can
be contacted to confirm whether the change is stable.
2016-04-26 19:26:22 -04:00
teor (Tim Wilson-Brown)
8b90d45f78 Add fallbacks to white/blacklist from operator responses
Also add misbehaving relays to updateFallbackDirs.py blacklist,
but leave them commented out in case it's a transient issue,
or it's been resolved by the download check fixes. (These
relays hang stem's downloader. It's unlikely they'll ever help
clients.)
2016-04-26 19:26:21 -04:00
teor (Tim Wilson-Brown)
c157a31ee8 Limit fallbacks from the same operator
Use IP address, effective family, and contact info to
discover and limit fallbacks to one per operator.

Also analyse netblock, ports, IP version, and Exit flag,
and print the results. Don't exclude any fallbacks from
the list because of netblocks, ports, IP version, or
Exit flag.
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
999834324b Simplify fallback list output
When creating the list of fallbacks for a release:
* don't output fallback name and contact
* sort fallbacks by fingerprint
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
78ec782f76 Select fallbacks by bandwidth rather than consensus weight
But as advertised bandwidth is controlled by relays,
use consensus weight and median weight to bandwidth ratio
to approximate measured bandwidth.

Includes minor comment changes and parameter reordering.
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
e115523c98 Only count active fallbacks in updateFallbackDirs.py
Previously, we would cut the list down to 100 fallbacks,
then check if they could serve a consensus, and comment
them out if they couldn't.
This would leave us with fewer than 100 active fallbacks.

Now, we stop when there are 100 active fallbacks.

Also count fallbacks with identical contact info.
Also fix minor logging issues.
2016-04-15 12:26:37 +10:00
teor (Tim Wilson-Brown)
205a641e1d Remove fallback directory weights
Give each fallback a set weight of 10.0 for client selection.

Fallbacks must have at least 3000 consensus weight.
This is (nominally) 100 times the expected extra load of
20 kilobytes per second (50 GB per month).

Fixes issue #17905.
2016-04-15 12:21:32 +10:00
teor (Tim Wilson-Brown)
7e1b8ae79c Improve fallback selection and output
Improve the download test:
* Allow IPv4 DirPort checks to be turned off.
* Add a timeout to stem's consensus download.
* Actually check for download errors, rather than ignoring them.
* Simplify the timeout and download error checking logic.

Tweak whitelist/blacklist checks to be more robust.
Improve logging, make it warn by default.

Cleanse fallback comments more thoroughly:
* non-printables (yes, ContactInfo can have these)
* // comments (don't rely on newlines to prevent // */ escapes)
2016-04-15 12:21:32 +10:00
Nick Mathewson
4043f2c95f Adopt the LCOV convention for marking lines as unreachable by tests.
Document this convention.

Add a script to post-process .gcov files in order to stop nagging us
about excluded lines.

Teach cov-diff to handle these post-processed files.

Closes ticket 16792
2016-04-12 21:12:10 -04:00
Nick Mathewson
560450f2fb helper script to highlight undocumented members 2016-03-26 10:31:59 -04:00
Nick Mathewson
52bc921402 Fix (most) lintChanges warnings on master. 2016-03-21 11:14:57 -04:00
Nick Mathewson
69fc025e95 Merge remote-tracking branch 'teor/fallbacks-201602-v2' 2016-02-28 15:51:22 +01:00
Nick Mathewson
57699de005 Update the copyright year. 2016-02-27 18:48:19 +01:00
teor (Tim Wilson-Brown)
e2202146d1 Update default fallback directories for 0.2.8.2-alpha (Feb 2016)
Allow fallback directories which have been stable for 7 days
to work around #18050, which causes relays to submit descriptors
with 0 DirPorts when restarted. (Particularly during Tor version
upgrades.)

Ignore low fallback directory count in alpha builds.
Set the target count to 50.
2016-02-27 10:04:00 +01:00
teor (Tim Wilson-Brown)
a6017d61e1 Fix minor fallback directory script issues with time zones 2016-02-27 10:03:59 +01:00
teor (Tim Wilson-Brown)
9580036986 Check if fallback directories can serve the consensus fast enough 2016-02-27 10:03:59 +01:00
Nick Mathewson
2123f0a2f9 Make the callgraph module-aware 2016-02-10 15:09:10 -05:00
Nick Mathewson
f09d03c384 use sortChanges to fold changes files into changelog. No additional editing or formatting yet. 2016-02-01 16:56:20 -05:00
Nick Mathewson
deea08c13f Fix all the warnings from lintChanges.py 2016-02-01 16:46:29 -05:00
teor (Tim Wilson-Brown)
ab3c86479a Add default fallback directories for the 0.2.8 alpha releases
Allow fallback directories which have been stable for 30 days
to work around #18050, which causes relays to submit descriptors
with 0 DirPorts when restarted. (Particularly during Tor version
upgrades.)

Ignore low fallback directory count in alpha builds.
2016-01-18 20:15:59 -05:00
teor (Tim Wilson-Brown)
efd2c9d05b Fix a display issue in updateFallbackDirs.py output 2016-01-18 12:36:49 +11:00
Nick Mathewson
d062baac86 Merge remote-tracking branch 'teor/bug17888' 2016-01-12 12:26:06 -05:00
Nick Mathewson
7616922e81 Merge remote-tracking branch 'teor/bug17887-v2' 2016-01-12 10:47:54 -05:00
teor (Tim Wilson-Brown)
1c139f60d2 Don't limit fallback directory candidates to 500
Only impose the limit when selecting fallback directories for
inclusion in tor.
2016-01-12 10:10:05 -05:00
teor (Tim Wilson-Brown)
da968e558e Fix various issues with fallback directory data handling
* support maximum history age in _avg_generic_history()
* fix division-by-zero trap in _avg_generic_history()
* skip missing (i.e. null/None) intervals in _avg_generic_history()
* Python timedelta.total_seconds() function not available in 2.6;
  replace with equivalent expression
* set DEBUG logging level to make relay exclusion reasons visible
* move CUTOFF_GUARD test to end in order to expose more exclusion
  reasons

Patch by "starlight", merge modifications by "teor".
2016-01-12 10:10:05 -05:00
teor (Tim Wilson-Brown)
bc2bed8979 Don't reduce the weight of exits selected as fallback directories
When selecting exits as fallback directories, don't reduce
their weights.

Closes ticket #17888.
2016-01-12 13:56:45 +11:00
teor (Tim Wilson-Brown)
430181fad2 Allow fallback directory selection to use day-old data
Allow cached or outdated Onionoo data to be used to choose
fallback directories, as long as it's less than a day old.

Modify last modified date checks in preparation for Onionoo change
2016-01-12 13:51:25 +11:00
teor
4c1c2a313d Add Fallback Directory Candidate Selection Script
"Tor has included a feature to fetch the initial consensus from nodes
 other than the authorities for a while now. We just haven't shipped a
 list of alternate locations for clients to go to yet.

 Reasons why we might want to ship tor with a list of additional places
 where clients can find the consensus is that it makes authority
 reachability and BW less important.

 We want them to have been around and using their current key, address,
 and port for a while now (120 days), and have been running, a guard,
 and a v2 directory mirror for most of that time."

Features:
* whitelist and blacklist for an opt-in/opt-out trial.
* excludes BadExits, tor versions that aren't recommended, and low
  consensus weight directory mirrors.
* reduces the weighting of Exits to avoid overloading them.
* places limits on the weight of any one fallback.
* includes an IPv6 address and orport for each FallbackDir, as
  implemented in #17327. (Tor won't bootstrap using IPv6 fallbacks
  until #17840 is merged.)
* generated output includes timestamps & Onionoo URL for traceability.
* unit test ensures that we successfully load all included default
  fallback directories.

Closes ticket #15775. Patch by "teor".
OnionOO script by "weasel", "teor", "gsathya", and "karsten".
2015-12-16 05:54:40 +11:00
Nick Mathewson
a02f6e3d23 update codegen script to work with latest mozilla 2015-10-07 16:24:30 -04:00
Nick Mathewson
216024261b add more clang checkers; find no new bugs. 2015-09-22 11:37:50 -04:00
Nick Mathewson
742514be2f scan-build is more mature now; no need to hardcode the one in my /opt/clang-3.4 2015-09-22 09:58:16 -04:00
Nick Mathewson
1eb2106375 Document callgraph analysis code 2015-08-25 11:53:20 -04:00
Sebastian Hahn
1633d1ad1d Remove tor-fw-helper more thoroughly 2015-08-21 10:36:53 -04:00
Sebastian Hahn
38601f06b7 Observe workqueue_reply_t in check-spaces 2015-08-21 10:36:53 -04:00
Nick Mathewson
bb46630513 Hack up the scripts/maint/*callgraph* scripts to do more, better
These scripts are now a little more bulletproof, cache data a little
better, and generate more information.  Notably, they search for the
vectors or edges to cut that would lower the size of the largest
SCC.
2015-08-15 22:37:32 -04:00
Nick Mathewson
def5883bbb Update callgraph code to find and output strongly connected components 2015-08-10 12:11:34 -04:00
Nick Mathewson
ebff18d18e Allow cov-display to take directories as arguments 2015-08-05 11:06:41 -04:00
Nick Mathewson
26c9691186 Check in some scripts I use for callgraph analysis. 2015-07-30 12:23:28 -04:00
Nick Mathewson
90701c1de4 Add my script to dump per-file or per-function coverage 2015-07-16 11:55:44 -04:00
Nick Mathewson
e71c8801cf more generic scan-build script 2015-05-29 13:50:05 -04:00
Nick Mathewson
1b52e95028 Merge branch '12498_ed25519_keys_v6'
Fixed numerous conflicts, and ported code to use new base64 api.
2015-05-28 11:04:33 -04:00
Nick Mathewson
79db24b3d5 Sign extrainfo documents with ed25519
Extrainfo documents are now ed-signed just as are router
descriptors, according to proposal 220.  This patch also includes
some more tests for successful/failing parsing, and fixes a crash
bug in ed25519 descriptor parsing.
2015-05-28 10:42:22 -04:00
Nick Mathewson
b600b68b20 Revise makedesc.py: teach it how to emit ed signatures and crosscerts
Also, add a trivial ed25519-signed routerinfo to the tests.
2015-05-28 10:41:50 -04:00