From d1a94a3a7ff3cb4fd5886fe9feec553fd2196e77 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Tue, 22 Sep 2020 14:17:26 -0400 Subject: [PATCH 1/5] Improve doc/state-contents.txt Part of a fix for #40136. This patch adds all the state file entries to the documentation, and documents the ones that I understand well. --- doc/state-contents.txt | 179 +++++++++++++++++++++++++++++++---------- 1 file changed, 135 insertions(+), 44 deletions(-) diff --git a/doc/state-contents.txt b/doc/state-contents.txt index 44716efc0c..9fe41f67a1 100644 --- a/doc/state-contents.txt +++ b/doc/state-contents.txt @@ -14,6 +14,21 @@ Recognized fields are: Time when this state file was written. Given in ISO format (YYYY-MM-DD HH:MM:SS) + + MinutesSinceUserActivity (integer) + Dormant (0, 1, or "auto") + + These values are used to keep track of how long Tor has been idle, + for the purpose of becoming 'dormant' after a long period without + any user-initiated requests. + + "MinutesSinceUserActivity" is the number of minutes since the last + time the user asked us to do something. It is set to zero if we're + dormant. + + "Dormant" is 1 if Tor was dormant when it wrote its state file, 0 if + Tor was active, and "auto" if Tor was starting for the first time. + AccountingBytesReadInInterval (memory unit) AccountingBytesWrittenInInterval (memory unit) AccountingExpectedUsage (memory unit) @@ -36,12 +51,121 @@ Recognized fields are: BytesAtSoftLimit. If we hit the soft limit already, we did so at SoftLimitHitAt. + TransportProxy + + One or more of these may be present. + + The format is "transportname addr:port", to remember the address + at which a pluggable transport was listening. Tor bridges use + this information to spawn pluggable transport listeners in the + same IP address and TCP port even after tor client restarts. + + BWHistory___Ends (ISO time) + BWHistory___Interval (integer, number of seconds) + BWHistory___Values (comma-separated list of integer) + BWHistory___Maxima (comma-separated list of integer) + + These values record bandwidth history. The "Values" fields are a list, + for some number of "Intervals", of the total amount read/written during + that integer. The "Maxima" are the highest burst for each interval. + + Interval duration is set by the "Interval" field, in seconds. The + "Ends" field is the ending time of the last interval in each list. + + Recognized values for "___" are: + Read -- total bytes read + Write -- total bytes written + DirRead -- total bytes read for directory connections. + DirWrite -- total bytes written for directory connections. + IPv6Read -- total bytes read on IPv6 connections + IPv6Write -- total bytes written on IPv6 connections + + LastRotatedOnionKey + + The last time that we changed our onion key for a new one. + Given in ISO format (YYYY-MM-DD HH:MM:SS) + + This field is used to ensure that onion key rotations happen with the + appropriate frequency. + + TotalBuildTimes + CircuitBuildAbandonedCount + CircuitBuildTimeBin + BuildTimeHistorgram + + XXXX writeme. + + Guard [key=value] [key=value]... + + Describes a single entry guard used by the client. Key=value + entries with unrecognized keys are persisted. Order is not + significant. For more information about terminology used here, + system, see guard-spec.txt in the tor specifications repository. + + Recognized keys are: + + in (string) + + The name of a guard selection that this guard is in. + + rsa_id (string) + + RSA fingerprint of this guard, without spaces. + + nickname (string) + + Declared nickname of this guard. + + sampled_on (Time in ISO YYYY-MM-DDTHH:MM:SS format) + + When was this guard added to the Guard sample? + + sampled_by (tor version) + + Which version of Tor added this Guard to the sample? + (Used to help with debugging.) + + sampled_idx (integer) + + Index of this guard among sampled guards. + + listed (boolean) + + Did this guard appear in the most recent consensus? + + unlisted_since (Time in ISO YYYY-MM-DDTHH:MM:SS format) + + If this guard is not listed, when is the earliest + consensus in which we found it unlisted? + + confirmed_on (Time in ISO YYYY-MM-DDTHH:MM:SS format) + + When did this guard become confirmed? + + confirmed_idx (integer) + + Index of this guard among confirmed guards. + + bridge_addr (address) + + If this guard is a bridge, its current address. + + pb_use_attempts + pb_use_successes + pb_circ_attempts + pb_successful_circuits_closed + pb_collapsed_circuits + pb_timeouts + + +Obsolete fields include: + EntryGuard EntryGuardDownSince EntryGuardUnlistedSince EntryGuardAddedBy - These lines form sections related to entry guards. Each section + These lines formed sections related to entry guards. Each section starts with a single EntryGuard line, and is then followed by information on the state of the Entry guard. @@ -56,50 +180,17 @@ Recognized fields are: space-separated fields: the identity of the entry guard, the version of Tor that added it, and the ISO time at which it was added. - TransportProxy + EntryGuardPathBias and EntryGuardPathUseBias are superseded by + the `pb_...` elements in the Guard flag, and served a similar purpose. - One or more of these may be present. + These entries have all been superseded by the Guard line type, + since Tor 0.3.0.1-alpha. - The format is "transportname addr:port", to remember the address - at which a pluggable transport was listening. Tor bridges use - this information to spawn pluggable transport listeners in the - same IP address and TCP port even after tor client restarts. + HidServRevCounter - BWHistoryReadEnds (ISO time) - BWHistoryReadInterval (integer, number of seconds) - BWHistoryReadValues (comma-separated list of integer) - BWHistoryReadMaxima (comma-separated list of integer) - BWHistoryWriteEnds - BWHistoryWriteInterval - BWHistoryWriteValues - BWHistoryWriteMaxima - BWHistoryDirReadEnds - BWHistoryDirReadInterval - BWHistoryDirReadValues - BWHistoryDirReadMaxima - BWHistoryDirWriteEnds - BWHistoryDirWriteInterval - BWHistoryDirWriteValues - BWHistoryDirWriteMaxima + It was once used to ensure that v3 onion service directory revision + numbers were strictly increasing; we now use an order-preserving + encryption scheme for that purpose. - These values record bandwidth history. The "Values" fields are a list, for - some number of "Intervals", of the total amount read/written during that - integer. The "Maxima" are the highest burst for each interval. - - Interval duration is set by the "Interval" field, in seconds. The - "Ends" field is the ending time of the last interval in each list. - - The *Read* and *Write* fields are the total amount read and - written; the *DirRead* and *DirWrite* variants are for directory - traffic only. - - LastRotatedOnionKey - - The last time that we changed our onion key for a new one. - Given in ISO format (YYYY-MM-DD HH:MM:SS) - - TotalBuildTimes - CircuitBuildAbandonedCount - CircuitBuildTimeBin - - XXXX writeme. + This option could appear multiple times; each time it does, it + applies to a different hidden service. From 371cd7ac4f538f6f70ae9c35680e78d39f9e01dd Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Tue, 22 Sep 2020 14:35:11 -0400 Subject: [PATCH 2/5] Try to document pathbias fields in state file. This will require feedback from @mikeperry, since these are a bit subtle and I've likely mangled some of these. --- doc/state-contents.txt | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/doc/state-contents.txt b/doc/state-contents.txt index 9fe41f67a1..58c77bc292 100644 --- a/doc/state-contents.txt +++ b/doc/state-contents.txt @@ -155,8 +155,39 @@ Recognized fields are: pb_circ_attempts pb_successful_circuits_closed pb_collapsed_circuits + pb_unusable_circuits pb_timeouts + Used by the pathbias subsystem to keep a record of the + behavior of circuits built through this guard, in hopes of + detecting guards try to that interfere with traffic. + + All of these fields are floating-point integers which + represent a count of circuits that have been trated in + various ways. These counts decay with time. + + "use_attempts" is a count of the circuits that we've built + and tried to use for traffic. + + "successful_circuits_closed" is a count of circuits that + have closed "naturally" without timeout or error. + + "use_successes" is a count of circuits that we've sent + traffic on, and which closed "naturally" without timeout + or error. + + "circ_attempts" is a count of circuits we've tried to + build through this guard. + + "collapsed_circuits" is a count of circuits that failed + after having been built, but before sending traffic. + + "unusable_circuits" is a count of circuits that we + launched as path-bias probes but which failed to build, + or which encountered questionable errors. + + "timeouts" is a count of circuits that encountered a + timeout while we were building them. Obsolete fields include: From 5ffd8bf2b8b0c051647179968218432ae5e5eb87 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Tue, 22 Sep 2020 14:44:30 -0400 Subject: [PATCH 3/5] state-contents.txt: try to document circ-build-timeout fields This should also get feedback from @mikeperry. Closes #40136 --- doc/state-contents.txt | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/doc/state-contents.txt b/doc/state-contents.txt index 58c77bc292..7c3ba99659 100644 --- a/doc/state-contents.txt +++ b/doc/state-contents.txt @@ -91,9 +91,24 @@ Recognized fields are: TotalBuildTimes CircuitBuildAbandonedCount CircuitBuildTimeBin - BuildTimeHistorgram - XXXX writeme. + These fields are used by the Circuit Build Timeout code, which + tries to learn what times are reasonable for circuit construction, + so that it can reject circuits that take too long to build. + + CircuitBuildTimeBin is a count of circuits that were build + successfully in some timeframe. This entry can repeat; each of + these represents some bar on a histogram. The first integer is a + number of milliseconds; it tells the position of the center of the + histogram bin on the time axis. The second number is a count of + circuits in that bin. + + CircuitBuildTimeAbandonedCount is a count of circuits that we + simply gave up on building because they were taking far too long. + + TotalBuildTimes is the number of circuit build times that we + observed in order to build the above measurements fields. If it + reaches a cap, then older measurements get thrown away. Guard [key=value] [key=value]... From ce1dfb46fcf7f1cf46235ac953a8ef6e33d5e85d Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Tue, 22 Sep 2020 14:46:41 -0400 Subject: [PATCH 4/5] Add a changes file for 40136 (state-contents.txt updates) --- doc/ticket40136 | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 doc/ticket40136 diff --git a/doc/ticket40136 b/doc/ticket40136 new file mode 100644 index 0000000000..cd1c0682f6 --- /dev/null +++ b/doc/ticket40136 @@ -0,0 +1,3 @@ + o Documentation: + - Update doc/state-contents.txt to more accurately explain the + current contents of a Tor state file. Closes ticket 40136. From d0b83c7423935ad9fd3bcb9b7b5faf732a2c4066 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Tue, 22 Sep 2020 16:51:26 -0400 Subject: [PATCH 5/5] state-contents: correcct the description of unusable_circuits --- doc/state-contents.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/state-contents.txt b/doc/state-contents.txt index 7c3ba99659..772e108377 100644 --- a/doc/state-contents.txt +++ b/doc/state-contents.txt @@ -198,7 +198,7 @@ Recognized fields are: after having been built, but before sending traffic. "unusable_circuits" is a count of circuits that we - launched as path-bias probes but which failed to build, + built, but where streams or probes but which failed, or which encountered questionable errors. "timeouts" is a count of circuits that encountered a