Commit Graph

24309 Commits

Author SHA1 Message Date
Nick Mathewson
d8c129a11a Avoid all needless memory copies when computing consensus diffs.
Previously, we operated on smartlists of NUL-terminated strings,
which required us to copy both inputs to produce the NUL-terminated
strings.  Then we copied parts of _those_ inputs to produce an
output smartlist of NUL-terminated strings.  And finally, we
concatenated everything into a final resulting string.

This implementation, instead, uses a pointer-and-extent pattern to
represent each line as a pointer into the original inputs and a
length.  These line objects are then added by reference into the
output. No actual bytes are copied from the original strings until
we finally concatenate the final result together.

Bookkeeping structures and newly allocated strings (like ed
commands) are allocated inside a memarea, to avoid needless mallocs
or complicated should-I-free-this-or-not bookkeeping.

In my measurements, this improves CPU performance by something like
18%.  The memory savings should be much, much higher.
2017-03-16 14:42:56 -04:00
Nick Mathewson
52fa6bb947 Swap memory allocation strategy for lists of lines for diffs
Now we use a single allocation block for all the lines, rather than
calling strdup on them one at a time.  This should help performance
a tiny bit.
2017-03-16 14:42:56 -04:00
Nick Mathewson
b36e311eab Fill in the missing documentation on the new functions 2017-03-16 14:42:56 -04:00
Nick Mathewson
12d428aaff Prop140: Fix a crash bug.
Found while fuzzing: this could occur if we tried to copy a
nonexistent "line 0" while applying a diff.
2017-03-16 14:42:56 -04:00
Nick Mathewson
dd92579b63 Add fuzzers for consensus diff backend code
This takes two fuzzers: one which generates a diff and makes sure it
works, and one which applies a diff.

So far, they won't crash, but there's a bug in my
string-manipulation code someplace that I'm having to work around,
related to the case where you have a blank line at the end of a
file, or where you diff a file with itself.
2017-03-16 14:42:55 -04:00
Nick Mathewson
653c6d129e Make consensus diff sha3 operations mockable.
(We'll want this for fuzzing)
2017-03-16 14:40:33 -04:00
Nick Mathewson
6a36e5ff3b String-based API for consensus diffs.
Also, add very strict split/join functions, and totally forbid
nonempty files that end with somethig besides a newline.  This
change is necessary to ensure that diff/apply are actually reliable
inverse operations.
2017-03-16 14:39:54 -04:00
Nick Mathewson
eff9fbd17d Fix an abstraction violation.
Don't alias the insides of smartlist_t; that way lies madness.
2017-03-16 14:38:29 -04:00
Nick Mathewson
69b3e11e59 Use "const" in consdiff.[ch] 2017-03-16 14:38:29 -04:00
Nick Mathewson
3647751c2a prop140: Use sha3-256, not sha2-256
This is a protocol update from recent prop140 changes.

Also, per #21673, we need to check the entire document, including
signatures.
2017-03-16 14:38:29 -04:00
Nick Mathewson
d6594bfa8c Add a couple more cases to reach 100% coverage. 2017-03-16 14:38:29 -04:00
Nick Mathewson
c86e77ac20 Cover two more failing cases with unit tests 2017-03-16 14:38:29 -04:00
Nick Mathewson
e1418c09fc Fix an unreachable memory leak.
Also add a missing newline.
2017-03-16 14:38:29 -04:00
Nick Mathewson
f193b666cd Remove digest[12]_hex 2017-03-16 14:38:29 -04:00
Nick Mathewson
c6046f4db8 Tweak&test log messages on apply_diff 2017-03-16 14:38:29 -04:00
Nick Mathewson
5766eed38f Fixes when applying diffs: Allow 2-line diffs, fix bogus free
The 2-line diff changs is needed to make the unit tests actually
test the cases that they thought they were testing.

The bogus free was found while testing those cases
2017-03-16 14:38:29 -04:00
Nick Mathewson
ab1fd85c99 Mark some warnings as bugs, and as (hopefully) unreachable. 2017-03-16 14:38:28 -04:00
Nick Mathewson
bb536a2e73 Check for expected warnings in apply_ed_diff 2017-03-16 14:38:28 -04:00
Nick Mathewson
06017f35e8 Fix some logging on failed apply_ed_diff 2017-03-16 14:38:28 -04:00
Nick Mathewson
97620cf18f No need to end a log message with newline. 2017-03-16 14:38:28 -04:00
Nick Mathewson
05901f2c6d test operator cleanup 2017-03-16 14:38:28 -04:00
Nick Mathewson
687df259c6 Enforce correct log messages on diff generation failure tests 2017-03-16 14:38:28 -04:00
Nick Mathewson
ccb789fe34 Test two more base64cmp cases. 2017-03-16 14:38:28 -04:00
Nick Mathewson
360d043ac7 Use "STATIC" to export consdiff fns for testing
Previously test_consdiff.c just did #include "consdiff.c", which is
not great style, and messes up coverage testing.
2017-03-16 14:38:28 -04:00
Daniel Martí
590ffdb2c9 Consensus diff backend from Daniel Martí GSOC project.
(This commit was extracted by nickm based on the final outcome of
the project, taking only the changes in the files touched by this
commit from the consdiff_rebased branch.  The directory-system
changes are going to get worked on separately.)
2017-03-16 14:38:28 -04:00
Nick Mathewson
ad19f1507a Merge branch 'maint-0.3.0' 2017-03-07 08:08:28 -05:00
Nick Mathewson
552bc39c32 Merge branch 'bug21594_030_squashed' into maint-0.3.0 2017-03-07 08:05:16 -05:00
teor
93ede051c2 Remove delay in hidden service introduction point checks
Make hidden services with 8 to 10 introduction points check for failed
circuits immediately after startup. Previously, they would wait for 5
minutes before performing their first checks.

Fixes bug 21594; bugfix on commit 190aac0eab in Tor 0.2.3.9-alpha.
Reported by alecmuffett.
2017-03-07 08:04:57 -05:00
Nick Mathewson
88b91d7753 Merge remote-tracking branch 'ahf/bugs/20988' 2017-03-06 12:04:58 -05:00
Nick Mathewson
2c8d2cab13 Merge remote-tracking branch 'teor/bug19699' 2017-03-06 11:52:17 -05:00
Nick Mathewson
0a54e5d148 Fix a function name in a comment. Closes 21580 2017-03-06 11:27:50 -05:00
Nick Mathewson
00d1093daf Merge branch 'feature21598_squashed' 2017-03-04 23:22:46 -05:00
teor
f24638aa49 Log a message when a hidden service has fewer intro points than expected
Closes ticket 21598.
2017-03-04 23:22:34 -05:00
Nick Mathewson
958ec0f5f8 Merge branch 'bug21599_squashed' 2017-03-04 23:16:29 -05:00
teor
684778e705 Simplify hidden service descriptor creation
Use an existing flag to check if an introduction point is established.

Cleanup after 21596.

Fixes bug 21599; bugfix on 0.2.7.2-alpha.
2017-03-04 23:15:55 -05:00
Nick Mathewson
fe17955661 Merge remote-tracking branch 'teor/bug21596_030' 2017-03-04 23:10:40 -05:00
Nick Mathewson
3a1cba7d90 Merge branch 'maint-0.3.0' 2017-03-04 20:24:02 -05:00
Nick Mathewson
333d5d0f2a Merge remote-tracking branch 'teor/bug21576_029_v2' into maint-0.3.0 2017-03-04 20:23:38 -05:00
Nick Mathewson
c04209c31b Forward-port changelogs and releaesenotes to master.
These are from today's stable releases in 0.2.4 through 0.2.8
2017-03-03 15:00:33 -05:00
teor
3e2d06bd3d
Make hidden services always check for failed intro point connections
Previously, they would stop checking when they exceeded their intro point
creation limit.

Fixes bug 21596; bugfix on commit d67bf8b2f2 in Tor 0.2.7.2-alpha.
Reported by alecmuffett.
2017-03-02 15:57:58 +11:00
teor
e0486c9371
Make hidden services always check for failed intro point connections
Previously, they would stop checking when they exceeded their intro point
creation limit.

Fixes bug 21596; bugfix on commit d67bf8b2f2 in Tor 0.2.7.2-alpha.
Reported by alecmuffett.
2017-03-02 15:34:45 +11:00
teor
4b5cdb2c30
Fix a crash when a connection tries to open just after it has been unlinked
Fixes bug 21576; bugfix on Tor 0.2.9.3-alpha.
Reported by alecmuffett.
2017-03-02 11:10:30 +11:00
Nick Mathewson
9af76a9583 Forward-port 0.3.0.4-rc changelog 2017-03-01 15:39:52 -05:00
Nick Mathewson
bc052d4978 Merge branch 'maint-0.3.0'
"ours" to avoid version bump
2017-03-01 15:39:01 -05:00
Nick Mathewson
528dcd113c Bump to 0.3.0.4-rc-dev 2017-03-01 15:38:36 -05:00
Alexander Færøy
ae91e64fa8
Reset buf after each succesful test to avoid artifacts.
This patch resets `buf` in test_util_fgets_eagain() after each succesful
ivocation to avoid stray artifacts left in the buffer by erroneous
tor_fgets() calls.
2017-03-01 21:26:27 +01:00
Alexander Færøy
3dca5a6e71
Use tor_fgets() instead of fgets().
This patch changes our use of fgets() to tor_fgets() for more consistent
error handling across different versions of the C library.
2017-03-01 21:26:27 +01:00
Alexander Færøy
498287b3c3
Add compatibility function for fgets(3).
This patch adds the `tor_fgets()` function to our compatibility layer.

`tor_fgets()` adds an additional check for whether the error-bit have
been enabled for the given file stream, if that is the case and `errno`
is set to `EAGAIN` we make sure that we always return NULL.

Unfortunately `fgets(3)` behaves differently on different versions of
the C library.

See: https://bugs.torproject.org/21416
See: https://bugs.torproject.org/20988
2017-03-01 21:26:26 +01:00
Nick Mathewson
7d3883d084 Merge branch 'maint-0.3.0' 2017-03-01 15:11:23 -05:00
Nick Mathewson
5298ab5917 Restore correct behavior of 0.3.0.4-rc with bridges+ipv6-min
In that chutney test, the bridge client is configured to connect to
the same bridge at 127.0.0.1:5003 _and_ at [::1]:5003, with no
change in transports.

That meant, I think, that the descriptor is only assigned to the
first bridge when it arrives, and never the second.
2017-03-01 15:02:16 -05:00