Commit Graph

683 Commits

Author SHA1 Message Date
Nick Mathewson
65e63e7461 annotate_ifdef_directives: remove some cases of double negation
This change should reduce the number of cases where we say
"/* !(!defined(foo)) */" .

This only does cases where we can use a regex to make sure that the
simplification is guaranteed to be correct.  Full boolean
simplification would require this script to parse C, and nobody
wants that.
2019-09-26 15:52:45 -04:00
Nick Mathewson
ffd857bbe1 Merge remote-tracking branch 'tor-github/pr/1308' 2019-09-26 14:36:37 -04:00
George Kadianakis
7157d80c6f Merge branch 'tor-github/pr/1337' 2019-09-25 14:19:49 +03:00
George Kadianakis
6677eae579 Merge branch 'tor-github/pr/1355' 2019-09-25 14:17:03 +03:00
George Kadianakis
e77ad795c6 Merge branch 'tor-github/pr/1322' 2019-09-25 14:13:44 +03:00
George Kadianakis
99f75373de Merge branch 'tor-github/pr/1309' 2019-09-25 14:12:42 +03:00
teor
22c33ea7f7
practracker: Fix a typo in the usage message
Fixes bug 31828; bugfix on 0.4.2.1-alpha.
2019-09-24 14:37:43 +10:00
AmreshVenugopal
3e45260594 fix: Ticket #31589
- The function `decrypt_desc_layer` has a cleaner interface.
- `is_superencrypted_layer` changed from `int` -> `bool`

[ticket details](https://trac.torproject.org/projects/tor/ticket/31589)

add(changes/*): changes file
fix(src/features/hs): is_superencrypted changed from `int` -> `bool`
fix(changes/ticket31589): header
add(changes/ticket31589): subsystem(onion services) to change
2019-09-23 23:52:01 +05:30
Nick Mathewson
16890839d3 annotate_ifdef_directives: obey an 80-column line-limit
If we would add a comment making a line longer than 80 columns,
instead truncate the variable portion of the comment until it just
fits into 80 columns, with an ellipsis.
2019-09-23 08:48:53 -04:00
Nick Mathewson
f36e743e5d annotate_ifdef_directives: introduce a function to make commented lines
No functional change in this commit.
2019-09-23 08:48:53 -04:00
Nick Mathewson
0b367f3386 Add comments to annotate_ifdef_directives 2019-09-23 08:48:53 -04:00
Nick Mathewson
2e1989dacc Remove practracker exception for microdesc parsing function 2019-09-19 14:37:24 -04:00
Nick Mathewson
387cfccee4 Add a coccinelle script to look for {inc,dec}rements in log_debug
We want to forbid this pattern since, unlike the other log_*()
macros, log_debug() conditionally evaluates its arguments only if
debug-level logging is enabled.  Thus, a call to
    log_debug("%d", x++);
will only increment x if debugging logs are enabled, which is
probably not what the programmer intended.

One bug caused by this pattern was #30628.

This script detects log_debug( ) calls with any of E++, E--, ++E,
or --E in their arguments, where E is an arbitrary expression.

Closes ticket 30743.
2019-09-18 13:43:42 -04:00
Nick Mathewson
1c67d261d5 practracker: Integration test for --list-overbroad
Make sure that we list overbroad exceptions, and nothing else
2019-09-18 10:28:33 -04:00
Nick Mathewson
d1ffac3f8a practracker: An exception is "used" even when it is violated.
The purpose of tracking whether an exception is used is so that we
can tell whether it is overbroad.  This means that an _underbroad_
exception is still a used one.  Fixes bug 31338.
2019-09-18 10:28:33 -04:00
Nick Mathewson
25d55fa01e Practracker: do not list problems when told to --list-overbroad. 2019-09-18 10:28:33 -04:00
Daniel Pinto
619178bb3a Fix add_c_file.py script to support paths starting with ./ #31336 2019-09-16 22:57:20 +01:00
George Kadianakis
8f2ab45229 Merge branch 'tor-github/pr/1313' 2019-09-16 19:45:20 +03:00
George Kadianakis
9877423736 Merge branch 'tor-github/pr/1312' 2019-09-16 19:44:33 +03:00
Nick Mathewson
25ed698fb8 Add some more of our trickier macros to tor-coccinelle.h
Note that this header file behaves a bit strangely.  It is used by
coccinelle just for the purpose of knowing how to parse
difficult-to-parse stuff.  It doesn't need to produce good C -- just
grammatical C.
2019-09-11 18:45:52 -04:00
Nick Mathewson
c6191983e9 Add a script to tell whether a file can be perfectly parsed by spatch
spatch can let us know whether a file has parsed "perfectly" or
not.  The more perfect it parses, the likelier any semantic patches
are to apply.  I've used this script to identify problem areas in
our code.
2019-09-11 18:44:10 -04:00
Nick Mathewson
9a101c2c0f Add a script to run spatch with appropriate arguments
It's a bit tricky to remember the right incantation to get the
proper include paths and incantations for coccinelle, but without
it, coccinelle is less effective at parsing our C.
2019-09-11 18:43:16 -04:00
Nick Mathewson
e7565855c0 Merge branch 'ticket31578' into ticket31578_merged 2019-09-11 08:59:29 -04:00
teor
af12b6fd29
scripts: Make checkShellScripts.sh error exit if it can't find src
Closes 31679.
2019-09-10 12:24:08 +10:00
teor
8c37bf738d
scripts: Allow checkShellScripts.sh to be run from its parent directory
... on systems that don't have realpath.

Part of 31679.
2019-09-10 12:23:12 +10:00
teor
f0044e44f2
scripts: Make the git scripts show usage messages
All of the git scripts now have usage messages on:
* a new -h option, and
* usage errors.

Closes 31677.
2019-09-10 12:14:49 +10:00
teor
785391a934
scripts: reformat the comments in git-pull-all.sh for consistency
Preparation for 31677.
2019-09-10 12:14:00 +10:00
teor
97da440f9e
scripts: Improve a usage message in git-push-all.sh
Part of 31677.
2019-09-10 12:12:04 +10:00
teor
51b792b000
scripts: Allow git-push-all.sh to be run from any directory
Closes 31678.
2019-09-10 12:10:28 +10:00
David Goulet
796a9b37ea Merge branch 'tor-github/pr/1264' 2019-09-09 14:53:12 -04:00
Nick Mathewson
a642a4cbd7 Merge branch 'ticket30924_042_04_squashed' into ticket30924_042_04_squashed_merged 2019-09-09 11:10:53 -04:00
David Goulet
a8a1ea4e0e practracker: Make you happy funny script
Signed-off-by: David Goulet <dgoulet@torproject.org>
2019-09-09 11:06:51 -04:00
Nick Mathewson
bc4ddbf4ac New practracker exceptions for dependency violations in headers
I've done this manually, since I don't want to override the existing
exceptions in this branch.
2019-09-05 16:17:50 -04:00
Nick Mathewson
318de94e49 Fix a bug in practracker's handling of .may_include in headers
I was expecting our filter code to work in a way it didn't.  I
thought that saying that DependencyViolation applied to "*" would
hit all of the files -- but actually, "*" wasn't implemented.  I had
to say "*.c" and "*.h"
2019-09-05 16:16:20 -04:00
teor
5c2941e69f
shellcheck: Add shellcheck to the pre-commit hook
* Move the shellcheck script from the Makefile to its own script file
* Reformat the shellcheck script so it's easier to read and modify
* Call the shellcheck script from the pre-commit hook

Fixes bug 30967; not in any released version of Tor.
2019-09-05 11:15:26 +10:00
David Goulet
6210c75076 Merge branch 'tor-github/pr/1246' 2019-09-04 08:27:36 -04:00
Nick Mathewson
03040903e6 practracker: handle empty files.
Previously practracker would fail on a file with no lines.
2019-09-02 15:40:35 -04:00
Nick Mathewson
3a2964577e practracker: Fix indentation in util.py
This commit only changes whitespace and removes a temporary comment.
2019-09-02 15:40:35 -04:00
Nick Mathewson
651bbe8a0d Practracker: only consider files under "src/" 2019-09-02 15:40:35 -04:00
Nick Mathewson
e3f7e5e65e practracker: make filename terminology uniform
Previously we often referred to "C files" and "H files", which is
more ambiguous than ".c files" and ".h files".
2019-08-29 09:20:27 -04:00
teor
e155598fe3
scripts/git: fix shellcheck issues in git-push-all.sh
Part of 31314.
2019-08-29 23:13:29 +10:00
teor
340ff7f5f8
scripts/git: fix an env var typo in git-merge-forward.sh
Part of 31314.
2019-08-29 22:58:53 +10:00
teor
d0e31b4d1f
scripts/git: Quote shell arguments where possible
Most shell arguments should be quoted to avoid mistakes.

But since all branch names are hard-coded, or supplied by the script user,
we don't need to be too concerned about command injection.

Quoting all shell arguments would take a major refactor.
(Probably using arrays.)

Part of 31314.
2019-08-29 22:53:44 +10:00
teor
664e6a392e
scripts/git: Improve usage documentation for merge-forward and push-all
Part of 31314.
2019-08-29 22:52:21 +10:00
teor
b47b71ad2f
scripts/git: Let git-push-all.sh skip unchanged test branches
Skip test branches that are the same as remote maint/release/master
branches.

Add a TOR_PUSH_SAME and -s argument to git-push-all.sh to change this
default.

Part of 31314.
2019-08-29 22:50:47 +10:00
teor
70387054b9
scripts/git: Make the git push command and args configurable
TOR_GIT_PUSH provides the git push command and default arguments.

Also fix handling of git-push-all.sh script arguments and arguments that
are passed through to $TOR_GIT_PUSH, using a "--" argument as a separator.

Fix on 29879.
2019-08-29 22:50:37 +10:00
teor
15782758c7
scripts/git: Allow git-merge-forward.sh to re-use existing test branches
Add a -u argument to git-merge-forward.sh, so that the script can
re-use existing test branches after a merge failure and fix.

Part of 31314.
2019-08-29 22:47:50 +10:00
teor
667311ebbd
scripts/git: Add test branch support to the git scripts
Add a -t <test-branch-prefix> argument to git-merge-forward.sh and
git-push-all.sh, which makes these scripts create, merge forward, and
push test branches.

Add a -r <remote-name> argument to git-push-all.sh, so the script can
push test branches to a personal remote.

Closes ticket 31314.
2019-08-29 22:47:43 +10:00
teor
c98724b022
scripts/git: Cleanup and fix minor git scripts issues
Fix typos, clean up formatting, rewrite some comments, add headings.

Preparation for 31314.
2019-08-29 22:42:37 +10:00
Nick Mathewson
c710518825 Add integration tests for new practracker features
These tests check our .may_include checking, and our header file
checking.

They do not pass yet: we have a bug in our filtering code.
2019-08-26 12:28:46 -04:00
George Kadianakis
859514d477 Merge branch 'tor-github/pr/1263' 2019-08-26 17:35:34 +03:00
Nick Mathewson
eff95429fd Merge remote-tracking branch 'tor-github/pr/1241' 2019-08-26 10:15:25 -04:00
Nick Mathewson
ca667b9a8a Fix/suppress shellcheck warnings in git-push-all.sh
(I've chosen to suppress some instances rather than 'fix' them,
since the fix would require arrays or major refactoring.)

Fixes bug 31519; bug not in any released Tor.
2019-08-26 09:58:38 -04:00
David Goulet
24bc2cd7b5 Merge branch 'tor-github/pr/1254' 2019-08-26 09:38:24 -04:00
Nick Mathewson
7a45c9c1d2 Merge remote-tracking branch 'tor-github/pr/1215' 2019-08-24 16:30:55 -04:00
Nick Mathewson
2780cbb9cb Merge branch 'ticket30935' into ticket30935_merged 2019-08-22 17:25:35 -04:00
David Goulet
d475d7c2fb Merge branch 'tor-github/pr/1244' 2019-08-22 17:10:22 -04:00
Nick Mathewson
5b3741e05a Document new practracker violation types, and add a practracker readme
Closes ticket 31476.
2019-08-21 10:17:26 -04:00
Nick Mathewson
cc48eff2d3 Merge branch 'ticket31176' into ticket31176_merged 2019-08-21 09:46:20 -04:00
Nick Mathewson
edf5a327c5 Merge branch 'ticket30914' into ticket30914_merged 2019-08-20 13:57:05 -04:00
teor
bdcccb9776
scripts/git: Remove a duplicate practracker call from the pre-push hook
The pre-push hook already calls the pre-commit hook, which calls
practracker.

Also update the script comments to avoid similar issues in future.

Fixes bug 31462; bugfix on 0.4.1.1-alpha.
2019-08-20 12:21:02 +10:00
Nick Mathewson
29bd43ab03 Merge remote-tracking branch 'tor-github/pr/1224' 2019-08-19 11:26:58 -04:00
Nick Mathewson
e7aabfce61 Merge remote-tracking branch 'tor-github/pr/1235' 2019-08-19 11:10:21 -04:00
George Kadianakis
e411fcabd7 Merge branch 'tor-github/pr/1220' 2019-08-19 18:02:44 +03:00
George Kadianakis
b257e8e553 Merge remote-tracking branch 'mike/bug31356+logs-rebased2' into bug31356_final 2019-08-19 17:53:00 +03:00
David Goulet
81e37b81f1 practracker: Make exceptions.txt happy
Signed-off-by: David Goulet <dgoulet@torproject.org>
2019-08-19 09:47:52 -04:00
Mike Perry
6deba34fee Practracker 2019-08-12 14:06:08 -05:00
Neel
64d2133a61 Move the connection_edge_process_relay_cell() assignments out of if statements in circuit_receive_relay_cell() 2019-08-08 13:05:46 -04:00
Nick Mathewson
71e5af0221 pre-push hook: Only run practracker when a special file is present
Closes ticket 30979.
2019-08-08 11:36:05 -04:00
teor
d6202d3128
scripts/git: add TOR_PUSH_DELAY to git-push-all.sh
Add a TOR_PUSH_DELAY variable to git-push-all.sh, which makes the script
push master and maint branches with a delay between each branch. These
delays trigger the CI jobs in a set order, which should show the most
likely failures first.

Also:
* make pushes atomic by default, and
* make the script pass any command-line arguments to git push.

Closes ticket 29879.
2019-08-08 18:59:44 +10:00
Nick Mathewson
35cfe2e776 Merge remote-tracking branch 'tor-github/pr/1208' 2019-08-07 09:49:49 -04:00
David Goulet
2b225aaa77 practracker: Make it happy after rebase
Signed-off-by: David Goulet <dgoulet@torproject.org>
2019-08-06 08:01:26 -04:00
David Goulet
c5b00c5a51 hs-v3: Add consensus parameters for DoS defenses
Part of #15516

Signed-off-by: David Goulet <dgoulet@torproject.org>
2019-08-06 07:58:14 -04:00
George Kadianakis
d7afdb3b0f Merge branch 'tor-github/pr/1195' 2019-08-06 14:51:47 +03:00
Nick Mathewson
0f4b245b20 update exceptions file for depencency violations 2019-08-05 17:48:06 -04:00
Nick Mathewson
a5971d732e Move include-violation checking into its own function. 2019-08-05 17:48:06 -04:00
Nick Mathewson
2a3c727dfe Make includes interface more like the rest of practracker
Everything else assumes that somebody else will open the file for it.
2019-08-05 17:48:06 -04:00
Nick Mathewson
6b26281b50 practracker: a violation of a .may_include rule is now a problem.
We treat "0" as the expected number, and warn about everything
else.  The problem type is "dependency-violation".
2019-08-05 17:48:06 -04:00
Nick Mathewson
720951f056 Teach include-checker about advisory rules
A .may_includes file can be "advisory", which means that some
violations of the rules are expected.  We will track these
violations with practracker, not as automatic errors.
2019-08-05 17:04:00 -04:00
Nick Mathewson
9abbde2c24 Update pre-commit hook to find checkIncludes in its new location
Also add a temporary script to redirect the hook, if people don't
upgrade for a bit.
2019-08-05 14:12:39 -04:00
Nick Mathewson
6fb74753c2 Move checkIncludes inside practracker
Update the makefile accordingly.
2019-08-05 14:10:40 -04:00
Nick Mathewson
9eb12dde18 checkIncludes: add a real main function and CLI 2019-08-05 13:40:59 -04:00
Nick Mathewson
3f4e89a7ab checkIncludes: refactor to use error-iteration style
This makes checkIncludes match practracker more closely, and lets us
eliminate a global.
2019-08-05 13:40:59 -04:00
Nick Mathewson
47d9bcfef8 checkIncludes: Separate file-handling from rule-handling
This is our shift from directory-at-a-time processing to
file-at-a-time processing.
2019-08-05 13:40:59 -04:00
Nick Mathewson
65a69f861e checkIncludes.py: extract topological sort code
Our topological sort code really deserves a function of its own.

Additionally, don't print from inside the topological sort code:
instead, return a result, and let the caller print it.
2019-08-05 13:40:59 -04:00
Nick Mathewson
3f35ac772b checkIncludes: introduce rules-file caching.
We'll want this so that we can have each file evaluated
independently, rather than a directory at a time.
2019-08-05 11:35:13 -04:00
Nick Mathewson
475749351d Move the executable part of checkIncludes.py inside an if block.
I'll want to make this block into a series of functions in a
subsequent commit, but I'm doing this separately to get the
indentation change out of the way.

This branch will end up with making checkIncludes.py an integrated
part of practracker, for ticket 31176.
2019-08-05 11:30:22 -04:00
George Kadianakis
ce477da8a7 Ignore regular cells in padding circuits.
Padding circuits were regular cells that got closed before their padding
machine could finish. This means that they can still receive regular cells from
their past life, but they have no way or reason to answer them anymore. Hence
let's ignore them before they even get to the proper subsystems.
2019-08-05 18:03:23 +03:00
Nick Mathewson
d5ccd0fa84 Add exceptions for current .h practracker problems. 2019-08-05 10:32:39 -04:00
Nick Mathewson
9a1e9b1d6c Teach practracker about .h files
I'm using 500 as a file size limit, and 15 as an include limit.
This affects comparatively few files, but I think they are the worst
ones.

Closes ticket 31175.
2019-08-05 10:31:02 -04:00
Nick Mathewson
1440c2cb34 Adjust test_practracker.sh to work on windows
The required change is to ignore trailing CRs when diffing files.
2019-08-01 15:43:54 -04:00
Nick Mathewson
fa60fee8d5 practracker: Add unit tests to test script, and test script to makefile
This makes all of the practracker tests get run by make check, and
hence by our CI.

Closes ticket 31304.
2019-08-01 14:01:43 -04:00
Nick Mathewson
5d98b54725 Port practracker unit tests to python 3 2019-08-01 14:00:48 -04:00
Nick Mathewson
e57209dc32 Regenerate the practracker exceptions.txt file 2019-08-01 10:25:42 -04:00
Nick Mathewson
a4e4896e66 practracker: restore exceptions.txt header when running --regen 2019-08-01 10:25:20 -04:00
David Goulet
c4864de573 Merge branch 'tor-github/pr/1177' 2019-08-01 10:18:10 -04:00
Nick Mathewson
19536fd18d practracker: replaces "overstrict" with "overbroad"
I had the logic reversed here.
2019-08-01 09:35:33 -04:00
Nick Mathewson
3221dc1b32 Lower check of TOR_DISABLE_PRACTRACKER
Since we sometimes call practracker directly, that's where we should
check the TOR_DISABLE_PRACTRACKER envvar.
2019-08-01 08:40:56 -04:00
Nick Mathewson
8d3f3e5d30 Practracker: add an integration test.
This test runs practracker with a set of 0 thresholds, to make sure
that it enumerates all its values right.  It tries running with an
empty exceptions file, and with an exceptions file that covers
_some_ of the data, and it makes sure that the outputs are as expected.
2019-07-30 12:14:46 -04:00
Nick Mathewson
a79e2c2975 practracker: better warning/regen handling
Now that there is only one toplevel place where we print problems,
we can redirect just that one print to a file when we are
regenerating the exceptions.txt file.  Previously we redirected
sys.stdout, which is naughty, and forced us to send warnings (and
warnings alone) to stderr.
2019-07-30 11:54:05 -04:00