Commit Graph

3741 Commits

Author SHA1 Message Date
Alexander Færøy
ca7a2ecc51 Avoid breaking the event loop prematurely.
This patch makes sure that we terminate the event loop from the event
loop timer instead of directly in the process' exit handler. This allows
us to run the event loop an additional time to ensure that the SleepEx()
call on Windows is called and the data from stdout/stderr is delivered
to us.

Additionally we ensure that we don't try to read or write data from a
Unix process that have been terminated in the main loop, since its file
descriptors are closed at that time.

See: https://bugs.torproject.org/28179
2018-12-18 13:35:29 -05:00
Nick Mathewson
e969d9c6b4 Merge branch 'ticket28179_squashed' into ticket28179_squashed_merged 2018-12-17 16:41:01 -05:00
Alexander Færøy
cacdd29087 Use const char * instead of char * for line parameter for process callbacks.
This patch changes the type definition of the process callbacks to use
`const char *` instead of `char *`.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
ec2ae3ed8b Change EVENT_TRANSPORT_LOG to EVENT_PT_LOG.
This patch changes our EVENT_TRANSPORT_LOG event to be EVENT_PT_LOG. The
new message includes the path to the PT executable instead of the
transport name, since one PT binary can include multiple transport they
sometimes might need to log messages that are not specific to a given
transport.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
5585cbd08f Change the Process exit_callback to return bool.
This patch changes our process_t's exit_callback to return a boolean
value.  If the returned value is true, the process subsystem will call
process_free() on the given process_t.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
bc6983afed Use run_main_loop_until_done() for process_t tests.
This patch changes the slow process_t tests to use
run_main_loop_until_done() instead of do_main_loop() since
do_main_loop() initializes a lot of subsystem callbacks that we don't
need to run in our tests.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
6e508e9eb4 Fix tests on kqueue() based platforms.
This patch disables fork()'ing of the slow process tests. This fixes the
tests on the MacOS and other kqueue() based platforms.

Without this patch the main loop exits eearly with EBADF as error.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
ccc1963890 Move remaining code from subprocess.{h,c} to more appropriate places.
This patch moves the remaining code from subprocess.{h,c} to more
appropriate places in the process.c and process_win32.c module.

We also delete the now empty subprocess module files.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
f7d13425fc Delete old process_handle_t code.
This patch removes the old process_handle_t code. Everything should by
now be using the process_t interface.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
289ed0849d Add tests for process environment functionality of process_t.
This patch adds tests for the process_environment_t interaction in
process_t.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
9b6a10a26f Add slow test for process_t for main loop interaction.
This patch adds test cases for process_t which uses Tor's main loop.
This allows us to test that the callbacks are actually invoked by the
main loop when we expect them.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
e3ceaebba2 Add support for logging messages from pluggable transports.
This patch adds support for the "LOG" protocol message from a pluggable
transport. This allows pluggable transport developers to relay log
messages from their binary to Tor, which will both emit them as log
messages from the Tor process itself, but also pass them on via the
control port.

See: https://bugs.torproject.org/28180
See: https://bugs.torproject.org/28181
See: https://bugs.torproject.org/28182
2018-12-17 16:39:28 -05:00
Alexander Færøy
bfb94dd2ca Use process_t for managed proxies.
This patch makes the managed proxy subsystem use the process_t data
structure such that we can get events from the PT process while Tor is
running and not just when the PT process is being configured.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
89393a77e5 Add process_get_pid() to the Process subsystem.
This patch adds support for getting the unique process identifier from a
given process_t. This patch implements both support for both the Unix
and Microsoft Windows backend.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
bb784cf4f3 Add Windows backend for the Process subsystem.
This patch adds support for Microsoft Windows in the Process subsystem.

Libevent does not support mixing different types of handles (sockets,
named pipes, etc.) on Windows in its core event loop code. This have
historically meant that Tor have avoided attaching any non-networking
handles to the event loop. This patch uses a slightly different approach
to roughly support the same features for the Process subsystem as we do
with the Unix backend.

In this patch we use Windows Extended I/O functions (ReadFileEx() and
WriteFileEx()) which executes asynchronously in the background and
executes a completion routine when the scheduled read or write operation
have completed. This is much different from the Unix backend where the
operating system signals to us whenever a file descriptor is "ready" to
either being read from or written to.

To make the Windows operating system execute the completion routines of
ReadFileEx() and WriteFileEx() we must get the Tor process into what
Microsoft calls an "alertable" state. To do this we execute SleepEx()
with a zero millisecond sleep time from a main loop timer that ticks
once a second.  This moves the process into the "alertable" state and
when we return from the zero millisecond timeout all the outstanding I/O
completion routines will be called and we can schedule the next reads
and writes.

The timer loop is also responsible for detecting whether our child
processes have terminated since the last timer tick.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
2e957027e2 Add Unix backend for the Process subsystem.
This patch adds the Unix backend for the Process subsystem. The Unix
backend attaches file descriptors from the child process's standard in,
out and error to Tor's libevent based main loop using traditional Unix
pipes. We use the already available `waitpid` module to get events
whenever the child process terminates.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
35509978dd Add new Process subsystem.
This patch adds a new Process subsystem for running external programs in
the background of Tor. The design is focused around a new type named
`process_t` which have an API that allows the developer to easily write
code that interacts with the given child process. These interactions
includes:

- Easy API for writing output to the child process's standard input
  handle.
- Receive callbacks whenever the child has output on either its standard
  output or standard error handles.
- Receive callback when the child process terminates.

We also support two different "protocols" for handling output from the
child process. The default protocol is the "line" protocol where the
process output callbacks will be invoked only when there is complete
lines (either "\r\n" or "\n" terminated). We also support the "raw"
protocol where the read callbacks will get whatever the operating system
delivered to us in a single read operation.

This patch does not include any operating system backends, but the Unix
and Windows backends will be included in separate commits.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Nick Mathewson
69264f96f3 Merge branch 'dormant_persist_squashed' 2018-12-13 08:26:10 -05:00
Nick Mathewson
e3b7fd2a81 Unit tests for back-end functions for persistent dormant state 2018-12-13 08:25:54 -05:00
Nick Mathewson
845e8dbe59 Fuzzing module for various string operations, currently focusing on
encoding and decoding.

There are bunches of places where we don't want to invest in a full
fuzzer, but we would like to make sure that some string operation
can handle all its possible inputs.  This fuzzer uses the first byte
of its input to decide what to do with the rest of the input.  Right
now, all the possibilities are decoding a string, and seeing whether
it is decodeable.  If it is, we try to re-encode it and do the whole
thing again, to make sure we get the same result.

This turned up a lot of bugs in the key-value parser, and I think it
will help in other cases too.

Closes ticket 28808.
2018-12-12 11:07:08 -05:00
Nick Mathewson
f0a8664677 Add code to parse K=V lines into config_line_t format.
Closes ticket 28755
2018-12-12 11:07:08 -05:00
Nick Mathewson
1eb3719a62 Merge remote-tracking branch 'public/prop298' 2018-12-05 09:43:03 -05:00
Nick Mathewson
1f95e80351 Merge branch 'prop293_squashed' 2018-12-05 09:24:51 -05:00
Nick Mathewson
a2f81b644b Write tests for mark_my_descriptor_dirty_if_too_old() 2018-12-05 09:24:45 -05:00
Nick Mathewson
00509aaafa Merge remote-tracking branch 'tor-github/pr/553' 2018-12-05 08:28:54 -05:00
Nick Mathewson
ca4b86f90a Merge remote-tracking branch 'tor-github/pr/508' 2018-12-05 08:19:02 -05:00
Nick Mathewson
31a6d9f499 Add tests for parsing each routerstatus flag. 2018-12-03 13:40:08 -05:00
Nick Mathewson
92af8e5113 Add a framework for testing set_routerstatus_from_routerinfo().
Additionally, use it to test that is_staledesc is set correctly.

Eventually we'll want to test all the other flags, but I'm aiming
for only adding coverage on the changed code here.
2018-12-03 13:22:23 -05:00
Nick Mathewson
32213fa9ad Keep list of dirauth flags in sync between dirvote.c and fuzz_vrs.c
Suggested by Teor on PR
2018-12-03 12:18:45 -05:00
rl1987
db9ab3754a Print error message we get from socket.connect_ex when it fails 2018-12-03 14:49:33 +02:00
Nick Mathewson
2b2b97484a Merge branch 'ticket27490a_squashed' 2018-12-01 20:32:18 -05:00
Neel Chauhan
ad031b64ce Add regression test for ClientAutoIPv6ORPort 2018-12-01 14:55:57 -05:00
Nick Mathewson
7e9985b75a Merge remote-tracking branch 'tor-github/pr/536' 2018-12-01 11:24:02 -05:00
Nick Mathewson
af9dc12fab Merge branch 'maint-0.3.5' 2018-12-01 11:20:10 -05:00
rl1987
945c4dfda0 Also log a Tor log entry when it has a substring we are waiting for 2018-12-01 11:18:03 -05:00
rl1987
4c4ed413ee 1 ms. resolution for Tor logs 2018-12-01 11:18:03 -05:00
rl1987
0bb25931dc Log everything from tor down to debug loglevel 2018-12-01 11:18:03 -05:00
rl1987
320f5f30b3 In test_rebind.py, log stuff with timestamps 2018-12-01 11:18:03 -05:00
Nick Mathewson
cf3f7753c3 Merge branch 'maint-0.3.5' 2018-12-01 11:15:09 -05:00
Nick Mathewson
e82023d2f7 Merge branch 'maint-0.3.4' into maint-0.3.5 2018-12-01 11:15:09 -05:00
Nick Mathewson
e3a19b1c78 Merge branch 'maint-0.3.3' into maint-0.3.4 2018-12-01 11:15:09 -05:00
David Goulet
a51dad4272 test: Fix a warning underflow in rend_cache/clean
Because the test is adding entries to the "rend_cache" directly, the
rend_cache_increment_allocation() was never called which made the
rend_cache_clean() call trigger that underflow warning:

rend_cache/clean: [forking] Nov 29 09:55:04.024 [warn] rend_cache_decrement_allocation(): Bug: Underflow in rend_cache_decrement_allocation (on Tor 0.4.0.0-alpha-dev 2240fe63feb9a8cf)

The test is still good and valid.

Fixes #28660

Signed-off-by: David Goulet <dgoulet@torproject.org>
2018-11-29 10:02:50 -05:00
Nick Mathewson
c292e505ff Merge remote-tracking branch 'tor-github/pr/539' 2018-11-26 17:22:37 -05:00
Nick Mathewson
7d8e0cc9ab Merge branch 'dormant_v2_squashed' 2018-11-26 16:33:31 -05:00
Nick Mathewson
02843c4a4e Test for check_network_participation_callback() 2018-11-26 16:32:40 -05:00
Nick Mathewson
55512ef022 Test netstatus.c tracking of user participation status 2018-11-26 16:32:40 -05:00
Nick Mathewson
3743f79695 Add options to control dormant-client feature.
The DormantClientTimeout option controls how long Tor will wait before
going dormant.  It also provides a way to disable the feature by setting
DormantClientTimeout to e.g. "50 years".

The DormantTimeoutDisabledByIdleStreams option controls whether open but
inactive streams count as "client activity".  To implement it, I had to
make it so that reading or writing on a client stream *always* counts as
activity.

Closes ticket 28429.
2018-11-26 16:32:40 -05:00
Nick Mathewson
05dee063c8 Emit router families in canonical form
This patch has routers use the same canonicalization logic as
authorities when encoding their family lists.  Additionally, they
now warn if any router in their list is given by nickname, since
that's error-prone.

This patch also adds some long-overdue tests for family formatting.
2018-11-24 16:35:58 -05:00
Nick Mathewson
0a0c612b79 Add a consensus method in which md families get canonicalized.
Implements prop298. Closes ticket 28266.
2018-11-24 12:01:09 -05:00
Nick Mathewson
d29e3a02d5 Add a function to canonicalize nodefamilies per prop298
This is the same as the regular canonical nodefamily format, except
that unrecognized elements are preserved.
2018-11-24 10:53:38 -05:00