Commit Graph

480 Commits

Author SHA1 Message Date
Nick Mathewson
70dd6d07bb Merge branch 'orconn-tracker_squashed' 2018-12-21 14:22:11 -05:00
Nick Mathewson
fd58e5e498 Fix priority on process subsystem level: it uses "net" 2018-12-21 14:12:20 -05:00
Nick Mathewson
ab4395d082 Merge branch 'ticket28847' 2018-12-21 13:26:47 -05:00
Alexander Færøy
2322b56389 Fix typo in time_sys.h. 2018-12-21 13:26:38 -05:00
Alexander Færøy
cf4b3dbd44 Use the subsystem list to initialize and shutdown process module.
This patch makes the process module use the subsystem list for
initializing and shutting down.

See: https://bugs.torproject.org/28847
2018-12-21 13:26:38 -05:00
Taylor Yu
b0f974633a Add LD_BTRACK log domain for bootstrap tracker
Part of ticket 27167.
2018-12-20 18:46:17 -06:00
Nick Mathewson
5c85ba3077 Merge remote-tracking branch 'tor-github/pr/608' 2018-12-20 11:42:26 -05:00
Alexander Færøy
f7e175db57 Forward declare smartlist_t in process.h
This allows other libraries to include process.h without including
the smartlist_t headers first.

See: https://bugs.torproject.org/28847
2018-12-20 14:36:04 +01:00
Alexander Færøy
7762088967 No need to log ordinary EOF conditions as LOG_WARN.
Let's not use log_warn() when a pipe is closed under what should be
considered normal conditions.

See: https://bugs.torproject.org/28179
2018-12-20 13:12:53 +01:00
Alexander Færøy
412fbe9f17 Make example CancelIoEx() code use CancelIo().
This patch changes the CancelIoEx() example code to use CancelIo(),
which is available for older versions of Windows too. I still think the
kernel handles this nicely by sending broken pipes if either side
closes the pipe while I/O operations are pending.

See: https://bugs.torproject.org/28179
2018-12-20 13:11:24 +01:00
Alexander Færøy
f58e597d42 Handle ERROR_BROKEN_PIPE in completion routines.
Handle `ERROR_BROKEN_PIPE` from ReadFileEx() and WriteFileEx() in
process_win32_stdin_write_done() and
process_win32_handle_read_completion() instead of in the early handler.
This most importantmly makes sure that `reached_eof` is set to true when
these errors appears.

See: https://bugs.torproject.org/28179
2018-12-20 13:04:49 +01:00
Alexander Færøy
36e24782f8 Remember to set reached_eof when our handles are reporting errors.
This patch adds some missing calls to set `reached_eof` of our handles
when various error conditions happens or when we close our handle (which
happens at `process_terminate()`.

See: https://bugs.torproject.org/28179
2018-12-20 13:02:22 +01:00
Alexander Færøy
c6e041e3d8 Handle errors even after success from ReadFileEx() and WriteFileEx().
This patch adds some additional error checking after calls to
ReadFileEx() and WriteFileEx(). I have not managed to get this code to
reach the branch where `error_code` is NOT `ERROR_SUCCESS`, but MSDN
says one should check for this condition so we do so just to be safe.

See: https://bugs.torproject.org/28179
2018-12-20 12:57:20 +01:00
Alexander Færøy
44586a89ef Delay checking process for termination until both stdout and stderr are closed.
This patch makes us delay checking for whether we have an exit code
value (via GetExitCodeProcess()) until both stdout and stderr have been
closed by the operating system either by the process itself or by
process cleanup after termination.

See: https://bugs.torproject.org/28179
2018-12-20 12:53:28 +01:00
Alexander Færøy
1d8dcb416c Remember to close the child process' ends of the pipes.
This prevents us from leaking the HANDLE for stdout, stderr, and stdin.

See: https://bugs.torproject.org/28179
2018-12-20 12:47:04 +01:00
Nick Mathewson
bb091da1e7 Merge branch 'ticket28839_v2_squashed' 2018-12-18 18:59:05 -05:00
Alexander Færøy
ca7a2ecc51 Avoid breaking the event loop prematurely.
This patch makes sure that we terminate the event loop from the event
loop timer instead of directly in the process' exit handler. This allows
us to run the event loop an additional time to ensure that the SleepEx()
call on Windows is called and the data from stdout/stderr is delivered
to us.

Additionally we ensure that we don't try to read or write data from a
Unix process that have been terminated in the main loop, since its file
descriptors are closed at that time.

See: https://bugs.torproject.org/28179
2018-12-18 13:35:29 -05:00
Nick Mathewson
8a01f0eaab lib/process may include lib/buf. 2018-12-17 17:58:49 -05:00
Nick Mathewson
4ad59bfbc2 Update location of buffers.h 2018-12-17 17:01:50 -05:00
Nick Mathewson
e969d9c6b4 Merge branch 'ticket28179_squashed' into ticket28179_squashed_merged 2018-12-17 16:41:01 -05:00
Alexander Færøy
c8b8b15f0e Ensure that line_size >= 1 before trying to trim input string.
See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
fab22509d7 Make Windows process event timer API available for dormant interface.
This patch changes the API of the Windows backend of the Process
subsystem to allow the dormant interface to disable the Process event
timer.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
a33a77d9cd Document the format of process_t::arguments.
See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
0d796cce17 Use errno directly if we are not reading/writing from/to a socket.
See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
cacdd29087 Use const char * instead of char * for line parameter for process callbacks.
This patch changes the type definition of the process callbacks to use
`const char *` instead of `char *`.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
5585cbd08f Change the Process exit_callback to return bool.
This patch changes our process_t's exit_callback to return a boolean
value.  If the returned value is true, the process subsystem will call
process_free() on the given process_t.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
22cb3c6ce9 Call close() on stdin/stdout/stderr in process_terminate().
Call close() on all process handles after we have called kill(pid,
SIGTERM).

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
ccc1963890 Move remaining code from subprocess.{h,c} to more appropriate places.
This patch moves the remaining code from subprocess.{h,c} to more
appropriate places in the process.c and process_win32.c module.

We also delete the now empty subprocess module files.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
f7d13425fc Delete old process_handle_t code.
This patch removes the old process_handle_t code. Everything should by
now be using the process_t interface.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
ad4cc89c5d Add "PT" log domain.
See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
b0d268a822 Add process_reset_environment() to the Process subsystem.
This patch adds a new function that allows us to reset the environment
of a given process_t with a list of key/value pairs.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
4f611a1df7 Add process_terminate().
This patch adds support for process termination to the Process
subsystem.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
338137221c Make sure we call process_notify_event_exit() as the last thing in different callbacks.
This patch makes sure that we call process_notify_event_exit() after we
have done any modifications we need to do to the state of a process_t.
This allows application developers to call process_free() in the
exit_callback of the process.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
e982fb1dae Add documentation for the is_socket and error argument of read_to_chunk().
See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
89393a77e5 Add process_get_pid() to the Process subsystem.
This patch adds support for getting the unique process identifier from a
given process_t. This patch implements both support for both the Unix
and Microsoft Windows backend.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
bb784cf4f3 Add Windows backend for the Process subsystem.
This patch adds support for Microsoft Windows in the Process subsystem.

Libevent does not support mixing different types of handles (sockets,
named pipes, etc.) on Windows in its core event loop code. This have
historically meant that Tor have avoided attaching any non-networking
handles to the event loop. This patch uses a slightly different approach
to roughly support the same features for the Process subsystem as we do
with the Unix backend.

In this patch we use Windows Extended I/O functions (ReadFileEx() and
WriteFileEx()) which executes asynchronously in the background and
executes a completion routine when the scheduled read or write operation
have completed. This is much different from the Unix backend where the
operating system signals to us whenever a file descriptor is "ready" to
either being read from or written to.

To make the Windows operating system execute the completion routines of
ReadFileEx() and WriteFileEx() we must get the Tor process into what
Microsoft calls an "alertable" state. To do this we execute SleepEx()
with a zero millisecond sleep time from a main loop timer that ticks
once a second.  This moves the process into the "alertable" state and
when we return from the zero millisecond timeout all the outstanding I/O
completion routines will be called and we can schedule the next reads
and writes.

The timer loop is also responsible for detecting whether our child
processes have terminated since the last timer tick.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
2e957027e2 Add Unix backend for the Process subsystem.
This patch adds the Unix backend for the Process subsystem. The Unix
backend attaches file descriptors from the child process's standard in,
out and error to Tor's libevent based main loop using traditional Unix
pipes. We use the already available `waitpid` module to get events
whenever the child process terminates.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Alexander Færøy
35509978dd Add new Process subsystem.
This patch adds a new Process subsystem for running external programs in
the background of Tor. The design is focused around a new type named
`process_t` which have an API that allows the developer to easily write
code that interacts with the given child process. These interactions
includes:

- Easy API for writing output to the child process's standard input
  handle.
- Receive callbacks whenever the child has output on either its standard
  output or standard error handles.
- Receive callback when the child process terminates.

We also support two different "protocols" for handling output from the
child process. The default protocol is the "line" protocol where the
process output callbacks will be invoked only when there is complete
lines (either "\r\n" or "\n" terminated). We also support the "raw"
protocol where the read callbacks will get whatever the operating system
delivered to us in a single read operation.

This patch does not include any operating system backends, but the Unix
and Windows backends will be included in separate commits.

See: https://bugs.torproject.org/28179
2018-12-17 16:39:28 -05:00
Nick Mathewson
29254812a3 Remove strcmp_len(): it is now unused
(See 28856.)
2018-12-17 09:04:25 -05:00
Nick Mathewson
3c35c0d441 Add a function to provide an upper bound on base64 decoded length 2018-12-14 13:51:51 -05:00
Nick Mathewson
041e9235c1 Lower the loop_max constant in curve25519_basepoint_spot_check()
The point of this function is to make sure that the ed25519-based
implementation of curve25519_basepoint() actually works when we
start tor, and use the regular fallback implementation if it
doesn't.  But it accounts for 9% of our startup time in the case
when we have directory information, and I think it's safe to make
the test shorter.  After all, it has yet to find any actual bugs in
curved25519_scalarmult_basepoint_donna() on any platforms.

Closes ticket 28838.
2018-12-13 11:26:09 -05:00
Nick Mathewson
69264f96f3 Merge branch 'dormant_persist_squashed' 2018-12-13 08:26:10 -05:00
Nick Mathewson
b5c04173c8 Change interaction between dormant mode and clock jumps.
When the clock jumps, and we have a record of last user activity,
adjust that record.  This way if I'm inactive for 10 minutes and
then the laptop is sleeping for an hour, I'll still count as having
been inactive for 10 minutes.

Previously, we treat every jump as if it were activity, which is
ridiculous, and would prevent a Tor instance with a jumpy clock from
ever going dormant.
2018-12-13 08:25:54 -05:00
Nick Mathewson
f0a8664677 Add code to parse K=V lines into config_line_t format.
Closes ticket 28755
2018-12-12 11:07:08 -05:00
Nick Mathewson
ca4b86f90a Merge remote-tracking branch 'tor-github/pr/508' 2018-12-05 08:19:02 -05:00
Nick Mathewson
701eaef980 Move net.inet.ip.random_id code to lib/net/ 2018-12-01 11:36:03 -05:00
Alexander Færøy
2b41b857bd Add LD_PROCESS as log domain.
See: https://bugs.torproject.org/28179
2018-11-27 19:31:08 +01:00
Alexander Færøy
31b3a6577c Add buf_flush_to_pipe() and buf_read_from_pipe().
This patch adds two new functions: buf_flush_to_pipe() and
buf_read_from_pipe(), which makes use of our new buf_flush_to_fd() and
buf_read_from_fd() functions.

See: https://bugs.torproject.org/28179
2018-11-27 19:31:08 +01:00
Alexander Færøy
771930b84c Refactor buf_read_from_socket() into buf_read_from_fd().
This patch refactors buf_read_from_socket() into buf_read_from_fd(), and
creates a specialized function for buf_read_from_socket(), which uses
buf_read_from_fd().

See: https://bugs.torproject.org/28179
2018-11-27 19:31:08 +01:00
Alexander Færøy
c71f9df07b Refactor buf_flush_to_socket() into buf_flush_to_fd().
This patch refactors buf_flush_to_socket() into buf_flush_to_fd() and
creates a specialization function for buf_flush_to_socket() that makes
use of buf_flush_to_fd().

See: https://bugs.torproject.org/28179
2018-11-27 19:31:08 +01:00
Alexander Færøy
340260281a Refactor flush_chunk() to work on pipes as well as sockets.
See: https://bugs.torproject.org/28179
2018-11-27 19:31:08 +01:00
Alexander Færøy
5f26ae833e Refactor read_to_chunk() such that it supports both pipes and sockets.
See: https://bugs.torproject.org/28179
2018-11-27 19:31:08 +01:00
Nick Mathewson
8a15d0f69b Merge branch 'maint-0.3.5' 2018-11-26 17:25:28 -05:00
teor
44ced9b750 Merge branch 'bug28096-029-squashed' into bug28096-035-squashed
Move the get_uname() changes from src/common/compat.c to
src/lib/osinfo/uname.c
2018-11-15 12:23:29 +10:00
Nick Mathewson
eaff47352a Make sure sandbox-related getaddrinfo() functions always exist. 2018-11-14 16:12:29 -05:00
Nick Mathewson
e429e31ad1 Normalize .may_include to always have paths, and paths to include 2018-11-14 16:07:35 -05:00
Nick Mathewson
c0a7527eb8 Remove dependency on lib/net from lib/sandbox.
This was trivial, and the easiest way to remove the remaining
.may_include circularities.
2018-11-14 16:07:03 -05:00
Nick Mathewson
f6b8c7da66 Move buffers.c out of lib/containers to resolve a circularity. 2018-11-14 16:07:03 -05:00
Nick Mathewson
c9f9c9bc49 Make memarea use smartlist_core, not container. 2018-11-14 16:07:03 -05:00
Nick Mathewson
a6a7a1f3ed Merge branch 'maint-0.3.5' 2018-11-13 16:48:26 -05:00
Nick Mathewson
021187f915 Merge branch 'bug28183_029' into maint-0.3.5 2018-11-13 16:48:21 -05:00
Nick Mathewson
d000e798ac Merge branch 'maint-0.3.5' 2018-11-13 08:27:29 -05:00
Nick Mathewson
ae4c94bb64 Merge branch 'maint-0.3.4' into maint-0.3.5 2018-11-13 08:27:29 -05:00
Nick Mathewson
d1e9285b1d Merge branch 'maint-0.3.5' 2018-11-11 18:05:45 -05:00
Nick Mathewson
896d0ebb99 Merge branch 'maint-0.3.4' into maint-0.3.5 2018-11-11 18:05:45 -05:00
Nick Mathewson
bf4f55a13d Merge branch 'subsystems' 2018-11-09 15:01:49 -05:00
Nick Mathewson
c6336727ca Rename subsystem callback functions to make them consistent 2018-11-09 11:12:12 -05:00
Nick Mathewson
e80595f562 fixup! Make initialization for the "err" library into a subsystem.
Check for failure to install backtrace handler.
2018-11-09 11:00:31 -05:00
Nick Mathewson
61695e3d62 Document that subsystem callbacks are optional. 2018-11-09 10:58:20 -05:00
Nick Mathewson
4fe4bcf8a1 Explain that configuration should happen elsewhere, but not init. 2018-11-09 10:55:18 -05:00
Nick Mathewson
5d73f87ced Merge branch 'maint-0.3.5' 2018-11-09 10:49:48 -05:00
Nick Mathewson
2ac2d0a426 Merge branch 'maint-0.3.4' into maint-0.3.5 2018-11-09 10:49:47 -05:00
Roger Dingledine
6e828ced56 simplify now that it uses tor's copyright and license 2018-11-07 11:02:26 -05:00
Nick Mathewson
a7a060a637 Switch ctassert.h to 3bsd (with permission) 2018-11-07 10:37:02 -05:00
Nick Mathewson
770653ff45 Allow lib/cc to include its own files. 2018-11-06 16:59:39 -05:00
Nick Mathewson
d9508d8ede Change copyright statement (with permission) 2018-11-06 15:42:18 -05:00
Nick Mathewson
6b706bcf19 Remove a tab. 2018-11-06 15:41:32 -05:00
Nick Mathewson
3c9dd9ef2d Add parentheses to the ctassert macro expansions 2018-11-06 15:41:14 -05:00
Nick Mathewson
e69a4ad6b3 Add a user of CTASSERT(). 2018-11-06 15:39:58 -05:00
Taylor R Campbell
1a6060fa42 New macro CTASSERT(condition) to assert condition at compile-time.
To get it, use: #include "lib/cc/ctassert.h"
2018-11-06 15:36:58 -05:00
Nick Mathewson
8020d6fb05 Merge remote-tracking branch 'tor-github/pr/464' 2018-11-06 15:33:25 -05:00
Nick Mathewson
bcba6a4918 Merge branch 'maint-0.3.5' 2018-11-06 15:17:52 -05:00
Kris Katterjohn
daaa2751ed Include sys/time.h in timers.c and time_fmt.c
This fixes compilation on OpenBSD.

Signed-off-by: Kris Katterjohn <katterjohn@gmail.com>
2018-11-06 15:15:22 -05:00
Nick Mathewson
32b23a4c40 Make tortls use the subsystems interface
This one only needs a shutdown right now.
2018-11-05 09:22:02 -05:00
Nick Mathewson
019a044e5e Turn "compress" into a subsystem. 2018-11-05 09:22:02 -05:00
Nick Mathewson
207253df8d Move monotonic time setup into a subsystem 2018-11-05 09:22:02 -05:00
Nick Mathewson
cad61f0f6d Move prefork, postfork, and thread-exit hooks into subsys
So far, crypto is the only module that uses them, but others are
likely to do so in the future.
2018-11-05 09:22:02 -05:00
Nick Mathewson
50436ccea4 Add crypto module as a subsystem. 2018-11-05 09:22:02 -05:00
Nick Mathewson
cfe5b35edb Move networking startup/cleanup logic into a subsystem. 2018-11-05 09:22:02 -05:00
Nick Mathewson
05b54f6a6a Use subsystems manager for subsystems used in tests. 2018-11-05 09:22:02 -05:00
Nick Mathewson
a0ee54549f Turn the wallclock module into a subsystem.
(This may be slightly gratuitous.)
2018-11-05 09:22:02 -05:00
Nick Mathewson
d3e4afcc9b Turn the logging code into a subsystem 2018-11-05 09:22:02 -05:00
Nick Mathewson
b8c50eabfe Add a subsystem for our threading support 2018-11-05 09:22:02 -05:00
Nick Mathewson
178c1821b2 Make the windows process parameter initialization a subsystem
Also, move it from "main" into lib/process
2018-11-05 09:22:02 -05:00
Nick Mathewson
175153a329 Make initialization for the "err" library into a subsystem. 2018-11-05 09:22:02 -05:00
Nick Mathewson
6e7ff8cba0 Move the code that knows our tor version into a lowest-level lib 2018-11-05 09:22:02 -05:00
Nick Mathewson
7bb76b24cf Code to manage the list of subsystems. 2018-11-05 09:22:02 -05:00
Nick Mathewson
a6c7e01584 Merge branch 'bug27963_wallclock' into maint-0.3.5 2018-11-02 09:39:13 -04:00
Nick Mathewson
084a5a13c7 Merge branch 'bug27963_wallclock' 2018-11-02 09:38:26 -04:00
Nick Mathewson
0ce1f2d466 Declare the subsystem structure. 2018-11-01 12:55:52 -04:00