From baddb30889fd281962f014614a43f323b5df6ae6 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Sun, 10 Nov 2019 15:49:45 -0500 Subject: [PATCH 1/4] doxygen: add a command for linking to our bugtracker --- Doxyfile.in | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Doxyfile.in b/Doxyfile.in index a087012c71..2ede044473 100644 --- a/Doxyfile.in +++ b/Doxyfile.in @@ -258,6 +258,8 @@ ALIASES = ALIASES += refdir{1}="\ref src/\1 \"\1\"" +ALIASES += ticket{1}="[ticket \1](https://bugs.torproject.org/\1)" + # This tag can be used to specify a number of word-keyword mappings (TCL only). # A mapping has the form "name=value". For example adding "class=itcl::class" # will allow you to use the command class in the itcl::class meaning. From a6d22d7fc2c5f5c868fda4ed1224216a16c25f84 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Sun, 10 Nov 2019 16:11:40 -0500 Subject: [PATCH 2/4] Improve subsys documentation; add initialization documentation. --- src/lib/subsys/initialization.dox | 76 +++++++++++++++++++++++++++++++ src/lib/subsys/subsys.h | 58 ++++++++++++++++++++++- src/mainpage.dox | 3 ++ 3 files changed, 135 insertions(+), 2 deletions(-) create mode 100644 src/lib/subsys/initialization.dox diff --git a/src/lib/subsys/initialization.dox b/src/lib/subsys/initialization.dox new file mode 100644 index 0000000000..3b500f9620 --- /dev/null +++ b/src/lib/subsys/initialization.dox @@ -0,0 +1,76 @@ +/** + +@page initialization Initialization and shutdown + +@tableofcontents + +@section overview Overview + +Tor has a single entry point: tor_run_main() in main.c. All the ways of +starting a Tor process (ntmain.c, tor_main.c, and tor_api.c) work by invoking tor_run_main(). + +The tor_run_main() function normally exits (\ref init_exceptwhen "1") by +returning: not by calling abort() or exit(). Before it returns, it calls +tor_cleanup() in shutdown.c. + +Conceptually, there are several stages in running Tor. + +1. First, we initialize those modules that do not depend on the + configuration. This happens in the first half of tor_run_main(), and the + first half of tor_init(). (\ref init_pending_refactor "2") + +2. Second, we parse the command line and our configuration, and configure + systems that depend on our configuration or state. This configuration + happens midway through tor_init(), which invokes + options_init_from_torrc(). We then initialize more systems from the + second half of tor_init(). + +3. At this point we may exit early if we have been asked to do something + requiring no further initialization, like printing our version number or + creating a new signing key. Otherwise, we proceed to run_tor_main_loop(), + which initializes some network-specific parts of Tor, grabs some + daemon-only resources (like the data directory lock) and starts Tor itself + running. + + +> \anchor init_exceptwhen 1. tor_run_main() _can_ terminate with a call to +> abort() or exit(), but only when crashing due to a bug, or when forking to +> run as a daemon. + +> \anchor init_pending_refactor 2. The pieces of code that I'm describing as +> "the first part of tor_init()" and so on deserve to be functions with their +> own name. I'd like to refactor them, but before I do so, there is some +> slight reorganization that needs to happen. Notably, the +> nt_service_parse_options() call ought logically to be later in our +> initialization sequence. See \ticket{32447} for our refactoring progress. + + +@section subsys Subsystems and initialization + +Our current convention is to use the subsystem mechanism to initialize and +clean up pieces of Tor. The more recently updated pieces of Tor will use +this mechanism. For examples, see e.g. time_sys.c or log_sys.c. + +In simplest terms, a **subsytem** is a logically separate part of Tor that +can be initialized, shut down, managed, and configured somewhat independently +of the rest of the program. + +To define a subsystem, we declare a `static const` instance of subsys_fns_t, +describing the subsystem and a set of functions that initialize it, +deconstruct it, and so on. See the documentation for subsys_fns_t for a full +list of these functions. + +After defining a subsytem, it must be inserted in subsystem_list.c. At that +point, table-driven mechanisms in subsysmgr.c will invoke its functions when +appropriate. + +@subsection vsconfig Initialization versus configuration + +We note that the initialization phase of Tor occurs before any configuration +is read from disk -- and therefore before any other files are read from +disk. Therefore, any behavior that depends on Tor's configuration or state +must occur _after_ the initialization process, during configuration. + + + +**/ diff --git a/src/lib/subsys/subsys.h b/src/lib/subsys/subsys.h index 29a90c049d..258d060bb8 100644 --- a/src/lib/subsys/subsys.h +++ b/src/lib/subsys/subsys.h @@ -25,6 +25,13 @@ struct config_format_t; * * You should use c99 named-field initializers with this structure: we * will be adding more fields, often in the middle of the structure. + * + * See \ref initialization for more information about initialization and + * shutdown in Tor. + * + * To make a new subsystem, you declare a const instance of this type, and + * include it on the list in subsystem_list.c. The code that manages these + * subsystems is in subsysmgr.c. **/ typedef struct subsys_fns_t { /** @@ -55,7 +62,7 @@ typedef struct subsys_fns_t { * it is only for global state or pre-configuration state. * * (If you need to do any setup that depends on configuration, you'll need - * to declare a configuration callback. (Not yet designed)) + * to declare a configuration callback instead. (Not yet designed)) * * This function MUST NOT have any parts that can fail. **/ @@ -63,22 +70,49 @@ typedef struct subsys_fns_t { /** * Connect a subsystem to the message dispatch system. + * + * This function should use the macros in \refdir{lib/pubsub} to register a + * set of messages that this subsystem may publish, and may subscribe to. + * + * See pubsub_macros.h for more information, and for examples. **/ int (*add_pubsub)(struct pubsub_connector_t *); /** * Perform any necessary pre-fork cleanup. This function may not fail. + * + * On Windows (and any other platforms without fork()), this function will + * never be invoked. Otherwise it is used when we are about to start + * running as a background daemon, or when we are about to run a unit test + * in a subprocess. Unlike the subsys_fns_t.postfork callback, it is run + * from the parent process. + * + * Note that we do not invoke this function when the child process's only + * purpose is to call exec() and run another program. */ void (*prefork)(void); /** * Perform any necessary post-fork setup. This function may not fail. + * + * On Windows (and any other platforms without fork()), this function will + * never be invoked. Otherwise it is used when we are about to start + * running as a background daemon, or when we are about to run a unit test + * in a subprocess. Unlike the subsys_fns_t.prefork callback, it is run + * from the child process. + * + * Note that we do not invoke this function when the child process's only + * purpose is to call exec() and run another program. */ void (*postfork)(void); /** * Free any thread-local resources held by this subsystem. Called before * the thread exits. + * + * This function is not allowed to fail. + * + * \bug Note that this callback is currently buggy: See \ticket{32103}. */ void (*thread_cleanup)(void); @@ -86,18 +120,28 @@ typedef struct subsys_fns_t { * Free all resources held by this subsystem. * * This function is not allowed to fail. + * + * Subsystems are shut down when Tor is about to exit or return control to + * an embedding program. This callback must return the process to a state + * such that subsys_fns_t.init will succeed if invoked again. **/ void (*shutdown)(void); /** * A config_format_t describing all of the torrc fields owned by this * subsystem. + * + * This object, if present, is registered in a confmgr_t for Tor's options, + * and used to parse option fields from the command line and torrc file. **/ const struct config_format_t *options_format; /** * A config_format_t describing all of the DataDir/state fields owned by * this subsystem. + * + * This object, if present, is registered in a confmgr_t for Tor's state, + * and used to parse state fields from the DataDir/state file. **/ const struct config_format_t *state_format; @@ -106,7 +150,11 @@ typedef struct subsys_fns_t { * on success, -1 on failure. * * It is safe to store the pointer to the object until set_options() - * is called again. */ + * is called again. + * + * This function is only called after all the validation code defined + * by subsys_fns_t.options_format has passed. + **/ int (*set_options)(void *); /* XXXX Add an implementation for options_act_reversible() later in this @@ -118,6 +166,12 @@ typedef struct subsys_fns_t { * * It is safe to store the pointer to the object; set_state() is only * called on startup. + * + * This function is only called after all the validation code defined + * by subsys_fns_t.state_format has passed. + * + * This function will only be called once per invocation of Tor, since + * Tor does not reload its state while it is running. **/ int (*set_state)(void *); diff --git a/src/mainpage.dox b/src/mainpage.dox index eb29eb5fa2..575d8b7f4e 100644 --- a/src/mainpage.dox +++ b/src/mainpage.dox @@ -30,7 +30,10 @@ Tor repository. @subpage intro +@subpage initialization + @subpage dataflow + **/ /** From 2d508f8fa56e180428337750788a155579ba84c8 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Sun, 10 Nov 2019 16:14:38 -0500 Subject: [PATCH 3/4] Not const. --- src/lib/subsys/initialization.dox | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/subsys/initialization.dox b/src/lib/subsys/initialization.dox index 3b500f9620..ed2d5abf27 100644 --- a/src/lib/subsys/initialization.dox +++ b/src/lib/subsys/initialization.dox @@ -55,7 +55,7 @@ In simplest terms, a **subsytem** is a logically separate part of Tor that can be initialized, shut down, managed, and configured somewhat independently of the rest of the program. -To define a subsystem, we declare a `static const` instance of subsys_fns_t, +To define a subsystem, we declare a `const` instance of subsys_fns_t, describing the subsystem and a set of functions that initialize it, deconstruct it, and so on. See the documentation for subsys_fns_t for a full list of these functions. @@ -73,4 +73,5 @@ must occur _after_ the initialization process, during configuration. + **/ From 8746fedce4a440cccde2b4683d7aa72e7c0b0c89 Mon Sep 17 00:00:00 2001 From: Nick Mathewson Date: Fri, 15 Nov 2019 09:00:54 -0500 Subject: [PATCH 4/4] Initialization documents: incorporate feedback from review. (Thanks, Taylor!) --- src/lib/subsys/initialization.dox | 18 +++++++++--------- src/lib/subsys/subsys.h | 9 +++++---- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/src/lib/subsys/initialization.dox b/src/lib/subsys/initialization.dox index ed2d5abf27..561cd17bab 100644 --- a/src/lib/subsys/initialization.dox +++ b/src/lib/subsys/initialization.dox @@ -9,7 +9,7 @@ Tor has a single entry point: tor_run_main() in main.c. All the ways of starting a Tor process (ntmain.c, tor_main.c, and tor_api.c) work by invoking tor_run_main(). -The tor_run_main() function normally exits (\ref init_exceptwhen "1") by +The tor_run_main() function normally exits (@ref init_exceptwhen "1") by returning: not by calling abort() or exit(). Before it returns, it calls tor_cleanup() in shutdown.c. @@ -17,7 +17,7 @@ Conceptually, there are several stages in running Tor. 1. First, we initialize those modules that do not depend on the configuration. This happens in the first half of tor_run_main(), and the - first half of tor_init(). (\ref init_pending_refactor "2") + first half of tor_init(). (@ref init_pending_refactor "2") 2. Second, we parse the command line and our configuration, and configure systems that depend on our configuration or state. This configuration @@ -33,16 +33,16 @@ Conceptually, there are several stages in running Tor. running. -> \anchor init_exceptwhen 1. tor_run_main() _can_ terminate with a call to +> @anchor init_exceptwhen 1. tor_run_main() _can_ terminate with a call to > abort() or exit(), but only when crashing due to a bug, or when forking to > run as a daemon. -> \anchor init_pending_refactor 2. The pieces of code that I'm describing as +> @anchor init_pending_refactor 2. The pieces of code that I'm describing as > "the first part of tor_init()" and so on deserve to be functions with their > own name. I'd like to refactor them, but before I do so, there is some > slight reorganization that needs to happen. Notably, the > nt_service_parse_options() call ought logically to be later in our -> initialization sequence. See \ticket{32447} for our refactoring progress. +> initialization sequence. See @ticket{32447} for our refactoring progress. @section subsys Subsystems and initialization @@ -55,10 +55,10 @@ In simplest terms, a **subsytem** is a logically separate part of Tor that can be initialized, shut down, managed, and configured somewhat independently of the rest of the program. -To define a subsystem, we declare a `const` instance of subsys_fns_t, -describing the subsystem and a set of functions that initialize it, -deconstruct it, and so on. See the documentation for subsys_fns_t for a full -list of these functions. +The subsys_fns_t type describes a subsystem and a set of functions that +initialize it, desconstruct it, and so on. To define a subsystem, we declare +a `const` instance of subsys_fns_t. See the documentation for subsys_fns_t +for a full list of these functions. After defining a subsytem, it must be inserted in subsystem_list.c. At that point, table-driven mechanisms in subsysmgr.c will invoke its functions when diff --git a/src/lib/subsys/subsys.h b/src/lib/subsys/subsys.h index 258d060bb8..324f4f2947 100644 --- a/src/lib/subsys/subsys.h +++ b/src/lib/subsys/subsys.h @@ -23,10 +23,11 @@ struct config_format_t; * All callbacks are optional -- if a callback is set to NULL, the subsystem * manager will treat it as a no-op. * - * You should use c99 named-field initializers with this structure: we - * will be adding more fields, often in the middle of the structure. + * You should use c99 named-field initializers with this structure, for + * readability and safety. (There are a lot of functions here, all of them + * optional, and many of them with similar signatures.) * - * See \ref initialization for more information about initialization and + * See @ref initialization for more information about initialization and * shutdown in Tor. * * To make a new subsystem, you declare a const instance of this type, and @@ -71,7 +72,7 @@ typedef struct subsys_fns_t { /** * Connect a subsystem to the message dispatch system. * - * This function should use the macros in \refdir{lib/pubsub} to register a + * This function should use the macros in @refdir{lib/pubsub} to register a * set of messages that this subsystem may publish, and may subscribe to. * * See pubsub_macros.h for more information, and for examples.