From 0d6c8eed49a9104597a907664675cd4095fb852f Mon Sep 17 00:00:00 2001 From: Antoine Veuiller Date: Wed, 21 Aug 2019 16:22:08 +0200 Subject: [PATCH 1/2] doc(hacking): enhance markdown style --- doc/HACKING/CodeStructure.md | 151 +++++++++++++++++------------------ 1 file changed, 72 insertions(+), 79 deletions(-) diff --git a/doc/HACKING/CodeStructure.md b/doc/HACKING/CodeStructure.md index 736d6cd484..fffafcaed1 100644 --- a/doc/HACKING/CodeStructure.md +++ b/doc/HACKING/CodeStructure.md @@ -2,128 +2,121 @@ TODO: revise this to talk about how things are, rather than how things have changed. -TODO: Make this into good markdown. - - - -For quite a while now, the program "tor" has been built from source -code in just two directories: src/common and src/or. +For quite a while now, the program *tor* has been built from source +code in just two directories: **src/common** and **src/or**. This has become more-or-less untenable, for a few reasons -- most notably of which is that it has led our code to become more spaghetti-ish than I can endorse with a clean conscience. So to fix that, we've gone and done a huge code movement in our git -master branch, which will land in a release once Tor 0.3.5.1-alpha is +master branch, which will land in a release once Tor `0.3.5.1-alpha` is out. Here's what we did: - * src/common has been turned into a set of static libraries. These -all live in the "src/lib/*" directories. The dependencies between + * **src/common** has been turned into a set of static libraries. These +all live in the **src/lib/*** directories. The dependencies between these libraries should have no cycles. The libraries are: - arch -- Headers to handle architectural differences - cc -- headers to handle differences among compilers - compress -- wraps zlib, zstd, lzma - container -- high-level container types - crypt_ops -- Cryptographic operations. Planning to split this into + - **arch** -- Headers to handle architectural differences + - **cc** -- headers to handle differences among compilers + - **compress** -- wraps zlib, zstd, lzma + - **container** -- high-level container types + - **crypt_ops** -- Cryptographic operations. Planning to split this into a higher and lower level library - ctime -- Operations that need to run in constant-time. (Properly, + - **ctime** -- Operations that need to run in constant-time. (Properly, data-invariant time) - defs -- miscelaneous definitions needed throughout Tor. - encoding -- transforming one data type into another, and various + - **defs** -- miscelaneous definitions needed throughout Tor. + - **encoding** -- transforming one data type into another, and various data types into strings. - err -- lowest-level error handling, in cases where we can't use + - **err** -- lowest-level error handling, in cases where we can't use the logs because something that the logging system needs has broken. - evloop -- Generic event-loop handling logic - fdio -- Low-level IO wrapper functions for file descriptors. - fs -- Operations on the filesystem - intmath -- low-level integer math and misc bit-twiddling hacks - lock -- low-level locking code - log -- Tor's logging module. This library sits roughly halfway up + - **evloop** -- Generic event-loop handling logic + - **fdio** -- Low-level IO wrapper functions for file descriptors. + - **fs** -- Operations on the filesystem + - **intmath** -- low-level integer math and misc bit-twiddling hacks + - **lock** -- low-level locking code + - **log** -- Tor's logging module. This library sits roughly halfway up the library dependency diagram, since everything it depends on has to be carefully crafted to *not* log. - malloc -- Low-level wrappers for the platform memory allocation functions. - math -- Higher-level mathematical functions, and floating-point math - memarea -- An arena allocator - meminfo -- Functions for querying the current process's memory + - **malloc** -- Low-level wrappers for the platform memory allocation functions. + - **math** -- Higher-level mathematical functions, and floating-point math + - **memarea** -- An arena allocator + - **meminfo** -- Functions for querying the current process's memory status and resources - net -- Networking compatibility and convenience code - osinfo -- Querying information about the operating system - process -- Launching and querying the status of other processes - sandbox -- Backend for the linux seccomp2 sandbox - smartlist_core -- The lowest-level of the smartlist_t data type. + - **net** -- Networking compatibility and convenience code + - **osinfo** -- Querying information about the operating system + - **process** -- Launching and querying the status of other processes + - **sandbox** -- Backend for the linux seccomp2 sandbox + - **smartlist_core** -- The lowest-level of the smartlist_t data type. Separated from the rest of the containers library because the logging subsystem depends on it. - string -- Compatibility and convenience functions for manipulating + - **string** -- Compatibility and convenience functions for manipulating C strings. - term -- Terminal-related functions (currently limited to a getpass + - **term** -- Terminal-related functions (currently limited to a getpass function). - testsupport -- Macros for mocking, unit tests, etc. - thread -- Higher-level thread compatibility code - time -- Higher-level time management code, including format + - **testsupport** -- Macros for mocking, unit tests, etc. + - **thread** -- Higher-level thread compatibility code + - **time** -- Higher-level time management code, including format conversions and monotonic time - tls -- Our wrapper around our TLS library - trace -- Formerly src/trace -- a generic event tracing API - wallclock -- Low-level time code, used by the log module. + - **tls** -- Our wrapper around our TLS library + - **trace** -- Formerly src/trace -- a generic event tracing API + - **wallclock** -- Low-level time code, used by the log module. - * To ensure that the dependency graph in src/common remains under -control, there is a tool that you can run called "make -check-includes". It verifies that each module in Tor only includes + * To ensure that the dependency graph in **src/common** remains under +control, there is a tool that you can run called `make +check-includes`. It verifies that each module in Tor only includes the headers that it is permitted to include, using a per-directory -".may_include" file. +*.may_include* file. - * The src/or/or.h header has been split into numerous smaller + * The **src/or/or.h** header has been split into numerous smaller headers. Notably, many important structures are now declared in a -header called foo_st.h, where "foo" is the name of the structure. +header called *foo_st.h*, where "foo" is the name of the structure. - * The src/or directory, which had most of Tor's code, had been split + * The **src/or** directory, which had most of Tor's code, had been split up into several directories. This is still a work in progress: This code has not itself been refactored, and its dependency graph is still a tangled web. I hope we'll be working on that over the coming releases, but it will take a while to do. - The new top-level source directories are: - - src/core -- Code necessary to actually perform or use onion routing. - src/feature -- Code used only by some onion routing + - The new top-level source directories are: + - **src/core** -- Code necessary to actually perform or use onion routing. + - **src/feature** -- Code used only by some onion routing configurations, or only for a special purpose. - src/app -- Top-level code to run, invoke, and configure the + - **src/app** -- Top-level code to run, invoke, and configure the lower-level code - The new second-level source directories are: - src/core/crypto -- High-level cryptographic protocols used in Tor - src/core/mainloop -- Tor's event loop, connection-handling, and + - The new second-level source directories are: + - **src/core/crypto** -- High-level cryptographic protocols used in Tor + - **src/core/mainloop** -- Tor's event loop, connection-handling, and traffic-routing code. - src/core/or -- Parts related to handling onion routing itself - src/core/proto -- support for encoding and decoding different + - **src/core/or** -- Parts related to handling onion routing itself + - **src/core/proto** -- support for encoding and decoding different wire protocols - - src/feature/api -- Support for making Tor embeddable - src/feature/client -- Functionality which only Tor clients need - src/feature/control -- Controller implementation - src/feature/dirauth -- Directory authority - src/feature/dircache -- Directory cache - src/feature/dirclient -- Directory client - src/feature/dircommon -- Shared code between the other directory modules - src/feature/hibernate -- Hibernating when Tor is out of bandwidth + - **src/feature/api** -- Support for making Tor embeddable + - **src/feature/client** -- Functionality which only Tor clients need + - **src/feature/control** -- Controller implementation + - **src/feature/dirauth** -- Directory authority + - **src/feature/dircache** -- Directory cache + - **src/feature/dirclient** -- Directory client + - **src/feature/dircommon** -- Shared code between the other directory modules + - **src/feature/hibernate** -- Hibernating when Tor is out of bandwidth or shutting down - src/feature/hs -- v3 onion service implementation - src/feature/hs_common -- shared code between both onion service + - **src/feature/hs** -- v3 onion service implementation + - **src/feature/hs_common** -- shared code between both onion service implementations - src/feature/nodelist -- storing and accessing the list of relays on + - **src/feature/nodelist** -- storing and accessing the list of relays on the network. - src/feature/relay -- code that only relay servers and exit servers need. - src/feature/rend -- v2 onion service implementation - src/feature/stats -- statistics and history + - **src/feature/relay** -- code that only relay servers and exit servers need. + - **src/feature/rend** -- v2 onion service implementation + - **src/feature/stats** -- statistics and history + - **src/app/config** -- configuration and state for Tor + - **src/app/main** -- Top-level functions to invoke the rest or Tor. - src/app/config -- configuration and state for Tor - src/app/main -- Top-level functions to invoke the rest or Tor. - - * The "tor" executable is now built in src/app/tor rather than src/or/tor. + * The `tor` executable is now built in **src/app/tor** rather than **src/or/tor**. * There are more static libraries than before that you need to build into your application if you want to embed Tor. Rather than -maintaining this list yourself, I recommend that you run "make -show-libs" to have Tor emit a list of what you need to link. +maintaining this list yourself, I recommend that you run `make +show-libs` to have Tor emit a list of what you need to link. From 3bf90e704ca14fd0b50f46ff150205b3fbb97a82 Mon Sep 17 00:00:00 2001 From: Antoine Veuiller Date: Wed, 21 Aug 2019 16:22:37 +0200 Subject: [PATCH 2/2] doc(hacking): update = to # on sections --- doc/HACKING/EndOfLifeTor.md | 4 ++-- doc/HACKING/Fuzzing.md | 14 +++++++------- doc/HACKING/ReleasingTor.md | 12 ++++++------ 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/doc/HACKING/EndOfLifeTor.md b/doc/HACKING/EndOfLifeTor.md index c7492f0ca0..2fece2ca9d 100644 --- a/doc/HACKING/EndOfLifeTor.md +++ b/doc/HACKING/EndOfLifeTor.md @@ -7,7 +7,7 @@ series reaches End of Life. Note that they are _only_ for entire series that have reached their planned EOL: they do not apply to security-related deprecations of individual versions. -=== 0. Preliminaries +### 0. Preliminaries 0. A few months before End of Life: Write a deprecation announcement. @@ -17,7 +17,7 @@ deprecations of individual versions. Send the announcement to tor-announce, tor-talk, tor-relays, and the packagers. -=== 1. On the day +### 1. On the day 1. Open tickets to remove the release from: - the jenkins builds diff --git a/doc/HACKING/Fuzzing.md b/doc/HACKING/Fuzzing.md index 2039d6a4c0..c2db7e9853 100644 --- a/doc/HACKING/Fuzzing.md +++ b/doc/HACKING/Fuzzing.md @@ -1,6 +1,6 @@ -= Fuzzing Tor +# Fuzzing Tor -== The simple version (no fuzzing, only tests) +## The simple version (no fuzzing, only tests) Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place where you checked it out. @@ -12,7 +12,7 @@ This won't actually fuzz Tor! It will just run all the fuzz binaries on our existing set of testcases for the fuzzer. -== Different kinds of fuzzing +## Different kinds of fuzzing Right now we support three different kinds of fuzzer. @@ -37,7 +37,7 @@ In all cases, you'll need some starting examples to give the fuzzer when it starts out. There's a set in the "fuzzing-corpora" git repository. Try setting TOR_FUZZ_CORPORA to point to a checkout of that repository -== Writing Tor fuzzers +## Writing Tor fuzzers A tor fuzzing harness should have: * a fuzz_init() function to set up any necessary global state. @@ -52,7 +52,7 @@ bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect "interesting" cases. -== Guided Fuzzing with AFL +## Guided Fuzzing with AFL There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so its integrity can't be verified. That said, you really shouldn't fuzz on a @@ -101,7 +101,7 @@ macOS (OS X) requires slightly more preparation, including: * using afl-clang (or afl-clang-fast from the llvm directory) * disabling external crash reporting (AFL will guide you through this step) -== Triaging Issues +## Triaging Issues Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code. @@ -115,7 +115,7 @@ To see what fuzz-http is doing with a test case, call it like this: (Logging is disabled while fuzzing to increase fuzzing speed.) -== Reporting Issues +## Reporting Issues Please report any issues discovered using the process in Tor's security issue policy: diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index 15c8ac55d8..f40e2af573 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -5,7 +5,7 @@ Putting out a new release Here are the steps that the maintainer should take when putting out a new Tor release: -=== 0. Preliminaries +### 0. Preliminaries 1. Get at least two of weasel/arma/Sebastian to put the new version number in their approved versions list. Give them a few @@ -18,7 +18,7 @@ new Tor release: date of a TB that contains it. See note below in "commit, upload, announce". -=== I. Make sure it works +### I. Make sure it works 1. Make sure that CI passes: have a look at Travis (https://travis-ci.org/torproject/tor/branches), Appveyor @@ -52,7 +52,7 @@ new Tor release: memory leaks.) -=== II. Write a changelog +### II. Write a changelog 1a. (Alpha release variant) @@ -139,7 +139,7 @@ new Tor release: text of existing entries, though.) -=== III. Making the source release. +### III. Making the source release. 1. In `maint-0.?.x`, bump the version number in `configure.ac` and run `make update-versions` to update version numbers in other @@ -165,7 +165,7 @@ new Tor release: If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see item 0.1 at the start of this document. -=== IV. Commit, upload, announce +### IV. Commit, upload, announce 1. Sign the tarball, then sign and push the git tag: @@ -241,7 +241,7 @@ new Tor release: For templates to use when announcing, see: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates -=== V. Aftermath and cleanup +### V. Aftermath and cleanup 1. If it's a stable release, bump the version number in the `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`