nihilist/tor
1
0
Fork 0
mirror of https://gitlab.torproject.org/tpo/core/tor.git synced 2024-11-24 04:13:28 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'tor-github/pr/1247'

Browse Source
This commit is contained in:
Nick Mathewson 2019-08-24 16:32:50 -04:00
commit 6ba05eea8e
4 changed files with 87 additions and 94 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

151
doc/HACKING/CodeStructure.md
View File

@ -2,128 +2,121 @@
TODO: revise this to talk about how things are, rather than how things
have changed.
TODO: Make this into good markdown.
For quite a while now, the program "tor" has been built from source
code in just two directories: src/common and src/or.
For quite a while now, the program *tor* has been built from source
code in just two directories: **src/common** and **src/or**.
This has become more-or-less untenable, for a few reasons -- most
notably of which is that it has led our code to become more
spaghetti-ish than I can endorse with a clean conscience.
So to fix that, we've gone and done a huge code movement in our git
master branch, which will land in a release once Tor 0.3.5.1-alpha is
master branch, which will land in a release once Tor `0.3.5.1-alpha` is
out.
Here's what we did:
* src/common has been turned into a set of static libraries. These
all live in the "src/lib/*" directories. The dependencies between
* **src/common** has been turned into a set of static libraries. These
all live in the **src/lib/*** directories. The dependencies between
these libraries should have no cycles. The libraries are:
arch -- Headers to handle architectural differences
cc -- headers to handle differences among compilers
compress -- wraps zlib, zstd, lzma
container -- high-level container types
crypt_ops -- Cryptographic operations. Planning to split this into
- **arch** -- Headers to handle architectural differences
- **cc** -- headers to handle differences among compilers
- **compress** -- wraps zlib, zstd, lzma
- **container** -- high-level container types
- **crypt_ops** -- Cryptographic operations. Planning to split this into
a higher and lower level library
ctime -- Operations that need to run in constant-time. (Properly,
- **ctime** -- Operations that need to run in constant-time. (Properly,
data-invariant time)
defs -- miscelaneous definitions needed throughout Tor.
encoding -- transforming one data type into another, and various
- **defs** -- miscelaneous definitions needed throughout Tor.
- **encoding** -- transforming one data type into another, and various
data types into strings.
err -- lowest-level error handling, in cases where we can't use
- **err** -- lowest-level error handling, in cases where we can't use
the logs because something that the logging system needs has broken.
evloop -- Generic event-loop handling logic
fdio -- Low-level IO wrapper functions for file descriptors.
fs -- Operations on the filesystem
intmath -- low-level integer math and misc bit-twiddling hacks
lock -- low-level locking code
log -- Tor's logging module. This library sits roughly halfway up
- **evloop** -- Generic event-loop handling logic
- **fdio** -- Low-level IO wrapper functions for file descriptors.
- **fs** -- Operations on the filesystem
- **intmath** -- low-level integer math and misc bit-twiddling hacks
- **lock** -- low-level locking code
- **log** -- Tor's logging module. This library sits roughly halfway up
the library dependency diagram, since everything it depends on has to
be carefully crafted to *not* log.
malloc -- Low-level wrappers for the platform memory allocation functions.
math -- Higher-level mathematical functions, and floating-point math
memarea -- An arena allocator
meminfo -- Functions for querying the current process's memory
- **malloc** -- Low-level wrappers for the platform memory allocation functions.
- **math** -- Higher-level mathematical functions, and floating-point math
- **memarea** -- An arena allocator
- **meminfo** -- Functions for querying the current process's memory
status and resources
net -- Networking compatibility and convenience code
osinfo -- Querying information about the operating system
process -- Launching and querying the status of other processes
sandbox -- Backend for the linux seccomp2 sandbox
smartlist_core -- The lowest-level of the smartlist_t data type.
- **net** -- Networking compatibility and convenience code
- **osinfo** -- Querying information about the operating system
- **process** -- Launching and querying the status of other processes
- **sandbox** -- Backend for the linux seccomp2 sandbox
- **smartlist_core** -- The lowest-level of the smartlist_t data type.
Separated from the rest of the containers library because the logging
subsystem depends on it.
string -- Compatibility and convenience functions for manipulating
- **string** -- Compatibility and convenience functions for manipulating
C strings.
term -- Terminal-related functions (currently limited to a getpass
- **term** -- Terminal-related functions (currently limited to a getpass
function).
testsupport -- Macros for mocking, unit tests, etc.
thread -- Higher-level thread compatibility code
time -- Higher-level time management code, including format
- **testsupport** -- Macros for mocking, unit tests, etc.
- **thread** -- Higher-level thread compatibility code
- **time** -- Higher-level time management code, including format
conversions and monotonic time
tls -- Our wrapper around our TLS library
trace -- Formerly src/trace -- a generic event tracing API
wallclock -- Low-level time code, used by the log module.
- **tls** -- Our wrapper around our TLS library
- **trace** -- Formerly src/trace -- a generic event tracing API
- **wallclock** -- Low-level time code, used by the log module.
* To ensure that the dependency graph in src/common remains under
control, there is a tool that you can run called "make
check-includes". It verifies that each module in Tor only includes
* To ensure that the dependency graph in **src/common** remains under
control, there is a tool that you can run called `make
check-includes`. It verifies that each module in Tor only includes
the headers that it is permitted to include, using a per-directory
".may_include" file.
*.may_include* file.
* The src/or/or.h header has been split into numerous smaller
* The **src/or/or.h** header has been split into numerous smaller
headers. Notably, many important structures are now declared in a
header called foo_st.h, where "foo" is the name of the structure.
header called *foo_st.h*, where "foo" is the name of the structure.
* The src/or directory, which had most of Tor's code, had been split
* The **src/or** directory, which had most of Tor's code, had been split
up into several directories. This is still a work in progress: This
code has not itself been refactored, and its dependency graph is still
a tangled web. I hope we'll be working on that over the coming
releases, but it will take a while to do.
The new top-level source directories are:
src/core -- Code necessary to actually perform or use onion routing.
src/feature -- Code used only by some onion routing
- The new top-level source directories are:
- **src/core** -- Code necessary to actually perform or use onion routing.
- **src/feature** -- Code used only by some onion routing
configurations, or only for a special purpose.
src/app -- Top-level code to run, invoke, and configure the
- **src/app** -- Top-level code to run, invoke, and configure the
lower-level code
The new second-level source directories are:
src/core/crypto -- High-level cryptographic protocols used in Tor
src/core/mainloop -- Tor's event loop, connection-handling, and
- The new second-level source directories are:
- **src/core/crypto** -- High-level cryptographic protocols used in Tor
- **src/core/mainloop** -- Tor's event loop, connection-handling, and
traffic-routing code.
src/core/or -- Parts related to handling onion routing itself
src/core/proto -- support for encoding and decoding different
- **src/core/or** -- Parts related to handling onion routing itself
- **src/core/proto** -- support for encoding and decoding different
wire protocols
src/feature/api -- Support for making Tor embeddable
src/feature/client -- Functionality which only Tor clients need
src/feature/control -- Controller implementation
src/feature/dirauth -- Directory authority
src/feature/dircache -- Directory cache
src/feature/dirclient -- Directory client
src/feature/dircommon -- Shared code between the other directory modules
src/feature/hibernate -- Hibernating when Tor is out of bandwidth
- **src/feature/api** -- Support for making Tor embeddable
- **src/feature/client** -- Functionality which only Tor clients need
- **src/feature/control** -- Controller implementation
- **src/feature/dirauth** -- Directory authority
- **src/feature/dircache** -- Directory cache
- **src/feature/dirclient** -- Directory client
- **src/feature/dircommon** -- Shared code between the other directory modules
- **src/feature/hibernate** -- Hibernating when Tor is out of bandwidth
or shutting down
src/feature/hs -- v3 onion service implementation
src/feature/hs_common -- shared code between both onion service
- **src/feature/hs** -- v3 onion service implementation
- **src/feature/hs_common** -- shared code between both onion service
implementations
src/feature/nodelist -- storing and accessing the list of relays on
- **src/feature/nodelist** -- storing and accessing the list of relays on
the network.
src/feature/relay -- code that only relay servers and exit servers need.
src/feature/rend -- v2 onion service implementation
src/feature/stats -- statistics and history
- **src/feature/relay** -- code that only relay servers and exit servers need.
- **src/feature/rend** -- v2 onion service implementation
- **src/feature/stats** -- statistics and history
- **src/app/config** -- configuration and state for Tor
- **src/app/main** -- Top-level functions to invoke the rest or Tor.
src/app/config -- configuration and state for Tor
src/app/main -- Top-level functions to invoke the rest or Tor.
* The "tor" executable is now built in src/app/tor rather than src/or/tor.
* The `tor` executable is now built in **src/app/tor** rather than **src/or/tor**.
* There are more static libraries than before that you need to build
into your application if you want to embed Tor. Rather than
maintaining this list yourself, I recommend that you run "make
show-libs" to have Tor emit a list of what you need to link.
maintaining this list yourself, I recommend that you run `make
show-libs` to have Tor emit a list of what you need to link.

4
doc/HACKING/EndOfLifeTor.md
View File

@ -7,7 +7,7 @@ series reaches End of Life. Note that they are _only_ for entire series that
have reached their planned EOL: they do not apply to security-related
deprecations of individual versions.
=== 0. Preliminaries
### 0. Preliminaries
0. A few months before End of Life:
Write a deprecation announcement.
@ -17,7 +17,7 @@ deprecations of individual versions.
Send the announcement to tor-announce, tor-talk, tor-relays, and the
packagers.
=== 1. On the day
### 1. On the day
1. Open tickets to remove the release from:
- the jenkins builds

14
doc/HACKING/Fuzzing.md
View File

@ -1,6 +1,6 @@
= Fuzzing Tor
# Fuzzing Tor
== The simple version (no fuzzing, only tests)
## The simple version (no fuzzing, only tests)
Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place
where you checked it out.
@ -12,7 +12,7 @@ This won't actually fuzz Tor! It will just run all the fuzz binaries
on our existing set of testcases for the fuzzer.
== Different kinds of fuzzing
## Different kinds of fuzzing
Right now we support three different kinds of fuzzer.
@ -37,7 +37,7 @@ In all cases, you'll need some starting examples to give the fuzzer when it
starts out. There's a set in the "fuzzing-corpora" git repository. Try
setting TOR_FUZZ_CORPORA to point to a checkout of that repository
== Writing Tor fuzzers
## Writing Tor fuzzers
A tor fuzzing harness should have:
* a fuzz_init() function to set up any necessary global state.
@ -52,7 +52,7 @@ bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
"interesting" cases.
== Guided Fuzzing with AFL
## Guided Fuzzing with AFL
There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so
its integrity can't be verified. That said, you really shouldn't fuzz on a
@ -101,7 +101,7 @@ macOS (OS X) requires slightly more preparation, including:
* using afl-clang (or afl-clang-fast from the llvm directory)
* disabling external crash reporting (AFL will guide you through this step)
== Triaging Issues
## Triaging Issues
Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code.
@ -115,7 +115,7 @@ To see what fuzz-http is doing with a test case, call it like this:
(Logging is disabled while fuzzing to increase fuzzing speed.)
== Reporting Issues
## Reporting Issues
Please report any issues discovered using the process in Tor's security issue
policy:

12
doc/HACKING/ReleasingTor.md
View File

@ -5,7 +5,7 @@ Putting out a new release
Here are the steps that the maintainer should take when putting out a
new Tor release:
=== 0. Preliminaries
### 0. Preliminaries
1. Get at least two of weasel/arma/Sebastian to put the new
version number in their approved versions list. Give them a few
@ -18,7 +18,7 @@ new Tor release:
date of a TB that contains it. See note below in "commit, upload,
announce".
=== I. Make sure it works
### I. Make sure it works
1. Make sure that CI passes: have a look at Travis
(https://travis-ci.org/torproject/tor/branches), Appveyor
@ -52,7 +52,7 @@ new Tor release:
memory leaks.)
=== II. Write a changelog
### II. Write a changelog
1a. (Alpha release variant)
@ -139,7 +139,7 @@ new Tor release:
text of existing entries, though.)
=== III. Making the source release.
### III. Making the source release.
1. In `maint-0.?.x`, bump the version number in `configure.ac` and run
`make update-versions` to update version numbers in other
@ -165,7 +165,7 @@ new Tor release:
If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see
item 0.1 at the start of this document.
=== IV. Commit, upload, announce
### IV. Commit, upload, announce
1. Sign the tarball, then sign and push the git tag:
@ -241,7 +241,7 @@ new Tor release:
For templates to use when announcing, see:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates
=== V. Aftermath and cleanup
### V. Aftermath and cleanup
1. If it's a stable release, bump the version number in the
`maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`