nihilist/tor
1
0
Fork 0
mirror of https://gitlab.torproject.org/tpo/core/tor.git synced 2024-11-24 04:13:28 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'tor-github/pr/1247'

Browse Source
This commit is contained in:
Nick Mathewson 2019-08-24 16:32:50 -04:00
commit 6ba05eea8e
4 changed files with 87 additions and 94 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

151
doc/HACKING/CodeStructure.md
View File

@ -2,128 +2,121 @@
TODO: revise this to talk about how things are, rather than how things TODO: revise this to talk about how things are, rather than how things
have changed. have changed.
TODO: Make this into good markdown. For quite a while now, the program *tor* has been built from source
code in just two directories: **src/common** and **src/or**.
For quite a while now, the program "tor" has been built from source
code in just two directories: src/common and src/or.
This has become more-or-less untenable, for a few reasons -- most This has become more-or-less untenable, for a few reasons -- most
notably of which is that it has led our code to become more notably of which is that it has led our code to become more
spaghetti-ish than I can endorse with a clean conscience. spaghetti-ish than I can endorse with a clean conscience.
So to fix that, we've gone and done a huge code movement in our git So to fix that, we've gone and done a huge code movement in our git
master branch, which will land in a release once Tor 0.3.5.1-alpha is master branch, which will land in a release once Tor `0.3.5.1-alpha` is
out. out.
Here's what we did: Here's what we did:
* src/common has been turned into a set of static libraries. These * **src/common** has been turned into a set of static libraries. These
all live in the "src/lib/*" directories. The dependencies between all live in the **src/lib/*** directories. The dependencies between
these libraries should have no cycles. The libraries are: these libraries should have no cycles. The libraries are:
arch -- Headers to handle architectural differences - **arch** -- Headers to handle architectural differences
cc -- headers to handle differences among compilers - **cc** -- headers to handle differences among compilers
compress -- wraps zlib, zstd, lzma - **compress** -- wraps zlib, zstd, lzma
container -- high-level container types - **container** -- high-level container types
crypt_ops -- Cryptographic operations. Planning to split this into - **crypt_ops** -- Cryptographic operations. Planning to split this into
a higher and lower level library a higher and lower level library
ctime -- Operations that need to run in constant-time. (Properly, - **ctime** -- Operations that need to run in constant-time. (Properly,
data-invariant time) data-invariant time)
defs -- miscelaneous definitions needed throughout Tor. - **defs** -- miscelaneous definitions needed throughout Tor.
encoding -- transforming one data type into another, and various - **encoding** -- transforming one data type into another, and various
data types into strings. data types into strings.
err -- lowest-level error handling, in cases where we can't use - **err** -- lowest-level error handling, in cases where we can't use
the logs because something that the logging system needs has broken. the logs because something that the logging system needs has broken.
evloop -- Generic event-loop handling logic - **evloop** -- Generic event-loop handling logic
fdio -- Low-level IO wrapper functions for file descriptors. - **fdio** -- Low-level IO wrapper functions for file descriptors.
fs -- Operations on the filesystem - **fs** -- Operations on the filesystem
intmath -- low-level integer math and misc bit-twiddling hacks - **intmath** -- low-level integer math and misc bit-twiddling hacks
lock -- low-level locking code - **lock** -- low-level locking code
log -- Tor's logging module. This library sits roughly halfway up - **log** -- Tor's logging module. This library sits roughly halfway up
the library dependency diagram, since everything it depends on has to the library dependency diagram, since everything it depends on has to
be carefully crafted to *not* log. be carefully crafted to *not* log.
malloc -- Low-level wrappers for the platform memory allocation functions. - **malloc** -- Low-level wrappers for the platform memory allocation functions.
math -- Higher-level mathematical functions, and floating-point math - **math** -- Higher-level mathematical functions, and floating-point math
memarea -- An arena allocator - **memarea** -- An arena allocator
meminfo -- Functions for querying the current process's memory - **meminfo** -- Functions for querying the current process's memory
status and resources status and resources
net -- Networking compatibility and convenience code - **net** -- Networking compatibility and convenience code
osinfo -- Querying information about the operating system - **osinfo** -- Querying information about the operating system
process -- Launching and querying the status of other processes - **process** -- Launching and querying the status of other processes
sandbox -- Backend for the linux seccomp2 sandbox - **sandbox** -- Backend for the linux seccomp2 sandbox
smartlist_core -- The lowest-level of the smartlist_t data type. - **smartlist_core** -- The lowest-level of the smartlist_t data type.
Separated from the rest of the containers library because the logging Separated from the rest of the containers library because the logging
subsystem depends on it. subsystem depends on it.
string -- Compatibility and convenience functions for manipulating - **string** -- Compatibility and convenience functions for manipulating
C strings. C strings.
term -- Terminal-related functions (currently limited to a getpass - **term** -- Terminal-related functions (currently limited to a getpass
function). function).
testsupport -- Macros for mocking, unit tests, etc. - **testsupport** -- Macros for mocking, unit tests, etc.
thread -- Higher-level thread compatibility code - **thread** -- Higher-level thread compatibility code
time -- Higher-level time management code, including format - **time** -- Higher-level time management code, including format
conversions and monotonic time conversions and monotonic time
tls -- Our wrapper around our TLS library - **tls** -- Our wrapper around our TLS library
trace -- Formerly src/trace -- a generic event tracing API - **trace** -- Formerly src/trace -- a generic event tracing API
wallclock -- Low-level time code, used by the log module. - **wallclock** -- Low-level time code, used by the log module.
* To ensure that the dependency graph in src/common remains under * To ensure that the dependency graph in **src/common** remains under
control, there is a tool that you can run called "make control, there is a tool that you can run called `make
check-includes". It verifies that each module in Tor only includes check-includes`. It verifies that each module in Tor only includes
the headers that it is permitted to include, using a per-directory the headers that it is permitted to include, using a per-directory
".may_include" file. *.may_include* file.
* The src/or/or.h header has been split into numerous smaller * The **src/or/or.h** header has been split into numerous smaller
headers. Notably, many important structures are now declared in a headers. Notably, many important structures are now declared in a
header called foo_st.h, where "foo" is the name of the structure. header called *foo_st.h*, where "foo" is the name of the structure.
* The src/or directory, which had most of Tor's code, had been split * The **src/or** directory, which had most of Tor's code, had been split
up into several directories. This is still a work in progress: This up into several directories. This is still a work in progress: This
code has not itself been refactored, and its dependency graph is still code has not itself been refactored, and its dependency graph is still
a tangled web. I hope we'll be working on that over the coming a tangled web. I hope we'll be working on that over the coming
releases, but it will take a while to do. releases, but it will take a while to do.
The new top-level source directories are: - The new top-level source directories are:
- **src/core** -- Code necessary to actually perform or use onion routing.
src/core -- Code necessary to actually perform or use onion routing. - **src/feature** -- Code used only by some onion routing
src/feature -- Code used only by some onion routing
configurations, or only for a special purpose. configurations, or only for a special purpose.
src/app -- Top-level code to run, invoke, and configure the - **src/app** -- Top-level code to run, invoke, and configure the
lower-level code lower-level code
The new second-level source directories are: - The new second-level source directories are:
src/core/crypto -- High-level cryptographic protocols used in Tor - **src/core/crypto** -- High-level cryptographic protocols used in Tor
src/core/mainloop -- Tor's event loop, connection-handling, and - **src/core/mainloop** -- Tor's event loop, connection-handling, and
traffic-routing code. traffic-routing code.
src/core/or -- Parts related to handling onion routing itself - **src/core/or** -- Parts related to handling onion routing itself
src/core/proto -- support for encoding and decoding different - **src/core/proto** -- support for encoding and decoding different
wire protocols wire protocols
- **src/feature/api** -- Support for making Tor embeddable
src/feature/api -- Support for making Tor embeddable - **src/feature/client** -- Functionality which only Tor clients need
src/feature/client -- Functionality which only Tor clients need - **src/feature/control** -- Controller implementation
src/feature/control -- Controller implementation - **src/feature/dirauth** -- Directory authority
src/feature/dirauth -- Directory authority - **src/feature/dircache** -- Directory cache
src/feature/dircache -- Directory cache - **src/feature/dirclient** -- Directory client
src/feature/dirclient -- Directory client - **src/feature/dircommon** -- Shared code between the other directory modules
src/feature/dircommon -- Shared code between the other directory modules - **src/feature/hibernate** -- Hibernating when Tor is out of bandwidth
src/feature/hibernate -- Hibernating when Tor is out of bandwidth
or shutting down or shutting down
src/feature/hs -- v3 onion service implementation - **src/feature/hs** -- v3 onion service implementation
src/feature/hs_common -- shared code between both onion service - **src/feature/hs_common** -- shared code between both onion service
implementations implementations
src/feature/nodelist -- storing and accessing the list of relays on - **src/feature/nodelist** -- storing and accessing the list of relays on
the network. the network.
src/feature/relay -- code that only relay servers and exit servers need. - **src/feature/relay** -- code that only relay servers and exit servers need.
src/feature/rend -- v2 onion service implementation - **src/feature/rend** -- v2 onion service implementation
src/feature/stats -- statistics and history - **src/feature/stats** -- statistics and history
- **src/app/config** -- configuration and state for Tor
- **src/app/main** -- Top-level functions to invoke the rest or Tor.
src/app/config -- configuration and state for Tor * The `tor` executable is now built in **src/app/tor** rather than **src/or/tor**.
src/app/main -- Top-level functions to invoke the rest or Tor.
* The "tor" executable is now built in src/app/tor rather than src/or/tor.
* There are more static libraries than before that you need to build * There are more static libraries than before that you need to build
into your application if you want to embed Tor. Rather than into your application if you want to embed Tor. Rather than
maintaining this list yourself, I recommend that you run "make maintaining this list yourself, I recommend that you run `make
show-libs" to have Tor emit a list of what you need to link. show-libs` to have Tor emit a list of what you need to link.

4
doc/HACKING/EndOfLifeTor.md
View File

@ -7,7 +7,7 @@ series reaches End of Life. Note that they are _only_ for entire series that
have reached their planned EOL: they do not apply to security-related have reached their planned EOL: they do not apply to security-related
deprecations of individual versions. deprecations of individual versions.
=== 0. Preliminaries ### 0. Preliminaries
0. A few months before End of Life: 0. A few months before End of Life:
Write a deprecation announcement. Write a deprecation announcement.
@ -17,7 +17,7 @@ deprecations of individual versions.
Send the announcement to tor-announce, tor-talk, tor-relays, and the Send the announcement to tor-announce, tor-talk, tor-relays, and the
packagers. packagers.
=== 1. On the day ### 1. On the day
1. Open tickets to remove the release from: 1. Open tickets to remove the release from:
- the jenkins builds - the jenkins builds

14
doc/HACKING/Fuzzing.md
View File

@ -1,6 +1,6 @@
= Fuzzing Tor # Fuzzing Tor
== The simple version (no fuzzing, only tests) ## The simple version (no fuzzing, only tests)
Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place
where you checked it out. where you checked it out.
@ -12,7 +12,7 @@ This won't actually fuzz Tor! It will just run all the fuzz binaries
on our existing set of testcases for the fuzzer. on our existing set of testcases for the fuzzer.
== Different kinds of fuzzing ## Different kinds of fuzzing
Right now we support three different kinds of fuzzer. Right now we support three different kinds of fuzzer.
@ -37,7 +37,7 @@ In all cases, you'll need some starting examples to give the fuzzer when it
starts out. There's a set in the "fuzzing-corpora" git repository. Try starts out. There's a set in the "fuzzing-corpora" git repository. Try
setting TOR_FUZZ_CORPORA to point to a checkout of that repository setting TOR_FUZZ_CORPORA to point to a checkout of that repository
== Writing Tor fuzzers ## Writing Tor fuzzers
A tor fuzzing harness should have: A tor fuzzing harness should have:
* a fuzz_init() function to set up any necessary global state. * a fuzz_init() function to set up any necessary global state.
@ -52,7 +52,7 @@ bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
"interesting" cases. "interesting" cases.
== Guided Fuzzing with AFL ## Guided Fuzzing with AFL
There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so
its integrity can't be verified. That said, you really shouldn't fuzz on a its integrity can't be verified. That said, you really shouldn't fuzz on a
@ -101,7 +101,7 @@ macOS (OS X) requires slightly more preparation, including:
* using afl-clang (or afl-clang-fast from the llvm directory) * using afl-clang (or afl-clang-fast from the llvm directory)
* disabling external crash reporting (AFL will guide you through this step) * disabling external crash reporting (AFL will guide you through this step)
== Triaging Issues ## Triaging Issues
Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code. Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code.
@ -115,7 +115,7 @@ To see what fuzz-http is doing with a test case, call it like this:
(Logging is disabled while fuzzing to increase fuzzing speed.) (Logging is disabled while fuzzing to increase fuzzing speed.)
== Reporting Issues ## Reporting Issues
Please report any issues discovered using the process in Tor's security issue Please report any issues discovered using the process in Tor's security issue
policy: policy:

12
doc/HACKING/ReleasingTor.md
View File

@ -5,7 +5,7 @@ Putting out a new release
Here are the steps that the maintainer should take when putting out a Here are the steps that the maintainer should take when putting out a
new Tor release: new Tor release:
=== 0. Preliminaries ### 0. Preliminaries
1. Get at least two of weasel/arma/Sebastian to put the new 1. Get at least two of weasel/arma/Sebastian to put the new
version number in their approved versions list. Give them a few version number in their approved versions list. Give them a few
@ -18,7 +18,7 @@ new Tor release:
date of a TB that contains it. See note below in "commit, upload, date of a TB that contains it. See note below in "commit, upload,
announce". announce".
=== I. Make sure it works ### I. Make sure it works
1. Make sure that CI passes: have a look at Travis 1. Make sure that CI passes: have a look at Travis
(https://travis-ci.org/torproject/tor/branches), Appveyor (https://travis-ci.org/torproject/tor/branches), Appveyor
@ -52,7 +52,7 @@ new Tor release:
memory leaks.) memory leaks.)
=== II. Write a changelog ### II. Write a changelog
1a. (Alpha release variant) 1a. (Alpha release variant)
@ -139,7 +139,7 @@ new Tor release:
text of existing entries, though.) text of existing entries, though.)
=== III. Making the source release. ### III. Making the source release.
1. In `maint-0.?.x`, bump the version number in `configure.ac` and run 1. In `maint-0.?.x`, bump the version number in `configure.ac` and run
`make update-versions` to update version numbers in other `make update-versions` to update version numbers in other
@ -165,7 +165,7 @@ new Tor release:
If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see
item 0.1 at the start of this document. item 0.1 at the start of this document.
=== IV. Commit, upload, announce ### IV. Commit, upload, announce
1. Sign the tarball, then sign and push the git tag: 1. Sign the tarball, then sign and push the git tag:
@ -241,7 +241,7 @@ new Tor release:
For templates to use when announcing, see: For templates to use when announcing, see:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates
=== V. Aftermath and cleanup ### V. Aftermath and cleanup
1. If it's a stable release, bump the version number in the 1. If it's a stable release, bump the version number in the
`maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours` `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`