Merge remote-tracking branch 'tor-github/pr/1832/head'

This commit is contained in:
Nick Mathewson 2020-03-27 09:15:29 -04:00
commit d9e042983f
17 changed files with 175 additions and 291 deletions

5
README
View File

@ -24,9 +24,8 @@ Making applications work with Tor:
Frequently Asked Questions:
https://www.torproject.org/docs/faq.html
Release timeline:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases
To get started working on Tor development:
See the doc/HACKING directory.
Release timeline:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases

View File

@ -47,6 +47,7 @@ If you want to install on your localsystem, run `make install`. For our case we
just want the tor binary at `src/app/tor`.
## Use tor in TB and at a relay
Download and install a fresh Tor Browser (TB) from torproject.org. Make sure it
works. From the command line, relative to the folder created when you extracted
TB, run `./Browser/start-tor-browser --verbose` to get some basic log output.
@ -75,6 +76,7 @@ relay. Start TB, visit a website, and manually confirm that the middle is used
by looking at the circuit display.
## Add a bare-bones APE padding machine
Now the fun part. We have several resources at our disposal (mind that links
might be broken in the future, just search for the headings):
- The official [Circuit Padding Developer
@ -192,6 +194,7 @@ register our machines:
We run `make` to get a new `tor` binary and copy it to our local TB.
## Run the machine
To be able
to view circuit info events in the console as we launch TB, we add `Log
[circ]info notice stdout` to `torrc` of TB.

View File

@ -1,3 +1,4 @@
# Code Structure
TODO: revise this to talk about how things are, rather than how things
have changed.

View File

@ -1,5 +1,4 @@
Coding conventions for Tor
==========================
# Coding conventions for Tor
tl;dr:
@ -10,8 +9,7 @@ tl;dr:
- Run `make distcheck` if you have made changes to build system components
- Add a file in `changes` for your branch.
Patch checklist
---------------
## Patch checklist
If possible, send your patch as one of these (in descending order of
preference)
@ -34,7 +32,7 @@ Did you remember...
If you are submitting a major patch or new feature, or want to in the future...
- Set up Chutney and Stem, see HACKING/WritingTests.md
- Set up Chutney and Stem, see `doc/HACKING/WritingTests.md`
- Run `make test-full` to test against all unit and integration tests.
If you have changed build system components:
@ -42,9 +40,7 @@ If you have changed build system components:
- For example, if you have changed Makefiles, autoconf files, or anything
else that affects the build system.
License issues
==============
## License issues
Tor is distributed under the license terms in the LICENSE -- in
brief, the "3-clause BSD license". If you send us code to
@ -58,9 +54,7 @@ Some compatible licenses include:
- 2-clause BSD
- CC0 Public Domain Dedication
How we use Git branches
=======================
## How we use Git branches
Each main development series (like 0.2.1, 0.2.2, etc) has its main work
applied to a single branch. At most one series can be the development series
@ -91,9 +85,7 @@ conflicts in the ChangeLog when it comes time to merge your branch into Tor.
Best advice: don't try to keep an independent branch forked for more than 6
months and expect it to merge cleanly. Try to merge pieces early and often.
How we log changes
==================
## How we log changes
When you do a commit that needs a ChangeLog entry, add a new file to
the `changes` toplevel subdirectory. It should have the format of a
@ -199,8 +191,7 @@ Why use changes files instead of entries in the ChangeLog?
* Having every single commit touch the ChangeLog file tended to create
zillions of merge conflicts.
Whitespace and C conformance
----------------------------
## Whitespace and C conformance
Invoke `make check-spaces` from time to time, so it can tell you about
deviations from our C whitespace style. Generally, we use:
@ -231,8 +222,7 @@ you're using gcc, you should invoke the configure script with the
option `--enable-fatal-warnings`. This will tell the compiler
to make all warnings into errors.
Functions to use; functions not to use
--------------------------------------
## Functions to use; functions not to use
We have some wrapper functions like `tor_malloc`, `tor_free`, `tor_strdup`, and
`tor_gettimeofday;` use them instead of their generic equivalents. (They
@ -250,7 +240,6 @@ available containers in `src/lib/containers/*.h`. You should probably
familiarize yourself with these modules before you write too much code, or
else you'll wind up reinventing the wheel.
We don't use `strcat` or `strcpy` or `sprintf` of any of those notoriously
broken old C functions. We also avoid `strncat` and `strncpy`. Use
`strlcat`, `strlcpy`, or `tor_snprintf/tor_asprintf` instead.
@ -281,9 +270,7 @@ version prefixed with `tor_` instead: strtok_r, memmem, memstr,
asprintf, localtime_r, gmtime_r, inet_aton, inet_ntop, inet_pton,
getpass, ntohll, htonll. (This list is incomplete.)
What code can use what other code?
----------------------------------
## What code can use what other code?
We're trying to simplify Tor's structure over time. In the long run, we want
Tor to be structured as a set of modules with *no circular dependencies*.
@ -300,8 +287,7 @@ included except those specifically permitted by the `.may_include` file.
When editing one of these files, please make sure that you are not
introducing any cycles into Tor's dependency graph.
Floating point math is hard
---------------------------
## Floating point math is hard
Floating point arithmetic as typically implemented by computers is
very counterintuitive. Failure to adequately analyze floating point
@ -360,8 +346,7 @@ For more detailed (and math-intensive) background, see [What Every
Computer Scientist Should Know About Floating-Point
Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html).
Other C conventions
-------------------
## Other C conventions
The `a ? b : c` trinary operator only goes inside other expressions;
don't use it as a replacement for if. (You can ignore this inside macro
@ -370,8 +355,7 @@ definitions when necessary.)
Assignment operators shouldn't nest inside other expressions. (You can
ignore this inside macro definitions when necessary.)
Binary data and wire formats
----------------------------
## Binary data and wire formats
Use pointer to `char` when representing NUL-terminated string. To represent
arbitrary binary data, use pointer to `uint8_t`. (Many older Tor APIs ignore
@ -390,8 +374,7 @@ for more information about trunnel.
For information on adding new trunnel code to Tor, see src/trunnel/README
Calling and naming conventions
------------------------------
## Calling and naming conventions
Whenever possible, functions should return -1 on error and 0 on success.
@ -414,17 +397,15 @@ probably time to create an enum. If you find that you are passing three or
more flags to a function, it's probably time to create a flags argument that
takes a bitfield.
What To Optimize
----------------
## What To Optimize
Don't optimize anything if it's not in the critical path. Right now, the
critical path seems to be AES, logging, and the network itself. Feel free to
do your own profiling to determine otherwise.
Log conventions
---------------
## Log conventions
`https://www.torproject.org/docs/faq#LogLevel`
[FAQ - Log Levels](https://www.torproject.org/docs/faq#LogLevel)
No error or warning messages should be expected during normal OR or OP
operation.
@ -438,8 +419,7 @@ end-users that they aren't expected to understand the message (perhaps
with a string like "internal error"). Option (A) is to be preferred to
option (B).
Assertions In Tor
-----------------
## Assertions In Tor
Assertions should be used for bug-detection only. Don't use assertions to
detect bad user inputs, network errors, resource exhaustion, or similar
@ -458,8 +438,7 @@ macro, as in:
if (BUG(ptr == NULL))
return -1;
Allocator conventions
---------------------
## Allocator conventions
By convention, any tor type with a name like `abc_t` should be allocated
by a function named `abc_new()`. This function should never return
@ -500,8 +479,7 @@ to use it as a function callback), define it with a name like
When deallocating, don't say e.g. `if (x) tor_free(x)`. The convention is to
have deallocators do nothing when NULL pointer is passed.
Doxygen comment conventions
---------------------------
## Doxygen comment conventions
Say what functions do as a series of one or more imperative sentences, as
though you were telling somebody how to be the function. In other words, DO

View File

@ -1,39 +1,36 @@
# Rust Coding Standards
Rust Coding Standards
=======================
You MUST follow the standards laid out in `.../doc/HACKING/CodingStandards.md`,
You MUST follow the standards laid out in `doc/HACKING/CodingStandards.md`,
where applicable.
Module/Crate Declarations
---------------------------
## Module/Crate Declarations
Each Tor C module which is being rewritten MUST be in its own crate.
See the structure of `.../src/rust` for examples.
See the structure of `src/rust` for examples.
In your crate, you MUST use `lib.rs` ONLY for pulling in external
crates (e.g. `extern crate libc;`) and exporting public objects from
other Rust modules (e.g. `pub use mymodule::foo;`). For example, if
you create a crate in `.../src/rust/yourcrate`, your Rust code should
live in `.../src/rust/yourcrate/yourcode.rs` and the public interface
to it should be exported in `.../src/rust/yourcrate/lib.rs`.
you create a crate in `src/rust/yourcrate`, your Rust code should
live in `src/rust/yourcrate/yourcode.rs` and the public interface
to it should be exported in `src/rust/yourcrate/lib.rs`.
If your code is to be called from Tor C code, you MUST define a safe
`ffi.rs`. See the "Safety" section further down for more details.
For example, in a hypothetical `tor_addition` Rust module:
In `.../src/rust/tor_addition/addition.rs`:
In `src/rust/tor_addition/addition.rs`:
pub fn get_sum(a: i32, b: i32) -> i32 {
a + b
}
In `.../src/rust/tor_addition/lib.rs`:
In `src/rust/tor_addition/lib.rs`:
pub use addition::*;
In `.../src/rust/tor_addition/ffi.rs`:
In `src/rust/tor_addition/ffi.rs`:
#[no_mangle]
pub extern "C" fn tor_get_sum(a: c_int, b: c_int) -> c_int {
@ -42,7 +39,7 @@ In `.../src/rust/tor_addition/ffi.rs`:
If your Rust code must call out to parts of Tor's C code, you must
declare the functions you are calling in the `external` crate, located
at `.../src/rust/external`.
at `src/rust/external`.
<!-- XXX get better examples of how to declare these externs, when/how they -->
<!-- XXX are unsafe, what they are expected to do —isis -->
@ -54,8 +51,7 @@ If you have any external modules as dependencies (e.g. `extern crate
libc;`), you MUST declare them in your crate's `lib.rs` and NOT in any
other module.
Dependencies and versions
---------------------------
## Dependencies and versions
In general, we use modules from only the Rust standard library
whenever possible. We will review including external crates on a
@ -81,8 +77,7 @@ Currently, Tor requires that you use the latest stable Rust version. At
some point in the future, we will freeze on a given stable Rust version,
to ensure backward compatibility with stable distributions that ship it.
Updating/Adding Dependencies
------------------------------
## Updating/Adding Dependencies
To add/remove/update dependencies, first add your dependencies,
exactly specifying their versions, into the appropriate *crate-level*
@ -101,14 +96,13 @@ Next, run `/scripts/maint/updateRustDependencies.sh`. Then, go into
`src/ext/rust` and commit the changes to the `tor-rust-dependencies`
repo.
Documentation
---------------
## Documentation
You MUST include `#![deny(missing_docs)]` in your crate.
For function/method comments, you SHOULD include a one-sentence, "first person"
description of function behaviour (see requirements for documentation as
described in `.../src/HACKING/CodingStandards.md`), then an `# Inputs` section
described in `src/HACKING/CodingStandards.md`), then an `# Inputs` section
for inputs or initialisation values, a `# Returns` section for return
values/types, a `# Warning` section containing warnings for unsafe behaviours or
panics that could happen. For publicly accessible
@ -118,14 +112,12 @@ types/constants/objects/functions/methods, you SHOULD also include an
You MUST document your module with _module docstring_ comments,
i.e. `//!` at the beginning of each line.
Style
-------
## Style
You SHOULD consider breaking up large literal numbers with `_` when it makes it
more human readable to do so, e.g. `let x: u64 = 100_000_000_000`.
Testing
---------
## Testing
All code MUST be unittested and integration tested.
@ -134,7 +126,7 @@ describing how the function/object is expected to be used.
Integration tests SHOULD go into a `tests/` directory inside your
crate. Unittests SHOULD go into their own module inside the module
they are testing, e.g. in `.../src/rust/tor_addition/addition.rs` you
they are testing, e.g. in `src/rust/tor_addition/addition.rs` you
should put:
#[cfg(test)]
@ -148,8 +140,7 @@ should put:
}
}
Benchmarking
--------------
## Benchmarking
The external `test` crate can be used for most benchmarking. However, using
this crate requires nightly Rust. Since we may want to switch to a more
@ -173,7 +164,7 @@ for basic benchmarks, is only used when running benchmarks via `cargo
bench --features bench`.
Finally, to write your benchmark code, in
`.../src/rust/tor_addition/addition.rs` you SHOULD put:
`src/rust/tor_addition/addition.rs` you SHOULD put:
#[cfg(all(test, features = "bench"))]
mod bench {
@ -186,23 +177,20 @@ Finally, to write your benchmark code, in
}
}
Fuzzing
---------
## Fuzzing
If you wish to fuzz parts of your code, please see the
[`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz) crate, which uses
[cargo fuzz](https://github.com/rust-fuzz/cargo-fuzz) crate, which uses
[libfuzzer-sys](https://github.com/rust-fuzz/libfuzzer-sys).
Whitespace & Formatting
-------------------------
## Whitespace & Formatting
You MUST run `rustfmt` (https://github.com/rust-lang-nursery/rustfmt)
on your code before your code will be merged. You can install rustfmt
by doing `cargo install rustfmt-nightly` and then run it with `cargo
fmt`.
Safety
--------
## Safety
You SHOULD read [the nomicon](https://doc.rust-lang.org/nomicon/) before writing
Rust FFI code. It is *highly advised* that you read and write normal Rust code

View File

@ -1,13 +1,11 @@
End of Life on an old release series
------------------------------------
# End of Life on an old release series
Here are the steps that the maintainer should take when an old Tor release
series reaches End of Life. Note that they are _only_ for entire series that
have reached their planned EOL: they do not apply to security-related
deprecations of individual versions.
### 0. Preliminaries
## 0. Preliminaries
0. A few months before End of Life:
Write a deprecation announcement.
@ -17,7 +15,7 @@ deprecations of individual versions.
Send the announcement to tor-announce, tor-talk, tor-relays, and the
packagers.
### 1. On the day
## 1. On the day
1. Open tickets to remove the release from:
- the jenkins builds

View File

@ -11,7 +11,6 @@ To run the fuzzing test cases in a deterministic fashion, use:
This won't actually fuzz Tor! It will just run all the fuzz binaries
on our existing set of testcases for the fuzzer.
## Different kinds of fuzzing
Right now we support three different kinds of fuzzer.
@ -51,7 +50,6 @@ But the fuzzing harness should crash if tor fails an assertion, triggers a
bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
"interesting" cases.
## Guided Fuzzing with AFL
There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so

View File

@ -1,24 +1,19 @@
Getting started in Tor development
==================================
# Getting started in Tor development
Congratulations! You've found this file, and you're reading it! This
means that you might be interested in getting started in developing Tor.
(This guide is just about Tor itself--the small network program at the
(_This guide is just about Tor itself--the small network program at the
heart of the Tor network--and not about all the other programs in the
whole Tor ecosystem.)
whole Tor ecosystem._)
If you are looking for a more bare-bones, less user-friendly information
dump of important information, you might like reading the
doxygen output at https://src-ref.docs.torproject.org/tor/index.html .
You should probably skim some of the topic headings there before you write
[doxygen output](https://src-ref.docs.torproject.org/tor/index.html).
You probably should skim some of the topic headings there before you write
your first patch.
Required background
-------------------
## Required background
First, I'm going to assume that you can build Tor from source, and that
you know enough of the C language to read and write it. (See the README
@ -27,22 +22,20 @@ and any high-quality guide to C for information on programming.)
I'm also going to assume that you know a little bit about how to use
Git, or that you're able to follow one of the several excellent guides
at http://git-scm.org to learn.
at [git-scm](http://git-scm.org) to learn.
Most Tor developers develop using some Unix-based system, such as Linux,
BSD, or OSX. It's okay to develop on Windows if you want, but you're
Most Tor developers develop using some Unix-based system, such as GNU/Linux,
BSD, or macOS. It's okay to develop on Windows if you want, but you're
going to have a more difficult time.
Getting your first patch into Tor
---------------------------------
## Getting your first patch into Tor
Once you've reached this point, here's what you need to know.
1. Get the source.
We keep our source under version control in Git. To get the latest
version, run
version, run:
git clone https://git.torproject.org/git/tor
@ -52,18 +45,16 @@ Once you've reached this point, here's what you need to know.
git checkout maint-0.4.3
2. Find your way around the source
2. Find your way around the source.
Our overall code structure is explained in our source documentation,
currently at
https://src-ref.docs.torproject.org/tor/index.html
Our overall code structure is explained in our
[source documentation](https://src-ref.docs.torproject.org/tor/index.html).
Find a part of the code that looks interesting to you, and start
looking around it to see how it fits together!
We do some unusual things in our codebase. Our testing-related
practices and kludges are explained in doc/WritingTests.txt.
practices and kludges are explained in `doc/HACKING/WritingTests.md`.
If you see something that doesn't make sense, we love to get
questions!
@ -75,10 +66,10 @@ Once you've reached this point, here's what you need to know.
Many people have gotten started by looking for an area where they
personally felt Tor was underperforming, and investigating ways to
fix it. If you're looking for ideas, you can head to our bug
tracker at trac.torproject.org and look for tickets that have
received the "easy" tag: these are ones that developers think would
be pretty simple for a new person to work on. For a bigger
fix it. If you're looking for ideas, you can head to
[trac](https://trac.torproject.org) our bug tracking tool and look for
tickets that have received the "easy" tag: these are ones that developers
think would be pretty simple for a new person to work on. For a bigger
challenge, you might want to look for tickets with the "lorax"
keyword: these are tickets that the developers think might be a
good idea to build, but which we have no time to work on any time
@ -97,7 +88,7 @@ Once you've reached this point, here's what you need to know.
4. Meet the developers!
We discuss stuff on the tor-dev mailing list and on the #tor-dev
We discuss stuff on the tor-dev mailing list and on the `#tor-dev`
IRC channel on OFTC. We're generally friendly and approachable,
and we like to talk about how Tor fits together. If we have ideas
about how something should be implemented, we'll be happy to share
@ -114,8 +105,8 @@ Once you've reached this point, here's what you need to know.
protocols, there needs to be a written design proposal before it
can be merged. (We use this process to manage changes in the
protocols.) To write one, see the instructions at
https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt
. If you'd like help writing a proposal, just ask! We're happy to
[the Tor proposal process](https://gitweb.torproject.org/torspec.git/plain/proposals/001-process.txt).
If you'd like help writing a proposal, just ask! We're happy to
help out with good ideas.
You might also like to look around the rest of that directory, to
@ -126,7 +117,7 @@ Once you've reached this point, here's what you need to know.
As you write your code, you'll probably want it to fit in with the
standards of the rest of the Tor codebase so it will be easy for us
to review and merge. You can learn our coding standards in
doc/HACKING.
`doc/HACKING` directory.
If your patch is large and/or is divided into multiple logical
components, remember to divide it into a series of Git commits. A
@ -138,16 +129,16 @@ Once you've reached this point, here's what you need to know.
ensure that it runs correctly. Also, all code should actually be
_run_ by somebody, to make sure it works.
See doc/WritingTests.txt for more information on how we test things
See `doc/HACKING/WritingTests.md` for more information on how we test things
in Tor. If you'd like any help writing tests, just ask! We're
glad to help out.
8. Submitting your patch
We review patches through tickets on our bugtracker at
trac.torproject.org. You can either upload your patches there, or
[trac](https://trac.torproject.org). You can either upload your patches there, or
put them at a public git repository somewhere we can fetch them
(like github or bitbucket) and then paste a link on the appropriate
(like gitlab, github or bitbucket) and then paste a link on the appropriate
trac ticket.
Once your patches are available, write a short explanation of what
@ -164,17 +155,17 @@ Once you've reached this point, here's what you need to know.
When your patch is reviewed, one of these things will happen:
* The reviewer will say "looks good to me" and your
* The reviewer will say "_looks good to me_" and your
patch will get merged right into Tor. [Assuming we're not
in the middle of a code-freeze window. If the codebase is
frozen, your patch will go into the next release series.]
* OR the reviewer will say "looks good, just needs some small
changes!" And then the reviewer will make those changes,
* OR the reviewer will say "_looks good, just needs some small
changes!_" And then the reviewer will make those changes,
and merge the modified patch into Tor.
* OR the reviewer will say "Here are some questions and
comments," followed by a bunch of stuff that the reviewer
* OR the reviewer will say "_Here are some questions and
comments,_" followed by a bunch of stuff that the reviewer
thinks should change in your code, or questions that the
reviewer has.

View File

@ -1,12 +1,9 @@
# Hacking on Rust in Tor
Hacking on Rust in Tor
========================
Getting Started
-----------------
## Getting Started
Please read or review our documentation on Rust coding standards
(`.../doc/HACKING/CodingStandardsRust.md`) before doing anything.
(`doc/HACKING/CodingStandardsRust.md`) before doing anything.
Please also read
[the Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html). We
@ -23,8 +20,7 @@ Please be patient with the other people who are working on getting more
Rust code into Tor, because they are graciously donating their free time
to contribute to this effort.
Resources for learning Rust
-----------------------------
## Resources for learning Rust
**Beginning resources**
@ -49,8 +45,7 @@ is
For learning more about FFI and Rust, see Jake Goulding's
[Rust FFI Omnibus](http://jakegoulding.com/rust-ffi-omnibus/).
Compiling Tor with Rust enabled
---------------------------------
## Compiling Tor with Rust enabled
You will need to run the `configure` script with the `--enable-rust`
flag to explicitly build with Rust. Additionally, you will need to
@ -79,7 +74,7 @@ you are in the top level of the repository) configure tor with:
TOR_RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust
(Note that TOR_RUST_DEPENDENCIES must be the full path to the directory; it
(Note that `TOR_RUST_DEPENDENCIES` must be the full path to the directory; it
cannot be relative.)
Assuming you used the above `git submodule` commands and you're in the
@ -87,9 +82,7 @@ topmost directory of the repository, this would be:
TOR_RUST_DEPENDENCIES=`pwd`/src/ext/rust/crates ./configure --enable-rust
Identifying which modules to rewrite
======================================
## Identifying which modules to rewrite
The places in the Tor codebase that are good candidates for porting to
Rust are:
@ -117,12 +110,11 @@ interconnected your target module is.
The output will tell you each module name, along with a set of every module that
the module calls. Modules which call fewer other modules are better targets.
Writing your Rust module
==========================
## Writing your Rust module
Strive to change the C API as little as possible.
We are currently targetting Rust stable. (See CodingStandardsRust.md for more
We are currently targetting Rust stable. (See `CodingStandardsRust.md` for more
details.)
It is on our TODO list to try to cultivate good
@ -134,19 +126,17 @@ If parts of your Rust code needs to stay in sync with C code (such as
handling enums across the FFI boundary), annonotate these places in a
comment structured as follows:
/// C_RUST_COUPLED: <path_to_file> `<name_of_c_object>`
`/// C_RUST_COUPLED: <path_to_file> <name_of_c_object>`
Where <name_of_c_object> can be an enum, struct, constant, etc. Then,
Where `<name_of_c_object>` can be an enum, struct, constant, etc. Then,
do the same in the C code, to note that rust will need to be changed
when the C does.
Adding your Rust module to Tor's build system
-----------------------------------------------
## Adding your Rust module to Tor's build system
0. Your translation of the C module should live in its own crate(s)
in the `.../tor/src/rust/` directory.
1. Add your crate to `.../tor/src/rust/Cargo.toml`, in the
in the `src/rust/` directory.
1. Add your crate to `src/rust/Cargo.toml`, in the
`[workspace.members]` section.
2. Add your crate's files to src/rust/include.am
@ -156,12 +146,11 @@ dependency of other Rust modules):
`src/rust/tor_util/Cargo.toml` and include it in
`src/rust/tor_rust/lib.rs`
How to test your Rust code
----------------------------
## How to test your Rust code
Everything should be tested full stop. Even non-public functionality.
Be sure to edit `.../tor/src/test/test_rust.sh` to add the name of your
Be sure to edit `src/test/test_rust.sh` to add the name of your
crate to the `crates` variable! This will ensure that `cargo test` is
run on your crate.
@ -177,7 +166,6 @@ Tor's integration tests should also pass:
make test-stem
Submitting a patch
=====================
## Submitting a patch
Please follow the instructions in `.../doc/HACKING/GettingStarted.md`.
Please follow the instructions in `doc/HACKING/GettingStarted.md`.

View File

@ -1,11 +1,10 @@
Useful tools
============
# Useful tools
These aren't strictly necessary for hacking on Tor, but they can help track
down bugs.
Travis/Appveyor CI
------------------
## Travis/Appveyor CI
It's CI.
Looks like this:
@ -29,8 +28,7 @@ for your fork to build commits outside of PRs too:
Builds should show up on the web at travis-ci.com and on IRC at #tor-ci on
OFTC. If they don't, ask #tor-dev (also on OFTC).
Jenkins
-------
## Jenkins
It's CI/builders. Looks like this: https://jenkins.torproject.org
@ -43,8 +41,7 @@ Builds Linux and Windows cross-compilation. Runs Linux tests.
Builds should show up on the web at jenkins.torproject.org and on IRC at
#tor-bots on OFTC. If they don't, ask #tor-dev (also on OFTC).
Valgrind
--------
## Valgrind
valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/app/tor
@ -52,16 +49,14 @@ Valgrind
pass `--undef-value-errors=no` to valgrind, or rebuild your openssl
with `-DPURIFY`.)
Coverity
--------
## Coverity
Nick regularly runs the coverity static analyzer on the Tor codebase.
The preprocessor define `__COVERITY__` is used to work around instances
where coverity picks up behavior that we wish to permit.
clang Static Analyzer
---------------------
## clang Static Analyzer
The clang static analyzer can be run on the Tor codebase using Xcode (WIP)
or a command-line build.
@ -69,8 +64,7 @@ or a command-line build.
The preprocessor define `__clang_analyzer__` is used to work around instances
where clang picks up behavior that we wish to permit.
clang Runtime Sanitizers
------------------------
## clang Runtime Sanitizers
To build the Tor codebase with the clang Address and Undefined Behavior
sanitizers, see the file `contrib/clang/sanitize_blacklist.txt`.
@ -78,8 +72,7 @@ sanitizers, see the file `contrib/clang/sanitize_blacklist.txt`.
Preprocessor workarounds for instances where clang picks up behavior that
we wish to permit are also documented in the blacklist file.
Running lcov for unit test coverage
-----------------------------------
## Running lcov for unit test coverage
Lcov is a utility that generates pretty HTML reports of test code coverage.
To generate such a report:
@ -96,8 +89,7 @@ output directory, use `make coverage-html HTML_COVER_DIR=./funky_new_cov_dir`.
Coverage diffs using lcov are not currently implemented, but are being
investigated (as of July 2014).
Running the unit tests
----------------------
## Running the unit tests
To quickly run all the tests distributed with Tor:
@ -123,8 +115,7 @@ working connection to the internet:
make test-full-online
Running gcov for unit test coverage
-----------------------------------
## Running gcov for unit test coverage
./configure --enable-coverage
make
@ -164,8 +155,7 @@ lines?
If you run ./scripts/test/cov-exclude, it marks excluded unreached
lines with 'x', and excluded reached lines with '!!!'.
Running integration tests
-------------------------
## Running integration tests
We have the beginnings of a set of scripts to run integration tests using
Chutney. To try them, set CHUTNEY_PATH to your chutney source directory, and
@ -174,14 +164,12 @@ run `make test-network`.
We also have scripts to run integration tests using Stem. To try them, set
`STEM_SOURCE_DIR` to your Stem source directory, and run `test-stem`.
Profiling Tor
-------------
## Profiling Tor
Ongoing notes about Tor profiling can be found at
https://pad.riseup.net/p/profiling-tor
Profiling Tor with oprofile
---------------------------
## Profiling Tor with oprofile
The oprofile tool runs (on Linux only!) to tell you what functions Tor is
spending its CPU time in, so we can identify performance bottlenecks.
@ -206,8 +194,7 @@ Here are some basic instructions
* `opreport -l that_dir/*`
- Profit
Profiling Tor with perf
-----------------------
## Profiling Tor with perf
This works with a running Tor, and requires root.
@ -236,8 +223,7 @@ This works with a running Tor, and requires root.
report -g > <FILENAME>.out`. Then you can compress that and put it somewhere
public.
Profiling Tor with gperftools aka Google-performance-tools
----------------------------------------------------------
## Profiling Tor with gperftools aka Google-performance-tools
This should work on nearly any unixy system. It doesn't seem to be compatible
with RunAsDaemon though.
@ -255,8 +241,7 @@ performance! See the gperftools manual for more info, but basically:
3. Run `pprof src/app/tor /tmp/profile` to start the REPL.
Generating and analyzing a callgraph
------------------------------------
## Generating and analyzing a callgraph
0. Build Tor on linux or mac, ideally with -O0 or -fno-inline.
@ -266,8 +251,7 @@ Generating and analyzing a callgraph
Note that currently the callgraph generator can't detect calls that pass
through function pointers.
Getting emacs to edit Tor source properly
-----------------------------------------
## Getting emacs to edit Tor source properly
Nick likes to put the following snippet in his .emacs file:
@ -315,8 +299,7 @@ If you use emacs for editing Tor and nothing else, you could always just say:
There is probably a better way to do this. No, we are probably not going
to clutter the files with emacs stuff.
Building a tag file (code index)
--------------------------------
## Building a tag file (code index)
Many functions in tor use `MOCK_IMPL` wrappers for unit tests. Your
tag-building program must be told how to handle this syntax.
@ -340,8 +323,7 @@ instead:
A vim-compatible tag file will be generated by default. If you use emacs, add
the `-e` flag to generate an emacs-compatible tag file.
Doxygen
-------
## Doxygen
We use the 'doxygen' utility to generate documentation from our
source code. Here's how to use it:
@ -396,8 +378,7 @@ source code. Here's how to use it:
6. See the Doxygen manual for more information; this summary just
scratches the surface.
Style and best-practices checking
--------------------------------
## Style and best-practices checking
We use scripts to check for various problems in the formatting and style
of our source code. The "check-spaces" test detects a bunch of violations

View File

@ -1,5 +1,4 @@
How to review a patch
=====================
# How to review a patch
Some folks have said that they'd like to review patches more often, but they
don't know how.
@ -9,9 +8,7 @@ So, here are a bunch of things to check for when reviewing a patch!
Note that if you can't do every one of these, that doesn't mean you can't do
a good review! Just make it clear what you checked for and what you didn't.
Top-level smell-checks
----------------------
## Top-level smell-checks
(Difficulty: easy)
@ -37,10 +34,9 @@ Top-level smell-checks
- If this changes anything in the code, is there a "changes" file?
Let's look at the code!
-----------------------
## Let's look at the code!
- Does the code conform to CodingStandards.txt?
- Does the code conform to `CodingStandards.md`?
- Does the code leak memory?
@ -60,8 +56,7 @@ Let's look at the code!
- Is there duplicated code that could be turned into a function?
Let's look at the documentation!
--------------------------------
## Let's look at the documentation!
- Does the documentation conform to CodingStandards.txt?
@ -70,8 +65,7 @@ Let's look at the documentation!
- Can you predict what the function will do from its documentation?
Let's think about security!
---------------------------
## Let's think about security!
- If there are any arrays, buffers, are you 100% sure that they cannot
overflow?

View File

@ -1,9 +1,9 @@
# Modules in Tor #
# Modules in Tor
This document describes the build system and coding standards when writing a
module in Tor.
## What is a module? ##
## What is a module?
In the context of the tor code base, a module is a subsystem that we can
selectively enable or disable, at `configure` time.
@ -27,7 +27,7 @@ because you would actually want to run one without the other.)
To disable a module, pass `--disable-module-{dirauth,relay}` at configure
time. All modules are currently enabled by default.
## Build System ##
## Build System
The changes to the build system are pretty straightforward.
@ -55,7 +55,7 @@ Finally, your module will automatically be included in the
`TOR_MODULES_ALL_ENABLED` variable which is used to build the unit tests.
They always build everything in order to test everything.
## Coding ##
## Coding
As mentioned above, a module should be isolated in its own directories,
suffixed with the name of the module, in `src/*/`.

View File

@ -1,17 +1,18 @@
# README.1st
In this directory
-----------------
## In this directory
This directory has helpful information about what you need to know to
hack on Tor!
First, read `GettingStarted.md` to learn how to get a start in Tor
development.
First, read `GettingStarted.md` and `GettingStartedRust.md`
to learn how to get a start in Tor development.
If you've decided to write a patch, `CodingStandards.txt` will give
you a bunch of information about how we structure our code.
If you've decided to write a patch, `CodingStandards.md` and
`CodingStandardsRust.md` will give you a bunch of information
about how we structure our code.
It's important to get code right! Reading `WritingTests.md` will
It's important to get the code right! Reading `WritingTests.md` will
tell you how to write and run tests in the Tor codebase.
There are a bunch of other programs we use to help maintain and
@ -21,42 +22,26 @@ with Tor.
If it's your job to put out Tor releases, see `ReleasingTor.md` so
that you don't miss any steps!
-----------------------
## Additional Information
For full information on how Tor is supposed to work, look at the files in
`https://gitweb.torproject.org/torspec.git/tree`.
[Tor specification](https://gitweb.torproject.org/torspec.git/tree).
For an explanation of how to change Tor's design to work differently, look at
`https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/001-process.txt`.
[the Tor proposal process](https://gitweb.torproject.org/torspec.git/plain/proposals/001-process.txt).
For the latest version of the code, get a copy of git, and
git clone https://git.torproject.org/git/tor
## Stay in touch
We talk about Tor on the `tor-talk` mailing list. Design proposals and
discussion belong on the `tor-dev` mailing list. We hang around on
irc.oftc.net, with general discussion happening on #tor and development
irc.oftc.net, with general discussion happening on `#tor` and development
happening on `#tor-dev`.
The other files in this `HACKING` directory may also be useful as you
get started working with Tor.
Happy hacking!
-----------------------
XXXXX also describe
doc/HACKING/WritingTests.md
torguts.git
torspec.git
The design paper
freehaven.net/anonbib
XXXX describe these and add links.

View File

@ -1,11 +1,9 @@
Putting out a new release
-------------------------
# Putting out a new release
Here are the steps that the maintainer should take when putting out a
new Tor release:
### 0. Preliminaries
## 0. Preliminaries
1. Get at least two of weasel/arma/Sebastian to put the new
version number in their approved versions list. Give them a few
@ -18,7 +16,7 @@ new Tor release:
date of a TB that contains it. See note below in "commit, upload,
announce".
### I. Make sure it works
## I. Make sure it works
1. Make sure that CI passes: have a look at Travis
(https://travis-ci.org/torproject/tor/branches), Appveyor
@ -51,9 +49,7 @@ new Tor release:
libevent and openssl, so using valgrind will sometimes find extra
memory leaks.)
### II. Write a changelog
## II. Write a changelog
1a. (Alpha release variant)
@ -138,8 +134,7 @@ new Tor release:
to start sorting and condensing entries. (Generally, we don't edit the
text of existing entries, though.)
### III. Making the source release.
## III. Making the source release.
1. In `maint-0.?.x`, bump the version number in `configure.ac` and run
`make update-versions` to update version numbers in other
@ -165,7 +160,7 @@ new Tor release:
If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see
item 0.1 at the start of this document.
### IV. Commit, upload, announce
## IV. Commit, upload, announce
1. Sign the tarball, then sign and push the git tag:
@ -215,7 +210,6 @@ new Tor release:
Include a link to the changelog.
4. Add the version number to Trac. To do this, go to Trac, log in,
select "Admin" near the top of the screen, then select "Versions" from
the menu on the left. At the right, there will be an "Add version"
@ -241,7 +235,7 @@ new Tor release:
For templates to use when announcing, see:
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates
### V. Aftermath and cleanup
## V. Aftermath and cleanup
1. If it's a stable release, bump the version number in the
`maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours`

View File

@ -1,16 +1,16 @@
# Tracing #
# Tracing
This document describes how the event tracing subsystem works in tor so
developers can add events to the code base but also hook them to an event
tracing framework.
## Basics ###
## Basics
Event tracing is separated in two concepts, trace events and a tracer. The
tracing subsystem can be found in `src/trace`. The `events.h` header file is
the main file that maps the different tracers to trace events.
### Events ###
### Events
A trace event is basically a function from which we can pass any data that
we want to collect. In addition, we specify a context for the event such as
@ -39,7 +39,7 @@ How `argc` is collected or used has nothing to do with the instrumentation
the trace events and collection framework (tracer) are decoupled. You _can_
have trace events without a tracer.
### Tracer ###
### Tracer
In `src/trace/events.h`, we map the `tor_trace()` function to the right
tracer. A tracer support is only enabled at compile time. For instance, the
@ -47,7 +47,7 @@ file `src/trace/debug.h` contains the mapping of the generic tracing function
`tor_trace()` to the `log_debug()` function. More specialized function can be
mapped depending on the tracepoint.
## Build System ##
## Build System
This section describes how it is integrated into the build system of tor.
@ -66,7 +66,7 @@ configure option:
--enable-tracing-debug
## Instrument Tor ##
## Instrument Tor
This is pretty easy. Let's say you want to add a trace event in
`src/feature/rend/rendcache.c`, you only have to add this include statement:

View File

@ -1,6 +1,4 @@
Writing tests for Tor: an incomplete guide
==========================================
# Writing tests for Tor: an incomplete guide
Tor uses a variety of testing frameworks and methodologies to try to
keep from introducing bugs. The major ones are:
@ -19,8 +17,7 @@ keep from introducing bugs. The major ones are:
5. The Shadow network simulator.
How to run these tests
----------------------
## How to run these tests
### The easy version
@ -64,7 +61,7 @@ The former are those that should finish in a few seconds; the latter tend to
take more time, and may include CPU-intensive operations, deliberate delays,
and stuff like that.
### Finding test coverage
## Finding test coverage
Test coverage is a measurement of which lines your tests actually visit.
@ -112,7 +109,7 @@ To count new or modified uncovered lines in D2, you can run:
./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' | wc -l
### Marking lines as unreachable by tests
## Marking lines as unreachable by tests
You can mark a specific line as unreachable by using the special
string LCOV_EXCL_LINE. You can mark a range of lines as unreachable
@ -126,9 +123,7 @@ unreached lines with 'x', and excluded reached lines with '!!!'.
Note: you should never do this unless the line is meant to 100%
unreachable by actual code.
What kinds of test should I write?
----------------------------------
## What kinds of test should I write?
Integration testing and unit testing are complementary: it's probably a
good idea to make sure that your code is hit by both if you can.
@ -143,8 +138,7 @@ If your code adds new externally visible functionality to Tor, it would
be great to have a test for that functionality. That's where
integration tests more usually come in.
Unit and regression tests: Does this function do what it's supposed to?
-----------------------------------------------------------------------
## Unit and regression tests: Does this function do what it's supposed to?
Most of Tor's unit tests are made using the "tinytest" testing framework.
You can see a guide to using it in the tinytest manual at
@ -165,7 +159,7 @@ If you have created a new test file, you will need to:
I use the term "unit test" and "regression tests" very sloppily here.
### A simple example
## A simple example
Here's an example of a test function for a simple function in util.c:
@ -207,7 +201,7 @@ Finally, remember that by convention, all `*_free()` functions that
Tor defines are defined to accept NULL harmlessly. Thus, you don't
need to say `if (contents)` in the cleanup block.
### Exposing static functions for testing
## Exposing static functions for testing
Sometimes you need to test a function, but you don't want to expose
it outside its usual module.
@ -228,7 +222,7 @@ For example, `crypto_curve25519.h` contains:
The `crypto_curve25519.c` file and the `test_crypto.c` file both define
`CRYPTO_CURVE25519_PRIVATE`, so they can see this declaration.
### STOP! Does this test really test?
## STOP! Does this test really test?
When writing tests, it's not enough to just generate coverage on all the
lines of the code that you're testing: It's important to make sure that
@ -269,8 +263,7 @@ it's supposed to do, and fail otherwise. Try to design your tests so
that they check for the code's intended and documented functionality
as much as possible.
### Mock functions for testing in isolation
## Mock functions for testing in isolation
Often we want to test that a function works right, but the function to
be tested depends on other functions whose behavior is hard to observe,
@ -311,7 +304,7 @@ And later, you can restore the original function with:
For more information, see the definitions of this mocking logic in
`testsupport.h`.
### Okay but what should my tests actually do?
## Okay but what should my tests actually do?
We talk above about "test coverage" -- making sure that your tests visit
every line of code, or every branch of code. But visiting the code isn't
@ -382,8 +375,7 @@ Based on the implementation, we now see three more edge cases to test:
* Removing an element from the end of the list
* Removing an element from a position other than the end of the list.
### What should my tests NOT do?
## What should my tests NOT do?
Tests shouldn't require a network connection.
@ -401,8 +393,7 @@ When possible, tests should not be over-fit to the implementation. That is,
the test should verify that the documented behavior is implemented, but
should not break if other permissible behavior is later implemented.
### Advanced techniques: Namespaces
## Advanced techniques: Namespaces
Sometimes, when you're doing a lot of mocking at once, it's convenient to
isolate your identifiers within a single namespace. If this were C++, we'd
@ -414,9 +405,7 @@ them, you define `NS_MODULE` to a prefix to be used for your identifiers, and
then use other macros in place of identifier names. See `src/test/test.h` for
more documentation.
Integration tests: Calling Tor from the outside
-----------------------------------------------
## Integration tests: Calling Tor from the outside
Some tests need to invoke Tor from the outside, and shouldn't run from the
same process as the Tor test program. Reasons for doing this might include:
@ -436,8 +425,7 @@ wrapped, add a new shell script to `TESTS`, and the new program to
makefile (eg `${PYTHON}` for a python interpreter), then make sure that the
makefile exports them.
Writing integration tests with Stem
-----------------------------------
## Writing integration tests with Stem
The 'stem' library includes extensive tests for the Tor controller protocol.
You can run stem tests from tor with `make test-stem`, or see
@ -483,8 +471,7 @@ you notice any strange behaviour that seems totally unreasonable.
Check out the `test_exit_policy()` function in abovementioned file to see the
final implementation for this test.
System testing with Chutney
---------------------------
## System testing with Chutney
The 'chutney' program configures and launches a set of Tor relays,
authorities, and clients on your local host. It has a `test network`

View File

@ -95,4 +95,3 @@ was spend on the call.
Start Tor the normal way via Orbot and collect the logs from your computer using
$ adb logcat