diff --git a/README b/README index 9e0f4b8241..9bbf027d70 100644 --- a/README +++ b/README @@ -24,9 +24,8 @@ Making applications work with Tor: Frequently Asked Questions: https://www.torproject.org/docs/faq.html +Release timeline: + https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases To get started working on Tor development: See the doc/HACKING directory. - -Release timeline: - https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases diff --git a/doc/HACKING/CircuitPaddingQuickStart.md b/doc/HACKING/CircuitPaddingQuickStart.md index 167ff9f292..2780b5c6ea 100644 --- a/doc/HACKING/CircuitPaddingQuickStart.md +++ b/doc/HACKING/CircuitPaddingQuickStart.md @@ -47,6 +47,7 @@ If you want to install on your localsystem, run `make install`. For our case we just want the tor binary at `src/app/tor`. ## Use tor in TB and at a relay + Download and install a fresh Tor Browser (TB) from torproject.org. Make sure it works. From the command line, relative to the folder created when you extracted TB, run `./Browser/start-tor-browser --verbose` to get some basic log output. @@ -75,6 +76,7 @@ relay. Start TB, visit a website, and manually confirm that the middle is used by looking at the circuit display. ## Add a bare-bones APE padding machine + Now the fun part. We have several resources at our disposal (mind that links might be broken in the future, just search for the headings): - The official [Circuit Padding Developer @@ -192,6 +194,7 @@ register our machines: We run `make` to get a new `tor` binary and copy it to our local TB. ## Run the machine + To be able to view circuit info events in the console as we launch TB, we add `Log [circ]info notice stdout` to `torrc` of TB. diff --git a/doc/HACKING/CodeStructure.md b/doc/HACKING/CodeStructure.md index fffafcaed1..d387018f9b 100644 --- a/doc/HACKING/CodeStructure.md +++ b/doc/HACKING/CodeStructure.md @@ -1,3 +1,4 @@ +# Code Structure TODO: revise this to talk about how things are, rather than how things have changed. diff --git a/doc/HACKING/CodingStandards.md b/doc/HACKING/CodingStandards.md index 7999724166..a181759d60 100644 --- a/doc/HACKING/CodingStandards.md +++ b/doc/HACKING/CodingStandards.md @@ -1,5 +1,4 @@ -Coding conventions for Tor -========================== +# Coding conventions for Tor tl;dr: @@ -10,8 +9,7 @@ tl;dr: - Run `make distcheck` if you have made changes to build system components - Add a file in `changes` for your branch. -Patch checklist ---------------- +## Patch checklist If possible, send your patch as one of these (in descending order of preference) @@ -34,7 +32,7 @@ Did you remember... If you are submitting a major patch or new feature, or want to in the future... - - Set up Chutney and Stem, see HACKING/WritingTests.md + - Set up Chutney and Stem, see `doc/HACKING/WritingTests.md` - Run `make test-full` to test against all unit and integration tests. If you have changed build system components: @@ -42,9 +40,7 @@ If you have changed build system components: - For example, if you have changed Makefiles, autoconf files, or anything else that affects the build system. - -License issues -============== +## License issues Tor is distributed under the license terms in the LICENSE -- in brief, the "3-clause BSD license". If you send us code to @@ -58,9 +54,7 @@ Some compatible licenses include: - 2-clause BSD - CC0 Public Domain Dedication - -How we use Git branches -======================= +## How we use Git branches Each main development series (like 0.2.1, 0.2.2, etc) has its main work applied to a single branch. At most one series can be the development series @@ -91,9 +85,7 @@ conflicts in the ChangeLog when it comes time to merge your branch into Tor. Best advice: don't try to keep an independent branch forked for more than 6 months and expect it to merge cleanly. Try to merge pieces early and often. - -How we log changes -================== +## How we log changes When you do a commit that needs a ChangeLog entry, add a new file to the `changes` toplevel subdirectory. It should have the format of a @@ -199,8 +191,7 @@ Why use changes files instead of entries in the ChangeLog? * Having every single commit touch the ChangeLog file tended to create zillions of merge conflicts. -Whitespace and C conformance ----------------------------- +## Whitespace and C conformance Invoke `make check-spaces` from time to time, so it can tell you about deviations from our C whitespace style. Generally, we use: @@ -231,8 +222,7 @@ you're using gcc, you should invoke the configure script with the option `--enable-fatal-warnings`. This will tell the compiler to make all warnings into errors. -Functions to use; functions not to use --------------------------------------- +## Functions to use; functions not to use We have some wrapper functions like `tor_malloc`, `tor_free`, `tor_strdup`, and `tor_gettimeofday;` use them instead of their generic equivalents. (They @@ -250,7 +240,6 @@ available containers in `src/lib/containers/*.h`. You should probably familiarize yourself with these modules before you write too much code, or else you'll wind up reinventing the wheel. - We don't use `strcat` or `strcpy` or `sprintf` of any of those notoriously broken old C functions. We also avoid `strncat` and `strncpy`. Use `strlcat`, `strlcpy`, or `tor_snprintf/tor_asprintf` instead. @@ -281,9 +270,7 @@ version prefixed with `tor_` instead: strtok_r, memmem, memstr, asprintf, localtime_r, gmtime_r, inet_aton, inet_ntop, inet_pton, getpass, ntohll, htonll. (This list is incomplete.) - -What code can use what other code? ----------------------------------- +## What code can use what other code? We're trying to simplify Tor's structure over time. In the long run, we want Tor to be structured as a set of modules with *no circular dependencies*. @@ -300,8 +287,7 @@ included except those specifically permitted by the `.may_include` file. When editing one of these files, please make sure that you are not introducing any cycles into Tor's dependency graph. -Floating point math is hard ---------------------------- +## Floating point math is hard Floating point arithmetic as typically implemented by computers is very counterintuitive. Failure to adequately analyze floating point @@ -360,8 +346,7 @@ For more detailed (and math-intensive) background, see [What Every Computer Scientist Should Know About Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html). -Other C conventions -------------------- +## Other C conventions The `a ? b : c` trinary operator only goes inside other expressions; don't use it as a replacement for if. (You can ignore this inside macro @@ -370,8 +355,7 @@ definitions when necessary.) Assignment operators shouldn't nest inside other expressions. (You can ignore this inside macro definitions when necessary.) -Binary data and wire formats ----------------------------- +## Binary data and wire formats Use pointer to `char` when representing NUL-terminated string. To represent arbitrary binary data, use pointer to `uint8_t`. (Many older Tor APIs ignore @@ -390,8 +374,7 @@ for more information about trunnel. For information on adding new trunnel code to Tor, see src/trunnel/README -Calling and naming conventions ------------------------------- +## Calling and naming conventions Whenever possible, functions should return -1 on error and 0 on success. @@ -414,17 +397,15 @@ probably time to create an enum. If you find that you are passing three or more flags to a function, it's probably time to create a flags argument that takes a bitfield. -What To Optimize ----------------- +## What To Optimize Don't optimize anything if it's not in the critical path. Right now, the critical path seems to be AES, logging, and the network itself. Feel free to do your own profiling to determine otherwise. -Log conventions ---------------- +## Log conventions -`https://www.torproject.org/docs/faq#LogLevel` +[FAQ - Log Levels](https://www.torproject.org/docs/faq#LogLevel) No error or warning messages should be expected during normal OR or OP operation. @@ -438,8 +419,7 @@ end-users that they aren't expected to understand the message (perhaps with a string like "internal error"). Option (A) is to be preferred to option (B). -Assertions In Tor ------------------ +## Assertions In Tor Assertions should be used for bug-detection only. Don't use assertions to detect bad user inputs, network errors, resource exhaustion, or similar @@ -458,8 +438,7 @@ macro, as in: if (BUG(ptr == NULL)) return -1; -Allocator conventions ---------------------- +## Allocator conventions By convention, any tor type with a name like `abc_t` should be allocated by a function named `abc_new()`. This function should never return @@ -500,8 +479,7 @@ to use it as a function callback), define it with a name like When deallocating, don't say e.g. `if (x) tor_free(x)`. The convention is to have deallocators do nothing when NULL pointer is passed. -Doxygen comment conventions ---------------------------- +## Doxygen comment conventions Say what functions do as a series of one or more imperative sentences, as though you were telling somebody how to be the function. In other words, DO diff --git a/doc/HACKING/CodingStandardsRust.md b/doc/HACKING/CodingStandardsRust.md index b570e10dc7..97026c9b7c 100644 --- a/doc/HACKING/CodingStandardsRust.md +++ b/doc/HACKING/CodingStandardsRust.md @@ -1,39 +1,36 @@ +# Rust Coding Standards - Rust Coding Standards -======================= - -You MUST follow the standards laid out in `.../doc/HACKING/CodingStandards.md`, +You MUST follow the standards laid out in `doc/HACKING/CodingStandards.md`, where applicable. - Module/Crate Declarations ---------------------------- +## Module/Crate Declarations Each Tor C module which is being rewritten MUST be in its own crate. -See the structure of `.../src/rust` for examples. +See the structure of `src/rust` for examples. In your crate, you MUST use `lib.rs` ONLY for pulling in external crates (e.g. `extern crate libc;`) and exporting public objects from other Rust modules (e.g. `pub use mymodule::foo;`). For example, if -you create a crate in `.../src/rust/yourcrate`, your Rust code should -live in `.../src/rust/yourcrate/yourcode.rs` and the public interface -to it should be exported in `.../src/rust/yourcrate/lib.rs`. +you create a crate in `src/rust/yourcrate`, your Rust code should +live in `src/rust/yourcrate/yourcode.rs` and the public interface +to it should be exported in `src/rust/yourcrate/lib.rs`. If your code is to be called from Tor C code, you MUST define a safe `ffi.rs`. See the "Safety" section further down for more details. For example, in a hypothetical `tor_addition` Rust module: -In `.../src/rust/tor_addition/addition.rs`: +In `src/rust/tor_addition/addition.rs`: pub fn get_sum(a: i32, b: i32) -> i32 { a + b } -In `.../src/rust/tor_addition/lib.rs`: +In `src/rust/tor_addition/lib.rs`: pub use addition::*; -In `.../src/rust/tor_addition/ffi.rs`: +In `src/rust/tor_addition/ffi.rs`: #[no_mangle] pub extern "C" fn tor_get_sum(a: c_int, b: c_int) -> c_int { @@ -42,7 +39,7 @@ In `.../src/rust/tor_addition/ffi.rs`: If your Rust code must call out to parts of Tor's C code, you must declare the functions you are calling in the `external` crate, located -at `.../src/rust/external`. +at `src/rust/external`. @@ -54,8 +51,7 @@ If you have any external modules as dependencies (e.g. `extern crate libc;`), you MUST declare them in your crate's `lib.rs` and NOT in any other module. - Dependencies and versions ---------------------------- +## Dependencies and versions In general, we use modules from only the Rust standard library whenever possible. We will review including external crates on a @@ -81,8 +77,7 @@ Currently, Tor requires that you use the latest stable Rust version. At some point in the future, we will freeze on a given stable Rust version, to ensure backward compatibility with stable distributions that ship it. - Updating/Adding Dependencies ------------------------------- +## Updating/Adding Dependencies To add/remove/update dependencies, first add your dependencies, exactly specifying their versions, into the appropriate *crate-level* @@ -101,14 +96,13 @@ Next, run `/scripts/maint/updateRustDependencies.sh`. Then, go into `src/ext/rust` and commit the changes to the `tor-rust-dependencies` repo. - Documentation ---------------- +## Documentation You MUST include `#![deny(missing_docs)]` in your crate. For function/method comments, you SHOULD include a one-sentence, "first person" description of function behaviour (see requirements for documentation as -described in `.../src/HACKING/CodingStandards.md`), then an `# Inputs` section +described in `src/HACKING/CodingStandards.md`), then an `# Inputs` section for inputs or initialisation values, a `# Returns` section for return values/types, a `# Warning` section containing warnings for unsafe behaviours or panics that could happen. For publicly accessible @@ -118,14 +112,12 @@ types/constants/objects/functions/methods, you SHOULD also include an You MUST document your module with _module docstring_ comments, i.e. `//!` at the beginning of each line. - Style -------- +## Style You SHOULD consider breaking up large literal numbers with `_` when it makes it more human readable to do so, e.g. `let x: u64 = 100_000_000_000`. - Testing ---------- +## Testing All code MUST be unittested and integration tested. @@ -134,7 +126,7 @@ describing how the function/object is expected to be used. Integration tests SHOULD go into a `tests/` directory inside your crate. Unittests SHOULD go into their own module inside the module -they are testing, e.g. in `.../src/rust/tor_addition/addition.rs` you +they are testing, e.g. in `src/rust/tor_addition/addition.rs` you should put: #[cfg(test)] @@ -148,8 +140,7 @@ should put: } } - Benchmarking --------------- +## Benchmarking The external `test` crate can be used for most benchmarking. However, using this crate requires nightly Rust. Since we may want to switch to a more @@ -173,7 +164,7 @@ for basic benchmarks, is only used when running benchmarks via `cargo bench --features bench`. Finally, to write your benchmark code, in -`.../src/rust/tor_addition/addition.rs` you SHOULD put: +`src/rust/tor_addition/addition.rs` you SHOULD put: #[cfg(all(test, features = "bench"))] mod bench { @@ -186,23 +177,20 @@ Finally, to write your benchmark code, in } } - Fuzzing ---------- +## Fuzzing If you wish to fuzz parts of your code, please see the -[`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz) crate, which uses +[cargo fuzz](https://github.com/rust-fuzz/cargo-fuzz) crate, which uses [libfuzzer-sys](https://github.com/rust-fuzz/libfuzzer-sys). - Whitespace & Formatting -------------------------- +## Whitespace & Formatting You MUST run `rustfmt` (https://github.com/rust-lang-nursery/rustfmt) on your code before your code will be merged. You can install rustfmt by doing `cargo install rustfmt-nightly` and then run it with `cargo fmt`. - Safety --------- +## Safety You SHOULD read [the nomicon](https://doc.rust-lang.org/nomicon/) before writing Rust FFI code. It is *highly advised* that you read and write normal Rust code diff --git a/doc/HACKING/EndOfLifeTor.md b/doc/HACKING/EndOfLifeTor.md index 2fece2ca9d..0735dc311a 100644 --- a/doc/HACKING/EndOfLifeTor.md +++ b/doc/HACKING/EndOfLifeTor.md @@ -1,13 +1,11 @@ - -End of Life on an old release series ------------------------------------- +# End of Life on an old release series Here are the steps that the maintainer should take when an old Tor release series reaches End of Life. Note that they are _only_ for entire series that have reached their planned EOL: they do not apply to security-related deprecations of individual versions. -### 0. Preliminaries +## 0. Preliminaries 0. A few months before End of Life: Write a deprecation announcement. @@ -17,7 +15,7 @@ deprecations of individual versions. Send the announcement to tor-announce, tor-talk, tor-relays, and the packagers. -### 1. On the day +## 1. On the day 1. Open tickets to remove the release from: - the jenkins builds diff --git a/doc/HACKING/Fuzzing.md b/doc/HACKING/Fuzzing.md index c2db7e9853..41853a8a23 100644 --- a/doc/HACKING/Fuzzing.md +++ b/doc/HACKING/Fuzzing.md @@ -11,7 +11,6 @@ To run the fuzzing test cases in a deterministic fashion, use: This won't actually fuzz Tor! It will just run all the fuzz binaries on our existing set of testcases for the fuzzer. - ## Different kinds of fuzzing Right now we support three different kinds of fuzzer. @@ -51,7 +50,6 @@ But the fuzzing harness should crash if tor fails an assertion, triggers a bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect "interesting" cases. - ## Guided Fuzzing with AFL There is no HTTPS, hash, or signature for American Fuzzy Lop's source code, so diff --git a/doc/HACKING/GettingStarted.md b/doc/HACKING/GettingStarted.md index 7d5397d70c..c2ca74d960 100644 --- a/doc/HACKING/GettingStarted.md +++ b/doc/HACKING/GettingStarted.md @@ -1,24 +1,19 @@ - -Getting started in Tor development -================================== +# Getting started in Tor development Congratulations! You've found this file, and you're reading it! This means that you might be interested in getting started in developing Tor. -(This guide is just about Tor itself--the small network program at the +(_This guide is just about Tor itself--the small network program at the heart of the Tor network--and not about all the other programs in the -whole Tor ecosystem.) - +whole Tor ecosystem._) If you are looking for a more bare-bones, less user-friendly information dump of important information, you might like reading the -doxygen output at https://src-ref.docs.torproject.org/tor/index.html . -You should probably skim some of the topic headings there before you write +[doxygen output](https://src-ref.docs.torproject.org/tor/index.html). +You probably should skim some of the topic headings there before you write your first patch. - -Required background -------------------- +## Required background First, I'm going to assume that you can build Tor from source, and that you know enough of the C language to read and write it. (See the README @@ -27,22 +22,20 @@ and any high-quality guide to C for information on programming.) I'm also going to assume that you know a little bit about how to use Git, or that you're able to follow one of the several excellent guides -at http://git-scm.org to learn. +at [git-scm](http://git-scm.org) to learn. -Most Tor developers develop using some Unix-based system, such as Linux, -BSD, or OSX. It's okay to develop on Windows if you want, but you're +Most Tor developers develop using some Unix-based system, such as GNU/Linux, +BSD, or macOS. It's okay to develop on Windows if you want, but you're going to have a more difficult time. - -Getting your first patch into Tor ---------------------------------- +## Getting your first patch into Tor Once you've reached this point, here's what you need to know. 1. Get the source. We keep our source under version control in Git. To get the latest - version, run + version, run: git clone https://git.torproject.org/git/tor @@ -52,18 +45,16 @@ Once you've reached this point, here's what you need to know. git checkout maint-0.4.3 - 2. Find your way around the source + 2. Find your way around the source. - Our overall code structure is explained in our source documentation, - currently at - - https://src-ref.docs.torproject.org/tor/index.html + Our overall code structure is explained in our + [source documentation](https://src-ref.docs.torproject.org/tor/index.html). Find a part of the code that looks interesting to you, and start looking around it to see how it fits together! We do some unusual things in our codebase. Our testing-related - practices and kludges are explained in doc/WritingTests.txt. + practices and kludges are explained in `doc/HACKING/WritingTests.md`. If you see something that doesn't make sense, we love to get questions! @@ -75,10 +66,10 @@ Once you've reached this point, here's what you need to know. Many people have gotten started by looking for an area where they personally felt Tor was underperforming, and investigating ways to - fix it. If you're looking for ideas, you can head to our bug - tracker at trac.torproject.org and look for tickets that have - received the "easy" tag: these are ones that developers think would - be pretty simple for a new person to work on. For a bigger + fix it. If you're looking for ideas, you can head to + [trac](https://trac.torproject.org) our bug tracking tool and look for + tickets that have received the "easy" tag: these are ones that developers + think would be pretty simple for a new person to work on. For a bigger challenge, you might want to look for tickets with the "lorax" keyword: these are tickets that the developers think might be a good idea to build, but which we have no time to work on any time @@ -97,7 +88,7 @@ Once you've reached this point, here's what you need to know. 4. Meet the developers! - We discuss stuff on the tor-dev mailing list and on the #tor-dev + We discuss stuff on the tor-dev mailing list and on the `#tor-dev` IRC channel on OFTC. We're generally friendly and approachable, and we like to talk about how Tor fits together. If we have ideas about how something should be implemented, we'll be happy to share @@ -114,8 +105,8 @@ Once you've reached this point, here's what you need to know. protocols, there needs to be a written design proposal before it can be merged. (We use this process to manage changes in the protocols.) To write one, see the instructions at - https://gitweb.torproject.org/torspec.git/tree/proposals/001-process.txt - . If you'd like help writing a proposal, just ask! We're happy to + [the Tor proposal process](https://gitweb.torproject.org/torspec.git/plain/proposals/001-process.txt). + If you'd like help writing a proposal, just ask! We're happy to help out with good ideas. You might also like to look around the rest of that directory, to @@ -126,7 +117,7 @@ Once you've reached this point, here's what you need to know. As you write your code, you'll probably want it to fit in with the standards of the rest of the Tor codebase so it will be easy for us to review and merge. You can learn our coding standards in - doc/HACKING. + `doc/HACKING` directory. If your patch is large and/or is divided into multiple logical components, remember to divide it into a series of Git commits. A @@ -138,16 +129,16 @@ Once you've reached this point, here's what you need to know. ensure that it runs correctly. Also, all code should actually be _run_ by somebody, to make sure it works. - See doc/WritingTests.txt for more information on how we test things + See `doc/HACKING/WritingTests.md` for more information on how we test things in Tor. If you'd like any help writing tests, just ask! We're glad to help out. 8. Submitting your patch We review patches through tickets on our bugtracker at - trac.torproject.org. You can either upload your patches there, or + [trac](https://trac.torproject.org). You can either upload your patches there, or put them at a public git repository somewhere we can fetch them - (like github or bitbucket) and then paste a link on the appropriate + (like gitlab, github or bitbucket) and then paste a link on the appropriate trac ticket. Once your patches are available, write a short explanation of what @@ -164,17 +155,17 @@ Once you've reached this point, here's what you need to know. When your patch is reviewed, one of these things will happen: - * The reviewer will say "looks good to me" and your + * The reviewer will say "_looks good to me_" and your patch will get merged right into Tor. [Assuming we're not in the middle of a code-freeze window. If the codebase is frozen, your patch will go into the next release series.] - * OR the reviewer will say "looks good, just needs some small - changes!" And then the reviewer will make those changes, + * OR the reviewer will say "_looks good, just needs some small + changes!_" And then the reviewer will make those changes, and merge the modified patch into Tor. - * OR the reviewer will say "Here are some questions and - comments," followed by a bunch of stuff that the reviewer + * OR the reviewer will say "_Here are some questions and + comments,_" followed by a bunch of stuff that the reviewer thinks should change in your code, or questions that the reviewer has. diff --git a/doc/HACKING/GettingStartedRust.md b/doc/HACKING/GettingStartedRust.md index aa29c097da..247ea5c695 100644 --- a/doc/HACKING/GettingStartedRust.md +++ b/doc/HACKING/GettingStartedRust.md @@ -1,12 +1,9 @@ +# Hacking on Rust in Tor - Hacking on Rust in Tor -======================== - - Getting Started ------------------ +## Getting Started Please read or review our documentation on Rust coding standards -(`.../doc/HACKING/CodingStandardsRust.md`) before doing anything. +(`doc/HACKING/CodingStandardsRust.md`) before doing anything. Please also read [the Rust Code of Conduct](https://www.rust-lang.org/en-US/conduct.html). We @@ -23,8 +20,7 @@ Please be patient with the other people who are working on getting more Rust code into Tor, because they are graciously donating their free time to contribute to this effort. - Resources for learning Rust ------------------------------ +## Resources for learning Rust **Beginning resources** @@ -49,8 +45,7 @@ is For learning more about FFI and Rust, see Jake Goulding's [Rust FFI Omnibus](http://jakegoulding.com/rust-ffi-omnibus/). - Compiling Tor with Rust enabled ---------------------------------- +## Compiling Tor with Rust enabled You will need to run the `configure` script with the `--enable-rust` flag to explicitly build with Rust. Additionally, you will need to @@ -79,7 +74,7 @@ you are in the top level of the repository) configure tor with: TOR_RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust -(Note that TOR_RUST_DEPENDENCIES must be the full path to the directory; it +(Note that `TOR_RUST_DEPENDENCIES` must be the full path to the directory; it cannot be relative.) Assuming you used the above `git submodule` commands and you're in the @@ -87,9 +82,7 @@ topmost directory of the repository, this would be: TOR_RUST_DEPENDENCIES=`pwd`/src/ext/rust/crates ./configure --enable-rust - - Identifying which modules to rewrite -====================================== +## Identifying which modules to rewrite The places in the Tor codebase that are good candidates for porting to Rust are: @@ -117,12 +110,11 @@ interconnected your target module is. The output will tell you each module name, along with a set of every module that the module calls. Modules which call fewer other modules are better targets. - Writing your Rust module -========================== +## Writing your Rust module Strive to change the C API as little as possible. -We are currently targetting Rust stable. (See CodingStandardsRust.md for more +We are currently targetting Rust stable. (See `CodingStandardsRust.md` for more details.) It is on our TODO list to try to cultivate good @@ -134,19 +126,17 @@ If parts of your Rust code needs to stay in sync with C code (such as handling enums across the FFI boundary), annonotate these places in a comment structured as follows: - /// C_RUST_COUPLED: `` + `/// C_RUST_COUPLED: ` -Where can be an enum, struct, constant, etc. Then, +Where `` can be an enum, struct, constant, etc. Then, do the same in the C code, to note that rust will need to be changed when the C does. - - Adding your Rust module to Tor's build system ------------------------------------------------ +## Adding your Rust module to Tor's build system 0. Your translation of the C module should live in its own crate(s) - in the `.../tor/src/rust/` directory. -1. Add your crate to `.../tor/src/rust/Cargo.toml`, in the + in the `src/rust/` directory. +1. Add your crate to `src/rust/Cargo.toml`, in the `[workspace.members]` section. 2. Add your crate's files to src/rust/include.am @@ -156,12 +146,11 @@ dependency of other Rust modules): `src/rust/tor_util/Cargo.toml` and include it in `src/rust/tor_rust/lib.rs` - How to test your Rust code ----------------------------- +## How to test your Rust code Everything should be tested full stop. Even non-public functionality. -Be sure to edit `.../tor/src/test/test_rust.sh` to add the name of your +Be sure to edit `src/test/test_rust.sh` to add the name of your crate to the `crates` variable! This will ensure that `cargo test` is run on your crate. @@ -177,7 +166,6 @@ Tor's integration tests should also pass: make test-stem - Submitting a patch -===================== +## Submitting a patch -Please follow the instructions in `.../doc/HACKING/GettingStarted.md`. +Please follow the instructions in `doc/HACKING/GettingStarted.md`. diff --git a/doc/HACKING/HelpfulTools.md b/doc/HACKING/HelpfulTools.md index ae892c34a2..15bd153318 100644 --- a/doc/HACKING/HelpfulTools.md +++ b/doc/HACKING/HelpfulTools.md @@ -1,11 +1,10 @@ -Useful tools -============ +# Useful tools These aren't strictly necessary for hacking on Tor, but they can help track down bugs. -Travis/Appveyor CI ------------------- +## Travis/Appveyor CI + It's CI. Looks like this: @@ -29,8 +28,7 @@ for your fork to build commits outside of PRs too: Builds should show up on the web at travis-ci.com and on IRC at #tor-ci on OFTC. If they don't, ask #tor-dev (also on OFTC). -Jenkins -------- +## Jenkins It's CI/builders. Looks like this: https://jenkins.torproject.org @@ -43,8 +41,7 @@ Builds Linux and Windows cross-compilation. Runs Linux tests. Builds should show up on the web at jenkins.torproject.org and on IRC at #tor-bots on OFTC. If they don't, ask #tor-dev (also on OFTC). -Valgrind --------- +## Valgrind valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/app/tor @@ -52,16 +49,14 @@ Valgrind pass `--undef-value-errors=no` to valgrind, or rebuild your openssl with `-DPURIFY`.) -Coverity --------- +## Coverity Nick regularly runs the coverity static analyzer on the Tor codebase. The preprocessor define `__COVERITY__` is used to work around instances where coverity picks up behavior that we wish to permit. -clang Static Analyzer ---------------------- +## clang Static Analyzer The clang static analyzer can be run on the Tor codebase using Xcode (WIP) or a command-line build. @@ -69,8 +64,7 @@ or a command-line build. The preprocessor define `__clang_analyzer__` is used to work around instances where clang picks up behavior that we wish to permit. -clang Runtime Sanitizers ------------------------- +## clang Runtime Sanitizers To build the Tor codebase with the clang Address and Undefined Behavior sanitizers, see the file `contrib/clang/sanitize_blacklist.txt`. @@ -78,8 +72,7 @@ sanitizers, see the file `contrib/clang/sanitize_blacklist.txt`. Preprocessor workarounds for instances where clang picks up behavior that we wish to permit are also documented in the blacklist file. -Running lcov for unit test coverage ------------------------------------ +## Running lcov for unit test coverage Lcov is a utility that generates pretty HTML reports of test code coverage. To generate such a report: @@ -96,8 +89,7 @@ output directory, use `make coverage-html HTML_COVER_DIR=./funky_new_cov_dir`. Coverage diffs using lcov are not currently implemented, but are being investigated (as of July 2014). -Running the unit tests ----------------------- +## Running the unit tests To quickly run all the tests distributed with Tor: @@ -123,8 +115,7 @@ working connection to the internet: make test-full-online -Running gcov for unit test coverage ------------------------------------ +## Running gcov for unit test coverage ./configure --enable-coverage make @@ -164,8 +155,7 @@ lines? If you run ./scripts/test/cov-exclude, it marks excluded unreached lines with 'x', and excluded reached lines with '!!!'. -Running integration tests -------------------------- +## Running integration tests We have the beginnings of a set of scripts to run integration tests using Chutney. To try them, set CHUTNEY_PATH to your chutney source directory, and @@ -174,14 +164,12 @@ run `make test-network`. We also have scripts to run integration tests using Stem. To try them, set `STEM_SOURCE_DIR` to your Stem source directory, and run `test-stem`. -Profiling Tor -------------- +## Profiling Tor Ongoing notes about Tor profiling can be found at https://pad.riseup.net/p/profiling-tor -Profiling Tor with oprofile ---------------------------- +## Profiling Tor with oprofile The oprofile tool runs (on Linux only!) to tell you what functions Tor is spending its CPU time in, so we can identify performance bottlenecks. @@ -206,8 +194,7 @@ Here are some basic instructions * `opreport -l that_dir/*` - Profit -Profiling Tor with perf ------------------------ +## Profiling Tor with perf This works with a running Tor, and requires root. @@ -236,8 +223,7 @@ This works with a running Tor, and requires root. report -g > .out`. Then you can compress that and put it somewhere public. -Profiling Tor with gperftools aka Google-performance-tools ----------------------------------------------------------- +## Profiling Tor with gperftools aka Google-performance-tools This should work on nearly any unixy system. It doesn't seem to be compatible with RunAsDaemon though. @@ -255,8 +241,7 @@ performance! See the gperftools manual for more info, but basically: 3. Run `pprof src/app/tor /tmp/profile` to start the REPL. -Generating and analyzing a callgraph ------------------------------------- +## Generating and analyzing a callgraph 0. Build Tor on linux or mac, ideally with -O0 or -fno-inline. @@ -266,8 +251,7 @@ Generating and analyzing a callgraph Note that currently the callgraph generator can't detect calls that pass through function pointers. -Getting emacs to edit Tor source properly ------------------------------------------ +## Getting emacs to edit Tor source properly Nick likes to put the following snippet in his .emacs file: @@ -315,8 +299,7 @@ If you use emacs for editing Tor and nothing else, you could always just say: There is probably a better way to do this. No, we are probably not going to clutter the files with emacs stuff. -Building a tag file (code index) --------------------------------- +## Building a tag file (code index) Many functions in tor use `MOCK_IMPL` wrappers for unit tests. Your tag-building program must be told how to handle this syntax. @@ -340,8 +323,7 @@ instead: A vim-compatible tag file will be generated by default. If you use emacs, add the `-e` flag to generate an emacs-compatible tag file. -Doxygen -------- +## Doxygen We use the 'doxygen' utility to generate documentation from our source code. Here's how to use it: @@ -396,8 +378,7 @@ source code. Here's how to use it: 6. See the Doxygen manual for more information; this summary just scratches the surface. -Style and best-practices checking --------------------------------- +## Style and best-practices checking We use scripts to check for various problems in the formatting and style of our source code. The "check-spaces" test detects a bunch of violations diff --git a/doc/HACKING/HowToReview.md b/doc/HACKING/HowToReview.md index 2325e70175..7815e76632 100644 --- a/doc/HACKING/HowToReview.md +++ b/doc/HACKING/HowToReview.md @@ -1,5 +1,4 @@ -How to review a patch -===================== +# How to review a patch Some folks have said that they'd like to review patches more often, but they don't know how. @@ -9,9 +8,7 @@ So, here are a bunch of things to check for when reviewing a patch! Note that if you can't do every one of these, that doesn't mean you can't do a good review! Just make it clear what you checked for and what you didn't. - -Top-level smell-checks ----------------------- +## Top-level smell-checks (Difficulty: easy) @@ -37,10 +34,9 @@ Top-level smell-checks - If this changes anything in the code, is there a "changes" file? -Let's look at the code! ------------------------ +## Let's look at the code! -- Does the code conform to CodingStandards.txt? +- Does the code conform to `CodingStandards.md`? - Does the code leak memory? @@ -60,8 +56,7 @@ Let's look at the code! - Is there duplicated code that could be turned into a function? -Let's look at the documentation! --------------------------------- +## Let's look at the documentation! - Does the documentation conform to CodingStandards.txt? @@ -70,8 +65,7 @@ Let's look at the documentation! - Can you predict what the function will do from its documentation? -Let's think about security! ---------------------------- +## Let's think about security! - If there are any arrays, buffers, are you 100% sure that they cannot overflow? diff --git a/doc/HACKING/Module.md b/doc/HACKING/Module.md index 781bb978f2..f8a9773d47 100644 --- a/doc/HACKING/Module.md +++ b/doc/HACKING/Module.md @@ -1,9 +1,9 @@ -# Modules in Tor # +# Modules in Tor This document describes the build system and coding standards when writing a module in Tor. -## What is a module? ## +## What is a module? In the context of the tor code base, a module is a subsystem that we can selectively enable or disable, at `configure` time. @@ -27,7 +27,7 @@ because you would actually want to run one without the other.) To disable a module, pass `--disable-module-{dirauth,relay}` at configure time. All modules are currently enabled by default. -## Build System ## +## Build System The changes to the build system are pretty straightforward. @@ -55,7 +55,7 @@ Finally, your module will automatically be included in the `TOR_MODULES_ALL_ENABLED` variable which is used to build the unit tests. They always build everything in order to test everything. -## Coding ## +## Coding As mentioned above, a module should be isolated in its own directories, suffixed with the name of the module, in `src/*/`. diff --git a/doc/HACKING/README.1st.md b/doc/HACKING/README.1st.md index 8299fe634a..2278a61d6c 100644 --- a/doc/HACKING/README.1st.md +++ b/doc/HACKING/README.1st.md @@ -1,17 +1,18 @@ +# README.1st -In this directory ------------------ +## In this directory This directory has helpful information about what you need to know to hack on Tor! -First, read `GettingStarted.md` to learn how to get a start in Tor -development. +First, read `GettingStarted.md` and `GettingStartedRust.md` +to learn how to get a start in Tor development. -If you've decided to write a patch, `CodingStandards.txt` will give -you a bunch of information about how we structure our code. +If you've decided to write a patch, `CodingStandards.md` and +`CodingStandardsRust.md` will give you a bunch of information +about how we structure our code. -It's important to get code right! Reading `WritingTests.md` will +It's important to get the code right! Reading `WritingTests.md` will tell you how to write and run tests in the Tor codebase. There are a bunch of other programs we use to help maintain and @@ -21,42 +22,26 @@ with Tor. If it's your job to put out Tor releases, see `ReleasingTor.md` so that you don't miss any steps! - ------------------------ +## Additional Information For full information on how Tor is supposed to work, look at the files in -`https://gitweb.torproject.org/torspec.git/tree`. +[Tor specification](https://gitweb.torproject.org/torspec.git/tree). For an explanation of how to change Tor's design to work differently, look at -`https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/001-process.txt`. +[the Tor proposal process](https://gitweb.torproject.org/torspec.git/plain/proposals/001-process.txt). For the latest version of the code, get a copy of git, and git clone https://git.torproject.org/git/tor +## Stay in touch + We talk about Tor on the `tor-talk` mailing list. Design proposals and discussion belong on the `tor-dev` mailing list. We hang around on -irc.oftc.net, with general discussion happening on #tor and development +irc.oftc.net, with general discussion happening on `#tor` and development happening on `#tor-dev`. The other files in this `HACKING` directory may also be useful as you get started working with Tor. Happy hacking! - - ------------------------ - -XXXXX also describe - -doc/HACKING/WritingTests.md - -torguts.git - -torspec.git - -The design paper - -freehaven.net/anonbib - -XXXX describe these and add links. diff --git a/doc/HACKING/ReleasingTor.md b/doc/HACKING/ReleasingTor.md index 0f453ca2aa..2464d8afb4 100644 --- a/doc/HACKING/ReleasingTor.md +++ b/doc/HACKING/ReleasingTor.md @@ -1,11 +1,9 @@ - -Putting out a new release -------------------------- +# Putting out a new release Here are the steps that the maintainer should take when putting out a new Tor release: -### 0. Preliminaries +## 0. Preliminaries 1. Get at least two of weasel/arma/Sebastian to put the new version number in their approved versions list. Give them a few @@ -18,7 +16,7 @@ new Tor release: date of a TB that contains it. See note below in "commit, upload, announce". -### I. Make sure it works +## I. Make sure it works 1. Make sure that CI passes: have a look at Travis (https://travis-ci.org/torproject/tor/branches), Appveyor @@ -51,9 +49,7 @@ new Tor release: libevent and openssl, so using valgrind will sometimes find extra memory leaks.) - -### II. Write a changelog - +## II. Write a changelog 1a. (Alpha release variant) @@ -138,8 +134,7 @@ new Tor release: to start sorting and condensing entries. (Generally, we don't edit the text of existing entries, though.) - -### III. Making the source release. +## III. Making the source release. 1. In `maint-0.?.x`, bump the version number in `configure.ac` and run `make update-versions` to update version numbers in other @@ -165,7 +160,7 @@ new Tor release: If it is not, you'll need to poke Roger, Weasel, and Sebastian again: see item 0.1 at the start of this document. -### IV. Commit, upload, announce +## IV. Commit, upload, announce 1. Sign the tarball, then sign and push the git tag: @@ -215,7 +210,6 @@ new Tor release: Include a link to the changelog. - 4. Add the version number to Trac. To do this, go to Trac, log in, select "Admin" near the top of the screen, then select "Versions" from the menu on the left. At the right, there will be an "Add version" @@ -241,7 +235,7 @@ new Tor release: For templates to use when announcing, see: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/AnnouncementTemplates -### V. Aftermath and cleanup +## V. Aftermath and cleanup 1. If it's a stable release, bump the version number in the `maint-x.y.z` branch to "newversion-dev", and do a `merge -s ours` diff --git a/doc/HACKING/Tracing.md b/doc/HACKING/Tracing.md index 24fa761310..e1e97abe6d 100644 --- a/doc/HACKING/Tracing.md +++ b/doc/HACKING/Tracing.md @@ -1,16 +1,16 @@ -# Tracing # +# Tracing This document describes how the event tracing subsystem works in tor so developers can add events to the code base but also hook them to an event tracing framework. -## Basics ### +## Basics Event tracing is separated in two concepts, trace events and a tracer. The tracing subsystem can be found in `src/trace`. The `events.h` header file is the main file that maps the different tracers to trace events. -### Events ### +### Events A trace event is basically a function from which we can pass any data that we want to collect. In addition, we specify a context for the event such as @@ -39,7 +39,7 @@ How `argc` is collected or used has nothing to do with the instrumentation the trace events and collection framework (tracer) are decoupled. You _can_ have trace events without a tracer. -### Tracer ### +### Tracer In `src/trace/events.h`, we map the `tor_trace()` function to the right tracer. A tracer support is only enabled at compile time. For instance, the @@ -47,7 +47,7 @@ file `src/trace/debug.h` contains the mapping of the generic tracing function `tor_trace()` to the `log_debug()` function. More specialized function can be mapped depending on the tracepoint. -## Build System ## +## Build System This section describes how it is integrated into the build system of tor. @@ -66,7 +66,7 @@ configure option: --enable-tracing-debug -## Instrument Tor ## +## Instrument Tor This is pretty easy. Let's say you want to add a trace event in `src/feature/rend/rendcache.c`, you only have to add this include statement: diff --git a/doc/HACKING/WritingTests.md b/doc/HACKING/WritingTests.md index 05de8e0be8..d212020525 100644 --- a/doc/HACKING/WritingTests.md +++ b/doc/HACKING/WritingTests.md @@ -1,6 +1,4 @@ - -Writing tests for Tor: an incomplete guide -========================================== +# Writing tests for Tor: an incomplete guide Tor uses a variety of testing frameworks and methodologies to try to keep from introducing bugs. The major ones are: @@ -19,8 +17,7 @@ keep from introducing bugs. The major ones are: 5. The Shadow network simulator. -How to run these tests ----------------------- +## How to run these tests ### The easy version @@ -64,7 +61,7 @@ The former are those that should finish in a few seconds; the latter tend to take more time, and may include CPU-intensive operations, deliberate delays, and stuff like that. -### Finding test coverage +## Finding test coverage Test coverage is a measurement of which lines your tests actually visit. @@ -112,7 +109,7 @@ To count new or modified uncovered lines in D2, you can run: ./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' | wc -l -### Marking lines as unreachable by tests +## Marking lines as unreachable by tests You can mark a specific line as unreachable by using the special string LCOV_EXCL_LINE. You can mark a range of lines as unreachable @@ -126,9 +123,7 @@ unreached lines with 'x', and excluded reached lines with '!!!'. Note: you should never do this unless the line is meant to 100% unreachable by actual code. - -What kinds of test should I write? ----------------------------------- +## What kinds of test should I write? Integration testing and unit testing are complementary: it's probably a good idea to make sure that your code is hit by both if you can. @@ -143,8 +138,7 @@ If your code adds new externally visible functionality to Tor, it would be great to have a test for that functionality. That's where integration tests more usually come in. -Unit and regression tests: Does this function do what it's supposed to? ------------------------------------------------------------------------ +## Unit and regression tests: Does this function do what it's supposed to? Most of Tor's unit tests are made using the "tinytest" testing framework. You can see a guide to using it in the tinytest manual at @@ -165,7 +159,7 @@ If you have created a new test file, you will need to: I use the term "unit test" and "regression tests" very sloppily here. -### A simple example +## A simple example Here's an example of a test function for a simple function in util.c: @@ -207,7 +201,7 @@ Finally, remember that by convention, all `*_free()` functions that Tor defines are defined to accept NULL harmlessly. Thus, you don't need to say `if (contents)` in the cleanup block. -### Exposing static functions for testing +## Exposing static functions for testing Sometimes you need to test a function, but you don't want to expose it outside its usual module. @@ -228,7 +222,7 @@ For example, `crypto_curve25519.h` contains: The `crypto_curve25519.c` file and the `test_crypto.c` file both define `CRYPTO_CURVE25519_PRIVATE`, so they can see this declaration. -### STOP! Does this test really test? +## STOP! Does this test really test? When writing tests, it's not enough to just generate coverage on all the lines of the code that you're testing: It's important to make sure that @@ -269,8 +263,7 @@ it's supposed to do, and fail otherwise. Try to design your tests so that they check for the code's intended and documented functionality as much as possible. - -### Mock functions for testing in isolation +## Mock functions for testing in isolation Often we want to test that a function works right, but the function to be tested depends on other functions whose behavior is hard to observe, @@ -311,7 +304,7 @@ And later, you can restore the original function with: For more information, see the definitions of this mocking logic in `testsupport.h`. -### Okay but what should my tests actually do? +## Okay but what should my tests actually do? We talk above about "test coverage" -- making sure that your tests visit every line of code, or every branch of code. But visiting the code isn't @@ -382,8 +375,7 @@ Based on the implementation, we now see three more edge cases to test: * Removing an element from the end of the list * Removing an element from a position other than the end of the list. - -### What should my tests NOT do? +## What should my tests NOT do? Tests shouldn't require a network connection. @@ -401,8 +393,7 @@ When possible, tests should not be over-fit to the implementation. That is, the test should verify that the documented behavior is implemented, but should not break if other permissible behavior is later implemented. - -### Advanced techniques: Namespaces +## Advanced techniques: Namespaces Sometimes, when you're doing a lot of mocking at once, it's convenient to isolate your identifiers within a single namespace. If this were C++, we'd @@ -414,9 +405,7 @@ them, you define `NS_MODULE` to a prefix to be used for your identifiers, and then use other macros in place of identifier names. See `src/test/test.h` for more documentation. - -Integration tests: Calling Tor from the outside ------------------------------------------------ +## Integration tests: Calling Tor from the outside Some tests need to invoke Tor from the outside, and shouldn't run from the same process as the Tor test program. Reasons for doing this might include: @@ -436,8 +425,7 @@ wrapped, add a new shell script to `TESTS`, and the new program to makefile (eg `${PYTHON}` for a python interpreter), then make sure that the makefile exports them. -Writing integration tests with Stem ------------------------------------ +## Writing integration tests with Stem The 'stem' library includes extensive tests for the Tor controller protocol. You can run stem tests from tor with `make test-stem`, or see @@ -483,8 +471,7 @@ you notice any strange behaviour that seems totally unreasonable. Check out the `test_exit_policy()` function in abovementioned file to see the final implementation for this test. -System testing with Chutney ---------------------------- +## System testing with Chutney The 'chutney' program configures and launches a set of Tor relays, authorities, and clients on your local host. It has a `test network` diff --git a/doc/HACKING/android/Simpleperf.md b/doc/HACKING/android/Simpleperf.md index 25f39a3d23..c7e63a7c86 100644 --- a/doc/HACKING/android/Simpleperf.md +++ b/doc/HACKING/android/Simpleperf.md @@ -95,4 +95,3 @@ was spend on the call. Start Tor the normal way via Orbot and collect the logs from your computer using $ adb logcat -