finish up doc/WritingTests.txt

This commit is contained in:
Nick Mathewson 2015-10-08 10:55:31 -04:00
parent 3780a6b439
commit f3804a4d6f

View File

@ -66,6 +66,8 @@ and stuff like that.
=== Finding test coverage === Finding test coverage
Test coverage is a measurement of which lines your tests actually visit.
When you configure Tor with the --enable-coverage option, it should When you configure Tor with the --enable-coverage option, it should
build with support for coverage in the unit tests, and in a special build with support for coverage in the unit tests, and in a special
"tor-cov" binary. "tor-cov" binary.
@ -245,29 +247,155 @@ And later, you can restore the original function with:
For more information, see the definitions of this mocking logic in For more information, see the definitions of this mocking logic in
testsupport.h. testsupport.h.
=== Okay but what should my tests actually do?
We talk above about "test coverage" -- making sure that your tests visit
every line of code, or every branch of code. But visiting the code isn't
enough: we want to verify that it's correct.
So when writing tests, try to make tests that should pass with any correct
implementation of the code, and that should fail if the code doesn't do what
it's supposed to do.
You can write "black-box" tests or "glass-box" tests. A black-box test is
one that you write without looking at the structure of the function. A
glass-box one is one you implement while looking at how the function is
implemented.
In either case, make sure to consider common cases *and* edge cases; success
cases and failure csaes.
For example, consider testing this function:
/** Remove all elements E from sl such that E==element. Preserve
* the order of any elements before E, but elements after E can be
* rearranged.
*/
void smartlist_remove(smartlist_t *sl, const void *element);
In order to test it well, you should write tests for at least all of the
following cases. (These would be black-box tests, since we're only looking
at the declared behavior for the function:
* Remove an element that is in the smartlist.
* Remove an element that is not in the smartlist.
* Remove an element that appears in the smartlist more than once.
And your tests should verify that it behaves correct. At minimum, you should
test:
* That other elements before E are in the same order after you call the
functions.
* That the target element is really removed.
* That _only_ the target element is removed.
When you consider edge cases, you might try:
* Remove an element from an empty list.
* Remove an element from a singleton list containing that element.
* Remove an element for a list containing several instances of that
element, and nothing else.
Now let's look at the implementation:
void
smartlist_remove(smartlist_t *sl, const void *element)
{
int i;
if (element == NULL)
return;
for (i=0; i < sl->num_used; i++)
if (sl->list[i] == element) {
sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */
i--; /* so we process the new i'th element */
sl->list[sl->num_used] = NULL;
}
}
Based on the implementation, we now see three more edge cases to test:
* Removing NULL from the list.
* Removing an element from the end of the list
* Removing an element from a position other than the end of the list.
=== What should my tests NOT do?
Tests shouldn't require a network connection.
Whenever possible, tests shouldn't take more than a second. Put the test
into test/slow if it genuinely needs to be run.
Tests should not alter global state unless they run with TT_FORK: Tests
should not require other tests to be run before or after them.
Tests should not leak memory or other resources.
When possible, tests should not be over-fit to the implementation. That is,
the test should verify that the documented behavior is implemented, but
should not break if other permissible behavior is later implemented.
=== Advanced techniques: Namespaces === Advanced techniques: Namespaces
XXXX write this. danah boyd made us some really awesome stuff here. Sometimes, when you're doing a lot of mocking at once, it's convenient to
isolate your identifiers within a single namespace. If this were C++, we'd
already have namespaces, but for C, we do the best we can with macros and
token-pasting.
We have some macros defined for this purpose in src/test/test.h. To use
them, you define NS_MODULE to a prefix to be used for your identifiers, and
then use other macros in place of identifier names. See src/test/test.h for
more documentation.
Integration tests: Calling Tor from the outside Integration tests: Calling Tor from the outside
----------------------------------------------- -----------------------------------------------
XXXX WRITEME Some tests need to invoke Tor from the outside, and shouldn't run from the
same process as the Tor test program. Reasons for doing this might include:
* Testing the actual behavior of Tor when run from the command line
* Testing that a crash-handler correctly logs a stack trace
* Verifying that a violating a sandbox or capability requirement will
actually crash the program.
* Needing to run as root in order to test capability inheritance or
user switching.
To add one of these, you generally want a new C program in src/test. Add it
to TESTS and noinst_PROGRAMS if it can run on its own and return success or
failure. If it needs to be invoked multiple times, or it needs to be
wrapped, add a new shell script to TESTS, and the new program to
noinst_PROGRAMS. If you need access to any environment variable from the
makefile (eg ${PYTHON} for a python interpreter), then make sure that the
makefile exports them.
Writing integration tests with Stem Writing integration tests with Stem
----------------------------------- -----------------------------------
XXXX WRITEME The 'stem' library includes extensive unit tests for the Tor controller
protocol.
For more information on writing new tests for stem, have a look around
the tst/* directory in stem, and find a good example to emulate. You
might want to start with
https://gitweb.torproject.org/stem.git/tree/test/integ/control/controller.py
to improve Tor's test coverage.
You can run stem tests from tor with "make test-stem", or see
https://stem.torproject.org/faq.html#how-do-i-run-the-tests .
System testing with Chutney System testing with Chutney
--------------------------- ---------------------------
XXXX WRITEME The 'chutney' program configures and launches a set of Tor relays,
authorities, and clients on your local host. It has a 'test network'
functionality to send traffic through them and verify that the traffic
arrives correctly.
Who knows what evil lurks in the timings of networks? The Shadow knows! You can write new test networks by adding them to 'networks'. To add
----------------------------------------------------------------------- them to Tor's tests, add them to the test-network or test-network-all
targets in Makefile.am.
XXXX WRITEME
(Adding new kinds of program to chutney will still require hacking the
code.)