tor/doc/HACKING/design/01.00-lib-overview.md

Library code in Tor.

Most of Tor's utility code is in modules in the src/lib subdirectory. In general, this code is not necessarily Tor-specific, but is instead possibly useful for other applications.

This code includes:

• Compatibility wrappers, to provide a uniform API across different platforms.

• Library wrappers, to provide a tor-like API over different libraries that Tor uses for things like compression and cryptography.

• Containers, to implement some general-purpose data container types.

The modules in src/lib are currently well-factored: each one depends only on lower-level modules. You can see an up-to-date list of the modules sorted from lowest to highest level by running ./scripts/maint/practracker/includes.py --toposort.

As of this writing, the library modules are (from lowest to highest level):

• lib/cc -- Macros for managing the C compiler and language. Includes macros for improving compatibility and clarity across different C compilers.

• lib/version -- Holds the current version of Tor.

• lib/testsupport -- Helpers for making test-only code and test mocking support.

• lib/defs -- Lowest-level constants used in many places across the code.

• lib/subsys -- Types used for declaring a "subsystem". A subsystem is a module with support for initialization, shutdown, configuration, and so on.

• lib/conf -- Types and macros used for declaring configuration options.

• lib/arch -- Compatibility functions and macros for handling differences in CPU architecture.

• lib/err -- Lowest-level error handling code: responsible for generating stack traces, handling raw assertion failures, and otherwise reporting problems that might not be safe to report via the regular logging module.

• lib/malloc -- Wrappers and utilities for memory management.

• lib/intmath -- Utilities for integer mathematics.

• lib/fdio -- Utilities and compatibility code for reading and writing data on file descriptors (and on sockets, for platforms where a socket is not a kind of fd).

• lib/lock -- Compatibility code for declaring and using locks. Lower-level than the rest of the threading code.

• lib/ctime -- Constant-time implementations for data comparison and table lookup, used to avoid timing side-channels from standard implementations of memcmp() and so on.

• lib/string -- Low-level compatibility wrappers and utility functions for string manipulation.

• lib/wallclock -- Compatibility and utility functions for inspecting and manipulating the current (UTC) time.

• lib/osinfo -- Functions for inspecting the version and capabilities of the operating system.

• lib/smartlist_core -- The bare-bones pieces of our dynamic array ("smartlist") implementation. There are higher-level pieces, but these ones are used by (and therefore cannot use) the logging code.

• lib/log -- Implements the logging system used by all higher-level Tor code. You can think of this as the logical "midpoint" of the library code: much of the higher-level code is higher-level because it uses the logging module, and much of the lower-level code is specifically written to avoid having to log, because the logging module depends on it.

• lib/container -- General purpose containers, including dynamic arrays, hashtables, bit arrays, weak-reference-like "handles", bloom filters, and a bit more.

• lib/trace -- A general-purpose API for introducing function-tracing functionality into Tor. Currently not much used.

• lib/thread -- Threading compatibility and utility functionality, other than low-level locks (which are in lib/lock) and workqueue/threadpool code (which belongs in lib/evloop).

• lib/term -- Code for terminal manipulation functions (like reading a password from the user).

• lib/memarea -- A data structure for a fast "arena" style allocator, where the data is freed all at once. Used for parsing.

• lib/encoding -- Implementations for encoding data in various formats, datatypes, and transformations.

• lib/dispatch -- A general-purpose in-process message delivery system. Used by lib/pubsub to implement our inter-module publish/subscribe system.

• lib/sandbox -- Our Linux seccomp2 sandbox implementation.

• lib/pubsub -- Code and macros to implement our publish/subscribe message passing system.

• lib/fs -- Utility and compatibility code for manipulating files, filenames, directories, and so on.

• lib/confmgt -- Code to parse, encode, and manipulate our configuration files, state files, and so forth.

• lib/crypt_ops -- Cryptographic operations. This module contains wrappers around the cryptographic libraries that we support, and implementations for some higher-level cryptographic constructions that we use.

• lib/meminfo -- Functions for inspecting our memory usage, if the malloc implementation exposes that to us.

• lib/time -- Higher level time functions, including fine-gained and monotonic timers.

• lib/math -- Floating-point mathematical utilities, including compatibility code, and probability distributions.

• lib/buf -- A general purpose queued buffer implementation, similar to the BSD kernel's "mbuf" structure.

• lib/net -- Networking code, including address manipulation, compatibility wrappers,

• lib/compress -- A compatibility wrapper around several compression libraries, currently including zlib, zstd, and lzma.

• lib/geoip -- Utilities to manage geoip (IP to country) lookups and formats.

• lib/tls -- Compatibility wrappers around the library (NSS or OpenSSL, depending on configuration) that Tor uses to implement the TLS link security protocol.

• lib/evloop -- Tools to manage the event loop and related functionality, in order to implement asynchronous networking, timers, periodic events, and other scheduling tasks.

• lib/process -- Utilities and compatibility code to launch and manage subprocesses.

What belongs in lib?

In general, if you can imagine some program wanting the functionality you're writing, even if that program had nothing to do with Tor, your functionality belongs in lib.

If it falls into one of the existing "lib" categories, your functionality belongs in lib.

If you are using platform-specific #ifdefs to manage compatibility issues among platforms, you should probably consider whether you can put your code into lib.