nihilist/tor
1
0
Fork 0
mirror of https://gitlab.torproject.org/tpo/core/tor.git synced 2024-11-14 07:03:44 +01:00
Code Issues Packages Projects Releases Wiki Activity
e5601f14ed
tor/src/feature/dircommon/consdiff.h

100 lines
4.2 KiB
C
Raw Normal View History

Consensus diff backend from Daniel Martí GSOC project. (This commit was extracted by nickm based on the final outcome of the project, taking only the changes in the files touched by this commit from the consdiff_rebased branch. The directory-system changes are going to get worked on separately.)
2017-03-07 15:58:30 +01:00
/* Copyright (c) 2014, Daniel Martí
Update copyrights to 2018.
2018-06-20 14:13:28 +02:00
* Copyright (c) 2014-2018, The Tor Project, Inc. */
Consensus diff backend from Daniel Martí GSOC project. (This commit was extracted by nickm based on the final outcome of the project, taking only the changes in the files touched by this commit from the consdiff_rebased branch. The directory-system changes are going to get worked on separately.)
2017-03-07 15:58:30 +01:00
/* See LICENSE for licensing information */
#ifndef TOR_CONSDIFF_H
#define TOR_CONSDIFF_H
Fix every include path changed in the previous commit (automated) I am very glad to have written this script.
2018-07-05 22:34:59 +02:00
#include "core/or/or.h"
Consensus diff backend from Daniel Martí GSOC project. (This commit was extracted by nickm based on the final outcome of the project, taking only the changes in the files touched by this commit from the consdiff_rebased branch. The directory-system changes are going to get worked on separately.)
2017-03-07 15:58:30 +01:00
String-based API for consensus diffs. Also, add very strict split/join functions, and totally forbid nonempty files that end with somethig besides a newline. This change is necessary to ensure that diff/apply are actually reliable inverse operations.
2017-03-07 19:45:32 +01:00
char *consensus_diff_generate(const char *cons1,
const char *cons2);
char *consensus_diff_apply(const char *consensus,
const char *diff);
Request (and try to use) consensus diffs.
2017-05-03 21:08:59 +02:00
int looks_like_a_consensus_diff(const char *document, size_t len);
String-based API for consensus diffs. Also, add very strict split/join functions, and totally forbid nonempty files that end with somethig besides a newline. This change is necessary to ensure that diff/apply are actually reliable inverse operations.
2017-03-07 19:45:32 +01:00
#ifdef CONSDIFF_PRIVATE
Remove bloom filters, order statistics, and bitarrays from container.h
2018-06-21 22:48:54 +02:00
#include "lib/container/bitarray.h"
Avoid all needless memory copies when computing consensus diffs. Previously, we operated on smartlists of NUL-terminated strings, which required us to copy both inputs to produce the NUL-terminated strings. Then we copied parts of _those_ inputs to produce an output smartlist of NUL-terminated strings. And finally, we concatenated everything into a final resulting string. This implementation, instead, uses a pointer-and-extent pattern to represent each line as a pointer into the original inputs and a length. These line objects are then added by reference into the output. No actual bytes are copied from the original strings until we finally concatenate the final result together. Bookkeeping structures and newly allocated strings (like ed commands) are allocated inside a memarea, to avoid needless mallocs or complicated should-I-free-this-or-not bookkeeping. In my measurements, this improves CPU performance by something like 18%. The memory savings should be much, much higher.
2017-03-14 20:00:39 +01:00
struct memarea_t;
/** Line type used for constructing consensus diffs. Each of these lines
* refers to a chunk of memory allocated elsewhere, and is not necessarily
* NUL-terminated: this helps us avoid copies and save memory. */
typedef struct cdline_t {
const char *s;
uint32_t len;
} cdline_t;
prop140: Use sha3-256, not sha2-256 This is a protocol update from recent prop140 changes. Also, per #21673, we need to check the entire document, including signatures.
2017-03-07 19:02:00 +01:00
typedef struct consensus_digest_t {
uint8_t sha3_256[DIGEST256_LEN];
} consensus_digest_t;
String-based API for consensus diffs. Also, add very strict split/join functions, and totally forbid nonempty files that end with somethig besides a newline. This change is necessary to ensure that diff/apply are actually reliable inverse operations.
2017-03-07 19:45:32 +01:00
STATIC smartlist_t *consdiff_gen_diff(const smartlist_t *cons1,
const smartlist_t *cons2,
const consensus_digest_t *digests1,
Avoid all needless memory copies when computing consensus diffs. Previously, we operated on smartlists of NUL-terminated strings, which required us to copy both inputs to produce the NUL-terminated strings. Then we copied parts of _those_ inputs to produce an output smartlist of NUL-terminated strings. And finally, we concatenated everything into a final resulting string. This implementation, instead, uses a pointer-and-extent pattern to represent each line as a pointer into the original inputs and a length. These line objects are then added by reference into the output. No actual bytes are copied from the original strings until we finally concatenate the final result together. Bookkeeping structures and newly allocated strings (like ed commands) are allocated inside a memarea, to avoid needless mallocs or complicated should-I-free-this-or-not bookkeeping. In my measurements, this improves CPU performance by something like 18%. The memory savings should be much, much higher.
2017-03-14 20:00:39 +01:00
const consensus_digest_t *digests2,
struct memarea_t *area);
String-based API for consensus diffs. Also, add very strict split/join functions, and totally forbid nonempty files that end with somethig besides a newline. This change is necessary to ensure that diff/apply are actually reliable inverse operations.
2017-03-07 19:45:32 +01:00
STATIC char *consdiff_apply_diff(const smartlist_t *cons1,
const smartlist_t *diff,
const consensus_digest_t *digests1);
STATIC int consdiff_get_digests(const smartlist_t *diff,
char *digest1_out,
char *digest2_out);
Consensus diff backend from Daniel Martí GSOC project. (This commit was extracted by nickm based on the final outcome of the project, taking only the changes in the files touched by this commit from the consdiff_rebased branch. The directory-system changes are going to get worked on separately.)
2017-03-07 15:58:30 +01:00
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
/** Data structure to define a slice of a smarltist. */
typedef struct smartlist_slice_t {
/**
* Smartlist that this slice is made from.
* References the whole original smartlist that the slice was made out of.
* */
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
const smartlist_t *list;
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
/** Starting position of the slice in the smartlist. */
int offset;
/** Length of the slice, i.e. the number of elements it holds. */
int len;
} smartlist_slice_t;
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
STATIC smartlist_t *gen_ed_diff(const smartlist_t *cons1,
Avoid all needless memory copies when computing consensus diffs. Previously, we operated on smartlists of NUL-terminated strings, which required us to copy both inputs to produce the NUL-terminated strings. Then we copied parts of _those_ inputs to produce an output smartlist of NUL-terminated strings. And finally, we concatenated everything into a final resulting string. This implementation, instead, uses a pointer-and-extent pattern to represent each line as a pointer into the original inputs and a length. These line objects are then added by reference into the output. No actual bytes are copied from the original strings until we finally concatenate the final result together. Bookkeeping structures and newly allocated strings (like ed commands) are allocated inside a memarea, to avoid needless mallocs or complicated should-I-free-this-or-not bookkeeping. In my measurements, this improves CPU performance by something like 18%. The memory savings should be much, much higher.
2017-03-14 20:00:39 +01:00
const smartlist_t *cons2,
struct memarea_t *area);
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
STATIC smartlist_t *apply_ed_diff(const smartlist_t *cons1,
Fix an abstraction violation. Don't alias the insides of smartlist_t; that way lies madness.
2017-03-07 19:15:43 +01:00
const smartlist_t *diff,
int start_line);
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
STATIC void calc_changes(smartlist_slice_t *slice1, smartlist_slice_t *slice2,
bitarray_t *changed1, bitarray_t *changed2);
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
STATIC smartlist_slice_t *smartlist_slice(const smartlist_t *list,
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
int start, int end);
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
STATIC int next_router(const smartlist_t *cons, int cur);
STATIC int *lcs_lengths(const smartlist_slice_t *slice1,
const smartlist_slice_t *slice2,
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
int direction);
STATIC void trim_slices(smartlist_slice_t *slice1, smartlist_slice_t *slice2);
Avoid all needless memory copies when computing consensus diffs. Previously, we operated on smartlists of NUL-terminated strings, which required us to copy both inputs to produce the NUL-terminated strings. Then we copied parts of _those_ inputs to produce an output smartlist of NUL-terminated strings. And finally, we concatenated everything into a final resulting string. This implementation, instead, uses a pointer-and-extent pattern to represent each line as a pointer into the original inputs and a length. These line objects are then added by reference into the output. No actual bytes are copied from the original strings until we finally concatenate the final result together. Bookkeeping structures and newly allocated strings (like ed commands) are allocated inside a memarea, to avoid needless mallocs or complicated should-I-free-this-or-not bookkeeping. In my measurements, this improves CPU performance by something like 18%. The memory savings should be much, much higher.
2017-03-14 20:00:39 +01:00
STATIC int base64cmp(const cdline_t *hash1, const cdline_t *hash2);
STATIC int get_id_hash(const cdline_t *line, cdline_t *hash_out);
STATIC int is_valid_router_entry(const cdline_t *line);
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
STATIC int smartlist_slice_string_pos(const smartlist_slice_t *slice,
Avoid all needless memory copies when computing consensus diffs. Previously, we operated on smartlists of NUL-terminated strings, which required us to copy both inputs to produce the NUL-terminated strings. Then we copied parts of _those_ inputs to produce an output smartlist of NUL-terminated strings. And finally, we concatenated everything into a final resulting string. This implementation, instead, uses a pointer-and-extent pattern to represent each line as a pointer into the original inputs and a length. These line objects are then added by reference into the output. No actual bytes are copied from the original strings until we finally concatenate the final result together. Bookkeeping structures and newly allocated strings (like ed commands) are allocated inside a memarea, to avoid needless mallocs or complicated should-I-free-this-or-not bookkeeping. In my measurements, this improves CPU performance by something like 18%. The memory savings should be much, much higher.
2017-03-14 20:00:39 +01:00
const cdline_t *string);
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
STATIC void set_changed(bitarray_t *changed1, bitarray_t *changed2,
Use "const" in consdiff.[ch]
2017-03-07 19:11:38 +01:00
const smartlist_slice_t *slice1,
const smartlist_slice_t *slice2);
Avoid all needless memory copies when computing consensus diffs. Previously, we operated on smartlists of NUL-terminated strings, which required us to copy both inputs to produce the NUL-terminated strings. Then we copied parts of _those_ inputs to produce an output smartlist of NUL-terminated strings. And finally, we concatenated everything into a final resulting string. This implementation, instead, uses a pointer-and-extent pattern to represent each line as a pointer into the original inputs and a length. These line objects are then added by reference into the output. No actual bytes are copied from the original strings until we finally concatenate the final result together. Bookkeeping structures and newly allocated strings (like ed commands) are allocated inside a memarea, to avoid needless mallocs or complicated should-I-free-this-or-not bookkeeping. In my measurements, this improves CPU performance by something like 18%. The memory savings should be much, much higher.
2017-03-14 20:00:39 +01:00
STATIC int consensus_split_lines(smartlist_t *out, const char *s,
struct memarea_t *area);
STATIC void smartlist_add_linecpy(smartlist_t *lst, struct memarea_t *area,
const char *s);
STATIC int lines_eq(const cdline_t *a, const cdline_t *b);
STATIC int line_str_eq(const cdline_t *a, const char *b);
Make consensus diff sha3 operations mockable. (We'll want this for fuzzing)
2017-03-07 20:12:53 +01:00
MOCK_DECL(STATIC int,
consensus_compute_digest,(const char *cons,
consensus_digest_t *digest_out));
bug#22143/prop#140: identify input diffs by their digest-as-signed See may 3 changes to prop140 for more background.
2017-05-03 17:48:08 +02:00
MOCK_DECL(STATIC int,
consensus_compute_digest_as_signed,(const char *cons,
consensus_digest_t *digest_out));
Make consensus diff sha3 operations mockable. (We'll want this for fuzzing)
2017-03-07 20:12:53 +01:00
MOCK_DECL(STATIC int,
consensus_digest_eq,(const uint8_t *d1,
const uint8_t *d2));
Run our #else/#endif annotator on our source code.
2017-09-15 22:24:44 +02:00
#endif /* defined(CONSDIFF_PRIVATE) */
Use "STATIC" to export consdiff fns for testing Previously test_consdiff.c just did #include "consdiff.c", which is not great style, and messes up coverage testing.
2017-03-07 16:22:00 +01:00
Run our #else/#endif annotator on our source code.
2017-09-15 22:24:44 +02:00
#endif /* !defined(TOR_CONSDIFF_H) */
Reference in New Issue Copy Permalink