Filename: 104-short-descriptors.txt
Title: Long and Short Router Descriptors
Version: $Revision$
Last-Modified: $Date$
Author: Nick Mathewson
Created:
Status: Open

Overview:

  This document proposes moving unused-by-clients information from regular
  router descriptors into a special "long form" router descriptor.

  It presents options; it is not yet a complete proposal.

Proposal:

  Some of the costliest fields in the current directory protocol are ones
  that no client actually uses.  In particular, the "read-history" and
  "write-history" fields are used only by the authorities for monitoring the
  status of the network.  If we took them out, the size of a compressed list
  of all the routers would fall by about 60%.  (No other disposable field
  would save more than 2%.)

  One possible solution here is that routers should generate and upload a
  short-form and long-form descriptor.  Only the short-form descriptor should
  ever be used by anybody for routing.  The long-form descriptor should be
  used only for analytics and other tools.  (If we allowed people to route
  with long descriptors, we'd have to ensure that they stayed in sync with
  the short ones somehow.  So let's not do that.)  We can ensure that the
  short descriptors are used by only recommending those in the network
  statuses.

  Another possible solution would be to drop these fields from descriptors,
  and have them uploaded as a part of a separate "bandwidth report" to the
  authorities.  This could help prevent the mistake of using long descriptors
  in the place of short ones. It could also be generalized later to be an
  overall status report, to include sanitized GeoIP information and whatever
  else comes up.

Other disposable fields:

  Clients don't need these fields, but removing them doesn't help bandwidth
  enough to be worthwhile.
    contact (save about 1%)
    fingerprint (save about 3%)

  We could represent these fields more succinctly, but removing them would
  only save 1%.  (!)
    reject
    accept
  (Apparently, exit polices are highly compressible.)

  [Does size-on-disk matter to anybody? Some clients and servers don't
   have much disk, or have really slow disk (e.g. USB). And we don't
   store caches compressed right now. -RD]

Issues:

  Indexing long descriptor or bandwidth reports presents an issue: right now
  the way to make sure you have the same copy of a descriptor as everyone
  else is to request the descriptor by its digest, and to make sure that
  the digest you request is the one that the authorities like.

  Authorities should presumably list the digests of short descriptors, since
  that's what most everybody will be using.  Including a second digest for
  long descriptors/bandwidth reports in the networkstatus would only bloat it
  with information nobody wants.

  Possible solutions are:
   1) Drop the property that you can be sure of having the same long
      descriptor as others.  This seems unoptimal, but if nobody caches
      long descriptors so you have to go to the authority to get them,
      maybe it's not so bad.
   2) Have a separate extra-information-status that also gets generated by the
      authorities; use it to tell which long descriptors others have.  Also a
      pain.
   3) Have short descriptors include a hash of the corresponding long
      descriptor/extra-info.  This would keep the same order of magnitude
      performance increase (~59.2% savings as opposed to 61% savings.)
      This would require longdesc/extra-info downloaders to fetch
      router data before they could know which longdescs/extra info to fetch.
   4) Have each authority make a signed concatenated "extra info" document,
      and hope we never need to reconcile them.
   5) ????

Migration:

  For long/short descriptor approach:
     * First:
       * Authorities should accept both, now, and silently drop short
         descriptors.
       * Routers should upload both once authorities accept them.
       * There should be a "long descriptor" url named
         /tor/server/fp-detailed/ and the current "normal" URL.
         Authorities should serve long descriptors from both URLs.
         There's no such thing as asking for a long descriptor by
         its digest.
     * Once tools that want long descriptors support fetching them from the
       "long descriptor" URL:
       * Have authorities remember short descriptors, and serve them from the
         'normal' URL.
       These tools include:
         lefkada's exit.py script.
         tor26's noreply script and general directory cache.
         https://nighteffect.us/tns/ for its graphs
         and check with or-talk for the rest, once it's time.

  For bandwidth info approach:
     * First:
       * Rename it; it won't be just bandwidth forever.
       * Authorities should accept bandwidth info
       * Routers should upload bandwidth info once authorities accept it.
       * There should be a way to download bandwidth info
     * Once tools that want bandwidth info support fetching it:
       * Have routers stop including bandwidth info in their router
         descriptors.