Filename: 104-short-descriptors.txt
Title: Long and Short Router Descriptors
Version: $Revision$
Last-Modified: $Date$
Author: Nick Mathewson
Created:
Status: Open

Overview:

  This document proposes moving unused-by-clients information from regular
  router descriptors into a special "long form" router descriptor.

  It presents options; it is not yet a complete proposal.

Proposal:

  Some of the costliest fields in the current directory protocol are ones
  that no client actually uses.  In particular, the "read-history" and
  "write-history" fields are used only by the authorities for monitoring the
  status of the network.  If we took them out, the size of a compressed list
  of all the routers would fall by about 60%.  (No other disposable field
  would save more than 2%.)

  One possible solution here is that routers should generate and upload a
  short-form and long-form descriptor.  Only the short-form descriptor should
  ever be used by anybody for routing.  The long-form descriptor should be
  used only for analytics and other tools.  (If we allowed people to route
  with long descriptors, we'd have to ensure that they stayed in sync with
  the short ones somehow.  So let's not do that.)  We can ensure that the
  short descriptors are used by only recommending those in the network
  statuses.

  Another possible solution would be to drop these fields from descriptors,
  and have them uploaded as a part of a separate "bandwidth report" to the
  authorities.  This could help prevent the mistake of using long descriptors
  in the place of short ones.

Other disposable fields:

  Clients don't need these fields, but removing them doesn't help bandwidth
  enough to be worthwhile.
    contact (save about 1%)
    fingerprint (save about 3%)

  We could represent these fields more succinctly, but removing them would
  only save 1%.  (!)
    reject
    accept
  (Apparently, exit polices are highly compressible.)

Issues:

  Indexing long descriptor or bandwidth reports presents an issue: right now
  the way to make sure you have the same copy of a descriptor as everyone
  else is to request the descriptor by its digest, and to make sure to that
  the digest you request is the one that the authorities like.

  Authorities should presumably list the digests of short descriptors, since
  that's what most everybody will be using.  Including a second digest for
  long descriptors/bandwidth reports in the networkstatus would only bloat it
  with information nobody wants.

  Possible solutions are:
    - Drop the property that you can be sure of having the same long
      descriptor as others.  This seems unoptimal.
    - Have a separate extra-information-status that also gets generated by the
      authorities; use it to tell which long descriptors others have.  Also a
      pain.
    - Have short descriptors include a hash of the corresponding long
      descriptor/extra-info.  This would keep the same order of magnitude
      performance increase (~59.2% savings as opposed to 61% savings.)
      This would require longdesc/extra-info downloaders to fetch
      router data before they could know which longdescs/extra info to fetch.
    - Have each authority make a signed concatenated "extra info" document,
      and hope we never need to reconcile them.
    - ????

Migration:

  For long/short descriptor approach:
     * First:
       * Authorities should accept both, now, and silently drop short
         descriptors.
       * Routers should upload both once authorities accept them.
       * There should be a "long descriptor" url and the current "normal" URL.
         Authorities should serve long descriptors from both URLs.
     * Once tools that want long descriptors support fetching them from the
       "long descriptor" URL:
       * Have authorities remember short descriptors, and serve them from the
         'normal' URL.

  For bandwidth info approach:
     * First:
       * Rename it; it won't be just bandwidth forever.
       * Authorities should accept bandwidth info
       * Routers should upload bandwidth info once authorities accept it.
       * There should be a way to download bandwidth info
     * Once tools that want bandwidth info support fetching it:
       * Have routers stop including bandwidth info in their router
         descriptors.