The most expensive part of packing a repository for transport to
another system is enumerating all of the objects in the repository.
Once this gets to the size of the linux-2.6 repository (1.8 million
objects), enumeration can take several CPU minutes and costs a lot
of temporary working set memory.

Teach PackWriter to efficiently reuse an existing "cached pack"
by answering a clone request with a thin pack followed by a larger
cached pack appended to the end.  This requires the repository
owner to first construct the cached pack by hand, and record the
tip commits inside of $GIT_DIR/objects/info/cached-packs:

  cd $GIT_DIR
  root=$(git rev-parse master)
  tmp=objects/.tmp-$$
  names=$(echo $root | git pack-objects --keep-true-parents --revs $tmp)
  for n in $names; do
    chmod a-w $tmp-$n.pack $tmp-$n.idx
    touch objects/pack/pack-$n.keep
    mv $tmp-$n.pack objects/pack/pack-$n.pack
    mv $tmp-$n.idx objects/pack/pack-$n.idx
  done

  (echo "+ $root";
   for n in $names; do echo "P $n"; done;
   echo) >>objects/info/cached-packs

  git repack -a -d

When a clone request needs to include $root, the corresponding
cached pack will be copied as-is, rather than enumerating all of
the objects that are reachable from $root.

For a linux-2.6 kernel repository that should be about 376 MiB,
the above process creates two packs of 368 MiB and 38 MiB[1].
This is a local disk usage increase of ~26 MiB, due to reduced
delta compression between the large cached pack and the smaller
recent activity pack.  The overhead is similar to 1 full copy of
the compressed project sources.

With this cached pack in hand, JGit daemon completes a clone request
in 1m17s less time, but a slightly larger data transfer (+2.39 MiB):

  Before:
    remote: Counting objects: 1861830, done
    remote: Finding sources: 100% (1861830/1861830)
    remote: Getting sizes: 100% (88243/88243)
    remote: Compressing objects: 100% (88184/88184)
    Receiving objects: 100% (1861830/1861830), 376.01 MiB | 19.01 MiB/s, done.
    remote: Total 1861830 (delta 4706), reused 1851053 (delta 1553844)
    Resolving deltas: 100% (1564621/1564621), done.

    real  3m19.005s

  After:
    remote: Counting objects: 1601, done
    remote: Counting objects: 1828460, done
    remote: Finding sources: 100% (50475/50475)
    remote: Getting sizes: 100% (18843/18843)
    remote: Compressing objects: 100% (7585/7585)
    remote: Total 1861830 (delta 2407), reused 1856197 (delta 37510)
    Receiving objects: 100% (1861830/1861830), 378.40 MiB | 31.31 MiB/s, done.
    Resolving deltas: 100% (1559477/1559477), done.

    real 2m2.938s

Repository owners can periodically refresh their cached packs by
repacking their repository, folding all newer objects into a larger
cached pack.  Since repacking is already considered to be a normal
Git maintenance activity, this isn't a very big burden.

[1] In this test $root was set back about two weeks.

Change-Id: Ib87131d5c4b5e8c5cacb0f4fe16ff4ece554734b
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Similar to what we did with the file code, move the pack writer
into its own package so the related classes and their package
private methods are hidden from the rest of the library.

Change-Id: Ic1b5c7c8c8d266e90c910d8d68dfc8e93586854f
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Rather than making a loader, and then using that to fill the object
representation, parse the header and set up our data directly.
This saves some time, as we don't waste cycles on information we
won't use right now.

The weight computed for a representation is now its actual stored
size in the pack file, rather than its inflated size.  This accounts
for changes made when the compression level is modified on the
repository.  It is however more costly to determine the weight of
the object, since we have to find its length in the pack.  To try and
recover that cost we now cache the length as part of our ObjectToPack
record, so it doesn't have to be found during the output phase.

A LocalObjectToPack now costs us (assuming 32 bit pointers):

                   (32 bit)     (64 bit)
  vm header:         8 bytes      8 bytes
  ObjectId:         20 bytes     20 bytes
  PackedObjectInfo: 12 bytes     12 bytes
  ObjectToPack:      8 bytes     12 bytes
  LocalOTP:         20 bytes     24 bytes
                 -----------    ---------
                    68 bytes     74 bytes

Change-Id: I923d2736186eb2ac8ab498d3eb137e17930fcb50
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This move isolates all of the local file specific implementation code
into a single package, where their package-private methods and support
classes are properly hidden away from the rest of the core library.

Because of the sheer number of files impacted, I have limited this
change to only the renames and the updated imports.

Change-Id: Icca4884e1a418f83f8b617d0c4c78b73d8a4bd17
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The new selection implementation uses a public API on the
ObjectReader, allowing the storage library to enumerate its
candidates and select the best one for this packer without
needing to build a temporary list of the candidates first.

Change-Id: Ie01496434f7d3581d6d3bbb9e33c8f9fa649b6cd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Replace the old crude event listener system with a much more generic
implementation, patterned after the event dispatch techniques used
in Google Web Toolkit 1.5 and later.

Each event delivers to an interface that defines a single method,
and the event itself is what performs the delivery in a type-safe
way through its own dispatch method.

Listeners are registered in a generic listener list, indexed by
the interface they implement and wish to receive an event for.
Delivery of events is performed by looping through all listeners
implementing the event's corresponding listener interface, and using
the event's own dispatch method to deliver the event.  This is the
classical "double dispatch" pattern for event delivery.

Listeners can be unregistered by invoking remove() on their
registration handle.  This change therefore requires application
code to track the handle if it wishes to remove the listener at a
later point in time.

Event delivery is now exposed as a generic public method on the
Repository class, making it easier for any type of message to
be sent out to any type of listener that has registered, without
needing to pre-arrange for type-safe fireFoo() methods.

New event types can be added in the future simply by defining a
new RepositoryEvent subclass and a corresponding RepositoryListener
interface that it dispatches to.  By always adding new events through
a new interface, we never need to worry about defining an Adapter
to provide default no-op implementations of new event methods.

Change-Id: I651417b3098b9afc93d91085e9f0b2265df8fc81
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Per CQ 3448 this is the initial contribution of the JGit project
to eclipse.org.  It is derived from the historical JGit repository
at commit 3a2dd9921c.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>