Shawn O. Pearce [Mon, 28 Feb 2011 23:39:31 +0000 (15:39 -0800)]
PackWriter: Don't reuse commit or tag deltas
JGit doesn't generate deltas for commit or tag objects when it packs
a repository from scratch. This is an explicit design decision that
is (mostly) justified by the fact that these objects do not delta
compress well.
Annotated tags are made once on stable points of the project history,
it is unlikely they will ever appear again with sufficient common
text to justify using a delta over just deflating the raw content.
JGit never tries to delta compress annotated tags and I take the
stance that these are best stored as non-deltas given how frequently
they might be accessed by repository viewers.
Commits only have sufficient common text when they are cherry-picked
to forward-port or back-port a change from one branch to another.
Even in these cases the distance between the commits as returned
by the log traversal has to be small enough that they would both
appear in the delta search window at the same time in order to
delta compress one of the messages against the other. JGit never
tries to delta compress commits, as it requires a lot of CPU time
but typically does not produce a smaller pack file.
Avoid reusing deltas for either of these types when constructing a
new pack. To avoid killing performance during serving of network
clients, UploadPack disables this code change by allowing PackWriter
to reuse delta commits. Repositories that were already repacked by
C Git will not have their delta commits decompressed and recompressed
on the fly during object writing, saving server-side CPU resources.
Change-Id: I749407e7c5c677e05e4d054b40db7656cfa7fca8 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 1 Mar 2011 17:28:11 +0000 (09:28 -0800)]
PackWriter: Do not delta compress already packed objects
This is a tiny optimization to how delta search works. Checking for
isReuseAsIs() avoids doing delta compression search on non-delta
objects already stored in packs within the repository. Such objects
are not likely to be delta compressable, as they were already delta
searched when their containing pack was generated and they were
not delta compressed at that time. Doing delta compression now is
unlikely to produce a different result, but would waste a lot of CPU.
The isReuseAsIs() flag is checked before isDoNotDelta() because it
is very common to reuse objects in the output pack. Most objects
get reused, and only a handful have the isDoNotDelta() bit set.
Moving the check earlier allows the loop to more quickly skip
through objects that will never need to be considered.
Change-Id: Ied757363f775058177fc1befb8ace20fe9759bac Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 1 Mar 2011 18:06:39 +0000 (10:06 -0800)]
Paper bag fix BatchingProgressMonitor alarm queue
The alarm queue threads were started with an empty task body, which
meant the thread started and terminated immediately, leaving the
queue itself with no worker.
Change-Id: I2a9b5fe9c2bdff4a5e0f7ec7ad41a54b41a4ddd6 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 1 Mar 2011 03:34:06 +0000 (19:34 -0800)]
ProgressMonitor: Refactor to use background alarms
Instead of polling the system clock on every update(1) method call,
use a scheduled executor to toggle a volatile once per second until
the task is done. Check the volatile on each update(int), looking
to see if output should occur.
This limits progress output to either once per 1% complete, or once
per second. To save time during update calls the timer isn't reset
during each 1% of output, which means we may see one unnecessary
output trigger if at least 1% completed during the one second of the
alarm time.
Change-Id: I8fdd7e31c37bef39a5d1b3da7105da0ef879eb84 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Matthias Sohn [Mon, 28 Feb 2011 23:21:14 +0000 (00:21 +0100)]
Fix NPE on checkout of remote tracking branch
Checkout of remote tracking branch failed when no local branch
existed. Also enhance RepositoryTestCase to enable checking index
state of another test repository.
Bug: 337695
Change-Id: Idf4c05bdf23b5161688818342b2bf9a45b49f479 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn O. Pearce [Sat, 26 Feb 2011 01:24:55 +0000 (17:24 -0800)]
Merge branch 'stable-0.11'
* stable-0.11:
JGit 0.11.3
Fix NullPointer when pulling from a deleted local branch
smart-http: Fix recognition of gzip encoding
Fix processing of broken symbolic references in RefDirectory
CreateBranchCommand: Wrong existence check
Qualify post 0.11.1 builds
Shawn O. Pearce [Sat, 26 Feb 2011 01:20:14 +0000 (17:20 -0800)]
UnpackedObject: Fix readSome() when initial read is short
JDK7 changed behavior slightly on some InputStream types, resulting in
the first read being shorter than the count requested. That caused us
to overwrite the earlier part of the buffer with later data, as the
offset index wasn't updated in the loop.
Fix the loop to increment offset by the number of bytes read in this
iteration, so the next read appends to the buffer rather than doing an
overwrite.
Bug: 338119
Change-Id: I222fb2f993cd9b637b6b8d93daab5777ef7ec7a6 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Matthias Sohn [Thu, 24 Feb 2011 12:52:24 +0000 (13:52 +0100)]
FetchCommand: do not set a null credentials provider
FetchCommand now does not set a null credentials provider on
Transport because in this case the default provider is replaced with
null and the default mechanism for providing credentials is not
working.
Change-Id: I44096aa856f031545df39d4b09af198caa2c21f6 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn O. Pearce [Wed, 23 Feb 2011 20:00:25 +0000 (12:00 -0800)]
RevWalk: Don't release during inMergeBase()
In bc1af8459e ("RevWalk: Don't reset ObjectReader when stopping") we
stopped releasing the reader when the current log traversal is over.
This should have also been applied to the merge base logic that is
buried within MergeGenerator, but got missed.
Change-Id: I8328f43f02cba06fd545e22134872e781b9d4d36 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Wed, 23 Feb 2011 02:56:51 +0000 (18:56 -0800)]
PackWriter: Add missing timers to Statistics
We did not record the time spent on the object reuse search or the
object size lookup, both of which occur between the counting phase and
the compressing phase. If there are enough objects involved, these
times can be significant so its worth timing them and recording it.
Change-Id: I89084acfc598bb6533d75d90cb8de459f0ed93be Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Sasa Zivkov [Mon, 21 Feb 2011 15:43:06 +0000 (16:43 +0100)]
Show notes in Log CLI command
Support for --no-standard-notes and --show-notes=REF options is added
to the Log command. The --show-notes option can be specified more than
once if more than one notes branch should be used for showing notes.
The notes are displayed from note branches in the order how the note
branches are specified in the command line. However, the standard note,
from the refs/notes/commits, is always displayed as first unless
the --no-standard-notes options is given.
Change-Id: I4e7940804ed9d388b625b8e8a8e25bfcf5ee15a6 Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Stefan Lay [Wed, 16 Feb 2011 14:46:26 +0000 (15:46 +0100)]
Fix NullPointer when pulling from a deleted local branch
A checked Exception is thrown instead.
The reason for throwing an Exception is that the state of the
repository is inconsistent in this case: There is a merge
configuration containing a non-existing local branch. Ideally the
deletion of a local branch should also delete the corresponding
merge configuration.
Bug: 337315
Change-Id: I8ed57d5aaed60aaab685fc11a8695e474e60215f Signed-off-by: Stefan Lay <stefan.lay@sap.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn O. Pearce [Tue, 15 Feb 2011 22:09:42 +0000 (14:09 -0800)]
smart-http: Fix recognition of gzip encoding
Some clients coming through proxies may advertise a different
Accept-Encoding, for example "Accept-Encoding: gzip(proxy)".
Matching by substring causes us to identify this as a false positive;
that the client understands gzip encoding and will inflate the
response before reading it.
In this particular case however it doesn't. Its the reverse proxy
server in front of JGit letting us know the proxy<->JGit link can
be gzip compressed, while the client<->proxy part of the link is not:
client <-- no gzip --> proxy <-- gzip --> JGit
Use a more standard method of parsing by splitting the value into
tokens, and only using gzip if one of the tokens is exactly the
string "gzip". Add a unit test to make sure this isn't broken in
the future.
Change-Id: Ib4c40f9db177322c7a2640808a6c10b3c4a73819 Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn O. Pearce [Sat, 19 Feb 2011 01:55:53 +0000 (17:55 -0800)]
PackWriter: Hoist and cluster reference targets
Many source browsers and network related tools like UploadPack need
to find and parse the target of all branches and annotated tags
within the repository during their startup phase. Clustering these
together into the same part of the pack file will improve locality,
reducing thrashing when an application starts and needs to load
all of these into memory at once.
To prevent bottlenecking basic log viewing tools that are scannning
backwards from the tip of a current branch (and don't need tags)
we place this cluster of older targets after 4096 newer commits
have already been placed into the pack stream. 4096 was chosen as
a rough guess, but was based on a few factors:
- log viewers typically show 5-200 commits per page
- users only view the first page or two
- DHT can cram 2200-4000 commits per 1 MiB chunk
thus these will fall into the second commit chunk (roughly)
Unfortunately this placement hurts history tools that are scanning
backwards through the commit graph and completely ignored tags or
branch heads when they started.
An ancient tagged commit is no longer positioned behind its first
child (its now much earlier), resulting in a page fault for the
parser to reload this cluster of objects on demand. This may be
an acceptable loss. If a user is walking backwards and has already
scanned through more than 4096 commits of history, waiting for the
region to reload isn't really that bad compared to the amount of
time already spent.
If the repository is so small that there are less than 4096 commits,
this change has no impact on the placement of objects.
Change-Id: If3052e430d305e17878d94145c93754f56b74c61 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sat, 19 Feb 2011 01:31:32 +0000 (17:31 -0800)]
PackWriter: Parse tag target objects in a batch
If the underlying storage has a high latency per SHA-1 lookup
(e.g. the DHT support we are working on), parsing each wanted
annotated tag object back to its underlying commit is too slow,
its a sequential lookup for each tag. With hundreds of tags in
a repository this takes far too long.
Instead queue up a list of the tags whose objects need to be found,
and then locate all of those in one parseAny batch. This works
for the common case of annotated tag to single tree or commit.
For the less often used tag->tag->commit, it at least gets us
one level parsed in the larger batch before we have to go back to
sequential lookups.
Change-Id: I94beef3f14281406f15c8cf9fa02d83faf102a19 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sat, 19 Feb 2011 01:06:36 +0000 (17:06 -0800)]
PackWriter: Short-circuit counting on full cached pack reuse
If one or more cached packs fully covers the request, don't bother
with looking up the objects and trying to walk the graph. Just use
the cached packs and return immediately.
This helps clones of quiet repositories that have not been modified
since their last repack, its likely the cached packs are accurate
and no graph walking is required.
Change-Id: I9062a5ac2f71b525322590209664a84051fd5f8a Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 18 Feb 2011 22:14:56 +0000 (14:14 -0800)]
BundleWriter: Always use OFS_DELTA
CGit just learned to always use OFS_DELTA when writing out bundle
files. This makes sense because bundle came about well after
OFS_DELTA was established, so any version of CGit that can read a
bundle file can also read OFS_DELTA. Since OFS_DELTA is smaller,
always use it when writing bundles.
Change-Id: I44f9921494798ea0c99e16eab58b87bebeb9aff5 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 17 Feb 2011 01:41:35 +0000 (17:41 -0800)]
PackWriter: Sort commits by parse order to improve locality
RevWalk in JGit and the revision code in C Git both parse commits out
of the pack file in an order that differs from strict timestamp and
topological sorting. Both implementations pop a commit from the head
of a date queue, and then immediately parse all of its parents in
order to insert those into the date queue at the proper positions as
determined by their committer timestamp field. This implies that the
parents are parsed when their most recent child is popped from the
queue, and not where they are popped during traversal.
Hoisting a parent commit to be immediately behind its child improves
locality by making sure all parents of a merge are clustered together,
and thus can be paged into the parser by the pack file buffering
system (aka WindowCache in JGit) together.
Change-Id: I80f9e64cafa2e8f082776b43845edf23065386a2 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Stefan Lay [Wed, 16 Feb 2011 14:46:26 +0000 (15:46 +0100)]
Fix NullPointer when pulling from a deleted local branch
A checked Exception is thrown instead.
The reason for throwing an Exception is that the state of the
repository is inconsistent in this case: There is a merge
configuration containing a non-existing local branch. Ideally the
deletion of a local branch should also delete the corresponding
merge configuration.
Bug: 337315
Change-Id: I71e56ffb90e11e6e3c1bbd964ad63972d67990c0 Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Shawn O. Pearce [Tue, 15 Feb 2011 22:46:30 +0000 (14:46 -0800)]
smart-http: Support progress in ReceivePack
As PackParser supports a progress meter for the "Resolving deltas"
phase of its work, we should export this to smart HTTP clients so
they know the server is still working on their (large) upload.
However this isn't as simple as just dropping in a binding for
the SmartOutputStream to flush when its told to. We want to
avoid spurious flushes triggered by the use of sideband, or the
status report formatting in the send-pack/receive-pack protocol.
Change-Id: Ibd88022a298c5fed0edb23dfaf2e90278807ba8b Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 15 Feb 2011 22:09:42 +0000 (14:09 -0800)]
smart-http: Fix recognition of gzip encoding
Some clients coming through proxies may advertise a different
Accept-Encoding, for example "Accept-Encoding: gzip(proxy)".
Matching by substring causes us to identify this as a false positive;
that the client understands gzip encoding and will inflate the
response before reading it.
In this particular case however it doesn't. Its the reverse proxy
server in front of JGit letting us know the proxy<->JGit link can
be gzip compressed, while the client<->proxy part of the link is not:
client <-- no gzip --> proxy <-- gzip --> JGit
Use a more standard method of parsing by splitting the value into
tokens, and only using gzip if one of the tokens is exactly the
string "gzip". Add a unit test to make sure this isn't broken in
the future.
Change-Id: I30cda8a6d11ad235b56457adf54a2d27095d964e Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 15 Feb 2011 22:13:59 +0000 (14:13 -0800)]
http.test: Delete badly named JUnit configurations
We also have org.eclipse.jgit.http--All-Tests, which matches the
style of the org.eclipse.jgit.core--All-Tests name. Drop the others
as these are just redundant duplicates.
Change-Id: I8600a343f6a85d21dc07bda68a8cb834c82946b5 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 15 Feb 2011 17:40:16 +0000 (09:40 -0800)]
PackWriter: Try for accurate delta reuse on cached pack
If a cached pack is used, it might know how many deltas are contained
within it. Record that count as part of our reusedDeltas field
for the stats line we show clients.
Change-Id: I1c61fb817305a95eeac654cccf132cba20b2339c Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Mon, 14 Feb 2011 17:02:57 +0000 (09:02 -0800)]
UploadPack: Expose advertised refs to callers
Like ReceivePack, callers that embed UploadPack within their
service may wish to see the set of references that were sent
to the client. We already have the map on hand, it just needs
to be exposed with a getter.
Change-Id: I123b23e475860d5bb968906bef59068985088b7b Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 13 Feb 2011 02:44:39 +0000 (18:44 -0800)]
RepositoryBuilder: Allow callers to require repository exists
The setMustExist() method allows callers to require the repository
exists in order for build() to succeed. This is useful within a
RepositoryResolver where existence is required.
Change-Id: I6a1154551435cf0da6c2b4a7f4dce266abea5dff Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Shawn O. Pearce [Mon, 7 Feb 2011 01:42:28 +0000 (17:42 -0800)]
pgm: Make --git-dir a string
DHT based repository types don't use a java.io.File to name the
repository. Moving the type to a string starts to open up more types
of repository names, making the standard pgm package easier to reuse
on other storage systems.
Change-Id: I262ccc8c01cd6db88f832ef317b0e1e5db2d016a Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Shawn O. Pearce [Mon, 7 Feb 2011 00:38:02 +0000 (16:38 -0800)]
daemon: Use HTTP's resolver and factory pattern
Using a resolver and factory pattern for the anonymous git:// Daemon
class makes transport.Daemon more useful on non-file storage systems,
or in embedded applications where the caller wants more precise
control over the work tasks constructed within the daemon.
Rather than defining new interfaces, move the existing HTTP ones
into transport.resolver and make them generic on the connection
handle type. For HTTP, continue to use HttpServletRequest, and
for transport.Daemon use DaemonClient.
To remain compatible with transport.Daemon, FileResolver needs to
learn how to use multiple base directories, and how to export any
Repository instance at a fixed name.
Change-Id: I1efa6b2bd7c6567e983fbbf346947238ea2e847e Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 23:58:24 +0000 (15:58 -0800)]
UploadPack: Expose PackWriter activity to a logger
The UploadPackLogger interface allows applications that embed
GitServlet or otherwise use UploadPack to service clients to
track and log how PackWriter was used, and what it sent. This
provides more granularity into the request activity than might
be available from the HTTP server logs, helping administrators
to better understand utilization and Git server performance.
Change-Id: I1d36b060eb3385339d5f986e68192789ef70fc4e Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 8 Feb 2011 00:30:49 +0000 (16:30 -0800)]
RevWalk: Avoid unnecessary re-parsing of commit bodies
If the RevFilter doesn't actually require the commit body,
we shouldn't reparse it if the body was disposed. This happens
often inside of UploadPack during common ancestor negotation, the
RevWalk is reset and re-run over roughly the same commit space,
but the bodies are discarded because the commit message is not
relevant to the process.
Change-Id: I87b6b6a5fb269669867047698abf718d366bd002 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 09:15:33 +0000 (01:15 -0800)]
RevWalk: Don't reset ObjectReader when stopping
Applications like UploadPack reset() and reuse the same RevWalk
multiple times in very rapid succession. Releasing the ObjectReader's
internal state on each use, only to allocate it again on the next
cycle kills performance if the ObjectReader has internal caches, or
even if the Inflater gets returned and pulled from the InflaterCache
too frequently.
Making releasing the ObjectReader the application's responsibility
when it is done with the RevWalk, which most already do by wrapping
their loop in a try/finally block.
Change-Id: I3ad188a719e8d7f6bf27d1a7ca16d465534713f4 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 09:00:44 +0000 (01:00 -0800)]
UploadPack: Donate parsed commits to PackWriter
When UploadPack has computed the merge base between the client's have
set and the want set, its already loaded and parsed all of the
interesting commits that PackWriter needs to transmit to the client.
Switching the RevWalk and its object pool over to be an ObjectWalk
saves PackWriter from needing to re-parse these same commits from the
ObjectDatabase, reducing the startup latency for the enumeration
phase of packing.
UploadPack doesn't want to use an ObjectWalk for the okToGiveUp()
tests because its slower, during each commit popped it needs to cache
the tree into the pendingObjects list, and during each reset() it
discards a bunch of ObjectWalk specific state and reallocates some
internal collections. ObjectWalk was never meant to be rapidly
reset() like UploadPack does, so its perhaps somewhat cleaner to allow
"upgrading" a RevWalk to an ObjectWalk.
Bug: 301639
Change-Id: I97ef52a0b79d78229c272880aedb7f74d0f7532f Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 8 Feb 2011 00:46:06 +0000 (16:46 -0800)]
UploadPack: Rely on peeled ref data for include-tag
The peeled reference information for tags is more efficient to
work with than parsing the tag objects, as usually its coming from
the packed-refs file, which stores the peeled information for us.
Rely on the peeled information to decide if the tag should be
included or not, instead of using our RevWalk to parse the object.
Change-Id: I6714a8560a1c04b5578e9c5b469ea3c77188dff3 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 08:42:23 +0000 (00:42 -0800)]
UploadPack: Assume okToGiveUp is initially false
When negotiate() starts there is at least one want, but no haves, and
thus no common base exists. Its not ok to give up yet, the client
should try to find a common base with the server. Avoid scanning our
history along the want chains until we have found at least one commit
in common with the client, this will trigger okToGiveUp to be set to
null, enabling okToGiveUp() to perform the scan.
Bug: 301639
Change-Id: I98a82a5424fd4c9995924375c7910f76ca4f03af Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 03:00:15 +0000 (19:00 -0800)]
UploadPack: Avoid walking the entire project history
If the client presents a common commit on a side branch, and there is
a want for a disconnected branch UploadPack was walking back on the
entire history of the disconnected branch because it never would find
the common commit.
Limit our search back along any given want to be no earlier than the
oldest common commit received via a "have" line from our client. This
prevents us from looking at all of the project history.
Bug: 301639
Change-Id: Iffaaa2250907150d6efa1cf2f2fcf59851d5267d Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Mathias Kinzler [Tue, 8 Feb 2011 07:56:19 +0000 (08:56 +0100)]
PullCommand: use default remote instead of throwing Exception
When pulling into a local branch that has no upstream configuration,
pull should try to used the default remote ("origin") instead of
throwing an Exception.
Shawn O. Pearce [Sun, 6 Feb 2011 22:04:39 +0000 (14:04 -0800)]
Remove quoting of command over SSH
If the command contains spaces, it needs to be evaluated by the remote
shell. Quoting the command breaks this, making it impossible to run a
remote command that needs additional options.
Bug: 336301
Change-Id: Ib5d88f0b2151df2d1d2b4e08d51ee979f6da67b5 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 02:47:42 +0000 (18:47 -0800)]
UploadPack: Tag non-commits SATISIFIED earlier
This gets non-commits out of the wantSatisfied() main loop by making
use of the cached SATISIFIED flag and its existing bypass. Anything
that isn't a commit cannot be discovered by the have negotiation, so
its always assumed to be SATISIFIED by the server.
Bug: 301639
Change-Id: I1ef354fbf2e2ed44c9020a4069d7179f2159f19f Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When the walker resets, its going to scrub the COMMON and SATISIFIED
flags off a commit if the commit is contained within another commit
the client wants. This is common if the client asks for both a
'maint' and 'master' branch, and 'maint' is also fully merged into
'master'.
COMMON shouldn't be scrubbed during reset because its used to control
membership of the commonBase collection, which is a List. commonBase
should technically be a set, but membership is cheaper with a RevFlag.
COMMON appears on a commit reachable from a WANT when there is also a
PEER_HAS flag present, as this is a merge base. Scrubbing this off
when another branch is tested isn't useful.
SATISIFIED is a cache to tell us if wantSatisified() has already
completed for this particular WANT. If it has, there isn't a need to
recompute on that branch. Scrubbing it off 'maint' when we test
'master' just means we would later need to re-test 'maint', wasting
CPU time on the server.
Bug: 301639
Change-Id: I3bb67d68212e4f579e8c5dfb138f007b406d775f Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 6 Feb 2011 01:49:01 +0000 (17:49 -0800)]
UploadPack: Fix want-is-satisfied test
okToGiveUpImp() has been missing a ! for a long time. This loop over
wantAll() is looking for an object where wantSatisfied() returns
false, because there is no common merge base present. Unfortunately
it was missing a !, causing the loop to break and return false after
at least one want was satisified.
Bug: 301639
Change-Id: Ifdbe0b22c9cd0a9181546d090b4990d792d70c82 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 4 Feb 2011 13:51:27 +0000 (05:51 -0800)]
Fix JGit --upload-pack, --receive-pack options
JGit did not use sh -c to run the receive-pack or upload-pack programs
locally, which caused errors if these strings contained spaces and
needed the local shell to evaluate them.
Win32 support using cmd.exe /c is completely untested, but seems like
it should work based on the limited information I could get through
Google search results.
Bug: 336301
Change-Id: I22e5e3492fdebbae092d1ce6b47ad411e57cc1ba Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 3 Feb 2011 20:37:39 +0000 (12:37 -0800)]
UploadPack: Avoid parsing want list on clone
If a client wants to perform a clone of the repository, it sends
wants, but no haves. There is no point in parsing the want list
within UploadPack, as there won't be a common merge base search.
Instead just defer the parsing to PackWriter, which will do its
own parsing and object enumeration.
If the client does have a "have" set, defer parsing of the want list
until the have list is also parsed, and parse them together in a
single batch queue. This lets the underlying storage system use a
larger lookup batch if there is significant latency involved when
resolving an ObjectId to a RevObject.
Change-Id: I9c30d34f8e344da05c8a2c041a6dc181d8e8bc19 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Mon, 31 Jan 2011 16:35:17 +0000 (08:35 -0800)]
PackWriter: Support reuse of entire packs
The most expensive part of packing a repository for transport to
another system is enumerating all of the objects in the repository.
Once this gets to the size of the linux-2.6 repository (1.8 million
objects), enumeration can take several CPU minutes and costs a lot
of temporary working set memory.
Teach PackWriter to efficiently reuse an existing "cached pack"
by answering a clone request with a thin pack followed by a larger
cached pack appended to the end. This requires the repository
owner to first construct the cached pack by hand, and record the
tip commits inside of $GIT_DIR/objects/info/cached-packs:
cd $GIT_DIR
root=$(git rev-parse master)
tmp=objects/.tmp-$$
names=$(echo $root | git pack-objects --keep-true-parents --revs $tmp)
for n in $names; do
chmod a-w $tmp-$n.pack $tmp-$n.idx
touch objects/pack/pack-$n.keep
mv $tmp-$n.pack objects/pack/pack-$n.pack
mv $tmp-$n.idx objects/pack/pack-$n.idx
done
(echo "+ $root";
for n in $names; do echo "P $n"; done;
echo) >>objects/info/cached-packs
git repack -a -d
When a clone request needs to include $root, the corresponding
cached pack will be copied as-is, rather than enumerating all of
the objects that are reachable from $root.
For a linux-2.6 kernel repository that should be about 376 MiB,
the above process creates two packs of 368 MiB and 38 MiB[1].
This is a local disk usage increase of ~26 MiB, due to reduced
delta compression between the large cached pack and the smaller
recent activity pack. The overhead is similar to 1 full copy of
the compressed project sources.
With this cached pack in hand, JGit daemon completes a clone request
in 1m17s less time, but a slightly larger data transfer (+2.39 MiB):
Repository owners can periodically refresh their cached packs by
repacking their repository, folding all newer objects into a larger
cached pack. Since repacking is already considered to be a normal
Git maintenance activity, this isn't a very big burden.
[1] In this test $root was set back about two weeks.
Change-Id: Ib87131d5c4b5e8c5cacb0f4fe16ff4ece554734b Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Mon, 31 Jan 2011 16:58:23 +0000 (08:58 -0800)]
PackWriter: Display totals after sending objects
CGit pack-objects displays a totals line after the pack data
was fully written. This can be useful to understand some of
the decisions made by the packer, and has been a great tool
for helping to debug some of that code.
Track some of the basic values, and send it to the client when
packing is done:
Shawn O. Pearce [Wed, 2 Feb 2011 21:48:26 +0000 (13:48 -0800)]
RefAdvertiser: Avoid object parsing
It isn't strictly necessary to validate every reference's target
object is reachable in the repository before advertising it to a
client. This is an expensive operation when there are thousands of
references, and its very unlikely that a reference uses a missing
object, because garbage collection proceeds from the references and
walks down through the graph. So trying to hide a dangling reference
from clients is relatively pointless.
Even if we are trying to avoid giving a client a corrupt repository,
this simple check isn't sufficient. It is possible for a reference to
point to a valid commit, but that commit to have a missing blob in its
root tree. This can be caused by staging a file into the index,
waiting several weeks, then committing that file while also racing
against a prune. The prune may delete the blob, since its
modification time is more than 2 weeks ago, but retain the commit,
since its modification time is right now.
Such graph corruption is already caught during PackWriter as it
enumerates the graph from the client's want list and digs back
to the roots or common base. Leave the reference validation also
for that same phase, where we know we have to parse the object to
support the enumeration.
Change-Id: Iee70ead0d3ed2d2fcc980417d09d7a69b05f5c2f Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Jens Baumgart [Wed, 2 Feb 2011 12:13:28 +0000 (13:13 +0100)]
PushCommand: do not set a null credentials provider
PushCommand now does not set a null credentials provider on
Transport because in this case the default provider is replaced with
null and the default mechanism for providing credentials is not
working.
Robin Stocker [Tue, 1 Feb 2011 21:27:33 +0000 (22:27 +0100)]
Don't print "into HEAD" when merging refs/heads/master
When MergeMessageFormatter was given a symbolic ref HEAD which points to
refs/heads/master (which is the case when merging a branch in EGit), it
would result in a merge message like the following:
Merge branch 'a' into HEAD
But it should print the following (as C Git does):
Merge branch 'a'
The solution is to use the leaf ref when checking for refs/heads/master.
Shawn O. Pearce [Mon, 31 Jan 2011 07:40:08 +0000 (23:40 -0800)]
PackWriter: Make thin packs more efficient
There is no point in pushing all of the files within the edge
commits into the delta search when making a thin pack. This floods
the delta search window with objects that are unlikely to be useful
bases for the objects that will be written out, resulting in lower
data compression and higher transfer sizes.
Instead observe the path of a tree or blob that is being pushed
into the outgoing set, and use that path to locate up to WINDOW
ancestor versions from the edge commits. Push only those objects
into the edgeObjects set, reducing the number of objects seen by the
search window. This allows PackWriter to only look at ancestors
for the modified files, rather than all files in the project.
Limiting the search to WINDOW size makes sense, because more than
WINDOW edge objects will just skip through the window search as
none of them need to be delta compressed.
To further improve compression, sort edge objects into the front
of the window list, rather than randomly throughout. This puts
non-edges later in the window and gives them a better chance at
finding their base, since they search backwards through the window.
These changes make a significant difference in the thin-pack:
The resulting pack is 13.63 MiB smaller, even though it contains the
same exact objects. 82,543 fewer objects had to have their sizes
looked up, which saved about 8s of server CPU time. 2,796 more
objects from the client were used as part of the base object set,
which contributed to the smaller transfer size.
Change-Id: Id01271950432c6960897495b09deab70e33993a9 Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Sigend-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Shawn O. Pearce [Sun, 30 Jan 2011 23:10:51 +0000 (15:10 -0800)]
PackWriter: Cleanup findObjectToPack method
Some of this code predates making ObjectId.equals() final
and fixing RevObject.equals() to match ObjectId.equals().
It was therefore more complex than it needs to be, because
it tried to work around RevObject's broken equals() rules
by converting to ObjectId in a different collection.
Also combine setUpWalker() and findObjectsToPack() methods,
these can be one method and the code is actually cleaner.
Change-Id: I0f4cf9997cd66d8b6e7f80873979ef1439e507fe Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
projects / jgit.git / log