Shawn O. Pearce [Fri, 28 May 2010 22:06:29 +0000 (15:06 -0700)]
eclipse-iplog: Use contribution rather than bug element
Wayne changed the schema to no longer be dependent upon the Bugzilla
notion of a contribution, but instead be more generic and better
support systems like Gerrit Code Review. Update our output to
use the <contribution> element and include a link to the change
in Gerrit.
Change-Id: Ibc8a436918bd8e7597dc17743824201a74bce09b Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 28 May 2010 21:30:27 +0000 (14:30 -0700)]
eclipse-ipzilla: Correctly parse result with empty last field
If the last field of our IPzilla query comes back empty, we were
skipping over and not including it in the result List, causing an
IndexOutOfBoundsException when it was read into our data model.
If the last field is empty, actually add the empty string.
Change-Id: Ib18b335990c73e036b185199d0004f4ffc395867 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 27 May 2010 02:00:06 +0000 (19:00 -0700)]
Don't use interruptable pread() to access pack files
The J2SE NIO APIs require that FileChannel close the underlying file
descriptor if a thread is interrupted while it is inside of a read or
write operation on that channel. This is insane, because it means we
cannot share the file descriptor between threads. If a thread is in
the middle of the FileChannel variant of IO.readFully() and it
receives an interrupt, the pack will be automatically closed on us.
This causes the other threads trying to use that same FileChannel to
receive IOExceptions, which leads to the pack getting marked as
invalid. Once the pack is marked invalid, JGit loses access to its
entire contents and starts to report MissingObjectExceptions.
Because PackWriter must ensure that the chosen pack file stays
available until the current object's data is fully copied to the
output, JGit cannot simply reopen the pack when its automatically
closed due to an interrupt being sent at the wrong time. The pack may
have been deleted by a concurrent `git gc` process, and that open file
descriptor might be the last reference to the inode on disk. Once its
closed, the PackWriter loses access to that object representation, and
it cannot complete sending the object the client.
Fortunately, RandomAccessFile's readFully method does not have this
problem. Interrupts during readFully() are ignored. However, it
requires us to first seek to the offset we need to read, then issue
the read call. This requires locking around the file descriptor to
prevent concurrent threads from moving the pointer before the read.
This reduces the concurrency level, as now only one window can be
paged in at a time from each pack. However, the WindowCache should
already be holding most of the pages required to handle the working
set for a process, and its own internal locking was already limiting
us on the number of concurrent loads possible. Provided that most
concurrent accesses are getting hits in the WindowCache, or are for
different repositories on the same server, we shouldn't see a major
performance hit due to the more serialized loading.
I would have preferred to use a pool of RandomAccessFiles for each
pack, with threads borrowing an instance dedicated to that thread
whenever they needed to page in a window. This would permit much
higher levels of concurrency by using multiple file descriptors (and
file pointers) for each pack. However the code became too complex to
develop in any reasonable period of time, so I've chosen to retrofit
the existing code with more serialization instead.
Bug: 308945
Change-Id: I2e6e11c6e5a105e5aef68871b66200fd725134c9 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Stefan Lay [Thu, 20 May 2010 13:09:39 +0000 (15:09 +0200)]
Add a merge command to the jgit API
Merges the current head with one other commit.
In this first iteration the merge command supports
only fast forward and already up-to-date.
Change-Id: I0db480f061e01b343570cf7da02cac13a0cbdf8f Signed-off-by: Stefan Lay <stefan.lay@sap.com> Signed-off-by: Christian Halstrick <christian.halstrick@sap.com> Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
The CommitCommand should take care to create a merge commit if the file
$GIT_DIR/MERGE_HEAD exists. It should then read the parents for the merge
commit out of this file. It should also take care that when commiting
a merge and no commit message was specified to read the message from
$GIT_DIR/MERGE_MSG.
Finally the CommitCommand should remove these files if the commit
succeeded.
Change-Id: I4e292115085099d5b86546d2021680cb1454266c Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Sasa Zivkov [Wed, 19 May 2010 14:59:28 +0000 (16:59 +0200)]
Externalize strings from JGit
The strings are externalized into the root resource bundles.
The resource bundles are stored under the new "resources" source
folder to get proper maven build.
Strings from tests are, in general, not externalized. Only in
cases where it was necessary to make the test pass the strings
were externalized. This was typically necessary in cases where
e.getMessage() was used in assert and the exception message was
slightly changed due to reuse of the externalized strings.
Change-Id: Ic0f29c80b9a54fcec8320d8539a3e112852a1f7b Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Shawn O. Pearce [Sun, 16 May 2010 00:01:49 +0000 (17:01 -0700)]
Fix SSH deadlock during OutOfMemoryError
In close() method of SshFetchConnection and SshPushConnection
errorThread.join() can wait forever if JSch will not close the
channel's error stream. Join with a timeout, and interrupt the
copy thread if its blocked on data that will never arrive.
Dmitry Neverov [Wed, 19 May 2010 18:39:17 +0000 (11:39 -0700)]
Fix race condition in StreamCopyThread
If we get an interrupt during an IO operation (src.read or dst.write)
caused by the flush() method incrementing the flush counter, ensure
we restart the proper section of code. Just ignore the interrupt
and continue running.
Bug: 313082
Change-Id: Ib2b37901af8141289bbac9807cacf42b4e2461bd Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 16 May 2010 02:10:47 +0000 (19:10 -0700)]
Remove unnecessary truncation of in-pack size during copy
The number of bytes to copy was truncated to an int, but the
pack's copyToStream() method expected to be passed a long here.
Pass through the long so we don't truncate a giant object.
Change-Id: I0786ad60a3a33f84d8746efe51f68d64e127c332 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 16 May 2010 00:51:03 +0000 (17:51 -0700)]
Reduce the size of PackWriter's ObjectToPack instances
Rather than holding onto the PackedObjectLoader, only hold the
PackFile and the object offset. During a reuse copy that is all
we should need to complete a reuse, and the other parts of the
PackedObjectLoader just waste memory.
This change reduces the per-object memory usage of a PackWriter by
32 bytes on a 32 bit JVM using only OFS_DELTA formatted objects.
The savings is even larger (by another 20 bytes) for REF_DELTAs.
This is close to a 50% reduction in the size of ObjectToPack,
making it rather worthwhile to do.
Beyond the memory reduction, this change will help to make future
refactoring work easier. We need to redo the API used to support
copying data, and disconnecting it from the PackedObjectLoader is
a good first step.
Change-Id: I24ba4e621e101f14e79a16463aec5379f447aa9b Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 16 May 2010 00:37:14 +0000 (17:37 -0700)]
Reduce size of PackedObjectLoader by dropping long to int
Rather than keep track of both the position of the object, and the
position of its data, just keep track of the number of bytes used
by the object's header in the pack. This shaves 4 bytes out of the
size of the PackedObjectLoader instances.
We also can defer the addition instruction to the materialize()
operation, avoiding it entirely if the caller never actually uses
the loader. This may be relevant for PackWriter invocations,
where only 1 loader gets chosen for a given object, even though
the object may appear on disk in more than one pack file.
Error reporting is now simplified, as we can rely on the object
offset rather than its data offset. This is the value displayed
by pack debugging tools like `git verify-pack -v`, so its better
to use that in our own errors.
Because nobody needs getDataOffset() now, we can drop that from
the public API.
Change-Id: Ic639c0d5a722315f4f5c8ffda6e26643d90e5f42 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sat, 15 May 2010 23:18:44 +0000 (16:18 -0700)]
Factor out duplicate Inflater setup in WindowCursor
Since we use this code twice, pull it into a private method. Let
the compiler/JIT worry about whether or not this logic should be
inlined into the call sites.
Change-Id: Ia44fb01e0328485bcdfd7af96835d62b227a0fb1 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 14 May 2010 22:02:31 +0000 (15:02 -0700)]
Squash OffsetCache into WindowCache
Originally when I wrote this code I had hoped to use OffsetCache
to also implement the UnpackedObjectCache. But it turns out they
need rather different code, and it just wasn't worth trying to
reuse the OffsetCache base class.
Before doing any major refactoring or code cleanups here, squash the
two classes together and delete OffsetCache. As WindowCache is our
only subclass, this is pretty simple to do. We also get a minor
code reduction due to less duplication between the two classes,
and the JIT should be able to do a better job of optimization here
as we can define types up front rather than relying on generics
that erase back to java.lang.Object.
Change-Id: Icac8bda01260e405899efabfdd274928e98f3521 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 14 May 2010 21:03:32 +0000 (14:03 -0700)]
Avoid unnecessary second read on OBJ_OFS_DELTA headers
When we read the object header we copy 20 bytes from the pack data,
then start parsing out the type and the inflated size. For most
objects, this is only going to require 3 bytes, which is sufficient
to represent objects with inflated sizes of up to 2^16. The local
buffer however still has 17 bytes remaining in it, and that can be
used to satisfy the OBJ_OFS_DELTA header.
We shouldn't need to worry about walking off the end of the buffer
here, because delta offsets cannot be larger than 64 bits, and that
requires only 9 bytes in the OFS_DELTA encoding.
Assuming worst-case scenarios of 9 bytes for the OFS_DELTA encoding,
the pack file itself must be approaching 2^64 bytes, an infeasible
size to store on any current technology. However, even if this
were the case we still have 11 bytes for the type/size header.
In that encoding we can represent an object as large as 2^74 bytes,
which is also an infeasible size to process in JGit.
So drop the second read here.
The data offsets we pass into the ObjectLoaders being constructed
need to be computed individually now. This saves a local variable,
but pushes the addition operation into each branch of the switch.
Change-Id: I6cf64697a9878db87bbf31c7636c03392b47a062 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 13 May 2010 17:23:33 +0000 (10:23 -0700)]
Fix hang when fetching over SSH
JSch may hang or abort with the timeout if JGit connects before
its obtained the streams. Instead defer the connect() call until
after the streams have been configured.
Shawn O. Pearce [Thu, 13 May 2010 16:56:15 +0000 (09:56 -0700)]
Fix interrupted write in StreamCopyThread
If a flush() gets delivered at the same time that we are blocking
while writing to an interruptable stream, the copy thread will
abort assuming its a stream error. Instead ignore the interrupt,
and retry the write.
Change-Id: Icbf62d1b8abe0fabbb532dbee088020eecf4c6c2 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Dmitry Neverov [Thu, 13 May 2010 16:50:44 +0000 (09:50 -0700)]
Fix missing flush in StreamCopyThread
It is possible to miss flush() invocation in StreamCopyThread.
In this case some data will not be sent to remote host and we will
wait forever (or until timeout) in src.read().
Use a counter to keep track of the flush requests.
Change-Id: Ia818be9b109a1674d9e2a9c78e125ab248cfb75b Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Jonathan Gossage [Tue, 11 May 2010 23:06:28 +0000 (18:06 -0500)]
Fix Maven Javadoc generation problem
There is a serious problem with the Maven Javadoc plugin. Please see
http://jira.codehaus.org/browse/MJAVADOC-275
for details. This problem is fixed by using maven-javadoc-plugin V2.7
instead of maven-javadoc-plugin v2.6.1.
Matthias Sohn [Tue, 11 May 2010 12:35:26 +0000 (14:35 +0200)]
Expose org.eclipse.jgit.junit via jgit p2 repository
EGit Tycho builds on build.eclipse.org frequently hit corrupted artifacts
which leads to broken builds. Cleaning up these corrupted files is tedious
since it requires file system access on the build server. Hence we want to
switch to use job-local m2 repositories. This requires that build artifacts
are shared between the jgit and egit build jobs via p2. Therefore the
bundle org.eclipse.jgit.junit needs to be exposed via p2 repository.
Change-Id: I0ccd7763eede117cb68240fdd25f13d6e6f6a2c1 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add builder-style API to jgit and Commit & Log cmd
Added a new package org.eclipse.jgit.api and a builder-style API for
jgit. Added also the first implementation for two git commands: Commit
and Log.
This API is intended to be used by external components when
functionalities of the standard git commands are required. It will also
help to ease writing JGit tests.
For internal usages this API may often not be optimal because the git
commands are doing much more than required or they expect parameters of
an unappropriate type.
Change-Id: I71ac4839ab9d2f848307eba9252090c586b4146b Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Robin Rosenberg [Sat, 8 May 2010 21:12:19 +0000 (23:12 +0200)]
A stages field and getter for GitIndex entry introduced
Currently, if the Index contains a file in more than one stage, only
the last entry (containing the highest stage) will be registered in
GitIndex. For applications it can be useful to not only know about the
highest stage, but also which other stages are present, e.g. to detect
the type of conflict the file is in.
Change-Id: I2d4ff9f6023335d9ba6ea25d8e77c8e283ae53cb Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
The repository state tells in which state the repo is and also which actions
are currently allowed. The state MERGING is telling that a commit is not
possible. But this is only true in the case of unmerged paths in the index.
When we are merging but have resolved all conflicts then we are in a special
state: We are still merging (means the next commit should have multiple
parents) but a commit is now allowed.
Since the MERGING state "canCommit()" cannot be enhanced to return true/false
based on the index state (MERGING is an enum value which does not have a
reference to the repository its state it is representing) I had to introduce a new
state MERGING_RESOLVED. This new state will report that a commit is possible.
CAUTION: there might be the chance that users of jgit previously blindly did a
plain commit (with only one parent) when the RepositoryState allowed them to
do so. With this change these users will now be confronted with a RepositoryState
which says a commit is possible but before they can commit they'll have to
check the MERGE_MESSAGE and MERGE_HEAD files and use the info from these
files.
Change-Id: I0a885e2fe8c85049fb23722351ab89cf2c81a431 Signed-off-by: Christian Halstrick <christian.halstrick@sap.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn O. Pearce [Tue, 4 May 2010 23:25:20 +0000 (16:25 -0700)]
Fix FooterLine.matches(FooterKey) on same length keys
If two keys are the same length, but don't share the same sequence
of characters, we were incorrectly claiming they still matched due
to a bug in the for loop condition. I used the wrong variable and
the loop never executed, resulting in equality anytime the two keys
being compared were the same length.
Use the proper local variable to loop through the arrays, and add
a JUnit test to verify equality works as expected.
Change-Id: I4a02400e65a9b2e0da925b05a2cc4b579e1dd33a Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a concurrent thread picks up a newly created PackFile and adds
it to the pack list before the IndexPack thread itself can insert
the item onto the front of the list, do nothing and use the item
that was picked up by that other concurrent scanning thread.
This avoids a potential condition where the same pack exists in
memory twice, which causes confusion later during a rescan of the
directory because we don't know exactly which PackFile instance
should be retained into the new list, and which should be discarded.
We can stop searching through the old pack list as soon as the
sort function declares that the item to insert should be before
the item already in the list. Because the list is always sorted
by modification time (in seconds), we should never encounter a
case where the pack is positioned at the wrong spot in the list.
This early break out still permits an efficient implementation of
the common case, inserting a new pack at the head of the list.
Change-Id: Ice4459bbd4ee9487078aff5257893883d04f05fb Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Favor earlier PackFile instances over later duplicates
There is a potential race condition during insertPack that can lead
to us having the same pack file open twice in the same directory.
A different thread can miss an object on disk, and trigger a scan
of the directory, and notice the pack that was put in by IndexPack.
So the pack winds up in the newly created PackList.
The IndexPack thread then wakes up and finishes its insertPack by
creating a new PackFile and inserting it into position 0 of the list.
We now have the same pack listed twice.
Readers will favor the earlier PackFile instance, because its the
first one they come across as they iterate through the list.
Keep that earlier one when we scan the pack directory again, as
this will avoid needing to purge out all of the windows that may
have been cached.
Of course we should also fix that race condition, but this block
was taking the wrong resolution if this error ever shows up, so
lets first fix the block to use a more sane resolution.
Change-Id: I0d339b9fd1dd8012e8fe5a564b893c0f69109e28 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Cleanup duplicated object reuse code in PackWriter
This reuse line was identical between the two branches related to
reusing a delta, or reusing a whole object. Either way they reuse
the body of the object as-is. So just make that a common function
after the header is written.
Change-Id: I0e6673b8e813c8c08c594ea2ba546fd366339d5d Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Fix NPE during InflaterCache return after corrupt loose object
If a corrupt loose object is read, UnpackedObjectLoader was disposing
of the Inflater, and then attempting to return the disposed Inflater
to the InflaterCache. Since the disposed Inflater had its native
libz resource deallocated and its reference cleared out, the Inflater
threw NullPointerException and refused to reset itself before being
put back into the cache.
Instead of disposing of the Inflater when corruption is found, do
nothing, and allow it to be returned to the cache. The instance
will get reset, and should be usable by a future caller.
Bug: 310291
Change-Id: I44f2247c08b6e04fa62f8399609341b07508c096 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
* receive-pack-filter:
ReceivePack: Clarify the check reachable option
ReceivePack: Micro-optimize object lookup when checking connectivity
ReceivePack: Correct type of not provided object
IndexPack: Tighten up new and base object bookkeeping
ReceivePack: Remove need new,base object id properties
ReceivePack: Discard IndexPack as soon as possible
ReceivePack: fix ensureProvidedObjectsVisible on thin packs
This option was mis-named from day 1. Its not checking that the
objects provided by the client are reachable, its actually doing
a scan to prove that objects referenced by the client are already
reachable through another reference on the server, or were sent
as part of the pack from the client.
Rename it checkReferencedObjectsAreReachable, since we really are
trying to validate that objects referenced by the client's actions
are reachable to the client.
We also need to ensure we run checkConnectivity() anytime this is
enabled, even if the caller didn't turn on fsck for object formats.
Otherwise the check would be completely bypassed.
Change-Id: Ic352ddb0ca8464d407c6da5c83573093e018af19 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
ReceivePack: Micro-optimize object lookup when checking connectivity
If we are checking the visibility of everything referenced in the
pack that isn't already reachable by a reference, it needs to be
in the provided set. Since the provided set lists everything that
is in this pack, we can avoid checking to see if the blob exists
on disk, because we know it should be there, it was found in the
pack we just consumed.
Change-Id: Ie3c7746f734d13077242100a68e048f1ac18c34a Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
IndexPack: Tighten up new and base object bookkeeping
The only current consumer of these collections is ReceivePack,
where it needs to test ObjectId equality between a RevObject and an
ObjectId. There we were copying from a traditional HashSet<ObjectId>
into an ObjectIdSubclassMap<ObjectId>, as the latter can perform
hashing using ObjectId's native value support, bypassing RevObject's
override on hashCode() and equals(). Instead of doing that copy,
directly create ObjectIdSubclassMap instances inside of ReceivePack.
We also only need to record the objects that do not appear in the
incoming pack, and were therefore copied from the local repositiory
in order to complete delta resolution. Instead of listing everything
that used an OBJ_REF_DELTA format, list only the objects that we
pulled from the destination repository via a normal ObjectLoader.
ReceivePack can now discard the IndexPack object, and all of its
other data, as soon as these collections are held by the check
connectivity method. This frees up memory for the ObjectWalk's
own RevObject pool.
Change-Id: I22ef71b45c2045a0202e7fd550a770ee1f6f38a6 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
ReceivePack: Remove need new,base object id properties
These are more like internal implementation details of how IndexPack
works with ReceivePack to validate the incoming object stream.
Callers who are embedding the ReceivePack logic in their own
application don't really need to know the details of which objects
were used for delta bases in the incoming thin pack, or exactly
which objects were newly transmitted.
Hide these from the API, as exposing them through ReceivePack was
an early mistake.
Change-Id: I7ee44a314fa19e6a8520472ce05de92c324ad43e Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
ReceivePack: Discard IndexPack as soon as possible
The IndexPack object carries a good bit of state within itself about
the objects received over the wire. The earlier we can discard it,
the sooner the GC is able to reclaim this chunk of memory for other
uses. So drop it as soon as we are certain the pack is valid and we
have no connectivity concerns.
Change-Id: I1e8bc87c2e9183733043622237a064e55957891f Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
ReceivePack: fix ensureProvidedObjectsVisible on thin packs
If ensureProvidedObjectsVisible is enabled we expected any trees or
blobs directly reachable from an advertised reference to be marked
with UNINTERESTING. Unfortunately ObjectWalk doesn't bother setting
this until the traversal is complete. Even then it won't necessarily
set it on every tree if the corresponding commit wasn't popped.
When we are going to check the base objects for the received pack,
ensure the UNINTERESTING flag gets carried into every immediately
reachable tree or blob, because these are the ones that the client
might try to use as delta bases in a thin pack.
Change-Id: I5d5fdcf07e25ac9fc360e79a25dff491925e4101 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
ObjectIdSubclassMap: Correct Iterator to throw NoSuchElementException
The Iterator contract says next() shall throw NoSuchElementException
if there are no more items remaining in the iteration. We got this
wrong when I originally wrote the implementation, so fix it.
Change-Id: Iea25e6569ead5c8b3128b8a368c5b2caebec7ecc Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This class behaves like a cross between a Set and a Map, sometimes
we might expect to use the method isEmpty() to test for size() == 0.
So implement it, reducing the surprise folks get when they are given
one of these objects.
Change-Id: I0d68e1243da8e62edf79c6ba4fd925f643e80a88 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
IndexPack: Correct thin pack fix using less than 20 bytes
If we need to append less than 20 bytes in order to fix a thin pack
and make it complete, we need to set the length of our file back to
the actual number of bytes used because the original SHA-1 footer was
not completely overwritten. That extra data will confuse the header
and footer fixup logic when it tries to read to the end of the file.
This isn't a very common case to occur, which is why we've never
seen it before. Getting a delta that requires a whole object which
uses less than 20 bytes in pack representation is really hard.
Generally a delta generator won't make these, because the delta
would be bigger than simply deflating the whole object. I only
managed to do this with a hand-crafted pack file where a 1 byte
delta was pointed to a 1 byte whole object.
Normally we try really hard to avoid truncating, because its
typically not safe across network filesystems. But the odds of
this occurring are very low. This truncation is done on a file
we have open for writing, will append more content onto, and is
a temporary file that we won't move into position for others to
see until we've validated its SHA-1 is sane. I don't think the
truncate on NFS issue is something we need to worry about here.
Change-Id: I102b9637dfd048dc833c050890d142f43c1e75ae Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Fix unit tests using MockSystemReader with user configuation
Since cc905e7d4be "Make Repository.getConfig aware of changed config"
its invalid to have a null result from FileBasedConfig.getFile(), as
the path is used to stat the location on disk before returning the
Config object from Repository.getConfig().
Mock out the isOutdated() method to return false all of the time
in the mock test environment, so we don't crash with an NPE when
this mock user configuration is being called.
Change-Id: I0b4d9cbd346d5dc225ec12674da905c35457fa7c Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Robin Rosenberg [Tue, 13 Apr 2010 21:00:53 +0000 (23:00 +0200)]
Optimize ref scanning
We can avoid one stat call by trying to perform a directory
listing without checking if the reference File is a directory.
Attempting a directory listing is defined to return. The other
case for null returns from list is when an I/O error occcurs.
Both cases are now intepreted as a possible plain reference. I/O
errors when reading plain references will be handled (ignored)
in scanRef().
Change-Id: I9906ed8c42eab4d6029c781aab87b3b07c1a1d2c Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
In the current implementation Repository reads user and repository
config only at creation point of time.
The new implementatiopn checks in Repository.getConfig if user or
repository config have changed on disk and reload the config if
required.
Robin Rosenberg [Sat, 10 Apr 2010 15:05:36 +0000 (17:05 +0200)]
Speed up check for modifications of tracked resources
We only need to check file existense if some other stat returns
a value that may mean that the file does not exist. File.length() == 0
or File.lastModified() == 0 are two such properties. We use length
here.
Change-Id: If626b12e7bb4da994b5c086f6a5b7a12c187261c Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Shawn O. Pearce [Sat, 20 Mar 2010 22:33:26 +0000 (15:33 -0700)]
Fix EGit deadlock listing branches of SSH remote
When listing branches, EGit only reads the advertisement and
then disconnects. When it closes down the pack channel the remote
side is waiting for the client to send our list of commands, or a
flush-pkt to let it know there is nothing to do.
However if an error thread is open watching the SSH stderr stream,
we ask for it to finish before we send the flush-pkt. Unfortunately
the thread won't terminate until the main output stream closes,
which is waiting for the flush-pkt. A classic network deadlock.
If the output stream needs a flush-pkt we send it before we wait
for the error stream to close. If the flush-pkt is rejected, we
close down the output stream early, assuming that the remote side
is broken and we will get error information soon.
Change-Id: I8d078a339077756220c113f49d206b1bf295d434 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sun, 21 Mar 2010 02:06:58 +0000 (19:06 -0700)]
Qualify builds as 0.8.0
Since the API is changing relative to 0.7.0, we'll call our next
release 0.8.1. But until that gets released, builds from master
will be 0.8.0.qualifier.
Change-Id: I921e984f51ce498610c09e0db21be72a533fee88 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Nico Sallembien [Sat, 13 Mar 2010 00:07:19 +0000 (16:07 -0800)]
Add a paranoid 'must be provided' option to ReceivePack
By default a receive pack assumes that its user will only provide
references to objects that the user already has access to on their
local client. In certain cases, an additional check to verify the
references point only to reachable objects is necessary.
This additional checking is useful when the code doesn't trust
the client not to provide a forged SHA-1 reference to an object,
in an attempt to access parts of the DAG that they weren't allowed
to see by the configured RefFilter.
Change-Id: I3e4b8505cb2992e3e4be253abb14a1501e47b970 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Sat, 13 Mar 2010 01:04:48 +0000 (17:04 -0800)]
Merge branch 'stable-0.7'
* stable-0.7:
Reuse the line buffer between strings in PacketLineIn
http.server: Use TemporaryBuffer and compress some responses
Reduce multi-level buffered streams in transport code
Fix smart HTTP client buffer alignment
Use "ERR message" for early ReceivePack problems
Catch and report "ERR message" during remote advertisements
Wait for EOF on stderr before finishing SSH channel
Capture non-progress side band #2 messages and put in result
ReceivePack: Enable side-band-64k capability for status reports
Use more restrictive patterns for sideband progress scraping
Prefix remote progress tasks with "remote: "
Decode side-band channel number as unsigned integer
Refactor SideBandInputStream construction
Refactor SideBandOutputStream to be buffered
Shawn O. Pearce [Sat, 13 Mar 2010 01:00:50 +0000 (17:00 -0800)]
Merge branch 'push-sideband' into stable-0.7
* push-sideband:
Reuse the line buffer between strings in PacketLineIn
http.server: Use TemporaryBuffer and compress some responses
Reduce multi-level buffered streams in transport code
Fix smart HTTP client buffer alignment
Use "ERR message" for early ReceivePack problems
Catch and report "ERR message" during remote advertisements
Wait for EOF on stderr before finishing SSH channel
Capture non-progress side band #2 messages and put in result
ReceivePack: Enable side-band-64k capability for status reports
Use more restrictive patterns for sideband progress scraping
Prefix remote progress tasks with "remote: "
Decode side-band channel number as unsigned integer
Refactor SideBandInputStream construction
Refactor SideBandOutputStream to be buffered
Shawn O. Pearce [Fri, 12 Feb 2010 15:00:32 +0000 (07:00 -0800)]
Reuse the line buffer between strings in PacketLineIn
When reading pkt-lines off an InputStream we are quite likely to
consume a whole group of fairly short lines in rapid succession, such
as in the have exchange that occurs in the fetch-pack/upload-pack
protocol. Rather than allocating a throwaway buffer for each
line's raw byte sequence, reuse a buffer that is equal to the small
side-band packet size, which is 1000 bytes. Text based pkt-lines
are required to be less than this size because many widely deployed
versions of C Git use a statically allocated array of this length.
Change-Id: Ia5c8e95b85020f7f80b6d269dda5059b092d274d Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 12 Feb 2010 02:53:22 +0000 (18:53 -0800)]
http.server: Use TemporaryBuffer and compress some responses
The HTTP server side code now uses the same approach that the smart
HTTP client code uses when preparing a request body. The payload
is streamed into a TemporaryBuffer of limited size. If the entire
data fits, its compressed with gzip if the user agent supports that,
and a Content-Length header is used to transmit the fixed length
body to the peer. If however the data overflows the limited memory
segment, its streamed uncompressed to the peer.
One might initially think that larger contents which overflow
the buffer should also be compressed, rather than sent raw, since
they were deemed "large". But usually these larger contents are
actually a pack file which has been already heavily compressed by
Git specific routines. Trying to deflate that with gzip is probably
going to take up more space, not less, so the compression overhead
isn't worthwhile.
This buffer and compress optimization helps repositories with a
large number of references, as their text based advertisements
compress well. For example jgit's own native repository currently
requires 32,628 bytes for its full advertisement of 489 references.
Most repositories have fewer references, and thus could compress
their entire response in one buffer.
Change-Id: I790609c9f763339e0a1db9172aa570e29af96f42 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 12 Feb 2010 02:10:45 +0000 (18:10 -0800)]
Reduce multi-level buffered streams in transport code
Some transports actually provide stream buffering on their own,
without needing to be wrapped up inside of a BufferedInputStream in
order to smooth out system calls to read or write. A great example
of this is the JSch SSH client, or the Apache MINA SSHD server.
Both use custom buffering to packetize the streams into the encrypted
SSH channel, and wrapping them up inside of a BufferedInputStream
or BufferedOutputStream is relatively pointless.
Our SideBandOutputStream implementation also provides some fairly
large buffering, equal to one complete side-band packet on the main
data channel. Wrapping that inside of a BufferedOutputStream just to
smooth out small writes from PackWriter causes extra data copies, and
provides no advantage. We can save some memory and some CPU cycles
by letting PackWriter dump directly into the SideBandOutputStream's
internal buffer array.
Instead we push the buffering streams down to be as close to the
network socket (or operating system pipe) as possible. This allows
us to smooth out the smaller reads/writes from pkt-line messages
during advertisement and negotation, but avoid copying altogether
when the stream switches to larger writes over a side band channel.
Change-Id: I2f6f16caee64783c77d3dd1b2a41b3cc0c64c159 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Fri, 12 Feb 2010 02:02:22 +0000 (18:02 -0800)]
Fix smart HTTP client buffer alignment
This proved to be a pretty difficult to find bug. If we read exactly
the number of response bytes from the UnionInputStream and didn't
try to read beyond that length, the last connection's InputStream is
still inside of the UnionInputStream, and UnionInputStream.isEmpty()
returns false. But there is no data present, so the next read
request to our UnionInputStream returns EOF at a point where the
HTTP client code should have started a new request in order to get
more data.
Instead of wrapping the UnionInputStream, push an dummy stream onto
the end of it which when invoked always starts the next request and
then returns EOF. The UnionInputStream will automatically pop that
dummy stream out, and then read the next request's stream.
This way we never get into the state where we don't think we need
to run another request in order to satisfy the current read request,
but we really do.
The bug was hidden for so long because BasePackConnection.init()
was always wrapping the InputStream into a BufferedInputStream
with an 8 KiB buffer. This made the odds of us reading from the
UnionInputStream the exact number of available bytes quite low, as
the BufferedInputStream would always try to read a full buffer size.
Change-Id: I02b5ec3ef6853688687d91de000a5fbe2354915d Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 11 Feb 2010 18:58:22 +0000 (10:58 -0800)]
Use "ERR message" for early ReceivePack problems
If the application wants to, it can use sendError(String) to send one
or more error messages to clients before the advertisements are sent.
These will cause a C Git client to break out of the advertisement
parsing loop, display "remote error: message\n", and terminate.
Servers can optionally use this to send a detailed error to a client
explaining why it cannot use the ReceivePack service on a repository.
Over smart HTTP these errors are sent in a 200 OK response, and
are in the payload, allowing the Git client to give the end-user
the custom message rather than the generic error "403 Forbidden".
Change-Id: I03f4345183765d21002118617174c77f71427b5a Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 11 Feb 2010 19:43:29 +0000 (11:43 -0800)]
Catch and report "ERR message" during remote advertisements
GitHub broke the native git protocol a while ago by interjecting an
"ERR message" line into the upload-pack or receive-pack advertisement
list. This didn't match the expected pattern, so it caused existing
C Git clients to abort with a protocol exception.
These days, C Git clients actually look for this message and abort
with a more graceful notice to the end-user. JGit should do the
same, including setting up a custom exception type that makes it
easier for higher-level UIs to identify a message from the remote
site and present it to the user.
Change-Id: I51ab62a382cfaf1082210e8bfaa69506fd0d9786 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Thu, 11 Feb 2010 03:54:07 +0000 (19:54 -0800)]
Wait for EOF on stderr before finishing SSH channel
JSch will allow us to close the connection and then just drop
any late messages coming over the stderr stream for the command.
This makes it easy to lose final output on a command, like from
Gerrit Code Review's post receive hook.
Instead spawn a background thread to copy data from JSch's pipe
into our own buffer, and wait for that thread to receive EOF on the
pipe before we declare the connection closed. This way we don't
have a race condition between the stderr data arriving and JSch
just tearing down the channel.
Change-Id: Ica1ba40ed2b4b6efb7d5e4ea240efc0a56fb71f6 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Wed, 10 Feb 2010 19:49:27 +0000 (11:49 -0800)]
Capture non-progress side band #2 messages and put in result
Any messages received on side band #2 that aren't scraped as a
progress message into our ProgressMonitor are now forwarded to a
buffer which is later included into the OperationResult object.
Application callers can use this buffer to present the additional
messages from the remote peer after the push or fetch operation
has concluded.
The smart push connections using the native send-pack/receive-pack
protocol now request side-band-64k capability if it is available
and forward any messages received through that channel onto this
message buffer. This makes hook messages available over smart HTTP,
or even over SSH.
The SSH transport was modified to redirect the remote command's
stderr stream into the message buffer, interleaved with any data
received over side band #2. Due to buffering between these two
different channels in the SSH channel mux itself the order of any
writes between the two cannot be ensured, but it tries to stay close.
The local fork transport was also modified to redirect the local
receive-pack's stderr into the message buffer, rather than going to
the invoking JVM's System.err. This gives applications a chance
to log the local error messages, rather than needing to redirect
their JVM's stderr before startup.
To keep things simple, the application has to wait for the entire
operation to complete before it can see the messages. This may
be a downside if the user is trying to debug a remote hook that is
blocking indefinitely, the user would need to abort the connection
before they can inspect the message buffer in any sort of UI built
on top of JGit.
Change-Id: Ibc215f4569e63071da5b7e5c6674ce924ae39e11 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 9 Feb 2010 03:18:51 +0000 (19:18 -0800)]
ReceivePack: Enable side-band-64k capability for status reports
We now advertise the side-band-64k capability inside of ReceivePack,
allowing hooks to echo status messages down the side band channel
instead of over the optional stderr stream.
This change permits hooks running inside of an http:// based push
invocation to still message the end-user with more detailed errors
than the small per-command string in the status report.
Change-Id: I64f251ef2d13ab3fd0e1a319a4683725455e5244 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 9 Feb 2010 18:46:01 +0000 (10:46 -0800)]
Use more restrictive patterns for sideband progress scraping
To avoid scraping a non-progress message as though it were a progress
item for the progress monitor, use a more restrictive pattern to
watch the remote side's messages. These two regexps should match
any message produced by C Git since 42e18fbf5f94 ("more compact
progress display", Oct 2007), and which first appeared in Git 1.5.4.
Change-Id: I57e34cf59d42c1dbcbd1a83dd6f499ce5e39d15d Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Shawn O. Pearce [Tue, 9 Feb 2010 18:39:15 +0000 (10:39 -0800)]
Prefix remote progress tasks with "remote: "
When we pull task messages off the remote peer via sideband #2
prefix them with the string "remote: " to make it clear to the
user these are coming from the other system, and not from their
local client.
Change-Id: I02c5e67c6be67e30e40d3bc4be314d6640feb519 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
projects / jgit.git / log