It is possible that StreamCopyThread will not flush everything
from it's src to it's dst.  In most cases StreamCopyThread works
like this:

  in loop:
    n = src.read(buf);
    dst.write(buf, 0, n);

and when we want to flush, we interrupt() StreamCopyThread and it
flushes everything it wrote to dst.

The problem is that our interrupt() could interrupt reading. In this
case we will flush everything we wrote to dst, but not everything
we wrote to src.

Change-Id: Ifaf4d8be87535c7364dd59b217dfc631460018ff
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Added a check in Diff to ensure that files that are most likely
not text are not line-by-line diffed. Files are determined to be
binary by checking the first 8000 bytes for a null character. This
is a similar heuristic to what C Git uses.

Change-Id: I2b6f05674c88d89b3f549a5db483f850f7f46c26

Added code to support ignoring leading, trailing, and changed
whitespace when performing a diff operation. I also added command
line options to Diff to enable the various whitespace ignoring
methods. These match the flags for git diff.

Change-Id: Ie56301aafad59ee3f0fe5de62719f5023cd702c8

JGit did not have support for skipping whitespace when comparing
lines in RawText objects. I added a subclass of RawText that skips
whitespace in its equals and hashCode methods. I used a subclass
rather than adding functionality into RawText so that performance
would not be impacted by extra logic.

This class only supports ignoring all whitespace. Others will follow
that allow other forms of whitespace ignoring.

Change-Id: Ic2f79e85215e48d3fd53ec1b4ad13373dd183a4a

Under smart HTTP the biDirectionalPipe flag is false, and we return
back immediately at this point in the negotiation process.  There is
no need to flush the stream to the client, the request is over and
it will be automatically flushed out by the higher level servlet
that invoked us.  Avoiding flush here allows us to only use flush
after a progress message is sent during pack generation.

Change-Id: Id0c8b7e95e3be6ca4c1b479e096bed6b0283b828
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This simplifies the PackIndex code, which is trying to quickly copy
an existing ObjectId into a MutableObjectId.  Rather than having
the PackIndex violate the ObjectId's internals, expose a copy from
function similar to the other ones for copying from raw byte arrays
or hex formatted strings.

Change-Id: I142635cbece54af2ab83c58477961ce925dc8255
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Storage systems can use these implementations to compare a passed
AnyObjectId with a stored representation of an ObjectId in the
canonical network byte order format.  This can be useful to do a
binary search, or just linear scan, over an encoded storage file.

Change-Id: I8c72993c4f4c6e98d599ac2c9867453752f25fd2
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

An implementation might prefer to use the RefList type here, and
RefList is part of our public API.  Expose the constructor so callers
who have a RefList can take advantage of the existing sorting.

Change-Id: I545867f85aa2c479d2d610024ebbe318144709c8
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

When we finally move RefDirectory to the new storage.file package,
its associated RefDirectoryUpdate will need visiblity to this
constructor in order to initialize itself.  This is true of any
other repository implementation, so make it protected rather than
package level visible.

Change-Id: If838aec9baeb80ee2f12dcbca717657c725a9242
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Repository implementations outside of .lib need to be able to
create these events and deliver them to listening application code.

Expose and document the constructors so that they are visible when
we move FileRepository into storage.file.FileRepository.

Change-Id: I7fb6e8f4f5fdab683c5ebb5267673aa6d5b560bb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

A Git reference name must never end with ".lock", as it would
confuse any existing C client that tries to obtain a clone of the
repository over the network.  Even if the repository isn't on a
local filesystem, it still should ban that suffix.

Because I plan to move LockFile to storage.file and make it a private
implementation detail of the local file system storage model,
we can't rely on its package level SUFFIX field here.  Making it
public probably won't work long-term either, as I also plan to
pull storage.file into its own separate project that depends on
the core library.

So, just inline the constant here.  Its as foribidden as ":" is.

Change-Id: If85076861baeacc183b82696375a13e935ba8836
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Modifications to various classes in order to allow serialization
for use of JGit in Hudson's git plugin.

Change-Id: If088717d3da7483538c00a927e433a74085ae9e6

If a repository is "bare", it currently still returns a working directory.
This conflicts with the specification of "bare"-ness.

Bug: 311902

Change-Id: Ib54b31ddc80b9032e6e7bf013948bb83e12cfd88
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>

Currently, there is no way to read the content
of the Git Configuration in a  way that would
allow to list all configured values generically.
This change extends the Config class in such a
way as to being able to get a list of sections and
to get a list of names for any given section or
subsection.
This is required in able to implement proper
configuration handling in EGit (show all the
content of a given configuration similar to 
"git config -l").

Change-Id: Idd4bc47be18ed0e36b11be8c23c9c707159dc830
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Apparently this was the intention, but never happened

Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

Refactor and extend the internals so users can override and
intervene during formatting, e.g. to colorize output.

Change-Id: Ia1cf40cfd4a5ed7dfb6503f8dfc617237bee0659
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

On Windows, FS_Win32_Cygwin has been used if a Cygwin Git installation
is present in the PATH. Assuming that the user works with the Cygwin
Git installation may result in unnecessary overhead if he actually
does not.

Applications built on top of jgit may have more knowledge on the
actually used Git client (Cygwin or not) and hence should be able to
configure which FS to use accordingly.

Change-Id: Ifc4278078b298781d55cf5421e9647a21fa5db24

A Change-Id helps tools like Gerrit Code Review to keeps different
versions of a patch together. The Change-Id is computed as a SHA-1
hash of some of the same basic information as a commit id on the first
commit intended to solve a particular problem and then reused for
updated solutions.

Change-Id: I04334f84e76e83a4185283cb72ea0308b1cb4182
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

The J2SE NIO APIs require that FileChannel close the underlying file
descriptor if a thread is interrupted while it is inside of a read or
write operation on that channel.  This is insane, because it means we
cannot share the file descriptor between threads.  If a thread is in
the middle of the FileChannel variant of IO.readFully() and it
receives an interrupt, the pack will be automatically closed on us.
This causes the other threads trying to use that same FileChannel to
receive IOExceptions, which leads to the pack getting marked as
invalid.  Once the pack is marked invalid, JGit loses access to its
entire contents and starts to report MissingObjectExceptions.

Because PackWriter must ensure that the chosen pack file stays
available until the current object's data is fully copied to the
output, JGit cannot simply reopen the pack when its automatically
closed due to an interrupt being sent at the wrong time.  The pack may
have been deleted by a concurrent `git gc` process, and that open file
descriptor might be the last reference to the inode on disk.  Once its
closed, the PackWriter loses access to that object representation, and
it cannot complete sending the object the client.

Fortunately, RandomAccessFile's readFully method does not have this
problem.  Interrupts during readFully() are ignored.  However, it
requires us to first seek to the offset we need to read, then issue
the read call.  This requires locking around the file descriptor to
prevent concurrent threads from moving the pointer before the read.

This reduces the concurrency level, as now only one window can be
paged in at a time from each pack.  However, the WindowCache should
already be holding most of the pages required to handle the working
set for a process, and its own internal locking was already limiting
us on the number of concurrent loads possible.  Provided that most
concurrent accesses are getting hits in the WindowCache, or are for
different repositories on the same server, we shouldn't see a major
performance hit due to the more serialized loading.

I would have preferred to use a pool of RandomAccessFiles for each
pack, with threads borrowing an instance dedicated to that thread
whenever they needed to page in a window.  This would permit much
higher levels of concurrency by using multiple file descriptors (and
file pointers) for each pack.  However the code became too complex to
develop in any reasonable period of time, so I've chosen to retrofit
the existing code with more serialization instead.

Bug: 308945
Change-Id: I2e6e11c6e5a105e5aef68871b66200fd725134c9
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Merges the current head with one other commit.
In this first iteration the merge command supports
only fast forward and already up-to-date.

Change-Id: I0db480f061e01b343570cf7da02cac13a0cbdf8f
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>

The CommitCommand should take care to create a merge commit if the file
$GIT_DIR/MERGE_HEAD exists. It should then read the parents for the merge
commit out of this file. It should also take care that when commiting
a merge and no commit message was specified to read the message from
$GIT_DIR/MERGE_MSG.
Finally the CommitCommand should remove these files if the commit
succeeded.

Change-Id: 	I4e292115085099d5b86546d2021680cb1454266c
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>

The strings are externalized into the root resource bundles.
The resource bundles are stored under the new "resources" source
folder to get proper maven build.

Strings from tests are, in general, not externalized. Only in
cases where it was necessary to make the test pass the strings
were externalized. This was typically necessary in cases where
e.getMessage() was used in assert and the exception message was
slightly changed due to reuse of the externalized strings.

Change-Id: Ic0f29c80b9a54fcec8320d8539a3e112852a1f7b
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>

In close() method of SshFetchConnection and SshPushConnection
errorThread.join() can wait forever if JSch will not close the
channel's error stream.  Join with a timeout, and interrupt the
copy thread if its blocked on data that will never arrive.

Bug: 312863
Change-Id: I763081267653153eed9cd7763a015059338c2df8
Reported-by: Dmitry Neverov <dmitry.neverov@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If we get an interrupt during an IO operation (src.read or dst.write)
caused by the flush() method incrementing the flush counter, ensure
we restart the proper section of code.  Just ignore the interrupt
and continue running.

Bug: 313082
Change-Id: Ib2b37901af8141289bbac9807cacf42b4e2461bd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The number of bytes to copy was truncated to an int, but the
pack's copyToStream() method expected to be passed a long here.
Pass through the long so we don't truncate a giant object.

Change-Id: I0786ad60a3a33f84d8746efe51f68d64e127c332
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Rather than holding onto the PackedObjectLoader, only hold the
PackFile and the object offset.  During a reuse copy that is all
we should need to complete a reuse, and the other parts of the
PackedObjectLoader just waste memory.

This change reduces the per-object memory usage of a PackWriter by
32 bytes on a 32 bit JVM using only OFS_DELTA formatted objects.
The savings is even larger (by another 20 bytes) for REF_DELTAs.
This is close to a 50% reduction in the size of ObjectToPack,
making it rather worthwhile to do.

Beyond the memory reduction, this change will help to make future
refactoring work easier.  We need to redo the API used to support
copying data, and disconnecting it from the PackedObjectLoader is
a good first step.

Change-Id: I24ba4e621e101f14e79a16463aec5379f447aa9b
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Rather than keep track of both the position of the object, and the
position of its data, just keep track of the number of bytes used
by the object's header in the pack.  This shaves 4 bytes out of the
size of the PackedObjectLoader instances.

We also can defer the addition instruction to the materialize()
operation, avoiding it entirely if the caller never actually uses
the loader.  This may be relevant for PackWriter invocations,
where only 1 loader gets chosen for a given object, even though
the object may appear on disk in more than one pack file.

Error reporting is now simplified, as we can rely on the object
offset rather than its data offset.  This is the value displayed
by pack debugging tools like `git verify-pack -v`, so its better
to use that in our own errors.

Because nobody needs getDataOffset() now, we can drop that from
the public API.

Change-Id: Ic639c0d5a722315f4f5c8ffda6e26643d90e5f42
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Since we use this code twice, pull it into a private method.  Let
the compiler/JIT worry about whether or not this logic should be
inlined into the call sites.

Change-Id: Ia44fb01e0328485bcdfd7af96835d62b227a0fb1
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Originally when I wrote this code I had hoped to use OffsetCache
to also implement the UnpackedObjectCache.  But it turns out they
need rather different code, and it just wasn't worth trying to
reuse the OffsetCache base class.

Before doing any major refactoring or code cleanups here, squash the
two classes together and delete OffsetCache.  As WindowCache is our
only subclass, this is pretty simple to do.  We also get a minor
code reduction due to less duplication between the two classes,
and the JIT should be able to do a better job of optimization here
as we can define types up front rather than relying on generics
that erase back to java.lang.Object.

Change-Id: Icac8bda01260e405899efabfdd274928e98f3521
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

When we read the object header we copy 20 bytes from the pack data,
then start parsing out the type and the inflated size.  For most
objects, this is only going to require 3 bytes, which is sufficient
to represent objects with inflated sizes of up to 2^16.  The local
buffer however still has 17 bytes remaining in it, and that can be
used to satisfy the OBJ_OFS_DELTA header.

We shouldn't need to worry about walking off the end of the buffer
here, because delta offsets cannot be larger than 64 bits, and that
requires only 9 bytes in the OFS_DELTA encoding.

Assuming worst-case scenarios of 9 bytes for the OFS_DELTA encoding,
the pack file itself must be approaching 2^64 bytes, an infeasible
size to store on any current technology.  However, even if this
were the case we still have 11 bytes for the type/size header.
In that encoding we can represent an object as large as 2^74 bytes,
which is also an infeasible size to process in JGit.

So drop the second read here.

The data offsets we pass into the ObjectLoaders being constructed
need to be computed individually now.  This saves a local variable,
but pushes the addition operation into each branch of the switch.

Change-Id: I6cf64697a9878db87bbf31c7636c03392b47a062
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

JSch may hang or abort with the timeout if JGit connects before
its obtained the streams.  Instead defer the connect() call until
after the streams have been configured.

Bug: 312383
Change-Id: I7c3a687ba4cb69a41a85e2b60d381d42b9090e3f
Reported-by: Dmitry Neverov <dmitry.neverov@gmail.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If a flush() gets delivered at the same time that we are blocking
while writing to an interruptable stream, the copy thread will
abort assuming its a stream error.  Instead ignore the interrupt,
and retry the write.

Change-Id: Icbf62d1b8abe0fabbb532dbee088020eecf4c6c2
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

It is possible to miss flush() invocation in StreamCopyThread.
In this case some data will not be sent to remote host and we will
wait forever (or until timeout) in src.read().

Use a counter to keep track of the flush requests.

Change-Id: Ia818be9b109a1674d9e2a9c78e125ab248cfb75b
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Added a new package org.eclipse.jgit.api and a builder-style API for
jgit. Added also the first implementation for two git commands: Commit
and Log.

This API is intended to be used by external components when
functionalities of the standard git commands are required. It will also
help to ease writing JGit tests.

For internal usages this API may often not be optimal because the git
commands are doing much more than required or they expect parameters of
an unappropriate type.

Change-Id: I71ac4839ab9d2f848307eba9252090c586b4146b
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>

Currently, if the Index contains a file in more than one stage, only
the last entry (containing the highest stage) will be registered in
GitIndex. For applications it can be useful to not only know about the
highest stage, but also which other stages are present, e.g. to detect
the type of conflict the file is in.

Change-Id: I2d4ff9f6023335d9ba6ea25d8e77c8e283ae53cb
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

The repository state tells in which state the repo is and also which actions
are currently allowed. The state MERGING is telling that a commit is not
possible. But this is only true in the case of unmerged paths in the index.
When we are merging but have resolved all conflicts then we are in a special
state: We are still merging (means the next commit should have multiple
parents) but a commit is now allowed.

Since the MERGING state "canCommit()" cannot be enhanced to return true/false
based on the index state (MERGING is an enum value which does not have a
reference to the repository its state it is representing) I had to introduce a new
state MERGING_RESOLVED. This new state will report that a commit is possible.

CAUTION: there might be the chance that users of jgit previously blindly did a
plain commit (with only one parent) when the RepositoryState allowed them to
do so. With this change these users will now be confronted with a RepositoryState
which says a commit is possible but before they can commit they'll have to
check the MERGE_MESSAGE and MERGE_HEAD files and use the info from these
files.

Change-Id: I0a885e2fe8c85049fb23722351ab89cf2c81a431
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

If two keys are the same length, but don't share the same sequence
of characters, we were incorrectly claiming they still matched due
to a bug in the for loop condition.  I used the wrong variable and
the loop never executed, resulting in equality anytime the two keys
being compared were the same length.

Use the proper local variable to loop through the arrays, and add
a JUnit test to verify equality works as expected.

Change-Id: I4a02400e65a9b2e0da925b05a2cc4b579e1dd33a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If a loose object was corrupted by truncation, JGit would hang.

Change-Id: I7e4c14f44183a5fcb37c1562e81682bddeba80ad
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

If a concurrent thread picks up a newly created PackFile and adds
it to the pack list before the IndexPack thread itself can insert
the item onto the front of the list, do nothing and use the item
that was picked up by that other concurrent scanning thread.

This avoids a potential condition where the same pack exists in
memory twice, which causes confusion later during a rescan of the
directory because we don't know exactly which PackFile instance
should be retained into the new list, and which should be discarded.

We can stop searching through the old pack list as soon as the
sort function declares that the item to insert should be before
the item already in the list.  Because the list is always sorted
by modification time (in seconds), we should never encounter a
case where the pack is positioned at the wrong spot in the list.
This early break out still permits an efficient implementation of
the common case, inserting a new pack at the head of the list.

Change-Id: Ice4459bbd4ee9487078aff5257893883d04f05fb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

There is a potential race condition during insertPack that can lead
to us having the same pack file open twice in the same directory.

A different thread can miss an object on disk, and trigger a scan
of the directory, and notice the pack that was put in by IndexPack.
So the pack winds up in the newly created PackList.

The IndexPack thread then wakes up and finishes its insertPack by
creating a new PackFile and inserting it into position 0 of the list.
We now have the same pack listed twice.

Readers will favor the earlier PackFile instance, because its the
first one they come across as they iterate through the list.

Keep that earlier one when we scan the pack directory again, as
this will avoid needing to purge out all of the windows that may
have been cached.

Of course we should also fix that race condition, but this block
was taking the wrong resolution if this error ever shows up, so
lets first fix the block to use a more sane resolution.

Change-Id: I0d339b9fd1dd8012e8fe5a564b893c0f69109e28
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This reuse line was identical between the two branches related to
reusing a delta, or reusing a whole object.  Either way they reuse
the body of the object as-is.  So just make that a common function
after the header is written.

Change-Id: I0e6673b8e813c8c08c594ea2ba546fd366339d5d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If a corrupt loose object is read, UnpackedObjectLoader was disposing
of the Inflater, and then attempting to return the disposed Inflater
to the InflaterCache.  Since the disposed Inflater had its native
libz resource deallocated and its reference cleared out, the Inflater
threw NullPointerException and refused to reset itself before being
put back into the cache.

Instead of disposing of the Inflater when corruption is found, do
nothing, and allow it to be returned to the cache.  The instance
will get reset, and should be usable by a future caller.

Bug: 310291
Change-Id: I44f2247c08b6e04fa62f8399609341b07508c096
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This option was mis-named from day 1.  Its not checking that the
objects provided by the client are reachable, its actually doing
a scan to prove that objects referenced by the client are already
reachable through another reference on the server, or were sent
as part of the pack from the client.

Rename it checkReferencedObjectsAreReachable, since we really are
trying to validate that objects referenced by the client's actions
are reachable to the client.

We also need to ensure we run checkConnectivity() anytime this is
enabled, even if the caller didn't turn on fsck for object formats.
Otherwise the check would be completely bypassed.

Change-Id: Ic352ddb0ca8464d407c6da5c83573093e018af19
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If we are checking the visibility of everything referenced in the
pack that isn't already reachable by a reference, it needs to be
in the provided set.  Since the provided set lists everything that
is in this pack, we can avoid checking to see if the blob exists
on disk, because we know it should be there, it was found in the
pack we just consumed.

Change-Id: Ie3c7746f734d13077242100a68e048f1ac18c34a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If a tree was referenced but not provided in the pack, report it
as a missing tree and not as a missing blob.

Change-Id: Iab05705349cdf0d30cc3f8afc6698a8d2a941343
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The only current consumer of these collections is ReceivePack,
where it needs to test ObjectId equality between a RevObject and an
ObjectId.  There we were copying from a traditional HashSet<ObjectId>
into an ObjectIdSubclassMap<ObjectId>, as the latter can perform
hashing using ObjectId's native value support, bypassing RevObject's
override on hashCode() and equals().  Instead of doing that copy,
directly create ObjectIdSubclassMap instances inside of ReceivePack.

We also only need to record the objects that do not appear in the
incoming pack, and were therefore copied from the local repositiory
in order to complete delta resolution.  Instead of listing everything
that used an OBJ_REF_DELTA format, list only the objects that we
pulled from the destination repository via a normal ObjectLoader.

ReceivePack can now discard the IndexPack object, and all of its
other data, as soon as these collections are held by the check
connectivity method.  This frees up memory for the ObjectWalk's
own RevObject pool.

Change-Id: I22ef71b45c2045a0202e7fd550a770ee1f6f38a6
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>