Shawn Pearce [Fri, 15 Aug 2014 17:21:32 +0000 (10:21 -0700)]
ResolveMerger: Use the ObjectReader to access objects
This is necessary to ensure objects accessed by the TreeWalk come from
the associated ObjectInserter when the merger is a RecursiveMerger
instance and a virtual common base was constructed but not flushed.
Shawn Pearce [Wed, 13 Aug 2014 23:54:22 +0000 (16:54 -0700)]
Cleanup use of java.util.Inflater, fixing rare infinite loops
The native implementation of inflate() can set finished to return
true at the same time as it copies the last bytes into the buffer.
Check for finished on each iteration, terminating as soon as libz
knows the stream was completely inflated.
If not finished, it is likely input is required before the next
native call could do any useful work. Most invocations are passing
in a buffer large enough to store the entire result. A partial return
from inflate() will need more input before it can continue. Checking
right away that needsInput() is true saves a native call to determine
no bytes can be inflated without more input.
This should fix a rare infinite loop condition inside of inflation
when an object ends exactly at the end of a block boundary, and
the next block contains only the 20 byte trailing SHA-1.
When the stream is finished each new attempt to inflate() returns
n == 0, as no additional bytes were output. The needsInput() test
tries to add the length of the footer block to itself, but then loops
back around an reloads the same block as the block is smaller than
a full block size. A zero length input is set to the inflater,
which triggers needsInput() condition again.
openInputStream() is never used in JGit, nor is it used by any
known working DFS implementation. The method was added as a
utility for reading back from a DfsInserter, but the final
implementation of that feature does not requrire this method.
Dave Borowitz [Fri, 3 May 2013 18:45:36 +0000 (11:45 -0700)]
Add a Merger.merge method that makes flushing optional
This allows callers performing multiple separate merges to reuse a
single ObjectInserter without flushing the inserter on each iteration
(which can be slow in the DFS case).
Dave Borowitz [Fri, 3 May 2013 17:54:38 +0000 (10:54 -0700)]
Teach mergers to use ObjectInserter.newReader()
This allows the RecursiveMerger to iteratively create new merge bases
without necessarily flushing packs to storage in the DFS case;
flushing only need happen at the end of the whole merge process.
Since Merger's walk now depends on its inserter, we need to construct
an inserter at Merger construction time. This should not be a
significant increase in overhead since unused inserters don't use any
resources (beyond a reference to the Repository).
We also must release and recreate the walk whenever setObjectInserter
is called, which can break usages where setObjectInserter is called in
the middle of stateful operations on the walk. No usages of this
method within JGit currently do this; the inserter is only ever set
before any stateful walk operations happen.
Dave Borowitz [Wed, 1 May 2013 17:20:31 +0000 (10:20 -0700)]
Add a method to ObjectInserter to read back inserted objects
In the DFS implementation, flushing an inserter writes a new pack to
the storage system and is potentially very slow, but was the only way
to ensure previously-inserted objects were available. For some tasks,
like performing a series of three-way merges, the total size of all
inserted objects may be small enough to avoid flushing the in-memory
buffered data.
DfsOutputStream already provides a read method to read back from the
not-yet-flushed data, so use this to provide an ObjectReader in the
DFS case.
In the file-backed case, objects are written out loosely on the fly,
so the implementation can just return the existing WindowCursor.
Shawn Pearce [Tue, 12 Aug 2014 21:57:28 +0000 (14:57 -0700)]
DfsInserter: buffer up to streamFileThreshold from InputStream
Since 2badedcbe0f87c0a in-core merges can write up to 10 MiB
into a TemporaryBuffer.Heap strategy, where the data is stored
as a chain of byte[] blocks.
Support the inserter reading up to the streamFileThreshold (default 50
MiB) from the supplied input stream and hash the content to determine
if the merged result blob is already present in the repository. This
allows the inserter to avoid creating duplicate objects in more cases,
reducing repository pack file churn.
Shawn Pearce [Sat, 9 Aug 2014 00:53:49 +0000 (17:53 -0700)]
Process most in-core merges without local temp files
Instead of always writing to disk use TemporaryBuffer.LocalFile to
store up to 10 MiB of merge result in RAM. Most source code will
fit into this limit, avoiding local disk IO for simple merges.
Larger files will automatically spool to a temporary file that
can be cleaned up in the finally, reducing the risk of leaving
them on disk and consuming space in /tmp.
Shawn Pearce [Fri, 8 Aug 2014 22:04:57 +0000 (15:04 -0700)]
ResolveMerger: push result file creation into updateIndex()
The only caller of writeMergedFile is updateIndex, and the only
user of this path object is the code within the method. This is
a no-op change that opens the door to refactoring the way temp
files are handled for inCore merges.
Shawn Pearce [Fri, 8 Aug 2014 18:19:45 +0000 (11:19 -0700)]
RecursiveMerger: do not insert virtual commits into repository
When merging common ancestors to create a single virtual common
ancestor the commit does not need to be inserted into the Git
repository. Instead just mock it out in memory as part of the
merger's RevWalk pool.
Make the author and committer stable and predictable for any
given pair of merge bases. It is not necessary for the caller's
name or email to be used as the commit will not be written out.
Currently if the remote defined in repo manifest xml is non-relative (e.g.
"https://chromium.googlesource.com"), our code will break. This change fixed
that.
It also makes that remotes are ending with "/".
Change-Id: Icef46360b32227a9db1d9bb9e6d929c72aeaa8df Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
Robin Stocker [Tue, 5 Aug 2014 13:38:07 +0000 (23:38 +1000)]
Fix CheckoutCommand not setting up tracking
Instead of passing on the start point as is to CreateBranchCommand, the
resolved ObjectId was used. Given this, CreateBranchCommand did not set
up tracking.
This also fixes CreateBranchCommand with setStartPoint(null) to use HEAD
(instead of NPEing), as documented in the Javadoc.
Bug: 441153
Change-Id: I5ed82b4a4b4a32a81a7fa2854636b921bcb3d471 Signed-off-by: Robin Stocker <robin@nibor.org>
Matthias Sohn [Sun, 3 Aug 2014 23:04:26 +0000 (01:04 +0200)]
Run unit tests in parallel
Configure Maven surefire plugin to run tests concurrently using
multiple processes. By default use one process per core available
on the machine running the tests.
Running this on my MacBook Pro i7 with 8 cores 16 GB RAM SSD
I get the following results:
forkCount "mvn clean install" on org.eclipse.jgit.test
--------------------------------------------------------
1 02:36 min
2 01:26 min
4 00:56 min
6 00:49 min
8 00:49 min
10 00:51 min
When on-access scan of McAfee virus scanner is enabled the optimum
is at half the number of cores (4 of 8) on my MacBook.
Set system property "test-fork-count" to override the default.
Either set it to an absolute number e.g. "4" or in units of
cores available on the system running the build. E.g. "0.5C" to
use 4 cores on a 8 core system or "1C" to use all 8 cores on
a 8 core system.
E.g.
$ mvn clean install -Dtest-fork-count=0.5C
to use half the number of cores processes to run tests.
Change-Id: Ifdbd1ae8153f2033922eae6ae8942c36db1f9806 Signed-off-by: Sohn <matthias.sohn@sap.com>
IndexDiffFilter should never filter entries with stage!=0
If the IndexDiffFilter is asked whether it should include or filter out
a certain path and for that path there is a dircache entry with a stage
different from 0, then the filter should never filter out this entry.
IndexDiffFilter is an optimized version of AnyDiffFilter and there is no
case where the index contains non-0 stages but we still don't see any
diff for that path.
Change-Id: I25915880f304090fe90584c79bddf021231227a2 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix API errors raised on ResolveMerger affecting API providers only
In change If45bc3d078b3d3de87b758e71d7379059d709603 a new parameter was
added to 3 protected methods of ResolveMerger. This breaks the code of
developers which have subclassed ResolveMerger. The API baseline check
in Eclipse reports this as API breakage.
Since this will break only providers but not consumers of the API this
should be allowed also in minor versions. According to OSGi semantic
versioning
http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf
breaking providers in a minor version update is ok.
Therefore silence these errors using API filter rules.
Bug: 440757
Change-Id: Icabbd0e1de7e877c66a5c4a2c8391473f992a1aa Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Small performance optimization for ignore rules/fnmatcher
- don't check empty segments generated by String.split()
- don't continue to add segments if the matcher fails to match input
- don't add empty heads
- don't iterate over empty heads.
Apparently repo allows projects overlapping, e.g. one project's path is "foo"
and another project's path is "foo/bar". This is not supported in git submodule.
At JGit repo side we'll skip all the submodules that are in subdirectories of
other submodules, and on repo side we'll make them submodules to resolve this
problem.
Change-Id: I6820c4ef400c530a36150b1228706adfcc43ef64 Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
RecursiveMerger should not fail on content-merge conflicts of parents
Previously when RecursiveMerger was trying to create a single virtual
common base for the merge it was failing when this lead to content-merge
conflicts. This is different from what native git is doing. When native
git's recursive merge algorithm creates a new common base it will merge
the multiple parents and simply take the merge result (potentially
including conflict markers) as common base. See my discussion with Shawn
here: http://www.spinics.net/lists/git/msg234959.html :
> - How should workingtree, index (stage1,2,3) look like if during
that
> merge of common ancestors a conflict occurs? Will I see in stage2
and
> stage3 really see content of X1 and X2?
Its done entirely in memory and never touches the working tree or
index. When a conflict exists in the X1-X2 merge the conflict is
preserved into the new virtual base.
There is still the possibility that the merge of parents lead to
conflicts. File/Folder conclicts, conflicts on filemodes. This commit
only fixes the situation for conflicts when merging content.
Fixed message for exception thrown during recursive merge
During recursive merge jgit potentially has to merge multiple
common ancestors. If this fails because there are conflicts then
the exception thrown for that should have a message which states
this clearly. Previously a wrong message was given ("More than 200
merge bases ...")
HistogramDiff: Convert stack recursion to heap managed queue
Each time the longest common substring is found the diff algorithm
recurses to reprocess the regions before and after the common string.
Large files with many edits can trigger StackOverflowError as the
algorithm attempts to process a deeply split tree of regions. This
is especially prone to happen in servers where the Java stack size
may have been limited to 1M or even 256K.
To keep edits produced in order a queue is used to process edits
in a depth-first strategy.
Fix RecursiveMerger's internal use of merge to find a merge base
When RecursiveMerger tried to determine a common base tree it was
recursively tried to merge multiple common bases. But these intermediate
merges which have just been done to determine a single common base for
the final merge already filled some important fields (toBeCheckedOut,
toBeDeleted, ...). These side effects of the intermediate merges led to
wrong results of the final merge. One symptom was that after a recursive
merge which should be succesful you could still see leftover files in
the worktree: files which existed in the (virtual) common base but which
don't exist anymore in the branches to be merged.
The solution is easy: Clear the appropriate fields after common base
determination and start the final merge with a clean state.
Change-Id: I644ea9e1cb15360f7901bc0483cdb9286308c226 Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
As described in native gits file "git-read-tree.txt" git has in a
special mode when doing the "initial" checkout. "Initial" means that the
index is empty before the checkout. This was not handled correctly in
JGit and is fixed in this commit. Also see
https://github.com/git/git/blob/master/Documentation/git-read-tree.txt#L181
Konrad Kügler [Fri, 23 May 2014 21:09:17 +0000 (23:09 +0200)]
Preserve merges during pull if configured to do so
Setting branch.<name>.rebase or pull.rebase to 'preserve' will preserve
merges during rebase. Also, pull.rebase is now consulted if there is no
branch-specific configuration.
Bug: 429664
Change-Id: I345fa295c7e774e0d0a8e6aba30fbfc3552e0084 Signed-off-by: Konrad Kügler <swamblumat-eclipsebugs@yahoo.de>
Konrad Kügler [Mon, 19 May 2014 19:47:27 +0000 (21:47 +0200)]
Rebase: Add --preserve-merges support
With --preserve-merges C Git re-does merges using the rewritten merge
parents, discarding the old merge commit. For the common use-case of
pull with rebase this is unfortunate, as it loses the merge conflict
resolution (and other fixes in the merge), which may have taken quite
some time to get right in the first place.
To overcome this we use a two-fold approach:
If any of the (non-first) merge parents of a merge were rewritten, we
also redo the merge, to include the (potential) new changes in those
commits.
If only the first parent was rewritten, i.e. we are merging a branch
that is otherwise unaffected by the rebase, we instead cherry-pick the
merge commit at hand. This is done with the --mainline 1 and --no-commit
options to apply the changes introduced by the merge. Then we set up an
appropriate MERGE_HEAD and commit the result, thus effectively forging a
merge.
Apart from the approach taken to rebase merge commits, this
implementation closely follows C Git. As a result, both Git
implementations can continue rebases of each other.
Preserving merges works for both interactive and non-interactive rebase,
but as in C Git it is easy do get undesired outcomes with interactive
rebase.
CommitCommand supports committing merges during rebase now.
Bug: 439421
Change-Id: I4cf69b9d4ec6109d130ab8e3f42fcbdac25a13b2 Signed-off-by: Konrad Kügler <swamblumat-eclipsebugs@yahoo.de>
Robin Rosenberg [Sat, 28 Jun 2014 13:31:11 +0000 (15:31 +0200)]
Try to fix intermittent test failure related to file timestamps
Improve how tests do a "touch" operation on files. Instead of doing
"f.setLastModified(System.currentTimeMillis)" open a Outputstream on the
file and directly close it again. This makes this method rely only on
one clock - the clock of the filesystem.
Bug: 436917
Change-Id: I68ef3c2878f28b12daebf2ef6a9fa0a5d6e0964d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Robin Rosenberg [Sat, 28 Jun 2014 08:24:15 +0000 (10:24 +0200)]
Fix formatting errors in error messages
{} is plain wrong and is not accepted by MessageFormat, the other risk
becoming wrong if another single quote is introduced in the future and
sets a bad example.
Stefan Lay [Wed, 18 Jun 2014 09:33:34 +0000 (11:33 +0200)]
Allow retrying connecting SshSession in case of an exception
Connecting to an SshSession may fail due to different reasons. Jsch for
example often throws an com.jcraft.jsch.JschException: verify: false.[1]
The issue is still not fixed in JSch 0.1.51.
In such a case it is worth retrying to connect. The number of connection
attempts can be configured using ssh_config parameter
"ConnectionAttempts" [2].
Dave Borowitz [Thu, 12 Jun 2014 17:04:01 +0000 (10:04 -0700)]
Rename RewriteTreeFilter to TreeRevFilter and make it public
The current behavior of passing a TreeFilter to RevWalk has limited
usefulness, since the RevFilter derived from the TreeFilter is always
ANDed together with any other RevFilters. It is also tied fairly
tightly to the parent rewriting mechanism.
Make TreeRevFilter a generic RevFilter that matches modified paths
against any TreeFilter. This allows for more complex logic like
(modified this path OR authored by this person).
Leave the rewrite flag logic in this class, since it's closely tied to
the parent comparison code, but hidden behind a protected constructor.
projects / jgit.git / log