Pretty printing the response is useful for human readers, but most
(if not all) of the time, the response will be read by programs.
Remove it to avoid the additional overhead of the formatting and
extra bytes in the response. Adjust the test accordingly.
Note that LfsProtocolServlet already doesn't use pretty printing,
so this change makes FileLfsServlet's behavior consistent. In fact,
both classes now have duplicate Gson handling; this will be cleaned
up in a separate change.
Change-Id: I113a23403f9222f16e2c0ddf39461398b721d064 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
This class is an inner class, but does not use its embedded reference
to the object which created it. This reference makes the instances
of the class larger, and may keep the reference to the creator object
alive longer than necessary. If possible, the class should be made
static.
Change-Id: I245e44678166176de0cfb275e22ddd159f88e0bd Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Marc Strapetz [Sat, 18 Nov 2017 16:50:30 +0000 (17:50 +0100)]
FileBasedConfig: support for relative includes
Relative include.path are now resolved against the config's parent
directory. include.path starting with ~/ are resolved against the
user's home directory
Change-Id: I91911ef404126618b1ddd3589294824a0ad919e6 Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Marc Strapetz [Mon, 20 Nov 2017 17:21:22 +0000 (18:21 +0100)]
ConfigTest: fix on Windows
Change-Id: I37a2ef611aef97faf1b891a9660c1745435a915d Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sun, 3 Dec 2017 12:54:28 +0000 (13:54 +0100)]
Silence API warnings for reintroduced ResolveMerger#processEntry
This was silenced before but suppression was unintentionally lost in
merge commit 6858339c1e2878d5c5dc6cc9b422f9802be950ae.
This method was removed in 4.9.0 and reintroduced in 4.9.1 to avoid
breaking EMF compare versions which were built against older versions.
Matthias Sohn [Thu, 30 Nov 2017 12:37:30 +0000 (13:37 +0100)]
Merge branch 'stable-4.9'
* stable-4.9:
GC: Delete stale temporary packs and indexes
Change-Id: I49b37845ee8a465404b801a2d8de0205a2e7ba30 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Jonathan Nieder [Tue, 28 Nov 2017 16:16:56 +0000 (08:16 -0800)]
RepoCommand: Skip RemoteReader when encountering a full SHA-1
There is no point in calling back to the RemoteReader to resolve a
40-digit hex SHA-1 to itself. We already skip that call when not
ignoring remote failures; skip it when ignoring remote failures, too.
This should simplify RemoteReader implementations.
When a GC operation is interrupted, temporary packs and indexes can be
left on the pack folder. In big, busy repositories this can lead to
significant amounts of wasted disk space if this interruption is done
with a certain frequency.
Remove stale temporary packs and indexes at the end of the GC process so
they do not accumulate. To avoid interfering with a possible concurrent
JGit GC process in the same repository, only delete temporary files that
are older than one day.
Stephen Lawson [Wed, 22 Nov 2017 12:41:05 +0000 (12:41 +0000)]
Performance improvement on writing a large index
The index header consists of a 4-byte version number. The current
supported version numbers are 2 and 3. The code checks if any entries
are extended. If it finds any entries that are extended it picks version
'3', otherwise it chooses version '2'.
DirCache.java
-Changed the 'extended' check to exit early when any entry is considered
'extended' in the index.
(Of course, I maybe missing a bitwise optimization that is made in
the Java bytecode.)
Change-Id: If70db9454befe683319b974ebd3774060be9445d Signed-off-by: Stephen Lawson <slawson@ptc.com>
Matthias Sohn [Fri, 24 Nov 2017 00:18:13 +0000 (01:18 +0100)]
Merge branch 'stable-4.9'
* stable-4.9:
Ignore warning for minor version change without API change
Silence boxing warning
Prepare 4.5.5-SNAPSHOT builds
JGit v4.5.4.201711221230-r
Fix LockFile semantics when running on NFS
Honor trustFolderStats also when reading packed-refs
Prepare 4.5.4-SNAPSHOT builds
JGit v4.5.3.201708160445-r
When running on NFS there was a chance that JGits LockFile
semantic is broken because File#createNewFile() may allow
multiple clients to create the same file in parallel. This
change provides a fix which is only used when the new config
option core.supportsAtomicCreateNewFile is set to false. The
default for this option is true. This option can only be set in the
global or the system config file. The repository config file is not
taken into account in this case.
If the config option core.supportsAtomicCreateNewFile is true
then File#createNewFile() is trusted and the behaviour doesn't
change.
But if core.supportsAtomicCreateNewFile is set to false then after
successful creation of the lock file a hardlink to that lock file is
created and the attribute nlink of the lock file is checked to be 2. If
multiple clients manage to create the same lock file nlink would be
greater than 2 showing the error.
This expensive workaround is described in
https://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
section III.d) "Exclusive File Creation"
Honor trustFolderStats also when reading packed-refs
Then list of packed refs was cached in RefDirectory based on mtime of
the packed-refs file. This may fail on NFS when attributes are cached.
A cached mtime of the packed-refs file could cause JGit to trust the
cached content of this file and to overlook that the file is modified.
Honor the config option trustFolderStats and always read the packed-refs
content if the option is false. By default this option is set to true
and this fix is not active.
Thomas Wolf [Sat, 11 Nov 2017 10:41:10 +0000 (11:41 +0100)]
Yet another work-around for a Jsch bug: timeouts
Jsch 0.1.54 passes on the values from ~/.ssh/config for
"ServerAliveInterval" and "ConnectTimeout" as read from
the config file to java.net.Socket.setSoTimeout(). That
method expects milliseconds, but the values in the config
file are seconds!
The missing conversion in Jsch means that the timeout is
set way too low, and if the server doesn't respond within
that very short time frame, Jsch kills the connection and
then throws an exception with a message such as "session is
down" or "timeout in waiting for rekeying process".
As a work-around, do the conversion to milliseconds in the
Jsch-facing Config interface of OpenSshConfig. That way Jsch
already gets these values as milliseconds.
Bug: 526867
Change-Id: Ibc9b93f7722fffe10f3e770dfe7fdabfb3b97e74 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
David Pursehouse [Mon, 20 Nov 2017 06:38:43 +0000 (15:38 +0900)]
Merge branch 'stable-4.9'
* stable-4.9:
Fix NPE in TransportGitSsh.ExtSession.exec()
Add missing help text for rev-parse's --verify option
Remove final modifier on args4j argument field in RevParse
Change-Id: I5ac9e2f185f2210ee76970501710b99b12e93e75 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
So far, in order to get the pack directory it was necessary to resolve
it from the object directory. This resolution is already done when
creating the object directory, so simplify the call by just adding a
getter to the pack directory.
Matthias Sohn [Mon, 13 Nov 2017 21:41:57 +0000 (22:41 +0100)]
Silence invalid @since 3.5 API warning on ResolveMerge#processEntry
This method was removed in 4.9 and reintroduced in
I48ba4308dee73925fa32d6c2fd6b5fd89632c571 as deprecated in 4.9.1 in
order to help EMF Compare to avoid breakage.
Change-Id: Ia638517178313da42ae13ebcf88ad535d9a02723 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* changes:
BitmapWalker: do not revisit objects in bitmap
Use bitmaps for non-commit reachability checks
Make PackWriterBitmapWalker public
UploadPackTest: construct commits in test method
Jonathan Tan [Thu, 2 Nov 2017 01:20:54 +0000 (18:20 -0700)]
BitmapWalker: do not revisit objects in bitmap
Currently, BitmapWalker walks through every object returned by the
internal ObjectWalk, regardless of whether that object has already
been marked in the bitmap. Set an object filter to ensure that only
bitmap-unmarked objects are walked through.
Change-Id: I22a8874b1e571df3c33643b365036d95f52fe7c7 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Thu, 2 Nov 2017 00:36:18 +0000 (17:36 -0700)]
Use bitmaps for non-commit reachability checks
Currently, unless RequestPolicy#ANY is used, UploadPack rejects all
non-commit "want" lines unless they were advertized. This is fine,
except when "uploadpack.allowreachablesha1inwant" is true
(corresponding to RequestPolicy#REACHABLE_COMMIT), in which case one
would expect that "want"-ing anything reachable would work.
(There is no restriction that "want" lines must only contain commits -
it is allowed for refs to directly point to trees and blobs, and
requesting for them using "want" lines works.)
This commit has been written to avoid performance regressions as much
as possible. In the usual (and currently working) case where the only
unadvertized things requested are commits, we do a standard RevWalk in
order to avoid incurring the cost of loading bitmaps. However, if
unadvertized non-commits are requested, bitmaps are used instead, and
if there are no bitmaps, a WantNotValidException is thrown (as is
currently done).
Change-Id: I68ed4abd0e477ff415c696c7544ccaa234df7f99 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Wed, 25 Oct 2017 21:09:17 +0000 (14:09 -0700)]
Make PackWriterBitmapWalker public
Make PackWriterBitmapWriter class public and move it to a more central
location, in preparation for its use by another class (in a subsequent
commit).
One of its inner static classes, AddUnseenToBitmapFilter, previously
package-private, is also used directly in its former package. Therefore,
AddUnseenToBitmapFilter and its sibling class have been moved to an
internal package instead.
Change-Id: I740bc4bfc4e4e3c857d1ee7d25fe45e90cd22a75 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Thomas Wolf [Tue, 7 Nov 2017 07:19:56 +0000 (08:19 +0100)]
Work around a Jsch bug: ensure the user name is set from URI
JSch unconditionally overrides the user name given in the connection
URI by the one found in ~/.ssh/config (if that does specify one for
the used host). If the SSH config file has a different user name,
we'll end up using the wrong name, which typically results in an
authentication failure or in Eclipse/EGit asking for a password for
the wrong user.
Unfortunately there is no way to prevent or circumvent this Jsch
behavior up front; it occurs already in the Session constructor at
com.jcraft.jsch.Session() and the Session.applyConfig() method. And
while there is a Session.setUserName() that would enable us to correct
this, that latter method has package visibility only.
So resort to reflection to invoke that setUserName() method to ensure
that Jsch uses the user name from the URI, if there is one.
Bug: 526778
Change-Id: Ia327099b5210a037380b2750a7fd76ff25c41a5a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Shawn Pearce [Thu, 9 Nov 2017 17:12:59 +0000 (09:12 -0800)]
Reject pack if delta exceeds array size limit
JGit's delta handling code requires the target to be a single byte
array. Any attempt to inflate a delta larger than fits in the 2GiB
limit will fail with some form of array index exceptions. Check for
this overflow early and abort pack parsing.
Suppress "Unlikely argument type for equals()" warnings in tests
This new warning was introduced in Eclipse 4.7 Oxygen [1].
The only instances of the warning are in test code that is asserting
that some class does not compare equal to Strings. As in the Gerrit
project [2] these asserts are arguably overkill, but arguably also
a reasonable test of an equals implementation. Ignore the warning in
these cases.
Note that if the project is opened in an earlier version of Eclipse,
a warning "Unsupported @SuppressWarnings" will be emitted.
* stable-4.9:
PackInserter: Implement newReader()
Move some strings from DfsText to JGitText
FileRepository: Add pack-based inserter implementation
ObjectDirectory: Factor a method to close open pack handles
ObjectDirectory: Remove last modified check in insertPack
Applications that use ObjectInserters to create lots of individual
objects may prefer to avoid cluttering up the object directory with
loose objects. Add a specialized inserter implementation that produces a
single pack file no matter how many objects. This inserter is loosely
based on the existing DfsInserter implementation, but is simpler since
we don't need to buffer blocks in memory before writing to storage.
An alternative for such applications would be to write out the loose
objects and then repack just those objects later. This operation is not
currently supported with the GC class, which always repacks existing
packs when compacting loose objects. This in turn requires more
CPU-intensive reachability checks and extra I/O to copy objects from old
packs to new packs.
So, the choice was between implementing a new variant of repack, or not
writing loose objects in the first place. The latter approach is likely
less code overall, and avoids unnecessary I/O at runtime.
The current implementation does not yet support newReader() for reading
back objects.
Dave Borowitz [Wed, 1 Nov 2017 14:19:38 +0000 (10:19 -0400)]
ObjectDirectory: Remove last modified check in insertPack
GC explicitly handles the case where a new pack has the same name as an
existing pack due to it containing the exact same set of objects. In
this case, the pack passed to insertPack will have the same name as an
existing pack, but it will also almost certainly have a later mtime than
the existing pack.
The loop in insertPack tried to short-circuit when inserting a new pack,
to avoid walking more of the pack list than necessary. Unfortunately,
this means it will never get to the check for an identical name,
resulting in a duplicate entry for the same PackFile in the pack list.
Remove the short-circuit so that insertPack does not insert a duplicate
entry.
Saša Živkov [Fri, 27 Oct 2017 10:29:52 +0000 (12:29 +0200)]
Move loggers to the top of their class
There is a possibility of hitting NPE on a logger if it is not the first
statically initialized member. For example, if another static
initializer creates an instance of its class and the logger is used
from the constructor.
Matthias Sohn [Tue, 24 Oct 2017 21:13:37 +0000 (23:13 +0200)]
Reintroduce protected method which removal broke EMF Compare
So far we follow OSGi semantic versioning [1] which says the following:
"A change in the second (minor) part of the version signals that the
change is backward compatible with consumers of the API package but not
with the providers of that API. That is, when the API package goes from
version 1.5 to 1.6 it is no longer compatible with a provider of that
API but consumers of that API are backward compatible with that API
package."
The change Ib5fbf17bdaf727bc5d0e106ce88f2620d9f87a6f broke EMF Compare
which subclasses ResolveMerger since we added a new parameter to the
protected ResolveMerger.processEntry() method. According to the above
cited OSGi semantic versioning this is ok, implementers should expect
that they break on minor version changes of the API they implement.
This change reintroduces the old processEntry() method in order to help
avoid breakage for existing EMF Compare versions which expect breakage
also for the implementer case only for major version change (in this
case from JGit 4.x to 5.x).
Han-Wen Nienhuys [Tue, 17 Oct 2017 13:17:34 +0000 (15:17 +0200)]
Throw BinaryBlobException from RawParseUtils#lineMap.
This makes detection of binaries exact for ResolveMerger and
DiffFormatter: they will classify files as binary regardless of where
the '\0' occurs in the text.
This method creates a RawText from a blob, but avoids reading the blob
if the start contains null bytes. This should reduce the amount of
garbage that Gerrit produces for changes with binaries.
Michael Keppler [Thu, 12 Oct 2017 07:38:31 +0000 (09:38 +0200)]
Avoid bad rounding "1 year, 12 months" in date formatter
Round first, then calculate the labels. This avoids "x years, 12 months"
and instead produces "x+1 years".
One test case has been added for the original example the bug was found
with, and one assertion has been moved from an existing test case to the
new test case, since it also triggered the bug.
Bug: 525907
Change-Id: I3270af3850c4fb7bae9123a0a6582f93055c9780 Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn Pearce [Sat, 12 Aug 2017 21:31:16 +0000 (14:31 -0700)]
dfs: Switch InMemoryRepository to DfsReftableDatabase
This ensure DfsReftableDatabase is tested by the same test suites that
use/test InMemoryRepository. It also simplifies the logic of
InMemoryRepository and brings its compatibility story closer to any
other DFS repository that uses reftables for its reference storage.
Change-Id: I881469fd77ed11a9239b477633510b8c482a19ca Signed-off-by: Minh Thai <mthai@google.com> Signed-off-by: Terry Parker <tparker@google.com>
Shawn Pearce [Sat, 12 Aug 2017 19:19:30 +0000 (12:19 -0700)]
dfs: reftable backed DfsRefDatabase
DfsReftableDatabase is a new alternative for DfsRefDatabase that
handles more operations for the implementor by delegating through
reftables. All reftable files are stored in sibling DfsObjDatabase
using PackExt.REFTABLE and PackSource.INSERT.
Its assumed the DfsObjDatabase periodically runs compactions and GCs
using DfsPackCompactor and DfsGarbageCollector. Those passes are
essential to collapsing the stack of reftables.
Change-Id: Ia03196ff6fd9ae2d0623c3747cfa84357c6d0c79 Signed-off-by: Minh Thai <mthai@google.com> Signed-off-by: Terry Parker <tparker@google.com>
Thomas Wolf [Wed, 18 Oct 2017 20:45:36 +0000 (22:45 +0200)]
Ensure that ~ in ssh config is replaced before Jsch sees it
Do tilde replacement for values from the ssh config file that are
file names in all cases to make sure that they are already replaced
when Jsch tries to get the values.
Previously, OpenSshConfig did tilde replacement only for the
IdentityFile in the JGit-facing "Host" interface and left the
replacement in the Jsch-facing "Config" interface to Jsch.
But on Windows the JGit notion of what should be used to replace the
tilde differs from Jsch's replacement. Jsch always replaces the tilde
by the value of the system property "user.home", whereas JGit also
considers some environment variables like %HOME%. This can lead to
rather surprising failures as in the case of bug 526175 where
%HOME% != user.home.
Prior to commit 9d24470 (i.e.,prior to JGit 4.9.0) this problem never
occurred because Jsch was completely unaware of the ssh config file
and all host and IdentityFile handling happened exclusively in JGit.
Bug: 526175
Change-Id: I1511699664ffea07cb58ed751cfdb79b15e3a99e Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Shawn Pearce [Sat, 12 Aug 2017 15:25:16 +0000 (08:25 -0700)]
Support symbolic references in ReceiveCommand
Allow creating symbolic references with link, and deleting them or
switching to ObjectId with unlink. How this happens is up to the
individual RefDatabase.
The default implementation detaches RefUpdate if a symbolic reference
is involved, supporting these command instances on RefDirectory.
Unfortunately the packed-refs file does not support storing symrefs,
so atomic transactions involving more than one symref command are
failed early.
Updating InMemoryRepository is deferred until reftable lands, as I
plan to switch InMemoryRepository to use reftable for its internal
storage representation.
Change-Id: Ibcae068b17a2fc6d958f767f402a570ad88d9151 Signed-off-by: Minh Thai <mthai@google.com> Signed-off-by: Terry Parker <tparker@google.com>
projects / jgit.git / log