Markus Duft [Tue, 5 Dec 2017 10:16:30 +0000 (11:16 +0100)]
LFS: Enable LFS support for the CLI, better error handling
Enable LFS support for the CLI by registering the according filters.
Errors during filter creation must be propagated up the call stack, as a
failure to create a filter should be treated as fatal if the filter is
required.
Change-Id: I3833757839bdda97cd01b6c21c1613d199e2692d Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Markus Duft [Fri, 2 Mar 2018 09:13:05 +0000 (10:13 +0100)]
Cleanup stream usage WRT filters
As it is right now some streams leak out of the filter construct. This
change clarifies responsibilities and fixes stream leaks
Change-Id: Ib9717d43a701a06a502434d64214d13a392de5ab Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Markus Duft [Fri, 2 Mar 2018 09:11:42 +0000 (10:11 +0100)]
LFS: support merge/rebase/cherry-pick/diff/compare with LFS files
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).
Does not implement general support for merge/diff attributes for any
other use case apart from LFS.
Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Markus Duft [Tue, 5 Dec 2017 09:46:43 +0000 (10:46 +0100)]
LFS: Dramatically improve checkout speed with SSH authentication
SSH Authentication is quite expensive (~120ms on localhost against
Gerrit with LFS plugin). The SSH authentication typically also sends a
validity time of the returned token, which allows to re-use it for a
certain time, avoiding the expensive authentication on every download
request. This improves checkout times by large factors depending on the
LFS object amount/sizes.
Also make sure that all instances of Gson used by LFS are configured in
the same way.
Change-Id: I422c94c37021b4322789b3829fa0185e25d683f2 Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Terry Parker [Wed, 28 Feb 2018 23:10:23 +0000 (15:10 -0800)]
Correctly exit fetch negotiation for stateless RPC connections
Commit 9530c10192cf033c021802a3b295b06864654464 (2018-02-11)
"Add a minimum negotiation feature for fetch" made fetch
negotiation cheaper for repos with huge numbers of
references (we are seeing a 15x reduction in maximum fetch
times for chromium/chromium/src on trans-Pacific links).
But it inadvertently broke the handling of stateless RPC
connections, so fix that here.
Change-Id: I0090aa76ffecc55801ebb833ac2e0c933a4a7c54 Signed-off-by: Terry Parker <tparker@google.com>
Markus Duft [Fri, 10 Nov 2017 10:10:28 +0000 (11:10 +0100)]
LFS: pre-push upload support
If JGit built in LFS support is enabled for the current repository (or
user/system), any existing pre-push hook will cause an exception for the
time beeing, as only a single pre-push hook is supported.
Thus either native pre-push hooks OR JGit built-in LFS support may be
enabled currently, but not both.
Change-Id: Ie7d2b90e26e948d9cca3d05a7a19489488c75895 Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Wed, 21 Feb 2018 03:40:47 +0000 (12:40 +0900)]
Add ObjectIdSerializer
This is based on the ObjectIdSerialization class written by Shawn Pearce
for the Gerrit Code Review project in 2009 [1]. As mentioned in the
commit message there, it should be part of core JGit.
This implementation is slightly different to Shawn's version. Rather
than having separate methods for null/non-null ids, single methods are
implemented with @Nullable annotations.
Change-Id: I7599cf8bd1ecd546e2252783d6d672eb76804060 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Marc Strapetz [Fri, 23 Feb 2018 12:34:23 +0000 (13:34 +0100)]
Fix processing of gitignore negations
Processing of negated rules, like !bin/ was not working correctly: they
were interpreted too broad, resulting in unexpected untracked files
which should actually be ignored
Bug: 409664
Change-Id: I0a422fd6607941461bf2175c9105a0311612efa0 Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Dave Borowitz [Thu, 11 Jan 2018 18:14:02 +0000 (13:14 -0500)]
DfsReaderIoStats: Clarify that "blocks" are really cache misses
The previous wording was ambiguous as to whether these were blocks
requested from the cache (hits + misses) or read from underlying storage
(misses only).
They are in fact recording only misses:
Accumulator#{readBlock,readBlockBytes,readBlockMicros} are only
incremented from BlockBasedFile#readOneBlock, which is only called from
the cache miss path in DfsBlockCache#getOrLoad (line 391).
David Pursehouse [Thu, 22 Feb 2018 05:37:48 +0000 (00:37 -0500)]
Merge changes Id616611e,Ib6509e81,I52f5d3f2
* changes:
RepoCommand: generate relative submodule URLs from absolute URLs.
RepoCommand: don't record new commit if tree did not change
RepoCommand: persist unreadable submodules in .gitmodules
Shawn Pearce [Wed, 9 Aug 2017 12:22:14 +0000 (05:22 -0700)]
dfs: Try to avoid searching UNREACHABLE_GARBAGE during packing
If an object can be found in a non-garbage pack, favor that pack over
paging in the garbage pack's idx and pack content.
Only fall back to garbage packs if an object cannot be found and there
are garbage packs present in the repository. This fallback is
required to correct race conditions during GC.
Han-Wen Nienhuys [Wed, 14 Feb 2018 17:23:03 +0000 (18:23 +0100)]
RepoCommand: persist unreadable submodules in .gitmodules
In cases where a manifest file mixes different remotes, a Gerrit
server process may not have access to all remotes, and won't be able
to produce a full submodule tree.
Preserving this information in .gitmodules will let downstream clients
reconstruct the full tree.
Markus Duft [Fri, 7 Oct 2016 10:39:45 +0000 (12:39 +0200)]
LFS: Add remote download to SmudgeFilter
Transfer data in chunks of 8k Transferring data byte per byte is slow,
running checkout with CleanFilter on a 2.9MB file takes 20 seconds.
Using a buffer of 8k shrinks this time to 70ms.
Also register the filter commands in a way that the native GIT LFS can
be used alongside with JGit.
Implements auto-discovery of LFS server URL when cloning from a Gerrit
LFS server.
Change-Id: I452a5aa177dcb346d92af08b27c2e35200f246fd Also-by: Christian Halstrick <christian.halstrick@sap.com> Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Thomas Wolf [Wed, 14 Feb 2018 12:39:28 +0000 (13:39 +0100)]
Fix ssh host name handling for Jsch
If we give Jsch access to the ssh config file, we must _not_ resolve
the host name from the alias. Instead we must give the alias (i.e.,
the host name as is in the URI) to Jsch, so that it finds the same
ssh config entry.
Otherwise if the hostname in the URI, which is taken as an alias in
ssh config ("Host" line), is unequal to the "Hostname" line, and
there happens to be another ssh config entry with that translated
host name as alias, Jsch will pick up that second entry, and we end
up with a strange mixture of both.
Add tests for this case.
Bug: 531118
Change-Id: I249d8c073b0190ed110a69dca5b9be2a749822c3 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Wed, 14 Feb 2018 12:05:17 +0000 (13:05 +0100)]
Jsch overrides the port in the URI with the one in ~/.ssh/config
Jsch unconditionally overwrites the port from the ssh config
file (if a port is specified there), even if the URI explicitly does
give a different port.
Fix this, and add tests.
Change-Id: I7b014543c7ece26270e366db39d7647f82d64f0d Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
David Pursehouse [Wed, 14 Feb 2018 01:13:46 +0000 (10:13 +0900)]
InitCommand: Don't leave Repository open after Git is closed
The InitCommand returns a Git that is instantiated with the newly
created Repository, but the Repository is not closed with the Git
resulting in resource leaks.
Create the Git with `closeRepo` set to true, such that the Repository
is also closed when the Git is closed.
Adjust the tests to use try-with-resource on the Git instance.
Change-Id: Ib26e7428c7d8840956d1edb09e53b93e23e6fe5a Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Terry Parker [Sun, 11 Feb 2018 22:37:22 +0000 (14:37 -0800)]
Add a minimum negotiation feature for fetch
Android an Chrome have several repos with >300k refs. We sometimes see
negotiations of >100k rounds. This change provides a "minimal negotiation"
feature on the client side that limits how many "have" lines the client
sends. The client extracts the current SHA-1 values for the refs in its
wants set, and terminates negotiation early when all of those values have
been sent as haves. If a new branch is being fetched then that set will
be empty and the client will terminate after current default minimum
of two rounds.
This feature is gated behind a "fetch.useminimalnegotiation" configuration
flag, which defaults to false.
Change-Id: Ib12b095cac76a59da6e8f72773c4129e3b32ff2b Signed-off-by: Terry Parker <tparker@google.com>
Terry Parker [Sun, 11 Feb 2018 21:29:38 +0000 (13:29 -0800)]
Add negotiation statistics to PackStatistics
Add fetch statistics for the counts of advertised refs, wants and haves.
Also add the duration in milliseconds for the negotiation phase. For
non-bidirectional transports like HTTP, this is the time for the final
round that sends the pack back to the user.
Change-Id: I1af7ffd3cb7b62182340682e2a243691ea24ec2e Signed-off-by: Terry Parker <tparker@google.com>
Fix CleanCommand not to throw FileNotFoundExceptions
When CleanCommand is collecting the files and folders to be deleted
it may happen that the list of directories contains obsolete entries.
E.g. a folder and its parent folder may be in the list. Only the
parent folder would be sufficient.
This was a reason for hitting FileNotFoundExceptions when finally
trying to delete the files and folders. Improve CleanCommand
to ignore files to be deleted which are already gone.
Bug: 514434
Change-Id: I10caa01bfb9cec5967dfdaea50c6e4a713eeeabd Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.
Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.
David Ostrovsky [Wed, 13 Dec 2017 22:41:45 +0000 (23:41 +0100)]
Bazel: Support building with Java 9
Bazel@HEAD supports Java 9.
The current code has one single issue with Java 9 compliance: the usage
of javax.xml.bind.DatatypeConverter class for printHexBinary() method.
This class is not available on Java 9. One alternative is to use guava
library. Something similar was done here: [1]. But unlike the case with
checkstyle library, JGit currently doesn't use guava. Instead, we add
java.xml.bind module with --add-modules compiler option.
To build (or test) with Java 9, build custom bazel version and issue:
The Files.exists method has noticeably poor performance in JDK 8 and can
slow an application significantly when used to check files that do not
actually exist. The same goes for Files.notExists, Files.isDirectory and
Files.isRegularFile [1].
David Turner [Thu, 12 Oct 2017 22:43:15 +0000 (18:43 -0400)]
Basic submodule merge handling
This doesn't handle the really hard thing, which is merging spurious
conflicts inside .gitmodules files. That's OK: git.git doesn't
either. Users can resolve the conflict themselves and then commit
the merge.
Previously, jgit would crash when attempting to merge conflicting
submodule changes. Even if there was no conflict, after a merge which
adds submodules, the repository would have been missing empty
directories for newly-added submodules.
This patch fixes the crash, and adds the empty directories where
necessary. It ensures that the index is in a conflicted state when
submodule changes conflict.
Reported-by: Alexey Korobkov
Bug: 494551
Change-Id: I79db6798c2bdcc1159b5b2589b02da198dc906a1 Signed-off-by: David Turner <dturner@twosigma.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Fri, 2 Feb 2018 07:16:13 +0000 (08:16 +0100)]
Fix index blob for merges with CRLF translations
Commit fc7d407 corrected line endings for working tree files resulting
from merges when CRLF translations are to be done. However, that also
resulted in the file content being put as-is into the index, which is
wrong. The index must contain the file content with reverse CRLF
translations applied.
With core.autocrlf=true, the working tree file should have CR-LF, but
the index blob must still contain only LF.
Fix this oversight and apply the inverse translation when updating the
index, similar to what is done in AddCommand.
Bug: 499615
Change-Id: I3a33931318bdb580b2390f3450f91ea8f258a6a4 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
David Pursehouse [Wed, 24 Jan 2018 01:15:56 +0000 (10:15 +0900)]
Fix version of slf4j in manifest files
Consistently require 1.7.0. We ship 1.7.2 with our p2 repository but
there is no need to require 1.7.2 since it should be API compatible with
1.7.0.
Change-Id: I8467bb14316cb24daa79e89275332107d2716190 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Wed, 24 Jan 2018 01:09:32 +0000 (10:09 +0900)]
Update gson version in org.eclipse.jgit.lfs.server manifest
Commit f3bb0e2 upgraded gson to 2.8.2 but did not update the LFS
server manifest file to require 2.8. Also accept 2.8.0 which should be
API compatible with 2.8.2 which is the version we ship with the p2
repository.
Change-Id: I734496c4826739f9bc6a790b9d2c70e549c0e559 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Thu, 1 Feb 2018 20:58:23 +0000 (21:58 +0100)]
Honor CRLF settings when writing merge results
Merges are performed using the raw text as stored in the git
repository. When we write the merge result, we must apply the
correct CRLF settings. Otherwise the line endings in the result
will be wrong.
Bug: 499615
Change-Id: I37a9b987e9404c97645d2720cd1c7c04c076a96b Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Sat, 27 Jan 2018 15:06:15 +0000 (16:06 +0100)]
Minor improvements in git config file inclusions
* Section and key names in git config files are case-insensitive.
* If an include directive is invalid, include the line in the
exception message.
* If inclusion of the included file fails, put the file name into
the exception message so that the user knows in which file the
problem is.
Change-Id: If920943af7ff93f5321b3d315dfec5222091256c Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Markus Duft [Tue, 5 Dec 2017 08:30:48 +0000 (09:30 +0100)]
Progress reporting for checkout
The reason for the change is LFS: when using a lot of LFS files,
checkout can take quite some time on larger repositories. To avoid
"hanging" UI, provide progress reporting.
Also implement (partial) progress reporting for cherry-pick, reset,
revert which are using checkout internally.
The feature is also useful without LFS, so it is independent of it.
Change-Id: I021e764241f3c107eaf2771f6b5785245b146b42 Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Mon, 15 Jan 2018 00:45:38 +0000 (09:45 +0900)]
Upgrade commons-compress to 1.15
Also update xz to 1.6 as this is the version commons-compress 1.15
wants. Since xz 1.6 is an optional dependency for commons-compress we
need to add a non-optional dependency to xz for
org.eclipse.jgit.pgm.test since one of the tests explicitly requires xz.
Related change adding commons-compress to Orbit:
https://git.eclipse.org/r/#/c/115366/
CQ: 15356
CQ: 15360
Change-Id: I0d61c71bc541cc30464a0fff93775b079dd3ba88 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Reported-by: Thomas Wolf <thomas.wolf@paranor.ch>
Change-Id: I96869f80dd11ee238911706581b224bca4fb12cd Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Zhen Chen [Wed, 17 Jan 2018 23:14:43 +0000 (15:14 -0800)]
Skip broken symbolic ref in DfsReftableDatabase
We skipped the broken symbolic reference in other implementation like
DfsRefDatabase, RefDirectory. The broken symbolic reference may cause
NPE when caller forget to have a null check against the object id before
calling parse it.
Thomas Wolf [Fri, 12 Jan 2018 11:51:22 +0000 (12:51 +0100)]
Revert handling of ssh IdentityFile to pre-4.9 behavior
Jsch caches keys (aka identities) specified in ~/.ssh/config via
IndentityFile only for the current Jsch Session. This results in
multiple password prompts for successive sessions.
Do the handling of IdentityFile exclusively in JGit, as it was before
4.9. JGit uses different Jsch instances per host and caches the
IdentityFile there, allowing it to be re-used in different sessions
for the same host.
* Add comments to explain this.
* Move the JschBugFixingConfig from OpenSshConfig to
JschConfigSessionFactory to have all these Jsch work-arounds
in one place.
* Make that config hide the IdentityFile config from Jsch to avoid
that Jsch overrides the JGit behavior.
Bug: 529173
Change-Id: Ib36c34a2921ba736adeb64de71323c2b91151613 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Dave Borowitz [Fri, 5 Jan 2018 18:02:47 +0000 (13:02 -0500)]
Ensure DirectoryStream is closed promptly
From the javadoc for Files.list:
"The returned stream encapsulates a DirectoryStream. If timely disposal
of file system resources is required, the try-with-resources construct
should be used to ensure that the stream's close method is invoked
after the stream operations are completed."
This is the only call to Files#newDirectoryStream that is not already in
a try-with-resources.
Matthias Sohn [Wed, 3 Jan 2018 00:09:57 +0000 (01:09 +0100)]
Add com.jcraft.jzlib 1.1.1
com.jcraft.jsch requires com.jcraft.jzlib to provide optional zlib
packet compression support. Add this library so that jgit can handle
packet compression.
CQ: 15292
Bug: 529129
Change-Id: I0297bd0488753547a5f5d47dbf0db508a79dd265 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Tim Hosey [Thu, 4 Jan 2018 01:58:05 +0000 (02:58 +0100)]
Fix file handle leak in FetchCommand#fetchSubmodules
The private fetchSubmodules method in the FetchCommand class creates a
Repository instance for each submodule being fetched, but never calls
closes on it.
This leads to the leaking of file handles.
Bug: 526494
Change-Id: I7070388b8b62063d9d5cd31afae3015a8388044f Signed-off-by: Tim Hosey <timhoseydev@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log