DirCacheCheckout has a warning about non-localised string "lfs". Other
classes use org.eclipse.jgit.lfs.lib.Constants but that is not visible
to DirCacheCheckout.
Add a new constant in ConfigConstants and use that in DirCacheCheckout.
Replace existing uses of org.eclipse.jgit.lfs.lib.Constants.LFS with
the new constant, except where it is referring to the folder name.
Change-Id: I0f21b951babff9a2e579d68c4de0c62ee4bc23d4 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Markus Duft [Tue, 5 Dec 2017 10:16:30 +0000 (11:16 +0100)]
LFS: Enable LFS support for the CLI, better error handling
Enable LFS support for the CLI by registering the according filters.
Errors during filter creation must be propagated up the call stack, as a
failure to create a filter should be treated as fatal if the filter is
required.
Change-Id: I3833757839bdda97cd01b6c21c1613d199e2692d Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Markus Duft [Fri, 2 Mar 2018 09:13:05 +0000 (10:13 +0100)]
Cleanup stream usage WRT filters
As it is right now some streams leak out of the filter construct. This
change clarifies responsibilities and fixes stream leaks
Change-Id: Ib9717d43a701a06a502434d64214d13a392de5ab Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Markus Duft [Fri, 2 Mar 2018 09:11:42 +0000 (10:11 +0100)]
LFS: support merge/rebase/cherry-pick/diff/compare with LFS files
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).
Does not implement general support for merge/diff attributes for any
other use case apart from LFS.
Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Markus Duft [Tue, 5 Dec 2017 09:46:43 +0000 (10:46 +0100)]
LFS: Dramatically improve checkout speed with SSH authentication
SSH Authentication is quite expensive (~120ms on localhost against
Gerrit with LFS plugin). The SSH authentication typically also sends a
validity time of the returned token, which allows to re-use it for a
certain time, avoiding the expensive authentication on every download
request. This improves checkout times by large factors depending on the
LFS object amount/sizes.
Also make sure that all instances of Gson used by LFS are configured in
the same way.
Change-Id: I422c94c37021b4322789b3829fa0185e25d683f2 Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Terry Parker [Wed, 28 Feb 2018 23:10:23 +0000 (15:10 -0800)]
Correctly exit fetch negotiation for stateless RPC connections
Commit 9530c10192cf033c021802a3b295b06864654464 (2018-02-11)
"Add a minimum negotiation feature for fetch" made fetch
negotiation cheaper for repos with huge numbers of
references (we are seeing a 15x reduction in maximum fetch
times for chromium/chromium/src on trans-Pacific links).
But it inadvertently broke the handling of stateless RPC
connections, so fix that here.
Change-Id: I0090aa76ffecc55801ebb833ac2e0c933a4a7c54 Signed-off-by: Terry Parker <tparker@google.com>
Markus Duft [Fri, 10 Nov 2017 10:10:28 +0000 (11:10 +0100)]
LFS: pre-push upload support
If JGit built in LFS support is enabled for the current repository (or
user/system), any existing pre-push hook will cause an exception for the
time beeing, as only a single pre-push hook is supported.
Thus either native pre-push hooks OR JGit built-in LFS support may be
enabled currently, but not both.
Change-Id: Ie7d2b90e26e948d9cca3d05a7a19489488c75895 Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Wed, 21 Feb 2018 03:40:47 +0000 (12:40 +0900)]
Add ObjectIdSerializer
This is based on the ObjectIdSerialization class written by Shawn Pearce
for the Gerrit Code Review project in 2009 [1]. As mentioned in the
commit message there, it should be part of core JGit.
This implementation is slightly different to Shawn's version. Rather
than having separate methods for null/non-null ids, single methods are
implemented with @Nullable annotations.
Change-Id: I7599cf8bd1ecd546e2252783d6d672eb76804060 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Marc Strapetz [Fri, 23 Feb 2018 12:34:23 +0000 (13:34 +0100)]
Fix processing of gitignore negations
Processing of negated rules, like !bin/ was not working correctly: they
were interpreted too broad, resulting in unexpected untracked files
which should actually be ignored
Bug: 409664
Change-Id: I0a422fd6607941461bf2175c9105a0311612efa0 Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Dave Borowitz [Thu, 11 Jan 2018 18:14:02 +0000 (13:14 -0500)]
DfsReaderIoStats: Clarify that "blocks" are really cache misses
The previous wording was ambiguous as to whether these were blocks
requested from the cache (hits + misses) or read from underlying storage
(misses only).
They are in fact recording only misses:
Accumulator#{readBlock,readBlockBytes,readBlockMicros} are only
incremented from BlockBasedFile#readOneBlock, which is only called from
the cache miss path in DfsBlockCache#getOrLoad (line 391).
David Pursehouse [Thu, 22 Feb 2018 05:37:48 +0000 (00:37 -0500)]
Merge changes Id616611e,Ib6509e81,I52f5d3f2
* changes:
RepoCommand: generate relative submodule URLs from absolute URLs.
RepoCommand: don't record new commit if tree did not change
RepoCommand: persist unreadable submodules in .gitmodules
Shawn Pearce [Wed, 9 Aug 2017 12:22:14 +0000 (05:22 -0700)]
dfs: Try to avoid searching UNREACHABLE_GARBAGE during packing
If an object can be found in a non-garbage pack, favor that pack over
paging in the garbage pack's idx and pack content.
Only fall back to garbage packs if an object cannot be found and there
are garbage packs present in the repository. This fallback is
required to correct race conditions during GC.
Han-Wen Nienhuys [Wed, 14 Feb 2018 17:23:03 +0000 (18:23 +0100)]
RepoCommand: persist unreadable submodules in .gitmodules
In cases where a manifest file mixes different remotes, a Gerrit
server process may not have access to all remotes, and won't be able
to produce a full submodule tree.
Preserving this information in .gitmodules will let downstream clients
reconstruct the full tree.
Markus Duft [Fri, 7 Oct 2016 10:39:45 +0000 (12:39 +0200)]
LFS: Add remote download to SmudgeFilter
Transfer data in chunks of 8k Transferring data byte per byte is slow,
running checkout with CleanFilter on a 2.9MB file takes 20 seconds.
Using a buffer of 8k shrinks this time to 70ms.
Also register the filter commands in a way that the native GIT LFS can
be used alongside with JGit.
Implements auto-discovery of LFS server URL when cloning from a Gerrit
LFS server.
Change-Id: I452a5aa177dcb346d92af08b27c2e35200f246fd Also-by: Christian Halstrick <christian.halstrick@sap.com> Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Thomas Wolf [Wed, 14 Feb 2018 12:39:28 +0000 (13:39 +0100)]
Fix ssh host name handling for Jsch
If we give Jsch access to the ssh config file, we must _not_ resolve
the host name from the alias. Instead we must give the alias (i.e.,
the host name as is in the URI) to Jsch, so that it finds the same
ssh config entry.
Otherwise if the hostname in the URI, which is taken as an alias in
ssh config ("Host" line), is unequal to the "Hostname" line, and
there happens to be another ssh config entry with that translated
host name as alias, Jsch will pick up that second entry, and we end
up with a strange mixture of both.
Add tests for this case.
Bug: 531118
Change-Id: I249d8c073b0190ed110a69dca5b9be2a749822c3 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Wed, 14 Feb 2018 12:05:17 +0000 (13:05 +0100)]
Jsch overrides the port in the URI with the one in ~/.ssh/config
Jsch unconditionally overwrites the port from the ssh config
file (if a port is specified there), even if the URI explicitly does
give a different port.
Fix this, and add tests.
Change-Id: I7b014543c7ece26270e366db39d7647f82d64f0d Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
David Pursehouse [Wed, 14 Feb 2018 01:13:46 +0000 (10:13 +0900)]
InitCommand: Don't leave Repository open after Git is closed
The InitCommand returns a Git that is instantiated with the newly
created Repository, but the Repository is not closed with the Git
resulting in resource leaks.
Create the Git with `closeRepo` set to true, such that the Repository
is also closed when the Git is closed.
Adjust the tests to use try-with-resource on the Git instance.
Change-Id: Ib26e7428c7d8840956d1edb09e53b93e23e6fe5a Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Terry Parker [Sun, 11 Feb 2018 22:37:22 +0000 (14:37 -0800)]
Add a minimum negotiation feature for fetch
Android an Chrome have several repos with >300k refs. We sometimes see
negotiations of >100k rounds. This change provides a "minimal negotiation"
feature on the client side that limits how many "have" lines the client
sends. The client extracts the current SHA-1 values for the refs in its
wants set, and terminates negotiation early when all of those values have
been sent as haves. If a new branch is being fetched then that set will
be empty and the client will terminate after current default minimum
of two rounds.
This feature is gated behind a "fetch.useminimalnegotiation" configuration
flag, which defaults to false.
Change-Id: Ib12b095cac76a59da6e8f72773c4129e3b32ff2b Signed-off-by: Terry Parker <tparker@google.com>
Terry Parker [Sun, 11 Feb 2018 21:29:38 +0000 (13:29 -0800)]
Add negotiation statistics to PackStatistics
Add fetch statistics for the counts of advertised refs, wants and haves.
Also add the duration in milliseconds for the negotiation phase. For
non-bidirectional transports like HTTP, this is the time for the final
round that sends the pack back to the user.
Change-Id: I1af7ffd3cb7b62182340682e2a243691ea24ec2e Signed-off-by: Terry Parker <tparker@google.com>
Fix CleanCommand not to throw FileNotFoundExceptions
When CleanCommand is collecting the files and folders to be deleted
it may happen that the list of directories contains obsolete entries.
E.g. a folder and its parent folder may be in the list. Only the
parent folder would be sufficient.
This was a reason for hitting FileNotFoundExceptions when finally
trying to delete the files and folders. Improve CleanCommand
to ignore files to be deleted which are already gone.
Bug: 514434
Change-Id: I10caa01bfb9cec5967dfdaea50c6e4a713eeeabd Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.
Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.
David Ostrovsky [Wed, 13 Dec 2017 22:41:45 +0000 (23:41 +0100)]
Bazel: Support building with Java 9
Bazel@HEAD supports Java 9.
The current code has one single issue with Java 9 compliance: the usage
of javax.xml.bind.DatatypeConverter class for printHexBinary() method.
This class is not available on Java 9. One alternative is to use guava
library. Something similar was done here: [1]. But unlike the case with
checkstyle library, JGit currently doesn't use guava. Instead, we add
java.xml.bind module with --add-modules compiler option.
To build (or test) with Java 9, build custom bazel version and issue:
The Files.exists method has noticeably poor performance in JDK 8 and can
slow an application significantly when used to check files that do not
actually exist. The same goes for Files.notExists, Files.isDirectory and
Files.isRegularFile [1].
projects / jgit.git / log