David Pursehouse [Thu, 30 May 2019 11:31:39 +0000 (20:31 +0900)]
PacketLineIn: Add helper methods to check for END and DELIM
These methods will allow clients to check for END and DELIM without
doing a reference comparison on the String objects, which raises
warnings from Error Prone.
Change-Id: I9e7e59843553ed4488ee8e864033198bbb60d67c Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Ivan Frade [Thu, 23 May 2019 17:39:31 +0000 (10:39 -0700)]
BitmapCalculator: javadoc fixes
Fill out the description of when IOException is thrown.
Also fix a typo in the description for IncorrectObjectTypeException.
Change-Id: I9fafd19d68ddc4fe4e95e8516c2c38484b941a3a Signed-off-by: Ivan Frade <ifrade@google.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Andrey Loskutov [Sat, 25 May 2019 07:53:47 +0000 (09:53 +0200)]
RevWalkUtils: add progress callback to findBranchesReachableFrom
Offer a version of findBranchesReachableFrom method with progress
monitor callback. This is required to allow UI clients to cancel long
running operations and show progress.
Dave Borowitz [Thu, 23 May 2019 20:56:00 +0000 (13:56 -0700)]
SystemReader: Respect passed-in parent when no system config is present
The default implementation of openSystemConfig has special handling for
when the FS returns null from getGitSystemConfig: it should return a
"real" FileBasedConfig instance that never actually tries to load a
file. However, this codepath was not respecting the passed-in parent
config.
Ivan Frade [Tue, 21 May 2019 00:14:22 +0000 (17:14 -0700)]
RevWalk: new method createReachabilityChecker()
Every caller would need to check if bitmaps are available in the repo to
instantiate a reachability checker.
Offer a method to create the reachability checker in the walk: the
caller has already a walk over the repo, and the walk has all the
information required.
This allows us to make the implementation classes package-private.
Change-Id: I355e47486fcd9d55baa7cb5700ec08dcc230eea5 Signed-off-by: Ivan Frade <ifrade@google.com>
Matthias Sohn [Wed, 22 May 2019 17:37:40 +0000 (19:37 +0200)]
Merge branch 'master' into stable-5.4
* master:
ReachabilityCheckers: @since 5.4
Update Orbit to S20190521195709 for 2019-06 M3
Tune max heap size for tests
GPG: check that the key found is a signing key
GPG: use key fingerprint suffix to compare id for signing key
Recognize ReachabilityCheckerTestCase as helper
UploadPack: restore inadvertently deleted line
UploadPack: Use reachability checker to validate non-advertised wants
Bazel: Simplify names of bouncy castle libraries
Convert to lambda or member reference
BitmappedReachabilityChecker: Reachability check using bitmaps
BitmapCalculator: Get the reachability bitmap of a commit
ReachabilityChecker: Default implementation with a RevWalk
Matthias Sohn [Wed, 22 May 2019 12:42:43 +0000 (14:42 +0200)]
Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].
Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test
Decrease it to 768m for
- org.eclipse.jgit.test
[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console
Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Tue, 21 May 2019 16:08:48 +0000 (18:08 +0200)]
GPG: use key fingerprint suffix to compare id for signing key
Check whether the value of the git config user.signingKey is a suffix
of the full fingerprint of the key. This was already used for finding
keys in secring.gpg, but not in pubring.kbx. This mechanism allows a
user to use any unique suffix to identify keys; to avoid needless
collisions it's recommended to use at least the last 16 characters of
the hex representation of the fingerprint, which is the key id.[1]
Jonathan Tan [Tue, 21 May 2019 00:02:07 +0000 (17:02 -0700)]
Recognize ReachabilityCheckerTestCase as helper
4e196faa1b ("ReachabilityChecker: Default implementation with a
RevWalk", 2019-05-15) added ReachabilityCheckerTestCase but did not add
it as a test helper in the corresponding BUILD file, making tests no
longer runnable with Bazel. Resolve this issue.
Change-Id: Iccc00b0d169dbaa137e130ce2ddd1b7669960b52 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Mon, 20 May 2019 23:22:31 +0000 (16:22 -0700)]
UploadPack: restore inadvertently deleted line
In 7b96bd812e ("UploadPack: Use reachability checker to validate
non-advertised wants", 2019-05-16), a "walk.setRetainBody(false);"
statement was inadvertently deleted. (An earlier version of this commit
had this line in another part of the code and a review comment suggested
to move it back here; the line was then deleted from the other part of
the code but not readded.) Restore this line.
Change-Id: I96ff6106ba9e4eef429388c83e898b3363295f69 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Mon, 20 May 2019 20:04:28 +0000 (16:04 -0400)]
Merge changes from topic 'reachability-2'
* changes:
UploadPack: Use reachability checker to validate non-advertised wants
BitmappedReachabilityChecker: Reachability check using bitmaps
BitmapCalculator: Get the reachability bitmap of a commit
ReachabilityChecker: Default implementation with a RevWalk
Ivan Frade [Wed, 8 May 2019 23:43:04 +0000 (16:43 -0700)]
UploadPack: Use reachability checker to validate non-advertised wants
In "Reachable commit" request validators, we need to check that a "want"
in the request, that hasn't been advertised, is reachable from the refs
visible to the user.
Current code has intermixed the translation of ObjectIds to RevCommits
(and its error handling) with the actual walk, with the delegation to
bitmaps in restricted circunstances.
Refactor the code to make it "flatter" and more readable. Move ObjectIds
to RevCommits translation to its own functions. Use the reachability
checker instead of a newly defined walk.
Before the non-advertised wants were validated with bitmaps only if any
"want" refered to an non-commit. Now they will be validated with bitmaps
also if the "wants" refer all to commits.
Change-Id: Ib925a48cde89672b07a88bba4e24d0457546baff Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Mon, 22 Apr 2019 22:38:45 +0000 (15:38 -0700)]
BitmappedReachabilityChecker: Reachability check using bitmaps
The "basic" reachability check walks the graph starting from the tips
marking things as "uninteresting". If the target commit is marked as
"uninteresting" it was reached; it is reachable from those tips.
This requires a lot of walking and can be solved directly with bitmaps.
Most of the time the bitmaps are already calculated or a short walk
away.
This should improve the performance of reachability checks, for example
in Gitiles.
Change-Id: I83d33271f58d95d2dc9ed151967b3eda513c99f7 Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Tue, 7 May 2019 03:39:18 +0000 (20:39 -0700)]
BitmapCalculator: Get the reachability bitmap of a commit
To make reachability checks with bitmaps, we need to get the
reachability bitmap of a commit, which is not always precalculated.
There is already a class returning such bitmap (BitmapWalker) but it
does too much unnecessary work: it calculates ALL reachable objects from
a commit (i.e. including trees and blobs), when for reachability the
commits are just enough.
Introduce BitmapCalculator to get the bitmap of a commit: either because
it is precalculated or generating it with a walk only over commits.
Change-Id: Ibb6c78affe9eeaf1fa362a06daf4fd2d91c1caea Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Mon, 22 Apr 2019 18:37:43 +0000 (11:37 -0700)]
ReachabilityChecker: Default implementation with a RevWalk
It is common to check if a certain commit is reachable from some
starting points. For example gitiles does it to check if a commit
is visible to a user based on its permissions.
Offer this functionality in JGit.
Split the interface as the next commit will introduce an implementation
using bitmap indices.
Change-Id: I0933b305c8d734f7a64502910ff4d9ef4fc92ae1 Signed-off-by: Ivan Frade <ifrade@google.com>
Thomas Wolf [Tue, 15 Jan 2019 18:50:12 +0000 (19:50 +0100)]
Apache MINA sshd client: test reading encrypted ed25519 keys
Add encrypted ed25519 keys in the tests; sshd 2.2.0 can finally
decrypt encrypted new-style OpenSSH key files. (Needs the "unlimited
strength" JCE, which is the default since Java 8u161. On older JREs,
users should install the policy files available from Oracle.)
The "expensive" key added has been generated with OpenSSH's
ssh-keygen -t ed25519 -a 256, i.e., with 256 bcrypt KDF rounds
instead of the default 16. On my machine it takes about 2sec to
decrypt.
Bug: 541703
Change-Id: Id3872ca2fd75d8f009cbc932eeb6357d3d1f267c Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Mon, 14 Jan 2019 16:30:03 +0000 (17:30 +0100)]
Apache MINA sshd client: adapt to sshd 2.2.0
Update target platforms, maven and bazel builds to use sshd 2.2.0.
Adapt internal classes to changed sshd interfaces and remove previous
work-arounds for asking repeatedly for key passwords and for loading
keys lazily; both are now done by sshd.
CQ: 19034
CQ: 19035
Bug: 541425
Change-Id: I85e1df6ebb8a94953a912d9b2b8a7b5bdfbd608a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 2 May 2019 22:23:01 +0000 (00:23 +0200)]
Use integer depth in PackWriter's DepthAwareVisitationPolicy
- ObjectWalk.getTreeDepth() returns int hence there is no need to use
long depths in the lowestDepthVisited map.
- Also fix boxing warnings introduced in 0a15cb3a.
Change-Id: I6d73b6f41d5d20975d02f376c8588e411eaff0ec Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 27 Apr 2019 23:09:55 +0000 (01:09 +0200)]
Merge branch 'stable-5.3'
* stable-5.3:
Prepare 5.3.2-SNAPSHOT builds
JGit v5.3.1.201904271842-r
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I8a8671f7767444a77b809bd66a27d776c8332736 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 27 Apr 2019 22:21:16 +0000 (00:21 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: Ie7e572ac7e346f21fe0c387d7448be168a9c127a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Mon, 22 Apr 2019 15:21:06 +0000 (17:21 +0200)]
Update target platform, maven and bazel builds to JSch 0.1.55
Same version as used in the latest Eclipse platform. Updating the
JGit target platform results in the new JSch version being included
in the JGit feature in the p2 repo, and thus ultimately in the EGit
update site.
CQ: 19588
Bug: 546130
Change-Id: Id7e4318096f6f29dc4573e15542abdf54582af34 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Sun, 10 Mar 2019 17:02:54 +0000 (18:02 +0100)]
Revert 4678f4b and provide another solution for bug 467631
Making gitlinks and folders match in a tree walk was the wrong
approach to fix bug 467631. The problem is that in such a conflict
the tree walk may then not descend into the folder.
Revert the changes to Paths.java and PathsTest.java from commit 4678f4b. Instead test for the problem case from bug 467631 explicitly
in IndexDiff. Add Daniel's test case from bug 545162, and add yet
another test case for DiffEntry.scan() that covers the problem
originally reported in bug 545162.
Bug: 545162
Change-Id: Ie2214c5d5ee32ac6596b621f0f1c7b86d38fa9b7 Also-by: Daniel Veihelmann <daniel.veihelmann@gmail.com> Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Tue, 2 Apr 2019 10:34:26 +0000 (12:34 +0200)]
Apache MINA sshd: make sendKexInit() work also for re-keying
The message delaying for the initial SSH messages (client
identification followed by the initial key exchange request)
was broken. sendKexInit() is _also_ called when a new key exchange
is requested. We inadvertently also re-sent the client identification
at that point, which is wrong and makes the server terminate the
connection.
Re-keying occurs from time to time during an SSH connection depending
on time, the number of messages (packets/blocks) exchanged, or the
amount of data exchanged. The net result was that for large
repositories data-intensive operations failed on the first re-keying.
Change the initial message delay such that the two messages for the
client identification and the initial key exchange can be buffered
individually while the proxy protocol is still in progress. The
AbstractClientProxyConnector can now buffer several commands, which
should also resolve bug 544715.
Bug: 545920
Change-Id: If09ee963a439b39098a0f52a1510237b428df8dd Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn [Sat, 20 Apr 2019 09:16:34 +0000 (11:16 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I76761002eedf360e93d0559942ebc927a40428d6 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 18 Apr 2019 12:43:23 +0000 (14:43 +0200)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Matthias Sohn [Tue, 16 Apr 2019 22:49:03 +0000 (00:49 +0200)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I7984f68833f2d615399296e53cb9a64e5b4ca6ed Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 16 Apr 2019 21:53:21 +0000 (23:53 +0200)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I6ea51dde6608a4163d681aa1ebf710f06da44b21 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 15 Apr 2019 23:04:52 +0000 (01:04 +0200)]
Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I0b4eaa521ebdea83ab18c05915d691c07a575a7d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthew DeVore [Mon, 25 Mar 2019 22:36:35 +0000 (15:36 -0700)]
tree:<depth> should not traverse overly-deep trees
If we are traversing a tree which is too deep, then there is no need to
traverse the children. Skipping children is much faster than traversing
the possibly thousands of objects which are directly or indirectly
referenced by the tree.
Change-Id: I6d68cc1d35da48e3288b9cc80356a281ab36863d Signed-off-by: Matthew DeVore <matvore@gmail.com>
Matthew DeVore [Mon, 18 Mar 2019 14:15:52 +0000 (07:15 -0700)]
Preliminary support for tree:<depth> filter
This is used when fetching, and in particular to populate a partial
clone or a virtual file system cache as the user navigates. With this,
a client can pre-fetch a few directories deeper than only the current
directory.
depth:0 will omit all trees, and is useful if you only want to fetch
the commits of a repository, or fetch just a single tree or blob object.
depth:1 will fetch only the root tree of all commits fetched. depth:2
will fetch the root tree and all blobs and tree objects directly
referenced from it. depth:3 gets one more level, and so on. depth:#
will not filter a blob or tree that is explicitly marked wanted.
Bitmaps are disabled when this filter is used.
This implementation is quite slow because it iterates over all omitted
objects rather than skipping them. This will be addressed in follow-up
commits.
Change-Id: Ic312fee22d60e32cfcad59da56980e90ae2cae6a Signed-off-by: Matthew DeVore <matvore@gmail.com>
Jonathan Nieder [Sun, 14 Apr 2019 15:00:46 +0000 (11:00 -0400)]
Revert "Use try-with-resources in SubmoduleWalk"
This reverts commit 39b0b51b1253f569888db3578b01708a14360b69. Before
that change, SubmoduleWalk.forPath transferred ownership to the caller
on success. Afterward, it returns a closed SubmoduleWalk to the caller,
which does not appear to be intentional.
DepthGenerator marks commits reinteresting for the ones that are
reachable from unshallow commits as it walks over the revisions. Those
unshallow commits won't necessarily be processed first. Because of this,
even if a commit is reachable from unshallow commits, if it's processed
before the uninteresting commits, it will not be processed as
reinteresting and processed as uninteresting. This causes unshallow
git-fetch to be failed.
This changes DepthGenerator to process unshallow commits first
independent to their depth. This makes uninteresting flag carry work
properly.
Keep track of the original cause for a packfile invalidation.
It is needed for the sysadmin to understand if there is a real
underlying filesystem problem and repository corruption or if it is
simply a consequence of a concurrency of Git operations (e.g. repack
or GC).
Change-Id: I06ddda9ec847844ec31616ab6d17f153a5a34e33 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log