Matthias Sohn [Wed, 22 May 2019 12:42:43 +0000 (14:42 +0200)]
Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].
Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test
Decrease it to 768m for
- org.eclipse.jgit.test
[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console
Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Tue, 21 May 2019 16:08:48 +0000 (18:08 +0200)]
GPG: use key fingerprint suffix to compare id for signing key
Check whether the value of the git config user.signingKey is a suffix
of the full fingerprint of the key. This was already used for finding
keys in secring.gpg, but not in pubring.kbx. This mechanism allows a
user to use any unique suffix to identify keys; to avoid needless
collisions it's recommended to use at least the last 16 characters of
the hex representation of the fingerprint, which is the key id.[1]
Jonathan Tan [Tue, 21 May 2019 00:02:07 +0000 (17:02 -0700)]
Recognize ReachabilityCheckerTestCase as helper
4e196faa1b ("ReachabilityChecker: Default implementation with a
RevWalk", 2019-05-15) added ReachabilityCheckerTestCase but did not add
it as a test helper in the corresponding BUILD file, making tests no
longer runnable with Bazel. Resolve this issue.
Change-Id: Iccc00b0d169dbaa137e130ce2ddd1b7669960b52 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Mon, 20 May 2019 23:22:31 +0000 (16:22 -0700)]
UploadPack: restore inadvertently deleted line
In 7b96bd812e ("UploadPack: Use reachability checker to validate
non-advertised wants", 2019-05-16), a "walk.setRetainBody(false);"
statement was inadvertently deleted. (An earlier version of this commit
had this line in another part of the code and a review comment suggested
to move it back here; the line was then deleted from the other part of
the code but not readded.) Restore this line.
Change-Id: I96ff6106ba9e4eef429388c83e898b3363295f69 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Mon, 20 May 2019 20:04:28 +0000 (16:04 -0400)]
Merge changes from topic 'reachability-2'
* changes:
UploadPack: Use reachability checker to validate non-advertised wants
BitmappedReachabilityChecker: Reachability check using bitmaps
BitmapCalculator: Get the reachability bitmap of a commit
ReachabilityChecker: Default implementation with a RevWalk
Ivan Frade [Wed, 8 May 2019 23:43:04 +0000 (16:43 -0700)]
UploadPack: Use reachability checker to validate non-advertised wants
In "Reachable commit" request validators, we need to check that a "want"
in the request, that hasn't been advertised, is reachable from the refs
visible to the user.
Current code has intermixed the translation of ObjectIds to RevCommits
(and its error handling) with the actual walk, with the delegation to
bitmaps in restricted circunstances.
Refactor the code to make it "flatter" and more readable. Move ObjectIds
to RevCommits translation to its own functions. Use the reachability
checker instead of a newly defined walk.
Before the non-advertised wants were validated with bitmaps only if any
"want" refered to an non-commit. Now they will be validated with bitmaps
also if the "wants" refer all to commits.
Change-Id: Ib925a48cde89672b07a88bba4e24d0457546baff Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Mon, 22 Apr 2019 22:38:45 +0000 (15:38 -0700)]
BitmappedReachabilityChecker: Reachability check using bitmaps
The "basic" reachability check walks the graph starting from the tips
marking things as "uninteresting". If the target commit is marked as
"uninteresting" it was reached; it is reachable from those tips.
This requires a lot of walking and can be solved directly with bitmaps.
Most of the time the bitmaps are already calculated or a short walk
away.
This should improve the performance of reachability checks, for example
in Gitiles.
Change-Id: I83d33271f58d95d2dc9ed151967b3eda513c99f7 Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Tue, 7 May 2019 03:39:18 +0000 (20:39 -0700)]
BitmapCalculator: Get the reachability bitmap of a commit
To make reachability checks with bitmaps, we need to get the
reachability bitmap of a commit, which is not always precalculated.
There is already a class returning such bitmap (BitmapWalker) but it
does too much unnecessary work: it calculates ALL reachable objects from
a commit (i.e. including trees and blobs), when for reachability the
commits are just enough.
Introduce BitmapCalculator to get the bitmap of a commit: either because
it is precalculated or generating it with a walk only over commits.
Change-Id: Ibb6c78affe9eeaf1fa362a06daf4fd2d91c1caea Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Mon, 22 Apr 2019 18:37:43 +0000 (11:37 -0700)]
ReachabilityChecker: Default implementation with a RevWalk
It is common to check if a certain commit is reachable from some
starting points. For example gitiles does it to check if a commit
is visible to a user based on its permissions.
Offer this functionality in JGit.
Split the interface as the next commit will introduce an implementation
using bitmap indices.
Change-Id: I0933b305c8d734f7a64502910ff4d9ef4fc92ae1 Signed-off-by: Ivan Frade <ifrade@google.com>
Thomas Wolf [Tue, 15 Jan 2019 18:50:12 +0000 (19:50 +0100)]
Apache MINA sshd client: test reading encrypted ed25519 keys
Add encrypted ed25519 keys in the tests; sshd 2.2.0 can finally
decrypt encrypted new-style OpenSSH key files. (Needs the "unlimited
strength" JCE, which is the default since Java 8u161. On older JREs,
users should install the policy files available from Oracle.)
The "expensive" key added has been generated with OpenSSH's
ssh-keygen -t ed25519 -a 256, i.e., with 256 bcrypt KDF rounds
instead of the default 16. On my machine it takes about 2sec to
decrypt.
Bug: 541703
Change-Id: Id3872ca2fd75d8f009cbc932eeb6357d3d1f267c Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Mon, 14 Jan 2019 16:30:03 +0000 (17:30 +0100)]
Apache MINA sshd client: adapt to sshd 2.2.0
Update target platforms, maven and bazel builds to use sshd 2.2.0.
Adapt internal classes to changed sshd interfaces and remove previous
work-arounds for asking repeatedly for key passwords and for loading
keys lazily; both are now done by sshd.
CQ: 19034
CQ: 19035
Bug: 541425
Change-Id: I85e1df6ebb8a94953a912d9b2b8a7b5bdfbd608a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 2 May 2019 22:23:01 +0000 (00:23 +0200)]
Use integer depth in PackWriter's DepthAwareVisitationPolicy
- ObjectWalk.getTreeDepth() returns int hence there is no need to use
long depths in the lowestDepthVisited map.
- Also fix boxing warnings introduced in 0a15cb3a.
Change-Id: I6d73b6f41d5d20975d02f376c8588e411eaff0ec Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 27 Apr 2019 23:09:55 +0000 (01:09 +0200)]
Merge branch 'stable-5.3'
* stable-5.3:
Prepare 5.3.2-SNAPSHOT builds
JGit v5.3.1.201904271842-r
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I8a8671f7767444a77b809bd66a27d776c8332736 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 27 Apr 2019 22:21:16 +0000 (00:21 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: Ie7e572ac7e346f21fe0c387d7448be168a9c127a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Mon, 22 Apr 2019 15:21:06 +0000 (17:21 +0200)]
Update target platform, maven and bazel builds to JSch 0.1.55
Same version as used in the latest Eclipse platform. Updating the
JGit target platform results in the new JSch version being included
in the JGit feature in the p2 repo, and thus ultimately in the EGit
update site.
CQ: 19588
Bug: 546130
Change-Id: Id7e4318096f6f29dc4573e15542abdf54582af34 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Sun, 10 Mar 2019 17:02:54 +0000 (18:02 +0100)]
Revert 4678f4b and provide another solution for bug 467631
Making gitlinks and folders match in a tree walk was the wrong
approach to fix bug 467631. The problem is that in such a conflict
the tree walk may then not descend into the folder.
Revert the changes to Paths.java and PathsTest.java from commit 4678f4b. Instead test for the problem case from bug 467631 explicitly
in IndexDiff. Add Daniel's test case from bug 545162, and add yet
another test case for DiffEntry.scan() that covers the problem
originally reported in bug 545162.
Bug: 545162
Change-Id: Ie2214c5d5ee32ac6596b621f0f1c7b86d38fa9b7 Also-by: Daniel Veihelmann <daniel.veihelmann@gmail.com> Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Tue, 2 Apr 2019 10:34:26 +0000 (12:34 +0200)]
Apache MINA sshd: make sendKexInit() work also for re-keying
The message delaying for the initial SSH messages (client
identification followed by the initial key exchange request)
was broken. sendKexInit() is _also_ called when a new key exchange
is requested. We inadvertently also re-sent the client identification
at that point, which is wrong and makes the server terminate the
connection.
Re-keying occurs from time to time during an SSH connection depending
on time, the number of messages (packets/blocks) exchanged, or the
amount of data exchanged. The net result was that for large
repositories data-intensive operations failed on the first re-keying.
Change the initial message delay such that the two messages for the
client identification and the initial key exchange can be buffered
individually while the proxy protocol is still in progress. The
AbstractClientProxyConnector can now buffer several commands, which
should also resolve bug 544715.
Bug: 545920
Change-Id: If09ee963a439b39098a0f52a1510237b428df8dd Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn [Sat, 20 Apr 2019 09:16:34 +0000 (11:16 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I76761002eedf360e93d0559942ebc927a40428d6 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 18 Apr 2019 12:43:23 +0000 (14:43 +0200)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Matthias Sohn [Tue, 16 Apr 2019 22:49:03 +0000 (00:49 +0200)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I7984f68833f2d615399296e53cb9a64e5b4ca6ed Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 16 Apr 2019 21:53:21 +0000 (23:53 +0200)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I6ea51dde6608a4163d681aa1ebf710f06da44b21 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 15 Apr 2019 23:04:52 +0000 (01:04 +0200)]
Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I0b4eaa521ebdea83ab18c05915d691c07a575a7d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthew DeVore [Mon, 25 Mar 2019 22:36:35 +0000 (15:36 -0700)]
tree:<depth> should not traverse overly-deep trees
If we are traversing a tree which is too deep, then there is no need to
traverse the children. Skipping children is much faster than traversing
the possibly thousands of objects which are directly or indirectly
referenced by the tree.
Change-Id: I6d68cc1d35da48e3288b9cc80356a281ab36863d Signed-off-by: Matthew DeVore <matvore@gmail.com>
Matthew DeVore [Mon, 18 Mar 2019 14:15:52 +0000 (07:15 -0700)]
Preliminary support for tree:<depth> filter
This is used when fetching, and in particular to populate a partial
clone or a virtual file system cache as the user navigates. With this,
a client can pre-fetch a few directories deeper than only the current
directory.
depth:0 will omit all trees, and is useful if you only want to fetch
the commits of a repository, or fetch just a single tree or blob object.
depth:1 will fetch only the root tree of all commits fetched. depth:2
will fetch the root tree and all blobs and tree objects directly
referenced from it. depth:3 gets one more level, and so on. depth:#
will not filter a blob or tree that is explicitly marked wanted.
Bitmaps are disabled when this filter is used.
This implementation is quite slow because it iterates over all omitted
objects rather than skipping them. This will be addressed in follow-up
commits.
Change-Id: Ic312fee22d60e32cfcad59da56980e90ae2cae6a Signed-off-by: Matthew DeVore <matvore@gmail.com>
Jonathan Nieder [Sun, 14 Apr 2019 15:00:46 +0000 (11:00 -0400)]
Revert "Use try-with-resources in SubmoduleWalk"
This reverts commit 39b0b51b1253f569888db3578b01708a14360b69. Before
that change, SubmoduleWalk.forPath transferred ownership to the caller
on success. Afterward, it returns a closed SubmoduleWalk to the caller,
which does not appear to be intentional.
DepthGenerator marks commits reinteresting for the ones that are
reachable from unshallow commits as it walks over the revisions. Those
unshallow commits won't necessarily be processed first. Because of this,
even if a commit is reachable from unshallow commits, if it's processed
before the uninteresting commits, it will not be processed as
reinteresting and processed as uninteresting. This causes unshallow
git-fetch to be failed.
This changes DepthGenerator to process unshallow commits first
independent to their depth. This makes uninteresting flag carry work
properly.
Keep track of the original cause for a packfile invalidation.
It is needed for the sysadmin to understand if there is a real
underlying filesystem problem and repository corruption or if it is
simply a consequence of a concurrency of Git operations (e.g. repack
or GC).
Change-Id: I06ddda9ec847844ec31616ab6d17f153a5a34e33 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix pack files scan when filesnapshot isn't modified
Do not reload packfiles when their associated filesnapshot is not
modified on disk compared to the one currently stored in memory.
Fix the regression introduced by fef78212 which, in conjunction with
core.trustfolderstats = false, caused any lookup of objects inside
the packlist to loop forever when the object was not found in the pack
list.
Patrick Hiesel [Tue, 22 Jan 2019 09:02:14 +0000 (10:02 +0100)]
Introduce RefDatabase#getTipsWithSha1 to list refs pointing to object
Add resolveTipSha1, an inverse of exactRef(String ...), to RefDatabase
and provide a default implementation that runs in O(n) time where n is
the number of refs. For RefTable, provide an implementation that runs
in O(log(n)) time.
[ifrade@google.com: with tests in InMemoryRepositoryTest to exercise
the reftable code path, too]
Change-Id: I2811ccd0339cdc1c74b42cce2ea003f07a2ce9e1 Signed-off-by: Patrick Hiesel <hiesel@google.com> Signed-off-by: Ivan Frade <ifrade@google.com>
LocalDiskRepositoryTestCase: Clarify semantics of repository creation methods
Several of the utility methods create a Repository without setting
the 'autoClose' flag, which means that the caller is responsible for
closing it. Update the Javadoc to explicitly mention this.
Change-Id: I2410dd8d230cd4519f756c38b17141d0daa6c314 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Thomas Wolf [Wed, 3 Apr 2019 15:32:53 +0000 (17:32 +0200)]
Attach deletion failure reason in FileUtils.delete()
Use Files.delete() instead of File.delete(), and if there is
an exception thrown propagate it unless errors are to be ignored so
that the actual deletion failure cause is available to the caller
(and will be logged).
Change-Id: I5fdb5a4052942437ab365289ad4bb1b563c29456 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthew DeVore [Tue, 26 Mar 2019 17:27:38 +0000 (10:27 -0700)]
.gitignore: remove editor- and OS-specific files
Whenever Vim opens a file, it creates a .<filename>.swp file in the same
directory as the file. Emacs adds *~ backup files. macOS creates
.DS_Store files. Other editors and OS' surely do their own thing. Rather
than add each one's own swap/backup file to this .gitignore, encourage
users to add the corresponding items to their system-wide gitignore
files.
Change-Id: I5535f5d2f1ebe896eef108cfda087dcb4c50f031 Signed-off-by: Matthew DeVore <matvore@gmail.com>
Thomas Wolf [Thu, 28 Mar 2019 21:28:47 +0000 (22:28 +0100)]
[releng] Fix dependencies between features
Add dependencies from source to binary features and enforce the same
version. Restrict the dependencies between binary features by version,
too.
Adapt the version.sh script to properly replace these version numbers.
Bug: 545906
Change-Id: I953616e58d27accdf61b49f6435a54895bcbf6da Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Ivan Frade [Wed, 27 Mar 2019 23:24:13 +0000 (16:24 -0700)]
DfsRefDatabase: Keep update index when peeling a reference
The new references created in the peeling do not receive the update
index. In other words, the update index of a reference (if set) is lost
in the peeling.
Pass-through the update index to the newly created references.
Tested via InMemoryRepository, which uses DfsReftableDatabase.
Change-Id: I7ff7c737a9c3366fdec296a4d9b2e51d10227957 Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Wed, 27 Mar 2019 22:21:50 +0000 (15:21 -0700)]
Ref: Add constant for undefined update index
Code that creates references and wants to support ref dbs with and
without update indexes needs to set this value. This leds to a
proliferation of "-1" in the code base.
Make the "undefined" value a constant in the ref interface.
Change-Id: I2622a37536a84b4a4036dd55792e185486fa0628 Signed-off-by: Ivan Frade <ifrade@google.com>
Matthias Sohn [Wed, 27 Mar 2019 15:26:28 +0000 (16:26 +0100)]
Fix API error filter settings
- UploadPack.getFilterBlobLimit() didn't have a wrong @since tag but was
marked final shortly after 5.3 was released. This was probably caused by
using an outdated API baseline. Currently we should use 5.3.0 as the
baseline.
- remove unused filter on FS.fileAttributes()
Change-Id: I9adc1703e99a9ddb3ea2a1c12a83dccbc1f69a99 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthew DeVore [Mon, 18 Mar 2019 16:45:59 +0000 (09:45 -0700)]
Expose and pass around the FilterSpec object rather than the raw blob limit
Use the FilterSpec object so that less code has to know about the make-up of
FilterSpecs. When fields are added to FilterSpec, these pieces of code won't
need updating again.
Change-Id: I2b9e59a9926ff112faf62a3fa2d33c961a1779e5 Signed-off-by: Matthew DeVore <matvore@gmail.com>
projects / jgit.git / log