]> source.dussan.org Git - jgit.git/log
jgit.git
3 years agoDocument http options supported by JGit 00/177700/1
Thomas Wolf [Sat, 13 Mar 2021 16:05:47 +0000 (17:05 +0100)]
Document http options supported by JGit

Change-Id: I0af4f9991fdb4f09de25f743d1e0dca67ceaa18b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoHTTP cookies: do tilde expansion on http.cookieFile 49/177649/1
Thomas Wolf [Tue, 9 Mar 2021 21:23:14 +0000 (22:23 +0100)]
HTTP cookies: do tilde expansion on http.cookieFile

Git config http.cookieFile must have ~ expansion, compare [1].

It also should be an absolute path. While a relative path is allowed,
C git just passes the value on to libcurl, so it'll be relative to the
current working directory and thus not work in all directories.

Log a warning if the path is relative.

(Alternatives would be to throw an exception, or to resolve the path
relative to the .git directory, or relative to the working tree root,
or relative to the config file it occurs in. But C git does not seem
to do either.)

[1] https://github.com/git/git/commit/e5a39ad8e

Bug: 571798
Change-Id: I5cdab6061d0613ac7d8cb7977e5b97f5b88f562d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoPrepare 5.12.0-SNAPSHOT builds 19/177519/2
Matthias Sohn [Wed, 10 Mar 2021 15:06:00 +0000 (16:06 +0100)]
Prepare 5.12.0-SNAPSHOT builds

Change-Id: I736de7c3deb11da75777d459f47332df0b486443
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge branch 'stable-5.11' 18/177518/1
Matthias Sohn [Wed, 10 Mar 2021 15:05:18 +0000 (16:05 +0100)]
Merge branch 'stable-5.11'

* stable-5.11:
  Update Orbit to R20210223232630
  Prepare 5.11.1-SNAPSHOT builds
  JGit v5.11.0.202103091610-r
  Prepare 5.11.0-SNAPSHOT builds
  JGit v5.11.0.202103031150-rc1

Change-Id: I808e53a3c54a49d0dd62c69818ea2f5672e16a91

3 years agoUpdate Orbit to R20210223232630 68/177468/1
Matthias Sohn [Tue, 9 Mar 2021 23:39:29 +0000 (00:39 +0100)]
Update Orbit to R20210223232630

Change-Id: I7577131b2c6e808b59f6c453233b261c64646d35
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoPrepare 5.11.1-SNAPSHOT builds 65/177465/1
Matthias Sohn [Tue, 9 Mar 2021 22:42:31 +0000 (23:42 +0100)]
Prepare 5.11.1-SNAPSHOT builds

Change-Id: I94628ccbb5099a65aa4345cfd28a141ff5555b68
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoJGit v5.11.0.202103091610-r 61/177461/1 v5.11.0.202103091610-r
Matthias Sohn [Tue, 9 Mar 2021 21:10:22 +0000 (22:10 +0100)]
JGit v5.11.0.202103091610-r

Change-Id: I8e6855eaf7228459f492036feb4e34ca085698a7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge branch 'master' into stable-5.11 44/177444/1
Matthias Sohn [Tue, 9 Mar 2021 17:00:55 +0000 (18:00 +0100)]
Merge branch 'master' into stable-5.11

* master:
  Manually set status of jmh dependencies
  Update DEPENDENCIES report for 5.11.0
  Add dependency to dash-licenses
  PackFile: Add id + ext based constructors
  GC: deleteOrphans: Use PackFile
  PackExt: Convert to Enum
  Restore preserved packs during missing object seeks
  Pack: Replace extensions bitset with bitmapIdx PackFile
  PackDirectory: Use PackFile to ensure we find preserved packs
  GC: Use PackFile to de-dup logic
  Create a PackFile class for Pack filenames

Change-Id: I1d56517cb6a95e10aed22cdb9e5f3e504872d110

3 years agoManually set status of jmh dependencies 25/177325/2
Matthias Sohn [Sun, 7 Mar 2021 17:41:05 +0000 (18:41 +0100)]
Manually set status of jmh dependencies

The following jmh dependencies were approved as works-with:
- jmh-core/1.21 has GPL-2.0 license and was approved in CQ20517
- jmh-generator-annprocess/1.21 has GPL-2.0 license and was approved in
CQ20518

Change-Id: Ibbe28c6e8359c576b23f40281e74f2e0d4a0dee0

3 years agoUpdate DEPENDENCIES report for 5.11.0 24/177324/3
Matthias Sohn [Sun, 7 Mar 2021 16:44:05 +0000 (17:44 +0100)]
Update DEPENDENCIES report for 5.11.0

Computed by dash license-tool-plugin [1].

[1] https://github.com/eclipse/dash-licenses

Change-Id: I28727115914613cefdcf9a9f462c195b6af23156

3 years agoAdd dependency to dash-licenses 23/177323/2
Matthias Sohn [Wed, 27 Jan 2021 23:25:06 +0000 (00:25 +0100)]
Add dependency to dash-licenses

This is required to run the license check using the license tool [1]
required by the Eclipse project handbook [2]:

mvn org.eclipse.dash:license-tool-plugin:license-check -Ddash.summary=DEPENDENCIES

Note: the tool still requires Java 11 hence it needs to be run in a
separate build step and is not yet integrated in the build which runs
on Java 8.

[1] https://github.com/eclipse/dash-licenses
[2] https://www.eclipse.org/projects/handbook/#ip-license-tool

Change-Id: Ib41d54de246c3c9499cc3be9f026294c39fdfd99

3 years agoPackFile: Add id + ext based constructors 92/177192/3
Nasser Grainawi [Thu, 4 Mar 2021 21:14:43 +0000 (14:14 -0700)]
PackFile: Add id + ext based constructors

Add new constructors to PackFile to improve a common use case where
callers know the directory, id, and extension, but previously needed to
construct a valid file name (with prefix, '.', etc) to create a
PackFile. Most callers can use the variant that has id as an ObjectId,
but provide an id as String variant too.

Change-Id: I39e4466abe8c9509f5916d5bfe675066570b8585
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoGC: deleteOrphans: Use PackFile 91/177191/1
Nasser Grainawi [Thu, 4 Mar 2021 00:05:21 +0000 (17:05 -0700)]
GC: deleteOrphans: Use PackFile

It's easier to follow the logic here when we can use our own objects
instead of Strings.

Change-Id: I6a166edcc67903fc1ca3544f458634c4cef8fde7
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoPackExt: Convert to Enum 53/176953/3
Nasser Grainawi [Fri, 26 Feb 2021 22:49:06 +0000 (15:49 -0700)]
PackExt: Convert to Enum

This class already looked very much like an Enum, but wasn't one.

As an Enum, we can use PackExt in EnumMaps and EnumSets. Convert the
Map key usage in PackDirectory to an EnumMap.

Change-Id: Ice097fd468a05805f914e6862fbd1d96ec8c45d1
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoRestore preserved packs during missing object seeks 88/122288/14
Martin Fick [Tue, 15 Dec 2020 21:20:44 +0000 (14:20 -0700)]
Restore preserved packs during missing object seeks

Provide a recovery path for objects being referenced during the pack
pruning race. Due to the pack pruning race, it is possible for objects
to become referenced after a pack has been deemed safe to prune, but
before it actually gets pruned. If this happened previously, the newly
referenced objects would be missing and potentially result in a
corrupted ref.

Add the ability to recover from this situation when an object is missing
but happens to still be available in a pack in the "preserved"
directory. This is likely only useful when used in conjunction with the
--preserve-old-packs GC option, which prunes packs by hard-linking to
the preserved directory. If an object is missing and found in a pack in
the preserved directory, immediately recover that pack and its
associated files (idx, bitmaps...) by moving them back to the original
pack directory, and then retry the operation that would have failed due
to the missing object. This retry can now succeed and the repository
may avoid corruption. This approach should drastically reduce the
chance of a corrupt repository during pack pruning at very little extra
cost. This extra cost should only be incurred when objects are missing
and a failure would normally occur.

Change-Id: I2a704e3276b88cc892159d9bfe2455c6eec64252
Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoPack: Replace extensions bitset with bitmapIdx PackFile 29/175129/8
Nasser Grainawi [Thu, 11 Feb 2021 06:26:17 +0000 (23:26 -0700)]
Pack: Replace extensions bitset with bitmapIdx PackFile

The only extension that was ever consulted from the bitmap was the
bitmap index. We can simplify the Pack code as well as the code of
all the callers if we focus on just that usage.

Change-Id: I799ddfdee93142af67ce5081d14a430d36aa4c15
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoPackDirectory: Use PackFile to ensure we find preserved packs 30/175130/11
Nasser Grainawi [Thu, 11 Feb 2021 06:33:43 +0000 (23:33 -0700)]
PackDirectory: Use PackFile to ensure we find preserved packs

Update scanPacksImpl and listPackDirectory (renamed to
getPackFilesByExtById) to use the new PackFile functionality to
validate file names and complete pack file sets (.pack, .idx, etc).

Most importantly, this allows a later change to rely on scanPacks() to
complete a packList that contains packs with the 'old-' prefix in their
extension.

This also eliminates duplication of logic for how to identify and
construct pack files.

Change-Id: I7175e5fefb187a29e0a7cf53c392aee922314f31
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoGC: Use PackFile to de-dup logic 33/176533/4
Nasser Grainawi [Fri, 19 Feb 2021 00:36:49 +0000 (17:36 -0700)]
GC: Use PackFile to de-dup logic

GC has several places where it tries to build files names for packs that
we can use the PackFile class for instead.

Change-Id: I99e5ceff9050f8583368fca35279251955e4644d
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoCreate a PackFile class for Pack filenames 45/176145/5
Nasser Grainawi [Thu, 11 Feb 2021 05:51:05 +0000 (22:51 -0700)]
Create a PackFile class for Pack filenames

The PackFile class is intended to be a central place to do all
common pack filename manipulation and parsing to help reduce repeated
code and bugs. Use the PackFile class in the Pack class and in many
tests to ensure it works well in a variety of situations. Later changes
will expand use of PackFiles to even more areas.

Change-Id: I921b30f865759162bae46ddd2c6d669de06add4a
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoPrepare 5.11.0-SNAPSHOT builds 63/177163/1
Matthias Sohn [Thu, 4 Mar 2021 15:27:51 +0000 (16:27 +0100)]
Prepare 5.11.0-SNAPSHOT builds

Change-Id: I89ed49a6acc53dd75d16f40c99e1140e0c18f646
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoJGit v5.11.0.202103031150-rc1 07/177107/1 v5.11.0.202103031150-rc1
Matthias Sohn [Wed, 3 Mar 2021 16:48:00 +0000 (17:48 +0100)]
JGit v5.11.0.202103031150-rc1

Change-Id: I0a86fa59645888f9f36ea6938c9121e095f02fc6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge branch 'master' into stable-5.11 03/177103/1
Matthias Sohn [Wed, 3 Mar 2021 16:44:20 +0000 (17:44 +0100)]
Merge branch 'master' into stable-5.11

* master:
  HTTP: cookie file stores expiration in seconds
  Update Orbit to S20210223232630
  LFS: handle invalid pointers better
  Fix errorprone configuration for maven-compiler-plugin with javac

Change-Id: Ib76e754bd36789de0a3c6b85a4814aa1fe9cb401

3 years agoHTTP: cookie file stores expiration in seconds 83/176983/3
Thomas Wolf [Mon, 1 Mar 2021 07:30:09 +0000 (08:30 +0100)]
HTTP: cookie file stores expiration in seconds

A cookie file stores the expiration in seconds since the Linux Epoch,
not in milliseconds. Correct reading and writing cookie files; with
a backwards-compatibility hack to read files that contain a millisecond
timestamp.

Add a test, and fix tests not to rely on the actual current time so
that they will also run successfully after 2030-01-01 noon.

Bug: 571574
Change-Id: If3ba68391e574520701cdee119544eedc42a1ff2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoMerge "Update Orbit to S20210223232630"
Matthias Sohn [Tue, 2 Mar 2021 16:07:49 +0000 (11:07 -0500)]
Merge "Update Orbit to S20210223232630"

3 years agoUpdate Orbit to S20210223232630 19/177019/1
Matthias Sohn [Mon, 1 Mar 2021 21:10:22 +0000 (22:10 +0100)]
Update Orbit to S20210223232630

Change-Id: Ida7a54cfe0bd15c1c28b892dea3452958924f0c0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoLFS: handle invalid pointers better 22/175522/6
Thomas Wolf [Fri, 29 Jan 2021 22:03:44 +0000 (23:03 +0100)]
LFS: handle invalid pointers better

Make sure that SmudgeFilter calls LfsPointer.parseLfsPointer() with
a stream that supports mark/reset, and make sure that parseLfsPointer()
resets the stream properly if it decides that the stream content is not
a LFS pointer.

Add a test.

Bug: 570758
Change-Id: I2593d67cff31b2dfdfaaa48e437331f0ed877915
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoFix errorprone configuration for maven-compiler-plugin with javac 27/174127/3
Matthias Sohn [Sun, 27 Dec 2020 01:11:47 +0000 (02:11 +0100)]
Fix errorprone configuration for maven-compiler-plugin with javac

See https://errorprone.info/docs/installation.

Add new profile jdk8 to enable running errorprone with javac on java 8
and java 11. Remove errorprone configuration from benchmark module,
didn't find a way to make it work and this module does not contain any
productive code.

Change-Id: I6a84195af05e6cea9e7c04ad5cd4c79742e80cb3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge branch 'stable-5.11' 80/176880/1
Matthias Sohn [Thu, 25 Feb 2021 08:09:57 +0000 (09:09 +0100)]
Merge branch 'stable-5.11'

* stable-5.11:
  Prepare 5.11.0-SNAPSHOT builds
  JGit v5.11.0.202102240950-m3

Change-Id: Ia216a698dd4f0dd235dfe4de4d2cc127aa530eed

3 years agoPrepare 5.11.0-SNAPSHOT builds 57/176857/1
Matthias Sohn [Wed, 24 Feb 2021 20:21:34 +0000 (21:21 +0100)]
Prepare 5.11.0-SNAPSHOT builds

Change-Id: If3dbe084ee37ae4b993d3a10ec48b14e8709ff6d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoJGit v5.11.0.202102240950-m3 39/176839/1 v5.11.0.202102240950-m3
Matthias Sohn [Wed, 24 Feb 2021 14:50:41 +0000 (15:50 +0100)]
JGit v5.11.0.202102240950-m3

Change-Id: Iea6b3515fa63db497989194b6bf50fe7324086d0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge branch 'master' into stable-5.11 13/176813/1
Matthias Sohn [Wed, 24 Feb 2021 13:54:52 +0000 (14:54 +0100)]
Merge branch 'master' into stable-5.11

* master: (35 commits)
  [releng] japicmp: update last release version
  IgnoreNode: include path to file for invalid .gitignore patterns
  FastIgnoreRule: include bad pattern in log message
  init: add config option to set default for the initial branch name
  init: allow specifying the initial branch name for the new repository
  Fail clone if initial branch doesn't exist in remote repository
  GPG: fix reading unprotected old-format secret keys
  Update Orbit to S20210216215844
  Add missing bazel dependency for o.e.j.gpg.bc.test
  GPG: handle extended private key format
  dfs: handle short copies
  [GPG] Provide a factory for the BouncyCastleGpgSigner
  Fix boxing warnings
  GPG: compute the keygrip to find a secret key
  GPG signature verification via BouncyCastle
  Post commit hook failure should not cause commit failure
  Allow to define additional Hook classes outside JGit
  GitHook: use default charset for output and error streams
  GitHook: use generic OutputStream instead of PrintStream
  Update jetty to 9.4.36.v20210114
  ...

Change-Id: I1cf5ab262c67b986e82422c48dfc103e335d28cc

3 years ago[releng] japicmp: update last release version 55/176755/3
Thomas Wolf [Tue, 23 Feb 2021 21:17:07 +0000 (22:17 +0100)]
[releng] japicmp: update last release version

The baseline for the 5.11 release is 5.10.0.202012080955-r.

Change-Id: Ied9b42dc58ba981e5586fa58d1b3e70a39c78a10
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoIgnoreNode: include path to file for invalid .gitignore patterns 44/176744/2
Thomas Wolf [Tue, 23 Feb 2021 17:10:08 +0000 (18:10 +0100)]
IgnoreNode: include path to file for invalid .gitignore patterns

Include the full file path of the .gitignore file and the line number
of the invalid pattern. Also include the pattern itself.

.gitignore files inside the repository are reported with their
repository-relative path; files outside (from git config
core.excludesFile or .git/info/exclude) are reported with their
full absolute path.

Bug: 571143
Change-Id: Ibe5969679bc22cff923c62e3ab9801d90d6d06d1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoFastIgnoreRule: include bad pattern in log message 31/176731/2
Thomas Wolf [Tue, 23 Feb 2021 12:11:56 +0000 (13:11 +0100)]
FastIgnoreRule: include bad pattern in log message

When a .gitignore pattern cannot be parsed include the pattern in the
log message. Just reporting "not closed bracket" isn't helpful if the
user doesn't know in which pattern the problem occurred.

Even better would be to include the full path of the .gitignore file
that contained the offending pattern. This is not implemented in this
change; it may need new API and needs more thought.

Bug: 571143
Change-Id: Id5b16d9cf550544ba3ad409a02041946fa8516ab
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoinit: add config option to set default for the initial branch name 97/175297/7
Matthias Sohn [Mon, 25 Jan 2021 01:43:18 +0000 (02:43 +0100)]
init: add config option to set default for the initial branch name

We introduced the option --initial-branch=<branch-name> to allow
initializing a new repository with a different initial branch.

To allow users to override the initial branch name more permanently
(i.e. without having to specify the name manually for each 'git init'),
introduce the 'init.defaultBranch' option.

This option was added to git in 2.28.0.

See https://git-scm.com/docs/git-config#Documentation/git-config.txt-initdefaultBranch

Bug: 564794
Change-Id: I679b14057a54cd3d19e44460c4a5bd3a368ec848
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoinit: allow specifying the initial branch name for the new repository 96/175296/6
Matthias Sohn [Mon, 25 Jan 2021 00:54:03 +0000 (01:54 +0100)]
init: allow specifying the initial branch name for the new repository

Add option --initial-branch/-b to InitCommand and the CLI init command.
This is the first step to implement support for the new option
init.defaultBranch. Both were added to git in release 2.28.

See https://git-scm.com/docs/git-init#Documentation/git-init.txt--bltbranch-namegt

Bug: 564794
Change-Id: Ia383b3f90b5549db80f99b2310450a7faf6bce4c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoFail clone if initial branch doesn't exist in remote repository 64/175264/8
Matthias Sohn [Sat, 23 Jan 2021 23:17:13 +0000 (00:17 +0100)]
Fail clone if initial branch doesn't exist in remote repository

jgit clone --branch foo <url>

did not fail if the remote branch "foo" didn't exist in the remote
repository being cloned.

Bug: 546580
Change-Id: I55648ad3a39da4a5711dfa8e6d6682bb8190a6d6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoGPG: fix reading unprotected old-format secret keys 95/176595/1
Thomas Wolf [Mon, 22 Feb 2021 08:29:12 +0000 (09:29 +0100)]
GPG: fix reading unprotected old-format secret keys

Fix code and add a test case. The old code passed on the original input
stream, which has already been consumed.

Bug: 570501
Change-Id: I81f60698ce42443df57e59b1d1ab155574136fa8
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoUpdate Orbit to S20210216215844 69/176569/1
Matthias Sohn [Sat, 20 Feb 2021 23:01:22 +0000 (00:01 +0100)]
Update Orbit to S20210216215844

Change-Id: Ic3af137e4aad0e6f7fd32c910766f547562442d6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoAdd missing bazel dependency for o.e.j.gpg.bc.test 64/176564/2
Matthias Sohn [Sat, 20 Feb 2021 18:05:59 +0000 (19:05 +0100)]
Add missing bazel dependency for o.e.j.gpg.bc.test

This was missed in 64cbea8a9794047fe576d03ab8a46e4eaf7eabee.

Change-Id: I0b2b234b9888a7dd8b7ace624233b141fb7c4394
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoGPG: handle extended private key format 32/175332/11
Thomas Wolf [Sun, 24 Jan 2021 01:13:43 +0000 (02:13 +0100)]
GPG: handle extended private key format

Add detection for the key-value pair format that was available in
gpg-agent for some time already and that has become the default since
gpg-agent 2.2.20. If a secret key in the .gnupg/private-keys-v1.d
directory is found to have this format, extract the human-readable key
from it, convert it to the binary serialized form and hand that to
BouncyCastle.

Encrypted keys in the new format may use AES/OCB. OCB is a patent-
encumbered algorithm; although there is a license for open-source
software, that may not be good enough and OCB may not be available in
Java. It is not available in the default security provider in Java,
and it is also not available in the BouncyCastle version included in
Eclipse.

Implement AES/OCB decryption, throwing a PGPException with a nice
message if the algorithm is not available. Include a copy of the normal
s-expression parser of BouncyCastle and fix it to properly handle data
from such keys: such keys do not contain an internal hash since the
AES/OCB cipher includes and checks a MAC already.

Bug: 570501
Change-Id: Ifa6391a809a84cfc6ae7c6610af6a79204b4143b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agodfs: handle short copies 47/173947/4
wh [Thu, 17 Dec 2020 18:14:32 +0000 (18:14 +0000)]
dfs: handle short copies

`copy` is documented as possibly returning a smaller number of bytes
than requested. In practice, this can occur if a block is cached and the
reader never pulls in the file to check its size.

Bug: 565874
Change-Id: I1e53b3d2f4ab09334178934dc0ef74ea99045cd3
Signed-off-by: wh <wh9692@protonmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge "[GPG] Provide a factory for the BouncyCastleGpgSigner"
Thomas Wolf [Fri, 19 Feb 2021 07:57:38 +0000 (02:57 -0500)]
Merge "[GPG] Provide a factory for the BouncyCastleGpgSigner"

3 years ago[GPG] Provide a factory for the BouncyCastleGpgSigner 73/176473/3
Thomas Wolf [Thu, 18 Feb 2021 16:06:27 +0000 (17:06 +0100)]
[GPG] Provide a factory for the BouncyCastleGpgSigner

Otherwise client code has no way to ever create an instance without
using internal non-API.

Change-Id: I6201f98d4b1704a053159967b8adacd98e368522
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoMerge "Rename PackFile to Pack"
Jonathan Nieder [Thu, 18 Feb 2021 22:04:07 +0000 (17:04 -0500)]
Merge "Rename PackFile to Pack"

3 years agoFix boxing warnings 03/176303/2
Matthias Sohn [Tue, 16 Feb 2021 00:13:35 +0000 (01:13 +0100)]
Fix boxing warnings

Change-Id: Idf4887a99e87c375ec32e2fd289cfce82d78cbce
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoGPG: compute the keygrip to find a secret key 55/174955/13
Thomas Wolf [Sun, 17 Jan 2021 15:21:28 +0000 (16:21 +0100)]
GPG: compute the keygrip to find a secret key

The gpg-agent stores secret keys in individual files in the secret
key directory private-keys-v1.d. The files have the key's keygrip
(in upper case) as name and extension ".key".

A keygrip is a SHA1 hash over the parameters of the public key. By
computing this keygrip, we can pre-compute the expected file name and
then check only that one file instead of having to iterate over all
keys stored in that directory.

This file naming scheme is actually an implementation detail of
gpg-agent. It is unlikely to change, though. The keygrip itself is
computed via libgcrypt and will remain stable according to the GPG
main author.[1]

Add an implementation for calculating the keygrip and include tests.
Do not iterate over files in BouncyCastleGpgKeyLocator but only check
the single file identified by the keygrip.

Ideally upstream BouncyCastle would provide such a getKeyGrip() method.
But as it re-builds GPG and libgcrypt internals, it's doubtful it would
be included there, and since BouncyCastle even lacks a number of curve
OIDs for ed25519/curve25519 and uses the short-Weierstrass parameters
instead of the more common Montgomery parameters, including it there
might be quite a bit of work.

[1] http://gnupg.10057.n7.nabble.com/GnuPG-2-1-x-and-2-2-x-keyring-formats-tp54146p54154.html

Bug: 547536
Change-Id: I30022a0e7b33b1bf35aec1222f84591f0c30ddfd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoGPG signature verification via BouncyCastle 06/174406/12
Thomas Wolf [Thu, 7 Jan 2021 16:11:57 +0000 (17:11 +0100)]
GPG signature verification via BouncyCastle

Add a GpgSignatureVerifier interface, plus a factory to create
instances thereof that is provided via the ServiceLoader mechanism.

Implement the new interface for BouncyCastle. A verifier maintains
an internal LRU cache of previously found public keys to speed up
verifying multiple objects (tag or commits). Mergetags are not handled.

Provide a new VerifySignatureCommand in org.eclipse.jgit.api together
with a factory method Git.verifySignature(). The command can verify
signatures on tags or commits, and can be limited to accept only tags
or commits. Provide a new public WrongObjectTypeException thrown when
the command is limited to either tags or commits and a name resolves
to some other object kind.

In jgit.pgm, implement "git tag -v", "git log --show-signature", and
"git show --show-signature". The output is similar to command-line
gpg invoked via git, but not identical. In particular, lines are not
prefixed by "gpg:" but by "bc:".

Trust levels for public keys are read from the keys' trust packets,
not from GPG's internal trust database. A trust packet may or may
not be set. Command-line GPG produces more warning lines depending
on the trust level, warning about keys with a trust level below
"full".

There are no unit tests because JGit still doesn't have any setup to
do signing unit tests; this would require at least a faked .gpg
directory with pre-created key rings and keys, and a way to make the
BouncyCastle classes use that directory instead of the default. See
bug 547538 and also bug 544847.

Tested manually with a small test repository containing signed and
unsigned commits and tags, with signatures made with different keys
and made by command-line git using GPG 2.2.25 and by JGit using
BouncyCastle 1.65.

Bug: 547751
Change-Id: If7e34aeed6ca6636a92bf774d893d98f6d459181
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoPost commit hook failure should not cause commit failure 76/154476/10
Tim Neumann [Fri, 13 Dec 2019 14:20:02 +0000 (15:20 +0100)]
Post commit hook failure should not cause commit failure

As the post commit hook is run after a commit is finished, it can not
abort the commit and the exit code of this hook should not have any
effect.

This can be achieved by not throwing a AbortedByHookException exception.
The stderr output is not lost thanks to contributions for bug 553471.

Bug: 553428
Change-Id: I451a76e04103e632ff44e045561c5a41f7b7d558
Signed-off-by: Tim Neumann <Tim.Neumann@advantest.com>
Signed-off-by: Fabian Pfaff <fabian.pfaff@vogella.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoAllow to define additional Hook classes outside JGit 47/175947/5
Matthias Sohn [Sun, 7 Feb 2021 22:48:57 +0000 (23:48 +0100)]
Allow to define additional Hook classes outside JGit

EGit wants to add gitflow specific hooks in org.eclipse.egit.gitflow.
Make GitHook public to allow sub-classing outside of the
org.eclipse.jgit.hooks package.

Change-Id: I439575ec901e3610b5cf9d66f7641c8324faa865
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoGitHook: use default charset for output and error streams 17/176017/4
Matthias Sohn [Tue, 9 Feb 2021 01:42:29 +0000 (02:42 +0100)]
GitHook: use default charset for output and error streams

External scripts most probably expect the default charset.

Change-Id: I318a5e1d9f536a95e70c06ffb5b6f408cd40f73a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoGitHook: use generic OutputStream instead of PrintStream 16/176016/3
Matthias Sohn [Tue, 9 Feb 2021 01:34:10 +0000 (02:34 +0100)]
GitHook: use generic OutputStream instead of PrintStream

Change-Id: I15e64dc963c9d27dc9c8de4976dd63f74b918b15
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoUpdate jetty to 9.4.36.v20210114 34/176234/1
Matthias Sohn [Sat, 13 Feb 2021 20:56:03 +0000 (21:56 +0100)]
Update jetty to 9.4.36.v20210114

Change-Id: Iea57f0fddb0f10dbd1c9be886bfa5ad8c3ff5cb5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoRename PackFile to Pack 44/176144/1
Nasser Grainawi [Thu, 11 Feb 2021 05:42:41 +0000 (22:42 -0700)]
Rename PackFile to Pack

Pack better represents the purpose of the object and paves the way to
add a PackFile object that extends File.

Change-Id: I39b4f697902d395e9b6df5e8ce53078ce72fcea3
Signed-off-by: Nasser Grainawi <quic_nasserg@quicinc.com>
3 years agoAllow dir/file conflicts in virtual base commit on recursive merge. 66/175166/5
Marija Savtchouk [Thu, 21 Jan 2021 15:08:57 +0000 (15:08 +0000)]
Allow dir/file conflicts in virtual base commit on recursive merge.

If RecursiveMerger finds multiple base commits, it tries to compute
the virtual ancestor to use as a base for the three way merge.
Currently, the content conflicts between ancestors are ignored (file
staged with the conflict markers). If the path is a file in one ancestor
and a dir in the other, it results in NoMergeBaseException
(CONFLICTS_DURING_MERGE_BASE_CALCULATION).

Allow these conflicts by ignoring this unmerged path in the virtual
base. The merger will compute diff in the children instead and it
can be further fixed manually if needed.

Change-Id: Id59648ae1d6bdf300b26fff513c3204317b755ab
Signed-off-by: Marija Savtchouk <mariasavtchouk@google.com>
3 years agoMerge "GitHook: make fields outputStream and errorStream private"
Christian Halstrick [Tue, 9 Feb 2021 07:07:53 +0000 (02:07 -0500)]
Merge "GitHook: make fields outputStream and errorStream private"

3 years agoGitHook: make fields outputStream and errorStream private 15/176015/1
Matthias Sohn [Tue, 9 Feb 2021 01:17:14 +0000 (02:17 +0100)]
GitHook: make fields outputStream and errorStream private

Subclasses can use the corresponding getter methods.

Change-Id: Iaa9ab01f5a9731a264b28608d2418a9405b601d7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoGPG: support git config gpg.program 31/175331/4
Thomas Wolf [Sun, 24 Jan 2021 00:57:09 +0000 (01:57 +0100)]
GPG: support git config gpg.program

Add it to the GpgConfig. Change GpgConfig to load the values once only.
Add a parameter to the GpgObjectSigner interface's operations to pass
in a GpgConfig. Update CommitCommand and TagCommand to pass the value
to the signer. Let the signer decide whether it can actually produce
the wanted signature type (openpgp or x509).

No behavior change. But this makes it possible to implement different
signers that might support x509 signatures, or use gpg.program and
shell out to an external GPG executable for signing.

Change-Id: I427f83eb1ece81c310e1cddd85315f6f88cc99ea
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoBump bazel version to 4.0.0 39/175939/2
David Ostrovsky [Sun, 7 Feb 2021 14:38:20 +0000 (15:38 +0100)]
Bump bazel version to 4.0.0

Change-Id: I2faa67d5083f23b29f7a434e54c5e17360b1c0fe
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
3 years agoBazel: Remove unused resources variable 38/175938/2
David Ostrovsky [Sun, 7 Feb 2021 14:36:35 +0000 (15:36 +0100)]
Bazel: Remove unused resources variable

Change-Id: Iac2e547791929c26027ab4730ceac6177899ccf1
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
3 years agoFix DateRevQueue tie breaks with more than 2 elements 23/175523/11
Adithya Chakilam [Sat, 30 Jan 2021 01:35:04 +0000 (19:35 -0600)]
Fix DateRevQueue tie breaks with more than 2 elements

DateRevQueue is expected to give out the commits that have higher
commit time. But in case of tie(same commit time), it should give
the commit that is inserted first. This is inferred from the
testInsertTie test case written for DateRevQueue. Also that test
case, right now uses just two commits which caused it not to fail
with the current implementation, so added another commit to make
the test more robust.

By fixing the DateRevQueue, we would also match the behaviour of
LogCommand.addRange(c1,c2) with git log c1..c2. A test case for
the same is added to show that current behaviour is not the
expected one.

By fixing addRange(), the order in which commits are applied during
a rebase is altered. Rebase logic should have never depended upon
LogCommand.addRange() since the intended order of addRange() is not
the order a rebase should use. So, modify the RebaseCommand to use
RevWalk directly with TopoNonIntermixSortGenerator.

Add a new LogCommandTest.addRangeWithMerge() test case which creates
commits in the following order:

         A - B - C - M
              \     /
                -D-

Using git 2.30.0, git log B..M outputs:  M C D
LogCommand.addRange(B, M) without this fix outputs: M D C
LogCommand.addRange(B, M) with this fix outputs: M C D

Change-Id: I30cc3ba6c97f0960f64e9e021df96ff276f63db7
Signed-off-by: Adithya Chakilam <achakila@codeaurora.org>
3 years agoField updateHead can be a local variable in RefDirectoryRename 03/175703/3
Lars Vogel [Wed, 3 Feb 2021 10:55:39 +0000 (11:55 +0100)]
Field updateHead can be a local variable in RefDirectoryRename

Keeping the field updateDate is unecessary, as it is set and used only
in the doRename method.

Change-Id: I1cdd1adf759b75c103480db7a74cec8c2d78b794
Signed-off-by: Lars Vogel <Lars.Vogel@vogella.com>
3 years agoFix FileRepository#convertToReftable which failed if no reflog existed 89/174189/3
Matthias Sohn [Sun, 3 Jan 2021 01:10:58 +0000 (02:10 +0100)]
Fix FileRepository#convertToReftable which failed if no reflog existed

Deleting non-existing files when converting to reftable without backup
caused convertToReftable to fail. Observed this on a mirrored repository
which had no reflogs. Fix this by skipping missing files during
deletion.

Change-Id: I3bb913d5bfddccc6813677b873006efb849a6ebc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoFix SeparateClassloaderTestRunner on Java 9 or higher 26/174126/3
Matthias Sohn [Sat, 26 Dec 2020 01:28:03 +0000 (02:28 +0100)]
Fix SeparateClassloaderTestRunner on Java 9 or higher

Since Java 9 the SystemClassLoader is no longer a URLClassLoader.

Change-Id: I3aa834f1075e611c86fc4684fda6a50c684b3729
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge "reftable: add random suffix to table names"
Matthias Sohn [Sat, 6 Feb 2021 00:43:42 +0000 (19:43 -0500)]
Merge "reftable: add random suffix to table names"

3 years agoTransportGitAnon: remove unnecessary socket bind to a local address 07/175907/2
Alina Djamankulova [Fri, 5 Feb 2021 18:12:25 +0000 (10:12 -0800)]
TransportGitAnon: remove unnecessary socket bind to a local address
before connecting.

A socket gets bound on connect in the next line.

Signed-off-by: Alina Djamankulova <adjama@google.com>
Change-Id: I69a423c592e2fdd582b3c40099137b4ef3d05b39

3 years agoMigrate to Apache MINA sshd 2.6.0 and Orbit I20210203173513 24/166824/21
David Ostrovsky [Sat, 25 Jul 2020 08:00:11 +0000 (10:00 +0200)]
Migrate to Apache MINA sshd 2.6.0 and Orbit I20210203173513

Re-enable DSA, DSA_CERT, and RSA_CERT public key authentication.
DSA is discouraged for a long time already, but it might still be
way too disruptive to completely drop it. RSA is discouraged for
far less long, and dropping that would be really disruptive.

Adapt to the changed property handling. Remove work-arounds for
shortcomings of earlier sshd versions.

Use Orbit I20210203173513, which includes sshd 2.6.0. This also bumps
apache.httpclient to 4.5.13 and apache.httpcore to 4.4.14.

Change-Id: I2d24a1ce4cc9f616a94bb5c4bdaedbf20dc6638e
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoLFSPointerTest: suppress errorprone error [SelfComparison] 70/175770/1
Matthias Sohn [Thu, 4 Feb 2021 00:17:00 +0000 (01:17 +0100)]
LFSPointerTest: suppress errorprone error [SelfComparison]

The test #testCompareToSame tests comparing against self intentionally.
Suppress the error raised by errorprone.

Change-Id: If8d70a51ab34ffb6d7f0c9d409746aee8b031408
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoFix bazel tests broken by classes moved in dbd05433 69/175769/1
Matthias Sohn [Thu, 4 Feb 2021 00:14:30 +0000 (01:14 +0100)]
Fix bazel tests broken by classes moved in dbd05433

Change-Id: I88a3547c4b52bcf28c0f0f548ba1bb41a7787704
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge branch 'stable-5.11' 63/175763/1
Matthias Sohn [Wed, 3 Feb 2021 21:57:33 +0000 (22:57 +0100)]
Merge branch 'stable-5.11'

* stable-5.11:
  Prepare 5.11.0-SNAPSHOT builds
  JGit v5.11.0.202102031030-m2

Change-Id: I3cada72e1917e6349ee415e29987165692d65c4b

3 years agoPrepare 5.11.0-SNAPSHOT builds 52/175752/1
Matthias Sohn [Wed, 3 Feb 2021 20:17:22 +0000 (21:17 +0100)]
Prepare 5.11.0-SNAPSHOT builds

Change-Id: I191674448c4a220e61ec5f0c181c0809eb873166
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoJGit v5.11.0.202102031030-m2 24/175724/1 v5.11.0.202102031030-m2
Matthias Sohn [Wed, 3 Feb 2021 15:29:21 +0000 (16:29 +0100)]
JGit v5.11.0.202102031030-m2

Change-Id: Ie14c162a7fc5e1e8f34bf4bbc944f4dbe13e4dd0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoIO: fix IO.readFully(InputStream, byte[], int) 21/175521/2
Thomas Wolf [Fri, 29 Jan 2021 22:02:30 +0000 (23:02 +0100)]
IO: fix IO.readFully(InputStream, byte[], int)

This would run into an endless loop if the offset given was not zero.
Fix the logic to exit the read loop when the buffer is full.

Luckily all existing uses of this method call it only with offset zero.

Change-Id: I0ec2a4fb43efe4a605d06ac2e88cf155d50e2f1e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoLFS: make pointer parsing more robust 94/175494/3
Thomas Wolf [Fri, 29 Jan 2021 11:12:28 +0000 (12:12 +0100)]
LFS: make pointer parsing more robust

Parsing an LFS pointer must check the input more to not run into
exceptions. LfsPoint.parseLfsPointer() is used in various places to
determine whether a blob is a LFS pointer; it is not only called with
valid LFS pointers. Tighten the validations and return null if they
fail. All callers already do check for a null return value.

Also, LfsPointer implemented Comparable but did not override equals().
This is rather unusual and actually warned against in the javadoc of
Comparable. Implement equals() and hashCode().

Add more tests.

Bug: 570744
Change-Id: I90ca264d0a250275cf1907e9dcfcee5eab80df0f
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years ago[pgm] add option --timeout to clone command 99/175499/1
Matthias Sohn [Fri, 29 Jan 2021 15:23:29 +0000 (16:23 +0100)]
[pgm] add option --timeout to clone command

Change-Id: I2ee74755045a8d9971ea0d9426db405829c7c679
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge "Move reachability checker generation into the ObjectReader object"
Jonathan Nieder [Fri, 29 Jan 2021 06:52:13 +0000 (01:52 -0500)]
Merge "Move reachability checker generation into the ObjectReader object"

3 years agoMove reachability checker generation into the ObjectReader object 00/175300/6
Terry Parker [Mon, 25 Jan 2021 06:18:21 +0000 (22:18 -0800)]
Move reachability checker generation into the ObjectReader object

Reachability checkers are retrieved from RevWalk and ObjectWalk objects:
* RevWalk.createReachabilityChecker()
* ObjectWalk.createObjectReachabilityChecker()

Since RevWalks and ObjectWalks are themselves directly instantiated
in hundreds of places (e.g. UploadPack...) overriding them in a
consistent way requires overloading 100s of methods, which isn't
feasible. Moving reachability checker generation to a more central
place solves that problem.

The ObjectReader object seems a good place from which to get
reachability checkers, because reachability checkers return
information about relationships between objects. ObjectDatabases
delegate many operations to ObjectReaders, and reachability bitmaps
are attached to ObjectReaders.

The Bitmapped and Pedestrian reachability checker objects were
package private in the org.eclipse.jgit.revwalk package. This change
makes them public and moves them to the
org.eclipse.jgit.internal.revwalk package. Corresponding tests are
also moved.

Motivation:
1) Reachability checking algorithms need to scale. One of the
   internal Android repositories has ~2.4 million refs/changes/*
   references, causing bad long tail performance in reachability
   checks.
2) Reachability check performance is impacted by repository
   topography: number of refs, number of objects, amounts of
   related vs. unrelated history.
3) Reachability check performance is also affected by per-branch
   access (Gerrit branch permissions) since different users can
   see different branches.
4) Reachability check performance isn't affected by any state in a
   RevWalk or ObjectWalk.

I don't yet know if a single algorithm will work for all cases in #2
and #3. We may need to evolve the ReachabilityChecker interfaces
over time to solve the Gerrit branch permissions case, or use
Gerrit-specific identity information to solve that in an efficient
way.

This change takes the existing public API and moves it to the
ObjectReader/whole repository level, which is where we can do
consistent customizations for #2 and #3. We intend to upstream the
best of whatever works, but anticipate the need for multiple rounds
of experimentation.

Change-Id: I9185feff43551fb387957c436112d5250486833d
Signed-off-by: Terry Parker <tparker@google.com>
3 years agoAdd the "compression-level" option to all ArchiveCommand formats 16/175216/9
Youssef Elghareeb [Fri, 22 Jan 2021 14:30:25 +0000 (15:30 +0100)]
Add the "compression-level" option to all ArchiveCommand formats

Different archive formats support a compression level in the range
[0-9]. The value 0 is for lowest compressions and 9 for highest. Highest
levels produce output files of smaller sizes but require more memory to
do the compression.

This change allows passing a "compression-level" option to the git
archive command and implements using it for different file formats.

Change-Id: I5758f691c37ba630dbac24db67bb7da827bbc8e1
Signed-off-by: Youssef Elghareeb <ghareeb@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoMerge changes I36d9b63e,I8c5db581,I2c02e89c
Jonathan Tan [Wed, 27 Jan 2021 18:36:43 +0000 (13:36 -0500)]
Merge changes I36d9b63e,I8c5db581,I2c02e89c

* changes:
  Compare getting all refs except specific refs with seek and with filter
  Add getsRefsByPrefixWithSkips (excluding prefixes) to ReftableDatabase
  Add seekPastPrefix method to RefCursor

3 years agoCompare getting all refs except specific refs with seek and with filter 98/173498/13
Gal Paikin [Mon, 7 Dec 2020 14:18:34 +0000 (15:18 +0100)]
Compare getting all refs except specific refs with seek and with filter

There are currently two ways to get all refs except a specific ref, we
add two methods that perform both and compare the two different approaches.

This change adds two methods that compares the two different approaches
of such query:
1. Get all the refs, and then filter by refs that don't start with the
prefix (current approach).
2. Get all refs until encountering a ref that is part of the prefix we
should exclude, skip using seekPastPrefix, and continue (new approach).
This works since the refs are sorted.

Specifically in Gerrit, we often have thousands of refs that are not
refs/changes, and millions of refs/changes, hence the second approach
should be much faster. In Jgit in general it's still expected to provide
a better result even if we're skipping a smaller chunk of the refs
since the complexity here is O(logn) with a binary search, rather than
O(number of skipped refs).

We ran this benchmark on a big chunk of chromium/src's reftable. To run
it, we first create the reftable:

  git ls-remote https://chromium.googlesource.com/chromium/src > lsr

  bazel build org.eclipse.jgit.pgm:jgit && rm -rf /tmp/reftable* && \
    ./bazel-bin/org.eclipse.jgit.pgm/jgit debug-benchmark-reftable \
    --test write_stack lsr /tmp/reftable

Then, we actually test the created reftable. Note that we can't test all
of them at once since there are multiple ones, but below is a good
example.

bazel build org.eclipse.jgit.pgm:jgit  && \
./bazel-bin/org.eclipse.jgit.pgm/jgit debug-benchmark-reftable \
--test get_refs_excluding_ref --ref refs/changes \
lsr /tmp/reftable/000000000001-0000001e0371.ref

Result:
total time the action took using seek:      36925 usec
total time the action took using filter:     874382 usec
number of refs that start with prefix: 4266.
number of refs that don't start with prefix: 1962695.

Similarly for Android's biggest repository, platform/frameworks/base
(still only partial result):
total time the action took using seek:       9020 usec
total time the action took using filter:     143166 usec
number of refs that start with prefix: 296.
number of refs that don't start with prefix: 60400.

In conclusion, it's easy to see an improvement of a factor of 15-20x for
large Gerrit repositories!

Signed-off-by: Gal Paikin <paiking@google.com>
Change-Id: I36d9b63eb259804c774864429cf2c761cd099cc3

3 years agoAdd getsRefsByPrefixWithSkips (excluding prefixes) to ReftableDatabase 22/173022/20
Gal Paikin [Mon, 30 Nov 2020 14:57:06 +0000 (15:57 +0100)]
Add getsRefsByPrefixWithSkips (excluding prefixes) to ReftableDatabase

We sometimes want to get all the refs except specific prefixes,
similarly to getRefsByPrefix that gets all the refs of a specific
prefix.

We now create a new method that gets all refs matching a prefix except a
set of specific prefixes.

One use-case is for Gerrit to be able to get all the refs except
refs/changes; in Gerrit we often have lots of refs/changes, but very
little other refs. Currently, to get all the refs except refs/changes we
need to get all the refs and then filter the refs/changes, which is very
inefficient. With this method, we can simply skip the unneeded prefix so
that we don't have to go over all the elements.

RefDirectory still uses the inefficient implementation, since there
isn't a simple way to use Refcursor to achieve the efficient
implementation (as done in ReftableDatabase).

Signed-off-by: Gal Paikin <paiking@google.com>
Change-Id: I8c5db581acdeb6698e3d3a2abde8da32f70c854c

3 years agoWrap the Files.list returned Stream in a try-with-resources block 48/175348/6
Terry Parker [Tue, 26 Jan 2021 02:46:25 +0000 (18:46 -0800)]
Wrap the Files.list returned Stream in a try-with-resources block

Adds a new FileUtils.hasFiles(Path) helper method to correctly handle
the Files.list returned Stream.

These errors were found by compiling the code using JDK11's
javac compiler.

Change-Id: Ie8017fa54eb56afc2e939a2988d8b2c5032cd00f
Signed-off-by: Terry Parker <tparker@google.com>
3 years agoAdd seekPastPrefix method to RefCursor 21/173021/17
Gal Paikin [Thu, 19 Nov 2020 17:05:04 +0000 (18:05 +0100)]
Add seekPastPrefix method to RefCursor

This method will be used by the follow-up change. This useful if we want
to go over all the changes after a specific ref.

For example, the new method allows us to create a follow-up that would
go over all the refs until we reach a specific ref (e.g refs/changes/),
and then we use seekPastPrefix(refs/changes/) to read the rest of the refs,
thus basically we return all refs except a specific prefix.

When seeking past a prefix, the previous condition that created the
RefCursor still applies. E.g, if the cursor was created by
seekRefsWithPrefix, we can skip some refs but we will not return refs
that are not starting with this prefix.

Signed-off-by: Gal Paikin <paiking@google.com>
Change-Id: I2c02e89c877fe90da8619cb8a4a9a0c865f238ef

3 years agoreftable: add random suffix to table names 27/175327/2
Han-Wen Nienhuys [Mon, 25 Jan 2021 15:54:52 +0000 (16:54 +0100)]
reftable: add random suffix to table names

In some circumstances (eg. compacting a stack that has deletions), the
result may have a {min, max} range that already exists. In these
cases, we would rename onto an already existing file, which does not
work on Windows. By adding a random suffix, we disambiguate the files,
and avoid this failure scenario.

Change-Id: I0273f99bb845cfbdbd8cdd582b55d3c310505d29
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
3 years agoTemporaryBuffer: fix toByteArray(limit) 47/175247/2
Thomas Wolf [Fri, 22 Jan 2021 21:45:48 +0000 (22:45 +0100)]
TemporaryBuffer: fix toByteArray(limit)

Heap always copied whole blocks, which leads to AIOOBEs. LocalFile
didn't overwrite the method and thus caused NPEs.

Change-Id: Ia37d4a875df9f25d4825e6bc95fed7f0dff42afb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoCorrect the minimum required version of Apache httpclient 99/174999/1
Thomas Wolf [Mon, 18 Jan 2021 15:05:58 +0000 (16:05 +0100)]
Correct the minimum required version of Apache httpclient

org.eclipse.jgit.http.apache uses several features that exist only
since httpclient 4.4, but its MANIFEST.MF still had a lower bound of
4.3.0. Bump this to 4.4.0 for all packages from httpclient. 4.3.0 for
the packages from httpcore is fine.

Do a similar clean-up in the other bundles using packages from Apache
httpclient (http.test, lfs, lfs.server, lfs.server.test)

Bug: 570451
Change-Id: Iffdde2a9bd0d65db2e5201a08cffbf03597e2866
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoMerge "pgm: add missing dependency to org.apache.commons.logging"
Matthias Sohn [Mon, 18 Jan 2021 07:44:39 +0000 (02:44 -0500)]
Merge "pgm: add missing dependency to org.apache.commons.logging"

3 years agoMerge "[spotbugs]: Fix potential NPE in FileSnapshot constructor"
Matthias Sohn [Mon, 18 Jan 2021 07:44:13 +0000 (02:44 -0500)]
Merge "[spotbugs]: Fix potential NPE in FileSnapshot constructor"

3 years agopgm: add missing dependency to org.apache.commons.logging 90/174190/2
Matthias Sohn [Sun, 3 Jan 2021 01:06:24 +0000 (02:06 +0100)]
pgm: add missing dependency to org.apache.commons.logging

Without this dependency I get class loading exceptions when trying to
run org.eclipse.jgit.pgm.Clone in Eclipse.

Change-Id: Ia9ecb385d3baccbcd041114287af5076fefd3d71
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years ago[pgm] Use Apache sshd by default 75/174875/2
Thomas Wolf [Fri, 15 Jan 2021 11:45:35 +0000 (12:45 +0100)]
[pgm] Use Apache sshd by default

Change-Id: I2a00059415fc2674469bc921827362801583af83
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoTransportHttp: support preemptive Basic authentication 20/173320/3
Thomas Wolf [Wed, 2 Dec 2020 22:07:01 +0000 (23:07 +0100)]
TransportHttp: support preemptive Basic authentication

If the caller knows already HTTP Basic authentication will be needed
and if it also already has the username and password, preemptive
authentication is a little bit more efficient since it avoids the
initial 401 response.

Add a setPreemptiveBasicAuthentication(username, password) method
to TransportHttp. Client code could call this for instance in a
TransportConfigCallback. The method throws an IllegalStateException
if it is called after an HTTP request has already been made.

Additionally, a URI can include userinfo. Although it is not
recommended to put passwords in URIs, JGit's URIish and also the
Java URL and URI classes still allow it. The underlying HTTP
connection may omit these fields though. If present, take these
fields as additional source for preemptive Basic authentication if
setPreemptiveBasicAuthentication() has not been called.

No preemptive authentication will be done if the connection is
redirected to a different host.

Add tests.

Bug: 541327
Change-Id: Id00b975e56a15b532de96f7bbce48106d992a22b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoTransportHttp: shared SSLContext during fetch or push 09/173009/5
Thomas Wolf [Mon, 30 Nov 2020 11:14:22 +0000 (12:14 +0100)]
TransportHttp: shared SSLContext during fetch or push

TransportHttp makes several HTTP requests. The SSLContext and socket
factory must be shared over these requests, otherwise authentication
information may not be propagated correctly from one request to the
next. This is important for authentication mechanisms that rely on
client-side state, like NEGOTIATE (either NTLM, if the underlying HTTP
library supports it, or Kerberos). In particular, SPNEGO cannot
authenticate on a POST request; the authentication must come from the
initial GET request, which implies that the POST request must use the
same SSLContext and socket factory that was used for the GET.

Change the way HTTPS connections are configured. Introduce the concept
of a GitSession, which is a client-side HTTP session over several HTTPS
requests. TransportHttp creates such a session and uses it to configure
all HTTP requests during that session (fetch or push). This gives a way
to abstract away the differences between JDK and Apache HTTP connections
and to configure SSL setup outside.

A GitSession can maintain state and thus give all HTTP requests in a
session the same socket factory.

Introduce an extension interface HttpConnectionFactory2 that adds a
method to obtain a new GitSession. Implement this for both existing
HTTP connection factories. Change TransportHttp to use the new
GitSession to configure HTTP connections.

The old methods for disabling SSL verification still exist to support
possibly external connection and connection factory implementations
that do not make use of the new GitSession yet.

Bug: 535850
Change-Id: Iedf67464e4e353c1883447c13c86b5a838e678f1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoTransportHttp: make the connection factory configurable 08/173008/4
Thomas Wolf [Sun, 29 Nov 2020 15:22:39 +0000 (16:22 +0100)]
TransportHttp: make the connection factory configurable

Previously, TransportHttp always used the globally set connection
factory. This is problematic if that global factory is changed in
the middle of a fetch or push operation. Initialize the factory to
use in the constructor, then use that factory for all HTTP requests
made through this transport. Provide a setter and a getter for it
so that client code can customize the factory, if needed, in a
TransportConfigCallback.

Once a factory has been used on a TransportHttp instance it cannot
be changed anymore.

Make the global static factory reference volatile.

Change-Id: I7c6ee16680407d3724e901c426db174a3125ba1c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoAdd target platform for eclipse 4.19 staging 12/174712/2
Matthias Sohn [Tue, 12 Jan 2021 21:39:51 +0000 (22:39 +0100)]
Add target platform for eclipse 4.19 staging

Change-Id: I172aa01203edc8cd069c93762c482a3f21950acc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoUpdate orbit to S20210105214148 and com.google.gson to 2.8.6 11/174711/2
Matthias Sohn [Tue, 12 Jan 2021 21:39:08 +0000 (22:39 +0100)]
Update orbit to S20210105214148 and com.google.gson to 2.8.6

Also fix target name of jgit-4.18 target.

Change-Id: Ib5a13281398e7c5589de2e4a34fae0d08346db9c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoUpdate spotbugs-maven-plugin to 4.2.0 11/174511/2
Marco Miller [Fri, 8 Jan 2021 21:11:30 +0000 (16:11 -0500)]
Update spotbugs-maven-plugin to 4.2.0

Change-Id: I094d92f95ec2ab1cefa8f1504f2a8c1aa6e28d96
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
3 years agoAdd org.eclipse.jetty.util.ajax to target platform and bazel deps 65/174665/3
Matthias Sohn [Mon, 11 Jan 2021 23:28:23 +0000 (00:28 +0100)]
Add org.eclipse.jetty.util.ajax to target platform and bazel deps

It is required by org.eclipse.jetty.servlet [1].

[1] https://search.maven.org/artifact/org.eclipse.jetty/jetty-servlet/9.4.35.v20201120/jar

Change-Id: I307ec8bad17f3a703bf25cc16c87ab9b524a84ff
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoTag message must not include the signature 05/174405/3
Thomas Wolf [Thu, 7 Jan 2021 16:10:45 +0000 (17:10 +0100)]
Tag message must not include the signature

Signatures on tags are just tacked onto the end of the message.
Getting the message must not return the signature. Compare [1]
and [2] in C git, which both drop a signature at the end of an
object body.

[1] https://github.com/git/git/blob/21bf933/builtin/tag.c#L173
[2] https://github.com/git/git/blob/21bf933/ref-filter.c#L1276

Change-Id: Ic8a1062b8bc77f2d7c138c3fe8a7fd13b1253f38
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years agoUpdate jetty to 9.4.35.v20201120 12/174212/2
Matthias Sohn [Mon, 4 Jan 2021 01:02:27 +0000 (02:02 +0100)]
Update jetty to 9.4.35.v20201120

Change-Id: I203778ea0536defffb720bcf7cdcbc6258540e65
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 years agoProtocol V2: don't log spurious ACKs in UploadPack 18/174318/1
Thomas Wolf [Wed, 6 Jan 2021 11:13:48 +0000 (12:13 +0100)]
Protocol V2: don't log spurious ACKs in UploadPack

UploadPack may log ACKs in protocol V2 that it doesn't send (if it
got a "done" from the client), or may log ACKs twice. That makes
packet log analysis difficult.

Add a new constructor to PacketLineOut to omit all logging from an
instance, and use it in UploadPack.

Change-Id: Ic29ef5f9a05cbcf5f4858a4e1b206ef0e6421c65
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>