As discussed in the bug, jgit should not include a logging
implementation, and instead rely on the product containing jgit to
configure the logging.

We have recently run into the situation, that installing egit in a (non
eclipse.org) RCP application breaks all the logging due to incompatible
logging implementations. Removal of the jgit logging implementation
should fix this.

Following further changes have been done for jgit command line:
* added log4j.properties to binary build of jgit.pgm. That file existed
in the git repository, but was not included in the eclipse binary build.
(maybe it is in the bazel build)
* removed apache.commons.logging package import from jgit.pgm. That
import is not used, and makes the logging even more confusing.

Bug: 514326
Change-Id: I6dc7d1462f0acfca9e2b1ac87e705617179ffdda
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Motivation: JSch serves as 'default' implementations of the SSH
transport. If a client application does not use it then there is no need
to pull in this dependency.

Move the classes depending on JSch to an OSGi fragment extending the
org.eclipse.jgit bundle and keep them in the same package as before
since moving them to another package would break API. Defer moving them
to a separate package to the next major release.

Add a new feature org.eclipse.jgit.ssh.jsch feature to enable
installation. With that users can now decide which of the ssh client
integrations (JCraft JSch or Apache Mina SSHD) they want to install.
We will remove the JCraft JSch integration in a later step due to the
reasons discussed in bug 520927.

Bug: 553625
Change-Id: I5979c8a9dbbe878a2e8ac0fbfde7230059d74dc2
Also-by: Michael Dardis <git@md-5.net>
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Ostrovsky <david@ostrovsky.org>

Motivation: BouncyCastle serves as 'default' implementation of
the GPG Signer. If a client application does not use it there is no need
to pull in this dependency, especially since BouncyCastle is a large
library.

Move the classes depending on BouncyCastle to an OSGi fragment extending
the org.eclipse.jgit bundle. They are moved to a distinct internal
package in order to avoid split packages. This doesn't break public API
since these classes were already in an internal package before this
change.

Add a new feature org.eclipse.jgit.gpg.bc to enable installation. With
that users can now decide if they want to install it.

Attempts to sign a commit if org.eclipse.jgit.gpg.bc isn't available
will result in ServiceUnavailableException being thrown.

Bug: 559106
Change-Id: I42fd6c00002e17aa9a7be96ae434b538ea86ccf8
Also-by: Michael Dardis <git@md-5.net>
Signed-off-by: Michael Dardis <git@md-5.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: David Ostrovsky <david@ostrovsky.org>

If the determination of the user home directory produces a Java File
object with an invalid path, spurious exceptions may occur at the
most inopportune moments anytime later. In the case in the linked bug
report, start-up of EGit failed, leading to numerous user-visible
problems in Eclipse.

So validate the return value of FS.userHomeImpl(). If converting that
File to a Path throws an exception, log the problem and fall back to
Java system property user.home. If that also is not valid, use null.

(A null user home directory is allowed by FS, and calling in Java
new File(null, "some_string") is fine and produces a File relative
to the current working directory.)

Bug: 563739
Change-Id: If9eec0f9a31a45bd815231706285c71b09f8cf56
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Make it possible to programmatically suppress the JMX bean
registration. In EGit it is not needed but can be rather costly
because it occurs during plug-in activation and accesses the
git user config.

Bug: 563740
Change-Id: I07ef7ae2f0208d177d2a03862846a8efe0191956
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Only the presence or absence of whitespace is significant; but not the
actual whitespace characters. Don't compare whitespace bytes.

Compare the C git implementation at [1].

[1] https://github.com/git/git/blob/0d0e1e8/xdiff/xutils.c#L173

Bug: 563570
Change-Id: I2d0522b637ba6b5c8b911b3376a9df5daa9d4c27
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

This reverts commit 3aee92478c2cbc67cd921533437b824e43ed9798, which
increased fetch latency significantly.

Change-Id: Id31a94dff83bf7ab2121718ead819bd08306a0b6
Signed-off-by: Yunjie Li <yunjieli@google.com>

Change-Id: I6fe26c1efcf812de3102ee82ce67f9e0bc3b0135
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

* replace 4.15 staging by 4.15 release
* add 4.16 staging

Change-Id: I33cc701f0ea6fffeaec898fa50ef536b20102048
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Avoid using a key written to a file. This makes it clearer that
the test does not rely on files being present.

Change-Id: I31cf4f404aab5b891c32fc4bda906b7f8fe03777
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

A builder API provides a more convenient way to define a customized
SshdSessionFactory by hiding the subclassing.

Also provide a new interface SshConfigStore to abstract away the
specifics of reading a ssh config file, and provide a way to customize
the concrete ssh config implementation to be used. This facilitates
using an alternate ssh config implementation that may or may not be
based on files.

Change-Id: Ib9038e8ff2a4eb3a9ce7b3554d1450befec8e1e1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Avoid trying other authentication methods on SocketException or on
InterruptedIOException. SocketException is rather fatal, such as
nothing listening on the peer's port, connection reset, or it could
be a connection time-out.

Time-outs enforced by Timeout{Input,Output}Stream may result in
InterruptedIOException being thrown.

In both cases, it makes no sense to try other authentication methods,
and doing so may wrongly report "authentication not supported" or
"cannot open git-upload-pack" or some such instead of reporting a
time-out.

Bug: 563138
Change-Id: I0191b1e784c2471035e550205abd06ec9934fd00
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Config core.eol is to be ignored if core.autocrlf is true or input.[1]
JGit didn't do so when core.autocrlf=input was set.

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-coreeol

Bug: 561877
Change-Id: I5e62e0510d160b5113c1090319af09c2bc1bcb59
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

In Git 2.10.0 the interpretation of gitattributes changed or was fixed
such that "* text=auto eol=crlf" would indeed still do auto-detection
of text vs. binary content.[1] Previously this was identical to
"* text eol=crlf", i.e., treating all files as text.

JGit still did the latter, which caused surprises because it changed
binary files.

[1] https://github.com/git/git/blob/master/Documentation/RelNotes/2.10.0.txt#L248

Bug: 561341
Change-Id: I5b6fb97b5e86fd950a98537b6b8574f768ae30e5
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Bazel buildtools project includes in addition to buildifier also unused
deps and buildozer utilities, that detect unused dependencies and fix
them by applying the removal to the build files. This change is created
by installing unused_deps from buildtools@HEAD and running:

  $ unused_deps //...

and applying the suggested modifications.

Change-Id: Iad74ec2fa719475b29391586f40b13ae30477004
Signed-off-by: David Ostrovsky <david@ostrovsky.org>

Log the stack trace in order to help understanding the bug 563380

Bug: 563380
Change-Id: If993a63ccec5042b10e1d5e945b18f4b5f06d8ff
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: I5bc120a495deb2f3b29bd04bb1c9a2058394ba8a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Update dependency to Bouncy Castle to 1.65.

Add the IssuerFingerprint as a hashed sub-packet in the signature. If
added unhashed, GPG ignores it.

Bug: 553206
Change-Id: I6807e8e2385e6ec5790f388e4753a44aa9474ebb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Test plan:

  bazel build //...

Change-Id: Ibf1d0b56785d62150bbae49b553c856efbe6d197
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

OSGi semantic versioning allows breaking implementers in a minor
release.

Change-Id: Ib55dc43dd3b50b0ef39a7094190f230210aee4b6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This change was introduced in 5.8.

Change-Id: Ic74ebff5a0547bb55e0401b38f73ebc6e67cace9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

* changes:
  PackBitmapIndex: Set distance threshold
  PackBitmapIndex: Not buffer inflated bitmap in BasePackBitmapIndex
  PackBitmapIndex: Remove convertedBitmaps in the Remapper
  PackBitmapIndex: Reduce memory usage in GC
  PackBitmapIndex: Add AddToBitmapWithCacheFilter class
  PackBitmapIndex: Add util methods and builder to BitmapCommit
  PackBitmapIndex: Move BitmapCommit to a top-level class
  Refactor: Make retriveCompressed an method of the Bitmap class

Setting the distance threshold to 2000 in PackWriterBitmapPreparer to
reduce memory usage in garbage collection. When the threshold is 0, GC
for the msm repository would use about 37 GB memory to complete. After
setting it to 2000, GC can finish in 75 min with about 10 GB memory.

Change-Id: I39783eeecbae58261c883735499e61ee1cac75fe
Signed-off-by: Yunjie Li <yunjieli@google.com>

Currently we're buffering the inflated bitmap entry in BasePackBitmapIndex
to optimize running time. However, this will use lots of memory during
the construction of the pack bitmap index file which may cause failure of
garbage collection.

The running time didn't increase significantly, if there's any increase,
after removing the buffering here. The report about usage of time/memory
will come in the next commit.

Change-Id: I874503ecc85714acab7ca62a6a7968c2dc0b56b3
Signed-off-by: Yunjie Li <yunjieli@google.com>

The convertedBitmaps serves for time-optimization purpose. But it's
actually not saving time much but using lots of memory. So remove the
field here to save memory.

Currently the remapper class is only used in the construction of the
bitmap index file. And during the preparation of the file, we're only
getting bitmaps from the remapper when finding objects accessible from
a commit, so bitmap associated with each commit will only be fetched once
and thus the convertedBitmaps would hardly be read, which means that it's
not saving time.

Change-Id: Ic942a8e485135fb177ec21d09282d08ca6646fdb
Signed-off-by: Yunjie Li <yunjieli@google.com>

Currently, the garbage collection is consistently failing for some large
repositories in the building bitmap phase, e.g.Linux-MSM project:
https://source.codeaurora.org/quic/la/kernel/msm-3.18

Historically, bitmap index creation happened in 3 phases:
1. Select the commits to which bitmaps should be attached.
2. Create all bitmaps for these commits, stored in uncompressed format
in the PackBitmapIndexBuilder.
3. Deltify the bitmaps and write them to disk.

We investigated the process. For phase 2 it's most efficient to create
bitmaps starting with oldest commit and moving to the newest commit,
because the newer commits are able to reuse the work for the old ones.
But for bitmap deltification in phase 3, it's better when a newer
commit's bitmap is the base, and the current disk format writes bitmaps
out for the newest commits first.

This change introduces a new collection to hold the deltified and
compressed representations of the bitmaps, keeping a smaller subset of
commits in the PackBitmapIndexBuilder to help make the bitmap index
creation more memory efficient.

And in this commit, we're setting DISTANCE_THRESHOLD to 0 in the
PackWriterBitmapPreparer, which means the garbage collection will not
have much behavoir change and will still use as much memory as before.

Change-Id: I6ec2c3e8dde11805af47874d67d33cf1ef83660e
Signed-off-by: Yunjie Li <yunjieli@google.com>

Add a new revwalk filter, AddToBitmapWithCachedFilter. This filter updates
a client-provided {@code BitmapBuilder} as a side effect of a revwalk.
Similar to {@code AddToBitmapFilter}, it short circuits the walk when it
encounters a commit which is included in the provided bitmap's BitmapIndex.
It also short circuits the walk if it encounters the client-provided
cached commit.

Change-Id: I62cb503016f4d3995d648d92b82baab7f93549a9
Signed-off-by: Yunjie Li <yunjieli@google.com>

Add some utility methods and a builder class for BitmapCommit class in
preparation for improving the memory footprint of GC's bitmap generation
phase.

Change-Id: Ice3d257fc26f3917a65a64eaf53b508b89043caa
Signed-off-by: Yunjie Li <yunjieli@google.com>

Move BitmapCommit from inside the PackWriterBitmapPreparer to a new
top-level class in preparation for improving the memory footprint of GC's
bitmap generation phase.

Change-Id: I4d404a5b3a34998b441d23105197f33d32d39670
Signed-off-by: Yunjie Li <yunjieli@google.com>

Make retrieveCompressed() a method of Bitmap interface to avoid type
casting and later reuse in improving the memory footprint of GC's bitmap
generation phase.

Change-Id: I098d85105cf17af845d43b8c71b4ca48b02fd7da
Signed-off-by: Yunjie Li <yunjieli@google.com>

When downloading LFS objects also accept response code 203 as successful
download. This response may be seen when downloading via a proxy.

Bug: 563022
Change-Id: Iee85fdb451b33369d08859872e5bfc2a67dffa6d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

* changes:
  ReceivePack: adding IterativeConnectivityChecker
  Moving transport/internal -> internal/transport

By default, it will generate hostname using the aws region passed to the
constructor.

This will allow for easier testing, since you can just spin up a local
minio (or other s3-compatible storage service) instance and point the
application at that for the storage mechanism.

It will also allow for storing lfs objects on-prem.

Change-Id: I2566b1fcce58f3d306ddd23a8da702ef5a451c7b
Signed-off-by: Pat Long <pllong@arista.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Introduce an IterativeConnectivityChecker which runs a connectivity
check with a filtered set of references, and falls back to using the
full set of advertised references.

It uses references during first check attempt:
- References that are ancestors of an incoming commits (e.g., pushing
a commit onto an existing branch or pushing a new branch based on
another branch)
- Additional list of references we know client can be interested in
(e.g. list of open changes for Gerrit)

We tested it inside Google and it improves connectivity for certain
topologies. For example connectivity counts for
chromium.googlesource.com/chromium/src:

percentile_50: 1923 (was: 22777)
percentile_90: 23272 (was: 353003)
percentile_99: 345522 (was: 353435)

This saved ~2 seconds on every push to this repository.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I6543c2e10ed04622ca795b195665133e690d3b10

Moving transport related internal classes into dedicated subpackage in
o/e/j/internal package.

Signed-off-by: Demetr Starshov <dstarshov@google.com>
Change-Id: I21ed029d359f5f7d8298f102efbb4b1dcdf404ad

Fix NullPointerException occurring when calling
CheckoutCommand with forced == true option when
the branch isn't changed and there is deleted
uncommitted file.

Change-Id: I99bf1fc25e6889f07092320d7bc2772ec5d341b5
Signed-off-by: Nail Samatov <sanail@yandex.ru>

* stable-5.8:
  Prepare 5.8.0-SNAPSHOT builds
  JGit v5.8.0.202005061305-m2

Change-Id: I7e95ddbedfae23035cc132abdf574545ffcda091

Change-Id: Ic7d763b3b02b4d2a7a4141d2d9d715855dd69918
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: I9d9e5a9e000a74513896744cc3646430cb872893
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Bug: 553206
Change-Id: I867bbe88b847bf6d17f17f6cf8c565cd4a2437cc
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This should be faster.

Change-Id: I404ec5e66731b3cf7a8e621cf1ff8748d109ea69
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

When applying a patch that contains renames or copies using ApplyCommand,
also apply all hunks that apply to the renamed or copied file.

Change-Id: I9f3fa4370458bd7c14beeb2e2b49e846d70203cb
Signed-off-by: Jack Wickham <jwickham@palantir.com>

Before this change, applying a patch will fail if the destination directory
doesn't exist; after, the necessary parent directories are created.

If renaming the file fails, the directories won't be deleted, so this change
isn't atomic. However, ApplyCommand is already not atomic - if one hunk fails
to apply, other hunks still get applied - so I don't think that is a blocker.

Change-Id: Iea36138b806d4e7012176615bcc673756a82f365
Signed-off-by: Jack Wickham <jwickham@palantir.com>

ObjectReachabilityChecker interface is the only public API. The
implementation is instantiated by ObjectWalk and doesn't need to be
visible outside the package.

Change-Id: I5b97bb98990cded637686bdc15c9655330b7780f
Signed-off-by: Ivan Frade <ifrade@google.com>

Change-Id: Ie7c3481aba515d9c9b0a152db17b5a9dc74e7ede
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Change-Id: I020ff281eaa49794aeb1853711485bc8c7047dd8
Signed-off-by: David Ostrovsky <david@ostrovsky.org>

Change-Id: I41d927e53f040750bbf0160dc15d8ca8a3f9f7ca
Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>

It is only assigned on initialization.

Change-Id: I36f68b3f511236e66cc7a4f56ba49252b317276a
Signed-off-by: Ivan Frade <ifrade@google.com>