Enable UnusedException at ERROR level which causes the build to fail
in many places with:
[UnusedException] This catch block catches an symbol and re-throws
another, but swallows the caught symbol rather than setting it as a
cause. This can make debugging harder.
Fix it by setting the caught exception as cause on the subsequently
thrown exception.
Note: The grammatically incorrect error message is copy-pasted as-is
from the version of ErrorProne currently used in Bazel; it has been
fixed by [1] in the latest version.
[1] https://github.com/google/error-prone/commit/d57a39c
Change-Id: I11ed38243091fc12f64f1b2db404ba3f1d2e98b5
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Store filesystem timestamp resolution in extra jgit config
This avoids polluting hand-crafted user level config with
auto-configured options which might disturb in environments where
the user level config is replicated between different machines.
Add a jgit config as parent of the system level config. Persist
measured timestamp resolutions always in this jgit config and read it
via the user global config. This has the effect that auto-configured
timestamp resolution will be used by default and can be overridden in
either the system level or user level config.
Store the jgit config under the XDG_CONFIG_HOME directory following the
XDG base directory specification [1] in order to ensure that we have
write permissions to persist the file. This has the effect that each OS
user will use its jgit config since they typically use different
XDG_CONFIG_HOME directories.
If the environment variable XDG_CONFIG_HOME is defined the jgit config
file is located at $XDG_CONFIG_HOME/jgit/config otherwise the default is
~/.config/jgit/config.
If you want to avoid redundant measurement for different OS users
manually copy the values measured and auto-configured for one OS user to
the system level git config.
[1] https://wiki.archlinux.org/index.php/XDG_Base_Directory
Bug: 551850
Change-Id: I0022bd40ae62f82e5b964c2ea25822eb55d94687
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Enable and fix "Statement unnecessarily nested within else clause" warnings
Since [1] the gerrit project includes jgit as a submodule, and has this
warning enabled, resulting in 100s of warnings in the console.
Also enable the warning here, and fix them.
At the same time, add missing braces around adjacent and nearby one-line
blocks.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/227897
Change-Id: I81df3fc7ed6eedf6874ce1a3bedfa727a1897e4c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
RevWalk: Traverse all parents of UNINTERESTING commits
When firstParent is set, RevWalk traverses only the first parent of a
commit, even though that commit is UNINTERESTING. Since we want the
maximal UNINTERESTING set, we shouldn't prune any parents here. This
issue is apparent only when some of the commits being traversed are
unparsed, since walker.carryFlagsImpl() propagates the UNINTERESTING
flag to all parsed ancestors, masking the issue.
Therefore teach RevWalk to traverse all parents when a commit is
UNINTERESTING and not only the first parent. Since this issue is
masked by commit parsing, also test situations when the commits
involved are unparsed.
Signed-off-by: Alex Spradlin <alexaspradlin@google.com>
Change-Id: I95e2ad9ae8f1f50fbecae674367ee7e0855519b1
Fix error occurring when SecurityManager is enabled
It's expected that jgit should work without native git installation.
In such case Security Manager can be configured to deny access to the
files outside of git repository. JGit tries to find cygwin
installation. If Security manager restricts access to some folders
in PATH, it should be considered that those folders are absent
for jgit.
Also JGit tries to detect if symbolic links are supported by OS. If
security manager forbids creation of symlinks, it should be assumed
that symlinks aren't supported.
Bug: 550115
Change-Id: Ic4b243cada604bc1090db6cc1cfd74f0fa324b98
Signed-off-by: Nail Samatov <sanail@yandex.ru>
Cache user global and system-wide git configurations
So far the git configuration and the system wide git configuration were
always reloaded when jgit accessed these global configuration files to
access global configuration options which are not in the context of a
single git repository. Cache these configurations in SystemReader and
only reload them if their file metadata observed using FileSnapshot
indicates a modification.
Change-Id: I092fe11a5d95f1c5799273cacfc7a415d0b7786c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
FS determines FileStore attributes in a background thread and tries to
save the results to the global git configuration. This competed with
LocalDiskRepositoryTestCase#setup trying to save changes to the same
file requiring the same lock. This frequently led to one of the threads
failing to acquire the lock.
Fix this by first initiating determination of FileStore attributes which
then uses a MockSystemReader not using a userConfig stored to disk which
avoids this race for the lock.
Change-Id: I30fcd96bc15100f8ef9b2a9eb3320bb5ace97c67
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
FS#getFileStoreAttributes used the real userConfig and not the mocked
one. This led to test errors when running tests with Bazel since it
sandboxes tests which prevents they can write to ~/.gitconfig.
Fix this by first preparing the MockedSystemReader and the mocked config
before calling FS#getFileStoreAttributes.
Also fix ConfigTest which broke due to this change since it inherits
from LocalDiskRepositoryTestCase and calls its setup method which was
changed here. We can no longer assert by comparing plain text since FS
adds FileStoreAttributes to the mocked userConfig. Also the default
options seen by this test changed since we now use a mocked config.
Change-Id: I76bc7c94953fe979266147d3b309a68dda9d4dfe
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Override FileBasedConfig's save method in MockConfig
This ensures we don't try to persist MockConfig using its superclasses
save() method which fails with an NPE since MockConfig has no backing
file.
Change-Id: Ifba2d24c9438bb30d3828ed31a4c131f940b45eb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[error prone] suppress AmbiguousMethodReference in AnyObjectId
Move the implementation of the static equals() method to a new method
and suppress the error. Deprecate the old method to signal that we
intend to remove it in the next major release.
See https://errorprone.info/bugpattern/AmbiguousMethodReference
Change-Id: I5e29c97f4db3e11770be589a6ccd785e2c9ac7f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Persist minimal racy threshold and allow manual configuration
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vendor")
- Java version (system property "java.version")
- FileStore's name, on Windows we use the attribute volume:vsn instead
since the name is not necessarily unique.
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The same prefix is used as for filesystem timestamp resolution, so
both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
time value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
and Java version being used
If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.
When the minimal racy threshold was measured it is persisted in the user
global git configuration.
Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.
Example:
[filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"]
timestampResolution = 7000 nanoseconds
minRacyThreshold = 3440 microseconds
Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
Measure minimum racy interval to auto-configure FileSnapshot
By running FileSnapshotTest#detectFileModified we found that the sum of
measured filesystem timestamp resolution and measured clock resolution
may yield a too small interval after a file has been modified which we
need to consider racily clean. In our tests we didn't find this behavior
on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11
this effect was not observed.
On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the
following test results using Java 8 and 11:
In 23-98% of 10000 test runs (depending on filesystem type and Java
version) the test failed, which means the effective interval which needs
to be considered racily clean after a file was modified is larger than
the measured file timestamp resolution.
"delta" is the observed interval after a file has been modified but
FileSnapshot did not yet detect the modification:
"resolution" is the measured sum of file timestamp resolution and clock
resolution seen in Java.
Java version filesystem failures resolution min delta max delta
1.8.0_212-b04 btrfs 98.6% 1 ms 3.6 ms 6.6 ms
1.8.0_212-b04 ext4 82.6% 3 ms 1.1 ms 4.1 ms
1.8.0_212-b04 xfs 23.8% 4 ms 3.7 ms 3.9 ms
1.8.0_212-b04 zfs 23.1% 3 ms 4.8 ms 5.0 ms
11.0.3+7 btrfs 98.1% 3 us 0.7 ms 4.7 ms
11.0.3+7 ext4 98.1% 6 us 0.7 ms 4.7 ms
11.0.3+7 xfs 98.5% 7 us 0.1 ms 8.0 ms
11.0.3+7 zfs 98.4% 7 us 0.7 ms 5.2 ms
Mac OS
1.8.0_212 APFS 0% 1 s
11.0.3+7 APFS 0% 6 us
The observed delta is not distributed according to a normal gaussian
distribution but rather random in the observed range between "min delta"
and "max delta".
Run this test after measuring file timestamp resolution in
FS.FileAttributeCache to auto-configure JGit since it's unclear what
mechanism is causing this effect.
In FileSnapshot#isRacyClean use the maximum of the measured timestamp
resolution and the measured "delta" as explained above to decide if a
given FileSnapshot is to be considered racily clean. Add a 30% safety
margin to ensure we are on the safe side.
Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Reuse FileUtils to recursively delete files created by tests
Replace redundant complex implementation of recursive delete by the one
in FileUtils.
Change-Id: Iced1468b96c4f32381a9cf0c651b2bf6a9a9af35
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Enhance RepeatRule to report number of failures at the end
In order to enable counting how frequently a test fails if repeated add
option abortOnFailure. If it is true the test aborts on the first
failure. Otherwise it runs the configured number of repetitions and, if
there was any failure, throws a RepeatException reporting how many of
the test repetitions failed.
Change-Id: Ic47de44d4a6273fddf04b9993ad989903efb40c3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Measure filesystem timestamp resolution already in test setup
This helps to avoid some time critical tests can't prepare the test
fixture intended since measuring timestamp resolution takes time.
Change-Id: Ib34023e682a106070ca97e98ef16789a4dfb97b4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use Instant instead of milliseconds for filesystem timestamp handling
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.
Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution
Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
As reported by Error Prone:
An inner class should be static unless it references members of its
enclosing class. An inner class that is made non-static unnecessarily
uses more memory and does not make the intent of the class clear.
See https://errorprone.info/bugpattern/ClassCanBeStatic
Change-Id: Ib99d120532630dba63cf400cc1c61c318286fc41
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ee40efcea4)
Enhance fsTick() to use filesystem timer resolution
RepositoryTestCase.fsTick() was was waiting 64, 128, 256, ... milliseconds
until it detected that the filesystem timer has ticked. Make use of
the filesystemtimer resolution information in FS to sleep a fraction
of the filesystem timer resolution. That raises probability to wake up
shortly after the filesystem timer has ticked.
Change-Id: Ibcc38576e42ece13b2fd4423a29c459eed167a69
Measure file timestamp resolution used in FileSnapshot
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].
Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.
Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.
Measured Java file timestamp resolution: [2]
[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo
Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Replace simple uses of Iterator with a corresponding for-loop.
Also add missing braces on loops as necessary.
Change-Id: I708d82acdf194787e3353699c07244c5ac3de189
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use of String.equals("") can be replaced with with String.length() == 0
(for JDK5 and lower) or String.isEmpty() (for JDK6 and higher)
Change-Id: Id1462d22c5d249485d87993263a9239809e73c55
Signed-off-by: Carsten Hammer <carsten.hammer@t-online.de>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
LocalDiskRepositoryTestCase: Clarify semantics of repository creation methods
Several of the utility methods create a Repository without setting
the 'autoClose' flag, which means that the caller is responsible for
closing it. Update the Javadoc to explicitly mention this.
Change-Id: I2410dd8d230cd4519f756c38b17141d0daa6c314
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
LocalDiskRepositoryTestCase#createRepository: Default auto-close to false
Since 8ed59c5 ("Make TestRepository AutoCloseable", Jan 11, 2019) the
TestRepository class is auto-closeable, but instantiations of it were
not converted to use try-with-resource.
Converting to try-with-resource results, in several cases, in the
repository being closed twice because LocalDiskRepositoryTestCase has
logic to close created repositories in the tearDown method. This results
in several tests emitting a warning to the console:
close() called when useCnt is already zero
Change the default behavior of the createRepository method to not use
the auto-close logic in LocalDiskRepositoryTestCase, so that thy will
instead be closed (only once) using the AutoCloseable implementation.
Deprecate the method that has the autoClose parameter.
Change-Id: I63d62c9913f9b61271667861dae144e551d358c1
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Currently, unit tests need to either close the Repository underlying a
TestRepository manually, or not close it at all. Both are error prone.
The TestRepository holds a reference to 4 AutoCloseable objects:
Repository, ObjectInserter, Git, and RevWalk. The last two can escape
the TestRepository scope, so they are not closed when TestRepository is
closed.
Change-Id: I4461bb9104d517bd6bef09c38507c7c2ef5c31d4
Signed-off-by: Jackson Toeniskoetter <jackdt@google.com>
Move SshTestGitServer to new bundle org.eclipse.jgit.junit.ssh
Create the bundle and move the SshTestGitServer there. Verified that
the Eclipse build still works and ran JSchSshTest and ApacheSshTest as
junit tests inside Eclipse.
Update maven build and features to account for that. Verified by
running full maven build including packaging.
Update bazel build files to account for that. Verified by a
clean-slate bazel build :all, followed by running the JSchSshTest
and the ApacheSshTest via bazel.
Change-Id: Ia084942f4425b454529de148e00417e7da786a90
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Apache MINA sshd client: properly handle HostKeyAlgorithms config
By default sshd will use its default built-in list, which matches
the one of openssh (as far as the algorithms exist in sshd at all).
But it doesn't handle HostKeyAlgorithms from the ssh config at all.
Implement this as in openssh, including the '+' and '-' modifiers
and reordering the default if there are known host keys for a
server already.
Add tests for the reordering.
Also use a more robust reader for the known hosts file. The default
aborts on the first error.
Bug: 520927
Change-Id: Ib1684440bfe2e96140536aa1a93c4bd4a0d35916
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Apache MINA sshd client: add gssapi-with-mic authentication
sshd does support gssapi-with-mic on the server side, but has no
built-in client-side support for this authentication mechanism.
Add our own implementation for it, following RFC 4462.[1] To avoid
needlessly re-trying mechanisms that aren't even configured on the
client, we disable mechanisms that fail on the very first attempt
to use them.
Since we have no real Kerberos5 test setup, this cannot be fully
tested in CI. The disabling of the authentication mechanism and
that it is skipped when not successful _is_ tested.
[1] https://www.ietf.org/rfc/rfc4462.txt
Bug: 520927
Change-Id: I5d0cdb14103588a57c52f927df541b589ab88d88
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Add more ssh tests: pushing, known_host file handling, etc.
Add support for git-receive-pack to the ssh git server and add two
new tests for pushing.
This actually uncovered an undocumented requirement in TransportSftp:
the FTP rename operation assumes POSIX semantics, i.e., that the
target is removed. This works as written only for servers that
support and advertise the "posix-rename@openssh.com" FTP extension.
Our little Apache MINA server does not advertise this extension.
Fix the FtpChannel implementation for Jsch to handle this case in a
meaningful way so that it can pass the new "push over sftp" test.
Add more tests to test the behavior of server host key checking.
Also refactor the tests generally to separate better the test
framework from the actual tests.
Bug: 520927
Change-Id: Ia4bb85e17ddacde7b36ee8c2d5d454bbfa66dfc3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Ssh tests with an Apache MINA sshd test git server
Add a simple ssh git server based on Apache MINA sshd, and use it
in new tests that verify ssh operations and in particular a number
of bugs that had cropped up over time in JSch.
The git server supports fetching only, and sftp access.
The tests are all in an abstract base class; the concrete JschSshTest
class only provides ssh-specific test setup. So the same tests could
be run easily also with some other ssh client.
Bug: 520927
Change-Id: Ide6687b717fb497a29fc83f22b07390a26dfce1d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
As reported by Error Prone:
An inner class should be static unless it references members of its
enclosing class. An inner class that is made non-static unnecessarily
uses more memory and does not make the intent of the class clear.
See https://errorprone.info/bugpattern/ClassCanBeStatic
Change-Id: Ib99d120532630dba63cf400cc1c61c318286fc41
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Return parsed objects from TestRepository.commit/tree/blob()
It is convenient for TestRepository to return fully parsed
objects from its commit()/tree()/blob() methods, so that test
code doesn't have to remember to parse them before making
assertions about them.
Update TestRepostiory to return fully parsed objects.
Adjust the tests that are affected by this change in behavior.
Change-Id: I09d03d0c80ad22cb7092f4a2eaed99d40a10af63
Signed-off-by: Terry Parker <tparker@google.com>
Fix trivial usages of deprecated Repository#getAllRefs
Callers of getAllRefs that only iterate over the `values()` of the
returned map can be trivially fixed to call getRefDatabase().getRefs()
instead.
Only fix those where the calling method is already declared to throw
IOException, to avoid potential API changes.
Change-Id: I2b05f785077a1713953cfd42df7bf915f889f90b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Remove it from
* package private functions.
* try blocks
* for loops
this was done with the following python script:
$ cat f.py
import sys
import re
import os
def replaceFinal(m):
return m.group(1) + "(" + m.group(2).replace('final ', '') + ")"
methodDecl = re.compile(r"^([\t ]*[a-zA-Z_ ]+)\(([^)]*)\)")
def subst(fn):
input = open(fn)
os.rename(fn, fn + "~")
dest = open(fn, 'w')
for l in input:
l = methodDecl.sub(replaceFinal, l)
dest.write(l)
dest.close()
for root, dirs, files in os.walk(".", topdown=False):
for f in files:
if not f.endswith('.java'):
continue
full = os.path.join(root, f)
print full
subst(full)
Change-Id: If533a75a417594fc893e7c669d2c1f0f6caeb7ca
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Open auto-closeable resources in try-with-resource
When an auto-closeable resources is not opened in try-with-resource,
the warning "should be managed by try-with-resource" is emitted by
Eclipse.
Fix the ones that can be silenced simply by moving the declaration of
the variable into a try-with-resource.
In cases where we explicitly call the close() method, for example in
tests where we are testing specific behavior caused by the close(),
suppress the warning.
Leave the ones that will require more significant refcactoring to fix.
They can be done in separate commits that can be reviewed and tested
in isolation.
Change-Id: I9682cd20fb15167d3c7f9027cecdc82bc50b83c4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Replace hard-coded "UTF-8" string with the constant.
Change-Id: Ie812add2df28e984090563ec7c6e2c0366616424
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Merges are performed using the raw text as stored in the git
repository. When we write the merge result, we must apply the
correct CRLF settings. Otherwise the line endings in the result
will be wrong.
Bug: 499615
Change-Id: I37a9b987e9404c97645d2720cd1c7c04c076a96b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use constants from StandardCharsets instead of hard-coded strings
Instead of hard-coding the charset strings "US-ASCII", "UTF-8", and
"ISO-8859-1", use the corresponding constants from StandardCharsets.
UnsupportedEncodingException is not thrown when the StandardCharset
constants are used, so remove the now redundant handling.
Because the encoding names are no longer hard-coded strings, also
remove redundant $NON-NLS warning suppressions.
Also replace existing usages of the constants with static imports.
Change-Id: I0a4510d3d992db5e277f009a41434276f95bda4e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Add flag for keeping ref tombstones in GC reftable
A tombstone will prevent a delayed reference update from resurrecting the
deleted reference.
Change-Id: Id9f4df43d435a299ff16cef614821439edef9b11
Signed-off-by: Minh Thai <mthai@google.com>
David Pursehouse
Matthias Sohn
Alex Spradlin
Nail Samatov
Christian Halstrick
Carsten Hammer
Jackson Toeniskoetter
Thomas Wolf
Terry Parker
Han-Wen Nienhuys
Minh Thai