Matthias Sohn [Mon, 19 Aug 2019 22:15:58 +0000 (00:15 +0200)]
Merge branch 'stable-5.4'
* stable-5.4:
Fix NPE in RebaseTodoFile#parseComments
Fix NPE in ObjectIdOwnerMap#get
Fix NPE in CommitOnlyTest#getHead
FileUtils#lastModifiedInstant should not log error if path doesn't exist
Cache user global and system-wide git configurations
Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
Add missing dependencies for running FS_POSIXTest in Eclipse
Fix javadoc for SystemReader#getInstance
Upgrade tycho-extras to 1.4.0
Improve retry handling when saving FileStoreAttributes fails
Ensure FSTest uses MockSystemReader
Make supportsAtomicCreateNewFile return true as default
Update orbit to R20190602212107-2019-06 to enable backports from master
Handle InvalidPathException in FS_POSIX#createNewFileAtomic
Ensure root cause of lock creation failures is logged
Implement toString in MockSystemReader and MockConfig
LocalDiskRefTreeDatabaseTest shall use MockSystemReader
Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
Ensure we use MockSystemReader in tests
Override FileBasedConfig's save method in MockConfig
Remove FileBasedConfig.load(boolean) introduced in d45219ba
Disable debug log for FS in org.eclipse.jgit.test
Bazel: enable logging for tests in org.eclipse.jgit.test
LockFile: log exception if creation of lock file failed
Stop using deprecated Constants.CHARACTER_ENCODING
Change-Id: I709de5edb626536529a99220aae7751b127c9bff Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 19 Aug 2019 22:10:01 +0000 (00:10 +0200)]
Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
Fix NPE in RebaseTodoFile#parseComments
Fix NPE in ObjectIdOwnerMap#get
Fix NPE in CommitOnlyTest#getHead
FileUtils#lastModifiedInstant should not log error if path doesn't exist
Cache user global and system-wide git configurations
Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
Add missing dependencies for running FS_POSIXTest in Eclipse
Fix javadoc for SystemReader#getInstance
Improve retry handling when saving FileStoreAttributes fails
Ensure FSTest uses MockSystemReader
Make supportsAtomicCreateNewFile return true as default
Update orbit to R20190602212107-2019-06 to enable backports from master
Handle InvalidPathException in FS_POSIX#createNewFileAtomic
Ensure root cause of lock creation failures is logged
Implement toString in MockSystemReader and MockConfig
LocalDiskRefTreeDatabaseTest shall use MockSystemReader
Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
Ensure we use MockSystemReader in tests
Override FileBasedConfig's save method in MockConfig
Remove FileBasedConfig.load(boolean) introduced in d45219ba
Disable debug log for FS in org.eclipse.jgit.test
Bazel: enable logging for tests in org.eclipse.jgit.test
LockFile: log exception if creation of lock file failed
Stop using deprecated Constants.CHARACTER_ENCODING
Change-Id: I43c2ab8b44c3e87d48e4072907ad169c81e3ffe0 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 19 Aug 2019 21:41:04 +0000 (23:41 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Fix NPE in RebaseTodoFile#parseComments
Fix NPE in ObjectIdOwnerMap#get
Fix NPE in CommitOnlyTest#getHead
FileUtils#lastModifiedInstant should not log error if path doesn't exist
Cache user global and system-wide git configurations
Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
Add missing dependencies for running FS_POSIXTest in Eclipse
Fix javadoc for SystemReader#getInstance
Improve retry handling when saving FileStoreAttributes fails
Ensure FSTest uses MockSystemReader
Make supportsAtomicCreateNewFile return true as default
Update orbit to R20190602212107-2019-06 to enable backports from master
Handle InvalidPathException in FS_POSIX#createNewFileAtomic
Ensure root cause of lock creation failures is logged
Implement toString in MockSystemReader and MockConfig
LocalDiskRefTreeDatabaseTest shall use MockSystemReader
Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
Ensure we use MockSystemReader in tests
Override FileBasedConfig's save method in MockConfig
Remove FileBasedConfig.load(boolean) introduced in d45219ba
Disable debug log for FS in org.eclipse.jgit.test
Bazel: enable logging for tests in org.eclipse.jgit.test
LockFile: log exception if creation of lock file failed
Stop using deprecated Constants.CHARACTER_ENCODING
Change-Id: If0c5010a2cf151ebebb2f2088fac3ee02c5007b9 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 19 Aug 2019 15:52:02 +0000 (17:52 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Fix NPE in RebaseTodoFile#parseComments
Fix NPE in ObjectIdOwnerMap#get
Fix NPE in CommitOnlyTest#getHead
FileUtils#lastModifiedInstant should not log error if path doesn't exist
Cache user global and system-wide git configurations
Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
Add missing dependencies for running FS_POSIXTest in Eclipse
Fix javadoc for SystemReader#getInstance
Improve retry handling when saving FileStoreAttributes fails
Ensure FSTest uses MockSystemReader
Make supportsAtomicCreateNewFile return true as default
Update orbit to R20190602212107-2019-06 to enable backports from master
Handle InvalidPathException in FS_POSIX#createNewFileAtomic
Ensure root cause of lock creation failures is logged
Implement toString in MockSystemReader and MockConfig
LocalDiskRefTreeDatabaseTest shall use MockSystemReader
Ensure LocalDiskRepositoryTestCase#setup fully uses MockSystemReader
Ensure we use MockSystemReader in tests
Override FileBasedConfig's save method in MockConfig
Remove FileBasedConfig.load(boolean) introduced in d45219ba
Disable debug log for FS in org.eclipse.jgit.test
Bazel: enable logging for tests in org.eclipse.jgit.test
LockFile: log exception if creation of lock file failed
Stop using deprecated Constants.CHARACTER_ENCODING
Change-Id: I48c585f3c9287be7d6ddb6b01a1955444e13fa31 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
reftable: fix seeking to refs in reflog implementation
Small reftables omit the log index. Currently,
ReftableWriter#shouldHaveIndex does this if there is a single-block
log, but other writers could decide on different criteria.
In the case that the log index is missing, we have to linearly search
for the right block. It is never appropriate to use binary search on
blocks for log data, as the blocks are compressed and therefore
irregularly sized.
Matthias Sohn [Wed, 14 Aug 2019 23:25:28 +0000 (01:25 +0200)]
Cache user global and system-wide git configurations
So far the git configuration and the system wide git configuration were
always reloaded when jgit accessed these global configuration files to
access global configuration options which are not in the context of a
single git repository. Cache these configurations in SystemReader and
only reload them if their file metadata observed using FileSnapshot
indicates a modification.
Change-Id: I092fe11a5d95f1c5799273cacfc7a415d0b7786c Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn [Sun, 11 Aug 2019 00:43:02 +0000 (02:43 +0200)]
Avoid setup and saving FileStoreAttributes compete for ~/.gitconfig lock
FS determines FileStore attributes in a background thread and tries to
save the results to the global git configuration. This competed with
LocalDiskRepositoryTestCase#setup trying to save changes to the same
file requiring the same lock. This frequently led to one of the threads
failing to acquire the lock.
Fix this by first initiating determination of FileStore attributes which
then uses a MockSystemReader not using a userConfig stored to disk which
avoids this race for the lock.
Change-Id: I30fcd96bc15100f8ef9b2a9eb3320bb5ace97c67 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Thu, 15 Aug 2019 09:46:46 +0000 (11:46 +0200)]
GPG: also consider pubring.gpg when looking for keys
The algorithm for finding keys was already improved in commit db0eb9f8,
but that wasn't quite correct yet.
If there is no pubring.kbx but a private-keys-v1.d directory and a
pubring.gpg, GPG also uses pubring.gpg in combination with the
private-keys-v1.d directory. GPG has three ways to locate public and
private key pairs:
* pubring.kbx and private-keys-v1.d (GPG >= 2.1)
* pubring.gpg and private-keys-v1.d (GPG >= 2.1)
* pubring.gpg and secring.gpg (GPG < 2.1)
See [1] and [2]. pubring.kbx may not exist if the user migrated from
an older GPG installation and didn't run the agent. Since we don't
know which GPG version the user has we must try secring.gpg also if
we found the public key in pubring.gpg, but didn't find the secret
key in the private key directory. Note that GPG < 2.1 also may have
a private key directory, used by the agent. But it may also _not_ have
that directory.
Matthias Sohn [Mon, 12 Aug 2019 15:40:39 +0000 (17:40 +0200)]
Improve retry handling when saving FileStoreAttributes fails
- fix handling of interrupts in FileStoreAttributes#saveToConfig
- increase retry wait time to 100ms
- don't wait after last retry
- dont retry if failure is caused by another exception than
LockFailedException
Change-Id: I108c012717d2bcce71f2c6cb9cf0879de704ebc2 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Make supportsAtomicCreateNewFile return true as default
The method org.eclipse.jgit.util.FS.supportsAtomicCreateNewFile()
should default to true as mentioned in docs [1]
org.eclipse.jgit.util.FS_POSIX.supportsAtomicCreateNewFile() method
will set the value to false if the git config
core.supportsatomiccreatenewfile is not set.
It should default to true if the configuration is undefined.
Matthias Sohn [Mon, 12 Aug 2019 10:02:59 +0000 (12:02 +0200)]
Update orbit to R20190602212107-2019-06 to enable backports from master
update
- org.apache.httpcomponents.httpclient to 4.5.6.v20190503-0009
- org.apache.httpcomponents.httpcore to 4.4.10.v20190123-2214
- com.jcraft.jsch" version to 0.1.55.v20190404-1902
- org.mockito to 2.23.0.v20190527-1420
add its dependencies
- net.bytebuddy.byte-buddy 1.9.0.v20181107-1410
- net.bytebuddy.byte-buddy-agent 1.9.0.v20181106-1534
- org.objenesis to 2.6.0.v20180420-1519
FS#getFileStoreAttributes used the real userConfig and not the mocked
one. This led to test errors when running tests with Bazel since it
sandboxes tests which prevents they can write to ~/.gitconfig.
Fix this by first preparing the MockedSystemReader and the mocked config
before calling FS#getFileStoreAttributes.
Also fix ConfigTest which broke due to this change since it inherits
from LocalDiskRepositoryTestCase and calls its setup method which was
changed here. We can no longer assert by comparing plain text since FS
adds FileStoreAttributes to the mocked userConfig. Also the default
options seen by this test changed since we now use a mocked config.
Change-Id: I76bc7c94953fe979266147d3b309a68dda9d4dfe Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 10 Aug 2019 21:34:58 +0000 (23:34 +0200)]
Ensure we use MockSystemReader in tests
If we use the default system reader FileStoreAttributes cannot persist
attributes in userConfig when tests run in Bazel due to sandboxing.
Hence we need to ensure that all tests use MockSystemReader (and
especially a mocked userConfig).
Change-Id: Ic1ad8e2ec5a150c5433434a5f6667d6c4674c87d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 10 Aug 2019 21:28:07 +0000 (23:28 +0200)]
Remove FileBasedConfig.load(boolean) introduced in d45219ba
We can't add this method to the super class StoredConfig since that
abstracts from filesystem storage. MockSystemReader.MockConfig is a
StoredConfig and is also used by tests for dfs based storage. Hence
remove this leaky abstraction.
This implies we always use the fallback FileStoreAttributes which means
a config file modification is considered racy within the first 2
seconds. This should not be an issue since typically configs change
rarely and re-reading a config within the racy period is relatively
cheap since configs are small.
Change-Id: Ia2615addc24a7cadf3c566ee842c6f4f07e159a5 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 9 Aug 2019 12:12:57 +0000 (14:12 +0200)]
Merge branch 'stable-5.4'
* stable-5.4: (82 commits)
Export all packages of o.e.j.ant and o.e.j.archive bundles
Do not require test bundles to export all packages
Fix API problem filters
Increase severity of AmbiguousMethodReference to ERROR
[error prone] suppress AmbiguousMethodReference in AnyLongObjectId
[error prone] fix ReferenceEquality warning in CommitBuilder
[error prone] suppress NonAtomicVolatileUpdate warning in SimpleLruCache
[error prone] fix ReferenceEquality warning in CommitGraphPane#authorFor
[error prone] fix ReferenceEquality warning in RevWalk#isMergedInto
[error prone] fix ReferenceEquality warning in RefUpdate#updateImpl
[error prone] fix ReferenceEquality warning in static equals methods
[error prone] suppress AmbiguousMethodReference in AnyObjectId
[error prone] fix "FutureReturnValueIgnored" error in FS
Fix formatting and add missing braces in Repository#stripWorkDir
Repository: fix reference comparison of Files
MergeAlgorithm: Suppress Error Prone warning about reference equality
Fix NarrowingCompoundAssignment warnings from Error Prone
FS_POSIX: handle Files.getFileStore() failures
Fix OpenSshConfigTest#config
FileSnapshot: fix bug with timestamp thresholding
In LockFile#waitForStatChange wait in units of file time resolution
Cache FileStoreAttributeCache per directory
Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
Persist minimal racy threshold and allow manual configuration
Measure minimum racy interval to auto-configure FileSnapshot
Reuse FileUtils to recursively delete files created by tests
Fix FileAttributeCache.toString()
Add test for racy git detection in FileSnapshot
Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
Add missing javadoc in org.eclipse.jgit.junit
Enhance RepeatRule to report number of failures at the end
Fix FileSnapshotTests for filesystem with high timestamp resolution
Retry deleting test files in FileBasedConfigTest
Measure filesystem timestamp resolution already in test setup
Refactor FileSnapshotTest to use NIO APIs
Measure stored timestamp resolution instead of time to touch file
Handle CancellationException in FileStoreAttributeCache
Fix FileSnapshot#saveNoConfig
Use Instant for smudge time in DirCache and DirCacheEntry
Use Instant instead of milliseconds for filesystem timestamp handling
Workaround SecurityException in FS#getFsTimestampResolution
Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
FS: ignore AccessDeniedException when measuring timestamp resolution
Add debug trace for FileSnapshot
Use FileChannel.open to touch file and set mtime to now
Persist filesystem timestamp resolution and allow manual configuration
Increase bazel timeout for long running tests
Bazel: Fix lint warning flagged by buildifier
Update bazlets to latest version
Bazel: Add missing dependencies for ArchiveCommandTest
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
Add support for nanoseconds and microseconds for Config#getTimeUnit
Optionally measure filesystem timestamp resolution asynchronously
Delete unused FileTreeIteratorWithTimeControl
FileSnapshot#equals: consider UNKNOWN_SIZE
Timeout measuring file timestamp resolution after 2 seconds
Fix RacyGitTests#testRacyGitDetection
GlobalBundleCache: Fix ClassNewInstance warning from Error Prone
IncorrectObjectTypeException: Fix typos in constructors' Javadoc
Change RacyGitTests to create a racy git situation in a stable way
Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
Fix non-deterministic hash of archives created by ArchiveCommand
Update Maven plugins ecj, plexus, error-prone
Update Maven plugins and cleanup Maven warnings
Make inner classes static where possible
Error Prone: Increase severity of NonOverridingEquals to ERROR
Error Prone: Increase severity of ImmutableEnumChecker to ERROR
GitDateParser#ParseableSimpleDateFormat: Make formatStr private final
BatchRefUpdateTest: Suppress ImmutableEnumChecker warning
PacketLineIn: Suppress comparison warnings for END and DELIM
FileSnapshot#toString: Suppress ReferenceEquality warnings
Blame: Suppress ReferenceEquality warning for RevCommit instances
Fix API problem filters
pgm: add missing optional dependency to org.tukaani:xz
NetscapeCookieFile: Make hash static and group overloaded write
NetscapeCookieFile: Javadoc fixes
Config: Handle reference-equality warning (and empty javadoc)
Error Prone: Increase severity of ShortCircuitBoolean to ERROR
ObjectWalk: Prefer boolean operators over logical operators in comparisons
BasePackFetchConnection: Prefer boolean operators over logical operators in comparisons
PackWriter: Prefer boolean operators over logical operators in comparisons
Change-Id: I825fd55bcb5345fb7afe066bf54ca50325f40acb Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 9 Aug 2019 11:16:11 +0000 (13:16 +0200)]
Do not require test bundles to export all packages
Configure PDE to not warn about not exported packages for test bundles.
Reusable test code which other bundles may want to reuse should go into
one of the junit bundles:
- org.eclipse.jgit.junit
- org.eclipse.jgit.junit.http
- org.eclipse.jgit.junit.ssh
Change-Id: Ifbdf30f8552e8f4bad6bbdeb63f729c53c844315 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[error prone] suppress AmbiguousMethodReference in AnyLongObjectId
Move the implementation of the static equals() method to a new method
and suppress the error. Deprecate the old method to signal that we
intend to remove it in the next major release.
See https://errorprone.info/bugpattern/AmbiguousMethodReference
Change-Id: I712697a411ab44c6e05ae4604eb2dcb9c0f8abd3 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Marco Miller [Mon, 29 Jul 2019 16:32:07 +0000 (12:32 -0400)]
Upgrade maven-site-plugin to 3.8.2
Change-Id: I79a3032e2ff7af14a41e6eb2618d2b1e22d1fde2 Signed-off-by: Marco Miller <marco.miller@ericsson.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Matthias Sohn [Thu, 8 Aug 2019 23:01:27 +0000 (01:01 +0200)]
[error prone] fix ReferenceEquality warning in static equals methods
Implement a helper method suppressing the ReferenceEquality error prone
warning and use it to fix this warning in static equals methods where
this comparison is used to implement fast path of static equals
implementation.
See https://errorprone.info/bugpattern/ReferenceEquality
Change-Id: I33538a3406007d24efec3a504e031ca1069572ed Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 8 Aug 2019 13:09:46 +0000 (15:09 +0200)]
[error prone] suppress AmbiguousMethodReference in AnyObjectId
Move the implementation of the static equals() method to a new method
and suppress the error. Deprecate the old method to signal that we
intend to remove it in the next major release.
See https://errorprone.info/bugpattern/AmbiguousMethodReference
Change-Id: I5e29c97f4db3e11770be589a6ccd785e2c9ac7f2 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Thu, 18 Jul 2019 07:21:13 +0000 (16:21 +0900)]
MergeAlgorithm: Suppress Error Prone warning about reference equality
Error Prone reports:
[ReferenceEquality] Comparison using reference equality instead of
value equality
The END_EDIT instance is used as a marker, and thus it's OK to use
a reference equality comparison. Factor the comparison to a method
and add a suppression.
Change-Id: I7d9dc1fa21f46c984787056b0b5d163e313026a6 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Thomas Wolf [Thu, 4 Jul 2019 07:53:25 +0000 (09:53 +0200)]
FS_POSIX: handle Files.getFileStore() failures
Android unconditionally throws a SecurityException;[1] getFileStore()
is not supported. Catch the exception and don't attempt the hard-
linking atomic file mechanism.
Matthias Sohn [Thu, 8 Aug 2019 11:17:48 +0000 (13:17 +0200)]
Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
Fix OpenSshConfigTest#config
FileSnapshot: fix bug with timestamp thresholding
In LockFile#waitForStatChange wait in units of file time resolution
Cache FileStoreAttributeCache per directory
Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
Persist minimal racy threshold and allow manual configuration
Measure minimum racy interval to auto-configure FileSnapshot
Reuse FileUtils to recursively delete files created by tests
Fix FileAttributeCache.toString()
Add test for racy git detection in FileSnapshot
Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
Add missing javadoc in org.eclipse.jgit.junit
Enhance RepeatRule to report number of failures at the end
Fix FileSnapshotTests for filesystem with high timestamp resolution
Retry deleting test files in FileBasedConfigTest
Measure filesystem timestamp resolution already in test setup
Refactor FileSnapshotTest to use NIO APIs
Measure stored timestamp resolution instead of time to touch file
Handle CancellationException in FileStoreAttributeCache
Fix FileSnapshot#saveNoConfig
Use Instant for smudge time in DirCache and DirCacheEntry
Use Instant instead of milliseconds for filesystem timestamp handling
Workaround SecurityException in FS#getFsTimestampResolution
Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
FS: ignore AccessDeniedException when measuring timestamp resolution
Add debug trace for FileSnapshot
Use FileChannel.open to touch file and set mtime to now
Persist filesystem timestamp resolution and allow manual configuration
Increase bazel timeout for long running tests
Bazel: Fix lint warning flagged by buildifier
Update bazlets to latest version
Bazel: Add missing dependencies for ArchiveCommandTest
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
Add support for nanoseconds and microseconds for Config#getTimeUnit
Optionally measure filesystem timestamp resolution asynchronously
Delete unused FileTreeIteratorWithTimeControl
FileSnapshot#equals: consider UNKNOWN_SIZE
Timeout measuring file timestamp resolution after 2 seconds
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
Fix non-deterministic hash of archives created by ArchiveCommand
Update Maven plugins ecj, plexus, error-prone
Update Maven plugins and cleanup Maven warnings
Make inner classes static where possible
Fix API problem filters
Change-Id: Iec3ad6ccc194582cb844310dc172c3103dae4457 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 8 Aug 2019 10:30:47 +0000 (12:30 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Fix OpenSshConfigTest#config
FileSnapshot: fix bug with timestamp thresholding
In LockFile#waitForStatChange wait in units of file time resolution
Cache FileStoreAttributeCache per directory
Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
Persist minimal racy threshold and allow manual configuration
Measure minimum racy interval to auto-configure FileSnapshot
Reuse FileUtils to recursively delete files created by tests
Fix FileAttributeCache.toString()
Add test for racy git detection in FileSnapshot
Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
Add missing javadoc in org.eclipse.jgit.junit
Enhance RepeatRule to report number of failures at the end
Fix FileSnapshotTests for filesystem with high timestamp resolution
Retry deleting test files in FileBasedConfigTest
Measure filesystem timestamp resolution already in test setup
Refactor FileSnapshotTest to use NIO APIs
Measure stored timestamp resolution instead of time to touch file
Handle CancellationException in FileStoreAttributeCache
Fix FileSnapshot#saveNoConfig
Use Instant for smudge time in DirCache and DirCacheEntry
Use Instant instead of milliseconds for filesystem timestamp handling
Workaround SecurityException in FS#getFsTimestampResolution
Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
FS: ignore AccessDeniedException when measuring timestamp resolution
Add debug trace for FileSnapshot
Use FileChannel.open to touch file and set mtime to now
Persist filesystem timestamp resolution and allow manual configuration
Increase bazel timeout for long running tests
Bazel: Fix lint warning flagged by buildifier
Update bazlets to latest version
Bazel: Add missing dependencies for ArchiveCommandTest
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
Add support for nanoseconds and microseconds for Config#getTimeUnit
Optionally measure filesystem timestamp resolution asynchronously
Delete unused FileTreeIteratorWithTimeControl
FileSnapshot#equals: consider UNKNOWN_SIZE
Timeout measuring file timestamp resolution after 2 seconds
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Deprecate Constants.CHARACTER_ENCODING in favor of
StandardCharsets.UTF_8
Fix non-deterministic hash of archives created by ArchiveCommand
Update Maven plugins ecj, plexus, error-prone
Update Maven plugins and cleanup Maven warnings
Make inner classes static where possible
Fix API problem filters
Change-Id: I238adfd3080a5fed9d64c3c757297da6ea893918 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 8 Aug 2019 09:18:14 +0000 (11:18 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Fix OpenSshConfigTest#config
FileSnapshot: fix bug with timestamp thresholding
In LockFile#waitForStatChange wait in units of file time resolution
Cache FileStoreAttributeCache per directory
Fix FileSnapshot#save(long) and FileSnapshot#save(Instant)
Persist minimal racy threshold and allow manual configuration
Measure minimum racy interval to auto-configure FileSnapshot
Reuse FileUtils to recursively delete files created by tests
Fix FileAttributeCache.toString()
Add test for racy git detection in FileSnapshot
Repeat RefDirectoryTest.testGetRef_DiscoversModifiedLoose 100 times
Fix org.eclipse.jdt.core.prefs of org.eclipse.jgit.junit
Add missing javadoc in org.eclipse.jgit.junit
Enhance RepeatRule to report number of failures at the end
Fix FileSnapshotTests for filesystem with high timestamp resolution
Retry deleting test files in FileBasedConfigTest
Measure filesystem timestamp resolution already in test setup
Refactor FileSnapshotTest to use NIO APIs
Measure stored timestamp resolution instead of time to touch file
Handle CancellationException in FileStoreAttributeCache
Fix FileSnapshot#saveNoConfig
Use Instant for smudge time in DirCache and DirCacheEntry
Use Instant instead of milliseconds for filesystem timestamp handling
Workaround SecurityException in FS#getFsTimestampResolution
Fix NPE in FS$FileStoreAttributeCache.getFsTimestampResolution
FS: ignore AccessDeniedException when measuring timestamp resolution
Add debug trace for FileSnapshot
Use FileChannel.open to touch file and set mtime to now
Persist filesystem timestamp resolution and allow manual configuration
Increase bazel timeout for long running tests
Bazel: Fix lint warning flagged by buildifier
Update bazlets to latest version
Bazel: Add missing dependencies for ArchiveCommandTest
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
Add support for nanoseconds and microseconds for Config#getTimeUnit
Optionally measure filesystem timestamp resolution asynchronously
Delete unused FileTreeIteratorWithTimeControl
FileSnapshot#equals: consider UNKNOWN_SIZE
Timeout measuring file timestamp resolution after 2 seconds
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Deprecate Constants.CHARACTER_ENCODING in favor of StandardCharsets.UTF_8
Fix non-deterministic hash of archives created by ArchiveCommand
Update Maven plugins ecj, plexus, error-prone
Update Maven plugins and cleanup Maven warnings
Make inner classes static where possible
Fix API problem filters
Change-Id: Ia57385b2a60f48a5317c8d723721c235d7043a84 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 8 Aug 2019 08:10:12 +0000 (10:10 +0200)]
Fix OpenSshConfigTest#config
- use FS.DETECTED instead of db.getFS() since the ssh config is
typically in a different place than the repository, the same is used in
OpenSshConfig
- reduce unnecessary repeated writes by introducing wait for one tick of
the file time resolution
Change-Id: Ifac915e97ff420ec5cf8e2f162e351f9f51b6b14 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* changes:
sshd: add missing javadoc in SshTestGitServer
sshd: shared reference in JGitClientSession must be volatile
sshd: correct the protocol version exchange
Increase the safety factor to 2.5x for extra safety if max of measured
timestamp resolution and measured minimal racy threshold is < 100ms, use
1.25 otherwise since for large filesystem resolution values the
influence of finite resolution of the system clock should be negligible.
Before, not yet using the newly introduced minRacyThreshold measurement,
the threshold was 1.1x FS resolution, and we could issue the
following sequence of events,
In this case, the difference between create-file and read is 5ms,
which exceeded the 4ms FS resolution, even though the events together
took just 2ms of runtime.
Reproduce with:
bazel test --runs_per_test=100 \
//org.eclipse.jgit.test:org_eclipse_jgit_internal_storage_file_FileSnapshotTest
The file system timestamp resolution is 4ms in this case.
This code assumes that the kernel and the JVM use the same clock that
is synchronized with the file system clock. This seems plausible,
given the resolution of System.currentTimeMillis() and the latency for
a gettimeofday system call (typically ~1us), but it would be good to
justify this with specifications.
Also cover a source of flakiness: if the test runs under extreme load,
then we could have
start
create-file
<long delay>
read
end
which would register as an unmodified file. Avoid this by skipping the
test if end-start is too big.
[msohn]:
- downported from master to stable-5.1
- skip test if resolution is below 10ms
- adjust safety factor to 1.25 for resolutions above 100ms
Change-Id: I87d2cf035e01c44b7ba8364c410a860aa8e312ef Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 5 Aug 2019 16:00:35 +0000 (18:00 +0200)]
In LockFile#waitForStatChange wait in units of file time resolution
Since we now measure file time resolution we can use it to replace the
hard coded wait time of 25ms. FileSnapshot#equals will return true until
the mtime of the old (o) and the new FileSnapshot (n) differ by at least
one file time resolution.
Change-Id: Icb713a80ce9eb929242ed083406bfb6650c72223 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 19 Jul 2019 15:35:27 +0000 (17:35 +0200)]
Cache FileStoreAttributeCache per directory
Cache FileStoreAttributeCache entries since looking up FileStore for a
file may be expensive on some platforms.
Implement a simple LRU cache based on ConcurrentHashMap using a simple
long counter to order access to cache entries.
Change-Id: I4881fa938ad2f17712c05da857838073a2fc4ddb Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com> Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
Matthias Sohn [Wed, 17 Jul 2019 14:31:42 +0000 (16:31 +0200)]
Persist minimal racy threshold and allow manual configuration
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vendor")
- Java version (system property "java.version")
- FileStore's name, on Windows we use the attribute volume:vsn instead
since the name is not necessarily unique.
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The same prefix is used as for filesystem timestamp resolution, so
both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
time value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
and Java version being used
If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.
When the minimal racy threshold was measured it is persisted in the user
global git configuration.
Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.
Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
Matthias Sohn [Mon, 15 Jul 2019 13:00:09 +0000 (15:00 +0200)]
Measure minimum racy interval to auto-configure FileSnapshot
By running FileSnapshotTest#detectFileModified we found that the sum of
measured filesystem timestamp resolution and measured clock resolution
may yield a too small interval after a file has been modified which we
need to consider racily clean. In our tests we didn't find this behavior
on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11
this effect was not observed.
On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the
following test results using Java 8 and 11:
In 23-98% of 10000 test runs (depending on filesystem type and Java
version) the test failed, which means the effective interval which needs
to be considered racily clean after a file was modified is larger than
the measured file timestamp resolution.
"delta" is the observed interval after a file has been modified but
FileSnapshot did not yet detect the modification:
"resolution" is the measured sum of file timestamp resolution and clock
resolution seen in Java.
Java version filesystem failures resolution min delta max delta
1.8.0_212-b04 btrfs 98.6% 1 ms 3.6 ms 6.6 ms
1.8.0_212-b04 ext4 82.6% 3 ms 1.1 ms 4.1 ms
1.8.0_212-b04 xfs 23.8% 4 ms 3.7 ms 3.9 ms
1.8.0_212-b04 zfs 23.1% 3 ms 4.8 ms 5.0 ms
11.0.3+7 btrfs 98.1% 3 us 0.7 ms 4.7 ms
11.0.3+7 ext4 98.1% 6 us 0.7 ms 4.7 ms
11.0.3+7 xfs 98.5% 7 us 0.1 ms 8.0 ms
11.0.3+7 zfs 98.4% 7 us 0.7 ms 5.2 ms
Mac OS
1.8.0_212 APFS 0% 1 s
11.0.3+7 APFS 0% 6 us
The observed delta is not distributed according to a normal gaussian
distribution but rather random in the observed range between "min delta"
and "max delta".
Run this test after measuring file timestamp resolution in
FS.FileAttributeCache to auto-configure JGit since it's unclear what
mechanism is causing this effect.
In FileSnapshot#isRacyClean use the maximum of the measured timestamp
resolution and the measured "delta" as explained above to decide if a
given FileSnapshot is to be considered racily clean. Add a 30% safety
margin to ensure we are on the safe side.
Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Jonathan Tan [Wed, 3 Apr 2019 17:52:48 +0000 (10:52 -0700)]
UploadPack: move writing of "packfile" header
In a subsequent patch, in some cases, PackWriter#writePack will be
responsible for both the "packfile-uris" and "packfile" sections,
meaning that (in these cases) it must write the "packfile" section
header itself.
In preparation for that patch, move the writing of the "packfile"
section header closer to the invocation of PackWriter#writePack when the
entire fetch response is configured to use the sideband. This means that
"packfile" is written *after* objects are counted (and progress messages
sent to the client in sideband 2) when the "sideband-all" feature is
used (whether "packfile-uris" is used or not), and written *before*
objects are counted otherwise.
Having code to write "packfile" in two places is unfortunate but
necessary. When "sideband-all" is not used, object counting has to
happen after "packfile" is written, because "packfile" activates the
sideband that allows counting progress to be transmitted. When
"packfile-uris" is used, object counting has to happen before "packfile"
is written, because object counting determines whether to send
"packfile-uris" or "packfile". When "sideband-all" is used but
"packfile-uris" is not used, either way works; this commit uses
"packfile-uris" behavior in this case.
Also make the naming of the sideband-activating methods in PacketLineOut
more consistent.
Change-Id: Ifbfd26cc26af10c41b77758168833702d6983df1 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Matthias Sohn [Fri, 12 Jul 2019 07:49:13 +0000 (09:49 +0200)]
Add test for racy git detection in FileSnapshot
Repeat the test 10000 times to get statistics if measured
fsTimestampResolution is working in practice to detect racy git
situations.
Add a class to compute statistics for this test. Log delta between
lastModified and time when FileSnapshot failed to detect modification.
This happens if the racy git limit determined by measuring filesystem
timestamp resolution and clock resolution is too small. If it would be
correct FileSnapshot would always detect modification or mark it
modified if time since modification is smaller than the racy git limit.
Change-Id: Iabe7af1a7211ca58480f8902d4fa4e366932fc77 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Wed, 24 Jul 2019 18:36:51 +0000 (20:36 +0200)]
JGit pgm tests must quote paths on the command line
The JGit pgm tests parse the command line internally using Linux
semantics, treating '\' as an escape. File paths therefore must
be quoted, otherwise Windows paths are destroyed. See the
attachment[1] on bug 548598.
Matthias Sohn [Fri, 12 Jul 2019 07:45:09 +0000 (09:45 +0200)]
Enhance RepeatRule to report number of failures at the end
In order to enable counting how frequently a test fails if repeated add
option abortOnFailure. If it is true the test aborts on the first
failure. Otherwise it runs the configured number of repetitions and, if
there was any failure, throws a RepeatException reporting how many of
the test repetitions failed.
Change-Id: Ic47de44d4a6273fddf04b9993ad989903efb40c3 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 10 Jul 2019 14:17:21 +0000 (16:17 +0200)]
Fix FileSnapshotTests for filesystem with high timestamp resolution
When filesystem timestamp resolution is very high some tests don't work
since runtime of the test setup is too long to reach a racily clean
FileSnapshot. Hence skip these tests when timestamp resolution is higher
than 10 millisecond.
Change-Id: Ie47dd10eda22037b5c1ebff6b6becce0654ea807 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 8 Jul 2019 08:20:16 +0000 (10:20 +0200)]
Measure stored timestamp resolution instead of time to touch file
Measure granularity of timestamps stored in the filesystem by setting
and then getting lastModified timestamp until the read value changed.
Increase increment exponentially to limit number of iterations starting
with 1 microsecond since Java's FileTime (up to Java 12) truncates
timestamps to 1 microsecond resolution. The chosen algorithm yields 2000
steps between 1 ms and 2.5 s.
Also measure clock resolution and add that for the total timestamp
resolution. This avoids systematic measurement errors introduced by
doing IO to touch a file.
Change-Id: I9b37138619422452373e298d9d8c7cb2c384db3f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log