Use Instant instead of milliseconds for filesystem timestamp handling
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.
Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution
Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Workaround SecurityException in FS#getFsTimestampResolution
On Android FS#getFsTimestampResolution always throws a
SecurityException, handle this by falling back to the fallback timestamp
resolution.
Bug: 548947
Change-Id: I0ee6cb3c20e189bdc8d488434a930427ad6f2df2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
FS: ignore AccessDeniedException when measuring timestamp resolution
It seems on cygwin creating a file under a writable directory can fail
with AccessDeniedException. Log a warning in this case and fallback to
worst case timestamp resolution of 2 seconds.
Bug: 548648
Change-Id: Ic50c31ce9dc9ccadd4db5247df929418ac62d45c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Checking lastModified is time critical hence debug trace is the only way
to analyze issues since debugging is impractical.
Also add configuration for buffering of log4j output to reduce runtime
impact when debug trace is on. Limit buffer to 1MiB and comment this
configuration out since we may not always want to use buffering.
Change-Id: Ib1a0537b67c8dc3fac994a77b42badd974ce6c97
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use FileChannel.open to touch file and set mtime to now
Use options
- StandardOpenOption.CREATE to create touched file if not existing
- StandardOpenOption.SYNC to enforce synch of data and meta data changes
- StandardOpenOption.WRITE
Also set mtime explicitly in FileUtils#touch to the current system time.
This should fix that the previous implementation didn't work on
- locally cached Windows network share (CSC-CACHE filesystem) mapped as
a drive
- nfsv4 mounts on Linux
and that it didn't create unborn file like Linux command "touch".
Apache common's and Guava's touch() use the same approach.
Immediately after creating the probe file used to measure timestamp
resolution touch it. This ensures we always use the local system clock
when measuring filesystem timestamp resolution. This should prevent that
clock skew could influence the measured timestamp resolution in case of
a mounted network filesystem.
Bug: 548598
Change-Id: Iaeaf5967963f582395a195aa637b8188bfadac60
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Persist filesystem timestamp resolution and allow manual configuration
To enable persisting filesystem timestamp resolution per FileStore add a
new config section to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vm.vendor")
- runtime version (system property "java.vm.version")
- FileStore's name
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The prefix is needed since some Java versions do not expose the full
timestamp resolution of the underlying filesystem. This may also
depend on the underlying operating system hence concrete key values
may not be portable.
- Config key for timestamp resolution is "timestampResolution" as a time
value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
If timestamp resolution is already configured for a given FileStore
the configured value is used instead of measuring the resolution.
When timestamp resolution was measured it is persisted in the user
global git configuration.
Example:
[filesystem "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"]
timestampResolution = 1 seconds
If locking the git config file fails retry saving the resolution up to 5
times in order to workaround races with another thread.
In order to avoid stack overflow use the fallback filesystem timestamp
resolution when loading FileBasedConfig which creates itself a
FileSnapshot to help checking if the config changed.
Note:
- on some OSes Java 8,9 truncate to milliseconds or seconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes up to Java 12 truncates timestamp resolution to
microseconds when converting the internal representation to FileTime
exposed in the API, see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution up to
Java 12
Hence do not attempt to manually configure a higher timestamp resolution
than supported by the Java version being used at runtime.
Bug: 546891
Bug: 548188
Change-Id: Iff91b8f9e6e5e2295e1463f87c8e95edf4abbcf8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
EolRepositoryTest and GcCommitSelectionTest timed out frequently when
running unit tests using bazel with the default timeout "moderate"
(300s). Increase timeout of these tests to "long" (900s).
Change-Id: I43588cf950f55b50f868d9fe9c66d22bd428a54c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This change is fixing confusing name warning: [1].
./org.eclipse.jgit.test/tests.bzl:12: confusing-name:
Never use 'l', 'I', or 'O' as names (they're too easily confused
with 'I', 'l', or '0').
And is also fixing: "All calls to rules or macros should pass arguments
by keyword position argument" warning: [2].
./org.eclipse.jgit.test/BUILD:42: positional-args: All calls to rules
or macros should pass arguments by keyword (arg_name=value) syntax.
[1] https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#confusing-name
[2] https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#positional-args
Change-Id: If5c28ec8a1ddc1d1b1035bd07b838a2a564aea4f
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
This is needed to make build tool chain compatible with the latest
Bazel releases.
Change-Id: I9822b5fe5f934457e6069217d687b3cf4764b7b7
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Bazel: Add missing dependencies for ArchiveCommandTest
Dependencies on commons-compress, xz, and jgit-archive are required
for the build to succeed.
Change-Id: I42f3721078a240ad93b8dcab909e66b9bfff0b56
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
FileTreeIteratorWithTimeControl was deleted in a024759, but was
not removed from the BUILD file, thus causing the bazel build to
fail.
Change-Id: I892c0ffcac947298d0d6009374ee2c5d9afefb66
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
In order to avoid blocking on the main thread during measurement
interactive applications like EGit may want to measure the filesystem
timestamp resolution asynchronously.
In order to enable measurement in the background call
FileStoreAttributeCache.setAsyncfileStoreAttrCache(true)
before the first access to cached FileStore attributes.
Bug: 548188
Change-Id: I8c9a2dbfc3f1d33441edea18b90e36b1dc0156c7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The only usage of this test iterator was removed in df637928d. Hence
delete this iterator and associated test.
Change-Id: I47710133ec3edc675c21db210960c024982668c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add a unittest.
In commit I5485db55 ("Fix FileSnapshot's consideration of file size"),
the special casing of UNKNOWN_SIZE was forgotten.
This change, together with I493f3b57b ("Measure file timestamp
resolution used in FileSnapshot") introduced a regression that would
occasionally surface in Gerrit integration tests marked UseLocalDisk,
with the symptom that creating the Admin user in NoteDb failed with a
LOCK_FAILURE.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I7ffd972581f815c144f810481103c7985af5feb0
Timeout measuring file timestamp resolution after 2 seconds
It was reported that measuring file timestamp resolution may hang
indefinitely on nfs. Hence timeout this measurement at the known worst
filesystem timestamp resolution (FAT) of 2 seconds.
Bug: 548188
Change-Id: I17004b0aa49d5b0e76360a008af3adb911b289c0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This test case assumed file system timestamp resolution of 1 second. On
filesystems with a finer resolution this test fails since the index
entry is only smudged if the file index entry's lastModified and the
lastModified of the git index itself are within the same filesystem
timer tick. Fix this by ensuring that these timestamps are identical
which should work for any filesystem timer resolution.
Bug: 548188
Change-Id: Id84d59e1cfeb48fa008f8f27f2f892c4f73985de
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Change RacyGitTests to create a racy git situation in a stable way
By using File#setLastModified, we can create a racy git situation
stably.
Tested with --runs_per_test=100
Bug: 526111
Change-Id: I60b3632d353e19f335668325aa603640be423f58
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Fix non-deterministic hash of archives created by ArchiveCommand
Archives created by the ArchiveCommand didn't produce deterministic
archive hashes. For RevCommits RevWalk.parseTree returns the root tree
instead of the RevCommit hence retrieving the commit's timestamp didn't
work. Instead use RevWalk.parseAny and extract the tree manually.
Archive entries store timestamps with 1 second resolution hence we need
to wait longer when creating the same archive twice and compare archive
hashes. Otherwise hash comparison in tests wouldn't fail without this
patch.
Bug: 548312
Change-Id: I437d515de51cf68265584d28a8446cebe6341b79
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
update Maven plugins
- ecj to 3.17.0
- error_prone_core to 2.3.3
- plexus-compiler-eclipse to 2.8.5
- plexus-compiler-javac to 2.8.5
- plexus-compiler-javac-errorprone to 2.8.5
Change-Id: I51ecb44538915ed84db041510562394bce977a3e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
update Maven plugins
- jacoco-maven-plugin to 0.8.4
- japicmp-maven-plugin to 0.14.1
- maven-compiler-plugin to 3.8.1
- maven-deploy-plugin to 3.0.0-M1
- maven-enforcer-plugin to 3.0.0-M2
- maven-install-plugin to 3.0.0-M1
- maven-jar-plugin to 3.1.2
- maven-javadoc-plugin to 3.1.0
- maven-jxr-plugin to 3.0.0
- maven-pmd-plugin to 3.12.0
- maven-resources-plugin to 3.1.0
- maven-shade-plugin to 3.2.1
- maven-source-plugin to 3.1.0
- maven-surefire-plugin to 3.0.0-M3
- spotbugs-maven-plugin to 3.1.12
- tycho to 1.3.0
- tycho-pack200a-plugin to 1.3.0
- tycho-pack200b-plugin to 1.3.0
Cleanup Maven warnings
- pin version of all used Maven plugins
- remove deprecated way to declare minimum Maven version
Change-Id: If23e2e2bb03e5e1e7b1eb9d4924a8faa0aa3704e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
As reported by Error Prone:
An inner class should be static unless it references members of its
enclosing class. An inner class that is made non-static unnecessarily
uses more memory and does not make the intent of the class clear.
See https://errorprone.info/bugpattern/ClassCanBeStatic
Change-Id: Ib99d120532630dba63cf400cc1c61c318286fc41
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
(cherry picked from commit ee40efcea4)
* stable-5.2:
Prepare 5.1.9-SNAPSHOT builds
JGit v5.1.8.201906050907-r
Test detecting modified packfiles
Enhance fsTick() to use filesystem timer resolution
Add debug trace to measure time needed to open pack index
Extend FileSnapshot for packfiles to also use checksum to detect changes
Wait opening new packfile until it can't be racy anymore
Avoid null PackConfig in GC
Add FileSnapshot test testing recognition of file size changes
Capture reason for result of FileSnapshot#isModified
Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
Tune max heap size for tests
Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
ObjectDirectory: fix closing of obsolete packs
Include filekey file attribute when comparing FileSnapshots
Measure file timestamp resolution used in FileSnapshot
Fix FileSnapshot's consideration of file size
Fix API problem filters
Fix API problem filters
Change-Id: Id79a3a19f9a31dff94d10a406c2b6e08a506931a
* stable-5.1:
Prepare 5.1.9-SNAPSHOT builds
JGit v5.1.8.201906050907-r
Test detecting modified packfiles
Enhance fsTick() to use filesystem timer resolution
Add debug trace to measure time needed to open pack index
Extend FileSnapshot for packfiles to also use checksum to detect changes
Wait opening new packfile until it can't be racy anymore
Avoid null PackConfig in GC
Add FileSnapshot test testing recognition of file size changes
Capture reason for result of FileSnapshot#isModified
Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
Tune max heap size for tests
Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
ObjectDirectory: fix closing of obsolete packs
Include filekey file attribute when comparing FileSnapshots
Measure file timestamp resolution used in FileSnapshot
Fix FileSnapshot's consideration of file size
Fix API problem filters
Change-Id: I3ac77bfa03f7436de12ab86e1bba29afee5ccd01
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Test that JGit detects that packfiles have changed even if they are
repacked multiple times in one tick of the filesystem timer.
Test that this detection works also when repacking doesn't change the
length or the filekey of the packfile. In this case where a modified
file can't be detected by looking at file metadata JGit should still
detect too fast modification by racy git checks and trigger rescanning
the pack list and consequently rereading of packfile content.
Change-Id: I67682cfb807c58afc6de9375224ff7489d6618fb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Enhance fsTick() to use filesystem timer resolution
RepositoryTestCase.fsTick() was was waiting 64, 128, 256, ... milliseconds
until it detected that the filesystem timer has ticked. Make use of
the filesystemtimer resolution information in FS to sleep a fraction
of the filesystem timer resolution. That raises probability to wake up
shortly after the filesystem timer has ticked.
Change-Id: Ibcc38576e42ece13b2fd4423a29c459eed167a69
Extend FileSnapshot for packfiles to also use checksum to detect changes
If the attributes of FileSnapshot don't detect modification of a
packfile read the packfile's checksum and compare it against the
checksum cached in the loaded packfile.
Since reading the checksum needs less IO than reloading the complete
packfile this may help to reduce the overhead to detect modficiation
when a gc completes while ObjectDirectory scans for packfiles in another
thread.
Bug: 546891
Change-Id: I9811b497eb11b8a85ae689081dc5d949ca8c4be5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Wait opening new packfile until it can't be racy anymore
If
- pack.waitPreventRacyPack = true (default is false)
- packfile size > pack.minSizePreventRacyPack (default is 100 MB)
wait after a new packfile was written and before it is opened until it
cannot be racy anymore.
If a new packfile is accessed while it's still racy at least the pack's
index will be reread by ObjectDirectory.scanPacksImpl(). Hence it may
save resources to wait one tick of the file system timer to avoid this
reloading. On filesystems with a coarse timestamp resolution it may be
beneficial to skip this wait for small packfiles.
Bug: 546891
Change-Id: I0e8bf3d7677a025edd2e397dd2c9134ba59b1a18
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Initialize it using the repository's config already in the constructor.
Change-Id: I4ea620a7db72956e7109f739990f09644640206b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Capture reason for result of FileSnapshot#isModified
This allows to verify the expected behavior in
FileSnapshotTest#testSimulatePackfileReplacement and enables extending
FileSnapshot for packfiles to read the packfile's checksum as another
criterion to detect modifications without reading the full content.
Also add another field capturing the result of the last check if
lastModified was racily clean.
Remove unnecessary determination of raciness in the constructor. It was
determined twice in all relevant cases.
Change-Id: I100a2f49d7949693d7b72daa89437e166f1dc107
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
NTFS does not support FileKey hence ignore this test on Windows.
Change-Id: I7b53a591daa5e03eb5e401b5b26d612ab68ce10d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].
Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test
Decrease it to 768m for
- org.eclipse.jgit.test
[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console
Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit e19e859977)
Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
testNewFileNoWait() was identical to testNewFileWithWait() but claims it
doesn't wait at all. Hence remove the waits.
Change-Id: I49b8ca5cb49a43c55fe61870c18c42f32fb4b74d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This resolves a regression introduced in fef78212.
Change-Id: Ibb4521635a87012520566efc70870c59d11be874
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Based on recent work by Dave Borowitz on the Gerrit project [1, 2].
The warnings/error configuration is unchanged, but now the checks are
enabled by default during the build rather than having to be manually
invoked.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/225653
[2] https://gerrit-review.googlesource.com/c/gerrit/+/225714
Change-Id: I7f4849896af72de38880b13d64519cc93bb3d2a3
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
SystemReader: Respect passed-in parent when no system config is present
The default implementation of openSystemConfig has special handling for
when the FS returns null from getGitSystemConfig: it should return a
"real" FileBasedConfig instance that never actually tries to load a
file. However, this codepath was not respecting the passed-in parent
config.
Change-Id: Id0bcdc93bb42f9ebe3f5ee4c6b4be8863e0133f9
Include filekey file attribute when comparing FileSnapshots
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Some filesystems expose unique file identifiers, e.g. inodes in Posix
filesystems which are named filekeys in Java's BasicFileAttributes. Use
them as another means to detect file modifications based on stat
information.
Running git gc on a repository yields a new packfile with the same id as
a packfile which existed before the gc if these packfiles contain the
same set of objects. The content of the old and the new packfile might
differ if a different PackConfig was used when writing the packfile.
Considering filekeys in FileSnapshot may help to detect such packfile
modifications.
Bug: 546891
Change-Id: I711a80328c55e1a31171d540880b8e80ec1fe095
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Measure file timestamp resolution used in FileSnapshot
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].
Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.
Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.
Measured Java file timestamp resolution: [2]
[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo
Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn
David Ostrovsky
David Pursehouse
Han-Wen Nienhuys
Masaya Suzuki
Christian Halstrick
Marc Strapetz
Dave Borowitz