Persist minimal racy threshold and allow manual configuration
To enable persisting the minimal racy threshold per FileStore add a
new config option to the user global git configuration:
- Config section is "filesystem"
- Config subsection is concatenation of
- Java vendor (system property "java.vendor")
- Java version (system property "java.version")
- FileStore's name, on Windows we use the attribute volume:vsn instead
since the name is not necessarily unique.
- separated by '|'
e.g.
"AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1"
The same prefix is used as for filesystem timestamp resolution, so
both values are stored in the same config section
- The config key for minmal racy threshold is "minRacyThreshold" as a
time value, supported time units are those supported by
DefaultTypedConfigGetter#getTimeUnit
- measure for 3 seconds to limit runtime which depends on hardware, OS
and Java version being used
If the minimal racy threshold is configured for a given FileStore the
configured value is used instead of measuring it.
When the minimal racy threshold was measured it is persisted in the user
global git configuration.
Rename FileStoreAttributeCache to FileStoreAttributes since this class
is now declared public in order to enable exposing all attributes in one
object.
Example:
[filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"]
timestampResolution = 7000 nanoseconds
minRacyThreshold = 3440 microseconds
Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
Measure minimum racy interval to auto-configure FileSnapshot
By running FileSnapshotTest#detectFileModified we found that the sum of
measured filesystem timestamp resolution and measured clock resolution
may yield a too small interval after a file has been modified which we
need to consider racily clean. In our tests we didn't find this behavior
on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11
this effect was not observed.
On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the
following test results using Java 8 and 11:
In 23-98% of 10000 test runs (depending on filesystem type and Java
version) the test failed, which means the effective interval which needs
to be considered racily clean after a file was modified is larger than
the measured file timestamp resolution.
"delta" is the observed interval after a file has been modified but
FileSnapshot did not yet detect the modification:
"resolution" is the measured sum of file timestamp resolution and clock
resolution seen in Java.
Java version filesystem failures resolution min delta max delta
1.8.0_212-b04 btrfs 98.6% 1 ms 3.6 ms 6.6 ms
1.8.0_212-b04 ext4 82.6% 3 ms 1.1 ms 4.1 ms
1.8.0_212-b04 xfs 23.8% 4 ms 3.7 ms 3.9 ms
1.8.0_212-b04 zfs 23.1% 3 ms 4.8 ms 5.0 ms
11.0.3+7 btrfs 98.1% 3 us 0.7 ms 4.7 ms
11.0.3+7 ext4 98.1% 6 us 0.7 ms 4.7 ms
11.0.3+7 xfs 98.5% 7 us 0.1 ms 8.0 ms
11.0.3+7 zfs 98.4% 7 us 0.7 ms 5.2 ms
Mac OS
1.8.0_212 APFS 0% 1 s
11.0.3+7 APFS 0% 6 us
The observed delta is not distributed according to a normal gaussian
distribution but rather random in the observed range between "min delta"
and "max delta".
Run this test after measuring file timestamp resolution in
FS.FileAttributeCache to auto-configure JGit since it's unclear what
mechanism is causing this effect.
In FileSnapshot#isRacyClean use the maximum of the measured timestamp
resolution and the measured "delta" as explained above to decide if a
given FileSnapshot is to be considered racily clean. Add a 30% safety
margin to ensure we are on the safe side.
Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use Instant instead of milliseconds for filesystem timestamp handling
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.
Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution
Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Enhance fsTick() to use filesystem timer resolution
RepositoryTestCase.fsTick() was was waiting 64, 128, 256, ... milliseconds
until it detected that the filesystem timer has ticked. Make use of
the filesystemtimer resolution information in FS to sleep a fraction
of the filesystem timer resolution. That raises probability to wake up
shortly after the filesystem timer has ticked.
Change-Id: Ibcc38576e42ece13b2fd4423a29c459eed167a69
Measure file timestamp resolution used in FileSnapshot
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].
Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.
Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.
Measured Java file timestamp resolution: [2]
[1] https://github.com/git/git/blob/master/Documentation/technical/racy-git.txt
[2] https://docs.google.com/spreadsheets/d/1imy0y6WmRqBf0kjCxzxj2X7M50eIVfa7oaUIzEOHmjo
Bug: 546891
Change-Id: I493f3b57b6b306285ffa7d392339d253e5966ab8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Open auto-closeable resources in try-with-resource
When an auto-closeable resources is not opened in try-with-resource,
the warning "should be managed by try-with-resource" is emitted by
Eclipse.
Fix the ones that can be silenced simply by moving the declaration of
the variable into a try-with-resource.
In cases where we explicitly call the close() method, for example in
tests where we are testing specific behavior caused by the close(),
suppress the warning.
Leave the ones that will require more significant refcactoring to fix.
They can be done in separate commits that can be reviewed and tested
in isolation.
Change-Id: I9682cd20fb15167d3c7f9027cecdc82bc50b83c4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Replace hard-coded "UTF-8" string with the constant.
Change-Id: Ie812add2df28e984090563ec7c6e2c0366616424
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Merges are performed using the raw text as stored in the git
repository. When we write the merge result, we must apply the
correct CRLF settings. Otherwise the line endings in the result
will be wrong.
Bug: 499615
Change-Id: I37a9b987e9404c97645d2720cd1c7c04c076a96b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
RepositoryTestCase: Open autocloseable types in try-with-resource
ObjectInserter.Formatter and Git are autocloseable and can be
opened in try-with-resource to prevent a resource leak warning.
Change-Id: I48c4001aaa7d9c1e36369e9799bfbb7c3bb46d8b
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
When filters are defined for certain paths in gitattributes make
sure that clean filters are processed when adding new content to the
object database.
Change-Id: Iffd72914cec5b434ba4d0de232e285b7492db868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Delete non empty directories before checkout a path
If the checkout path is currently a non-empty directory (and was a link
or a regular file before), this directory will be removed before
performing checkout, but only if the checkout path is specified.
Bug: 474973
Change-Id: Ifc6c61592d9b54d26c66367163acdebea369145c
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Move `RepositoryTestCase.indexState` to parent class
The test helper method `indexState` in `RepositoryTestCase` is
very useful for writing tests, even in cases where we need to
do things like create more than one repository for a test and
thus we don't want to use the built-in `db` member variable that
exists in `RepositoryTestCase`. Since the method is static,
we can move it up to the parent class `LocalDiskRepositoryTestCase`,
where it can be used by tests that aren't a great fit for inheriting
directly from `RepositoryTestCase`.
Bug: 436200
Change-Id: I2b6de75c001d2d77ddb607488af246548784a67f
Signed-off-by: Chris Price <chris@puppetlabs.com>
The `indexState` method is no longer referring to any
member variables from the class, so it can be made static.
Bug: 436200
Change-Id: I013316de5c373417ea758ca6e17da29209fead53
Signed-off-by: Chris Price <chris@puppetlabs.com>
Use local variable in RepositoryTestCase.indexState
There is a signature of the test helper method `indexState`,
in `RepositoryTestCase`, that accepts a `Repository` object
as an argument. However, there was one line of code where
this variable was not being used, and the method was instead
referring to a member variable `db`. I believe this was
probably just an oversight in a previous refactor, and
that the correct behavior is to use the variable from
the argument list. This change also has the benefit
of making it possible to convert this method to a static
method, since it no longer relies on any state from the class.
Bug: 436200
Change-Id: Iac95b046dc5bd0b3756642e241c3637f1fad3609
Signed-off-by: Chris Price <chris@puppetlabs.com>
Fix RecursiveMerger's internal use of merge to find a merge base
When RecursiveMerger tried to determine a common base tree it was
recursively tried to merge multiple common bases. But these intermediate
merges which have just been done to determine a single common base for
the final merge already filled some important fields (toBeCheckedOut,
toBeDeleted, ...). These side effects of the intermediate merges led to
wrong results of the final merge. One symptom was that after a recursive
merge which should be succesful you could still see leftover files in
the worktree: files which existed in the (virtual) common base but which
don't exist anymore in the branches to be merged.
The solution is easy: Clear the appropriate fields after common base
determination and start the final merge with a clean state.
Change-Id: I644ea9e1cb15360f7901bc0483cdb9286308c226
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Try to fix intermittent test failure related to file timestamps
Improve how tests do a "touch" operation on files. Instead of doing
"f.setLastModified(System.currentTimeMillis)" open a Outputstream on the
file and directly close it again. This makes this method rely only on
one clock - the clock of the filesystem.
Bug: 436917
Change-Id: I68ef3c2878f28b12daebf2ef6a9fa0a5d6e0964d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix failed tests when the tmp directory is a symlink
On Mac OS X /tmp is by default a symbolic link.
Change-Id: I0913ee5ee8db5c5918a9e41abbbfe125b6c70783
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The most important difference is that in Java7 we have symbolic links
and for most operations in the work tree we want to operate on the link
itself rather than the link target, which the old File methods generally
do.
We also add support for the hidden attribute, which only makes sense
on Windows and exists, just since there are claims that Files.exists
is faster the File.exists.
A new bundle is only activated when run with a Java7 execution
environment. It is implemented as a fragment.
Tycho currently has no way to conditionally include optional features
based on the java version used to run the build, this means with this
change the jgit packaging build always needs to be run using java 7.
Change-Id: I3d6580d6fa7b22f60d7e54ab236898ed44954ffd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
JGit 3.0: move internal classes into an internal subpackage
This breaks all existing callers once. Applications are not supposed
to build against the internal storage API unless they can accept API
churn and make necessary updates as versions change.
Change-Id: I2ab1327c202ef2003565e1b0770a583970e432e9
Ensure a directory exists before trying to create/merge a file into it.
Since git doesn't keep track of empty directories, they should be
created first. Test case included demonstrates that using
StashApplyCommand(). Bugfix is applied to the DirCacheCheckout class,
because StashApplyCommand() uses it internally to apply a stash.
Change-Id: Iac259229ef919f9e92e7e51a671d877172bb88a8
Signed-off-by: Jevgeni Zelenkov <jevgeni.zelenkov@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Again teach ResolveMerger to create more correct DirCacheEntry's
Currently, after a merge/cherry-pick/rebase, all index entries are
smudged as the ResolveMerger never sets entry lengths and/or
modification times. This change teaches it to re-set them at least for
things it did not touch. The other entries are then repaired when the
index is persisted, or entries are checked out.
The first attempt to get this in was commit
3ea694c252 which has been reverted.
Since then some fixes to ResolveMerger and a few more tests have
been added which check situations where the index is not matching
HEAD before we merge.
Change-Id: I648fda30846615b3bf688c34274c6cf4bc857832
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Also-by: Markus Duft <markus.duft@salomon.at>
Teach ResolveMerger to create more correct DirCacheEntry's
Currently, after a merge/cherry-pick/rebase, all index entries are
smudged as the ResolveMerger never sets entry lengths and/or
modification times. This change teaches it to re-set them at least for
things it did not touch. The other entries are then repaired when the
index is persisted, or entries are checked out.
Change-Id: I0944f2017483d32043d0d09409b13055b5609a4b
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Move writeTrashFile and deleteFile into JGitTestUtil
Moves RepositoryTestCase.writeThashFile, RepositoryTestCase.deleteFile
and dependencies into JGitTestUtil for further reuse.
Required-by-EGit: If8dfa0251797aca56ddc825619500dc21885ba26
Change-Id: I6fc62c8e6626f907e544b5bbe5d64e864a2c323f
Signed-off-by: Dariusz Luksza <dariusz@luksza.org>
This change moves commonly used methods into the RepositoryTestCase
base class.
Change-Id: I56a46c31ee1661c7ce22eb755ab23da8bc9f5da2
Signed-off-by: Philipp Thun <philipp.thun@sap.com>
Checkout of remote tracking branch failed when no local branch
existed. Also enhance RepositoryTestCase to enable checking index
state of another test repository.
Bug: 337695
Change-Id: Idf4c05bdf23b5161688818342b2bf9a45b49f479
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Eclipse has some problem re-running single JUnit tests if
the tests are in Junit 3 format, but the JUnit 4 launcher
is used. This was quite unnecessary and the move was not
completed. We still have no JUnit4 test.
This completes the extermination of JUnit3. Most of the
work was global searce/replace using regular expression,
followed by numerous invocarions of quick-fix and organize
imports and verification that we had the same number of
tests before and after.
- Annotations were introduced.
- All references to JUnit3 classes removed
- Half-good replacement for getting the test name. This was
needed to make the TestRngs work. The initialization of
TestRngs was also made lazily since we can not longer find
out the test name in runtime in the @Before methods.
- Renamed test classes to end with Test, with the exception
of TestTranslateBundle, which fails from Maven
- Moved JGitTestUtil to the junit support bundle
Change-Id: Iddcd3da6ca927a7be773a9c63ebf8bb2147e2d13
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
java.io.File.delete() reports failure as an exceptional
return value false. Fix the code which silently ignored
this exceptional return value. Also remove some duplicate
deletion helper methods.
Change-Id: I80ed20ca1f07a2bc6e779957a4ad0c713789c5be
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When the assume unchanged flag is set the Add command must not update
the index for this file if any changes are present in the working
directory.
Bug: 331351
Change-Id: I255870f689225a1d88971182e0eb377952641b42
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Check for deletion of temporary files in .git folder.
Check for deletion and creation of files.
Change-Id: I60b0b2975724f2e3582e8674d9f876dcbf62b350
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Signed-off-by: Mathias Kinzler <mathias.kinzler@sap.com>
indexState() encodes the complete state of the index
into one readable String. This helps to write tests
against the index. indexState() is enhanced to optionally
also contain the content of the files in the index.
Change-Id: Ie988f93768d864f4cbd55809a786bd5759fc24a5
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Added a utility method to set the reset an index to match exactly
some content in the filesystem. This can be used by tests to prepare
commits in the working-tree and set the index in one shot.
[sp: Cleaned up formatting, added getEntryFile(), released inserter.]
Change-Id: If38b1f7cacaaf769f51b14541c5da0c1e24568a5
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This change is mainly done for a subsequent commit
which will introduce the "all" parameter to the Commit
command.
Bug: 318439
Change-Id: I85a8a76097d0197ef689a289288ba82addb92fc9
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Matthias Sohn
Christian Halstrick
Carsten Hammer
David Pursehouse
Han-Wen Nienhuys
Thomas Wolf
XinTong Wang
David Pursehouse
Andrey Loskutov
Chris Price
Robin Rosenberg
Shawn Pearce
Robin Stocker
Jevgeni Zelenkov
Shawn Pearce
Markus Duft
Dariusz Luksza
Philipp Thun
Stefan Lay