Matthias Sohn [Wed, 5 Jun 2019 19:18:21 +0000 (21:18 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Prepare 5.1.9-SNAPSHOT builds
JGit v5.1.8.201906050907-r
Test detecting modified packfiles
Enhance fsTick() to use filesystem timer resolution
Add debug trace to measure time needed to open pack index
Extend FileSnapshot for packfiles to also use checksum to detect changes
Wait opening new packfile until it can't be racy anymore
Avoid null PackConfig in GC
Add FileSnapshot test testing recognition of file size changes
Capture reason for result of FileSnapshot#isModified
Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
Tune max heap size for tests
Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
ObjectDirectory: fix closing of obsolete packs
Include filekey file attribute when comparing FileSnapshots
Measure file timestamp resolution used in FileSnapshot
Fix FileSnapshot's consideration of file size
Fix API problem filters
Fix API problem filters
Matthias Sohn [Wed, 5 Jun 2019 13:43:29 +0000 (15:43 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Prepare 5.1.9-SNAPSHOT builds
JGit v5.1.8.201906050907-r
Test detecting modified packfiles
Enhance fsTick() to use filesystem timer resolution
Add debug trace to measure time needed to open pack index
Extend FileSnapshot for packfiles to also use checksum to detect changes
Wait opening new packfile until it can't be racy anymore
Avoid null PackConfig in GC
Add FileSnapshot test testing recognition of file size changes
Capture reason for result of FileSnapshot#isModified
Skip FileSnapshotTest#testSimulatePackfileReplacement on Windows
Tune max heap size for tests
Fix FileSnapshotTest.testNewFileNoWait() to match its javadoc
ObjectDirectory: fix closing of obsolete packs
Include filekey file attribute when comparing FileSnapshots
Measure file timestamp resolution used in FileSnapshot
Fix FileSnapshot's consideration of file size
Fix API problem filters
Change-Id: I3ac77bfa03f7436de12ab86e1bba29afee5ccd01 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Test that JGit detects that packfiles have changed even if they are
repacked multiple times in one tick of the filesystem timer.
Test that this detection works also when repacking doesn't change the
length or the filekey of the packfile. In this case where a modified
file can't be detected by looking at file metadata JGit should still
detect too fast modification by racy git checks and trigger rescanning
the pack list and consequently rereading of packfile content.
Change-Id: I67682cfb807c58afc6de9375224ff7489d6618fb Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Enhance fsTick() to use filesystem timer resolution
RepositoryTestCase.fsTick() was was waiting 64, 128, 256, ... milliseconds
until it detected that the filesystem timer has ticked. Make use of
the filesystemtimer resolution information in FS to sleep a fraction
of the filesystem timer resolution. That raises probability to wake up
shortly after the filesystem timer has ticked.
Matthias Sohn [Sun, 26 May 2019 20:40:47 +0000 (22:40 +0200)]
Extend FileSnapshot for packfiles to also use checksum to detect changes
If the attributes of FileSnapshot don't detect modification of a
packfile read the packfile's checksum and compare it against the
checksum cached in the loaded packfile.
Since reading the checksum needs less IO than reloading the complete
packfile this may help to reduce the overhead to detect modficiation
when a gc completes while ObjectDirectory scans for packfiles in another
thread.
Bug: 546891
Change-Id: I9811b497eb11b8a85ae689081dc5d949ca8c4be5 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 8 May 2019 23:23:15 +0000 (01:23 +0200)]
Wait opening new packfile until it can't be racy anymore
If
- pack.waitPreventRacyPack = true (default is false)
- packfile size > pack.minSizePreventRacyPack (default is 100 MB)
wait after a new packfile was written and before it is opened until it
cannot be racy anymore.
If a new packfile is accessed while it's still racy at least the pack's
index will be reread by ObjectDirectory.scanPacksImpl(). Hence it may
save resources to wait one tick of the file system timer to avoid this
reloading. On filesystems with a coarse timestamp resolution it may be
beneficial to skip this wait for small packfiles.
Bug: 546891
Change-Id: I0e8bf3d7677a025edd2e397dd2c9134ba59b1a18 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 9 May 2019 22:58:42 +0000 (00:58 +0200)]
Capture reason for result of FileSnapshot#isModified
This allows to verify the expected behavior in
FileSnapshotTest#testSimulatePackfileReplacement and enables extending
FileSnapshot for packfiles to read the packfile's checksum as another
criterion to detect modifications without reading the full content.
Also add another field capturing the result of the last check if
lastModified was racily clean.
Remove unnecessary determination of raciness in the constructor. It was
determined twice in all relevant cases.
Change-Id: I100a2f49d7949693d7b72daa89437e166f1dc107 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 22 May 2019 12:42:43 +0000 (14:42 +0200)]
Tune max heap size for tests
This is an attempt to fix crashes observed on the new Jenkins
infrastructure running on Kubernetes [1].
Increase it to 512m for
- org.eclipse.jgit.ant.test
- org.eclipse.jgit.http.test
- org.eclipse.jgit.lfs.server.test
- org.eclipse.jgit.lfs.test
- org.eclipse.jgit.pgm.test
Decrease it to 768m for
- org.eclipse.jgit.test
[1] e.g. https://ci-staging.eclipse.org/jgit/job/stable/job/jgit.gerrit/16074/console
Change-Id: Id074ed0f7bcb8a13da649a547342af2a08439d9f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit e19e859977525c2a39aaa928dfdef20e5fab7e3c)
Dave Borowitz [Thu, 23 May 2019 20:56:00 +0000 (13:56 -0700)]
SystemReader: Respect passed-in parent when no system config is present
The default implementation of openSystemConfig has special handling for
when the FS returns null from getGitSystemConfig: it should return a
"real" FileBasedConfig instance that never actually tries to load a
file. However, this codepath was not respecting the passed-in parent
config.
Matthias Sohn [Tue, 7 May 2019 21:55:54 +0000 (23:55 +0200)]
Include filekey file attribute when comparing FileSnapshots
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Some filesystems expose unique file identifiers, e.g. inodes in Posix
filesystems which are named filekeys in Java's BasicFileAttributes. Use
them as another means to detect file modifications based on stat
information.
Running git gc on a repository yields a new packfile with the same id as
a packfile which existed before the gc if these packfiles contain the
same set of objects. The content of the old and the new packfile might
differ if a different PackConfig was used when writing the packfile.
Considering filekeys in FileSnapshot may help to detect such packfile
modifications.
Bug: 546891
Change-Id: I711a80328c55e1a31171d540880b8e80ec1fe095 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sun, 5 May 2019 01:18:23 +0000 (03:18 +0200)]
Measure file timestamp resolution used in FileSnapshot
FileSnapshot.notRacyClean() assumed a worst case filesystem timestamp
resolution of 2.5 sec (FAT has a resolution of 2 sec). Instead measure
timestamp resolution to avoid unnecessary IO caused by false positives
in detecting the racy git problem caused by finite filesystem timestamp
resolution [1].
Cache the measured resolution per FileStore since timestamp resolution
depends on the respective filesystem type. If timestamp resolution
cannot be measured or fails due to an exception fallback to the worst
case FAT timestamp resolution and avoid caching this value.
Add a 10% safety margin in FileSnapshot.notRacyClean(), though running
FsTest.testFsTimestampResolution() 1000 times which is not using a
safety margin didn't fail on Mac using APFS and Java 8, 11, 12.
Matthias Sohn [Sat, 27 Apr 2019 22:21:16 +0000 (00:21 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Prepare 5.2.3-SNAPSHOT builds
JGit v5.2.2.201904231744-r
Revert 4678f4b and provide another solution for bug 467631
Apache MINA sshd: make sendKexInit() work also for re-keying
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: Ie7e572ac7e346f21fe0c387d7448be168a9c127a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Sun, 10 Mar 2019 17:02:54 +0000 (18:02 +0100)]
Revert 4678f4b and provide another solution for bug 467631
Making gitlinks and folders match in a tree walk was the wrong
approach to fix bug 467631. The problem is that in such a conflict
the tree walk may then not descend into the folder.
Revert the changes to Paths.java and PathsTest.java from commit 4678f4b. Instead test for the problem case from bug 467631 explicitly
in IndexDiff. Add Daniel's test case from bug 545162, and add yet
another test case for DiffEntry.scan() that covers the problem
originally reported in bug 545162.
Bug: 545162
Change-Id: Ie2214c5d5ee32ac6596b621f0f1c7b86d38fa9b7 Also-by: Daniel Veihelmann <daniel.veihelmann@gmail.com> Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Thomas Wolf [Tue, 2 Apr 2019 10:34:26 +0000 (12:34 +0200)]
Apache MINA sshd: make sendKexInit() work also for re-keying
The message delaying for the initial SSH messages (client
identification followed by the initial key exchange request)
was broken. sendKexInit() is _also_ called when a new key exchange
is requested. We inadvertently also re-sent the client identification
at that point, which is wrong and makes the server terminate the
connection.
Re-keying occurs from time to time during an SSH connection depending
on time, the number of messages (packets/blocks) exchanged, or the
amount of data exchanged. The net result was that for large
repositories data-intensive operations failed on the first re-keying.
Change the initial message delay such that the two messages for the
client identification and the initial key exchange can be buffered
individually while the proxy protocol is still in progress. The
AbstractClientProxyConnector can now buffer several commands, which
should also resolve bug 544715.
Bug: 545920
Change-Id: If09ee963a439b39098a0f52a1510237b428df8dd Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn [Sat, 20 Apr 2019 09:16:34 +0000 (11:16 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Prepare 5.1.8-SNAPSHOT builds
JGit v5.1.7.201904200442-r
ObjectUploadListener: Add callback interface
Prepare 4.11.9-SNAPSHOT builds
JGit v4.11.8.201904181247-r
Prepare 4.9.11-SNAPSHOT builds
JGit v4.9.10.201904181027-r
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I76761002eedf360e93d0559942ebc927a40428d6 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 18 Apr 2019 12:43:23 +0000 (14:43 +0200)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Matthias Sohn [Tue, 16 Apr 2019 22:49:03 +0000 (00:49 +0200)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I7984f68833f2d615399296e53cb9a64e5b4ca6ed Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 16 Apr 2019 21:53:21 +0000 (23:53 +0200)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I6ea51dde6608a4163d681aa1ebf710f06da44b21 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 15 Apr 2019 23:04:52 +0000 (01:04 +0200)]
Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I0b4eaa521ebdea83ab18c05915d691c07a575a7d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Keep track of the original cause for a packfile invalidation.
It is needed for the sysadmin to understand if there is a real
underlying filesystem problem and repository corruption or if it is
simply a consequence of a concurrency of Git operations (e.g. repack
or GC).
Change-Id: I06ddda9ec847844ec31616ab6d17f153a5a34e33 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix pack files scan when filesnapshot isn't modified
Do not reload packfiles when their associated filesnapshot is not
modified on disk compared to the one currently stored in memory.
Fix the regression introduced by fef78212 which, in conjunction with
core.trustfolderstats = false, caused any lookup of objects inside
the packlist to loop forever when the object was not found in the pack
list.
Matthias Sohn [Wed, 20 Mar 2019 12:54:10 +0000 (13:54 +0100)]
Fix GC to delete empty fanout directories after repacking
The prune method did not delete empty fanout directories when loose
objects moved to a new pack file but only when loose unreferenced
objects were pruned.
Change-Id: Ia068f4914c54d9cf9f40b75e8ea50759402b5000 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 09:14:07 +0000 (10:14 +0100)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Prepare 5.1.7-SNAPSHOT builds
JGit v5.1.6.201903130242-r
Prepare 4.11.8-SNAPSHOT builds
JGit v4.11.7.201903122105-r
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: Ia866587a910e620cb297bcb2a81fe0786f4e3db2 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 06:51:34 +0000 (07:51 +0100)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Prepare 5.1.7-SNAPSHOT builds
JGit v5.1.6.201903130242-r
Prepare 4.11.8-SNAPSHOT builds
JGit v4.11.7.201903122105-r
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I8da2761ed27ff2817a4136f843b9b293e3c0f4e6 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 06:37:36 +0000 (07:37 +0100)]
Merge branch 'stable-5.0' into stable-5.1
* stable-5.0:
Prepare 4.11.8-SNAPSHOT builds
JGit v4.11.7.201903122105-r
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I71771dcc086a9a0097bad74d65ec71ad97106a38 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 01:21:54 +0000 (02:21 +0100)]
Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
Prepare 4.11.8-SNAPSHOT builds
JGit v4.11.7.201903122105-r
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I04c70f66c6f1e53bdc199fb5b24c9136c9a8e4f7 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:55:10 +0000 (01:55 +0100)]
Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: Ic66dc00455c05f7627919f682a7221ffeef1629d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:48:06 +0000 (01:48 +0100)]
Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: Ifcd8ea6a2b59f6f8e9d896d8407cfcaef0a9375d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:17:47 +0000 (01:17 +0100)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I9a47153831f8eb10d3cd91b4157cf45385e5b13a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:16:28 +0000 (01:16 +0100)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I1136eaa1590b225ceaf16ba7552af0374253a74e Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 12 Mar 2019 21:39:53 +0000 (22:39 +0100)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I3d1544d034783fe0fa1385dfe9b03ad8e9247c63 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 12 Mar 2019 20:01:55 +0000 (21:01 +0100)]
Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I029e1797447e6729de68bd89d4d69b324dbb3f5f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio [Sun, 10 Mar 2019 22:03:40 +0000 (22:03 +0000)]
Check for packfile validity and fd before reading
When reading from a packfile, make sure that is valid
and has a non-null file-descriptor.
Because of concurrency between a thread invalidating a packfile
and another trying to read it, the read() may result into a NPE
that won't be able to be automatically recovered.
Throwing a PackInvalidException would instead cause the packlist
to be refreshed and the read to eventually succeed.
Luca Milanesio [Wed, 6 Mar 2019 11:30:07 +0000 (11:30 +0000)]
Move throw of PackInvalidException outside the catch
When a packfile is invalid, throw an exception explicitly
outside any catch scope, so that is not accidentally caught
by the generic catch-all cause, which would set the packfile
as valid again.
Flagging an invalid packfile as valid again would have
dangerous consequences such as the corruption of the in-memory
packlist.
Luca Milanesio [Tue, 12 Mar 2019 07:00:01 +0000 (07:00 +0000)]
Use FileSnapshot to get lastModified on PackFile
Do not redundantly call File.lastModified() for extracting the
timestamp of the PackFile but rather use consistently the FileSnapshot
which reads all file attributes in a single bulk call.
Change-Id: I932675ae4fe56dcd3833dac249816f097303bb09 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio [Wed, 6 Mar 2019 17:51:59 +0000 (17:51 +0000)]
Include size when comparing FileSnapshot
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Read and consider file size also, so that differing file size can help
to more accurately detect file changes without reading the file content.
Use bulk read to avoid multiple stat calls to retrieve file attributes.
Change-Id: I974288fff78ac78c52245d9218b5639603f67a46 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio [Wed, 6 Mar 2019 00:31:45 +0000 (00:31 +0000)]
Do not reuse packfiles when changed on filesystem
The pack reload mechanism from the filesystem works only by name
and does not check the actual last modified date of the packfile.
This lead to concurrency issues where multiple threads were loading
and removing from each other list of packfiles when one of those
was failing the checksum.
Rely on FileSnapshot rather than directly checking lastModified
timestamp so that more checks can be performed.
Juergen Denner [Tue, 15 Jan 2019 12:22:54 +0000 (13:22 +0100)]
Reduce contention on PackFile.idx() function.
In case of concurrent pack file access, threads may wait on the idx()
function even for already open files. This happens especially with a
slow file system.