Matthias Sohn [Tue, 3 Sep 2019 12:23:29 +0000 (14:23 +0200)]
Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: If672b4f0c350f4e8ff7e1e706485cffd8137236d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 3 Sep 2019 11:24:28 +0000 (13:24 +0200)]
Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Id5bf44645655fca40ad22bb1f1ad20a7c2e8f6db Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Han-Wen Nienhuys [Sat, 31 Aug 2019 19:33:58 +0000 (21:33 +0200)]
BatchRefUpdate: repro racy atomic update, and fix it
PackedBatchRefUpdate was creating a new packed-refs list that was
potentially unsorted. This would be papered over when the list was
read back from disk in parsePackedRef, which detects unsorted ref
lists on reading, and sorts them. However, the BatchRefUpdate also
installed the new (unsorted) list in-memory in
RefDirectory#packedRefs.
With the timestamp granularity code committed to stable-5.1, we can
more often accurately decide that the packed-refs file is clean, and
will return the erroneous unsorted data more often. Unluckily timed
delays also cause the file to be clean, hence this problem was
exacerbated under load.
The symptom is that refs added by a BatchRefUpdate would stop being
visible directly after they were added. In particular, the Gerrit
integration tests uses BatchRefUpdate in its setup for creating the
Admin group, and then tries to read it out directly afterward.
The tests recreates one failure case. A better approach would be to
revise RefList.Builder, so it detects out-of-order lists and
automatically sorts them.
Fixes https://bugs.eclipse.org/bugs/show_bug.cgi?id=548716 and
https://bugs.chromium.org/p/gerrit/issues/detail?id=11373.
Matthias Sohn [Tue, 18 Jun 2019 09:32:59 +0000 (11:32 +0200)]
Fix RacyGitTests#testRacyGitDetection
This test case assumed file system timestamp resolution of 1 second. On
filesystems with a finer resolution this test fails since the index
entry is only smudged if the file index entry's lastModified and the
lastModified of the git index itself are within the same filesystem
timer tick. Fix this by ensuring that these timestamps are identical
which should work for any filesystem timer resolution.
Bug: 548188
Change-Id: Id84d59e1cfeb48fa008f8f27f2f892c4f73985de Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
(cherry picked from commit 1159f9dd7c80a53c2509cd75d997a6afed37f9a6)
Matthias Sohn [Thu, 18 Apr 2019 12:43:23 +0000 (14:43 +0200)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Matthias Sohn [Tue, 16 Apr 2019 22:49:03 +0000 (00:49 +0200)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.10-SNAPSHOT builds
JGit v4.7.9.201904161809-r
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I7984f68833f2d615399296e53cb9a64e5b4ca6ed Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 16 Apr 2019 21:53:21 +0000 (23:53 +0200)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I6ea51dde6608a4163d681aa1ebf710f06da44b21 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 15 Apr 2019 23:04:52 +0000 (01:04 +0200)]
Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
Prepare 4.5.8-SNAPSHOT builds
JGit v4.5.7.201904151645-r
Remember the cause for invalidating a packfile
Fix API problem filters
Fix pack files scan when filesnapshot isn't modified
Change-Id: I0b4eaa521ebdea83ab18c05915d691c07a575a7d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Keep track of the original cause for a packfile invalidation.
It is needed for the sysadmin to understand if there is a real
underlying filesystem problem and repository corruption or if it is
simply a consequence of a concurrency of Git operations (e.g. repack
or GC).
Change-Id: I06ddda9ec847844ec31616ab6d17f153a5a34e33 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix pack files scan when filesnapshot isn't modified
Do not reload packfiles when their associated filesnapshot is not
modified on disk compared to the one currently stored in memory.
Fix the regression introduced by fef78212 which, in conjunction with
core.trustfolderstats = false, caused any lookup of objects inside
the packlist to loop forever when the object was not found in the pack
list.
Matthias Sohn [Wed, 20 Mar 2019 12:54:10 +0000 (13:54 +0100)]
Fix GC to delete empty fanout directories after repacking
The prune method did not delete empty fanout directories when loose
objects moved to a new pack file but only when loose unreferenced
objects were pruned.
Change-Id: Ia068f4914c54d9cf9f40b75e8ea50759402b5000 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:55:10 +0000 (01:55 +0100)]
Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: Ic66dc00455c05f7627919f682a7221ffeef1629d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:48:06 +0000 (01:48 +0100)]
Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
Prepare 4.9.10-SNAPSHOT builds
JGit v4.9.9.201903122025-r
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: Ifcd8ea6a2b59f6f8e9d896d8407cfcaef0a9375d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:17:47 +0000 (01:17 +0100)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I9a47153831f8eb10d3cd91b4157cf45385e5b13a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 13 Mar 2019 00:16:28 +0000 (01:16 +0100)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.9-SNAPSHOT builds
JGit v4.7.8.201903121755-r
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I1136eaa1590b225ceaf16ba7552af0374253a74e Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 12 Mar 2019 21:39:53 +0000 (22:39 +0100)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I3d1544d034783fe0fa1385dfe9b03ad8e9247c63 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 12 Mar 2019 20:01:55 +0000 (21:01 +0100)]
Merge branch 'stable-4.5' into stable-4.6
* stable-4.5:
Prepare 4.5.7-SNAPSHOT builds
JGit v4.5.6.201903121547-r
Check for packfile validity and fd before reading
Move throw of PackInvalidException outside the catch
Use FileSnapshot to get lastModified on PackFile
Include size when comparing FileSnapshot
Do not reuse packfiles when changed on filesystem
Silence API warnings for new API introduced for fixes
Change-Id: I029e1797447e6729de68bd89d4d69b324dbb3f5f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio [Sun, 10 Mar 2019 22:03:40 +0000 (22:03 +0000)]
Check for packfile validity and fd before reading
When reading from a packfile, make sure that is valid
and has a non-null file-descriptor.
Because of concurrency between a thread invalidating a packfile
and another trying to read it, the read() may result into a NPE
that won't be able to be automatically recovered.
Throwing a PackInvalidException would instead cause the packlist
to be refreshed and the read to eventually succeed.
Luca Milanesio [Wed, 6 Mar 2019 11:30:07 +0000 (11:30 +0000)]
Move throw of PackInvalidException outside the catch
When a packfile is invalid, throw an exception explicitly
outside any catch scope, so that is not accidentally caught
by the generic catch-all cause, which would set the packfile
as valid again.
Flagging an invalid packfile as valid again would have
dangerous consequences such as the corruption of the in-memory
packlist.
Luca Milanesio [Tue, 12 Mar 2019 07:00:01 +0000 (07:00 +0000)]
Use FileSnapshot to get lastModified on PackFile
Do not redundantly call File.lastModified() for extracting the
timestamp of the PackFile but rather use consistently the FileSnapshot
which reads all file attributes in a single bulk call.
Change-Id: I932675ae4fe56dcd3833dac249816f097303bb09 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio [Wed, 6 Mar 2019 17:51:59 +0000 (17:51 +0000)]
Include size when comparing FileSnapshot
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Read and consider file size also, so that differing file size can help
to more accurately detect file changes without reading the file content.
Use bulk read to avoid multiple stat calls to retrieve file attributes.
Change-Id: I974288fff78ac78c52245d9218b5639603f67a46 Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Luca Milanesio [Wed, 6 Mar 2019 00:31:45 +0000 (00:31 +0000)]
Do not reuse packfiles when changed on filesystem
The pack reload mechanism from the filesystem works only by name
and does not check the actual last modified date of the packfile.
This lead to concurrency issues where multiple threads were loading
and removing from each other list of packfiles when one of those
was failing the checksum.
Rely on FileSnapshot rather than directly checking lastModified
timestamp so that more checks can be performed.
Juergen Denner [Tue, 15 Jan 2019 12:22:54 +0000 (13:22 +0100)]
Reduce contention on PackFile.idx() function.
In case of concurrent pack file access, threads may wait on the idx()
function even for already open files. This happens especially with a
slow file system.
* stable-4.10:
Fix error log message in ObjectDirectory.handlePackError()
Properly format pack checksums in PackFile.idx()
Cancel gc if thread was interrupted
PackFile: report correct message for checksum mismatch
ObjectDirectory: Clean up logging
Bazel: Stop using native.git_repository
ObjectDirectory: extra logging on packfile exceptions
Change-Id: I9052e318b5d920770f7c7121d36e3c58df9d5f5a Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
* stable-4.9:
Fix error log message in ObjectDirectory.handlePackError()
Properly format pack checksums in PackFile.idx()
Cancel gc if thread was interrupted
PackFile: report correct message for checksum mismatch
ObjectDirectory: Clean up logging
Bazel: Stop using native.git_repository
ObjectDirectory: extra logging on packfile exceptions
Change-Id: I0847251eb010616a705e0b91df4bdebc225fa95d Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Luca Milanesio [Sat, 23 Feb 2019 21:57:09 +0000 (21:57 +0000)]
PackFile: report correct message for checksum mismatch
When the packfile checksum does not match the expected one
report the correct checksum error instead of reporting that
the number of objects is incorrect.
Change-Id: I040f36dacc4152ae05453e7acbf8dfccceb46e0d Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
(cherry picked from commit 436c99ce5946f31f06b8704b1fd33136f39dc814)
Matthias Sohn [Mon, 24 Dec 2018 23:49:46 +0000 (00:49 +0100)]
Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
Prepare 4.9.9-SNAPSHOT builds
JGit v4.9.8.201812241815-r
UploadPack: Test filtering by AdvertiseRefsHook in stateless transports
Prepare 4.7.8-SNAPSHOT builds
JGit v4.7.7.201812240805-r
Fix feature versions imported by feature org.eclipse.jgit.pgm
Prepare 4.5.6-SNAPSHOT builds
JGit v4.5.5.201812240535-r
Call AdvertiseRefsHook before validating wants
Change-Id: I937e9a4547fc10e4de7c887163022d1ab0322d64 Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 24 Dec 2018 23:33:44 +0000 (00:33 +0100)]
Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
Prepare 4.9.9-SNAPSHOT builds
JGit v4.9.8.201812241815-r
UploadPack: Test filtering by AdvertiseRefsHook in stateless transports
Prepare 4.7.8-SNAPSHOT builds
JGit v4.7.7.201812240805-r
Fix feature versions imported by feature org.eclipse.jgit.pgm
Prepare 4.5.6-SNAPSHOT builds
JGit v4.5.5.201812240535-r
Call AdvertiseRefsHook before validating wants
Change-Id: I2e499f34b1c481af794fa9325b0dfebaccdf3cb0 Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Masaya Suzuki [Tue, 18 Dec 2018 17:20:54 +0000 (09:20 -0800)]
UploadPack: Test filtering by AdvertiseRefsHook in stateless transports
AdvertiseRefsHook is used to limit the visibility of the refs in Gerrit.
If this hook is not called, then all refs are treated as visible,
causing the server to serve commits reachable from branches the client
should not be able to access, if asked to via a request naming a guessed
object id.
Until 3a529361a76e8267467071e0b13ebb36b97d8fb2 (Call AdvertiseRefsHook
before validating wants, 2018-12-18), UploadPack would invoke this hook
at ref advertisement time but not during negotiation and when serving a
pack file. Add a test to avoid regressing. Stateful bidirectional
transports were not affected, so the test uses HTTP.
[jn: split out when backporting the fix to stable-4.5. The test passes
as long as v4.9.0.201710071750-r~169 (fetch: Accept any SHA-1 on lhs of
refspec, 2017-06-04) is cherry picked along with it.]
Change-Id: I8c017107336adc7cb4c826985779676bf043e648 Signed-off-by: Masaya Suzuki <masayasuzuki@google.com> Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Change-Id: Ib44e314a68bca2349b45f4937257aa1298c8d74b Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Change-Id: I68a21067705b580b40840f8039001ff1e5273c15 Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 24 Dec 2018 12:25:31 +0000 (13:25 +0100)]
Merge branch 'stable-4.6' into stable-4.7
* stable-4.6:
Fix feature versions imported by feature org.eclipse.jgit.pgm
Prepare 4.5.6-SNAPSHOT builds
JGit v4.5.5.201812240535-r
Call AdvertiseRefsHook before validating wants
Change-Id: If637694f80dbd1e774d60c672fe78a6500650bb8 Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Change-Id: I0fd67ddd9c4966c20d82cdfe78b2f9d4898b4665 Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Masaya Suzuki [Tue, 18 Dec 2018 17:20:54 +0000 (09:20 -0800)]
Call AdvertiseRefsHook before validating wants
AdvertiseRefsHook is used to limit the visibility of the refs in Gerrit.
If this hook is not called, then all refs are treated as visible,
causing the server to serve commits reachable from branches the client
should not be able to access, if asked to via a request naming a guessed
object id.
This bug was introduced in v2.0.0.201206130900-r~123 (Modify refs in
UploadPack/ReceivePack using a hook interface, 2012-02-08). Stateful
bidirectional transports are not affected.
Fix it by moving the AdvertiseRefsHook call to
getAdvertisedOrDefaultRefs, ensuring the hook is called in all cases.
[jn: backported to stable-4.5 by splitting out tests and the protocol v2
specific parts]
Change-Id: I159f396216354f2eda3968d17802e166d8c8ec2d Signed-off-by: Masaya Suzuki <masayasuzuki@google.com> Signed-off-by: Jonathan Nieder <jrn@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sun, 25 Nov 2018 22:15:11 +0000 (23:15 +0100)]
Fix DescribeCommand with multiple match options
when multiple match options are given in git describe the result must
not depend on the order of the match options. JGit wrongly picked the
first match using the match options in the order they were defined. Fix
this by concatenating the streams of matching tags for all match options
and then choosing the first match on the concatenated stream sorted in
tie break order.
See https://git-scm.com/docs/git-describe#git-describe---matchltpatterngt
Change-Id: Id01433d35fa16fb4c30526605bee041ac1d954b2 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Håvard Wall [Wed, 17 Oct 2018 13:34:51 +0000 (15:34 +0200)]
Fix git-describe tie-breakers
Correct behaviour as git 1.7.1.1 is to resolve tie-breakers to choose
the most recent tag.
https://github.com/git/git/blob/master/Documentation/RelNotes/1.7.1.1.txt:
* "git describe" did not tie-break tags that point at the same commit
correctly; newer ones are preferred by paying attention to the
tagger date now.
projects / jgit.git / log