Dariusz Luksza [Mon, 20 Nov 2023 11:53:19 +0000 (11:53 +0000)]
Improve handling of NFS stale handle errors
Mark packfile as invalid when NFS stale handle error occurs.
This should fix broken fetch operations when the repo is located on the
NFS system and is GC'ed on a separate system (or process). Which may
result in the index, pack or bitmap file being removed when they are
accessed from the fetch operation.
Dariusz Luksza [Mon, 20 Nov 2023 11:00:51 +0000 (11:00 +0000)]
Fix handling of missing pack index file
As demonstrated in
`UploadPackHandleDeletedPackFile.testV2IdxFileRemovedDuringUploadPack`
the fetch operation will fail when the pack index file is removed.
This is due to a wrapping of `FileNotFoundException` (which is a
subclass of `IOExeption`) in an `IOException` at PackIndex L#68. This
is then changing the behaviour of error handling in
`Pack.file.getBitmapIndex()` where the `FileNotFoundException` is
swallowed and allows the fetch process to continue. With FNFE being
wrapped in IOE, this blows up and breaks the fetch operation.
Simply rethrowing `FileNotFoundException` from `PackFile.open()` fixes
the broken fetch operation. This will also mark the whole pack as
invalid in the `IOException` handler in `Pack.idx()` method.
Dariusz Luksza [Fri, 17 Nov 2023 19:28:53 +0000 (19:28 +0000)]
Add tests for handling pack files removal during fetch
Although this could sound like a corner case, it really can occur out
there in the real world. Especially in the Gerrit world where the
repositories could be GC'ed on a separate process or system.
The `FileNotFoundException` seems to be handled correctly in
`PackFile#doOpen` (line 671) and it will mark the pack as invalid. But
triggering that code path was not an easy task.
First of all, we need to add a new commit to the `master` branch of the
test repository after `UploadPack` object is created.
Secondly, in the refspec for fetch, commit id instead of "regular"
refspec must be used.
With both in place, we can see a warning log statement about deleted
pack file. And the fetch succeeds!
Also, tests for the removal of *.idx and *.bitmap files were added.
This unveiled a corner for the *.idx file deletion while fetching, as
the test will fail with "Unreachable pack index" IOException only
when the HEAD commit is empty.
Luca Milanesio [Mon, 13 Jun 2022 22:09:55 +0000 (23:09 +0100)]
Introduce a PriorityQueue sorting RevCommits by commit timestamp
The DateRevQueue uses a tailor-made algorithm to keep
RevCommits sorted by reversed commit timestamp, which has a O(n*n/2)
complexity and caused the explosion of the Git fetch times to
tens of seconds.
The standard Java PriorityQueue provides a O(n*log(n)) complexity
and scales much better with the increase of the number of
RevCommits.
Introduce a new implementation DateRevPriorityQueue of the DateRevQueue
based on PriorityQueue.
Enable usage of the new DateRevPriorityQueue implementation by setting
the system property REVWALK_USE_PRIORITY_QUEUE=true. By default the old
implementation DateRevQueue is used.
Revert commit 170244d05977491271a1cc234583d2e5ba75145d
"Checkout: better directory handling" which is the downport of the
original fix Ie12864c54c9f901a2ccee7caddec73027f353111 which was done
on stable-6.6. Merging this up to stable-6.6 would be a lot of work and
these branches aren't maintained anymore hence revert this change here.
This way the fix is available on stable-5.13 for those who still need
Java 8 and everybody else should upgrade to 6.6.1 or higher.
Fabio Ponciroli [Wed, 6 Dec 2023 13:38:21 +0000 (14:38 +0100)]
Make sure ref to prune is in packed refs
RefDirectory:pack might raise an NPE when deleting loose
refs as final part of the RefDirectory.pack().
This is what the code does:
1) packed ref update: update the list of refs which will be
persisted in packed-refs
2) persit packed-refs: flush on file the refs computed in #1
3) prune loose refs: prune loose refs that have been packed in #2
The code correctly locks the packed-refs file during phases 1 to 3.
However, it makes the wrong assumption of considering
the loose refs set as immutable between phases 1 and 3.
The number and values of loose refs on the filesystem can mutate
at any time whilst the RefDirectory.pack() is in progress.
Assuming the contrary can lead to an NPE when retrieving refs
from the mutable loose refs list during phase #3.
Make sure that the ref is not null before dereferencing its
object-id value.
When checking out a file into the working tree ensure that all parent
directories of the file below the working tree root are actually
directories and do exist before we try to create the file.
When multiple files are to be checked out (or even a whole tree), this
may check the same directories over and over again. Asking the file
system every time for file attributes is a potentially expensive
operation. As a remedy, introduce an in-memory cache of directory
states for a particular check-out operation.
Apply the same fix also in the ResolveMerger, which may also check out
files, and also in the PatchApplier. In PatchApplier, also validate
paths.
Change-Id: Ie12864c54c9f901a2ccee7caddec73027f353111 Signed-off-by: Thomas Wolf <twolf@apache.org> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 13 Oct 2023 06:46:11 +0000 (08:46 +0200)]
Merge branch 'stable-6.5' into stable-6.6
* stable-6.5:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Matthias Sohn [Thu, 12 Oct 2023 23:52:43 +0000 (01:52 +0200)]
Merge branch 'stable-6.4' into stable-6.5
* stable-6.4:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Matthias Sohn [Thu, 12 Oct 2023 23:33:56 +0000 (01:33 +0200)]
Merge branch 'stable-6.3' into stable-6.4
* stable-6.3:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Matthias Sohn [Thu, 12 Oct 2023 22:51:46 +0000 (00:51 +0200)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Matthias Sohn [Thu, 12 Oct 2023 22:44:33 +0000 (00:44 +0200)]
Merge branch 'stable-6.1' into stable-6.2
* stable-6.1:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Matthias Sohn [Thu, 12 Oct 2023 22:33:19 +0000 (00:33 +0200)]
Merge branch 'stable-6.0' into stable-6.1
* stable-6.0:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Matthias Sohn [Thu, 12 Oct 2023 22:22:11 +0000 (00:22 +0200)]
Merge branch 'stable-5.13' into stable-6.0
* stable-5.13:
PackConfig: fix @since tags
Remove unused API problem filters
Add support for git config repack.packKeptObjects
Do not exclude objects in locked packs from bitmap processing
Antonio Barone [Thu, 5 Oct 2023 19:58:13 +0000 (21:58 +0200)]
Add support for git config repack.packKeptObjects
Change Ide3445e652 introduced the `--pack-kept-objects` option to GC for
including the objects contained in the locked packfiles during the
repack phase.
Whilst this allowed to explicitly pass a command line argument to the
jgit gc program, it did not allow the option to be read from
configuration.
Allow the pack kept objects option to be configured exactly as C-Git
documents [1], by introducing a new `repack.packKeptObjects`
configuration.
`repack.packKeptObjects` defaults to `true`, when the
`pack.buildBitmaps` is `true` (which is the default case), `false`
otherwise.
Luca Milanesio [Fri, 11 Aug 2023 19:15:17 +0000 (20:15 +0100)]
Do not exclude objects in locked packs from bitmap processing
Packfiles having an equivalent .keep file are associated with in-flight
pushes that haven't been completed, with potentially a set of git
objects not yet referenced by a ref.
If the Git client is not up-to-date, it may result in pushing a
packfile, generating a <packfile>.keep on the server, which
may also contain existing commits due to the lack of Git protocol
negotiation in the git-receive-pack.
The Git protocol negotiation is the phase where the client and the
server exchange the list of refs they have for trying to find a common
base and minimise the amount of objects to be transferred.
The repack phase in GC was previously skipping all objects that were
contained in all packfiles having a <packfile>.keep file associated
(aka "locked packfiles"), which did not take into consideration the
fact that excluding the existing commits would have resulted in the
generation of an invalid bitmap file.
The code for excluding the objects in the locked packfiles was written
well before the bitmap was introduced, hence could not consider a use
case that did not exist at that time.
However, when the bitmap was introduced, the exclusion of locked
packfiles was not changed, hence creating a potential problem.
The issue went unnoticed for many years because the bitmap generation
was disabled when JGit noticed any locked packfiles; however, the
bitmaps are enabled again since Id722e68d9f , and the the issue is now
visible and is impacting the GC repack phase.
Introduce the '--pack-kept-objects' option in GC for including the
objects contained in the locked packfiles during the repack phase,
which is not an issue because of the following:
- If there are any existing commits duplicated in the packfiles
they will be just considered once anyway because the repack doesn't
generate duplicates in the output packfile.
- If there are any new commits that do not have any ref pointing to
them, they will be automatically excluded from the output repacked
packfile.
The same identical solution is adopted in the C implementation of git
in repack.c.
Because the locked packfile is not pruned, any new commits not pointed
by any refs will remain in the repository and there will not be any
accidental pruning or object loss as it is today before this change.
As a side-effect of this change, it is now potentially possible to still
have duplicate BLOBs after GC when the keep packfile contained existing
objects. However, it is way better to keep the duplication until the
next GC phase rather than omitting existing objects from repacking and,
therefore generating an invalid bitmap and incorrect packfile.
Thomas Wolf [Fri, 11 Aug 2023 19:40:13 +0000 (21:40 +0200)]
Checkout: better directory handling
When checking out a file into the working tree ensure that all parent
directories of the file below the working tree root are actually
directories and do exist before we try to create the file.
When multiple files are to be checked out (or even a whole tree), this
may check the same directories over and over again. Asking the file
system every time for file attributes is a potentially expensive
operation. As a remedy, introduce an in-memory cache of directory
states for a particular check-out operation.
Apply the same fix also in the ResolveMerger, which may also check out
files, and also in the PatchApplier. In PatchApplier, also validate
paths.
Change-Id: Ie12864c54c9f901a2ccee7caddec73027f353111 Signed-off-by: Thomas Wolf <twolf@apache.org> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 3 Aug 2023 08:14:45 +0000 (10:14 +0200)]
Merge branch 'stable-6.5' into stable-6.6
* stable-6.5:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Matthias Sohn [Thu, 3 Aug 2023 08:12:57 +0000 (10:12 +0200)]
Update to Tycho 4.0.1
Tycho 4.0.0-SNAPSHOT is no longer available and it's a bad practice to
depend on any snapshot version (we had to since this was the only way
to get gpg signing to work in time for releasing 6.6.0).
Matthias Sohn [Wed, 2 Aug 2023 23:55:12 +0000 (01:55 +0200)]
Merge branch 'stable-6.4' into stable-6.5
* stable-6.4:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Matthias Sohn [Wed, 2 Aug 2023 23:51:36 +0000 (01:51 +0200)]
Merge branch 'stable-6.3' into stable-6.4
* stable-6.3:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Matthias Sohn [Wed, 2 Aug 2023 23:37:43 +0000 (01:37 +0200)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Matthias Sohn [Wed, 2 Aug 2023 23:28:07 +0000 (01:28 +0200)]
Merge branch 'stable-6.1' into stable-6.2
* stable-6.1:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Matthias Sohn [Wed, 2 Aug 2023 23:19:21 +0000 (01:19 +0200)]
Merge branch 'stable-6.0' into stable-6.1
* stable-6.0:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Matthias Sohn [Wed, 2 Aug 2023 23:17:17 +0000 (01:17 +0200)]
Merge branch 'stable-5.13' into stable-6.0
* stable-5.13:
Add verification in GcKeepFilesTest that bitmaps are generated
Express the explicit intention of creating bitmaps in GC
GC: prune all packfiles after the loosen phase
Prepare 5.13.3-SNAPSHOT builds
JGit v5.13.2.202306221912-r
Luca Milanesio [Sat, 10 Jun 2023 00:20:44 +0000 (01:20 +0100)]
Add verification in GcKeepFilesTest that bitmaps are generated
The packfiles with the .keep extensions are meant to prevent
a packfile from being processed or removed during GC.
From the point of view of the GC process then, the associated
packfile should be completely transparent:
- it should not included in the repacked file
- it should not pruned
- its objects should be left untouched, even if unreachable
- the GC process, including the bitmap generation should continue
as usual, as the the packfiles with .keep file did not exist
Add one explicit test for making sure that the management
of .keep file is also transparent to the generation of bitmaps,
which are still generated if a .keep file exists.
Luca Milanesio [Sat, 10 Jun 2023 01:21:07 +0000 (02:21 +0100)]
Express the explicit intention of creating bitmaps in GC
Add an explicit flag to PackWriter for allowing the
GC.repack() phase to explicitly generate bitmaps only for the
heads packfile and not for the others.
Previously the bitmap generation was conditioned to the
presence of object ids exclusion from the PackWriter.
The introduction of the bitmap generation in the PackWriter
done in Icdb0cdd66 has accidentally made the .keep files not
completely transparent, because their presence have disabled
the generation of the bitmap index, even if the generation
of bitmaps is enabled.
This bug has been an accidental consequence of the intention
of the bitmap generator to avoid generating bitmaps for the
non-heads packfile, however the implementation done by Colby
decided to use the excludeInPacks variable (see [1]) which
is unfortunately also used for excluding the packfiles having
an associated .keep file (see [2]).
Luca Milanesio [Sun, 11 Jun 2023 16:24:29 +0000 (17:24 +0100)]
GC: prune all packfiles after the loosen phase
When loosening the objects inside the packfiles to be pruned, make sure
that the packfile list is stable and prune all the files after the
loosening is done.
This prevents a series of exceptions previously thrown when loosening
the packfiles, due to the too early pruning of the packfiles that were
still in the pack list.
Reason for revert: This change was based on the false claim that the
packedrefs file lock is held while the CAS is being done, but it is
actually released before the CAS (the in memory lock is still held,
however that does not prevent external actors from updating the
packedrefs files and then another thread from subsequently re-reading it
and updating the in memory packedRefList). Although reverting this
change can cause the CAS to fail, it should not actually matter since
the failure would indicate that another thread has already updated the
in memory packedRefList to either the same version this thread was
trying to update it too, or to a more recent version. Either way,
failing the CAS is then appropriate and should not be problematic.
Although this change reverts the code in the RefDirectory class, it
keeps the "improvements" to the test so that it continues to pass
reliably. The reason for the quotes around the word "improvements" is
because I believe the test alteration actually dramatically changes the
intent of the test, and that the original intent of the test is
untestable with the GC and RefDirectory classes as is.
Bug: 582044
Change-Id: I3acee7527bb542996dcdfaddfb2bdb45ec444db5 Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
(cherry picked from commit c5617711a1b4d5d0807cc7eed702b78d114d46b3)
David Ostrovsky [Thu, 25 May 2023 06:49:44 +0000 (08:49 +0200)]
Bazel: Fix remote build execution for Java 17
Bazel build and test with remote17 configuration was actually executed
locally, and not remotely.
Rename remote configuration setting for Java 11 from remote to remote11
and derive the common remote configuration settings for remote11 and
remote17 configurations.
Also remove deprecated --host_javabase and --javabase options.
Test Plan:
Verify that the remote build for Java 17 is actually executed remotely:
$ bazel test --config=remote17 --remote_instance_name=$PROJECT_NAME \
org.eclipse.jgit.test/...
Matthias Sohn [Wed, 24 May 2023 13:50:27 +0000 (15:50 +0200)]
Merge branch 'master' into stable-6.6
* master:
GraphObjectIndex: fix search in findGraphPosition
Update to Tycho 4.0.0-SNAPSHOT
PGP sign p2 artefacts
Revert 'Use net.i2p.crypto:eddsa directly from Maven Central'
Update dash license-tool-plugin to 1.0.2
Also add suppressed exception if unchecked exception occurs in finally
Candidate: use "Objects.equals" instead of "=="
Use hamcrest 2.2 directly from Maven Central
Use commons-logging directly from Maven Central
Update jna to 5.13.0
Use bytebuddy directly from Maven Central
Use jna directly from Maven Central
Use net.i2p.crypto:eddsa directly from Maven Central
Use org.tukaani:xz directly from Maven Central
Use args4j directly from Maven Central
Use gson directly from Maven Central
Remove unused $NON-NLS-1$
Remove unused API filters
Switch to Apache MINA sshd 2.10.0
[releng] API filter for PackIndex.DEFAULT_WRITE_REVERSE_INDEX
PackExt: add a #getTmpExtension method
UploadPack: Record negotiation stats on fetchV2 call
RewriteGeneratorTest: Introduce test cases for the RewriteGenerator
PackWriter: write the PackReverseIndex file
Jonathan Tan [Mon, 8 May 2023 20:55:40 +0000 (13:55 -0700)]
GraphObjectIndex: fix search in findGraphPosition
In findGraphPosition, when there is no object whose OID starts with
the first byte of the sought OID, low equals high. This violates an
invariant of the loop, and when the sought OID is lexicographically
greater than every other OID in the repository, causes an
ArrayIndexOutOfBoundsException (because we're trying to read outside the
list of OIDs).
Therefore, check the "low < high" condition at the start of the loop,
not only after the first iteration.
Change-Id: Ic8ac198c151bd161c4996b9e7cb6e6660f151733 Helped-by: Ivan Frade <ifrade@google.com> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Matthias Sohn [Tue, 9 May 2023 07:44:03 +0000 (09:44 +0200)]
Update to Tycho 4.0.0-SNAPSHOT
We need to update to Tycho in order to force PGP signing of the
bouncycastle libraries which isn't supported by earlier Tycho versions.
For that we need to run Maven on Java 17 or higher.
In order to run tests on Java 11 add a `toolchain.xml` file into the
`~/.m2` directory providing the path to Java installations:
Reason: the maven artifact has a broken MANIFEST.MF with a mandatory
dependency to sun.security.x509, which is an internal package in the
JDK and moreover not needed by the bundle except for one test class
that isn't in the bundle at all.
This extra dependency makes the JGit tycho packaging build fail when
Tycho 4 is used.
We must keep using the Orbit re-packaging of this artifact, which does
not have this unnecessary mandatory dependency.
Change-Id: Ica15a5ddcada09686de3055b2b3daf081e3c5ffc Signed-off-by: Thomas Wolf <twolf@apache.org>
Matthias Sohn [Thu, 18 May 2023 10:02:48 +0000 (12:02 +0200)]
Also add suppressed exception if unchecked exception occurs in finally
If a method called in a finally block throws an exception we should add
exceptions caught earlier to the exception we throw in the finally block
not regarding if it's a checked or unchecked exception.
Thomas Wolf [Mon, 3 Apr 2023 17:33:24 +0000 (19:33 +0200)]
Switch to Apache MINA sshd 2.10.0
Bump the version numbers in pom.xml and in MANIFESTs, and in the bazel
WORKSPACE file. Update the target platforms. Remove work-arounds in
org.eclipse.jgit.ssh.apache that are no longer necessary.
The release notes for Apache MINA sshd are at [1].
Anna Papitto [Tue, 2 May 2023 20:45:05 +0000 (13:45 -0700)]
PackWriter: write the PackReverseIndex file
PackWriter offers the ability to write out the pack file and its various
index files, except for the newly introduced file-based reverse index.
Now that PackReverseIndexWriter can write reverse index files,
PackWriter#writeReverseIndex will write one for a pack if the
corresponding config flag PackConfig#writeReverseIndex is on.
Change-Id: Ib75dd2bbfb9ee9366d5aacb46700d8cf8af4823a Signed-off-by: Anna Papitto <annapapitto@google.com>
Matthias Sohn [Wed, 3 May 2023 01:01:03 +0000 (03:01 +0200)]
Update Maven plugins
- com.github.siom79.japicmp:japicmp-maven-plugin to 0.17.2
- com.github.spotbugs:spotbugs-maven-plugin to 4.7.3.4
- maven-clean-plugin to 3.2.0
- maven-compiler-plugin to 3.11.0
- maven-deploy-plugin to 3.1.1
- maven-enforcer-plugin to 3.3.0
- maven-javadoc-plugin to 3.5.0
- maven-project-info-reports-plugin to 3.4.3
- maven-resources-plugin to 3.3.1
- maven-surefire-plugin to 3.0.0
- maven-surefire-report-plugin to 3.0.0
- org.codehaus.mojo:build-helper-maven-plugin to 3.3.0
- org.jacoco:jacoco-maven-plugin to 0.8.10
- org.springframework.boot:spring-boot-maven-plugin to 2.7.11
Nasser Grainawi [Tue, 2 May 2023 22:30:44 +0000 (16:30 -0600)]
Fix inProcessPackedRefsLock not shared with copies of the instance
The in process lock is intended to manage contention on locking the
packed-refs file within a single process without acquiring the file
system lock. Not sharing it across RefDirectory instances of the same
repository undermines that intent and results in more contention at the
file system level.
projects / jgit.git / log