Matthias Sohn [Thu, 20 Apr 2023 22:52:18 +0000 (00:52 +0200)]
Merge branch 'stable-6.5'
* stable-6.5:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 22:40:02 +0000 (00:40 +0200)]
Merge branch 'stable-6.4' into stable-6.5
* stable-6.4:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 22:33:26 +0000 (00:33 +0200)]
Merge branch 'stable-6.3' into stable-6.4
* stable-6.3:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 22:25:51 +0000 (00:25 +0200)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 22:19:38 +0000 (00:19 +0200)]
Merge branch 'stable-6.1' into stable-6.2
* stable-6.1:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 22:11:40 +0000 (00:11 +0200)]
Merge branch 'stable-6.0' into stable-6.1
* stable-6.0:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 14:01:33 +0000 (16:01 +0200)]
Merge branch 'stable-5.13' into stable-6.0
* stable-5.13:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 13:40:36 +0000 (15:40 +0200)]
Merge branch 'stable-5.12' into stable-5.13
* stable-5.12:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 13:12:01 +0000 (15:12 +0200)]
Merge branch 'stable-5.11' into stable-5.12
* stable-5.11:
Add missing since tag for SshBasicTestBase
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on PackMismatchException
Remove blank in maven.config
Matthias Sohn [Thu, 20 Apr 2023 12:42:56 +0000 (14:42 +0200)]
Merge branch 'stable-5.10' into stable-5.11
* stable-5.10:
Add missing since tag for SshTestHarness#publicKey2
Silence API errors
Prevent infinite loop rescanning the pack list on
PackMismatchException
Remove blank in maven.config
Migrated "Prevent infinite loop rescanning the pack list on
PackMismatchException" to refactoring done in
https://git.eclipse.org/r/q/topic:restore-preserved-packs
Matthias Sohn [Thu, 30 Mar 2023 11:43:17 +0000 (13:43 +0200)]
Prevent infinite loop rescanning the pack list on PackMismatchException
We found, when analysing an incident where Gerrit's gc runner thread got
stuck, that we can end up in an infinite loop in
ObjectDirectory#openPackedObject which tries to rescan the pack
list and starts over trying to open a packed object in an unconfined
loop if it catches a PackMismatchException.
Here the relevant part of a thread dump we created while the gc runner
was stuck:
"WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@350812a3[Not
completed,
task = java.util.concurrent.Executors$RunnableAdapter@5425d7ee]]" #72
tid=0x00007f73cee1c800 nid=0x584
runnable [0x00007f7392d57000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.jgit.internal.storage.file.WindowCache.removeAll(WindowCache.java:716)
at org.eclipse.jgit.internal.storage.file.WindowCache.purge(WindowCache.java:399)
at org.eclipse.jgit.internal.storage.file.PackFile.close(PackFile.java:296)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.reuseMap(ObjectDirectory.java:973)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacksImpl(ObjectDirectory.java:904)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacks(ObjectDirectory.java:895)
- locked <0x000000050a498f60> (a
java.util.concurrent.atomic.AtomicReference)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.searchPacksAgain(ObjectDirectory.java:794)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:465)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:417)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:408)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader$1.open(ObjectReader.java:279)
at org.eclipse.jgit.revwalk.RevWalk$2.next(RevWalk.java:1031)
at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1911)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:960)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:876)
at org.eclipse.jgit.internal.storage.file.GC.writePack(GC.java:1168)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:852)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:269)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:220)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:112)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:75)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:76)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.18/Executors.java:515)
at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.18/FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.18/ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
The code in ObjectDirectory#openPackedObject [1] apparently assumes that
this is caused by a transient problem which it can resume from by
retrying. We use `core.trustFolderStat = false` on this server since it
uses NFS. The incident we had showed that we can enter into an infinite
loop here if there is a permanent mismatch between a pack file and its
corresponding pack index. I am not yet sure how this can happen.
Break the infinite loop by limiting the number of attempts rescanning
the pack list to 5 retries. When we exceed this threshold set the type
of the PackMismatchException to permanent and rethrow it which breaks
the infinite loop.
Also apply the same limit in #getPackedObjectSize
and #selectObjectRepresentation where we use similar retry loops.
Pat Patterson [Thu, 6 Apr 2023 15:05:56 +0000 (08:05 -0700)]
Add protocol configuration to Amazon S3 transport
Before this change, attempting to use the jgit Amazon S3 transport with an S3-compatible service that requires https (for example, Backblaze B2) results in an error:
$ jgit push b2
fatal: amazon-s3://metadaddy-jgit/repos/test/objects: error in packed-refs
This change adds a "protocol" property to the Amazon S3 transport configuration, defaulting to http, and uses that value when constructing the URL for the S3 service.
kylezhao [Tue, 9 Nov 2021 12:03:27 +0000 (20:03 +0800)]
RevWalk: use generation number to optimize getMergedInto()
A commit A can reach a commit B only if the generation number of A is
strictly larger than the generation number of B. This condition allows
significantly short-circuiting commit-graph walks.
On a copy of the Linux repository where HEAD is contained in v6.3-rc4
but no earlier tag, the command 'git tag --contains HEAD' of
ListTagCommand#call() had the following peformance improvement:
(excluded the startup time of the repo)
Ivan Frade [Tue, 11 Apr 2023 19:50:52 +0000 (12:50 -0700)]
DfsPackFile: Extract block aligment code
Loading of pack, bitmap and commit-graph copy the same code to adjust
the input stream buffering.
Extract to a common function. Besides reusing the code, the name hints
what it is doing.
This block aligment seems unnecessary as the reading is from storage
not dfs cache. The channel probably knows better. Left a TODO because
I don't know the original intention.
Matthias Sohn [Thu, 6 Apr 2023 20:16:41 +0000 (22:16 +0200)]
Merge branch 'stable-6.5'
* stable-6.5:
Ensure parsed RevCommitCG has derived data from commit-graph
Downgrade maven-site-plugin to 3.12.1
Use wagon-ssh-external to deploy Maven site
Ensure parsed RevCommitCG has derived data from commit-graph
If a RevCommitCG was newly created and called #parseCanonical(RevWalk,
byte[]) method immediately, its flag was marked as PARSED, but no
derived data was obtained from the commit-graph. This is different from
what we expected.
Jonathan Tan [Tue, 4 Apr 2023 23:49:30 +0000 (16:49 -0700)]
PatchApplierTest: specify charset to avoid warning
Running
bazel test //org.eclipse.jgit.test:org_eclipse_jgit_patch_PatchApplierTest
results in a DefaultCharset error on my machine, which can be avoided
by explicitly specifying a charset when calling getBytes on a string. In
these tests, the charset used doesn't really matter, so go with UTF_8.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Id7721cc5b7ea650e77c2db47042715487983cae6
Jonathan Tan [Wed, 5 Apr 2023 20:44:59 +0000 (13:44 -0700)]
GcConcurrentTest: @Ignore flaky testInterruptGc
During my development of Id7721cc5b7ea650e77c2db47042715487983cae6, I
have found this test to be flaky when run by CI. As a speculative fix,
mark this test as @Ignore so it won't be run.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Change-Id: Idfe04d7f1fb72a772d4c8d249ca86a9c2eec0b1a
Reason for revert: This change was based on the false claim that the
packedrefs file lock is held while the CAS is being done, but it is
actually released before the CAS (the in memory lock is still held,
however that does not prevent external actors from updating the
packedrefs files and then another thread from subsequently re-reading it
and updating the in memory packedRefList). Although reverting this
change can cause the CAS to fail, it should not actually matter since
the failure would indicate that another thread has already updated the
in memory packedRefList to either the same version this thread was
trying to update it too, or to a more recent version. Either way,
failing the CAS is then appropriate and should not be problematic.
Although this change reverts the code in the RefDirectory class, it
keeps the "improvements" to the test so that it continues to pass
reliably. The reason for the quotes around the word "improvements" is
because I believe the test alteration actually dramatically changes the
intent of the test, and that the original intent of the test is
untestable with the GC and RefDirectory classes as is.
Change-Id: I3acee7527bb542996dcdfaddfb2bdb45ec444db5 Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
RefDirectory.delete: Prevent failures when packed-refs is outdated
The in-memory copy of packed refs might be outdated by the time the
packed-refs lock is acquired, so ensure the one read from disk is
used after acquiring the lock to prevent commit packed-refs from
throwing an exception. As a side-effect, since this updates the
in-memory copy of packed-refs when it is re-read from disk, it can
prevent other callers needing to re-read if it had changed.
RefDirectory.pack: Only rely on packed refs from disk
Since packed-refs is read from disk anyway, don't rely on the
in-memory copy as that is racy and if outdated, could result in
commit of pack-refs throwing an exception. This change also avoids
a possible unnecessary double read of packed-refs from disk.
RefDirectory: Make pack() and commitPackRefs() void
There are no more callers (since Iae71cb3) of these methods that need
the returned value. These methods should not have been returning
anything in the first place as that can introduce bugs such as the
one described in Iae71cb3.
Implement a snapshotting RefDirectory for use in request scope
Introduce a SnapshottingRefDirectory class which allows users to get
a snapshot of the ref database and use it in a request scope (for
example a Gerrit query) instead of having to re-read packed-refs
several times in a request.
This can potentially be further improved to avoid scanning/reading a
loose ref several times in a request. This would especially help
repeated lookups of a packed ref, where we check for the existence of
a loose ref each time.
Since the first attempt to read a ref is not expected to trigger
a RefsChangedEvent, update the test to ensure 'lastNotifiedModCnt'
is not 0 before we start the actual work. The test has been passing
luckily because createBareRepository in setUp() happens to bump
'lastNotifiedModCnt'.
PackedBatchRefUpdate: Ensure updates are applied on latest packed refs
In the window between refs being packed (via refDb.pack) and obtaining
updates (via applyUpdates), packed-refs may have been updated by another
actor and relying on the previously read contents may lead to losing the
updates done by the other actor. To help avoid this, read packed-refs
from disk to ensure we have the latest copy after it is locked and
before committing updates to it.
Ronald Bhuleskar [Wed, 22 Mar 2023 22:07:19 +0000 (15:07 -0700)]
BasePackFetchConnection: support negotiationTip feature
By default, Git will report, to the server, commits reachable from all local refs to find common commits in an attempt to reduce the size of the to-be-received packfile. If specified with negotiation tip, Git will only report commits reachable from the given tips. This is useful to speed up fetches when the user knows which local ref is likely to have commits in common with the upstream ref being fetched.
When negotation-tip is on, use the wanted refs instead of all refs as source of the "have" list to send.
This is controlled by the `fetch.usenegotationtip` flag, false by default. This works only for programmatic fetches and there is no support for it yet in the CLI.
Matthias Sohn [Mon, 27 Mar 2023 20:23:11 +0000 (22:23 +0200)]
DirCache: support option index.skipHash
Support the new option index.skipHash which was introduced in git 2.40
[1]. If it is set to true skip computing the git index checksum. This
accelerates Git commands that manipulate the index, such as git add, git
commit, or git status. Instead of storing the checksum, write a trailing
set of bytes with value zero, indicating that the computation was
skipped.
Accept a skipped checksum consisting of 20 null bytes when reading the
index since the option could have been set to true at the time when the
index was written.
1. For general errors, throw IOException instead of wrapping them with
PatchApplyException. The wrapping was moved (back) to ApplyCommand.
2. For file specific errors, log the errors as part of
PatchApplier::Result.
3. Change applyPatch() to receive the parsed Patch object, so the caller
can decide how to handle parsing errors.
Background: this utility class was extracted from ApplyCommand on V6.4.0.
During the extraction, we left the exception wrapping by
PatchApplyException intact. This attitude made it harder for the callers to
distinguish between the actual error causes.
kylezhao [Mon, 27 Mar 2023 06:48:31 +0000 (14:48 +0800)]
Ensure FileCommitGraph scans commit-graph file if it already exists
When commit-graph file already exists in the repository, a newly
created FileCommitGraph didn't scan CommitGraph until the file was
modified, resulting in wrong result.
Xing Huang [Tue, 21 Mar 2023 22:27:49 +0000 (17:27 -0500)]
GC: Close File.lines stream
From File#lines javadoc: The returned stream from File Lines
encapsulates a Reader. If timely disposal of file system resources is
required, the try-with-resources construct should be used to ensure
that the stream's close method is
invoked after the stream operations are completed.
Xing Huang [Tue, 21 Mar 2023 22:27:49 +0000 (17:27 -0500)]
GC: Close File.lines stream
From File#lines javadoc: The returned stream from File Lines
encapsulates a Reader. If timely disposal of file system resources is
required, the try-with-resources construct should be used to ensure
that the stream's close method is
invoked after the stream operations are completed.
Matthias Sohn [Fri, 3 Mar 2023 15:04:00 +0000 (16:04 +0100)]
Merge branch 'stable-6.5'
* stable-6.5:
[errorprone] Suppress [Finally] warnings
Update Orbit to R20230302014618 for 2023-03
Improve test coverage when core.trustPackedRefsStat set to after_open
Prepare 6.5.0-SNAPSHOT builds
JGit v6.5.0.202302281825-rc1
Prepare 6.5.0-SNAPSHOT builds
JGit v6.5.0.202302221508-m3
Matthias Sohn [Thu, 2 Mar 2023 09:43:10 +0000 (10:43 +0100)]
[errorprone] Suppress [Finally] warnings
In these cases we use Throwable#addSuppressed to ensure the exception
thrown in the catch block preceding the finally block throwing another
exception isn't lost.
Improve test coverage when core.trustPackedRefsStat set to after_open
As of today, we don't have test coverage for RefDirectory when
core.trustPackedRefsStat config is set to after_open. Thus create new
test classes which set core.trustPackedRefsStat config to after_open in
setup and extend RefDirectoryTest and FileRepositoryBuilderTest
respectively.
Matthias Sohn [Tue, 28 Feb 2023 23:11:41 +0000 (00:11 +0100)]
Merge branch 'master' into stable-6.5
* master:
Change config pull.rebase=preserve to pull.rebase=merges
BatchingProgressMonitor: expose time spent per task
PackWriter: offer to write an object-size index for the pack
Fix formatting in GC#doGc
PackExt: Define new extension for the object size index
Matthias Sohn [Thu, 19 Jan 2023 00:46:50 +0000 (01:46 +0100)]
BatchingProgressMonitor: expose time spent per task
Display elapsed time per task if enabled via
ProgressMonitor#showDuration or if system property or environment
variable GIT_TRACE_PERFORMANCE is set to "true". If both the system
property and the environment variable are set the system property takes
precedence.
Ivan Frade [Tue, 28 Dec 2021 22:23:40 +0000 (14:23 -0800)]
PackWriter: offer to write an object-size index for the pack
PackWriter callers tell the writer what do the want to include in the
pack and invoke #writePack(). Afterwards, they can invoke #writeIndex()
to write the corresponding pack index.
Mirror this for the object-size index, adding a #writeObjectSizeIndex()
method.
Matthias Sohn [Wed, 22 Feb 2023 20:07:34 +0000 (21:07 +0100)]
Merge branch 'master' into stable-6.5
* master:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Matthias Sohn [Wed, 22 Feb 2023 20:06:41 +0000 (21:06 +0100)]
Merge branch 'stable-6.4'
* stable-6.4:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Matthias Sohn [Wed, 22 Feb 2023 20:04:31 +0000 (21:04 +0100)]
Merge branch 'stable-6.3' into stable-6.4
* stable-6.3:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Matthias Sohn [Wed, 22 Feb 2023 20:03:52 +0000 (21:03 +0100)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Matthias Sohn [Wed, 22 Feb 2023 20:03:22 +0000 (21:03 +0100)]
Merge branch 'stable-6.1' into stable-6.2
* stable-6.1:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Matthias Sohn [Wed, 22 Feb 2023 20:02:47 +0000 (21:02 +0100)]
Merge branch 'stable-6.0' into stable-6.1
* stable-6.0:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
Matthias Sohn [Wed, 22 Feb 2023 20:02:09 +0000 (21:02 +0100)]
Merge branch 'stable-5.13' into stable-6.0
* stable-5.13:
If tryLock fails to get the lock another gc has it
Fix GcConcurrentTest#testInterruptGc
Don't swallow IOException in GC.PidLock#lock
Check if FileLock is valid before using or releasing it
projects / jgit.git / log