Matthias Sohn [Wed, 25 Nov 2020 16:00:09 +0000 (17:00 +0100)]
Ensure that GC#deleteOrphans respects pack lock
If pack or index files are guarded by a pack lock (.keep file)
deleteOrphans() should not touch the respective files protected by the
lock file. Otherwise it may interfere with PackInserter concurrently
inserting a new pack file and its index.
The problem was caused by the following race.
All mentioned files are located in "objects/pack/".
File endings relevant in "pack" dir:
.pack
.keep
.idx
.bitmap
When ReceivePack receives a pack file it executes the following steps:
ReceivePack.service():
receivePackAndCheckConnectivity():
receivePack():
receive the pack
parse the pack, returns packLock (.keep file)
PackInserter.flush():
write tmpPck file: "insert_<random>.pack"
write tmpIdx file: "insert_<random>.idx"
real pack name: "pack-<SHA1>.pack"
real index name: "pack-<SHA1>.idx"
atomic rename tmpPack to realPack
atomic rename tmpIdx to tmpIdx
execute commands
unlock pack by removing .keep file
trigger auto gc if enabled
When PackInserter.flush() renames the temporary pack to the final
"pack-xxx.pack" file the temporary pack index file "insert_xxx.idx"
has no matching .pack file with the same base name for a short interval.
If deleteOrphans() ran during that interval it deduced the pack index
file was orphaned. Subsequently the missing pack index caused
MissingObjectExceptions since objects contained in the pack couldn't be
looked up anymore.
Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=13544
Change-Id: I559c81e4b1d7c487f92a751bd78b987d32c98719 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 1 Feb 2020 00:14:01 +0000 (01:14 +0100)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
David Pursehouse [Fri, 31 Jan 2020 00:12:08 +0000 (09:12 +0900)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Fix string format parameter for invalidRefAdvertisementLine
WindowCache: add metric for cached bytes per repository
pgm daemon: fallback to user and system config if no config specified
WindowCache: add option to use strong refs to reference ByteWindows
Change-Id: I741059a1d0d5950ab5bc16ec70352655ee926a24 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Thu, 30 Jan 2020 23:38:24 +0000 (08:38 +0900)]
Fix string format parameter for invalidRefAdvertisementLine
The externalized error message added in f4fc640 ("BasePackConnection:
Check for expected length of ref advertisement", Dec 18, 2019) uses a
malformed string format. Since there is only one formatting argument,
it should be referenced with '{0}' rather than '{1}'.
Change-Id: Ibda864dfb0bb902fe07ae4bba73117b212046e8a Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Matthias Sohn [Fri, 3 Jan 2020 17:16:42 +0000 (18:16 +0100)]
WindowCache: add metric for cached bytes per repository
Since ObjectDatabase and PackFile don't know their repository use the
packfile's grand-grand-parent directory as an identifier for the
repository the packfile resides in.
Remove metric for a repository if the number of cached bytes for the
repository drops to 0 in order to ensure the map of cached bytes per
repository doesn't contain repositories which have no data cached in the
WindowCache.
Change-Id: I969ab8029db0a292e7585cbb36ca0baa797da20b Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 27 Jan 2020 22:00:57 +0000 (23:00 +0100)]
pgm daemon: fallback to user and system config if no config specified
If a config file is passed via option --config-file then use only the
options defined in that file. This helps to concisely configure the
daemon without side effects from global and system level git configs.
Otherwise fallback to user and system level configs.
Change-Id: I242de248f257579874ad0bfe4882a22502353b1f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 13 Dec 2019 00:18:12 +0000 (01:18 +0100)]
WindowCache: add option to use strong refs to reference ByteWindows
Java GC evicts all SoftReferences when the used heap size comes close to
the maximum heap size. This means peaks in heap memory consumption can
flush the complete WindowCache which was observed to have negative
impact on performance of upload-pack in Gerrit.
Hence add a boolean option core.packedGitUseStrongRefs to allow using
strong references to reference packfile pages cached in the WindowCache.
If this option is set to true Java gc can no longer flush the
WindowCache to free memory if the used heap comes close to the maximum
heap size. On the other hand this provides more predictable performance.
Bug: 553573
Change-Id: I9de406293087ab0fa61130c8e0829775762ece8d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Wed, 15 Jan 2020 11:03:28 +0000 (20:03 +0900)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Add config constants for WindowCache configuration options
Change-Id: I17da3c5183eca536aa2be3972bc5df45c9d75f1b Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Patrick Hiesel [Thu, 9 Jan 2020 09:23:12 +0000 (10:23 +0100)]
Replace usage of ArrayIndexOutOfBoundsException in treewalk
Using exceptions during normal operations - for example with the
desire of expanding an array in the failure case - can have a
severe performance impact. When exceptions are instantiated,
a stack trace is collected. Generating stack trace can be expensive.
Compared to that, checking an array for length - even if done many
times - is cheap since this is a check that can run in just a
handful of CPU cycles.
Change-Id: Ifaf10623f6a876c9faecfa44654c9296315adfcb Signed-off-by: Patrick Hiesel <hiesel@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Thu, 9 Jan 2020 01:19:03 +0000 (02:19 +0100)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Fix API problem filters
Fix unclosed resource warning in SmartOutputStream
JschConfigSessionFactory: fix boxing warning
SshSupport#runSshCommand: don't throw exception in finally block
Don't override already managed maven-compiler-plugin version
Remove unused import from CreateFileSnapshotBenchmark
Remove duplicate ignore_optional_problems entry in .classpath
Update maven-site-plugin used by benchmark module to 3.8.2
Add dependency to enable site generation for benchmark module
Ignore warnings for generated source code in
org.eclipse.jgit.benchmark
Fix MBean registration
Enhance WindowCache statistics
Change-Id: I67a07d92718188bdf7f8a13b83e9f538ecf4b22f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 4 Jan 2020 14:44:19 +0000 (15:44 +0100)]
SshSupport#runSshCommand: don't throw exception in finally block
The CommandFailedException which was thrown in finally block is silently
discarded [1]. Refactor this method to throw the exception after the
finally block.
This fixes the warning "Null comparison always yields false: The
variable failure can only be null at this location".
Matthias Sohn [Mon, 16 Dec 2019 14:22:08 +0000 (15:22 +0100)]
Ignore warnings for generated source code in org.eclipse.jgit.benchmark
The source code in the folder .apt_generated is generated by the JMH
code generator, so there's no point in raising any warnings as this
could only be fixed in the upstream code generator.
Change-Id: I882888e7bf924f9ae74182598fcb91671a5c9818 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 30 Nov 2019 02:38:13 +0000 (03:38 +0100)]
Enhance WindowCache statistics
Add the following statistics
- cache hit count and hit ratio
- cache miss count and miss ratio
- count of successful and failed loads
- rate of failed loads
- load, eviction and request count
- average and total load time
Use LongAdder instead of AtomicLong to implement counters in order to
improve scalability.
Optionally expose these metrics via JMX, they are registered with the
platform MBean server if the config option jmx.WindowCacheStats = true
in the user or system level git config.
Bug: 553573
Change-Id: Ia2d5246ef69b9c2bd594a23934424bc5800774aa Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 14:24:54 +0000 (15:24 +0100)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Update API problem filters
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: I00615958ab6fbdff601e87a9792aba5606cda12a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 14:07:39 +0000 (15:07 +0100)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: If91c55a192d3b2c441d9c8d414f2e24a7261b1b6 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 3 Aug 2019 10:02:34 +0000 (12:02 +0200)]
JMH benchmark for SimpleLruCache
See [1] for JMH documentation and [2] how to use JMH in Eclipse.
The benchmarks pom currently cannot use the JGit parent pom due to an
ecj bug [3] regarding annotation processing. Hence for now do not
inherit from the JGit parent pom and copy the compiler plugin
configuration for javac from the parent pom.
After running the Maven build the benchmark can be run using Maven:
Matthias Sohn [Wed, 2 Oct 2019 08:56:15 +0000 (10:56 +0200)]
Merge branch 'stable-5.1' into stable-5.2
# By Matthias Sohn (2) and Saša Živkov (1)
* stable-5.1:
Prepare 5.1.13-SNAPSHOT builds
JGit v5.1.12.201910011832-r
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
Change-Id: Iaae4b171eaa0081f9142489de8df94ab455d65f7 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Do not rely on ArrayIndexOutOfBoundsException to detect end of input
In the Config#StringReader we relied on ArrayIndexOutOfBoundsException
to detect the end of the input. Creation of exception with (deep) stack
trace can significantly degrade performance in case when we read
thousands of config files, like in the case when Gerrit reads all
external ids from the NoteDb.
Use the buf.length to detect the end of the input.
Thomas Wolf [Thu, 12 Sep 2019 19:05:19 +0000 (21:05 +0200)]
WorkingTreeIterator: handle different timestamp resolutions
Older JGit stored only milliseconds timestamps in the index. Newer
JGit may get finer timestamps from the file system. This leads to
slow index diffs when a new JGit runs against an index produced
by older JGit because many timestamps will differ and JGit will
then do many content checks. See [1].
Handle this migration case by only comparing milliseconds if the
index entry has only millisecond precision.
The inverse may also occur; also compare only milliseconds if the
file timestamp has only millisecond precision.
Do the same also for microsecond resolution. On Windows, NTFS may
provide 100ns resolution and may be used by external programs writing
the index, but Java's WindowsFileAttributes may provide only
microseconds.
File timestamp precision in Java depends not only on the Java APIs
used by different JGit versions but may also change when running the
same Java code on different VMs. And of course the resolution may
vary among operating and file systems. Moreover, timestamp precision
in the index depends on the program that wrote the index. Canonical
git may use a different resolution, maybe even different between git
versions.
Change-Id: Idfd08606c883cb98787b2138f9baf0cc89a57b56 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 10 Sep 2019 14:08:45 +0000 (16:08 +0200)]
Fix WorkingTreeIterator.compareMetadata() for CheckStat.MINIMAL
If CheckStat is MINIMAL or timestamps have no nanosecond part
WorkingTreeIterator.compareMetaData only checks the second part of
timestamps and ignores nanoseconds which may have ended up in the index
by using native git.
we currently proceed comparing fileLastModified and cacheLastModified
with full precision which is wrong since we determined that we detected
reduced timestamp resolution.
Fix this and also handle smudged index entries for CheckStat.MINIMAL.
Change-Id: I6149885903ac63d79b42d234cc02aa4e19578f3c Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sun, 8 Sep 2019 09:17:32 +0000 (11:17 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
[error prone] Suppress NonAtomicVolatileUpdate in SimpleLruCache
Bazel: Format BUILD files with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: I0d8b7f79177a20dc00c89e2cf0005eb3d3039532 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 7 Sep 2019 09:07:21 +0000 (11:07 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Bazel: Fix warning about deprecated lib.bzl
Format lib/BUILD with buildifier
Bazel: Add fixes for --incompatible_load_java_rules_from_bzl
Bazel: Bump minimum supported version to 0.29.0
Bazel: Bump skylib library version to 0.8.0
Use bazelisk to switch between used bazel version
Bazel: Require minimum bazel version 0.17.1
Fix wrong placeholder index in error message packInaccessible
JGitText: Remove unused externalized strings
RepoText: Remove unused externalized string
CLI: Remove unused externalized strings
Change-Id: Iea37a8e39e9d4872dc607c9222dcf191ce4e4757 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Wed, 10 Oct 2018 01:53:35 +0000 (10:53 +0900)]
Bazel: Require minimum bazel version 0.17.1
Check the bazel version using the checker from bazel_skylib, and
require at least version 0.17.1 which is the minimum version that
does not suffer from the Java API mismatch issue [1].
The implementation is borrowed from the Gerrit project.
Matthias Sohn [Tue, 3 Sep 2019 22:32:42 +0000 (00:32 +0200)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Prepare 5.1.12-SNAPSHOT builds
JGit v5.1.11.201909031202-r
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: I7e88c7c7d202f1e3fb8e143277650aa5fefff439 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 3 Sep 2019 21:41:03 +0000 (23:41 +0200)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Prepare 5.1.12-SNAPSHOT builds
JGit v5.1.11.201909031202-r
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Iac5dc9683cea97db04d20f27c10f2e103d3ae7b5 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 3 Sep 2019 14:02:44 +0000 (16:02 +0200)]
Merge branch 'stable-5.0' into stable-5.1
* stable-5.0:
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: I172136a031ff0730e575327cafb3527c9650a71d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 3 Sep 2019 13:54:54 +0000 (15:54 +0200)]
Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
Prepare 4.11.10-SNAPSHOT builds
JGit v4.11.9.201909030838-r
Bazel: Update bazlets to the latest master revision
Bazel: Remove FileTreeIteratorWithTimeControl from BUILD file
BatchRefUpdate: repro racy atomic update, and fix it
Delete unused FileTreeIteratorWithTimeControl
Fix RacyGitTests#testRacyGitDetection
Change RacyGitTests to create a racy git situation in a stable way
Silence API warnings
Change-Id: Ifb6a4dbea2f48fd2ffa66eb737d61920aefedfbd Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log