RevWalk: integrate commit-graph with commit parsing
RevWalk#createCommit() will inspect the commit-graph file to find the
specified object's graph position and then return a new RevCommitCG
instance.
RevCommitGC is a RevCommit with an additional "pointer" (the position)
to the commit-graph, so it can load the headers and metadata from there
instead of the pack. This saves IO access in walks where the body is not
needed (i.e. #isRetainBody is false and #parseBody is not invoked).
RevWalk uses automatically the commit-graph if available, no action
needed from callers. The commit-graph is fetched on first access from
the reader (that internally can keep it loaded and reuse it between
walks).
The startup cost of reading the entire commit graph is small. After
testing, reading a commit-graph with 1 million commits takes less than
50ms. If we use RepositoryCache, it will not be initialized util the
commit-graph is rewritten.
kylezhao [Fri, 6 Jan 2023 07:24:14 +0000 (15:24 +0800)]
GC: disable writing commit-graph for shallow repos
In shallow repos, GC writes to the commit-graph that shallow commits
do not have parents. This won't be true after a "git fetch --unshallow"
(and before another GC).
Do not write the commit-graph from shallow clones of a repo. The
commit-graph must have the real metadata of commits and that is not
available in a shallow view of the repo.
Currently, we always read packed-refs file when 'trustFolderStat'
is false. Introduce a new config 'trustPackedRefsStat' which takes
precedence over 'trustFolderStat' when reading packed refs. Possible
values for this new config are:
* always: Trust packed-refs file attributes
* after_open: Same as 'always', but refresh the file attributes of
packed-refs before trusting it
* never: Always read the packed-refs file
* unset: Fallback to 'trustFolderStat' to determine if the file
attributes of packed-refs can be trusted
Folks whose repositories are on NFS and have traditionally been
setting 'trustFolderStat=false' can now get some performance improvement
with 'trustPackedRefsStat=after_open' as it refreshes the file
attributes of packed-refs (at least on some NFS clients) before
considering it.
For example, consider a repository on NFS with ~500k packed-refs. Here
are some stats which illustrate the improvement with this new config
when reading packed refs on NFS:
Matthias Sohn [Tue, 5 Oct 2021 02:08:49 +0000 (04:08 +0200)]
Add TernarySearchTree
A ternary search tree is a type of tree where nodes are arranged in a
manner similar to a binary search tree, but with up to three children
rather than the binary tree's limit of two.
Each node of a ternary search tree stores a single character, a
reference to a value object and references to its three children named
equal kid, lo kid and hi kid. The lo kid pointer must point to a node
whose character value is less than the current node. The hi kid pointer
must point to a node whose character is greater than the current
node.[1] The equal kid points to the next character in the word. Each
node in a ternary search tree represents a prefix of the stored strings.
All strings in the middle subtree of a node start with that prefix.
Like other prefix trees, a ternary search tree can be used as an
associative map with the ability for incremental string search. Ternary
search trees are more space efficient compared to standard prefix trees,
at the cost of speed.
They allow efficient prefix search which is important to implement
searching refs by prefix in a RefDatabase.
Searching by prefix returns all keys if the prefix is an empty string.
Thomas Wolf [Fri, 16 Dec 2022 22:23:03 +0000 (23:23 +0100)]
PatchApplier: fix handling of last newline in text patch
If the last line came from the patch, use the patch to determine whether
or not there should be a trailing newline. Otherwise use the old text.
Add test cases for
- no newline at end, last line not in patch hunk
- no newline at end, last line in patch hunk
- patch removing the last newline
- patch adding a newline at the end of file not having one
all for core.autocrlf false, true, and input.
Add a test case where the "no newline" indicator line is not the last
line of the last hunk. This can happen if the patch ends with removals
at the file end.
Bug: 581234
Change-Id: I09d079b51479b89400ad300d0662c1dcb50deab6 Also-by: Yuriy Mitrofanov <a2terminator@mail.ru> Signed-off-by: Thomas Wolf <twolf@apache.org>
CommitGraph: add commit-graph for FileObjectDatabase
This change makes JGit can read .git/objects/info/commit-graph file
and then get CommitGraph.
Loading a new commit-graph into memory requires additional time. After
testing, loading a copy of the Linux's commit-graph(1039139 commits)
is under 50ms.
Thomas Wolf [Fri, 16 Dec 2022 22:40:42 +0000 (23:40 +0100)]
Reformat PatchApplier and PatchApplierTest
Some lines were too long, unnecessary fully qualified class names,
and an assertEquals(actual, expected) when it should have been
assertEquals(expected, actual).
Change-Id: I3b3c46c963afe2fb82a79c1e93970e73778877e5 Signed-off-by: Thomas Wolf <twolf@apache.org>
Anna Papitto [Fri, 2 Dec 2022 23:56:56 +0000 (15:56 -0800)]
IO#readFully: provide overload that fills the full array
IO#readFully is often called with the intent to fill the destination
array from beginning to end. The redundant arguments for where to start
and stop filling are opportunities for bugs if specified incorrectly or
if not changed to match a changed array length.
Provide a overloaded method for filling the full destination array.
Change-Id: I964f18f4a061189cce1ca00ff0258669277ff499 Signed-off-by: Anna Papitto <annapapitto@google.com>
If 'core.commitGraph' and 'gc.writeCommitGraph' are both true, then gc
will rewrite the commit-graph file when 'git gc' is run. Defaults to
false while the commit-graph feature matures.
Git introduced a new file storing the topology and some metadata of
the commits in the repo (commitGraph). With this data, git can browse
commit history without parsing the pack, speeding up e.g.
reachability checks.
This change teaches JGit to read commit-graph-format file, following
the upstream format([1]).
JGit can read a commit-graph file from a buffered stream, which means
that we can provide this feature for both FileRepository and
DfsRepository.
Anna Papitto [Thu, 10 Nov 2022 22:48:22 +0000 (14:48 -0800)]
Gc#deleteOrphans: avoid dependence on PackExt alphabetical ordering
Deleting orphan files depends on .pack and .keep being reverse-sorted
to before the corresponding index files that could be orphans. The new
reverse index file extension (.rev) will break that frail dependency.
Rewrite Gc#deleteOrphans to avoid that dependence by tracking which pack
names have a .pack or .keep file and then deleting any index files that
without a corresponding one. This approach takes linear time instead of
the O(n logn) time needed for sorting.
Change-Id: If83c378ea070b8871d4b01ae008e7bf8270de763 Signed-off-by: Anna Papitto <annapapitto@google.com>
Ivan Frade [Tue, 6 Dec 2022 23:59:03 +0000 (15:59 -0800)]
GraphCommits: Remove unused getter by position
CommitGraphWriter uses the GraphCommits in for-each loops and doesn't
need the access by position anymore. This was a left-over from
https://git.eclipse.org/r/c/jgit/jgit/+/182832
Errorprone run in the bazel build raised this exception:
org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/commitgraph/CommitGraphWriter.java:105:
error: [UnusedException] This catch block catches an exception and
re-throws another, but swallows the caught exception rather than setting
it as a cause. This can make debugging harder.
} catch (InterruptedIOException e) {
^
(see https://errorprone.info/bugpattern/UnusedException)
Did you mean 'throw new
IOException(JGitText.get().commitGraphWritingCancelled, e);'?
Matthias Sohn [Tue, 13 Dec 2022 13:40:27 +0000 (14:40 +0100)]
Update jetty to 10.0.13
Since Oomph's p2 repo for jetty 10.0.13 doesn't have source bundles, we
remove them. Eclipse platform doesn't create p2 repos for jetty anymore
and we aren't yet ready to use maven dependencies like the platform
does.
Xing Huang [Fri, 9 Dec 2022 22:15:50 +0000 (16:15 -0600)]
PackExt: Add a commit graph extension.
There is no commit graph PackExt because the non-DFS stack is not writing using PackExt mechanism. The extension is needed in DFS to determine the stream to write the commit-graph.
Add a commit graph extension that matches the one in cgit
(https://git-scm.com/docs/commit-graph#_file_layout)
in preparation for adding DFS support for reading and writing commit graphs.
Sergey [Wed, 7 Dec 2022 11:49:47 +0000 (15:49 +0400)]
BatchRefUpdate: Consistent switch branches in ref update
The expression RefUpdate ru = newUpdate(cmd) is eagerly evaluated before the switch statement.
But it is not used in some switch cases and thus is calculated uselessly.
Move expression evaluation to the switch case where it is actually used.
After such a move, several cases became identical and thus were squashed.
Sergey [Wed, 7 Dec 2022 11:49:47 +0000 (15:49 +0400)]
RefWriter#writePackedRefs: Remove a redundant "if" check
After checking the variable, the same variable was checked again inside
the "if" block, and after the first check, this variable does not
change. Remove the second unnecessary check.
Dmitrii Filippov [Mon, 27 Jun 2022 18:03:53 +0000 (20:03 +0200)]
Fix crashes on rare combination of file names
The NameConflictTreeWalk class is used in merge for iterating over
entries in commits. The class uses a separate iterator for each
commit's tree. In rare cases it can incorrectly report the same entry
twice. As a result, duplicated entries are added to the merge result
and later jgit throws an exception when it tries to process merge
result.
The problem appears only when there is a directory-file conflict for
the last item in trees. Example from the bug:
Commit 1:
* subtree - file
* subtree-0 - file
Commit 2:
* subtree - directory
* subtree-0 - file
Here the names are ordered like this:
"subtree" file <"subtree-0" file < "subtree" directory.
The NameConflictTreeWalk handles similar cases correctly if there are
other files after subtree... in commits - this is processed in the
AbstractTreeIterator.min function. Existing code has a special
optimization for the case, when all trees are pointed to the same
entry name - it skips additional checks. However, this optimization
incorrectly skips checks if one of trees reached the end.
The fix processes a situation when some trees reached the end, while
others are still point to an entry.
Matthias Sohn [Wed, 23 Nov 2022 15:54:34 +0000 (16:54 +0100)]
Merge branch 'master' into stable-6.4
* master:
[pgm] Add options --name-only, --name-status to diff, log, show
Update Orbit to R20221123021534 for 2022-12
RBE: Update toolchain with bazel-toolchains 5.1.2 release
SshTestGitServer: : ensure UploadPack is closed to fix resource leak
UploadPackTest: ensure UploadPack is closed to fix resource leak
[pgm] Ensure UploadPack is closed to fix resource leak
UploadPackServlet#doPost use try-with-resource to ensure up is closed
Fix warnings in PatchApplierTest
Fix boxing warnings in TransportTest
Silence warnings about unclosed BasePackPushConnection
Fix warning about non-externalized String
Remove unused imports
Suppress non-externalized String warnings
Remove unused API problem filters
Silence API errors
Silence API errors
Silence API warnings
Add 4.26 target platform
Use "releases" repository for 4.25 target platform
Update Apache Mina SSHD to 2.9.2
Update Orbit to S20221118032057
DfsBlockCache: Report IndexEventConsumer metrics for reverse indexes.
DfsStreamKey: Replace ForReverseIndex to separate metrics.
RawText.isBinary(): handle complete buffer correctly
PackExt: Add a reverse index extension.
David Ostrovsky [Fri, 11 Nov 2022 10:18:38 +0000 (11:18 +0100)]
RBE: Update toolchain with bazel-toolchains 5.1.2 release
Due to this platform style migration: [1] the RBE toolchain needs to be
updated to use the latest rbe_config_gen from bazel-toolchains (at least
version 5.1.2 so that it contains: [2]).
This change makes RBE build forwards compatible so that Bazel could be
updated to the upcoming major 6.0 release.
Matthias Sohn [Sun, 20 Nov 2022 19:24:14 +0000 (20:24 +0100)]
Merge branch 'stable-6.3'
* stable-6.3:
Remove unused imports
Suppress non-externalized String warnings
Remove unused API problem filters
Silence API errors
Silence API errors
Silence API warnings
Matthias Sohn [Sun, 20 Nov 2022 19:22:24 +0000 (20:22 +0100)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
Remove unused imports
Suppress non-externalized String warnings
Remove unused API problem filters
Silence API errors
Silence API errors
Silence API warnings
Thomas Wolf [Mon, 25 Apr 2022 05:52:25 +0000 (07:52 +0200)]
Update Apache Mina SSHD to 2.9.2
Release notes for 2.9.2:
https://github.com/apache/mina-sshd/blob/master/docs/changes/2.9.2.md
Change-Id: I7809bcba1d45b76ab9dcc031f86beb2f69da3788 Signed-off-by: Thomas Wolf <twolf@apache.org> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 18 Nov 2022 21:09:04 +0000 (22:09 +0100)]
Update Orbit to S20221118032057
update
- com.jcraft.jsch to 0.1.55.v20221112-0806
- net.bytebuddy.byte-buddy to 1.12.18.v20221114-2102
- net.bytebuddy.byte-buddy-agent to 1.12.18.v20221114-2102
- org.apache.commons.codec to 1.14.0.v20221112-0806
- org.apache.httpcomponents.httpclient to 4.5.13.v20221112-0806
- org.slf4j.api to 1.7.30.v20221112-0806
- org.slf4j.binding.simple to 1.7.30.v20221112-0806
Anna Papitto [Fri, 11 Nov 2022 18:44:37 +0000 (10:44 -0800)]
DfsStreamKey: Replace ForReverseIndex to separate metrics.
Keys used for identifying reverse indexes in the DfsBlockCache use a
custom subclass ForReverseIndex because there was no PackExt for them.
This conflates BlockCacheMetrics for reverse indexes with those for
packs, since the key falls back onto 0 when there is no extension.
Replace the custom ForReverseIndex with a DfsStreamKey usage to bring
keys for the new REVERSE_INDEX extension in line with INDEX and BITMAP
and separate reverse index and pack BlockCacheMetrics.
Change-Id: I305e2c16d2a8cb2a824855ea92e0c9a9b188fce5 Signed-off-by: Anna Papitto <annapapitto@google.com>
Matthias Sohn [Wed, 16 Nov 2022 10:29:24 +0000 (11:29 +0100)]
Merge branch 'master' into stable-6.4
* master:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Fix API breakage caused by extracting WorkTreeUpdater
Update Orbit to S20221109014815
Use replace instead of replaceAll in toCleanString
Extract Exception -> HTTP status code mapping for reuse
Don't handle internal git errors as an HTTP error
Fix the path for JSchText.properties
Fix Maven SHA1 for Bazel build
UploadPack: Receive and parse client session-id
TransferConfig: Move reading advertisesid setting into TransferConfig
FirstWant: Parse client session-id if received.
ReceivePack: Receive and parse client session-id.
Ignore IllegalStateException if JVM is already shutting down
Allow to perform PackedBatchRefUpdate without locking loose refs
Matthias Sohn [Wed, 16 Nov 2022 09:15:30 +0000 (10:15 +0100)]
Merge branch 'stable-6.3'
* stable-6.3:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Fix API breakage caused by extracting WorkTreeUpdater
Extract Exception -> HTTP status code mapping for reuse
Don't handle internal git errors as an HTTP error
Ignore IllegalStateException if JVM is already shutting down
Allow to perform PackedBatchRefUpdate without locking loose refs
Matthias Sohn [Wed, 16 Nov 2022 09:14:13 +0000 (10:14 +0100)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
Extract Exception -> HTTP status code mapping for reuse
Don't handle internal git errors as an HTTP error
Allow to perform PackedBatchRefUpdate without locking loose refs
Matthias Sohn [Wed, 16 Nov 2022 09:13:20 +0000 (10:13 +0100)]
Merge branch 'stable-6.1' into stable-6.2
* stable-6.1:
Extract Exception -> HTTP status code mapping for reuse
Don't handle internal git errors as an HTTP error
Allow to perform PackedBatchRefUpdate without locking loose refs
Matthias Sohn [Wed, 16 Nov 2022 08:56:08 +0000 (09:56 +0100)]
Merge branch 'stable-6.2' into stable-6.3
* stable-6.2:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Matthias Sohn [Wed, 16 Nov 2022 08:55:22 +0000 (09:55 +0100)]
Merge branch 'stable-6.1' into stable-6.2
* stable-6.1:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Matthias Sohn [Wed, 16 Nov 2022 08:54:28 +0000 (09:54 +0100)]
Merge branch 'stable-6.0' into stable-6.1
* stable-6.0:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Anna Papitto [Thu, 10 Nov 2022 22:45:46 +0000 (14:45 -0800)]
PackExt: Add a reverse index extension.
There is no reverse index PackExt because the reverse index is not currently
written to a file. This prevents fine-grained performance reporting for reverse
indexes, which will be useful when introducing a reverse index file and
observing performance changes.
Add a reverse index extension that matches the one in cgit
(https://github.com/git/git/blob/9bf691b78cf906751e65d65ba0c6ffdcd9a5a12c/Documentation/gitformat-pack.txt#L302)
in preparation for adding a reverse index file while observing
performance before and after.
Change-Id: Iee53f1e01cf645a3c468892fcf97c8444f9a784a Signed-off-by: Anna Papitto <annapapitto@google.com>
Matthias Sohn [Tue, 15 Nov 2022 23:15:17 +0000 (00:15 +0100)]
Merge branch 'stable-5.13' into stable-6.0
* stable-5.13:
[benchmarks] Remove profiler configuration
Add SHA1 benchmark
[benchmarks] Set version of maven-compiler-plugin to 3.8.1
Fix running JMH benchmarks
Add option to allow using JDK's SHA1 implementation
Ignore IllegalStateException if JVM is already shutting down
Matthias Sohn [Tue, 4 Oct 2022 13:45:01 +0000 (15:45 +0200)]
Fix running JMH benchmarks
Without this I sometimes hit the error:
$ java -jar target/benchmarks.jar
Exception in thread "main" java.lang.RuntimeException: ERROR: Unable to
find the resource: /META-INF/BenchmarkList
at org.openjdk.jmh.runner.AbstractResourceReader.getReaders(AbstractResourceReader.java:98)
at org.openjdk.jmh.runner.BenchmarkList.find(BenchmarkList.java:124)
at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:253)
at org.openjdk.jmh.runner.Runner.run(Runner.java:209)
at org.openjdk.jmh.Main.main(Main.java:71)
Matthias Sohn [Fri, 11 Nov 2022 16:54:06 +0000 (17:54 +0100)]
Add option to allow using JDK's SHA1 implementation
The change If6da9833 moved the computation of SHA1 from the JVM's
JCE to a pure Java implementation with collision detection.
The extra security for public sites comes with a cost of slower
SHA1 processing compared to the native implementation in the JDK.
When JGit is used internally and not exposed to any traffic from
external or untrusted users, the extra cost of the pure Java SHA1
implementation can be avoided, falling back to the previous
native MessageDigest implementation.
Matthias Sohn [Mon, 14 Nov 2022 13:34:30 +0000 (14:34 +0100)]
Update Orbit to S20221109014815
and update
- com.sun.jna to 5.12.1.v20221103-2317
- com.sun.jna.platform to 5.12.1.v20221103-2317
- org.bouncycastle.bcpg to 1.72.0.v20221013-1810
- org.bouncycastle.bcpkix to 1.72.0.v20221013-1810
- org.bouncycastle.bcprov to 1.72.0.v20221013-1810
- org.bouncycastle.bcutil to 1.72.0.v20221013-1810
- org.mockito.mockito-core to 4.8.1.v20221103-2317
- org.objenesis to 3.3.0.v20221103-2317
projects / jgit.git / log