Matthias Sohn [Mon, 11 Nov 2024 23:06:18 +0000 (00:06 +0100)]
Merge branch 'master' into stable-7.1
* master:
errorprone: Disable javadoc checks in tests
Rename numberOfPackFilesAfterBitmap to numberOfPackFilesSinceBitmap
Replace custom encoder Constants#encodeASCII by JDK implementation
Replace custom encoder `Constants#encode` by JDK implementation
DfsGarbageCollector: #setReflogExpire with Instant instead of Date
ssh: Minor simplification in SerialRangeSet
DfsBlockCacheConfig: propagate hotmap configs to pack ext cache configs
SystemReader: Offer methods with java.time API
Add `numberOfPackFilesAfterBitmap` to RepoStatistics
Enhance CommitBuilder#parent to tolerate null parent
GPG: use BC PGP secret key parsing out of the box
[errorprone] bc: Remove unused SExprParser#parseSecretKey
Update bouncycastle to 1.79
Update bytebuddy to 1.15.10
DfsPackCompactor: write object size index
[errorprone] BaseRepositoryBuilder: Use #split(sep, limit)
[errorprone] Remove deprecated security manager
[errorprone] RefDatabase: #getConflictingNames immutable return
DfsGarbageCollector: Add setter for reflog expiration time.
[errorprone] SeparateClassloadertTestRunner: use #split(String,int)
[errorprone] HttpConnection: Add missing summary in java
[errorprone] PackWriter: Fix javadoc tag in new #writeIndex method
[errorprone] ByteArraySet: Add summary fragment to javadoc
[errorprone] util.Stats: Add summary fragment to javadoc
DfsInserter: Read minBytesForObjectSizeIndex only from repo config
PackWriter: make PackWriter.writeIndex() take a PackIndexWriter
dfs: update getBlockCacheStats to return a List of BlockCacheStats
Update mockito to 5.14.2
Update bytebuddy to 1.15.7
Remove unnecessary argument handler in MergeBase.java
Replace custom encoder Constants#encodeASCII by JDK implementation
Ivan Frade [Mon, 11 Nov 2024 19:07:21 +0000 (11:07 -0800)]
errorprone: Disable javadoc checks in tests
Errorprone finds many problems in the tests javadocs. This is noisy in
the logs, but fixing them also disturbs the project history and can
complicate merges.
Disable the javadoc checks in the tests packages. We can fix those
javadocs if some other modification happen in the file (as we fix
older coding style).
Ivan Frade [Fri, 8 Nov 2024 16:49:09 +0000 (08:49 -0800)]
DfsGarbageCollector: #setReflogExpire with Instant instead of Date
The Date API is full of major design flaws and pitfalls and should be
avoided at all costs. Prefer the java.time APIs, specifically,
java.time.Instant (for physical time) and java.time.LocalDate[Time]
(for civil time). [1]
Replace the Date with Instant in the
DfsGarbageCollector#setReflogExpire method.
Laura Hamelin [Wed, 6 Nov 2024 21:41:30 +0000 (13:41 -0800)]
DfsBlockCacheConfig: propagate hotmap configs to pack ext cache configs
CacheHotMap is currently only set on the base DfsBlockCacheConfig and is
not propagated down to PackExt specific caches.
Because CacheHotMap is set from a method call rather than from Configs,
this change sets per-PackExt CacheHotMap configs on PackExt cache
configs both when DfsBlockCacheConfig#setCacheHotMap(...) is called, and
when DfsBlockCacheConfig#configure(...) is called after setCacheHotMap.
The outer DfsBlockCacheConfig keeps the full CacheHotMap for the same
reason that the CacheHotMap config is propagated in both setCacheHotMap
and configure: the order of operations setting the configuration from
Configs and calling setCacheHotMap is not guaranteed.
Ivan Frade [Mon, 4 Nov 2024 22:21:24 +0000 (14:21 -0800)]
SystemReader: Offer methods with java.time API
Error prone explains: The Date API is full of major design flaws and
pitfalls and should be avoided at all costs. Prefer the java.time
APIs, specifically, java.time.Instant (for physical time) and
java.time.LocalDate[Time] (for civil time).
Add to SystemReader methods to get the time and timezone in the new
java.time classes (Instant/ZoneId) and mark as deprecated their old
counterparts.
The mapping of methods is:
* #getCurrentTime -> #now (returns Instant instead of int)
* #getTimezone -> #getTimeZoneAt (returns ZoneOffset intead of int)
* #getTimeZone -> #getTimeZoneId (return ZoneId instead of TimeZone)
Jacek Centkowski [Fri, 20 Sep 2024 06:47:13 +0000 (08:47 +0200)]
Add `numberOfPackFilesAfterBitmap` to RepoStatistics
Introduce a `numberOfPackFilesAfterBitmap` that contains the number of
packfiles created since the latest bitmap generation.
Notes:
* the `repo.getObjectDatabase().getPacks()` that obtains the list of
packs (in the existing `getStatistics` function) uses
`PackDirectory.scanPacks` that boils down to call
`PackDirectory.scanPacksImpl` which is sorting packs prior returning
them therefore the `numberOfPackFilesAfterBitmap` is just all packs
before the one that has bitmap attached
* the improved version of `packAndPrune` function (one that skips
non-existent packfiles) was introduced for testing
Thomas Wolf [Wed, 6 Nov 2024 18:14:47 +0000 (19:14 +0100)]
GPG: use BC PGP secret key parsing out of the box
Remove the custom S-expression parsing; BC has gotten many
improvements in 1.79 regarding PGP ed25519 keys, AES/OCB
encryption, and generally parsing key files. It now can do
all we need.
Change-Id: I392443e040cce150a9575d18795a7cb8195a3515 Signed-off-by: Thomas Wolf <twolf@apache.org>
errorprone complains about using Date in the SExprParser class. All
the usages are in a variant of the parseSecretKey method that doesn't
have any callers.
Matthias Sohn [Tue, 5 Nov 2024 00:29:08 +0000 (01:29 +0100)]
Merge branch 'stable-7.1'
* stable-7.1:
Add missing @since 7.1 to UploadPack#implies
ResolveMerger: Allow setting the TreeWalk AttributesNodeProvider
Add Union merge strategy support
Nasser Grainawi [Tue, 29 Oct 2024 23:22:15 +0000 (17:22 -0600)]
ResolveMerger: Allow setting the TreeWalk AttributesNodeProvider
When a merger is created without a Repository, no
AttributesNodeProvider is created in the TreeWalk. Since mergers are
often created with a custom ObjectInserter and no repo, they skip any
lookups of attributes from any of the gitattributes files (within a
tree, in the repo info/ dir, or user/global). Since there are
potentially merge-affecting attributes in those files, callers might
want to use both a custom ObjectInserter and an AttributesNodeProvider.
Ivan Frade [Fri, 1 Nov 2024 15:58:27 +0000 (08:58 -0700)]
DfsPackCompactor: write object size index
Currently the compactor is not writing the object size index for
packs. As it is using PackWriter to generate the packs, it needs to
explicitely call the writes of each extension.
Invoke writeObjectSizeIndex in the compactor. The pack writer will
write one if the configuration says so.
Ivan Frade [Thu, 31 Oct 2024 19:17:09 +0000 (12:17 -0700)]
[errorprone] Remove deprecated security manager
Errorprone warns about this deprecated classes. The recommendation is
stop using SecurityManager all together.
The Security Manager is deprecated and subject to removal in a future
release. There is no replacement for the Security Manager. See JEP 411
[1] for discussion and alternatives.
Errorprone reports that: This method returns both mutable and
immutable collections or maps from different paths. This may be
confusing for users of the method.
Saril Sudhakaran [Tue, 29 Oct 2024 05:17:01 +0000 (00:17 -0500)]
DfsGarbageCollector: Add setter for reflog expiration time.
JGit reftable writer/compator knows how to prune the history, but the
DfsGarbageCollector doesn't expose the time limit.
Add a method to DfsGarbageCollector to set the reflog time limit.
This value is then passed to the reftable compactor. Callers usually
pass here the value from gc.reflogExpire.
The reflog block length is stored in 24 bits [1], limiting the size to
16MB. I have observed that in repositories with frequent commits,
reflogs hit that size in 6-12 months.
Ivan Frade [Tue, 9 Jul 2024 18:10:02 +0000 (11:10 -0700)]
[errorprone] util.Stats: Add summary fragment to javadoc
Errorprone complains about missing summary in these javadocs.
[MissingSummary] A summary fragment is required; consider using the
value of the @return b lock as a summary fragment instead.
* @return variance of the added values
^
(see https://google.github.io/styleguide/javaguide.html#s7.2-summary-fragment)
Did you mean '*Returns variance of the added values.'?
Ivan Frade [Tue, 29 Oct 2024 17:57:28 +0000 (10:57 -0700)]
DfsInserter: Read minBytesForObjectSizeIndex only from repo config
In general, JGit reads the configuration it needs from the repository
configuration. minBytesForObjectSizeIndex is a special case with a
setter for subclasses but that is unnecessary.
Remove the setter and read the conf from the repo. Make the property
final and read it directly from the conf (it is clearer than parsing a
whole PackConfig to read a single value).
Sam Delmerico [Mon, 7 Oct 2024 21:34:03 +0000 (14:34 -0700)]
PackWriter: make PackWriter.writeIndex() take a PackIndexWriter
Previously, the PackWriter implementation required that indexes and
their extensions be writable to an OutputStream with a fixed binary
format. To support more general index storage formats, allow
PackWriter to accept an PackIndexWriter interface which accepts only
the objects to store. This allows implementors to choose their storage
format.
The implementation will be provided by the DfsObjectDatabase. The
DfsObjectDatabase is already responsible for providing the OutputStream
that was previously used to write indexes. Having it provide a writing
interface would be a natural generalization.
This idea was previously implemented for PackBitmapIndex writing in
https://gerrithub.io/c/eclipse-jgit/jgit/+/1177722.
Laura Hamelin [Wed, 24 Jul 2024 21:01:38 +0000 (14:01 -0700)]
dfs: update getBlockCacheStats to return a List of BlockCacheStats
Make available all underlying cache table stats for the used cache table
implementation.
The existing cache table stats implementation only allows a "global"
view of the cache table statistics; it does not differentiate between
all possible underlying cache tables used.
This change allows callers to get the block cache stats broken down
per underlying table. These cache stats are intended to be used for
monitoring all cache tables independently.
Existing usages of getBlockCacheStats now make use of
AggregatedBlockCacheStats.fromStatsList to aggregate the list of
BlockCacheStats into a single BlockCacheStats instance.
Matthias Sohn [Wed, 23 Oct 2024 20:17:29 +0000 (20:17 +0000)]
Merge changes from topic "ssh-signatures"
* changes:
SSH signing: implement a SignatureVerifier
SSH signing: implement a Signer
SSH signing: don't require a session in PasswordProviderWrapper
SSH signing: make OpenSSH pattern matching public
SSH signing: prepare config
ssh: add a factory for KeyPasswordProvider
Thomas Wolf [Sat, 28 Sep 2024 14:11:45 +0000 (16:11 +0200)]
SSH signing: implement a SignatureVerifier
Signature verification needs quite a bit of infrastructure. There are
two files to read: a list of allowed signers, and a list of revoked keys
or certificates. Introduce a SigningKeyDatabase abstraction for these,
and give client code the possibility to plug in its own implementation.
Loading these files afresh for every signature to be checked would be
prohibitively expensive. Introduce a cache of SigningKeyDatabases, and
have them reload the files only when they have changed.
Include a default implementation that works with the OpenSSH allowed
signers file and with OpenSSH revocation lists. Binary KRLs are parsed
according to [1]; the test data was generated using the OpenSSH test
script[2].
Thomas Wolf [Sat, 28 Sep 2024 13:58:20 +0000 (15:58 +0200)]
SSH signing: implement a Signer
Implement a Signer and its factory, and publish the factory for the
ServiceLoader. SSH signatures can be created directly if the key is
given via a file in user.signingKey and the private key can be found.
Otherwise, signing is delegated to an SSH agent, if available.
If a certificate is used as public key, the signer verifies the
certificate (correct signature, and valid at the commit time).
Sam Delmerico [Mon, 7 Oct 2024 22:20:26 +0000 (15:20 -0700)]
PackIndexWriter: create interface to write indexes
PackWriter assumes that the primary index goes to a file in a well-known
format. This cannot accomodate implementations in other storages or
formats (e.g. in a database).
Create an interface to write the index (PackIndexWriter). This interface
will be implemented by the existing pack index writer classes
(PackIndexWriterV1 etc.).
As the "PackIndexWriter" name was used by the previous superclass of the
file writers, we rename that class to "BasePackIndexWriter".
Jacek Centkowski [Fri, 20 Sep 2024 06:47:13 +0000 (08:47 +0200)]
Add `numberOfPackFilesAfterBitmap` to RepoStatistics
Introduce a `numberOfPackFilesAfterBitmap` that contains the number of
packfiles created since the latest bitmap generation.
Notes:
* the `repo.getObjectDatabase().getPacks()` that obtains the list of
packs (in the existing `getStatistics` function) uses
`PackDirectory.scanPacks` that boils down to call to
`PackDirectory.scanPacksImpl` which is sorting packs prior returning
them therefore the `numberOfPackFilesAfterBitmap` is just all packs
before the one that has bitmap attached
* the improved version of `packAndPrune` function (one that skips
non-existent packfiles) was introduced for testing
Thomas Wolf [Sat, 28 Sep 2024 13:52:31 +0000 (15:52 +0200)]
SSH signing: make OpenSSH pattern matching public
SSH signing needs the same pattern matching algorithm as is used for
host matching in host entries in ~/.ssh/config. So make that pattern
matching available via a static method.
Change-Id: Ia26f23666f323f44ce66f769fbcd6c85965eb219 Signed-off-by: Thomas Wolf <twolf@apache.org>
Thomas Wolf [Fri, 20 Sep 2024 19:59:49 +0000 (21:59 +0200)]
ssh: add a factory for KeyPasswordProvider
Introduce a global default factory to create KeyPasswordProvider.
Previously, their creation was tied to the SSH session, but for SSH
signatures, we will need to be able to create KeyPasswordProviders
without having an SSH session.
Change-Id: If4a69c4d4c4e8de390cb1ef3b65966d0e39c24ff Signed-off-by: Thomas Wolf <twolf@apache.org>
Laura Hamelin [Wed, 16 Oct 2024 22:16:00 +0000 (15:16 -0700)]
DfsBlockCache: refactor stats implementations.
The stats interface has an implementation in the interface itself and another inside the PackExtBlockCache class. This asymmetry gets on the way to implement stats-per-table later.
Make DfsBlockCacheStats (the stats of a single table) a top-level class and create an aggregator class to combine multiple stats. This makes the stats classes mirror the table classes structure (singles tables + composite).
This change is part of a refactor to support providing detailed stats
breakdowns for cache implementations using multiple table instances
while keeping the existing "aggregated" view of cache stats.
This will write out configuration values on a line by line basis to a
given PrintWriter.
Primary usage is as a semi-formatted debug print of the configuration
values used by dfs block cache.
Luca Milanesio [Fri, 11 Oct 2024 21:40:16 +0000 (22:40 +0100)]
Test advertised capabilities with protocol V0 and allow*Sha1InWant
The advertised capabilities with protocol V0 were untested
leading to potential regressions when advertising what
SHA1 should or should not be on the list of capabilities.
Verify that allow-tip-sha1-in-want and allow-reachable-sha1-in-want
are properly advertised with the allow*Sha1InWant is set in
jgit.config.
Luca Milanesio [Mon, 7 Oct 2024 22:16:58 +0000 (23:16 +0100)]
Align request policies with CGit
CGit defines the SHA request policies using a bitmask
that represents which policy is implied by another policy.
For example, in CGit the ALLOW_TIP_SHA1 is 0x01 and ALLOW_REACHABLE_SHA1
is 0x02, which are associated to two different bit in a 3-bit value.
The ALLOW_ANY_SHA1 value is 0x07 which denotes a different policy that
implies the previous two ones, because is represented with a 3-bit
bitmask having all ones.
Associate the JGit RequestPolicy enum to the same CGit bitmask values
and use the same logic for the purpose of advertising the server
capabilities.
The JGit code becomes easier to read and associate with its counterpart
in CGit, especially during the capabilities advertising phase.
Also add a new utility method RequestPolicy.implies() which is more
readable than a direct bitmask and operator.
Matthias Sohn [Wed, 9 Oct 2024 00:04:55 +0000 (02:04 +0200)]
Update Apache sshd to 2.14.0
This fixes an 'incorrect signature' error when trying to use the keys
generated by SSHD during server init with an OpenSSH client.
This also includes a few other changes since 2.13.2:
* GH-524 Performance improvements
* GH-533 Fix multi-step authentication
* GH-582 Fix filtering in NamedFactory
* GH-587 Prevent NullPointerExceptionon closed channel in NettyIoSession
* GH-590 Better support for FIPS
* GH-597 Pass on Charset in ClientSession.executeRemoteCommand()
Matthias Sohn [Wed, 9 Oct 2024 00:04:55 +0000 (02:04 +0200)]
Update Apache sshd to 2.14.0
This fixes an 'incorrect signature' error when trying to use the keys
generated by SSHD during server init with an OpenSSH client.
This also includes a few other changes since 2.13.2:
* GH-524 Performance improvements
* GH-533 Fix multi-step authentication
* GH-582 Fix filtering in NamedFactory
* GH-587 Prevent NullPointerExceptionon closed channel in NettyIoSession
* GH-590 Better support for FIPS
* GH-597 Pass on Charset in ClientSession.executeRemoteCommand()
Kamil Musin [Tue, 8 Oct 2024 11:36:00 +0000 (13:36 +0200)]
RevolveMerger: honor ignoreConflicts also for binary files
Currently difference in binary files during merge will cause them to be
added to unmergedPaths regardless of whether ignoreConflicts is true.
This creates an issue during merging with strategy "RECURSIVE", as it
makes it impossible to create a virtual commit if there is a difference
in a binary file. Resulting in the
CONFLICTS_DURING_MERGE_BASE_CALCULATION error being thrown.
This is especially problematic, since JGit has a
rather simplistic rules for considering file binary, which easily leads
to false positives.
What we should do instead is keep OURS. This will not lead to silently
ignoring difference in the final result. It will allow creation of
virtual merge-base commit, and then the difference would be presented
again in the final merge results. In essense it only affects what's
shown as BASE in 3-way merge.
Additionally, this is correct because
- It's consistent with treatment of other unmergeable entities, for
example Gitlinks
- It's consistent with behaviour of CGit:
- https://git-scm.com/docs/gitattributes#Documentation/gitattributes.txt-binary
states on diffs in binary OURS is picked by default.
- In code: https://git.kernel.org/pub/scm/git/git.git/tree/merge-ll.c#n81
- ignoreConflicts in CGit afterwards ignores all issues with content
merging https://git.kernel.org/pub/scm/git/git.git/tree/merge-ort.c#n5201
We also adjust the behaviour when .gitattributes tell us to treat the
file as binary for the purpose of the merge.
We only change the behaviour when ignoreConlicts = true, as otherwise
the current behaviour works as intended.
Kamil Musin [Tue, 8 Oct 2024 11:25:53 +0000 (13:25 +0200)]
Remove unnecessary argument handler in MergeBase.java
Currently the parsing of command line arguments always results in an
error.
```
fatal: Argument "commit-ish" is required
```
This is caused by the fact that the first method (commit_0) is invoked
twice. (As if the multiValued = true). I couldn't figure out, why that
would be the case, as it's defaults to false and RevCommit handler
doesn't seem to have a logic that would override that either.
In any case, defining Arguments like this is unnecessary as it's
sufficient to define a single ArrayList @Argument that will accumulate
all commits.
Laura Hamelin [Mon, 10 Jun 2024 20:42:03 +0000 (13:42 -0700)]
DfsBlockCache: use PackExtBlockCacheTable when configured
Adds the usage of PackExtBlockCacheTable to the
DfsBlockCache, replacing the current DfsBlockCacheTable
when PackExtCacheConfigurations exists.
When no PackExtCacheConfigurations exists the current
DfsBlockCacheTable implementation will be used.
projects / jgit.git / log