Ivan Frade [Thu, 14 Nov 2019 21:56:27 +0000 (13:56 -0800)]
BitmappedReachabilityChecker: Use only one bitmap for the whole check
The checker is creating a new bitmap per branch leading to excessive
memory consumption. For the reachability check one bitmap with the
reachability of all branches aggregated is enough.
Build the reachability bitmap with a filter. The filter itself uses it
to emit only commits not reached before and the caller to check what
targets have been reached already.
BitmapCalculator is not required anymore.
Change-Id: Ic5c62f77fe0f188913215b7eaa51d849a9aae6a5 Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Tue, 19 Nov 2019 20:08:53 +0000 (12:08 -0800)]
checkNotAdvertisedWants: Be lazy converting Ref to RevCommit
The ref points to an ObjectId that then is translated into a RevCommit.
This translation can be costly and with the incremental reachability
check is probably not needed for most of the elements.
Delay the translation from ObjectId to RevCommit to when it is needed.
Use Streams, that have the laziness built-in, all the way from Ref to
RevCommit.
This should reduce the latency for reachability checks over big sets of
references.
Change-Id: I28693087321b2beff3eaa1f3d2e7840ab0eedc6d Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Tue, 19 Nov 2019 18:40:21 +0000 (10:40 -0800)]
ReachabilityChecker: Receive a Stream instead of a Collection
Preparatory change. Converting ObjectIds to RevCommits is potentially
expensive and in the incremental reachability check, it is probably not
required for all elements in the collection.
Pass a Stream to the reachability checker. In the follow up we make
the conversion from ObjectId to RevCommit in the stream (i.e. on
demand). This should reduce the latency of reachability checks over big
sets of references.
Change-Id: I9f310e331de5b0bf8de34143bd7dcd34316d2fba Signed-off-by: Ivan Frade <ifrade@google.com>
Ivan Frade [Tue, 12 Nov 2019 22:33:29 +0000 (14:33 -0800)]
UploadPack: Prioritize references for non-advertised wants checks
UploadPack needs to check if object ids that weren't advertised before
are reachable from the references visible to the user. In the
bitmap-based reachability check, this is done incrementally: checking
against one reference, if anything remaining adding a second and so on.
It is more efficient to check first more common references (e.g. refs/heads/*)
Sort the references for the reachability checker. This should solve the
connectivity earlier and require less bitmap creation and less memory.
Change-Id: I48ac10d71e29fab2d346479802401eaea4aacb5c Signed-off-by: Ivan Frade <ifrade@google.com>
Han-Wen Nienhuys [Mon, 12 Aug 2019 15:33:51 +0000 (17:33 +0200)]
file: implement FileReftableDatabase
Reftable is a binary, block-based storage format for the ref-database.
It provides several advantages over the traditional packed + loose
storage format:
* O(1) write performance, even for deletions and transactions.
* atomic updates to the ref database.
* O(log N) lookup and prefix scans
* free from restrictions imposed by the file system: it is
case-sensitive even on case-insensitive file systems, and has
no inherent limitations for directory/file conflicts
* prefix compression reduces space usage for repetitive ref names,
such as gerrit's refs/changes/xx/xxxxx format.
FileReftableDatabase is based on FileReftableStack, which does
compactions inline. This is simple, and has good median performance,
but every so often it will rewrite the entire ref database.
For testing, a FileReftableTest (mirroring RefUpdateTest) is added to
check for Reftable specific behavior. This must be done separately, as
reflogs have different semantics.
Add a reftable flavor of BatchRefUpdateTest.
Add a FileReftableStackTest to exercise compaction.
Add FileRepository#convertToReftable so existing testdata can be
reused.
Michael Keppler [Sat, 16 Nov 2019 15:49:38 +0000 (16:49 +0100)]
Fix Maven missing version warning
[WARNING] Some problems were encountered while building the effective
model for org.eclipse.jgit:org.eclipse.jgit.benchmarks:jar:5.6.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-shade-plugin is missing. @ line 131,
column 15
Change-Id: Id25916c460bc4a10d0f797779d24f53292df2b77 Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
Michael Keppler [Sat, 16 Nov 2019 15:23:34 +0000 (16:23 +0100)]
Use Java 8 source and target in Maven
Building the jgit aggregator with Maven leads to a compiler level 1.6
for me, thereby failing the build. Multiple profiles specify
source/target level separately, but there seems no global setting.
Adding the global properties makes the jgit aggregator compile on my
system.
Change-Id: Ia5613cc0fcf6085265a9e48f940e5a7d3f998608 Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de>
David Pursehouse [Thu, 14 Nov 2019 00:40:14 +0000 (16:40 -0800)]
ReceivePack: Reduce visibility of methods
Several methods were protected when they were defined in BaseReceivePack
which has since been squashed into ReceivePack. Those methods no longer
need to be protected, and can now be private instead.
Change-Id: Ic6bb5e66f353fcbdc7919847423234af398c88b4 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Mon, 28 Oct 2019 17:11:57 +0000 (18:11 +0100)]
DiffFormatter: support core.quotePath = false
core.quotePath = false means that "bytes higher than 0x80 are not
considered "unusal" anymore"[1], i.e., they are not escaped. In
essence this preserves non-ASCII characters in path names in output.
Note that control characters and other special characters in the
ASCII range will still be escaped.
Add a new QuotedString.GIT_PATH_MINIMAL singleton implementing this.
Change the normal GIT_PATH algorithm to use bytes instead of characters
so it can be re-used. Provide a setter in DiffFormatter for the quoting
style so that an application can override the default, which is the
setting from the git config (and by default "true"). Use the new
QuotedString.GIT_PATH_MINIMAL when core.quotePath == false.
Thomas Wolf [Sat, 26 Oct 2019 17:58:07 +0000 (19:58 +0200)]
IndexDiff/SubmoduleWalk: make the RepositoryBuilder configurable
Some applications using JGit use their own repository caching. In
such applications, it may be needlessly inefficient to create new
submodule repositories from a SubmoduleWalk or in an IndexDiff. It
can be much more efficient to use an already cached repository
instance.
Provide a way to configure a SubmoduleWalk with a factory to create
BaseRepositoryBuilders to use to create repositories, and use it in
IndexDiff. Provide new IndexDiff.diff() operations that take such an
additional factory as parameter.
An application that caches Repository instances (for instance EGit)
can use a factory that provides builders that don't create a new
Repository instance but that return the already cached instance, if
one is available. Note that in such a case, the application may need
to be prepared to deal with IndexDiff.diff() also _closing_ the
obtained repository; if the application expects its cached Repository
instances to remain open while being cached, it'll have to use
Repository.incrementOpen() to prevent that the repository instance
gets closed.
Bug: 550878
Change-Id: Icc1b34dfc4cebd8ed4739dd09d37744d41adf711 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
IndexDiff would apply ignore mode ALL from .gitmodules to all remaining
submodules, and would ignore other settings from .gitignore and always
apply the setting defined on the IndexDiff instead. Correct that.
In canonical git the ignore setting from .gitmodules can also be
overridden by .git/config.[1] Implement that override in SubmoduleWalk.
I1ce92869435d5eebb7d671be44561e69c6233134 merged BaseReceivePack into
ReceivePack which breaks API but is only affecting the few jgit based
servers out in the wild.
Change-Id: Iad856a2afaf3cad95d01ad81a0116cebcd9de2d9 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Move the BaseReceivePack implementation back into ReceivePack. This is a
backward-incompatible change. For example, BaseReceivePack.FirstLine no
longer exists and cannot be referenced. However, most of the code
should just work by replacing BaseReceivePack with ReceivePack.
Although this is an API change, it only affects callers using JGit as a
server, and there are very few of those in the wild.
David Pursehouse [Wed, 13 Nov 2019 18:16:25 +0000 (10:16 -0800)]
BaseReceivePack: Add hasReceivedPack method
The getPackSize method will throw IllegalStateException if it is
called when the pack size is not set. This is the case for example
when the received commands are all DELETE and there is no pack.
Add a new method hasReceivedPack that can be called prior to calling
getPackSize, to avoid causing the IllegalStateException.
Thomas Wolf [Tue, 12 Nov 2019 08:38:04 +0000 (09:38 +0100)]
Fix NPE in SystemReader in tests
SystemReader.updateAll() must _not_ test whether the file exists. In
tests at least there are FileBasedConfigs with a null file. Test
configs should (and do) override isOutdated() to deal with this case.
Change-Id: I56303fe0d56afeb9f2203ee807a92c5dcf3809e9 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn [Mon, 11 Nov 2019 19:07:01 +0000 (11:07 -0800)]
Merge branch 'stable-5.5'
* stable-5.5:
BaseReceivePack: Fix the format
Prepend hostname to subsection used to store file timestamp resolution
Store filesystem timestamp resolution in extra jgit config
SystemReader: extract updating config and its parents if outdated
Change-Id: Iecfddce8081303af29badcdcd3d72a0da50c964f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 7 Oct 2019 17:05:07 +0000 (19:05 +0200)]
Prepend hostname to subsection used to store file timestamp resolution
This ensures the measured filesystem timestamp resolution will be only
used on the machine where it was measured and avoid errors in case the
~/.jgitconfig file is copied to another machine.
Bug: 551850
Change-Id: Iff2a11be62ca94c3bbe4a955182988dc50852f9f Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Mon, 7 Oct 2019 15:58:56 +0000 (17:58 +0200)]
Store filesystem timestamp resolution in extra jgit config
This avoids polluting hand-crafted user level config with
auto-configured options which might disturb in environments where
the user level config is replicated between different machines.
Add a jgit config as parent of the system level config. Persist
measured timestamp resolutions always in this jgit config and read it
via the user global config. This has the effect that auto-configured
timestamp resolution will be used by default and can be overridden in
either the system level or user level config.
Store the jgit config under the XDG_CONFIG_HOME directory following the
XDG base directory specification [1] in order to ensure that we have
write permissions to persist the file. This has the effect that each OS
user will use its jgit config since they typically use different
XDG_CONFIG_HOME directories.
If the environment variable XDG_CONFIG_HOME is defined the jgit config
file is located at $XDG_CONFIG_HOME/jgit/config otherwise the default is
~/.config/jgit/config.
If you want to avoid redundant measurement for different OS users
manually copy the values measured and auto-configured for one OS user to
the system level git config.
Change-Id: I0f2fd3a218ea31f6f6b58b1816af9361e4de54e4 Signed-off-by: David Ostrovsky <d.ostrovsky@gmx.de> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Sat, 21 Sep 2019 12:38:22 +0000 (14:38 +0200)]
SystemReader: extract updating config and its parents if outdated
Change-Id: Ia77f442e47c5670c2d6d279ba862044016aabd86 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 15:28:06 +0000 (16:28 +0100)]
Merge branch 'stable-5.5'
* stable-5.5:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Update API problem filters
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: If87a13d0f809d9968ad2921b786f4b18e1b494d0 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 15:10:48 +0000 (16:10 +0100)]
Merge branch 'stable-5.4' into stable-5.5
* stable-5.4:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Update API problem filters
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: I071536d630a95e89f5bbbf965a1571b9f5eb81ee Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 14:42:14 +0000 (15:42 +0100)]
Merge branch 'stable-5.3' into stable-5.4
* stable-5.3:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Update API problem filters
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: Ib735c4039e24ec8b045ae2cc81df1e9e5c9fa996 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 14:24:54 +0000 (15:24 +0100)]
Merge branch 'stable-5.2' into stable-5.3
* stable-5.2:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Update API problem filters
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: I00615958ab6fbdff601e87a9792aba5606cda12a Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 5 Nov 2019 14:07:39 +0000 (15:07 +0100)]
Merge branch 'stable-5.1' into stable-5.2
* stable-5.1:
Run JMH benchmarks using bazel
Benchmark for creating files and FileSnapshots
Implement benchmark for looking up FileStore of a given Path
JMH benchmark for SimpleLruCache
Remove unused API problem filters
Silence API errors for new API added since 5.1.0
Change-Id: If91c55a192d3b2c441d9c8d414f2e24a7261b1b6 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sat, 3 Aug 2019 10:02:34 +0000 (12:02 +0200)]
JMH benchmark for SimpleLruCache
See [1] for JMH documentation and [2] how to use JMH in Eclipse.
The benchmarks pom currently cannot use the JGit parent pom due to an
ecj bug [3] regarding annotation processing. Hence for now do not
inherit from the JGit parent pom and copy the compiler plugin
configuration for javac from the parent pom.
After running the Maven build the benchmark can be run using Maven:
Thomas Wolf [Sat, 2 Nov 2019 18:26:42 +0000 (19:26 +0100)]
WorkingTreeModifiedEvent: must be fired explicitly after merge
A merge may write files to the working tree. After a successful
merge one must fire a WorkingTreeModifiedEvent explicitly if
getModifiedFiles() is not empty.
Also, any touched files must be reported by the
WorkingTreeModifiedEvent fired by DirCacheCheckout.checkout().
Bug: 552636
Change-Id: I5fab8279ed8be8a4ae34cddfa726836b9277aea6 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Han-Wen Nienhuys [Sun, 13 Oct 2019 16:14:17 +0000 (18:14 +0200)]
reftable: enforce ascending order in sortAndWriteRefs
MergedReftableTest#scanDuplicates tests whether we can write duplicate
keys in a merged reftable. Apparently, the first key appearing should
get precedence, and this works because the sort() algorithm on ordered
collections is stable.
This is potentially confusing behavior, because you can write data
into the table that cannot be retrieved (Merged table can only have
one entry per key), and the APIs such as exactRef() only return a
single value.
Make this consistent with behavior introduced in I04f55c481 "reftable:
enforce ordering for ref and log writes" by considering a duplicate key
in sortAndWriteRefs as a fatal runtime error.
Thomas Wolf [Wed, 31 Jul 2019 17:40:40 +0000 (19:40 +0200)]
HttpSupport.proxyFor(): use only scheme, host, and port
Some URLs cannot be converted via URL.toURI(). So don't convert
the full URL but only the bits that are needed to find a proxy
via java.net.ProxySelector.
Bug: 549690
Change-Id: I55b5ecee70c6b52f72f9bdba9ce552fde7f33976 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
By using ${min_update}-${max_update} as file name template, we
guarantee that each file has a unique name.
This allows data from open files to be cached across reloads of the
stack.
This is in anticipation of Change I1837f268e
("file: implement FileReftableDatabase"), which is the first
implementation of reftable on a filesystem.
Michael Keppler [Fri, 25 Oct 2019 11:20:05 +0000 (13:20 +0200)]
Upgrade Jetty to 9.4.22.v20191022
Change-Id: Ie2ee0a978949f9cf70833645132eedf26c3cf1c8 Signed-off-by: Michael Keppler <Michael.Keppler@gmx.de> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Tue, 22 Oct 2019 22:55:28 +0000 (00:55 +0200)]
GPG: implement more OpenPGP UserId matching formats
Instead of just looking for a substring match of user.signingKey
in a key's user ID implement the GPG matching formats[1] for:
'=' Full exact match
'<' Full exact match of the e-mail address
'@' Substring match within the e-mail address only
'*' General case-insensitive substring match (default)
When user.signingKey is not set, the committer's e-mail address is
used by default. In that case, use '<', i.e., require an exact match
on the OpenPGP e-mail address.
Also handle the optional "0x" prefix for (partial) key fingerprints.
Jonathan Tan [Tue, 22 Oct 2019 00:26:22 +0000 (17:26 -0700)]
TreeRevFilterTest: Simplify filter used
TreeRevFilterTest uses an unncessarily complicated TreeFilter - an
AndTreeFilter - when it should be as simple as possible because this
class tests TreeRevFilter, not AndTreeFilter. Replace the filter with a
simpler one.
Change-Id: I3256a65f6e0042d32fd76a9224b79a835674ff3a Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Roan Hofland [Tue, 22 Oct 2019 00:16:43 +0000 (02:16 +0200)]
GpgKeyLocator: Return a signing key for a user instead of the master key
Currently when a GPG key is looked up using a user identity the first
key from the keyring that has this user identity is returned.
The code was changed to instead return the first signing [S] key in this
keyring and only return the master key if no such signing key was found.
If the master key also does not have the signing flag set null is
returned instead.
Roan Hofland [Tue, 22 Oct 2019 00:06:31 +0000 (02:06 +0200)]
GpgKeyLocator: Return subkeys instead of the first key for subkeys
Currently when a subkey is configured for signing via the git
user.signingkey configuration option the first key from the keyring for
this subkey would be returned for use (master key). The code has been
changed to return the requested key from the keyring instead.
Thomas Wolf [Fri, 20 Sep 2019 14:41:39 +0000 (16:41 +0200)]
Support for core.hooksPath
Support the core.hooksPath git config. This can be an absolute or
relative path of a directory where to find git hooks; a relative
path is resolved relative to the directory the hook will run in.
Bug: 500266
Change-Id: I671999a6386a837e897c31718583c91d8035f3ba Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Matthias Sohn [Mon, 21 Oct 2019 07:29:50 +0000 (09:29 +0200)]
Remove meaningless comment in CloneCommandTest
This comment was probably copied from testCloneRepositoryWithBranch() to
testBareCloneRepositoryOnlyOneBranch() where it doesn't make sense.
Hence remove it.
Change-Id: I846debd084dd77fd473c3602a799f195a8390f77 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
This introduces ReftableBatchRefUpdate and ReftableDatabase, as
generic classes, with some code moved to DfsReftableBatchRefUpdate and
DfsReftableDatabase.
Clarify thread-safety requirements by asserting locked status in
accessors, and acquiring locks in callers. This does not fix threading
problems, because ReftableBatchRefUpdate already wraps the whole
transaction in a lock.
This also fixes a number of bugs in ReftableBatchRefUpdate:
* non-atomic updates should not bail on first failure
* isNameConflicting should also check for conflicts between names that
are added and removed in the BatchRefUpdate.
Change-Id: I5ec91173ea9a0aa19da444c8c0b2e0f4e8f88798 Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Han-Wen Nienhuys [Sun, 13 Oct 2019 17:21:06 +0000 (19:21 +0200)]
reftable: fix lookup by ID in merged reftables
On changing a ref, the old SHA1 is not updated in the object => ref
mapping. This means search by object ID may still turn up a ref from
deeper within the stack. To fix this, check all refs produced by the
merged iterator against the merged reftables.
projects / jgit.git / log