Use Map interface instead of ConcurrentHashMap class
On Android, the co-variant override of ConcurrentHashMap.keySet()
introduced in Java 8 was undone. [1] If compiled Java code calls that
co-variant override directly, one gets a NoSuchMethodError exception
at run-time on Android.
Making the code call that method via Map.keySet() side-steps this
problem.
This is similar to bug 496262, where the same problem cropped up when
compiling with Java 8 against a Java 7 target, but here we cannot use
bootclasspath. We build against Java 8, not against the Android version
of it.
Recent Android versions should have some bytecode "magic" that adds the
co-variant override in bytecode (see the commit referenced in [1]), but
on older Android version this problem may still occur. (Or perhaps the
"magic" is ineffective...) There are two pull requests on Github for
this problem, both from 2020, [2][3] while the Android commit [1] is
from March 2018. Apparently people still occasionally run into this
problem in the wild.
[1] 0e8b937ded/ojluni/src/main/java/java/util/concurrent/ConcurrentHashMap.java (1244)
[2] https://github.com/eclipse/jgit/pull/104
[3] https://github.com/eclipse/jgit/pull/100
Change-Id: I7c07e0cc59871cb7fe60795e22867827fa9c2458
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
[spotbugs] Don't use class from java.util.concurrent for locking
Use a dedicated Lock object to lock the scheduler in
RepositoryCache#configureEviction to fix spotbugs warning
JLM_JSR166_UTILCONCURRENT_MONITORENTER.
Change-Id: I003dcf0ed1a0a3f4eea5d8a2f51a07473d28a928
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Reorder modifiers to follow Java Language Specification
The Java Language Specification recommends listing modifiers in
the following order:
1. Annotations
2. public
3. protected
4. private
5. abstract
6. static
7. final
8. transient
9. volatile
10. synchronized
11. native
12. strictfp
Not following this convention has no technical impact, but will reduce
the code's readability because most developers are used to the standard
order.
This was detected using SonarLint.
Change-Id: I9cddecb4f4234dae1021b677e915be23d349a380
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
RepositoryCache: don't require HEAD in git repositories
Reftable-enabled repositories don't have a file called HEAD. Check for
reftable/ instead.
This fixes repository creation on reftable in Gerrit.
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I778c2be01d96aaf135affae4b457b5fe5b483bee
These are useful to avoid typos, and also for tab completion.
Change-Id: I0f2d267e46b36bc40297c9657c447f3fd8b9f831
Signed-off-by: David Turner <dturner@twosigma.com>
There is a possibility of hitting NPE on a logger if it is not the first
statically initialized member. For example, if another static
initializer creates an instance of its class and the logger is used
from the constructor.
Change-Id: I51fa855a8883c107f2e4ef5ac039dc12a571a7ae
Allow to use an external ExecutorService for background auto-gc
If set use the external executor, otherwise use JGit's own simple
WorkQueue. Move WorkQueue to an internal package so we can reuse it
without exposing it in the public API.
Change-Id: I060d62ffd6692362a88b4bf13ee07b0dc857abe9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Enable and fix warnings about redundant specification of type arguments
Since the introduction of generic type parameter inference in Java 7,
it's not necessary to explicitly specify the type of generic parameters.
Enable the warning in Eclipse, and fix all occurrences.
Change-Id: I9158caf1beca5e4980b6240ac401f3868520aad0
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The type parameters can now be inferred when creating
ConcurrentHashMap.
A for loop over the keys of a ConcurrentHashMap doesn't
need to use an Iterator<Map.Entry>; loop syntax handles
this just fine over keySet().
Change-Id: I1f85bb81b77f7cd1caec77197f2f0bf78e4a82a1
Now that RepositoryCache have a time based eviction strategy, get rid
of the strategy to evict cache entries if heap memory is running low,
i.e. soft references. Main reason why time based eviction was
implemented was to offer an alternative to the unpredictable soft
references.
Relying on soft references is not working, especially in large heap. The
JVM GC will consider collecting soft references as last resort before
throwing an out of memory error. For example, an application like Gerrit
configured with a 128GB heap, GC will wait until all 128GB is filled
before collecting the soft references so the application will be
suffering long pauses caused by GC for a long time already. In other
words, you will have to restart application because it's unusable before
JVM eviction kicks in.
Keeping the SoftReference in RepositoryCache is causing more harm than
good. If you use the time based eviction (which is the default strategy)
and want to tune JVM to release soft references more aggressively, it
will release repositories from the cache even though they are not
expired which defeats the purpose of the repository cache.
Gerrit uses Lucene library which uses soft references and this is
causing a "memory leak" except if you configure JVM to release soft
references more aggressively which have the nasty side effect of
evicting non expired repositories from the cache.
Change-Id: I9940bd800464c7f007696d0ccde52ea617b2ebce
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Unconditionally close repositories in RepositoryCache.clear()
Earlier we tried to close the repository before removing it from the
cache, so close only reduced refcount but didn't close it.
Now that we no longer leak usage count on purpose and the usage count is
now ignored anyway, there is no longer a need to run the removal twice.
Change-Id: I8b62cec6d8a3e88c096d1f37a1f7f5a5066c90a0
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix eviction of repositories with negative usage count
If the repository close method was called twice (or more) for one open,
the usage count became negative and the repository was never be evicted
from the cache because the method checking if repository is expired was
not considering negative usage count.
Change-Id: I18a80c415c54c37d1b9def2b311ff2d0afa455ca
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Unconditionally close repository in unregisterAndCloseRepository
Repository.close() method is used when reference counting and expiration
needs to be honored. The RepositoryCache.unregisterAndCloseRepository
method should close the repository unconditionally. This is also indicated
from its javadoc.
Change-Id: I19392d1eaa17f27ae44b55eea49dcff05a52f298
When Repository.close() decrements the useCount to 0 currently the cache
immediately evicts the repository from WindowCache and RepositoryCache.
This leads to I/O overhead on busy repositories because pack files and
references are inserted and deleted from the cache frequently.
This commit defers the eviction of a repository from the caches until
last use of the repository is older than time to live. The eviction is
handled by a background task running periodically.
Add two new configuration parameters:
* core.repositoryCacheExpireAfter: cache entries are evicted if the
cache entry wasn't accessed longer than this time in milliseconds
* core.repositoryCacheCleanupDelay: defines the interval in milliseconds
for running a background task evicting expired cache entries. If set to
-1 the delay is set to min(repositoryCacheExpireAfter, 10 minutes). If
set to 0 the time based eviction is switched off and no background task
is started. If time based eviction is switched off the JVM can still
evict cache entries if heap memory is running low.
Change-Id: I4a0214ad8b4a193985dda6a0ade63b70bdb948d7
Also-by: Matthias Sohn <matthias.sohn@sap.com>
Also-by: Hugo Arès <hugo.ares@ericsson.com>
Also-by: Sasa Zivkov <sasa.zivkov@sap.com>
RepositoryCache has 2 methods to remove a repository from the cache but
they are never called when a repository is closed. Users of the cache
were expected to call one of those 2 methods but how could they have
called them at proper time without having visibility of the repository
usage count.
Ideally, I would have reworked the RepositoryCache to wrap any
repository it opens in a class that would be responsible to unregister
them from the cache when it's really closed, i.e. when usage counter
reaches 0. The problem preventing the wrapping solution is the
RepositoryCache.register method that allows to register an already
opened repository in the cache. Such repositories cannot be wrapped
because callers are still holding a reference on the unwrapped
repository.
Document that RepositoryCache.close method is removing the repository
from the cache as well as closing it and rework
RepositoryCache.unregister method to only remove the repository from the
cache. Use the latter to unregister repository when Repository.doClose
is getting executed.
Change-Id: Ia364816e4da8d7b6cfa72f10758ca31aa8a1f9db
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repository has a usage counter that is initialized to 1 at
instantiation and this counter is decremented when Repository.close
method is called. There is also a Repository.incrementOpen method that
RepositoryCache uses to increment the usage count when it's returning a
repository that is already opened.
The problem was that RepositoryCache was incrementing the usage count
for repositories that it just opened or registered. The usage count was
2 when it should have been 1.
Incrementing usage count is now only be done for repository that are
served from the cache.
This bug is causing slow memory increase of our Gerrit server until the
server become slow. Even if the RepositoryCache is using SoftReference,
it seems that the JVM is not garbage collecting the repositories because
it's not yet on the edge of being out of memory.
To test this change, I replicated all repositories(11k) from Gerrit
master to one slave. The Gerrit master used memory after this test was
10GB without this change and 3.5GB with.
Change-Id: I86c7b36174e384f106b51fe92f306018fd1dbdf0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Add methods that allow to unregister repositories from the
RepositoryCache individually.
Bug: 470234
Change-Id: Ib918a634d829c9898072ae7bdeb22b099a32b1c9
Signed-off-by: Tobias Oberlies <tobias.oberlies@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
JGit 3.0: move internal classes into an internal subpackage
This breaks all existing callers once. Applications are not supposed
to build against the internal storage API unless they can accept API
churn and make necessary updates as versions change.
Change-Id: I2ab1327c202ef2003565e1b0770a583970e432e9
A few classes such as Constanrs are marked with @SuppressWarnings, as are
toString() methods with many liternal, but otherwise $NLS-n$ is used for
string containing text that should not be translated. A few literals may
fall into the gray zone, but mostly I've tried to only tag the obvious
ones.
Change-Id: I22e50a77e2bf9e0b842a66bdf674e8fa1692f590
Move FileRepository to storage.file.FileRepository
This move isolates all of the local file specific implementation code
into a single package, where their package-private methods and support
classes are properly hidden away from the rest of the core library.
Because of the sheer number of files impacted, I have limited this
change to only the renames and the updated imports.
Change-Id: Icca4884e1a418f83f8b617d0c4c78b73d8a4bd17
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some types of repositories might not be stored on local disk. For
these, they will most likely return null for getDirectory() as the
java.io.File type cannot describe where their storage is, its not
in the host's filesystem.
Document that getDirectory() can return null now, and update all
current non-test callers in JGit that might run into problems on
such repositories. For the most part, just act like its bare.
Change-Id: I061236a691372a267fd7d41f0550650e165d2066
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Make lib.Repository abstract and lib.FileRepository its implementation
To support other storage models other than just the local filesystem,
we split the Repository class into a nearly abstract interface and
then create a concrete subclass called FileRepository with the file
based IO implementation.
We are using an abstract class for Repository rather than the much
more generic interface, as implementers will want to inherit a large
array of utility functions, such as resolve(String). Having these in
a base class makes it easy to inherit them.
This isn't the final home for lib.FileRepository. Future changes
will rename it into storage.file.FileRepository, but to do that we
need to also move a number of other related class, which we aren't
quite ready to do.
Change-Id: I1bd54ea0500337799a8e792874c272eb14d555f7
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
On Windows, FS_Win32_Cygwin has been used if a Cygwin Git installation
is present in the PATH. Assuming that the user works with the Cygwin
Git installation may result in unnecessary overhead if he actually
does not.
Applications built on top of jgit may have more knowledge on the
actually used Git client (Cygwin or not) and hence should be able to
configure which FS to use accordingly.
Change-Id: Ifc4278078b298781d55cf5421e9647a21fa5db24
Introduce a named constant for the .git directory.
Not all occurrences of ".git" are replaced by this constant, only
those where it actually refers to the directory with that name, i.e
not the ".git" directory suffix.
Asserts and comment are also excluded from replacement.
Change-Id: I65a9da89aedd53817f2ea3eaab4f9c2bed35d7ee
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Move pure IO utility functions to a utility class of its own.
According the javadoc, and implied by the name of the class, NB
is about network byte order. The purpose of moving the IO only,
and non-byte order related functions to another class is to
make it easier for new contributors to understand that they
can use these functions in general and it's also makes it easier
to understand where to put new IO related utility functions
Change-Id: I4a9f6b39d5564bc8a694b366e7ff3cc758c5181b
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Per CQ 3448 this is the initial contribution of the JGit project
to eclipse.org. It is derived from the historical JGit repository
at commit 3a2dd9921c.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Thomas Wolf
Matthias Sohn
David Pursehouse
Han-Wen Nienhuys
David Turner
Carsten Hammer
Saša Živkov
Shawn Pearce
Hugo Arès
Christian Halstrick
Tobias Oberlies
Robin Rosenberg
Marc Strapetz
Git Development Community