Unconditionally close repository in unregisterAndCloseRepository
Repository.close() method is used when reference counting and expiration
needs to be honored. The RepositoryCache.unregisterAndCloseRepository
method should close the repository unconditionally. This is also indicated
from its javadoc.
Change-Id: I19392d1eaa17f27ae44b55eea49dcff05a52f298
Add configuration parameter to enable built-in hooks/filters
If the configuration parameter filter.<filterDriverName>.useJGitBuiltin
is set to true then for all corresponding filters JGit will try to
execute the built-in filter instead of the filter-command which is
defined in git configuration. It will fallback to the non-built-in
filters if no built-in filters are registered or if constructing them
leads to exceptions. If set to false JGit will not try to execute
built-in filters for the specified filter driver.
Example: The configuration contains the following lines
[filter "lfs"]
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
useJGitBuiltin = true
Addtionally the .gitattributes file in the root of the working tree
contains:
*.bin filter=lfs
In this case when new content is added similar to "git add 1.bin" then
the following will happen:
- jgit will check whether a built-in command factory was registered
for the command "jgit://builtin/lfs/clean". If that is true the
factory is used to create a built-in filter command and that
command is used to filter the content
- Otherwise jgit will call the external program "git lfs clean ..."
to do the filtering
Change-Id: Idadb1db06b1e89e7031d7ed6319904973c367d38
JGit supports clean filters defined in repository configuration. The
filters are implemented as external programs filtering content by
accepting the original content (as seen in the working tree) on stdin
and which emit the filtered content on stdout. To run such a filter JGit
has to start an external process and pump data into/from this process.
This commit adds support for clean filters which are implemented
in Java and which are executed by jgit's main thread. When a filter is
defined in the configuration as
"jgit://builtin/<filterDriverName>/clean" then JGit will lookup in a
static map whether a filter is registered under this name. If found
such a filter is called to do the filtering.
The functionality in this commit requires that a program using JGit
explicitly calls the JGit API to register built-in implementations for
specific clean filters. In follow-up commits configuration parameters
will be added which trigger such registrations. Other commits will add
implementations for lfs filters.
Change-Id: I0344d3c54801c9a46e5a606c5df17e5f2e17b2be
BranchConfig treated this config property as a boolean, but git also
allows the values "preserve" and "interactive". Config property
pull.rebase also allows the same values.
Replace private enum PullCommand.PullRebaseMode by new public enum
BranchConfig.BranchRebaseMode and adapt all uses. Add a new setter to
PullCommand.
Note: PullCommand will treat "interactive" like "true", i.e., as a
non-interactive rebase. Not sure how "interactive" should be handled.
At least it won't balk on it.
Bug: 499482
Change-Id: I7309360f5662b2c2efa1bd8ea6f112c63cf064af
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
When doing a detaching operation, JGit fakes a SymbolicRef as an
ObjectIdRef. This is because RefUpdate#updateImpl dereferences the
SymbolicRef when updating it. For example, assume that HEAD is
pointing to refs/heads/master. If I try to make a detached HEAD
pointing to a commit c0ffee, RefUpdate dereferences HEAD as
refs/heads/master first and changes refs/heads/master to c0ffee. The
detach argument of RefDatabase#newUpdate avoids this dereference by
faking HEAD as ObjectIdRef.
This faking is problematic for the linking operation of
DfsRefDatabase. It does a compare-and-swap operation on every
reference change because of its distributed systems nature. If a
SymbolicRef is faked as an ObjectRef, it thinks that there is a
racing change in the reference and rejects the update. Because of
this, DFS based repositories cannot change the link target of symbolic
refs. This has not been a problem for file-based repositories because
they have a file-lock based semantics instead of the CAS based one.
The reference implementation, InMemoryRepository, is not affected
because it only compares ObjectIds.
When [1] introduced this faking code, there was no way for RefUpdate
to distinguish the detaching operation. When [2] fixed the detaching
operation, it introduced a detachingSymbolicRef flag. This commit uses
this flag to control whether it needs to dereference the symbolic refs
by calling Ref#getLeaf. The same flag is used in the reflog update
operation.
This commit does not affect any operation that succeeds currently. In
some DFS repository implementations, this fixes a ref linking
operation, which is currently failing.
[1]: 01b5392cdb
[2]: 3a86868c08
Change-Id: I118f85f0414dbfad02250944e28d74dddd59469b
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Repository: Log negative useCnt message together with stack trace
The message "close() called when useCnt is already zero" is logged with
level warning, and then if debug logging is enabled, the stack trace is
logged separately with level debug.
Log the message and the stack trace in the same call, so that they always
appear together in the output rather than potentially interleaved with
other log statements.
Change-Id: I1b5c1557ddc2d19f3f5b29baec96e62bc467d88a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Example usage:
$ ./jgit push \
--push-option "Reviewer=j.doe@example.org" \
--push-option "<arbitrary string>" \
origin HEAD:refs/for/master
Stefan Beller has also made an equivalent change to CGit:
http://thread.gmane.org/gmane.comp.version-control.git/299872
Change-Id: I6797e50681054dce3bd179e80b731aef5e200d77
Signed-off-by: Dan Wang <dwwang@google.com>
This variable has been populated and never used since it was
introduced in commit 5cf53fdacf
(Speed up clone/fetch with large number of refs, 2013-02-18).
Noted by FindBugs:
"BatchRefUpdate.java:359, UC_USELESS_OBJECT, Priority: Normal"
Change-Id: I7aacb49540aaee4a83db3d38b15633bb6c4773d0
Signed-off-by: Dan Wang <dwwang@google.com>
We observe in Gerrit 2.12 that useCnt can become negative in rare cases.
Log this to help finding the bug.
Change-Id: Ie91c7f9d190a5d7cf4733d4bf84124d119ca20f7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When Repository.close() decrements the useCount to 0 currently the cache
immediately evicts the repository from WindowCache and RepositoryCache.
This leads to I/O overhead on busy repositories because pack files and
references are inserted and deleted from the cache frequently.
This commit defers the eviction of a repository from the caches until
last use of the repository is older than time to live. The eviction is
handled by a background task running periodically.
Add two new configuration parameters:
* core.repositoryCacheExpireAfter: cache entries are evicted if the
cache entry wasn't accessed longer than this time in milliseconds
* core.repositoryCacheCleanupDelay: defines the interval in milliseconds
for running a background task evicting expired cache entries. If set to
-1 the delay is set to min(repositoryCacheExpireAfter, 10 minutes). If
set to 0 the time based eviction is switched off and no background task
is started. If time based eviction is switched off the JVM can still
evict cache entries if heap memory is running low.
Change-Id: I4a0214ad8b4a193985dda6a0ade63b70bdb948d7
Also-by: Matthias Sohn <matthias.sohn@sap.com>
Also-by: Hugo Arès <hugo.ares@ericsson.com>
Also-by: Sasa Zivkov <sasa.zivkov@sap.com>
Config load should not fail on unsupported or nonexistent include path
1f86350 added initial support for include.path. Relative path and path
with tilde are not yet supported but config load was failing if one of
those 2 unsupported options was encountered. Another problem was that
config load was failing if the include.path file did not exist.
Change the behavior to be consistent with native git. Ignore unsupported
or nonexistent include.path.
Bug: 495505
Bug: 496732
Change-Id: I7285d0e7abb6389ba6983e9c46021bea4344af68
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Support git config [include] section with absolute path(s)
As per [1], but limited to absolute paths indeed. No support yet for
tilde or $HOME expansion. Support for the --[no-]includes options
([1]) is not part of this commit scope either, but those options'
defaults are in effect as described in [1].
[1] https://git-scm.com/docs/git-config
Included path can be a config file that includes other path-s in turn.
An exception is thrown if too many recursions (circular includes)
happen because of ill-specified config files.
Change-Id: I700bd7b7e1625eb7de0180f220c707d8e7b0930b
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix config value get to return last instead of 1st just like git
Before this fix, getting the value of 'key' below used to return
value1. This fix makes it so that value3 gets returned instead,
just like native git's get.
[section]
key = value1
key = value2
key = value3
Change-Id: Iccb24de9b63c3ad8646c909494ca3f8c9ed6e29c
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Remove UTF-8 checking duplication in Config lib subclasses
Change-Id: Ib9f0ae8207000a36c5bf1a92fcc2c32efc4c0984
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
PersonIdent: Strip some special chars from external strings
The special characters <> and '\n' interfere with parsing of
identities. C git strips these special characters, so we should too.
Rather than allocating extra strings by calling String#trim(), add a
few lines to our sanitization method to perform the same trimming as
described in String's Javadoc.
Change-Id: I96edcb93a2fc194ee354d60566d352299742a52f
PersonIdent: Document that name and email aren't trimmed
This might be somewhat surprising behavior to users who might
naturally assume the following invariant:
ident.equals(parseIdent(ident.toExternalString()))
This invariant does not hold since whitespace is only trimmed during
serialization. We don't want to mess with the strings during
initialization, as this is called during the highly-optimized commit
parsing codepath.
Change-Id: I081a603f0ac0e33167462244779b0ff3ad51e80c
Expose the ObjectInserter that created an ObjectReader
We've found in Gerrit Code Review that it is common to pass around
both an ObjectReader (or more commonly a RevWalk wrapping one) and an
ObjectInserter. These code paths often assume that the ObjectReader
can read back any objects created by the ObjectInserter without
flushing. However, we previously had no way to enforce that constraint
programmatically, leading to hard-to-spot problems.
Provide a solution by exposing the ObjectInserter that created an
ObjectReader, when known. Callers can either continue passing both
objects and check:
reader.getCreatedFromInserter() == inserter
or they can just pass around ObjectReader and extract the inserter
when it's needed (checking that it's not null at usage time).
Change-Id: Ibbf5d1968b506f6b47030ab1b046ffccb47352ea
Repurpose RefDatabase#performsAtomicTransactions() slightly, to
indicate that the backend _supports_ atomic transactions, rather than
the current definition, which is that the backend always _uses_ atomic
transactions regardless of whether or not the caller actually wants
them. Allow BatchRefUpdate callers to turn off atomic transactions by
calling setAtomic(false). Defaulting to true means this is backwards
compatible.
Change-Id: I6df78d7df65ab147b4cce7764bd3101db985491c
RepositoryCache has 2 methods to remove a repository from the cache but
they are never called when a repository is closed. Users of the cache
were expected to call one of those 2 methods but how could they have
called them at proper time without having visibility of the repository
usage count.
Ideally, I would have reworked the RepositoryCache to wrap any
repository it opens in a class that would be responsible to unregister
them from the cache when it's really closed, i.e. when usage counter
reaches 0. The problem preventing the wrapping solution is the
RepositoryCache.register method that allows to register an already
opened repository in the cache. Such repositories cannot be wrapped
because callers are still holding a reference on the unwrapped
repository.
Document that RepositoryCache.close method is removing the repository
from the cache as well as closing it and rework
RepositoryCache.unregister method to only remove the repository from the
cache. Use the latter to unregister repository when Repository.doClose
is getting executed.
Change-Id: Ia364816e4da8d7b6cfa72f10758ca31aa8a1f9db
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repository has a usage counter that is initialized to 1 at
instantiation and this counter is decremented when Repository.close
method is called. There is also a Repository.incrementOpen method that
RepositoryCache uses to increment the usage count when it's returning a
repository that is already opened.
The problem was that RepositoryCache was incrementing the usage count
for repositories that it just opened or registered. The usage count was
2 when it should have been 1.
Incrementing usage count is now only be done for repository that are
served from the cache.
This bug is causing slow memory increase of our Gerrit server until the
server become slow. Even if the RepositoryCache is using SoftReference,
it seems that the JVM is not garbage collecting the repositories because
it's not yet on the edge of being out of memory.
To test this change, I replicated all repositories(11k) from Gerrit
master to one slave. The Gerrit master used memory after this test was
10GB without this change and 3.5GB with.
Change-Id: I86c7b36174e384f106b51fe92f306018fd1dbdf0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Implement the DIR_NO_GITLINKS setting with the same functionality
it provides in cGit.
Bug: 436200
Change-Id: I8304e42df2d7e8d7925f515805e075a92ff6ce28
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>
Add config parameter gc.prunePackExpire for packfile expiration
JGit's Garbage Collector is repacking relevant objects into new
packfiles and is afterwards deleting the now obsolete packfiles. But to
prevent problems caused by race conditions JGit was not deleting
packfiles when they are too young. The same mechanism as for loose
objects and the config parameter gc.pruneExpire was used.
But JGit was reusing the parameter gc.pruneExpire also for packfiles
which may cause a lot of filesystem consumption if gc.pruneExpire was
set to the default of 2 weeks. Only two weeks after packfile creation gc
was allowed to delete this packfile.
This change introduces a new config paramter gc.prunePackExpire with a
default of "1.hour". This parameter is used when packfiles are deleted.
Only packfiles older than the specified time can be deleted.
For loose objects the behaviour is not changed and only the old
parameter gc.pruneExpire is relevant.
Change-Id: I6209efb05678b15153bd22479dc13486907a44f8
TreeWalk provides the new method getEolStreamType. This new method can
be used with EolStreamTypeUtil in order to create a wrapped InputStream
or OutputStream when reading / writing files. The implementation
implements support for the git configuration options core.crlf, core.eol
and the .gitattributes "text", "eol" and "binary"
CQ: 10896
Bug: 486563
Change-Id: Ie4f6367afc2a6aec1de56faf95120fff0339a358
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Remove deprecated Tree, TreeEntry, FileTreeEntry and friends
These types were deprecated in 0.9.1 (aka 384a19eee0).
If anyone is still using them, its time to stop.
Change-Id: I3f73347ba78c639e0c6a504812bc1a0702f829b1
Paths.pathCompare: Utility to sort paths from byte[]
Consolidate copies of this function into one location.
Add some unit tests to prevent bugs that were accidentally
introduced while trying to make this refactoring.
Change-Id: I82f64bbb8601ca2d8316ca57ae8119df32bb5c08
A group of updates can be applied by updating the tree in one step,
writing out a new root tree, and storing its SHA-1. If references
are stored in RefTrees, comparing two repositories is a matter of
checking if two SHA-1s are identical. Without RefTrees comparing two
repositories requires listing all references and comparing the sets.
Track the "refs/" directory as a root tree by storing references
that point directly at an object as a GITLINK entry in the tree.
For example "refs/heads/master" is written as "heads/master".
Annotated tags also store their peeled value with ^{} suffix, using
"tags/v1.0" and "tags/v1.0^{}" GITLINK entries.
Symbolic references are written as SYMLINK entries with the blob of
the symlink carrying the name of the symbolic reference target.
HEAD is outside of "refs/" namespace so it is stored as a special
"..HEAD" entry. This name is chosen because ".." is not valid in
a reference name and it almost looks like "../HEAD" which names
HEAD if the reader was inside of the "refs/" directory.
A new Command type is required to handle symbolic references and
peeled references.
Change-Id: Id47e5d4d32149a9e500854147edd7d93c1041a39
Accept some of the same section keys that fsck does in git-core,
allowing repositories to skip over specific kinds of acceptable
broken objects, e.g.:
[fsck]
duplicateEntries = ignore
zeroPaddedFilemode = ignore
The zeroPaddedFilemode = ignore is a synonym for the JGit specific
allowLeadingZeroFileMode = true. Only accept the JGit key if git-core
key was not specified.
Change-Id: Idaed9310e2a5ce5511670ead1aaea2b30aac903c
Some ancient objects may be broken, but in a relatively harmless way.
Allow the ObjectChecker caller to whitelist specific objects that are
going to fail checks, but that have been reviewed by a human and decided
the objects are OK enough to permit continued use of.
This avoids needing to rewrite history to scrub the broken objects out.
Honor the git-core fsck.skipList configuration setting when receiving a
push or fetching from a remote repository.
Change-Id: I62bd7c0b0848981f73dd7c752860fd02794233a6
Base Java version for JGit is now Java 7. The java.text.Normalizer
class was available in Java 6. Reflection is no longer required to
normalize strings for Mac OS X.
Change-Id: I98e14b72629a7a729a2d40a3aa275932841274e8
Hoist ObjectIdSet up to lib as part of the public API and add
the interface to some common types like PackIndex and JGit custom
ObjectId map types. This cleans up wrapper code in a number of
places by allowing direct use of the types as an ObjectIdSet.
Future commits can now rely on ObjectIdSet as a simple read-only
type to check a set of objects from a number of storage options.
Change-Id: Ib62b062421d475bd52abd6c84a73916ef36e084b
Null-annotated Ref class and fixed related compiler errors
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Change-Id: I24c8a600ec962d61d5f40abf73eac4203e115240
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
In I1f5dc07182dbf6bba2a9f4807fdd25b475da4ead, FileBasedConfig got
support for reading a configuration with UTF8 BOM. Apply the same
support to BlobBasedConfig, to make SubmoduleWalk able to parse
.gitmodules configurations with BOM.
Change-Id: I25b5474779952fe2c076180b96fc2869eef190a8
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
If defined in .gitattributes call smudge filter during checkout.
To support checkout where current HEAD,index do not contain attributes
we need to also consider attributes from the tree we checkout. Therefore
CanonicalTreeParser has to learn how to provide attributes.
Change-Id: I168fdb81a8e1a9f991587b3e95a36550ea845f0a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When filters are defined for certain paths in gitattributes make
sure that clean filters are processed when adding new content to the
object database.
Change-Id: Iffd72914cec5b434ba4d0de232e285b7492db868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add the new class Attributes holding multiple Attribute(s)
Attributes represents a semantic collector of Attribute(s) and replaces
the anonymous Map<String,Attribute>. This class will be returned by
TreeWalk.getAttributes(). It offers convenient access to the attributes
wrapped in the Attributes object. Adds preparations for a future
Attribute Macro Expansion
Change-Id: I8348c8c457a2a7f1f0c48050e10399b0fa1cdbe1
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Repository: Introduce exactRef and findRef, deprecate getRef
The Repository class provides only one method to look up a ref by
name, getRef. If I request refs/heads/master and that ref does not
exist, getRef will look further in the search path:
ref/refs/heads/master
refs/heads/refs/heads/master
refs/remotes/refs/heads/master
This behavior is counterintuitive, needlessly inexpensive, and usually
not what the caller expects.
Allow callers to specify whether to use the search path by providing
two separate methods:
- exactRef, which looks up a ref when its exact name is known
- findRef, which looks for a ref along the search path
For backward compatibility, keep getRef as a deprecated synonym for
findRef.
This change introduces findRef and exactRef but does not update
callers outside tests to use them yet.
Change-Id: I35375d942baeb3ded15520388f8ebb9c0cc86f8c
Signed-off-by: Jonathan Nieder <jrn@google.com>