RepositoryCache has 2 methods to remove a repository from the cache but
they are never called when a repository is closed. Users of the cache
were expected to call one of those 2 methods but how could they have
called them at proper time without having visibility of the repository
usage count.
Ideally, I would have reworked the RepositoryCache to wrap any
repository it opens in a class that would be responsible to unregister
them from the cache when it's really closed, i.e. when usage counter
reaches 0. The problem preventing the wrapping solution is the
RepositoryCache.register method that allows to register an already
opened repository in the cache. Such repositories cannot be wrapped
because callers are still holding a reference on the unwrapped
repository.
Document that RepositoryCache.close method is removing the repository
from the cache as well as closing it and rework
RepositoryCache.unregister method to only remove the repository from the
cache. Use the latter to unregister repository when Repository.doClose
is getting executed.
Change-Id: Ia364816e4da8d7b6cfa72f10758ca31aa8a1f9db
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repository has a usage counter that is initialized to 1 at
instantiation and this counter is decremented when Repository.close
method is called. There is also a Repository.incrementOpen method that
RepositoryCache uses to increment the usage count when it's returning a
repository that is already opened.
The problem was that RepositoryCache was incrementing the usage count
for repositories that it just opened or registered. The usage count was
2 when it should have been 1.
Incrementing usage count is now only be done for repository that are
served from the cache.
This bug is causing slow memory increase of our Gerrit server until the
server become slow. Even if the RepositoryCache is using SoftReference,
it seems that the JVM is not garbage collecting the repositories because
it's not yet on the edge of being out of memory.
To test this change, I replicated all repositories(11k) from Gerrit
master to one slave. The Gerrit master used memory after this test was
10GB without this change and 3.5GB with.
Change-Id: I86c7b36174e384f106b51fe92f306018fd1dbdf0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
TreeWalk provides the new method getEolStreamType. This new method can
be used with EolStreamTypeUtil in order to create a wrapped InputStream
or OutputStream when reading / writing files. The implementation
implements support for the git configuration options core.crlf, core.eol
and the .gitattributes "text", "eol" and "binary"
CQ: 10896
Bug: 486563
Change-Id: Ie4f6367afc2a6aec1de56faf95120fff0339a358
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
ReflogConfigTest: refactor commit method to avoid variable hiding
The author and committer parameters were hiding class members of
the same name.
Since the passed in PersonIdent instances were being created from
the class members anyway, remove the parameters and instead create
the PersonIdent instances inline in the method.
Change-Id: I66b057df388835d57f332fdcbadb8a9f4e1094a4
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Remove deprecated Tree, TreeEntry, FileTreeEntry and friends
These types were deprecated in 0.9.1 (aka 384a19eee0).
If anyone is still using them, its time to stop.
Change-Id: I3f73347ba78c639e0c6a504812bc1a0702f829b1
Paths.pathCompare: Utility to sort paths from byte[]
Consolidate copies of this function into one location.
Add some unit tests to prevent bugs that were accidentally
introduced while trying to make this refactoring.
Change-Id: I82f64bbb8601ca2d8316ca57ae8119df32bb5c08
Accept some of the same section keys that fsck does in git-core,
allowing repositories to skip over specific kinds of acceptable
broken objects, e.g.:
[fsck]
duplicateEntries = ignore
zeroPaddedFilemode = ignore
The zeroPaddedFilemode = ignore is a synonym for the JGit specific
allowLeadingZeroFileMode = true. Only accept the JGit key if git-core
key was not specified.
Change-Id: Idaed9310e2a5ce5511670ead1aaea2b30aac903c
Some ancient objects may be broken, but in a relatively harmless way.
Allow the ObjectChecker caller to whitelist specific objects that are
going to fail checks, but that have been reviewed by a human and decided
the objects are OK enough to permit continued use of.
This avoids needing to rewrite history to scrub the broken objects out.
Honor the git-core fsck.skipList configuration setting when receiving a
push or fetching from a remote repository.
Change-Id: I62bd7c0b0848981f73dd7c752860fd02794233a6
DirCacheEditor: Replace file-with-tree and tree-with-file
If a PathEdit tries to store a file where a subtree was, or a subtree
where a file was, replace the entry in the DirCache with the new
name(s). This supports switching between file and tree entry types
using a DirCacheEditor.
Add new unit tests to cover the conditions where these can happen.
Change-Id: Ie843d9388825f9e3d918a5666aa04e47cd6306e7
FileTreeIterator was calling by mistake
WorkingTreeIterator.idSubmodule(Entry). Instead it should always compute
idSubmodule on its own.
Change-Id: Id1b988aded06939b1d7edd2671e34bf756896c0e
If defined in .gitattributes call smudge filter during checkout.
To support checkout where current HEAD,index do not contain attributes
we need to also consider attributes from the tree we checkout. Therefore
CanonicalTreeParser has to learn how to provide attributes.
Change-Id: I168fdb81a8e1a9f991587b3e95a36550ea845f0a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repository: Introduce exactRef and findRef, deprecate getRef
The Repository class provides only one method to look up a ref by
name, getRef. If I request refs/heads/master and that ref does not
exist, getRef will look further in the search path:
ref/refs/heads/master
refs/heads/refs/heads/master
refs/remotes/refs/heads/master
This behavior is counterintuitive, needlessly inexpensive, and usually
not what the caller expects.
Allow callers to specify whether to use the search path by providing
two separate methods:
- exactRef, which looks up a ref when its exact name is known
- findRef, which looks for a ref along the search path
For backward compatibility, keep getRef as a deprecated synonym for
findRef.
This change introduces findRef and exactRef but does not update
callers outside tests to use them yet.
Change-Id: I35375d942baeb3ded15520388f8ebb9c0cc86f8c
Signed-off-by: Jonathan Nieder <jrn@google.com>
Let FS_Win32_Cygwin detect symlink support by creating temporary symlink
The class FS_Win32 was always trying out to create a temporary symlink
in order to find out whether symlinks are supported. FS_Win32_Cygwin was
overwriting this method and always returned true. But when the user
running JGit does not have administrative rights then the creation of
symlinks is forbidden even if he is running on FS_Win32_Cygwin. A lot of
tests failed only on the Windows platform because of this. It was
correctly detected that FS_Win32_Cygwin is the filesystem abstraction to
be used but creation of symlinks always failed because of lacking
privileges of the user running the tests.
This fix teaches FS_Win32_Cygwin to behave like FS_Win32 and to test
whether symlinks can be created in order to find out whether symlinks
are supported.
Change-Id: Ie2394631ffc4c489bd37c3ec142ed44bbfcac726
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Delete non empty directories before checkout a path
If the checkout path is currently a non-empty directory (and was a link
or a regular file before), this directory will be removed before
performing checkout, but only if the checkout path is specified.
Bug: 474973
Change-Id: Ifc6c61592d9b54d26c66367163acdebea369145c
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Add utility method allowing to check for empty folders in workdir
Previously the method DirCacheCheckoutTest#assertWorkDir() silently
skipped over empty folders. If tests would have left unexpected empty
folders in the worktree this would be overlooked. Now empty folders have
to be specified by something like mkmap("<foldername>", "/", ...]
Change-Id: Idb8b270e92daf02ecdc381d148a5958bd83ec057
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix the unit tests to not do boxing by using assertEquals(int, int)
instead of assertThat with a matcher.
Change-Id: I5412fe2f72c8ea0227b9ff3a3352ccb555e22231
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Throw InvalidObjectIdException from ObjectId.fromString("tooshort")
ObjectId.fromString already throws InvalidObjectIdException for most
malformed object ids, but for this kind it previously threw
IllegalArgumentException. Since InvalidObjectIdException is a child of
IllegalArgumentException, callers that catch IllegalArgumentException
will continue to work.
Change-Id: I24e1422d51607c86a1cb816a495703279e461f01
Signed-off-by: Jonathan Nieder <jrn@google.com>
Add methods that allow to unregister repositories from the
RepositoryCache individually.
Bug: 470234
Change-Id: Ib918a634d829c9898072ae7bdeb22b099a32b1c9
Signed-off-by: Tobias Oberlies <tobias.oberlies@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Revert "Config: Distinguish between empty and null strings"
This reverts commit 96eb3ee3976e7e9e3e118851fa614cce8a1f7d88, which
broke Gerrit tests that set a config value to 'null', serialize the
result, deserialize, and expect 'null' from Config.getString[1].
The intent of that commit was to make it possible to distinguish between
an absent and an empty config value, which we'll have to do with a new
method.
Revert the behavior change. Keep the tests from 428cb23f2de8, since
they test the behavior more precisely than the old tests did.
[1] https://gerrit-review.googlesource.com/68452
Change-Id: Ie8042f380ea0e34e3203e1991aa0feb2e6e44641
Signed-off-by: Jonathan Nieder <jrn@google.com>
Introduce exactRef to read a ref whose exact name is known
Unlike getRef(name), the new exactRef method does not walk the search
path. This should produce a less confusing result than getRef when the
exact ref name is known: it will not try to resolve refs/foo/bar to
refs/heads/refs/foo/bar even when refs/foo/bar does not exist.
It can be faster than both getRefs(ALL).get(name) and getRef(name)
because it only needs to examine a single ref.
A follow-up change will introduce a findRef synonym to getRef and
deprecate getRef to make the choice a caller is making more obvious
(exactRef or findRef, with the same semantics as getRefs(ALL).get and
getRefs(ALL).findRef).
Change-Id: If1bd09bcfc9919e7976a4d77f13184ea58dcda52
Signed-off-by: Jonathan Nieder <jrn@google.com>
Config: Allow ending a file with "=" and no newline
This is a perfectly valid construction according to C git:
$ echo -en '[a]\nx =' > foo.config
$ git config -f foo.config a.x; echo $?
0
Change-Id: Icfcf8304adb43c79e2b8b998f8d651b2a94f6acb
Config: Distinguish between empty and null strings
The C git API and command line tools distinguish between a key having
the empty string as a value and no key being present in the config
file:
$ echo -e '[a]\nx =' > foo.config
$ git config -f foo.config a.x; echo $?
0
$ git config -f foo.config a.y; echo $?
1
Make JGit make the same distinction. This is in line with the current
Javadoc of getString, which claims to return "a String value from the
config, null if not found". It is more reasonable to interpret "x ="
in the above example as "found" rather than "missing".
We need to maintain the special handling of a key name with no "="
resolving to a boolean true, but "=" with an empty string is still not
a valid boolean.
Change-Id: If0dbb7470c524259de0b167148db87f81be2d04a
Add fsck.allowInvalidPersonIdent to accept invalid author/committers
A larger than expected number of real-world repositories found on
the Internet contain invalid author, committer and tagger lines
in their history. Many of these seem to be caused by users misusing
the user.name and user.email fields, e.g.:
[user]
name = Au Thor <author@example.com>
email = author@example.com
that some version of Git (or a reimplementation thereof) copied
directly into the object header. These headers are not valid and
are rejected by a strict fsck, making it impossible to transfer
the repository with JGit/EGit.
Another form is an invalid committer line with double negative for
the time zone, e.g.
committer Au Thor <a@b> 1288373970 --700
The real world is messy. :(
Allow callers and users to weaken the fsck settings to accept these
sorts of breakages if they really want to work on a repo that has
broken history. Most routines will still function fine, however
commit timestamp sorting in RevWalk may become confused by a corrupt
committer line and sort commits out of order. This is mostly fine if
the corrupted chain is shorter than the slop window.
Change-Id: I6d529542c765c131de590f4f7ef8e7c1c8cb9db9
Merge bundle org.eclipse.jgit.java7 into org.eclipse.jgit
As we moved minimum Java version to 7 we don't need a separate bundle
and feature for JGit features depending on Java 7 anymore.
Change-Id: Ib5da61b0886ddbdea65298f1e8c6d65c9879ced1
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
According to [1] user name and email are taken first from the
environment variables:
GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL
GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL
In case (some of) these environment variables are not set, the
information is taken from the git configuration.
JGit doesn not yet support the environment variables GIT_AUTHOR_DATE and
GIT_COMMITTER_DATE.
[1] https://www.kernel.org/pub/software/scm/git/docs/git-commit-tree.html#_commit_information
Bug: 460586
Change-Id: I3ba582b4ae13674cf319652b5b13ebcbb96dd8ec
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Trim author/committer name and email in commit header
C Git trims name and email before inserting them into the commit header
so that " A U Thor " and " author@example.com " becomes
"A U Thor <author@example.com>" with a single separating space.
This changes PersonIdent#toExternalString() to trim name and email
before concatenating them.
Change-Id: Idd77b659d0db957626824f6632e2da38d7731625
Signed-off-by: Rüdiger Herrmann <ruediger.herrmann@gmx.de>
ObjectChecker: Disallow names potentially mapping to ".git" on HFS+
Mac's HFS+ folds concatentations of ".git" and ignorable Unicode
characters [1] to ".git" [2]. Hence we need to disallow all names which
could potentially be a shortname for ".git". Example: in an empty
directory create a folder ".g\U+200Cit". Now you can't create another
folder ".git".
The following characters are ignorable Unicode which are ignored on
HFS+:
unicode hex name
-------------------------------------------------
U+200C 0xe2808c ZERO WIDTH NON-JOINER
U+200D 0xe2808d ZERO WIDTH JOINER
U+200E 0xe2808e LEFT-TO-RIGHT MARK
U+200F 0xe2808f RIGHT-TO-LEFT MARK
U+202A 0xe280aa LEFT-TO-RIGHT EMBEDDING
U+202B 0xe280ab RIGHT-TO-LEFT EMBEDDING
U+202C 0xe280ac POP DIRECTIONAL FORMATTING
U+202D 0xe280ad LEFT-TO-RIGHT OVERRIDE
U+202E 0xe280ae RIGHT-TO-LEFT OVERRIDE
U+206A 0xe281aa INHIBIT SYMMETRIC SWAPPING
U+206B 0xe281ab ACTIVATE SYMMETRIC SWAPPING
U+206C 0xe281ac INHIBIT ARABIC FORM SHAPING
U+206D 0xe281ad ACTIVATE ARABIC FORM SHAPING
U+206E 0xe281ae NATIONAL DIGIT SHAPES
U+206F 0xe281af NOMINAL DIGIT SHAPES
U+FEFF 0xefbbbf ZERO WIDTH NO-BREAK SPACE
[1] http://www.unicode.org/versions/Unicode7.0.0/ch05.pdf#G40025http://www.unicode.org/reports/tr31/#Layout_and_Format_Control_Characters
[2] http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.html#UnicodeSubtleties
Change-Id: Ib6a1dd090b2649bdd8ec16387c994ed29de2860d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Windows creates shortnames for all non-8.3 files (see [1]). Hence we
need to disallow all names which could potentially be a shortname for
".git". Example: in an empty directory create a folder "GIT~1". Now you
can't create another folder ".git".
The path "GIT~1" may map to ".git" on Windows. A potential victim to
such an attack first has to initialize a git repository in order to
receive any git commits. Hence the .git folder created by init will get
the shortname "GIT~1". ".git" will only get a different shortname if the
user has created a file "GIT~1" before initialization of the git
repository.
[1] http://en.wikipedia.org/wiki/8.3_filename
Change-Id: I9978ab8f2d2951c46c1b9bbde57986d64d26b9b2
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Windows treats "foo." and "foo " as "foo". The ".git" directory is
special, as it contains metadata for a local Git repository. Disallow
variations that Windows considers to be the same.
Change-Id: I28eb48859a95a89111b4987c91de97557e3bb539
Always ignore case when forbidding .git in ObjectChecker
The component name ".GIT" inside a tree entry could confuse a
case insensitive filesystem into looking at a submodule and
not a directory entry.
Disallow any case permutations of ".git" to prevent this
confusion from entering a repository and showing up at a
later date on a case insensitive system.
Change-Id: Iaa3f768931d0d5764bf07ac5f6f3ff2b1fdda01b
Without explicitly closing repos we can't delete the test repositories
on windows.
Change-Id: Id5fa17bd764cbf28703c2f21639d7e969289c2d6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Make sure checkout doesn't report conflicts on ignored paths
In a situation where a certain path was ignored but a working tree file
with this path existed jgit didn't allow to checkout a branch which
didn't ignore this path but contained different content. JGit considered
this to be a checkout conflict to prevent overwriting the file in the
working tree and raised an error. This commit fixes this by ensuring
that ignored dirty working tree files don't lead to a checkout conflict.
Bug: 450169
Change-Id: I90288d314ffac73c24a9c70a5181f8243bd4679a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Support for Submodule configuration submodule.<name>.ignore
For each submodule native git allows to configure which modifications to
submodules should be ignored by the status command. It is possible to
ignore "none", "all", "dirty", "untracked" [1]. This configuration is
now supported by IndexDiff. The StatusCommand offers the possibility to
specify this mode.
[1] http://git-scm.com/docs/gitmodules
Change-Id: Ifd81d574a680f9b4152945ba70f8ec4af4f452c9
Strip "<", ">", and "\n" from name/email in UserConfig
This matches what C Git does, see "stripped" in `man git-commit-tree`.
It also fixes the bug of the user where an user.email like "<>" would
show up as "<<>>" in EGit.
Bug: 439844
Change-Id: I567a3c620e191ce9d37d318417e63cb5d4483419
Signed-off-by: Robin Stocker <robin@nibor.org>
Add IndexDiff tests for merge conflict state BOTH_ADDED
JGit handled this case improperly which these tests demonstrate. Fixed
by I25915880f304090fe90584c79bddf021231227a2.
Bug: 440537
Change-Id: Ia29c1d6cf8c0ce724cc3ff5ed9e0b396949b44bf
Signed-off-by: Laurent Goubet <laurent.goubet@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
As described in native gits file "git-read-tree.txt" git has in a
special mode when doing the "initial" checkout. "Initial" means that the
index is empty before the checkout. This was not handled correctly in
JGit and is fixed in this commit. Also see
https://github.com/git/git/blob/master/Documentation/git-read-tree.txt#L181
Change-Id: I9b9d1bd9ebf349cfca420c891c7b099a18d07ba4
Hugo Arès
Ivan Motsch
David Pursehouse
Andrey Loskutov
Shawn Pearce
Shawn Pearce
Christian Halstrick
Jonathan Nieder
Tobias Oberlies
Dave Borowitz
Matthias Sohn
Rüdiger Herrmann
Robin Stocker
Laurent Goubet