Change-Id: I2d17115843e453c71febd4a58527effe658b48fd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Git core learned about the submodule.<name>.shallow option in
.gitmodules files, which is a recommendation to clone a submodule
shallow. A repo manifest may record a clone depth recommendation as
an optional field, which contains more information than a binary
shallow/nonshallow recommendation, so any attempted conversion may be
lossy. In practice the clone depth recommendation is either '1' or doesn't
exist, which is the binary behavior we have in Git core.

Change-Id: I51aa9cb6d1d9660dae6ab6d21ad7bae9bc5325e6
Signed-off-by: Stefan Beller <sbeller@google.com>

Git core learned about attributes in pathspecs:

    pathspec: allow querying for attributes

    The pathspec mechanism is extended via the new
    ":(attr:eol=input)pattern/to/match" syntax to filter paths so that it
    requires paths to not just match the given pattern but also have the
    specified attrs attached for them to be chosen.

(177161a5f7, 2016-05-20)

We intend to use these pathspec attribute patterns for submodule
grouping, similar to the grouping in repo. So the RepoCommand which
translates repo manifest files into submodules should propagate this
information along. This requires writing information to the
.gitattributes file instead of the .gitmodules file. For now we just
overwrite any existing .gitattributes file and do not care about prior
attributes set. If this becomes an issue we need to figure out how to
correctly amend the grouping information to an existing .gitattributes
file.

Change-Id: I0f55b45786b6b8fc3d5be62d7f6aab9ac00ed60e
Signed-off-by: Stefan Beller <sbeller@google.com>

Change-Id: Iecf2b8deafc4991cc3333702fb9fa0638be7b914
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

JGits PushCommand and BasePackPushConnection were throwing a generic
exception when the pushed pack-file was rejected by the server since it
contained too large objects. Teach JGit to better analyze the server's
response to detect this situation and throw a more specific exception.

Detect this situation by parsing the status line sent by the server.
This change only recognizes the response sent by a JGit based server.
All other servers which report such problems in a different way still
lead to a generic TransportExceptions.

Also see https://git.eclipse.org/r/#/c/46348/

Change-Id: I8d6d65e4585ebb3846f7207e7d1a2f82fa9cbd86
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

File.listFiles() returns null if the File is not a directory, improve
validation of directory and gitDir to fix this.

Change-Id: I763d08835faf96a0beb8e706992df0908526bd2c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

JGit failed to do checkouts when the index contained smudged entries and
autocrlf was on. In such cases the WorkingTreeIterator calculated the
SHA1 sometimes on content which was not correctly filtered. The SHA1 was
computed on content which two times went through a lf->crlf conversion.

We used to tell the treewalk whether it is a checkin or checkout
operation and always use the related filters when reading any content.
If on windows and autocrlf is true and we do a checkout operation then
we always used a lf->crlf conversion on any text content. That's not
correct. Even during a checkout we sometimes need the crlf->lf
conversion. E.g. when calculating the content-id for working-tree
content we need to use crlf->lf filtering although the overall operation
type is checkout.

Often this bug does not have effects because we seldom compute the
content-id of filesystem content during a checkout. But we do need to
know whether a file is dirty or not before we overwrite it during a
checkout. And if the index entries are smudged we don't trust the index
and compute filesystem-content-sha1's explicitly.

This caused EGit not to be able to switch branches anymore on Windows
when autocrlf was true. EGit denied the checkout because it thought
workingtree files are dirty because content-sha1 are computed on wrongly
filtered content.

Bug: 493360
Change-Id: I1072a57b4c529ba3aaa50b7b02d2b816bb64a9b8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

As per [1], but limited to absolute paths indeed. No support yet for
tilde or $HOME expansion. Support for the --[no-]includes options
([1]) is not part of this commit scope either, but those options'
defaults are in effect as described in [1].

[1] https://git-scm.com/docs/git-config

Included path can be a config file that includes other path-s in turn.
An exception is thrown if too many recursions (circular includes)
happen because of ill-specified config files.

Change-Id: I700bd7b7e1625eb7de0180f220c707d8e7b0930b
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Added the option to retrieve either merge or non-merge commits in the
LogCommand.

Change-Id: Ie0e1c515a823f2392783f1a47d385c31230e8167
Signed-off-by: Alcemir Santos <alcemir.santos@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

BufferedReader.readLine() returns null if the end of the stream has been
reached.

Change-Id: I83102bbfb1316407247e0f29023077af9e8d9606
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

There was a bug regarding how JGit handled untracked files when applying
a stash. Problem was that untracked files are applied by doing a merge
of HEAD and untrackedFiles commit with a merge base of the stashed HEAD.
That's wrong because the untrackedFiles commit has no parent and
contains only the untracked files. Using stashed HEAD as merge base
leads to unneccessary conflicts on files not event included in the
untrackedFiles commit.

Imagine this graph directly before you want to apply a stash which was
based on 0. You want to apply the stash on current HEAD commit 5.

  5 (HEAD,master)
 /
0---+
 \   \
  1---3 (WIP on master)
     /
    2 (untracked files on master)

Imagine for a specific (tracked) file f
- commit 0 contains X
- HEAD contains Y
- commit 2 (the untracked files) does not contain file f

A merge of 2 and 5 with a merge base of 0 leads to a conflict. The 5
commit wants to modify the file and the 2 commit wants to delete the
file -> conflict.

If no merge base is set then the semantic is correct.

Thanks to Bow for finding this bug and providing the test case.

Bug: 485467
Change-Id: I453fa6ec337f81b2a52c4f51f23044faeec409e6
Also-by: Bow Ruggeri <bow@bow.net>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

There are references which are returned by
RefDatabase.getAdditionalRefs() which are allowed to point to
non-existing objects. These refs should not save objects from being
garbage collected. Examples for these references are ORIG_HEAD,
MERGE_HEAD, FETCH_HEAD and CHERRY_PICK_HEAD. Native git will not take
these references into account when doing a gc and therefore these
references may point to non-existing objects after a gc. Teach JGit's
GC to behave the same: ignore additional refs if they don't start with
"refs/". Examples for refs returned by getAdditionalRefs() which do
start with "refs/" are the bootstrap refs when using reftree's (see
commit 115f1ad397). See also
http://article.gmane.org/gmane.comp.version-control.git/294126.

Bug: 479697
Change-Id: I10e40589f13e72aacdd9f86f3b44696fd1cd068a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

When performing a "reset --hard" a checkout is done. The pathes are
checked for potential checkout conflicts. But in the end for all
remaining conflicts these files are simply deleted from the working
tree. That's not the right strategy to handle checkout conflicts during
"reset --hard". Instead for every conflict we should simply checkout the
merge commit's content.

This is different from native gits behavior which reports errors when
during a "checkout --hard" a file shows up where a folder was expected.

  "warning: unable to unlink d/c.txt: Not a directory"

Why it is like that in native git was asked in
http://permalink.gmane.org/gmane.comp.version-control.git/279482. Unless
it is explained why native git why this error is reported JGit should
overwrite the files.

Bug: 474842
Change-Id: I08e23822a577aaf22120c5137eb169b6bd08447b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

git-apply allows modifying file modes in patched files using either
"new mode" or "new file mode" headers. This patch adds support for
setting files as executables and vice-versa.

Change-Id: I24848966b46f686f540a8efa8150b42e0d9c3ad1
Signed-off-by: Nadav Cohen <nadavcoh@gmail.com>

Before this fix, getting the value of 'key' below used to return
value1. This fix makes it so that value3 gets returned instead,
just like native git's get.

[section]
  key = value1
  key = value2
  key = value3

Change-Id: Iccb24de9b63c3ad8646c909494ca3f8c9ed6e29c
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: Ib9f0ae8207000a36c5bf1a92fcc2c32efc4c0984
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Before this change, jgit used to read packed-refs before scanning
loose refs. That was not a problem if gc didn't run concurrently. When
gc did run concurrently with such refs reading, that order sometimes
broke the latter. This lead to reading an older version of a ref's
tip, which meant "losing" the real tip or commit. The specific
read-Vs-gc concurrency scenario which broke reading that way follows:

1. let ref R be in packed-refs and R' be in loose
2. jgit starts reading packed-refs
3. gc also starts its business around that very time
4. jgit still has the time to read R from packed-refs
5. as gc is not done yet updating packed-refs with R'
6. jgit then starts scanning loose refs (or is about to)
7. gc quickly ends up being done moving loose R' to packed-refs
8. so gc (quickly) removes loose refs
9. -while jgit is scanning loose refs, now gone
10. so jgit assumes no loose to consider => packed-refs winning
11. so jgit wrongfully returns R (from 4.) as the tip, instead of R'.

This fix switches the order so loose refs are scanned (secured) before
taking the time to read packed-refs. This way, knowledge of the
likelier tip is guaranteed for ref reading to return the true tip
- despite concurrent gc. If there is no loose ref to scan, jgit reads
packed-refs and lands on R' (or S), which it then returns, as
expected. The gerrit issue [1] should be solved by this fix.

[1] https://code.google.com/p/gerrit/issues/detail?id=2302

Change-Id: Ibd120120a361a3a6ed565f3836afc1db706fbcdd
Signed-off-by: Marco Miller <marco.miller@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This makes atomic more consistent with other properties like
allowNonFastForwards.

Change-Id: Ie3b41f14ffba1d0b206ffecc697377417daa6973

Change-Id: I0e4e6cef693e8138562f5ca345e0db61a8abef9c

Change-Id: I337c90fff3c013819a42a7622ac6bacc72828c07

Change-Id: Idccfa0dfd6db324236f7cb26ab28bdbd80581a8b
Signed-off-by: Stefan Beller <sbeller@google.com>

The special characters <> and '\n' interfere with parsing of
identities. C git strips these special characters, so we should too.

Rather than allocating extra strings by calling String#trim(), add a
few lines to our sanitization method to perform the same trimming as
described in String's Javadoc.

Change-Id: I96edcb93a2fc194ee354d60566d352299742a52f

This might be somewhat surprising behavior to users who might
naturally assume the following invariant:
  ident.equals(parseIdent(ident.toExternalString()))

This invariant does not hold since whitespace is only trimmed during
serialization. We don't want to mess with the strings during
initialization, as this is called during the highly-optimized commit
parsing codepath.

Change-Id: I081a603f0ac0e33167462244779b0ff3ad51e80c

There was a bug in JGit which caused NPEs being thrown when Transport
errors should be reported. Avoid the NPE to let the original error show
up.

Change-Id: I9e1e2b0195bd61b7e531a09d0fc7bce109bd6515

We've found in Gerrit Code Review that it is common to pass around
both an ObjectReader (or more commonly a RevWalk wrapping one) and an
ObjectInserter. These code paths often assume that the ObjectReader
can read back any objects created by the ObjectInserter without
flushing. However, we previously had no way to enforce that constraint
programmatically, leading to hard-to-spot problems.

Provide a solution by exposing the ObjectInserter that created an
ObjectReader, when known. Callers can either continue passing both
objects and check:
  reader.getCreatedFromInserter() == inserter
or they can just pass around ObjectReader and extract the inserter
when it's needed (checking that it's not null at usage time).

Change-Id: Ibbf5d1968b506f6b47030ab1b046ffccb47352ea

Change-Id: If34ad8185dc8192435e622995fdca598b86add7e

When CheckoutCommand or MergeCommand is called then not in all situation
the treewalks have been prepared to support clean/smudge filters. Fix
this

Bug: 491505
Change-Id: Iab5608049221c46d06812552ab97299e44d59e64

Repurpose RefDatabase#performsAtomicTransactions() slightly, to
indicate that the backend _supports_ atomic transactions, rather than
the current definition, which is that the backend always _uses_ atomic
transactions regardless of whether or not the caller actually wants
them. Allow BatchRefUpdate callers to turn off atomic transactions by
calling setAtomic(false). Defaulting to true means this is backwards
compatible.

Change-Id: I6df78d7df65ab147b4cce7764bd3101db985491c

Such hunks are identifiable by a zero value for "new start line". Prior
to the fix, JGit throws and ArrayIndexOutOfBoundsException on such
patches.

Change-Id: I4f3deb5e5f41a08af965fcc178d678c77270cddb
Signed-off-by: Jonathan Schneider <jkschneider@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Now if refs are unreadable when serving an upload pack the handler
will fail due to the actual underlying failure. Previously all wants
would be rejected as invalid because Repository.getAllRefs() returned
an empty map.

Testing this required a new subclass of InMemoryRepository so that
an IOException could be injected at the correct time.

Signed-off-by: Michael Edgar <adgar@google.com>
Change-Id: Iac708b1db9d0ccce08c4ef5ace599ea0b57afdc0

Change-Id: I5b3b7b0633354d5ccf0c6c320c0df9c93fdf8eeb
Signed-off-by: Ned Twigg <ned.twigg@diffplug.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

CommitCommand already provided a method to set the comment which should
be written into the reflog. The underlying RefUpdate class supported to
skip writing a reflog entry. But through the CommitCommand API it was
not possible to prevent writing a reflog entry. Fix this and allow
creating commits which don't occur in the reflog.

Change-Id: I193c53de71fb5958ea749c4bfa8360a51acc9b58

PackWriter.writeObject() can get into an infinite loop when corrupt
packs are present. When it finds a pack file with an object that can be
reused it calls DfsPackFile.copyAsIs(). If that method sees an invalid
CRC, it adds the object to the DfsPackFile's corrupt object list and
throws a CorruptObjectException, which it later catches as an
IOException and wraps in a
StoredObjectRepresentationNotAvailableException.
PackWriter.writeObjectImpl() catches that SORNAE and retries the
operation by calling DfsReader.selectObjectRepresentation(). But
currently that method returns the same object which was just seen to
be corrupt.

Change DfsPackFile.isCorrupt() from private to package private, and use
that method in DfsReader.selectObjectRepresentation() to filter out
corrupt objects.

The stack traces that show the problem are:
  org.eclipse.jgit.errors.CorruptObjectException.<init>(CorruptObjectException.java:113)
  org.eclipse.jgit.internal.storage.dfs.DfsPackFile.copyAsIs(DfsPackFile.java:624)
  org.eclipse.jgit.internal.storage.dfs.DfsReader.copyObjectAsIs(DfsReader.java:491)
  org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1478)
  org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1455)

  org.eclipse.jgit.internal.storage.dfs.DfsPackFile.getPackIndex(DfsPackFile.java:228)
  org.eclipse.jgit.internal.storage.dfs.DfsReader.findAllFromPack(DfsReader.java:476)
  org.eclipse.jgit.internal.storage.dfs.DfsReader.selectObjectRepresentation(DfsReader.java:455)
  org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1492)
  org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1455)

Change-Id: Iad7bbcaed1f11a6aa3b4f5af911a73a34c0fabfd
Signed-off-by: Terry Parker <tparker@google.com>

RepositoryCache has 2 methods to remove a repository from the cache but
they are never called when a repository is closed. Users of the cache
were expected to call one of those 2 methods but how could they have
called them at proper time without having visibility of the repository
usage count.

Ideally, I would have reworked the RepositoryCache to wrap any
repository it opens in a class that would be responsible to unregister
them from the cache when it's really closed, i.e. when usage counter
reaches 0. The problem preventing the wrapping solution is the
RepositoryCache.register method that allows to register an already
opened repository in the cache. Such repositories cannot be wrapped
because callers are still holding a reference on the unwrapped
repository.

Document that RepositoryCache.close method is removing the repository
from the cache as well as closing it and rework
RepositoryCache.unregister method to only remove the repository from the
cache. Use the latter to unregister repository when Repository.doClose
is getting executed.

Change-Id: Ia364816e4da8d7b6cfa72f10758ca31aa8a1f9db
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

When repositories are opened using the RepositoryCache, they are kept in
memory and when the repository usage counter reaches 0, the
Repository.close method is called which then calls close method on its
reference and object databases.

The problem is that RefDirectory.close method was a no-op and the
reference database was kept in memory. This problem is only happening
when opening a repository using the RepositoryCache because it never
evicts repositories, it's just calling the close method.

Change-Id: Iacb961de8e8b1f5b37824bf0d1a4caf4c6f1233f
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>

Repository has a usage counter that is initialized to 1 at
instantiation and this counter is decremented when Repository.close
method is called. There is also a Repository.incrementOpen method that
RepositoryCache uses to increment the usage count when it's returning a
repository that is already opened.

The problem was that RepositoryCache was incrementing the usage count
for repositories that it just opened or registered. The usage count was
2 when it should have been 1.

Incrementing usage count is now only be done for repository that are
served from the cache.

This bug is causing slow memory increase of our Gerrit server until the
server become slow. Even if the RepositoryCache is using SoftReference,
it seems that the JVM is not garbage collecting the repositories because
it's not yet on the edge of being out of memory.

To test this change, I replicated all repositories(11k) from Gerrit
master to one slave. The Gerrit master used memory after this test was
10GB without this change and 3.5GB with.

Change-Id: I86c7b36174e384f106b51fe92f306018fd1dbdf0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>

When checking out commits/branches JGit was triggering correctly
configured smudge filters. But when checking out paths (either from
index or from commits) JGit was not triggering smudge filters. Fix
CheckoutCommand to properly call filters.

Bug: 486560
Also-by: Pascal Krause <pascal.krausek@sap.com>
Change-Id: I5ff893054defe57ab12e201d901fe74e1376efea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Implement the DIR_NO_GITLINKS setting with the same functionality
it provides in cGit.

Bug: 436200
Change-Id: I8304e42df2d7e8d7925f515805e075a92ff6ce28
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>

JGit's Garbage Collector is repacking relevant objects into new
packfiles and is afterwards deleting the now obsolete packfiles. But to
prevent problems caused by race conditions JGit was not deleting
packfiles when they are too young. The same mechanism as for loose
objects and the config parameter gc.pruneExpire was used.
But JGit was reusing the parameter gc.pruneExpire also for packfiles
which may cause a lot of filesystem consumption if gc.pruneExpire was
set to the default of 2 weeks. Only two weeks after packfile creation gc
was allowed to delete this packfile.

This change introduces a new config paramter gc.prunePackExpire with a
default of "1.hour". This parameter is used when packfiles are deleted.
Only packfiles older than the specified time can be deleted.

For loose objects the behaviour is not changed and only the old
parameter gc.pruneExpire is relevant. 

Change-Id: I6209efb05678b15153bd22479dc13486907a44f8

Noticed by error-prone (http://errorprone.info/bugpattern/DepAnn).

Change-Id: If8dcc2dd6d7cabffc95e10f8357ce31c12cf6b1c

This change replaced some tabs with spaces introduced in
change I8b3765713599e34f1411f9bbc7f575ec7c2384e0.

Change-Id: Ia5c23b38c9fbbb46f150e527347b61c64c8d9e87
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>

With ignoreRemoteFailures set to true, we can ignore remote failures
(e.g. the branch of a project described in the manifest file does not
exist), skip that project and continue to the next one, instead of fail
the whole operation.

Change-Id: I8b3765713599e34f1411f9bbc7f575ec7c2384e0
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>

This commit introduces a FileModeStrategy to
the FileTreeIterator class.  This provides a way to
allow different modes of traversing a file tree;
for example, to control whether or not a nested
.git directory should be treated as a gitlink.

Bug: 436200
Change-Id: Ibf85defee28cdeec1e1463e596d0dcd03090dddd
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>

Allow access to the ObjectId of a DirCacheTree if known for low level
integration code.

Change-Id: I6f05b10c9ac781f5e8b38af4a19e653313c91fa8
Signed-off-by: Philipp Marx <smigfu@googlemail.com>

TreeWalk provides the new method getEolStreamType. This new method can
be used with EolStreamTypeUtil in order to create a wrapped InputStream
or OutputStream when reading / writing files. The implementation
implements support for the git configuration options core.crlf, core.eol
and the .gitattributes "text", "eol" and "binary"

CQ: 10896
Bug: 486563
Change-Id: Ie4f6367afc2a6aec1de56faf95120fff0339a358
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Bug: 483943
Change-Id: If28f64053d20ab1bee54245f223e952dc2fe392c
Signed-off-by: XinTong Wang <xintong@ca.ibm.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

The reason is that URIish(URL) and URIish(String) make different parsing
of path / rawPath with regard to drive letters. /C:/... for URL and
C:/... for String. This patch fixes the issue.

Change-Id: I8e2013fff30b7bb198ff733c038e21366667b8a0
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>

Capture the internal "want X not valid" state as a specific subclass
of PackProtocolException, allowing this to be more easily identified
in server stack traces and wrapper application code.

Change-Id: I4b1adb7497f396432da420b0f600ad25a261f912

If the client sends a SHA-1 that the server does not recognize echo
this back to the client with an explicit error message instead of
the generic "internal server error".

This was always the intent of the implementation but it was being
dropped on smart HTTP due to the UploadPackServlet catching the
PackProtocolException, discarding the buffered message UploadPack
meant to send, and sending along a generic message instead.

Change-Id: I8d96b064ec655aef64ac2ef3e01853625af32cd1

This reverts commit 979fa19110.
Breaks API.

Change-Id: I54af657898d49f64d6906fe3edfb6b08e996d901