Symlinks on MacOS are written as UTF-8 NFD, but
readSymbolicLink().toString() converts to NFC with potentially fewer
bytes. May occur in particular if the link target has non-ASCII
characters for which the NFC and NFD encodings differ. This may lead
to an EOFException: Short read of block.

This causes all kinds of weird effects in EGit, ranging from failing
rebases (which report the exception to the user) to EGit decorations in
the navigator silently disappearing (and never coming back).

* Rename readContentAsNormalizedString() to readSymlinkTarget() as it's
  called only for symlinks. Also make it protected.
* Fix by allowing the read to succeed even if less than the expected
  number of bytes are returned by the entry's input stream.
* Override in FileTreeIterator to use fs.readSymlink() directly.

Includes a new MacOS-only test.

Change-Id: I264c5972d67b1cbb1ed690580f5706e671b9affd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Adds a JGit built-in implementation of the "git lfs smudge" filter. This
filter should do the same as the one described in [1] besides that it
only supports the local case when the lfs objects are already present in
the media directory. Remote cases where download of LFS objects from an
LFS server is needed will be done in a later commit.

[1] https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-smudge.1.ronn

Change-Id: I8ff661d4edd3667ef7f86f3b4fa33e568eb4c8f4

JGit supports clean filters defined in repository configuration. The
filters are implemented as external programs filtering content by
accepting the original content (as seen in the working tree) on stdin
and which emit the filtered content on stdout. To run such a filter JGit
has to start an external process and pump data into/from this process.

This commit adds support for clean filters which are implemented
in Java and which are executed by jgit's main thread. When a filter is
defined in the configuration as
"jgit://builtin/<filterDriverName>/clean" then JGit will lookup in a
static map whether a filter is registered under this name. If found
such a filter is called to do the filtering.

The functionality in this commit requires that a program using JGit
explicitly calls the JGit API to register built-in implementations for
specific clean filters. In follow-up commits configuration parameters
will be added which trigger such registrations. Other commits will add
implementations for lfs filters.

Change-Id: I0344d3c54801c9a46e5a606c5df17e5f2e17b2be

The metadata comparison of submodules is not reliable because of the
last modified timestamp and directory length.

Bug: 498759
Change-Id: If5db69ef3868e475ac477d3e8a7750b268799b0c

JGit failed to do checkouts when the index contained smudged entries and
autocrlf was on. In such cases the WorkingTreeIterator calculated the
SHA1 sometimes on content which was not correctly filtered. The SHA1 was
computed on content which two times went through a lf->crlf conversion.

We used to tell the treewalk whether it is a checkin or checkout
operation and always use the related filters when reading any content.
If on windows and autocrlf is true and we do a checkout operation then
we always used a lf->crlf conversion on any text content. That's not
correct. Even during a checkout we sometimes need the crlf->lf
conversion. E.g. when calculating the content-id for working-tree
content we need to use crlf->lf filtering although the overall operation
type is checkout.

Often this bug does not have effects because we seldom compute the
content-id of filesystem content during a checkout. But we do need to
know whether a file is dirty or not before we overwrite it during a
checkout. And if the index entries are smudged we don't trust the index
and compute filesystem-content-sha1's explicitly.

This caused EGit not to be able to switch branches anymore on Windows
when autocrlf was true. EGit denied the checkout because it thought
workingtree files are dirty because content-sha1 are computed on wrongly
filtered content.

Bug: 493360
Change-Id: I1072a57b4c529ba3aaa50b7b02d2b816bb64a9b8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Implement the DIR_NO_GITLINKS setting with the same functionality
it provides in cGit.

Bug: 436200
Change-Id: I8304e42df2d7e8d7925f515805e075a92ff6ce28
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>

TreeWalk provides the new method getEolStreamType. This new method can
be used with EolStreamTypeUtil in order to create a wrapped InputStream
or OutputStream when reading / writing files. The implementation
implements support for the git configuration options core.crlf, core.eol
and the .gitattributes "text", "eol" and "binary"

CQ: 10896
Bug: 486563
Change-Id: Ie4f6367afc2a6aec1de56faf95120fff0339a358
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Consolidate copies of this function into one location.

Add some unit tests to prevent bugs that were accidentally
introduced while trying to make this refactoring.

Change-Id: I82f64bbb8601ca2d8316ca57ae8119df32bb5c08

When filters are defined for certain paths in gitattributes make
sure that clean filters are processed when adding new content to the
object database.

Change-Id: Iffd72914cec5b434ba4d0de232e285b7492db868
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

When checking out a branch we need to access the attributes stored
in the tree to be checked out. E.g. directly after a clone we checkout
the remote HEAD. In this case index and workingtree are still empty.
So we have to search the tree to be checked out for attributes.

Change-Id: I6d96f5d095ed2e3c259d4b12124e404f5215bd9f

Adds the getAttributes feature to the tree walk. The computation of
attributes needs to be done by the TreeWalk since it needs both a
WorkingTreeIterator and a DirCacheIterator.

Bug: 342372
CQ: 9120
Change-Id: I5e33257fd8c9895869a128bad3fd1e720409d361
Signed-off-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>

The WorkingTreeIterator.isEntryIgnored() should use originally requested
file mode while descending to the file tree root and checking ignore
rules. Original code asking isEntryIgnored() on a file was using
directory mode instead if the .gitignore was not located in the same
directory.

Bug: 473506
Change-Id: I9f16ba714c3ea9e6585e9c11623270dbdf4fb1df
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>

Change-Id: I687c392e5a625fd66c45998c94373aa59921b986
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Core classes to parse and process .gitattributes files including
support for reading attributes in WorkingTreeIterator and the
dirCacheIterator.

The implementation follows the git ignore implementation. It supports
lazy reading attributes while walking the working tree.

Bug: 342372
CQ: 9078
Change-Id: I05f3ce1861fbf9896b1bcb7816ba78af35f3ad3d
Also-by: Marc Strapetz <marc.strapetz@syntevo.com>
Also-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Also-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>
Signed-off-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>

The change tries to make jgit behave more like native CLI git regarding
the negation rules. According to [1] "... prefix "!" which negates the
pattern; any matching file excluded by a previous pattern will become
included again." Negating the pattern should not automatically make the
file *not ignored* - other pattern rules have to be considered too.

The fix adds test cases for both bugs 448094 and 407475.

[1] https://www.kernel.org/pub/software/scm/git/docs/gitignore.html

Bug: 448094
Bug: 407475
Change-Id: I322954200dd3c683e3d8f4adc48506eb99e56ae1
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

The current IgnoreRule/FileNameMatcher implementation scales not well
with huge repositories - it is both slow and memory expensive while
parsing glob expressions (bug 440732). Addtitionally, the "double star"
pattern (/**/) is not understood by the old parser (bug 416348).

The proposed implementation is a complete clean room rewrite of the
gitignore parser, aiming to add missing double star pattern support and
improve the performance and memory consumption.

The glob expressions from .gitignore rules are converted to Java regular
expressions (java.util.regex.Pattern). java.util.regex.Pattern code can
evaluate expression from gitignore rules considerable faster (and with
less memory consumption) as the old FileNameMatcher implementation.

CQ: 8828
Bug: 416348
Bug: 440732
Change-Id: Ibefb930381f2f16eddb9947e592752f8ae2b76e1
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Bug: 445363
Change-Id: Ia8428af84fb61ba0d572374a19e8e8c55b138a63
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>

Adds further tests where the working tree is dirty (differs from
index) and where we have staged but uncommitted changes.

Fixed the test case 9 for file/directory conflicts.

Bug: 428819
Change-Id: Ie44a288b052abe936ebb74272d0fefef3b218a7a
Signed-off-by: Axel Richard <axel.richard@obeo.fr>
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>

Change-Id: I09cd39c367f408b5a963ff004f235f558990f338
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Java spec requires the @Deprecated annotation on any deprecated
field or method. Add the missing annotation to fields and methods
already declared deprecated in the javadoc.

Change-Id: Ic0ef24b43cfd99ac947e771ef5a28e493c304274

While fixing an NPE, I introduced another one in a deprecated isModified
method. It cannot avoid NPE's entirely, which is the reason the method
is deprecated

Change-Id: I5147c1c94621586dd84bd11e6090a954523b6c1c

The method was added for symlink support, but isn't needed anymore.
Since it was added for this release it's like it never existed.

Change-Id: I422cd1dcdfa40b25ba3d6e08b112159dae9a4353
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

It's strange that we have that member since it is not so clear
when it it set or not.

Change-Id: I53903a264f46866d249901a3cd9f9295028aa6bd
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Java normalizes paths to NFC, but other source may not, e.g Eclipse.

Bug: 413390
Change-Id: I08649ac58c9b3cb8bf12794703e4137b1b4e94d5
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

If there is an unnormalized symbolic link in the index, lie that it
matches a normalized link in the working tree. This does not make the
case completely invisible everywhere though, but it helps to some
degree.

Change-Id: I599fb71648c41fa2310049d0e0040b3c9f09386b
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

The change includes comparing symbolic links between disk and index,
adding symbolic links to the index, creating/modifying links on
checkout. The behavior is controlled by the core.symlinks setting, just
as C Git does. When a new repository is created core.symlinks will be
set depending on the capabilities of the operating system and Java
runtime.

If core.symlinks is set to true, the assumption is that symlinks are
supported, which may result in runtime errors if this turns out not to
be the case.

Measuring the cost of jgit status on a repository with ~70000 files,
of which ~30000 are tracked reveals a penalty of about 10% for using
the Java7 (really NIO2) support module.

Bug: 354367
Change-Id: I12f0fdd9d26212324a586896ef7eb1f6ff89c39c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Allegedly this should improve performance, but I could not see it.

Change-Id: Id2057cb2cfcb46e94ff954483ce23f9c4a7edc5e

We should really pass the forceContentCheck parameter to
the real method.

Change-Id: I9ea439cf6340a18d0e931edde3b9e3486cafde93
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Bug: 426514
Change-Id: Ifb75a4fa12291aeeece3dda129a65f0c1fd5e0eb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This version does not attempt to unsmudge, unlike the first attempt
in Idafad150553df14827eccfde2e3b95760e16a8b6.

Bug: 372834
Change-Id: I9300e735cb16d6208e1df963abb1ff69f688155d
Also-by: Robin Stocker <robin@nibor.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Robin Stocker <robin@nibor.org>

This reverts commit 1def0a1257.

We found this fix uncovers problems with unsmudged DirCacheEntry's. This
surfaced because egit's ui test CreatePatchActionTest failed since jgit
computes a wrong status. JGit doesn't detect modified content in now
unsmudged entries. Hence revert this change until these problems are
fixed.

Change-Id: Ia04277ce316d35fc5b0d82c93d2078b856af24bb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>

Bug: 372834
Change-Id: Idafad150553df14827eccfde2e3b95760e16a8b6
Also-by: Robin Stocker <robin@nibor.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: Ifd537ce4e726cb9460ea332f683428689bd3d7f4

There is a huge performance issue when using both JGit (EGit) and Git
because JGit does not fill all dircache stat fields with the values Git
would expect. As a result thereof Git would typically revalidate a large
number of tracked files. This can take several minutes for large
repositories with many large files.

Since 1.8.2 Git will restrict stat checking to the size and whole second
part of the modification time stamp, if core.statinfo is set to
"minimal".

As JGit checks only size and modification time this is close to what
JGit already does. To make the match perfect ignore the sub-second part
of the modification time stamp if core.statinfo = minimal.

Change-Id: I8eaff1858a891571075a86db043f9d80da3d7503

This logic is similar to what we do on Windows, but in this case it's
Java that truncates the timestamps, not Git.

Bug: 395410
Change-Id: Ie55dcb9fa583f5c3dd10d7a1b582e5b04b45858d

A few classes such as Constanrs are marked with @SuppressWarnings, as are
toString() methods with many liternal, but otherwise $NLS-n$ is used for
string containing text that should not be translated. A few literals may
fall into the gray zone, but mostly I've tried to only tag the obvious
ones.

Change-Id: I22e50a77e2bf9e0b842a66bdf674e8fa1692f590

In Iff768422c the offset used for the content id was fixed to use  the
offset that applied to the dircache iterator. Unfortunately the index
for the dircache content id offset stuck for entries that were not in
the index. Few caller probably cared about that, unless it actually
caused an ArrayIndexOutOfBoundsException.

Change-Id: Ic9f0e77c8ea3a0770d88565e94392e76853e3006

idOffset is not zero when idBuffer comes from blob in the dircache

Change-Id: Iff768422cba140a5d6a776e2c627b852f079c1da

Add isModeDifferent method to WorkingTreeIterator
that compares mode with consideration of the
core.filemode setting in the config.

Bug: 379004
Change-Id: I07335300d787a69c3d1608242238991d5b5214ac
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>

This method only creates an EolCanonicalizingInputStream
which does not throw an IOException and so the throws
declaration on the method is unneeded.

Change-Id: Icae8b80006c5e3ffcf3b69790a1a45c505be0f05
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Allow adding files with size over 2 GB. The drawback is that the tests
for huge file support adds roughly 10 minutes of execution time.
For that reason we @Ignore the test in the standard test execution.

Change-Id: I5788e8009899203b346f353297166825b3744575

Use early return instead of nested if/else

Change-Id: I3b5048f9f5cfdfd01f916af550722532db3f9bb3

Content length is computed and cached (short term) in the working
tree iterator when core.autocrlf is set.

Hopefully this is a cleaner fix than my previous attempt to make
autocrlf work.

Change-Id: I1b6bbb643101a00db94e5514b5e2b069f338907a

Change-Id: I763590a45d75f00a09097ab6f89581a3bbd3c797

EolCanonicalizingInputStream may also be used in combination with
.gitattributes. If .gitattributes states that a file is of type text, line
endings have to be canonicalized even if the actual file content seems
to be binary.

Change-Id: Ie4ccdfc5cb91fbd55e06f51146cf5c7c84b8e18b

A '.git' file in a repository's working tree root is now parsed
as a ref to a folder located elsewhere.  This supports submodules
having their repository location outside of the parent repository's
working directory such as in the parent repository's '.git/modules'
directory.

This adds support to BaseRepositoryBuilder for repositories created
with the '--separate-git-dir' option specified to 'git init'.

Change-Id: I73c538f6d845bdbc0c4e2bce5a77f900cf36e1a9
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This patch introduces CRLF handling to the DirCacheCheckout and
WorkingTreeIterator supporting the AutoCRLF for add, checkout
reset and status and hopefully some other places that depende
on the underlying logic of the affected API's.

The patch includes test cases for the Status command provided by
Tomasz Zarna for bug 353867.

The core.eol and core.safecrlf options are not yet supported.

Bug: 301775
Bug: 353867
Change-Id: I2280a2dc0698829475de6a662a6c6e80b1df7663

Currently files in a repository marked as executable will have
that mode unset when modified and committed on systems that
do not support detection of this mode since the working tree
iterator will never report this mode for any entries.

This change updates WorkingTreeIterator to be able
to determine the target file mode to be used for the index
through consideration of the configured WorkingTreeOptions.

Bug: 364956
Change-Id: Iae496baa011b8a59d9329ec73615482b03d34a5a

Change-Id: I17c2709f2613536c02c9ea2977d936b87029ac0d

Open a repository for submodule entries that have a child .git
directory and use the resolved HEAD commit as the entry's id.

Change-Id: I68d6e127f018b24ee865865a2dd3011a0e21453c
Signed-off-by: Kevin Sawicki <kevin@github.com>