Update SHA1 class to include a Java port of sha1dc[1]'s ubc_check,
which can detect the attack pattern used by the SHAttered[2] authors.

Given the shattered example files that have the same SHA-1, this
modified implementation can identify there is risk of collision given
only one file in the pair:

  $ jgit ...
  [main] WARN org.eclipse.jgit.util.sha1.SHA1 - SHA-1 collision 38762cf7f5

When JGit detects probability of a collision the SHA1 class now warns
on the logger, reporting the object's SHA-1 hash, and then throws a
Sha1CollisionException to the caller.

From the paper[3] by Marc Stevens, the probability of a false positive
identification of a collision is about 14 * 2^(-160), sufficiently low
enough for any detected collision to likely be a real collision.

git-core[4] may adopt sha1dc before the system migrates to an entirely
new hash function.  This commit enables JGit to remain compatible with
that move to sha1dc, and help protect users by warning if similar
attacks as SHAttered are identified.

Performance declined about 8% (detection off), now:

  MessageDigest        238.41 MiB/s
  MessageDigest        244.52 MiB/s
  MessageDigest        244.06 MiB/s
  MessageDigest        242.58 MiB/s

  SHA1                 216.77 MiB/s (was ~240.83 MiB/s)
  SHA1                 220.98 MiB/s
  SHA1                 221.76 MiB/s
  SHA1                 221.34 MiB/s

This decline in throughput is attributed to the step loop unrolling in
compress(), which was necessary to easily fit the UbcCheck logic into
the hash function.  Using helper functions s1-s4 reduces the code
explosion, providing acceptable throughput.

With detection enabled (default):

  SHA1 detectCollision 180.12 MiB/s
  SHA1 detectCollision 181.59 MiB/s
  SHA1 detectCollision 181.64 MiB/s
  SHA1 detectCollision 182.24 MiB/s

  sha1dc (native C)   ~206.28 MiB/s
  sha1dc (native C)   ~204.47 MiB/s
  sha1dc (native C)   ~203.74 MiB/s

Average time across 100,000 calls to hash 4100 bytes (such as a commit
or tree) for the various algorithms available to JGit also shows SHA1
is slower than MessageDigest, but by an acceptable margin:

  MessageDigest        17 usec
  SHA1                 18 usec
  SHA1 detectCollision 22 usec

Time to index-pack for git.git (217982 objects, 69 MiB) has increased:

  MessageDigest   SHA1 w/ detectCollision
  -------------   -----------------------
         20.12s   25.25s
         19.87s   25.48s
         20.04s   25.26s

    avg  20.01s   25.33s    +26%

Being implemented in Java with these additional safety checks is
clearly a penalty, but throughput is still acceptable given the
increased security against object name collisions.

[1] https://github.com/cr-marcstevens/sha1collisiondetection
[2] https://shattered.it/
[3] https://marc-stevens.nl/research/papers/C13-S.pdf
[4] https://public-inbox.org/git/20170223230621.43anex65ndoqbgnf@sigill.intra.peff.net/

Change-Id: I9fe4c6d8fc5e5a661af72cd3246c9e67b1b9fee6

Symlinks on MacOS are written as UTF-8 NFD, but
readSymbolicLink().toString() converts to NFC with potentially fewer
bytes. May occur in particular if the link target has non-ASCII
characters for which the NFC and NFD encodings differ. This may lead
to an EOFException: Short read of block.

This causes all kinds of weird effects in EGit, ranging from failing
rebases (which report the exception to the user) to EGit decorations in
the navigator silently disappearing (and never coming back).

* Rename readContentAsNormalizedString() to readSymlinkTarget() as it's
  called only for symlinks. Also make it protected.
* Fix by allowing the read to succeed even if less than the expected
  number of bytes are returned by the entry's input stream.
* Override in FileTreeIterator to use fs.readSymlink() directly.

Includes a new MacOS-only test.

Change-Id: I264c5972d67b1cbb1ed690580f5706e671b9affd
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

git-apply allows modifying file modes in patched files using either
"new mode" or "new file mode" headers. This patch adds support for
setting files as executables and vice-versa.

Change-Id: I24848966b46f686f540a8efa8150b42e0d9c3ad1
Signed-off-by: Nadav Cohen <nadavcoh@gmail.com>

Such hunks are identifiable by a zero value for "new start line". Prior
to the fix, JGit throws and ArrayIndexOutOfBoundsException on such
patches.

Change-Id: I4f3deb5e5f41a08af965fcc178d678c77270cddb
Signed-off-by: Jonathan Schneider <jkschneider@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Bug: 483943
Change-Id: If28f64053d20ab1bee54245f223e952dc2fe392c
Signed-off-by: XinTong Wang <xintong@ca.ibm.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Building on top of https://git.eclipse.org/r/#/c/56391/

Here we preserve compatibility with JetS3t
and add 2 new native JGit encryption implementations.

For reference, see connection configuration files:
* Version 0: jgit-s3-connection-v-0.properties
* Version 1: jgit-s3-connection-v-1.properties
* Version 2: jgit-s3-connection-v-2.properties

Change-Id: I713290bcacbe92d88e5ef28ce137de73dd1abe2f
Signed-off-by: Andrei Pozolotin <andrei.pozolotin@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

See previous attempt: https://git.eclipse.org/r/#/c/16674/

Here we preserve as much of JetS3t mode as possible
while allowing to use new Java 8+ PBE algorithms
such as PBEWithHmacSHA512AndAES_256

Summary of changes:
* change pom.xml to control long tests
* add WalkEncryptionTest.launch to run long tests
* add AmazonS3.Keys to to normalize use of constants
* change WalkEncryption to support AES in JetS3t mode
* add WalkEncryptionTest to test remote encryption pipeline
* add support for CI configuration for live Amazon S3 testing
* add log4j based logging for tests in both Eclipse and Maven build

To test locally, check out the review branch, then:
* create amazon test configuration file
* located your home dir: ${user.home}
* named jgit-s3-config.properties
* file format follows AmazonS3 connection settings file:
	accesskey = your-amazon-access-key
	secretkey = your-amazon-secret-key
	test.bucket = your-bucket-for-testing
* finally:
* run in Eclipse: WalkEncryptionTest.launch
* or
* run in Shell: mvn test --define test=WalkEncryptionTest

Change-Id: I6f455fd9fb4eac261ca73d0bec6a4e7dae9f2e91
Signed-off-by: Andrei Pozolotin <andrei.pozolotin@gmail.com>

Previously the result of an application would have been \r\r\n in the
case of windows line endings, as RawText does not touch the \r, and
ApplyCommand adds "\r\n" if this is the ending of the first line in the
target file. Only always adding \n should be ok, since \r\n would be the
result if the file and the patch include windows line endings.

Also add according test.

Change-Id: Ibd4c4948d81bd1c511ecf5fd6c906444930d236e

Otherwise applying will fail with a FileNotFoundException, because
File.createNewFile() fails with missing parents.

Contains change & according test.

Change-Id: I970522b549b8bb260ca6720da11f12c57ee8a492
Signed-off-by: Chris Aniszczyk <zx@twitter.com>

Check for a '#' character in each line read and
parse the leading characters as the class name of
a TransportProtocol being registered via SPI.

Bug: 373439
Change-Id: If36cb62c07ecea78ba0f326a87edf1d80b7b42b6
Signed-off-by: Chris Aniszczyk <zx@twitter.com>

Bug: 361548
CQ: 6243
Change-Id: I08e1369e142bb19f42a8d7bbb5a7d062cc8533fc
Signed-off-by: Chris Aniszczyk <zx@twitter.com>

The test was never run from maven, because its name did not end
in a way that was recognized by the pom. After rename it failed
because it did not find its resources.

Rename test class and move resources to the resources folder

Change-Id: I74a7ef1373cd902e1d05ff6ea38f8648b5fc5700
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>

If the 'TREE' extension contains an invalid subtree that has
been removed, DirCacheIterator still tried to access it due to
an invalid childCnt field within the parent DirCacheTree object.
This is easy for a user to do, they just need to move all files
out of a subdirectory.

For example, the input for the JUnit test case for this bug was
built using the following C Git sequence:

  mkdir -p a/b
  touch a/b/c q
  git add a/b/c q
  git write-tree
  git mv a/b/c a/a

After the last step, the subdirectory a/b is empty, as its only
file was moved into the parent directory.  Because of the earlier
`git write-tree` operation, there is a 'TREE' extension present, but
the a and a/b subdirectories have been marked invalid by the rename.

When JGit tried to iterate over the a tree, it tried to correct
childCnt to be zero as a/b no longer exists, but it failed to
update childCnt.

Change-Id: I7a0f78fc48a36b1a83252d354618f6807fca0426
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Extended flags are processed and available via DirCacheEntry's
new isSkipWorkTree() and isIntentToAdd() methods.  "resolve-undo"
information is completely ignored since its an optional extension.

Change-Id: Ie6e9c6784c9f265ca3c013c6dc0e6bd29d3b7233

The working tree iterator has perfect knowledge of the path structure
as well as immediate information about whether or not an ignore file
even exists at this level.  We can exploit that to simplify the
logic and running time for testing ignored file status by pushing
all of the checks down into the iterator itself.

Change-Id: I22ff534853e8c5672cc5c2d9444aeb14e294070e
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Charley Wang <chwang@redhat.com>
CC: Chris Aniszczyk <caniszczyk@gmail.com>
CC: Stefan Lay <stefan.lay@sap.com>
CC: Matthias Sohn <matthias.sohn@sap.com>

This patch adds ignore compatibility to jgit. It encompasses
exclude files as well as .gitignore. Uses TreeWalk and
FileTreeIterator to find nodes and parses .gitignore
files when required. The patch includes a simple cache that
can be used to save results and avoid excessive gitignore
parsing.

CQ: 4302
Bug: 303925
Change-Id: Iebd7e5bb534accca4bf00d25bbc1f561d7cad11b
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

We didn't skip the correct number of bytes when we skipped over an
unrecognized but optional dircache extension.  We missed skipping
the 8 byte header that makes up the extension's name and length.

We also didn't include the skipped extension's payload as part of
our index checksum, resuting in a checksum failure when the index
was done reading.  So ensure we always scan through a skipped
section and include it in the checksum computation.

Add a test case for a currently unsupported index extension, 'ZZZZ',
to verify we can still read the DirCache object even though we
don't know what 'ZZZZ' is supposed to mean.

Bug: 301287
Change-Id: I4bdde94576fffe826d0782483fd98cab1ea628fa
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The test data is expected to have unix new lines by tests, but it
is converted to crlf on Windows platform (with msys git). As result
DiffFormatterReflowTest tests fail. To prevent this problem,
crlf conversion is disbled for test data related to that test.

Bug: 295077
Change-Id: I67d3ed543fcc38647041896146de12b1781ec6be
Signed-off-by: Constantine Plotnikov <constantine.plotnikov@gmail.com>

Per CQ 3448 this is the initial contribution of the JGit project
to eclipse.org.  It is derived from the historical JGit repository
at commit 3a2dd9921c.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>