This also includes a change to generating the jgit CLI jar. Shading is
no longer possible because it breaks the signature of BouncyCastle.
Instead, the Spring Boot Loader Maven plug-in is now used to generate an
executable jar.

Bug: 382212
Change-Id: I35ee3d4b06d9d479475ab2e51b29bed49661bbdc
Also-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>
Signed-off-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This change introduces the concept of a GpgSigner which will sign
commits. The GpgSigner will be of a specific implementation (eg.,
Bouncycastle or OpenPgP executable). The actual implementation is not
part of this change.

Bug: 382212
Change-Id: Iea5da1e885c039e06bc8d679d46b124cbe504c8e
Also-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Medha Bhargav Prabhala <mprabhala@salesforce.com>
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>

Change-Id: I22a244114ef973b3801cf3b639ff782f21c02f98
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

CheckoutCommand had a setForce() method. But this didn't correspond
to native git's 'git checkout -f' option. Deprecate the old setForce()
method and move its implementation to a new method setForceRefUpdate()
and use it to implement the -B option in the CLI class Checkout.

Add a setForced() method and use it to fix the associated '-f' option of
the CLI Checkout class to behave like native git's 'git checkout -f'
which overwrites dirty worktree files during checkout.

This is still not fully matching native git's behavior: updating
additionally dirty index entries is not done yet.

Bug: 530771
Change-Id: I776b78eb623b6ea0aca42f681788f2e4b1667f15
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Its implementation contains

		} catch (IOException e) {
			// Legacy API, assume error means "no"
			return false;
		}

Better to use ObjectDatabase#has, which throws IOException to report
errors.

Change-Id: I7de02f7ceb8f57b2a8ebdb16d2aa4376775ff933
Signed-off-by: Jonathan Nieder <jrn@google.com>

Using findRef instead of getRef makes it clearer that the caller wants
to search for the ref in the search path, instead of looking for a ref
that exactly matches the input.

This change introduces the new findRef method and deprecates getRef.
It updates Repository#findRef to use the new method, ensuring some
test coverage.  Other callers will be updated in followup changes.

A nice side effect of introducing the new findRef method is that it is
final and based on firstExactRef, so implementers can focus on
implementing the latter efficiently and do not have to carefully write
custom path search code respecting SEARCH_PATH.

Change-Id: Id3bb944344a9743705fd1f20193ab679298fa51c
Signed-off-by: Jonathan Nieder <jrn@google.com>

Psuedorefs like FETCH_HEAD and MERGE_HEAD are supposed to be directly
under the .git directory, not in other locations in the SEARCH_PATH
like refs/ and refs/heads/.  Use exactRef to access them.

Change-Id: Iab8ac47008822fa78fc0691e239e518c34d7a98e
Signed-off-by: Jonathan Nieder <jrn@google.com>

This is simpler to implement than getRef.  Make it abstract so
implementers remember to override it.

Change-Id: I5f319be1fb1206d7a0142ea939dc4e1039f850ab
Signed-off-by: Jonathan Nieder <jrn@google.com>

Now the reference carries its updateIndex, so the cursor doesn't need
to expose it.

Change-Id: Icbfca46f92a13f3d8215ad10b2a166a6f40b0b0f
Signed-off-by: Ivan Frade <ifrade@google.com>

In DFS implementations the reference table can fall out of sync, but
it is not possible to check this situation in the current API.

Add a property to the Refs indicating the order of its updates.  This
version is set only by RefDatabase implementations that support
versioning (e.g reftable based).

Caller is responsible to check if the reference db creates versioned
refs before accessing getUpdateIndex(). E.g:

   Ref ref = refdb.exactRef(...);
   if (refdb.hasVersioning()) {
       ref.getUpdateIndex();
   }

Change-Id: I0d5ec8e8df47c730301b2e12851a6bf3dac9d120
Signed-off-by: Ivan Frade <ifrade@google.com>

ProtocolV2Parser explains:

  // TODO(ifrade): This validation should be done after the
  // protocol parsing. It is not a protocol problem asking for an
  // unexisting ref and we wouldn't need the ref database here.

Do so.  This way all ref database accesses are in one place, in the
UploadPack class.

No user-visible change intended --- this is just to make the code
easier to manipulate.

Change-Id: I68e87dff7b9a63ccc169bd0836e8e8baaf5d1048
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

When executing a test with LANG environment variable set to non UTF-8
encoding, it seems that JRE cannot handle Unicode file paths. This
happens when this test is executed in Bazel as it unsets LANG
(https://docs.bazel.build/versions/master/test-encyclopedia.html#initial-conditions).

Skip the test if the runtime cannot handle Unicode file paths.

Change-Id: I16bd3cd959dbaf2335b9c5202873e2f12ed0ba21
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>

By using File#setLastModified, we can create a racy git situation
stably.

Tested with --runs_per_test=100

Bug: 526111
Change-Id: I60b3632d353e19f335668325aa603640be423f58
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>

Change-Id: I0fdf595cba93f5a5cdd0496cee07ac91db304532
Signed-off-by: Mincong Huang <mincong.h@gmail.com>

Set GIT_DIR and GIT_WORK_TREE when calling hooks.

Bug: 541622
Change-Id: I6153d8a6a934ec37a3a5e7319c2d0e516f539ab7
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

After cloning a repo with a submodule, non-recursively, JGit would
encounter in its TreeWalk in IndexDiff:

* first, a missing gitlink (in index & HEAD, not in working tree)
* second, the untracked folder (not in index and head, in working tree)

As a result, it would report the submodule as missing. Canonical git
reports a clean workspace.

The root cause of this is that the path of a gitlink "x" did
not compare equal to the path of a tree "x" in JGit.

Correct Paths.compare() to account for that. If two paths are otherwise
equal, then let gitlinks match both trees and files. Matching trees
solves the bug. Matching files is necessary to handle the case where
the gitlink directory was replaced by a file; see the new test case
IndexDiffSubmoduleTest.testSubmoduleReplacedByFile(). Comparisons of
unequal paths are left untouched, so the sort order is unchanged.

After the fix, another bug(?) in WorkingTreeIterator became apparent:
with core.dirNoGitLinks = true, it was no longer possible to overwrite
a gitlink in the index. This is now fixed in WorkingTreeIterator.

Add new test cases for the bug itself and for some related cases
(submodule directory deleted or replaced by a file) in
IndexDiffSubmoduleTest. Add a test for missing files in IndexDiffTest,
and adapt the PathsTest to test matching gitlinks.

Bug: 467631
Change-Id: I0549d10d46b1858e5eec3cc15095aa9f1d5f5280
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

when multiple match options are given in git describe the result must
not depend on the order of the match options. JGit wrongly picked the
first match using the match options in the order they were defined. Fix
this by concatenating the streams of matching tags for all match options
and then choosing the first match on the concatenated stream sorted in
tie break order.

See https://git-scm.com/docs/git-describe#git-describe---matchltpatterngt

Change-Id: Id01433d35fa16fb4c30526605bee041ac1d954b2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

According to RFC 2616 [1] header field names are case insensitive.
Header fields defined as a comma separated list can have multiple header
fields with the same field name. Add a method to HttpConnection which
retrieves all values with a given header field name with the field name
compared case insensitive.

[1] https://tools.ietf.org/html/rfc2616#section-4.2"

Change-Id: I7f601b21cda99e84f43f866c7c7cb4cb0e3cf5c3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This partially reverts commit a551b646: revert the changes in
RawParseUtils.lineMap(). Forcing all blobs containing a NUL byte
as a single line causes blame to produce useless results as soon
as it hits any version containing a NUL byte.

Doing binary detection at this level also has the problem that the
user cannot control it. Not by setting the text attribute nor in any
other way.

This came up in bug 541036, where a Java source inadvertently
contained NUL bytes in strings. Even fixing this by using escapes
"\000" will not fix JGit's blame for this file because the past
versions will still contain the NUL byte.

Native git can blame that file from bug 541036 fine.

Added new tests verifying that blaming a text file containing a NUL
byte produces sensible results.

Bug: 541036
Change-Id: I8991bec88e9827cc096868c6026ea1890b6d0d32
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Change-Id: Ifece38132e0390931d33b5678bbbd4a43f9099da
Signed-off-by: Gunnar Wagenknecht <gunnar@wagenknecht.org>

Add a new ssh client implementation based on Apach MINA sshd 2.0.0.

This implementation uses JGit's own config file parser and host entry
resolver. Code inspection of the Apache MINA implementation revealed
a few bugs or idiosyncrasies that immediately would re-introduce bugs
already fixed in the past in JGit.

Apache MINA sshd is not without quirks either, and I had to configure
and override more than I had expected. But at least it was all doable
in clean ways.

Apache MINA boasts support for Bouncy Castle, so in theory this should
open the way to using more ssh key algorithms, such as ed25519.

The implementation is in a separate bundle and is still not used in
the core org.eclipse.jgit bundle. The tests re-use the ssh tests from
the core test bundle.

Bug: 520927
Change-Id: Ib35e73c35799140fe050d1ff4fb18d0d3596580e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Add support for git-receive-pack to the ssh git server and add two
new tests for pushing.

This actually uncovered an undocumented requirement in TransportSftp:
the FTP rename operation assumes POSIX semantics, i.e., that the
target is removed. This works as written only for servers that
support and advertise the "posix-rename@openssh.com" FTP extension.

Our little Apache MINA server does not advertise this extension.

Fix the FtpChannel implementation for Jsch to handle this case in a
meaningful way so that it can pass the new "push over sftp" test.

Add more tests to test the behavior of server host key checking.

Also refactor the tests generally to separate better the test
framework from the actual tests.

Bug: 520927
Change-Id: Ia4bb85e17ddacde7b36ee8c2d5d454bbfa66dfc3
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Correct behaviour as git 1.7.1.1 is to resolve tie-breakers to choose
the most recent tag.

https://github.com/git/git/blob/master/Documentation/RelNotes/1.7.1.1.txt:
* "git describe" did not tie-break tags that point at the same commit
   correctly; newer ones are preferred by paying attention to the
   tagger date now.

Bug: 538610
Change-Id: Ib0b2a301997bb7f75935baf7005473f4de952a64
Signed-off-by: Håvard Wall <haavardw@gmail.com>

Suppose that a repository has the following commit graph:

 B   C
  \ /
   A

and it was cloned with --shallow-exclude=A. DepthGenerator does not mark
C as shallow, causing an invalid repository to be produced on the
client, because A is not sent. (A similar issue occurs when
--shallow-since is used to exclude A but neither B nor C.)

This happens whenever an excluded commit has more than one child that is
to be sent to the client. Fix DepthGenerator to handle this case
correctly.

While we're editing DepthWalk.Commit, fix the documentation of
DepthWalk.Commit#isBoundary.

Change-Id: I7068abf0fe0c864d1b0e56e1616dad1aa8719411
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>

Change-Id: Id5ba87b24e8a29157d49e9e5b151d65bdbcab37a
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Move the bulk of the basic parsing and host entry handling into a
new class OpenSshConfigFile that has no dependencies on any concrete
ssh implementation. Make the existing OpenSshConfig use the new
parser.

Introduce a new class SshConstants collecting all the various ssh-
related string literals. Also use TreeMaps with a case-insensitive
key comparator instead of converting keys to uppercase. Add a test
to verify that keys are matched case-insensitively.

Most of the parsing code was simply moved, except that the new
parser supports looking up entries given host name, port, and user
name, and can thus handle more %-substitutions correctly. This
feature is not yet used and cannot be used with JSch since JSch
only has a ConfigRepository.getConfig(String) interface.

The split is still worth the trouble as it opens the way to using
another ssh client altogether. Apache MINA sshd, for instance,
resolves host entries giving host name, port, and user name.

(Apache MINA has a built-in ssh config handling, but that has
problems, too: its pattern matching is case-insensitive, and its
merging of host entries if several match is not the same as in
OpenSsh. But with this refactoring, it will be possible to plug in
OpenSshConfigFile into an Apache MINA sshd  client without dragging
along JSch.)

One test case that doesn't make sense anymore has been removed. It
tested that repeatedly querying for a host entry returned the same
object. That is no longer true since the caching has been moved to
a deeper level.

Bug: 520927
Change-Id: I6381d52b29099595e6eaf8b05c786aeeaefbf9cc
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

Add a simple ssh git server based on Apache MINA sshd, and use it
in new tests that verify ssh operations and in particular a number
of bugs that had cropped up over time in JSch.

The git server supports fetching only, and sftp access.

The tests are all in an abstract base class; the concrete JschSshTest
class only provides ssh-specific test setup. So the same tests could
be run easily also with some other ssh client.

Bug: 520927
Change-Id: Ide6687b717fb497a29fc83f22b07390a26dfce1d
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

The copyfile entry in the manifest file copies the contents of the file
but doesn't keep the executable flag. This is inconsistent with repo
tool behaviour, plus is natural to expect that the copy of a executable
file is executable.

Transfer the executable bit when copying the file, aligning the
RepoCommand with repo tool and user expectations.

Change-Id: I01b24f482d5939e01d496f032388b3a5c02a912a
Signed-off-by: Ivan Frade <ifrade@google.com>

The RepoCommand.RemoteReader interface doesn't offer access to the mode
of a file. Caller can only default to mark the copied objects as regular
files, losing e.g. the executable bit (if set).

Add a new method readFileWithMode that returns the contents and mode of
the remote file. It supersedes the readFile method, that is marked as
deprecated.

Now callers can set correctly the file mode of the copied file.

Change-Id: I8fce01e4bc5707434c0cbc4aebbae1b6b64756f0
Signed-off-by: Ivan Frade <ifrade@google.com>

Many tests verify the contents of files in a try-with-resources
incantation that clutters the code.

Extract that verification to an "assertContents" method, that is easier
to read.

Change-Id: If430eac6f5b9ae352e42b2d43867ceb6cd618fbb
Signed-off-by: Ivan Frade <ifrade@google.com>

Patterns should treat \r in file names as normal characters

Change-Id: Ica3e0fa4a58acf5326db46bb28571fe5f20f6cd2
Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com>

ProtocolV2Parser has unit tests but protocol v0/v1 is not covered.

Change-Id: I96022e8f8eb60d4da748d1042474fd1efd67e882
Signed-off-by: Ivan Frade <ifrade@google.com>

Parsed requests represent object ids (SHA1) in ObjectId instances but tests
use strings for those ids because they are easier to define.

Create a custom matcher that hides the conversion from string to
ObjectId. Note that this reverses the existing code conversion (it was
transforming ObjectIds into string).

This produces more readable code, consistent with the other hamcrest
assertions.

Change-Id: I47ba1d25557d791fe74fb93c740ff7de9923cc00
Signed-off-by: Ivan Frade <ifrade@google.com>

This allows clients to use the --shallow-exclude parameter (producing a
"deepen-not <ref>" line when communicating with the server) in their fetch
commands when fetching against a JGit server using protocol v2.

Note that the implementation in this commit is somewhat inefficient, as
described in the TODO comment in DepthGenerator.

Change-Id: I9fad3ed9276b624d8f668356ffd99a067dc67ef7
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>

In C Git, when a client fetches with "git fetch --shallow-since=<date>
origin <ref>", and all commits reachable from <ref> are older than
<date>, the server dies with a message "no commits selected for shallow
requests". That is, (1) the --shallow-since filter applies to the commit
pointed to by the ref itself, and (2) there is a check that at least one
commit is not filtered out. (The pack-protocol.txt documentation does
not describe this, but the C implementation does this.)

The implementation in commit 1bb430dc21 ("UploadPack: support
deepen-since in protocol v2", 2018-09-27) does neither (1) nor (2), so
do both of these.

Change-Id: I9946327a71627626ecce34ca2d017d2add8867fc
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>

The request receives a list of capabilities and takes out the "agent" to
offer it on its own setter (getAgent).

Do this at parse time: when reading the line if the capability is
"agent" set it directly in the builder.

This makes the treatment of "agent" consistent in v0/v1 and v2.

Change-Id: Ie4f9f2cad8639adeeaef4921df49a30a8ce5b42f
Signed-off-by: Ivan Frade <ifrade@google.com>

All of the input lines passed to pre-push hook scripts must be properly
terminated by '\n', so that normal shell scripts like the git-supplied
pre-push.sample work properly, even when pushing just a single branch.

With the old code, hook scripts that use the following pattern didn't
process the last line, because 'read' has a non-zero exit status when
EOF is encountered:
  while read local_ref local_sha remote_ref remote_sha; do ... done

Change-Id: Id899662ed3fedef6c314fc4b2ddf91a6dcb98cbb
Signed-off-by: Markus Keller <markus.kell.r@gmail.com>

Before this commit, a force checkout would fail if there
were any conflicting files. After this commit, a force
checkout will overwrite the conflicting files, as expected.

Making this work required fixing a bug in DirCacheCheckout.
Before this commit, when DirCacheCheckout had
failOnConflict=false, it would delete all conflicting files
from the working copy and just leave them missing. After
this commit, DirCacheCheckout overwrites conflicting files
with the merge tree.

This change in DirCacheCheckout causes "reset --hard" and
"revert --abort" to behave as expected (previously they
would simply delete conflicting files, now they will be
overwritten from the merge tree).

Change-Id: If7e328ee792ef6511ab7d9c26d8d77c39210ec9f
Signed-off-by: Ned Twigg <ned.twigg@diffplug.com>

UploadPack.getPeerUserAgent() doesn't produce the expected results for
protocol v2 requests. In v2, the agent reported in the request (in an
"agent=" line) is not in the clientCapabilities but in a field on its
own. This makes getPeerUserAgent default to the transport user agent.

Making "agent" a shared property between protocol v0/v1 and v2 fixes the
problem, simplifies the function and harmonizes the implementation
between protocol versions.

In a follow up commit the "agent" will be identified on parsing time,
instead of taking it from the client capabilities.

Change-Id: Idf9825ec4e0b81a1458c8e3701f3e28aafd8a32a
Signed-off-by: Ivan Frade <ifrade@google.com>

In protocol v2, a command request can be followed by server options
(lines like "agent=<>" and "server-option=<>"), but current code
doesn't accept those lines.

Advertise the "server-option" capability, parse the lines and add
them to the request objects.

Other code in JGit can see this options and act accordingly via the
protocol v2 hooks.

This should not require any change in the client side.

Change-Id: If3946390f9cc02d29644b6ca52534b6f757bda9f
Signed-off-by: Ivan Frade <ifrade@google.com>

Error Prone reports that the unsynchronized method skip overrides the
synchronized method in ByteArrayInputStream [1].

This is a test class, so we can just suppress the warning as recommended
in [1].

Note that the suppression causes a warning in Eclipse, because it doesn't
recognize the "UnsynchronizedOverridesSynchronized" as a valid value for
the @SuppressWarnings annotation [2].

[1] https://errorprone.info/bugpattern/UnsynchronizedOverridesSynchronized
[2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=392045

Change-Id: I3e798b448211f1363729091f72fb0ef6a873e599
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

In FetchV0Request, the fields "wantsIds" and "options" are called
"wantIds" and "clientCapabilities". Those names describe them better.

Rename FetchV2Request fields to follow fetch v0. This will make easier
to extract a superclass later.

Take also the chance to polish the javadoc.

Change-Id: Ia17dbbab8084f39cc529fef9ca5c65e189073767
Signed-off-by: Ivan Frade <ifrade@google.com>

First-want line parsing accepts lines with an optional whitespace, when
the spec is strict requiring a white space.

Validate the line enforcing that there is a white space between oid and
capabilities list.

Change-Id: I45ada67030e0720f9b402c298be18c7518c799b1
Signed-off-by: Ivan Frade <ifrade@google.com>

In protocol v0/v1 pack negotiation, the first want line contains the
options the client wants in effect. This parsing is done in UploadPack
but it doesn't have any interaction with that class.

Move the code to its own class and package, mark the current one
as deprecated (it is public API) and add unit tests.

Take the chance to move the parsing code from the constructor to a
factory method, making the class a simple container of results.

Change-Id: I1757f535dda78a4111a1c12c3a3b455a4b6f0c51
Signed-off-by: Ivan Frade <ifrade@google.com>

Change-Id: Ibffb0dcaab6ec033b1b69aa27ccb7fdaa32fa925
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Previous commits block the addition to the repo of dangerous .gitmodules
files, but some could have been committed before those safeguards where
in place.

Add a check in DfsFsck to validate the .gitmodules files in the repo.
Use the same validation than the ReceivePack, translating the
results to FsckErrors.

Note that *all* .gitmodules files in the storage will be checked, not
only the latest version.

Change-Id: I040cf1f31a779419aad0292ba5e6e76eb7f32b66
Signed-off-by: Ivan Frade <ifrade@google.com>

The fsck test needs more detail about the error than an IOException
with an explanatory message.

Add an error identifier to the SubmoduleValidatorException and make
it the only throwable exception when parsing a file.

Change-Id: Ic3f0955b497e1681b25e681e1282e876cdf3d2c5
Signed-off-by: Ivan Frade <ifrade@google.com>

The main concern are submodule urls starting with '-' that could pass as
options to an unguarded tool.

Pass through the parser the ids of blobs identified as .gitmodules
files in the ObjectChecker. Load the blobs and parse/validate them
in SubmoduleValidator.

Change-Id: Ia0cc32ce020d288f995bf7bc68041fda36be1963
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

In order to validate .gitmodules files, we first need to find them
in the incoming pack.

Do it in the ObjectChecker stage. Check in the tree objects if they
point to a .gitmodules file and report the tree id and the .gitmodules
blob id.

This can be used later to check if the file is in the root of the
project and if the contents are good.

While we're here, make isMacHFSGit more accurate by detecting variants
of filenames that vary in case.

[jn: tweaked NTFS and HFS+ checking; added more tests]

Change-Id: I70802e7d2c1374116149de4f89836b9498f39582
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

In C git versions before 2.19.1, the submodule is fetched by running
"git clone <uri> <path>". A URI starting with "-" would be interpreted
as an option, causing security problems. See CVE-2018-17456.

Refuse to add submodules with URIs, names or paths starting with "-",
that could be confused with command line arguments.

[jn: backported to JGit 4.7.y, bringing portions of Masaya Suzuki's
 dotdot check code in v5.1.0.201808281540-m3~57 (Add API to specify
 the submodule name, 2018-07-12) along for the ride]

Change-Id: I2607c3acc480b75ab2b13386fe2cac435839f017
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>