Use the FilterSpec object so that less code has to know about the make-up of
FilterSpecs. When fields are added to FilterSpec, these pieces of code won't
need updating again.

Change-Id: I2b9e59a9926ff112faf62a3fa2d33c961a1779e5
Signed-off-by: Matthew DeVore <matvore@gmail.com>

Commit body contains the message that is not needed for reachability checks, and
takes up memory unnecessarily.

Change-Id: I0c7f6da249bf9c4fda9dc9e62e809322c68effce
Signed-off-by: Minh Thai <mthai@google.com>

Client will stop sending haves when the number of haves sent reaches maxhaves.

Change-Id: I1e5b1525be4c67f20a81ca24a2770c20eb5c1271
Signed-off-by: Minh Thai <mthai@google.com>

Remove it from

 * package private functions.

 * try blocks

 * for loops

this was done with the following python script:

$ cat f.py
import sys
import re
import os

def replaceFinal(m):
  return m.group(1) + "(" +  m.group(2).replace('final ', '') + ")"

methodDecl = re.compile(r"^([\t ]*[a-zA-Z_ ]+)\(([^)]*)\)")

def subst(fn):
  input = open(fn)
  os.rename(fn, fn + "~")

  dest = open(fn, 'w')
  for l in input:
    l = methodDecl.sub(replaceFinal, l)
    dest.write(l)
  dest.close()


for root, dirs, files in os.walk(".", topdown=False):
  for f in files:
    if not f.endswith('.java'):
      continue

    full = os.path.join(root, f)
    print full
    subst(full)

Change-Id: If533a75a417594fc893e7c669d2c1f0f6caeb7ca
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

Change-Id: Id924f79c8b2c720297ebc49bf9c5d4ddd6d52547
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

This is easier to type and makes it clearer that it only returns refs
and not the pseudo-refs returned by getAdditionalRefs. It also puts us
in a better position to add a method to the Repository class later
that delegates to this one without colliding with the existing
Repository#getAllRefs method that returns a Map<String, Ref>.

While at it, clarify the javadoc of getRefs and hasRefs to make the
same point.

Suggested-by: David Pursehouse <david.pursehouse@gmail.com>
Change-Id: I23497c66ac7b5e0c987b91efbc9e9cc29924ca66
Signed-off-by: Jonathan Nieder <jrn@google.com>

Currently to get all refs, callers must use:

  getRefsByPrefix(ALL)

Introduce getAllRefs, which does this, and migrate all existing
callers of getRefsByPrefix(ALL).

Change-Id: I7b1687c162c8ae836dc7db3ccc7ac847863f691d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Change-Id: I2d3e426a3391923f8a690ac68fcc33851f3eb419
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: I2e4544d715b4284f544a26f504d01159c5843160
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Teach UploadPack to advertise the filter capability and support a
"filter" line in the request, accepting blob sizes only, if the
configuration variable "uploadpack.allowfilter" is true. This feature is
currently in the "master" branch of Git, and as of the time of writing,
this feature is to be released in Git 2.17.

This is incomplete in that the filter-by-sparse-specification feature
also supported by Git is not included in this patch.

If a JGit server were to be patched with this commit, and a repository
on that server configured with RequestPolicy.ANY or
RequestPolicy.REACHABLE_COMMIT_TIP, a Git client built from the "master"
branch would be able to perform a partial clone.

Change-Id: If72b4b422c06ab432137e9e5272d353b14b73259
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>

Commit 9530c10192 (2018-02-11)
"Add a minimum negotiation feature for fetch" made fetch
negotiation cheaper for repos with huge numbers of
references (we are seeing a 15x reduction in maximum fetch
times for chromium/chromium/src on trans-Pacific links).
But it inadvertently broke the handling of stateless RPC
connections, so fix that here.

Change-Id: I0090aa76ffecc55801ebb833ac2e0c933a4a7c54
Signed-off-by: Terry Parker <tparker@google.com>

Android an Chrome have several repos with >300k refs. We sometimes see
negotiations of >100k rounds. This change provides a "minimal negotiation"
feature on the client side that limits how many "have" lines the client
sends. The client extracts the current SHA-1 values for the refs in its
wants set, and terminates negotiation early when all of those values have
been sent as haves. If a new branch is being fetched then that set will
be empty and the client will terminate after current default minimum
of two rounds.

This feature is gated behind a "fetch.useminimalnegotiation" configuration
flag, which defaults to false.

Change-Id: Ib12b095cac76a59da6e8f72773c4129e3b32ff2b
Signed-off-by: Terry Parker <tparker@google.com>

Change-Id: Id70e2c27e38bf12f87e66b28fbd0b06908494764
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Happily, most anonymous SectionParser implementations can be replaced
with FooConfig::new, as long as the constructor takes a single Config
arg. Many of these, the non-public ones, can in turn be inlined. A few
remaining SectionParsers can be lambdas.

Change-Id: I3f563e752dfd2007dd3a48d6d313d20e2685943a

Since the introduction of generic type parameter inference in Java 7,
it's not necessary to explicitly specify the type of generic parameters.

Enable the warning in Eclipse, and fix all occurrences.

Change-Id: I9158caf1beca5e4980b6240ac401f3868520aad0
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Set missingOverrideAnnotation=warning in Eclipse compiler preferences
which enables the warning:

  The method <method> of type <type> should be tagged with @Override
  since it actually overrides a superclass method

Justification for this warning is described in:

  http://stackoverflow.com/a/94411/381622

Enabling this causes in excess of 1000 warnings across the entire
code-base. They are very easy to fix automatically with Eclipse's
"Quick Fix" tool.

Fix all of them except 2 which cause compilation failure when the
project is built with mvn; add TODO comments on those for further
investigation.

Change-Id: I5772061041fd361fe93137fd8b0ad356e748a29c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Change-Id: I7c545d06b1bced678c020fab9af1382bc4416b6e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.

Change-Id: I24c8a600ec962d61d5f40abf73eac4203e115240
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>

C git 2.5 supports setting the equivalent of
RequestPolicy.REACHABLE_COMMIT with uploadpack.allowreachablesha1inwant.
Parse this into TransportConfig and use it from UploadPack. An explicitly
set RequestPolicy overrides the config, and the policy may still be
upgraded on a unidirectional connection to avoid races.

Change-Id: Id39771a6e42d8082099acde11249306828a053c0
Signed-off-by: Fredrik Medley <fredrik.medley@gmail.com>

Since git-core ff5effd (v1.7.12.1) the native wire protocol transmits
the server and client implementation and version strings using
capability "agent=git/1.7.12.1" or similar.

Support this in JGit and hang the implementation data off UploadPack
and ReceivePack.  On HTTP transports default to the User-Agent HTTP
header until the client overrides this with the optional capability
string in the first line.

Extract the user agent string into a UserAgent class under transport
where it can be specified to a different value if the application's
build process has broken the Implementation-Version header in the
JGit package.

Change-Id: Icfc6524d84a787386d1786310b421b2f92ae9e65

Change-Id: If90a6cca9a2ae80474f5ad023cb2a0a8905010be
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

RecievePack already honors fsck settings for safeForWindows and
safeForMacOS.  Allow those same checks to be performed during fetch
through a caller-configurable ObjectChecker.

Default the fetch fsck options to match the current platform, as
it can be reasonably assumed the repository will be accessed here.

Change-Id: I3c0f411fad209c6bd8fb9c4acf5c55a6799a6a2a

This avoids the server from referencing the client code directly.

Change-Id: Ie6ade781b5a689646ad8b0b2988ef2b544412195

Currently, Repository.getAllRefs() and Repository.getTags() silently
ignores an IOException and instead returns an empty map. Repository
is a public API and as such cannot be changed until the next major
revision change. Where possible, update the internal jgit APIs to
use the RefDatabase directly, since it propagates the error.

Change-Id: I4e4537d8bd0fa772f388262684c5c4ca1929dc4c

The NullProgressMonitor does not report progress anywhere. Inform the
server not to send progress by enabling the no-progress capability.

Change-Id: Id18dbc754c814d1a5534a284c947030bf201c569

Setting the walk and other fields to null will result in NPEs when the
user e.g. calls fetch on the connection, but at least the advertised
refs can be read like that without having a local repository.

Bug: 413389
Change-Id: I39c8363e81a1c7e6cb3412ba88542ead669e69ed
Signed-off-by: Robin Stocker <robin@nibor.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>

This protocol capability, new in C git 1.8.2, corresponds to
RequestPolicy.TIP, so advertise it if that request policy was set.

Change-Id: I0d52af8a7747e951a87f060a5124f822ce1b2b26

Allow users to provide their OutputStream (via Transport#
push(monitor, refUpdates, out)) so that server messages can be written
to it (in SideBandInputStream) while they're coming in.

CQ: 7065
Bug: 398404
Change-Id: I670782784b38702d52bca98203909aca0496d1c0
Signed-off-by: Andre Dietisheim <andre.dietisheim@gmail.com>
Signed-off-by: Chris Aniszczyk <zx@twitter.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This breaks all existing callers once. Applications are not supposed
to build against the internal storage API unless they can accept API
churn and make necessary updates as versions change.

Change-Id: I2ab1327c202ef2003565e1b0770a583970e432e9

A few classes such as Constanrs are marked with @SuppressWarnings, as are
toString() methods with many liternal, but otherwise $NLS-n$ is used for
string containing text that should not be translated. A few literals may
fall into the gray zone, but mostly I've tried to only tag the obvious
ones.

Change-Id: I22e50a77e2bf9e0b842a66bdf674e8fa1692f590

Change-Id: I0a86ce0e393dfde9bb27f0b29e036e76c856396e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Chris Aniszczyk <zx@twitter.com>

When fetching over smart HTTP the InputStream that gets fed into
a PackParser doesn't really support EOF at the end of the pack. It
instead tries to make a new HTTP request, which fails because there
is no request body currently buffered by the client.

Make EOF work correctly on the end of an HTTP derived InputStream
for the pack by denoting no more requests are expected as the higher
level code is now consuming the pack (or side-band embedded pack).
Smart HTTP support doesn't automatically enqueue execute support onto
the end of the UnionInputStream, which allows the UnionInputStream
to correctly reflect EOF when the HTTP response is consumed.

Change-Id: I975f1ab1c81ab1c1af925716970088bc7b8d6b1a

Change-Id: I763590a45d75f00a09097ab6f89581a3bbd3c797

Change-Id: Ib475dfc087705ea40e1db37ec9ef28a67fde6d72

If the no-done capability was enabled on the connection, don't
queue up the state vector again once the ACK %s ready message
is observed from the remote. The pack will be following in this
response stream, so the state vector is no longer required.

Change-Id: I7bd1e76957cb58c7ff1cdaeef227f1b02a7e5d24
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Change-Id: I926838058c1de2146e22faa08570406600457acb
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Smart HTTP clients may request both multi_ack_detailed and no-done in
the same request to prevent the client from needing to send a "done"
line to the server in response to a server's "ACK %s ready".

For smart HTTP, this can save 1 full HTTP RPC in the fetch exchange,
improving overall latency when incrementally updating a client that
has not diverged very far from the remote repository.

Unfortuantely this capability cannot be enabled for the traditional
bi-directional connections.  multi_ack_detailed has the client sending
more "have" lines at the same time that the server is creating the
"ACK %s ready" and writing out the PACK stream, resulting in some race
conditions and/or deadlock, depending on how the pipe buffers are
implemented.  For very small updates, a server might actually be able
to send "ACK %s ready", then the PACK, and disconnect before the
client even finishes sending its first batch of "have" lines.  This
may cause the client to fail with a broken pipe exception.  To avoid
all of these potential problems, "no-done" is restricted only to the
smart HTTP variant of the protocol.

Change-Id: Ie0d0a39320202bc096fec2e97cb58e9efd061b2d
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

When fetching from a remote peer, consider all of the refs of any
alternate repository to be reachable locally, in addition to the refs
of the local repository.  This mirrors the push protocol and may avoid
unnecessary object transfer when the local repository is empty, but
its alternate and the remote share a lot of common history.

Junio C Hamano recently proposed a similar change to C Git's fetch
client, in order to work around a performance bug I identified when
fetching between two repositories that actually shared the same
alternate repository on the local system.

Change-Id: Iffb0b70e1223901ce2caac3b87ba7e0d6634d265
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If the client is only following the remote repository and has not
created any new non-common commits, the client will wind up sending
a "have %s" line for each tag in the repository.  For some projects
like git.git, this is 339 tags and growing, resulting in more than
16 KiB needing to be POSTed over 12 HTTP requests.

Teach UploadPack (server side) to always execute the okToGiveUp()
logic at least once per negotiation round to determine if the server
can compute a pack right now.  If it can, shove in an "ACK %s ready"
message to tell the client this and try to prevent receiving ancient
tags in future negotiation rounds.

Teach BasePackFetchConnection (client side) to honor a "ACK %s ready"
from the remote and break out of its SEND_HAVE loop once the remote
knows it can create a pack.  This avoids sending the remaining 307
tags of git.git.

These two changes together reduce the number of HTTP RPCs from 13
down to 3 in order to fetch from git.git over smart HTTP.  If either
side is missing the change, the older behavior (and its 13 RPCs)
is used.

Change-Id: I64736318fd0abf9ee5e56bd0b737707adb580b37
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

If the RevFilter doesn't actually require the commit body,
we shouldn't reparse it if the body was disposed.  This happens
often inside of UploadPack during common ancestor negotation, the
RevWalk is reset and re-run over roughly the same commit space,
but the bodies are discarded because the commit message is not
relevant to the process.

Change-Id: I87b6b6a5fb269669867047698abf718d366bd002
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

By moving the logic that parses a pack stream from the network (or
a bundle) into a type that can be constructed by an ObjectInserter,
repository implementations have a chance to inject their own logic
for storing object data received into the destination repository.

The API isn't completely generic yet, there are still quite a few
assumptions that the PackParser subclass is storing the data onto
the local filesystem as a single file.  But its about the simplest
split of IndexPack I can come up with without completely ripping
the code apart.

Change-Id: I5b167c9cc6d7a7c56d0197c62c0fd0036a83ec6c
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>

These classes need to be visible if an application wants to define
its own native pack based protocol embedded within another layer,
much like we already support for smart HTTP.

Change-Id: I7e2ac3ad01d15b94d340128a395fe0b2f560ff35
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Update a number of calling sites of RevWalk to ensure the walker's
internal ObjectReader is released after the walk is no longer used.
Because the ObjectReader is likely to hold onto a native resource
like an Inflater, we don't want to leak them outside of their
useful scope.

Where possible we also try to share ObjectReaders across several
walk pools, or between a walker and a PackWriter.  This permits
the ObjectReader to actually do some caching if it felt inclined
to do so.

Not everything was updated, we'll probably need to come back and
update even more call sites, but these are some of the biggest
offenders.  Test cases in particular aren't updated.  My plan is to
move most storage-agnostic tests onto some purely in-memory storage
solution that doesn't do compression.

Change-Id: I04087ec79faeea208b19848939898ad7172b6672
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

This move isolates all of the local file specific implementation code
into a single package, where their package-private methods and support
classes are properly hidden away from the rest of the core library.

Because of the sheer number of files impacted, I have limited this
change to only the renames and the updated imports.

Change-Id: Icca4884e1a418f83f8b617d0c4c78b73d8a4bd17
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The strings are externalized into the root resource bundles.
The resource bundles are stored under the new "resources" source
folder to get proper maven build.

Strings from tests are, in general, not externalized. Only in
cases where it was necessary to make the test pass the strings
were externalized. This was typically necessary in cases where
e.getMessage() was used in assert and the exception message was
slightly changed due to reuse of the externalized strings.

Change-Id: Ic0f29c80b9a54fcec8320d8539a3e112852a1f7b
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>

Any messages received on side band #2 that aren't scraped as a
progress message into our ProgressMonitor are now forwarded to a
buffer which is later included into the OperationResult object.
Application callers can use this buffer to present the additional
messages from the remote peer after the push or fetch operation
has concluded.

The smart push connections using the native send-pack/receive-pack
protocol now request side-band-64k capability if it is available
and forward any messages received through that channel onto this
message buffer.  This makes hook messages available over smart HTTP,
or even over SSH.

The SSH transport was modified to redirect the remote command's
stderr stream into the message buffer, interleaved with any data
received over side band #2.  Due to buffering between these two
different channels in the SSH channel mux itself the order of any
writes between the two cannot be ensured, but it tries to stay close.

The local fork transport was also modified to redirect the local
receive-pack's stderr into the message buffer, rather than going to
the invoking JVM's System.err.  This gives applications a chance
to log the local error messages, rather than needing to redirect
their JVM's stderr before startup.

To keep things simple, the application has to wait for the entire
operation to complete before it can see the messages.  This may
be a downside if the user is trying to debug a remote hook that is
blocking indefinitely, the user would need to abort the connection
before they can inspect the message buffer in any sort of UI built
on top of JGit.

Change-Id: Ibc215f4569e63071da5b7e5c6674ce924ae39e11
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

Typically we refer to the raw InputStream (the stream without the
pkt-line headers on it) as rawIn, and the pkt-line header variant
as pckIn.  Refactor our fields to reflect that.  To ensure these
are actually the same underlying InputStream, we now create our own
PacketLineIn wrapper around the supplied raw InputStream.  Its a
very low-cost object since it has only the 4 byte length buffer.

Instead of hardcoding the header length as 5, use the constant from
SideBandOutputStream.  This makes it a bit more clear what we are
consuming, exactly here.

Change-Id: Iebd05538042913536b88c3ddc3adc3a86a841cc5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

During fetch over http:// clients now try to take advantage of
the info/refs?service=git-upload-pack URL to determine if the
remote side will support a standard upload-pack command stream.
If so each block of 32 have lines is sent in one POST request,
prefixed by all of the 'want' lines and any previously discovered
common bases as 'have' lines.

During push over http:// clients now try to take advantage of
the info/refs?service=git-receive-pack URL to determine if the
remote side will support a standard receive-pack command stream.
If so, commands are sent along with their pack in a single HTTP
POST request.

Bug: 291002
Change-Id: I8c69b16ac15c442e1a4c3bd60b4ea1a47882b851
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

The multi_ack_detailed extension breaks out the "ACK %s continue" status
code into "ACK %s common" and "ACK %s ready" states, making it easier to
discover which objects are truely common, and which objects are simply
on a chain the server doesn't care learning about.

Change-Id: Ie8e907424cfbbba84996ca205d49eacf339f9d04
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

As discussed on the egit-dev mailing list, we prefer not to have
trailing whitespace in our source code.  Correct all currently
offending lines by trimming them.

Change-Id: I002b1d1980071084c0bc53242c8f5900970e6845
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>