AdvertiseRefsHook is used to limit the visibility of the refs in Gerrit.
If this hook is not called, then all refs are treated as visible,
causing the server to serve commits reachable from branches the client
should not be able to access, if asked to via a request naming a guessed
object id.

This bug was introduced in v2.0.0.201206130900-r~123 (Modify refs in
UploadPack/ReceivePack using a hook interface, 2012-02-08).  Stateful
bidirectional transports are not affected.

Fix it by moving the AdvertiseRefsHook call to
getAdvertisedOrDefaultRefs, ensuring the hook is called in all cases.

[jn: backported to stable-4.5 by splitting out tests and the protocol v2
 specific parts]

Change-Id: I159f396216354f2eda3968d17802e166d8c8ec2d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: Id70e2c27e38bf12f87e66b28fbd0b06908494764
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Currently, unless RequestPolicy#ANY is used, UploadPack rejects all
non-commit "want" lines unless they were advertized. This is fine,
except when "uploadpack.allowreachablesha1inwant" is true
(corresponding to RequestPolicy#REACHABLE_COMMIT), in which case one
would expect that "want"-ing anything reachable would work.

(There is no restriction that "want" lines must only contain commits -
it is allowed for refs to directly point to trees and blobs, and
requesting for them using "want" lines works.)

This commit has been written to avoid performance regressions as much
as possible. In the usual (and currently working) case where the only
unadvertized things requested are commits, we do a standard RevWalk in
order to avoid incurring the cost of loading bitmaps. However, if
unadvertized non-commits are requested, bitmaps are used instead, and
if there are no bitmaps, a WantNotValidException is thrown (as is
currently done).

Change-Id: I68ed4abd0e477ff415c696c7544ccaa234df7f99
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>

Shadow commits in the RevWalk in the UploadPack object may cause the
UNINTERESTING flag not being carried over to their parents commits since
they were marked NO_PARENTS during the assumeShallow or
initializeShallowCommits call.

A new RevWalk needs to be created for this reason, but instead of
creating a new RevWalk from Repository, we can reuse the ObjectReader in
the RevWalk of UploadPack to load objects.

Change-Id: Ic3fee0512d35b4f555c60e696a880f8b192e4439
Signed-off-by: Zhen Chen <czhen@google.com>

This is continuation from https://git.eclipse.org/r/#/c/94249/. When an
error happens, we might not read the entire stream. Consume the request
body before we flush the buffer.

Change-Id: Ia473a04ace600653b2d1f2822e3023570d992410
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>

Commit db77610 ensured that all refs/tags commits are added to the
primary GC pack. It did that by adding all of the refs/tags commits
to the primary GC pack PackWriter's "interesting" object set.

Unfortunately, all commit objects in the "interesting" set are
selected as commits for which bitmap indices will be built. In a
repository like chromium with lots of tags, this changed the number of
bitmaps created from <700 to >10000. That puts huge memory pressure on
the GC task.

This change restores the original behavior of ignoring tags when
selecting commits for bitmaps.

In the "uninteresting" set, commits for refs/heads and refs/tags for
unannotated tags can not be differentiated. We instead identify
refs/tags commits by passing their ObjectIds as a new "noBitmaps"
parameter to the PackWriter.preparePack() methods.
PackWriterBitmapPreparer.setupTipCommitBitmaps() can then use that
"noBitmaps" parameter to exclude those commits.

Change-Id: Icd287c6b04fc1e48de773033fe432a9b0e904ac5
Signed-off-by: Terry Parker <tparker@google.com>

Introduced via commit 3b2508b514.

Change-Id: I2b6175c095aea2868a8c302103095accde5170e3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>

This is a continuation from https://git.eclipse.org/r/#/c/4716/. For a
non-bidirectional request, we need to consume the request before writing
any response. In UploadPack, we write "shallow"/"unshallow" responses
before parsing "have" lines. This has happened not to be a problem most
of the time in the smart HTTP protocol because the underlying
InputStream has a 32 KiB buffer in SmartOutputStream.

Change-Id: I7c61659e7c4e8bd49a8b17e2fe9be67bb32933d3
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>

Since the introduction of generic type parameter inference in Java 7,
it's not necessary to explicitly specify the type of generic parameters.

Enable the warning in Eclipse, and fix all occurrences.

Change-Id: I9158caf1beca5e4980b6240ac401f3868520aad0
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Set missingOverrideAnnotation=warning in Eclipse compiler preferences
which enables the warning:

  The method <method> of type <type> should be tagged with @Override
  since it actually overrides a superclass method

Justification for this warning is described in:

  http://stackoverflow.com/a/94411/381622

Enabling this causes in excess of 1000 warnings across the entire
code-base. They are very easy to fix automatically with Eclipse's
"Quick Fix" tool.

Fix all of them except 2 which cause compilation failure when the
project is built with mvn; add TODO comments on those for further
investigation.

Change-Id: I5772061041fd361fe93137fd8b0ad356e748a29c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Change-Id: I7c545d06b1bced678c020fab9af1382bc4416b6e
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

UploadPackLogger is deprecated but will not be removed until
JGit version 5.0.

Suppress the unavoidable deprecation warnings on usages of the
interface that are kept for backwards compatibility.

Add a TODO so that we don't forget to remove it in 5.0.

Change-Id: Id248002b9bdf23db192427196d54c722a012106c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

Since 84d2738ff2 (Don't skip want validation when the client sends no
haves, 2013-06-21), this branch is not taken.  Process the
"shallow"s anyway as a defensive measure in case the code path gets
revived.

Change-Id: Idfb834825d77f51e17191c1635c9d78c78738cfd
Signed-off-by: Jonathan Nieder <jrn@google.com>

d385a7a5e5 (Shallow fetch: Respect "shallow" lines, 2016-08-03) forgot
that UploadPack wasn't passing a DepthWalk to PackWriter in the first
place.  As a result, shallow clones fail:

  java.lang.IllegalArgumentException: Shallow packs require a DepthWalk
        at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:756)
        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1497)
        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1381)
        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:774)
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:667)
        at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:191)

Change-Id: Ib0d8c2946eebfea910a2b767fb92e23da15d4749

cgit changed the --depth parameter to mean the total depth of history
rather than the depth of ancestors to be returned [1]. JGit still uses
the latter meaning, so update it to match cgit.

depth=0 still means a non-shallow clone. depth=1 now means only the
wants rather than the wants and their direct parents.

This is accomplished by changing the semantic meaning of "depth" in
UploadPack and PackWriter to mean the total depth of history desired,
while keeping "depth" in DepthWalk.{RevWalk,ObjectWalk} to mean
the depth of traversal. Thus UploadPack and PackWriter always
initialize their DepthWalks with "depth-1".

[1] upload-pack: fix off-by-one depth calculation in shallow clone
https://code.googlesource.com/git/+/682c7d2f1a2d1a5443777237450505738af2ff1a

Change-Id: I87ed3c0f56c37e3491e367a41f5e555c4207ff44
Signed-off-by: Terry Parker <tparker@google.com>

A RefAdvertiser writing to the network includes both the reference's
ObjectId and its peeled ObjectId in the advertised set.  In smart HTTP
negotiation requests may bypass the RefAdvertiser and quickly build
the set based on current refs; include the peeled ObjectIds to match
behavior with the normal bidirectional protocols on git:// and SSH.

Change-Id: I5371bed60da36e8d12c4ad9a5c1d91a0f0ad486b

This field was being set twice within the block.  Setting it just once
is sufficient.  writeString() does not examine the field so it is fine
to set it after the call.

Change-Id: Ib4c74df4f1304e9df3015885bf360bf0d7bc6ca2

Change-Id: Ibab89fddda7ccfe3390348aa129ab2e5e3b7f538

Now if refs are unreadable when serving an upload pack the handler
will fail due to the actual underlying failure. Previously all wants
would be rejected as invalid because Repository.getAllRefs() returned
an empty map.

Testing this required a new subclass of InMemoryRepository so that
an IOException could be injected at the correct time.

Signed-off-by: Michael Edgar <adgar@google.com>
Change-Id: Iac708b1db9d0ccce08c4ef5ace599ea0b57afdc0

Capture the internal "want X not valid" state as a specific subclass
of PackProtocolException, allowing this to be more easily identified
in server stack traces and wrapper application code.

Change-Id: I4b1adb7497f396432da420b0f600ad25a261f912

If the client sends a SHA-1 that the server does not recognize echo
this back to the client with an explicit error message instead of
the generic "internal server error".

This was always the intent of the implementation but it was being
dropped on smart HTTP due to the UploadPackServlet catching the
PackProtocolException, discarding the buffered message UploadPack
meant to send, and sending along a generic message instead.

Change-Id: I8d96b064ec655aef64ac2ef3e01853625af32cd1

This reverts commit bbcbcab8d3.
These classes were part of the public API and should not be removed
until JGit 5.0.

Change-Id: Ife4bee69f82151de6ef8ea1a4c6c146d91bbf0d5

These classes make improper use of internal classes in the public API
and were replaced by corresponding classes in the JGit 4.1 release.

Change-Id: I3d474210e49089aa788314b4e08f505f0d26619b
Signed-off-by: Terry Parker <tparker@google.com>

If a client mistakenly tries to send a tag object as a shallow line
JGit blindly assumes this is a commit and tries to parse the tag
buffer using the commit parser. This can cause an obtuse error like:

  InvalidObjectIdException: Invalid id: t c0ff331234...

The "t" comes from the "object c0ff331234..." line of the tag tring
to be parsed as though it where the "tree" line of a commit.

Run any client supplied shallow lines through the RevWalk to lookup
the object types. Fail fast with a protocol exception if any of them
are non-commit.

Skip objects not known to this repository. This matches behavior
with git-core's upload-pack, which sliently skips over any shallow
line object named by the client but not known by the server.

Change-Id: Ic6c57a90a42813164ce65c2244705fc42e84d700

Change-Id: I976102e054369b045340d4b9209984b5243e3750
Signed-off-by: Terry Parker <tparker@google.com>

UploadPackLogger is incorrectly named--it can be used to trigger any
post upload action, such as GC/compaction. This change introduces
PostUploadHook/PostUploadHookChain to replace
UploadPackLogger/UploadPackLoggerChain and deprecates the latter.

It also introduces PackStatistics as a replacement for
PackWriter.Statistics, since the latter is not public API.
It changes PackWriter to use PackStatistics and reimplements
PackWriter.Statistics to delegate to PackStatistics.

Change-Id: Ic51df1613e471f568ffee25ae67e118425b38986
Signed-off-by: Terry Parker <tparker@google.com>

C git 2.5 supports setting the equivalent of
RequestPolicy.REACHABLE_COMMIT with uploadpack.allowreachablesha1inwant.
Parse this into TransportConfig and use it from UploadPack. An explicitly
set RequestPolicy overrides the config, and the policy may still be
upgraded on a unidirectional connection to avoid races.

Change-Id: Id39771a6e42d8082099acde11249306828a053c0
Signed-off-by: Fredrik Medley <fredrik.medley@gmail.com>

Change-Id: I4e79cefe15037df0e1c520956bf2482240e31a7d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

- use try-with-resource where possible
- replace use of deprecated release() by close()

Change-Id: I0f139c3535679087b7fa09649166bca514750b81
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Since git-core ff5effd (v1.7.12.1) the native wire protocol transmits
the server and client implementation and version strings using
capability "agent=git/1.7.12.1" or similar.

Support this in JGit and hang the implementation data off UploadPack
and ReceivePack.  On HTTP transports default to the User-Agent HTTP
header until the client overrides this with the optional capability
string in the first line.

Extract the user agent string into a UserAgent class under transport
where it can be specified to a different value if the application's
build process has broken the Implementation-Version header in the
JGit package.

Change-Id: Icfc6524d84a787386d1786310b421b2f92ae9e65

The clone or fetch depth is a valuable bit of information
for access logging. Create a public getter to faciliate access.
A precondition check prevents unintentional misuse when the
data isn't valid yet.

Change-Id: I4603d5fd3bd4a767e3e2419b0f2da3664cfbd7f8
Signed-off-by: David Pletcher <dpletcher@google.com>

cgit has this feature for some time. This will teach JGit to send symbolic refs,
too.

Change-Id: I7cb2ab4e6d31a838a0af92eac64535fdb66ed74a
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>

If the packer fails, still obtain the stats and make them available
to the logger and the caller. Failures can frequently happen when
a client disconnects in the middle of a pack stream. Server admins
may still want to examine the timing metrics from counting and
compressing phases.

Change-Id: Iceae4f68b5473f4223d85c9edfb57837fc818eed

In certain cases a JGit server updating an existing shallow client
selected a common ancestor that was behind the shallow edge of
the client. This allowed the server to assume the client had some
objects it did not have and allowed creation of pack deltas the
client could never inflate.

Any commit the client has advertised as shallow must be treated
by UploadPack server as though it has no parents. With no parents
the walker cannot visit graph history the client does not have,
and PackWriter cannot consider delta base candidates the client
is lacking.

Change-Id: I4922b9354df9f490966a586fb693762e897345a2

This avoids the server from referencing the client code directly.

Change-Id: Ie6ade781b5a689646ad8b0b2988ef2b544412195

Currently, Repository.getAllRefs() and Repository.getTags() silently
ignores an IOException and instead returns an empty map. Repository
is a public API and as such cannot be changed until the next major
revision change. Where possible, update the internal jgit APIs to
use the RefDatabase directly, since it propagates the error.

Change-Id: I4e4537d8bd0fa772f388262684c5c4ca1929dc4c

Change-Id: I5e80b3befca5cf1dcb06075862d6d48e6491cc0f

Change-Id: I9754e2124c0fe6ad2dbde5597c3ed10f1c3efef5
Signed-off-by: Lars Vogel <Lars.Vogel@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

With reference hiding, it is possible for a repository to appear
empty when all refs are hidden. This causes capabilities to not be
advertised either, since they are published with the first reference,
breaking fetch by SHA1 support.

Always advertise the capabilites by publishing the symbolic capabilities
reference when the repository has no references to advertise (similar to
the receive service).

Change-Id: I8060e430ee03571dc51239e702864c85e888505c

UploadPack can be invoked with no capabilities selected by the
client if the client is an ancient version of Git that nobody in
their right mind should still be using. Or if the client is very
broken and does not want to use any of the newer features added to
the protocol since its inception.

Change-Id: I3baa6f90e6a41a37a8eab8449a3cc41f4efcb91a

Instead of RevObject list, this allows a custom request validator to be called
on SHA-1's corresponding to objects that may not exist in repository storage

Change-Id: I19bb667beff0d0c144150a61d7a1dc6c9703be7f
Signed-off-by: Greg Hill <greghill@google.com>

Make the existing concrete implementations public as well so custom
implementations may delegate to them where appropriate. Treat all custom
implementations as providing allow-tip-sha1 in want.

Change-Id: If386fe25c0d3b4551a97c16a22350714453b03e9

Associate each RequestPolicy with an implementation of a
RequestValidator interface that contains the validation logic. The
checkWants method is only called if there are wants that were not
advertised, since clients may always request any advertised want
according to the git protocol. Calling the method only once at the
end of parsing the want list also means policy implementations can be
stateful, unlike the previous switch statement inside a loop.

For the special handling of unidirectional pipes, simply check
isBiDirectional() and delegate to other implementations if necessary.

Change-Id: I52a174999ac3a5aca46d3469cb0b81edd1710580

Teach TransportConfig to respect uploadpack.hiderefs, which is new in
C git 1.8.2.

Change-Id: Id12e7f00b9a60258e996410f67fa10616459f53f

C git 1.8.2 supports setting the equivalent of RequestPolicy.TIP with
uploadpack.allowtipsha1. Parse this into TransportConfig and use it
from UploadPack. An explicitly set RequestPolicy overrides the config,
and the policy may still be upgraded on a unidirectional connection to
avoid races.

Defer figuring out the effective RequestPolicy to later in the
process. This is a minor semantic change to fix a bug: previously,
calling setRequestPolicy(ADVERTISED) _after_ calling
setBiDirectionalPipe(true) would have reintroduced the race condition
otherwise fixed by 01888db892.

Change-Id: I264e028a76574434cecb34904d9f5944b290df78

This protocol capability, new in C git 1.8.2, corresponds to
RequestPolicy.TIP, so advertise it if that request policy was set.

Change-Id: I0d52af8a7747e951a87f060a5124f822ce1b2b26

Users of UploadPack may set a custom RefFilter or AdvertisedRefsHook
that limits which refs are advertised, but clients may learn of a
SHA-1 that the server should have as a ref tip through some
alternative means. Support serving such objects from the server side
with a new RequestPolicy.

As with ADVERTISED, we need a special relaxed RequestPolicy to allow
commits reachable from the set of valid tips for unidirectional
connections.

Change-Id: I0d0cc4f8ee04d265e5be8221b9384afb1b374315

Change-Id: I5e80b3befca5cf1dcb06075862d6d48e6491cc0f

Change-Id: I7f2c155cab4b5be55f7e849ff2595c8c5d804f05
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Change-Id: I9304cb3a6502ceb0cba21eb841e466f932ab3c01