UploadPack: support custom packfile-to-URI mapping
Teach UploadPack to take a provider of URIs corresponding to cached
packs. When fetching, if the client supports the packfile-uri feature,
and if such a cached pack were to be streamed, instead send the
corresponding URI.
This packfile-uri feature is implemented in the jt/fetch-cdn-offload
branch of Git. There is interest in this feature [1], but it is not yet
merged.
[1] https://public-inbox.org/git/cover.1552073690.git.jonathantanmy@google.com/
Change-Id: I9a32dae131c9c56ad2ff4a8a9638ae3b5e44dc15
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Cache user global and system-wide git configurations
So far the git configuration and the system wide git configuration were
always reloaded when jgit accessed these global configuration files to
access global configuration options which are not in the context of a
single git repository. Cache these configurations in SystemReader and
only reload them if their file metadata observed using FileSnapshot
indicates a modification.
Change-Id: I092fe11a5d95f1c5799273cacfc7a415d0b7786c
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
TransportHttp: resolve redirect URIs from the "Location" header
The "Location" header in a redirect response may contain a relative
URI. Resolve it against the URI the request was made.
Bug: 550033
Change-Id: I29de07dfbbbc794090821b7c190cb2cf662c5a60
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
[error prone] fix ReferenceEquality warning in static equals methods
Implement a helper method suppressing the ReferenceEquality error prone
warning and use it to fix this warning in static equals methods where
this comparison is used to implement fast path of static equals
implementation.
See https://errorprone.info/bugpattern/ReferenceEquality
Change-Id: I33538a3406007d24efec3a504e031ca1069572ed
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
[error prone] suppress AmbiguousMethodReference in AnyObjectId
Move the implementation of the static equals() method to a new method
and suppress the error. Deprecate the old method to signal that we
intend to remove it in the next major release.
See https://errorprone.info/bugpattern/AmbiguousMethodReference
Change-Id: I5e29c97f4db3e11770be589a6ccd785e2c9ac7f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
These are useful to avoid typos, and also for tab completion.
Change-Id: I0f2d267e46b36bc40297c9657c447f3fd8b9f831
Signed-off-by: David Turner <dturner@twosigma.com>
In a subsequent patch, in some cases, PackWriter#writePack will be
responsible for both the "packfile-uris" and "packfile" sections,
meaning that (in these cases) it must write the "packfile" section
header itself.
In preparation for that patch, move the writing of the "packfile"
section header closer to the invocation of PackWriter#writePack when the
entire fetch response is configured to use the sideband. This means that
"packfile" is written *after* objects are counted (and progress messages
sent to the client in sideband 2) when the "sideband-all" feature is
used (whether "packfile-uris" is used or not), and written *before*
objects are counted otherwise.
Having code to write "packfile" in two places is unfortunate but
necessary. When "sideband-all" is not used, object counting has to
happen after "packfile" is written, because "packfile" activates the
sideband that allows counting progress to be transmitted. When
"packfile-uris" is used, object counting has to happen before "packfile"
is written, because object counting determines whether to send
"packfile-uris" or "packfile". When "sideband-all" is used but
"packfile-uris" is not used, either way works; this commit uses
"packfile-uris" behavior in this case.
Also make the naming of the sideband-activating methods in PacketLineOut
more consistent.
Change-Id: Ifbfd26cc26af10c41b77758168833702d6983df1
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
PreUploadHookChain: Use list instead of array internally
The newly introduced ProtocolV2HookChain is implemented using lists
instead of arrays.
Update PostUploadHookChain to keep the hook chains implementation
consistent.
Change-Id: I5ae0c923f117ac48558a989464f5d5d868d81f76
Signed-off-by: Ivan Frade <ifrade@google.com>
PostUploadHookChain: Use a list instead of array internally
The newly introduced ProtocolV2HookChain is implemented using lists
instead of arrays.
Update PostUploadHookChain to keep hook chain implementations
consistent.
Change-Id: Ic5694feab943e8949896b93103dbf427716c9bd7
Signed-off-by: Ivan Frade <ifrade@google.com>
JschConfigSessionFactory: [findbugs] Set synchronized
Set synchronized to make the config access consistent.
> Inconsistent synchronization of
org.eclipse.jgit.transport.JschConfigSessionFactory.config; locked 80%
of time
In order to make JschConfigSessionFactory threadsafe, synchronize this
method as well.
Change-Id: I32d1bfc2e98363d254992144e795ce72fe1e8846
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
A caller cannot install a second hook in the UploadPack without
overwriting whatever is already there.
Offer a method to get the current protocol v2 hook, so it can be chained
with new hooks.
Change-Id: Icb06f94ec52b8c8714f509b5b8622d6db42960fb
Signed-off-by: Ivan Frade <ifrade@google.com>
ProtocolV2HookChain: Allow to create a chain of protocol V2 hooks
UploadPack only supports one protocol-v2 hook. There are already cases
where more than one is needed.
Offer a Chain class to compose ProtocolV2Hooks, as other hooks do. It
looks like a single hook but it calls all its members.
Change-Id: Idd173ca7df6672079ac0de03c67f77abac376538
Signed-off-by: Ivan Frade <ifrade@google.com>
Use Instant instead of milliseconds for filesystem timestamp handling
This enables higher file timestamp resolution on filesystems like ext4,
Mac APFS (1ns) or NTFS (100ns) providing high timestamp resolution on
filesystem level.
Note:
- on some OSes Java 8,9 truncate milliseconds, see
https://bugs.openjdk.java.net/browse/JDK-8177809, fixed in Java 10
- UnixFileAttributes truncates timestamp resolution to microseconds when
converting the internal representation to FileTime exposed in the API,
see https://bugs.openjdk.java.net/browse/JDK-8181493
- WindowsFileAttributes also provides only microsecond resolution
Change-Id: I25ffff31a3c6f725fc345d4ddc2f26da3b88f6f2
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Allow the client to specify "sideband-all" in a fetch v2 request,
indicating that the whole response is to be multiplexed (with a sideband
indicator on every non-flush and non-delim pkt) instead of only the
packfile being multiplexed. This allows, for example, progress messages
to be sent at any point in the response.
This implements the "sideband-all" feature documented in
Documentation/technical/protocol-v2.txt in Git.
Change-Id: I3e7f21c88ff0982b1b7ebb09c9ad6c742c4483c8
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
It is difficult to track what's happening with the pckOut instance
field, so replace it with a local variable in #upload instead.
Change-Id: Ibd9225b28334b7133eccdc6d82b26fc96cbde299
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Move ServiceMayNotContinueException handling code from sendPack
All other exceptions are handled in a wrapped sendPack method.
Consolidate the error handling code.
Change-Id: Ieac0ce64960534d009d1e6b025130b021b744794
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
By doing this, exceptions thrown by sendPack are also covered by the
same code.
Change-Id: I3509f2d832af1410f307e931577e4d07e32b014e
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
When another exception is thrown while handling another exception, that
exception can be attached to the original exception since Java 7
(Throwable#getSuppressed). Attach the secondary exception to the
original exception instead of throwing it away.
Change-Id: Ia093b8207714f2638e0343bc45a83d4342947505
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
If a client clones with "--filter=blob:none", the checkout that "git
clone" automatically does causes the client to fetch all blobs at HEAD.
When fetching from a non-bitmapped repository, this will fail if an
object walk is ever needed, because JGit currently rejects such requests
- see the commit message of d3021788d2 ("Use bitmaps for non-commit
reachability checks", 2017-11-10) for more information.
Rejecting such requests in the absence of bitmaps is probably
overzealous: it is true that the server would prefer to have bitmaps in
this case, but there might be a small proportion of repos (for example,
very small repos or newly created ones) that do not have bitmaps, yet
the server would still like to have partial clones for them.
So, allow such requests, performing the object walk reachability check
if necessary. Limit this to servers with "uploadpack.allowFilter"
configured, so that servers wanting to support partial clone have this
functionality, and servers that do not support partial clone do not have
to pay the object walk reachability check cost.
Change-Id: I51964bafec68696a799625d627615b4f45ddbbbf
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
[Java 11] Remove dependency on javax.xml.bind package
Replace javax.xml.bind.DatatypeConverter, that is not available any
more in Java 11 and later with Hex utility from non optional Bouncy
Castle library.
Bug: 540790
Change-Id: I9903c00ecc1a434e9795b8ba9267f02628fdc0e9
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
PacketLineIn: Suppress comparison warnings for END and DELIM
Reference comparison is intentional. The END and DELIM string
constants are used as sentinels and will always be the same
instances.
Suppress both ReferenceEquality and StringEquality warnings.
Change-Id: I4ce0495702c56b3911f42f26c2f81d28073cbe19
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
When cloning repository with --single-branch option, tag chains are not
packed and pack file is broken in some cases.
Typical test-case:
git tag -a test_tag <commit-id>
git tag -a test_prev_tag test_tag
git tag -d test_tag
git clone --single-branch <repository>
fatal: did not receive expected object <test_tag_id>
The reason for that is missing object for original test_tag reference,
which was deleted.
Problem description:
When pack-objects is given --include-tag, it peels each tag reference
down to a commit. If the commit is prepared to be packed, we we have to
include such tag too. The problem is when the tag points to through some
chain of other tag to commit. Then, the inner tags are not added leading
to broken pack.
Fix:
When going to commit, we have to check and add any of the tags on the
way (if they were not selected, which may happen with --single-branch
option).
Change-Id: I1682d4a2c52d674f90a1b021e0f6c3524c5ce5bc
Signed-off-by: Pavel Flaška <Pavel.Flaska@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
BasePackFetchConnection: Prefer boolean operators over logical operators in comparisons
Using the | and & operators in boolean conditions results in a warning
from Error Prone:
[ShortCircuitBoolean]
Prefer the short-circuiting boolean operators && and || to & and |.
see https://errorprone.info/bugpattern/ShortCircuitBoolean
Change-Id: I6cccca3fdd28bf93b302a9b8a66e68ac912cb60d
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Increase severity of OperatorPrecedence to ERROR, and fix instances
Fix all remaining instances of the OperatorPrededence warning, by adding
parentheses to make the precedence explicit.
Change-Id: Ib296dfed09f9be042d0ff0f7fad8214e4dd766b4
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Consistently use "!isEmpty()" to detect non-empty list
Replace "size() > 0" with "!isEmpty()" where appropriate.
In the Status implementation we can drop the check; the subsequent
loop will only execute when the list is non-empty anyway.
Change-Id: I355aff551a603373e702a9d44304f087b476263c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Error Prone reports:
[SizeGreaterThanOrEqualsZero] Comparison of a size >= 0 is always true,
did you intend to check for non-emptiness?
see https://errorprone.info/bugpattern/SizeGreaterThanOrEqualsZero
Change-Id: Ie964771cacca4b15569eb45f6e273ad2a7e2e49c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
The git config entries "http.cookieFile" and
"http.saveCookies" are correctly evaluated.
Bug: 488572
Change-Id: Icfeeea95e1a5bac3fa4438849d4ac2306d7d5562
Signed-off-by: Konrad Windszus <konrad_w@gmx.de>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Repository: Add getIdentifier() method to avoid instanceof operator
This change is needed to implement permission aware ref database in
Gerrit: [1], that is a pre-requisite to re-enable Git v2 protocol in
Gerrit: [2].
Background: Last year Git v2 protocol was enabled in Gerrit. The fact,
that JGit layer was not calling ref advertise filter for Git v2
protocol, introduced security vulnerability.
The lesson learned from this security incident: Gerrit should not rely
on ref advertise filter being called by JGit to implement crictical
security checks. Instead, the idea is to use the same approach as
currently used by Google's internal code on googlesource.com that
didn't suffer from this vulnerability: provide a custom repository to
JGit. The repository provides a RefDatabase that is permission-aware
and will only ever return refs that the user has access to.
However, due to hard coded instanceof operator usages in JGit code
base, some tests in Gerrit are failing with: [1] in place. This change
addresses this problem.
[1] https://gerrit-review.googlesource.com/c/gerrit/+/212874
[2] https://gerrit-review.googlesource.com/c/gerrit/+/226754
Change-Id: I67c0f53ca33b149442e7ee3e51910d19e3f348d5
Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Deprecate the constant with the intention of making it private in
a future release.
All existing usage of the constant within JGit code has already been
replaced with the recommended alternatives in preceding commits.
Change-Id: I10eb95f3f92cb74f93a26bf1a6edd24615b75c6f
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
PacketLineIn: Add an iterator over strings in the input stream
Allows callers to read all lines in the input stream until the
END marker is reached, without having to explicitly check for
the END marker.
Replace all remaining usage of the END marker with the new method.
Change-Id: I51f419c7f569ab7ed01e1aaaf6b40ed8cdc2116b
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Replace most usages of PacketLineIn.END with PacketLineIn.end()
PacketLineIn.END is only referenced in tests. Replace most of those
with a new package visible end() method.
Remaining usages of PacketLineIn.END are in the form:
while ((line = pckIn.readString()) != PacketLineIn.END) {
and are not trivial replacements, hence are not touched in this change.
Change-Id: Id77c5321ddcad127130b246bde8f08736e60e1ea
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Deprecate DELIM with the intention of making it private in a future
release.
Callers that want to test if a packet line string is the delimiter
should use the isDelimiter(String) method.
The only other references to DELIM in the JGit code are in tests. For
those, introduce a package visible delimiter() method.
Change-Id: I21e8bbac0ffb9ef710c9753e23435416b09a4891
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Replace trivial reference comparison of PacketLineIn.{DELIM,END}
Replace reference comparisons of PacketLineIn's DELIM and END strings
with usage of the helper methods isDelimiter() and isEnd().
Change-Id: I52dcfc4ee9097f1bd6970601c716701847d9eebd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
PacketLineIn: Add helper methods to check for END and DELIM
These methods will allow clients to check for END and DELIM without
doing a reference comparison on the String objects, which raises
warnings from Error Prone.
Change-Id: I9e7e59843553ed4488ee8e864033198bbb60d67c
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Every caller would need to check if bitmaps are available in the repo to
instantiate a reachability checker.
Offer a method to create the reachability checker in the walk: the
caller has already a walk over the repo, and the walk has all the
information required.
This allows us to make the implementation classes package-private.
Change-Id: I355e47486fcd9d55baa7cb5700ec08dcc230eea5
Signed-off-by: Ivan Frade <ifrade@google.com>
In 7b96bd812e ("UploadPack: Use reachability checker to validate
non-advertised wants", 2019-05-16), a "walk.setRetainBody(false);"
statement was inadvertently deleted. (An earlier version of this commit
had this line in another part of the code and a review comment suggested
to move it back here; the line was then deleted from the other part of
the code but not readded.) Restore this line.
Change-Id: I96ff6106ba9e4eef429388c83e898b3363295f69
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
UploadPack: Use reachability checker to validate non-advertised wants
In "Reachable commit" request validators, we need to check that a "want"
in the request, that hasn't been advertised, is reachable from the refs
visible to the user.
Current code has intermixed the translation of ObjectIds to RevCommits
(and its error handling) with the actual walk, with the delegation to
bitmaps in restricted circunstances.
Refactor the code to make it "flatter" and more readable. Move ObjectIds
to RevCommits translation to its own functions. Use the reachability
checker instead of a newly defined walk.
Before the non-advertised wants were validated with bitmaps only if any
"want" refered to an non-commit. Now they will be validated with bitmaps
also if the "wants" refer all to commits.
Change-Id: Ib925a48cde89672b07a88bba4e24d0457546baff
Signed-off-by: Ivan Frade <ifrade@google.com>
This is used when fetching, and in particular to populate a partial
clone or a virtual file system cache as the user navigates. With this,
a client can pre-fetch a few directories deeper than only the current
directory.
depth:0 will omit all trees, and is useful if you only want to fetch
the commits of a repository, or fetch just a single tree or blob object.
depth:1 will fetch only the root tree of all commits fetched. depth:2
will fetch the root tree and all blobs and tree objects directly
referenced from it. depth:3 gets one more level, and so on. depth:#
will not filter a blob or tree that is explicitly marked wanted.
Bitmaps are disabled when this filter is used.
This implementation is quite slow because it iterates over all omitted
objects rather than skipping them. This will be addressed in follow-up
commits.
Change-Id: Ic312fee22d60e32cfcad59da56980e90ae2cae6a
Signed-off-by: Matthew DeVore <matvore@gmail.com>
Expose and pass around the FilterSpec object rather than the raw blob limit
Use the FilterSpec object so that less code has to know about the make-up of
FilterSpecs. When fields are added to FilterSpec, these pieces of code won't
need updating again.
Change-Id: I2b9e59a9926ff112faf62a3fa2d33c961a1779e5
Signed-off-by: Matthew DeVore <matvore@gmail.com>
This increases type-safety and is ground work for support of the
"tree:<depth>" filter.
Change-Id: Id19eacdcdaddb9132064c642f6d554b1060efe9f
Signed-off-by: Matthew DeVore <matvore@gmail.com>
This code is not specific to V2 of the protocol, so it doesn't belong
there. Move it to its own class.
Change-Id: I8084764147d1d0e2bceaaead59b332a8ec1a0825
Signed-off-by: Matthew DeVore <matvore@gmail.com>
Jonathan Tan
Matthias Sohn
Thomas Wolf
David Turner
Ivan Frade
Masaya Suzuki
Michael Keppler
David Pursehouse
David Ostrovsky
Pavel Flaška
Konrad Windszus
Carsten Hammer
Matthew DeVore