Fix HTTP response processing for WWW-Authenticate headers
The original code was able to process only one WWW-Authenticate
header in an HTTP response, and if this header was not one of
two expected, authentication failed regardless of that there
could be other headers in the response.
All WWW-Authenticate headers in an HTTP response have to be
browsed to find one of supported, i.e. Basic or Digest.
By that if both are present, the Digest one should be used
as more preferable.
Bug: 357719
Change-Id: Icf601a41fec63f7d40308f3c85aaa4f71a7c095b
Signed-off-by: Alex Rukhlin <arukhlin@microsoft.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Can be used for listing remote refs for a repository on the file system
without having a local repository.
Bug: 413400
Change-Id: I397f5092c5eafb62236e9f9e74d9183f56903cc6
Signed-off-by: Robin Stocker <robin@nibor.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Fix HTTP response processing for WWW-Authenticate headers
The original code was able to process only one WWW-Authenticate
header in an HTTP response, and if this header was not one of
two expected, authentication failed regardless of that there
could be other headers in the response.
All WWW-Authenticate headers in an HTTP response have to be
browsed to find one of supported, i.e. Basic or Digest.
By that if both are present, the Digest one should be used
as more preferable.
Bug: 357719
Change-Id: Icf601a41fec63f7d40308f3c85aaa4f71a7c095b
Signed-off-by: Alex Rukhlin <arukhlin@microsoft.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Support refspecs with wildcard in middle (not only at end)
The following refspec, which can be used to fetch GitHub pull requests,
is supported by C Git but was not yet by JGit:
+refs/pull/*/head:refs/remotes/origin/pr/*
The reason is that the wildcard in the source is in the middle.
This change also includes more validation (e.g. "refs//heads" is not
valid) and test cases.
Bug: 405099
Change-Id: I9bcef7785a0762ed0a98ca95a0bdf8879d5702aa
Allowed ipv6-address in a uri like:
http://[::1]:8080/repo.git
Change-Id: Ia00a20f694b2e9314892df77f9b11f551bb1d34e
Signed-off-by: Chris Aniszczyk <zx@twitter.com>
Allow users to provide their OutputStream (via Transport#
push(monitor, refUpdates, out)) so that server messages can be written
to it (in SideBandInputStream) while they're coming in.
CQ: 7065
Bug: 398404
Change-Id: I670782784b38702d52bca98203909aca0496d1c0
Signed-off-by: Andre Dietisheim <andre.dietisheim@gmail.com>
Signed-off-by: Chris Aniszczyk <zx@twitter.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
JGit 3.0: move internal classes into an internal subpackage
This breaks all existing callers once. Applications are not supposed
to build against the internal storage API unless they can accept API
churn and make necessary updates as versions change.
Change-Id: I2ab1327c202ef2003565e1b0770a583970e432e9
For streams that should not be closed, i.e. don't own an underlying
stream, and in-memory streams that do not need to be closed we just
suppress the warning. This mostly apply to test cases. GC is enough.
For streams with external resources (i.e. files) we add the necessary
call to close().
Change-Id: I4d883ba2e7d07f199fe57ccb3459ece00441a570
Suppress boxing warnings where we know they are ok
Invoke the wrapper types' valueOf via static imports.
For booleans used in asserts, add a new assert in
the JUnit utility package since out current version of JUnit
does not have the assert(boolean, boolean) method.
Change-Id: I9099bd8efbc8c133479344d51ce7dabed8958a2b
Fix resolving of relative file URIs in TransportLocal
A configured remote url like "../repo" works with C Git.
In JGit, it only worked if Java's current working directory happened to
be the local repository working directory.
Change-Id: I33ba3f81b37d03cf17ca7ae25a90774a27e7e02b
Signed-off-by: Robin Stocker <robin@nibor.org>
When receiving a pack, data buffered after the pack can restored
to the InputStream if the stream supports mark and reset.
Change-Id: If04915c32c91be28db8df7e8491ed3e9fe0e1608
Use BatchRefUpdate for tracking refs in FetchProcess
If there are a lot of references to modify, using BatchRefUpdate can
save time if the underlying storage is able to combine these updates
together. This should speed up initial clone or fetch into an empty
repository, as some projects can have hundreds of release tags, or
hundreds of branch heads.
Change-Id: Iee9af8d5fa19080077d88357c18853540936e940
Check for a '#' character in each line read and
parse the leading characters as the class name of
a TransportProtocol being registered via SPI.
Bug: 373439
Change-Id: If36cb62c07ecea78ba0f326a87edf1d80b7b42b6
Signed-off-by: Chris Aniszczyk <zx@twitter.com>
Modify refs in UploadPack/ReceivePack using a hook interface
This is intended to replace the RefFilter interface (but does not yet,
for backwards compatibility). That interface required lots of extra
scanning and copying in filter cases such as only advertising a subtree
of the refs directory. Instead, provide a hook that can be executed
right before ref advertisement, using the public methods on
UploadPack/ReceivePack to explicitly set the map of advertised refs.
Change-Id: I0067019a191c8148af2cfb71a675f2258c5af0ca
Add percent-encoding of reserved characters in URIish
We do this for the the names that have an explicit scheme and
do it both ways. The URIish is parsed before decoding. Only
a few special characters are encoded for the path part of the
URI, i.e. space, non-ASCII and control characters. The percent
encoding is assumed to be a stream encoding so we interpret it
as UTF-8.
Change-Id: I82d1910df9472e21d7212a2b984ff7d8fb2cbf0f
ReceivePack (and PackParser) can be configured with the
maxObjectSizeLimit in order to prevent users from pushing too large
objects to Git. The limit check is applied to all object types
although it is most likely that a BLOB will exceed the limit. In all
cases the size of the object header is excluded from the object size
which is checked against the limit as this is the size of which a BLOB
object would take in the working tree when checked out as a file.
When an object exceeds the maxObjectSizeLimit the receive-pack will
abort immediately.
Delta objects (both offset and ref delta) are also checked against the
limit. However, for delta objects we will first check the size of the
inflated delta block against the maxObjectSizeLimit and abort
immediately if it exceeds the limit. In this case we even do not know
the exact size of the resolved delta object but we assume it will be
larger than the given maxObjectSizeLimit as delta is generally only
chosen if the delta can copy more data from the base object than the
delta needs to insert or needs to represent the copy ranges. Aborting
early, in this case, avoids unnecessary inflating of the (huge) delta
block.
Unfortunately, it is too expensive (especially for a large delta) to
compute SHA-1 of an object that causes the receive-pack to abort.
This would decrease the value of this feature whose main purpose is to
protect server resources from users pushing huge objects. Therefore
we don't report the SHA-1 in the error message.
Change-Id: I177ef24553faacda444ed5895e40ac8925ca0d1e
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Double ' characters are needed for variables to appear in
single quotes. Variables surrounded with a s single ' will
not be replaced when formatted
Change-Id: I0182c1f679ba879ca19dd81bf46924f415dc6003
Signed-off-by: Kevin Sawicki <kevin@github.com>
Actually this is not ok according to the RFC, but this implementation is
ment to be Git compatible. A '\' is needed when the authentication
requires or allows authentication to a Windows domain where the
user name can be specified as DOMAIN\user.
Change-Id: If02f258c032486f1afd2e09592a3c7069942eb8b
Some servlet containers require the servlet to read the EOF marker
from the input stream before a response can be output if the stream
is using "Transfer-Encoding: chunked"... which is typical for any
sort of large push to a repository over smart HTTP.
Ensure the EOF is always read by the PackParser when it is handling
the stream, and fail fast if there is more data present than expected
since this does indicate a protocol error.
Also ensure the EOF is read by UploadPack before it starts to output
a partial response using packing progress meters.
Change-Id: I131db9dea20b2324cb7c3272a814f21296bc64bd
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Since d1718a the method getHumanishName was broken on windows since
the URIish is not normalized anymore. For a path like
"C:\gitRepositories\egit" the whole path was returned instead of
"egit".
Bug: 343519
Change-Id: I95056009072b99d32f288966302d0f8188b47836
Signed-off-by: Stefan Lay <stefan.lay@sap.com>
We used to normalize URI's since it seems simple. This however causes
inconsistencies to the user and to out tests. Just pass backslashes
through and make sure our parser can handle them.
Bug: 341062
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Change-Id: I2c8e917a086faabcd8749160c2acc9dd05a42838
If the remote peer replies with "ERR %s" instead of "ACK %s common" or
"NAK" during ancestor negotiation in the fetch-pack/upload-pack
protocol, treat that as an exception that aborts processing with the
error text as supplied by the remote system.
This matches behavior with "ERR %s" during the advertisements, which
is also a way for the remote to abort processing.
Change-Id: I2fe818e75c7f46156744ef4f703c40173cbc76d0
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
TransportProtocol: Allow null Repository in canHandle()
This allows callers to determine if a URI is supported, before
worrying about the local repository.
Suggested-by: Dariusz Luksza <dariusz@luksza.org>
Change-Id: Ifc76a4ba841f2e2e7354bd51306b87b3b9d7f6ab
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Make the supported Transports extensible and discoverable
The new TransportProtocol type describes what a particular Transport
implementation wants in order to support a connection. 3rd parties
can now plug into the Transport.open() logic by implementing their
own TransportProtocol and Transport classes, and registering with
Transport.register().
GUI applications can help the user configure a connection by looking
at the supported fields of a particular TransportProtocol type, which
makes the GUI more dynamic and may better support new Transports.
Change-Id: Iafd8e3a6285261412aac6cba8e2c333f8b7b76a5
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
RemoteRefUpdate: Accept Ref and ObjectId arguments for source
Applications may already have a Ref or ObjectId on hand that they want
the remote to be updated to. Instead of converting these into a
String and relying on the parsing rules of resolve(), allow the
application to supply the Ref or ObjectId directly.
Bug: 338839
Change-Id: If5865ac9eb069de1c8f224090b6020fc422f9f12
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
[findbugs] Do not ignore exceptional return value of mkdir
java.io.File.mkdir() and mkdirs() report failure as an exceptional
return value false. Fix the code which silently ignored this
exceptional return value.
Change-Id: I41244f4b9d66176e68e2c07e2329cf08492f8619
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Refactor IndexPack to not require local filesystem
By moving the logic that parses a pack stream from the network (or
a bundle) into a type that can be constructed by an ObjectInserter,
repository implementations have a chance to inject their own logic
for storing object data received into the destination repository.
The API isn't completely generic yet, there are still quite a few
assumptions that the PackParser subclass is storing the data onto
the local filesystem as a single file. But its about the simplest
split of IndexPack I can come up with without completely ripping
the code apart.
Change-Id: I5b167c9cc6d7a7c56d0197c62c0fd0036a83ec6c
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
Eclipse has some problem re-running single JUnit tests if
the tests are in Junit 3 format, but the JUnit 4 launcher
is used. This was quite unnecessary and the move was not
completed. We still have no JUnit4 test.
This completes the extermination of JUnit3. Most of the
work was global searce/replace using regular expression,
followed by numerous invocarions of quick-fix and organize
imports and verification that we had the same number of
tests before and after.
- Annotations were introduced.
- All references to JUnit3 classes removed
- Half-good replacement for getting the test name. This was
needed to make the TestRngs work. The initialization of
TestRngs was also made lazily since we can not longer find
out the test name in runtime in the @Before methods.
- Renamed test classes to end with Test, with the exception
of TestTranslateBundle, which fails from Maven
- Moved JGitTestUtil to the junit support bundle
Change-Id: Iddcd3da6ca927a7be773a9c63ebf8bb2147e2d13
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Always use streaming (for SHA-checksum & collision detection)
when indexing whole blobs, regardless of their size.
Positives:
* benefits of bugfix #312868 will apply to all runtimes, without
additional conf for mem-constrained JVMs (5MB huge for some)
* no byte array allocation
(re-uses readBuffer instead of allocating new full-size array)
* mildly better overall performance
(given the usual blob-does-not-need-collision-checking case)
* removes unnecessary code
Negative:
* doubles the disk IO for a blob comparision
(comparitively rare occurance)
I perf-tested a range of threshold sizes against a random selection
of packfiles I found on my harddrive, the results are here:
https://spreadsheets.google.com/ccc?key=tLCQElyyd2RKN9QevfvgwGQ&hl=en_GB#gid=1
My interpretation of the results is that the streaming size threshold
isn't beneficial (actually seems to be very slightly detrimental) -so
we should just get rid of it. This tallies with some of the comments
Shawn & I had for the default value of streamFileThreshold in the
review for I862afd4c:
http://egit.eclipse.org/r/#patch,sidebyside,2040,2,org.eclipse.jgit/src/org/eclipse/jgit/transport/IndexPack.java
The perf-test code is here: https://gist.github.com/735402
It's a bit scruffy but basically does 10 runs (in randomised order)
for each threshold size on various packfiles, waiting a second
between each pack-indexing to allow GC to catch up. I know it's not
perfect - proper perf testing is hard to do :-)
When indexing large blobs that are stored whole (non-delta form),
avoid allocating the entire blob in memory and instead stream it
through the SHA-1 checksum computation. This reduces the size
of memory required by IndexPack when processing very big blobs,
such as a 500 MiB uncompressable binary.
If the large blob already exists in the local repository, its
contents needs to be compared byte-for-byte after the entire pack
has been indexed, to ensure there isn't an unexpected SHA-1 collision
which may result in later data corruption. This compare is performed
as a streaming compare, again avoiding the large object allocation.
This change doesn't improve on memory utilization for large objects
stored as deltas. The change also doesn't improve handling for
any large commits, trees or annotated tags. There isn't much to
be done here for those objects, because they need to be passed down
to the ObjectChecker as a byte[]. Fortunately it isn't common for
these object types to be that large,
Bug: 312868
Change-Id: I862afd4cb78013ee033d4ec68c067b1774a05be8
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Chris Aniszczyk <caniszczyk@gmail.com>
CC: Roberto Tyley <roberto.tyley@guardian.co.uk>
We stopped handling URIs such as "example.com:/some/p ath", because
this was confused with the Windows absolute path syntax of "c:/path".
Support absolute style scp URIs again, but only when the host name
is more than 2 characters long.
Change-Id: I9ab049bc9aad2d8d42a78c7ab34fa317a28efc1a
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
In bug 323571 it is mentioned that if you call
'toURI().toURL().toString()' on a java.io.File you cannot pass
that string to jgit as an URIish. Problem is that the passed
URI looks like 'file:/C:/a/b.txt' and that we where expecting
double slashes after scheme':'. This fix adds support for this
single-slash file URLs.
Bug: 323571
Change-Id: I866a76a4fcd0c3b58e0d26a104fc4564e7ba5999
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
There where quite some bugs regarding wrong URI parsing. In order
to solve them the parsing has to be refactored. We now have
specialized regexps for 'scheme://host/...', scp URIs and local
file names. Now we can detect problems while parsing 'git://host:/abc' which
was previously not possible.
Bug: 315571
Bug: 292897
Bug: 307017
Bug: 323571
Bug: 317388
Change-Id: If72576576ebb6b9d9dc8b7e51ddd87c9909e8b62
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The regular expression which should handle the
user/password part in an URI was potentially
processing too many chars. This led to problems
when user/pwd and port was specified
Change-Id: I87db02494c4b367283e1d00437b1c06d2c8fdd28
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix URIish tests to contain a hostname for git protocol
URIs for the git protocol have to have a hostname.
(see http://www.kernel.org/pub/software/scm/git/docs
/git-clone.html#_git_urls_a_id_urls_a) Some tests tested
URIs like git:/abc.git which is not allowed. Fixed this.
Change-Id: Ia3b8b681ad6592f03b090a874a6e91068a8301fe
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
git allows remotes to be relative paths, but the regex
validating urls wouldn't accept anything starting with "..".
Other functionality works fine with these paths.
Bug: 311300
Change-Id: Ib74de0450a1c602b22884e19d994ce2f52634c77
Fix checkReferencedIsReachable to use correct base list
When checkReferencedIsReachable is set in ReceivePack we are trying
to prove that the push client is permitted to access an object that
it did not send to us, but that the received objects link to either
via a link inside of an object (e.g. commit parent pointer or tree
member) or by a delta base reference.
To do this check we are making a list of every potential delta base,
and then ensuring that every delta base used appears on this list.
If a delta base does not appear on this list, we abort with an error,
letting the client know we are missing a particular object.
Preventing spurious errors about missing delta base objects requires
us to use the exact same list of potential delta bases as the remote
push client used. This means we must use TOPO ordering, and we
need to enable BOUNDARY sorting so that ObjectWalk will correctly
include any trees found during the enumeration back to the common
merge base between the interesting and uninteresting heads.
To ensure JGit's own push client matches this same potential delta
base list, we need to undo 60aae90d4d ("Disable topological
sorting in PackWriter") and switch back to using the conventional
TOPO ordering for commits in a pack file. This ensures that our
own push client will use the same potential base object list as
checkReferencedIsReachable uses on the receiving side.
Change-Id: I14d0a326deb62a43f987b375cfe519711031e172
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Alex Rukhlin
Robin Stocker
Robin Rosenberg
Andreas König
André Dietisheim
Shawn Pearce
Ian Wetherbee
Kevin Sawicki
Dave Borowitz
Tomasz Zarna
Sasa Zivkov
Ketan Padegaonkar
Stefan Lay
Matthias Sohn
roberto
Christian Halstrick
Chris West (Faux)