Ned Twigg [Fri, 18 Mar 2016 10:08:44 +0000 (03:08 -0700)]
CLI: implement option -d for deleting tags
Change-Id: I438456b76aefd361384729686271288186d3be3b Signed-off-by: Ned Twigg <ned.twigg@diffplug.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn Pearce [Mon, 29 Aug 2016 19:11:11 +0000 (12:11 -0700)]
ReceivePack: integrate push option parsing into recvCommands
This allows the same try/catch to handle parsing the command list,
push certificate and push options. Any errors will be caught and
handled by the same catch block, as the client is in the same state.
Shawn Pearce [Mon, 29 Aug 2016 18:54:15 +0000 (11:54 -0700)]
ReceivePack: allow push options to be set
Some embeddings of JGit require creating a ReceivePack instance in
another process from the one that handled the network socket with the
client. Similar to the PushCertificate add a setter to allow the
option list to be supplied.
Shawn Pearce [Fri, 26 Aug 2016 21:44:44 +0000 (14:44 -0700)]
ReceivePack: refactor push option parsing
Refactor all of the push option support code to allocate the list
immediately before parsing the options section off the stream.
Move option support down to ReceivePack instead of BaseReceivePack.
Push options are specific to the ReceivePack protocol and are not
likely to appear in the 4 year old subscription proposal. These
changes are OK before JGit 4.5 ships as no consumer should be relying
on these new APIs.
Shawn Pearce [Fri, 26 Aug 2016 01:59:15 +0000 (18:59 -0700)]
DfsReader: check object type during open
Do not open an OBJ_TREE if the caller is expecting an OBJ_BLOB or
OBJ_COMMIT; instead throw IncorrectObjectTypeException. This better
matches behavior of WindowCursor, the ObjectReader implementation of
the local file based object store.
Masaya Suzuki [Fri, 26 Aug 2016 01:25:48 +0000 (18:25 -0700)]
Clarify the semantics of DfsRefDatabase#compareAndPut
DfsRefDatabase#compareAndPut had a vague semantics for reference
matching. Because of this, an operation to make a symbolic
reference had been broken for some DFS implementations even if they
followed the contract of compareAndPut. The clarified semantics
requires the implementations to satisfy the followings:
* Matching references should be both symbolic references or both
object ID references.
* If both are symbolic references, both should have the same target
name.
* If both are object ID references, both should have the same object
ID.
This semantics is defined based on
https://git.eclipse.org/r/#/c/77416/. Before this commit,
DfsRefDatabase couldn't see the target of symbolic references.
InMemoryRepository is changed to comply with the new semantics. This
semantics change can affect the existing DFS implementations that only
checks object IDs. This commit adds two tests that the previous
InMemoryRepository couldn't pass.
Matthias Sohn [Tue, 24 May 2016 23:07:18 +0000 (01:07 +0200)]
Add a RepeatRule to help repeating flaky tests
A JUnit TestRule which enables to run the same JUnit test repeatedly.
This may help to identify the root cause why a flaky tests which succeed
most often does fail sometimes.
Add the RepeatRule to the test class containing the test to be repeated:
public class MyTest {
@Rule
public RepeatRule repeatRule = new RepeatRule();
...
}
and annotate the test to be repeated with the @Repeat(n=<repetitions>)
annotation:
@Test
@Repeat(n = 100)
public void test() {
...
}
then this test will be repeated 100 times. If any test execution fails
test repetition will be stopped.
Change-Id: I7c49ccebe1cb00bcde6b002b522d95c13fd3a35e Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When doing a detaching operation, JGit fakes a SymbolicRef as an
ObjectIdRef. This is because RefUpdate#updateImpl dereferences the
SymbolicRef when updating it. For example, assume that HEAD is
pointing to refs/heads/master. If I try to make a detached HEAD
pointing to a commit c0ffee, RefUpdate dereferences HEAD as
refs/heads/master first and changes refs/heads/master to c0ffee. The
detach argument of RefDatabase#newUpdate avoids this dereference by
faking HEAD as ObjectIdRef.
This faking is problematic for the linking operation of
DfsRefDatabase. It does a compare-and-swap operation on every
reference change because of its distributed systems nature. If a
SymbolicRef is faked as an ObjectRef, it thinks that there is a
racing change in the reference and rejects the update. Because of
this, DFS based repositories cannot change the link target of symbolic
refs. This has not been a problem for file-based repositories because
they have a file-lock based semantics instead of the CAS based one.
The reference implementation, InMemoryRepository, is not affected
because it only compares ObjectIds.
When [1] introduced this faking code, there was no way for RefUpdate
to distinguish the detaching operation. When [2] fixed the detaching
operation, it introduced a detachingSymbolicRef flag. This commit uses
this flag to control whether it needs to dereference the symbolic refs
by calling Ref#getLeaf. The same flag is used in the reflog update
operation.
This commit does not affect any operation that succeeds currently. In
some DFS repository implementations, this fixes a ref linking
operation, which is currently failing.
Dan Wang [Tue, 23 Aug 2016 02:16:51 +0000 (19:16 -0700)]
Packet logging for JGit
Imitate the packet tracing feature from C Git v1.7.5-rc0~58^2~1 (add
packet tracing debug code, 2011-02-24). Unlike C Git, use the log4j
log level setting instead of the GIT_TRACE_PACKET environment variable
to enable tracing.
Stefan Beller [Wed, 24 Aug 2016 19:47:10 +0000 (12:47 -0700)]
push: Do not use push options unless requested
Unless the user passed --push-option, the client does not intend to
pass push options to the server.
Without this change, all pushes to servers without push option support
fail.
Not enabling the feature (instead of enabling it and sending an empty
list of options) in this case is more intuitive and matches the
behavior of C git push's --push-option parameter better.
Bug: 500149
Change-Id: Ia4f13840cc54d8ba54e99b1432108f1c43022c53 Signed-off-by: Stefan Beller <sbeller@google.com>
HttpClientConnection uses a TemporaryBufferEntity which uses
TemporaryBuffer.LocalFile to buffer an HttpEntity. It was leaking
temporary files if the buffered entities were larger than 1MB since it
failed to destroy the TemporaryBuffer.LocalFile.
Bug: 500079
Change-Id: Ib963e04efc252bdd0420a5c69b1a19181e9e6169 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Masaya Suzuki [Sun, 21 Aug 2016 22:27:30 +0000 (15:27 -0700)]
Use FS#lastModified instead of File#lastModified
This fixes the tests failed in JDK8.
FS uses java.nio API to get file attributes. The timestamps obtained
from that API are more precise than the ones from
java.io.File#lastModified() since Java8.
This difference accidentally makes JGit detect newly added files as
smudged. Use the precised timestamp to avoid this false positive.
David Pursehouse [Fri, 19 Aug 2016 01:58:06 +0000 (10:58 +0900)]
LfsProtocolServlet: Add support for rate limit and bandwidth limit errors
The git-lfs specification [1] describes the following optional status codes
that may be returned:
429 - The user has hit a rate limit with the server. Though the API does
not specify any rate limits, implementors are encouraged to set some
for availability reasons.
509 - Returned if the bandwidth limit for the user or repository has been
exceeded. The API does not specify any bandwidth limit, but implementors
may track usage.
Add two new exception classes to represent these cases. Implementations may
throw these from #getLargeFileRepository(), causing the corresponding HTTP
status codes to be returned to the client.
Masaya Suzuki [Fri, 19 Aug 2016 20:51:05 +0000 (13:51 -0700)]
Ignore IOException thrown from close
AddCommandTest is flaky because IOException is thrown sometimes.
Caused by: java.io.IOException: Stream closed
at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
at java.io.OutputStream.write(OutputStream.java:116)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.eclipse.jgit.util.FS.runProcess(FS.java:993)
at org.eclipse.jgit.util.FS.execute(FS.java:1102)
at org.eclipse.jgit.treewalk.WorkingTreeIterator.filterClean(WorkingTreeIterator.java:470)
... 22 more
OpenJDK replaces the underlying OutputStream with NullOutputStream when
the process exits. This throws IOException for all write operation. When
it exits before JGit writes the input to the pipe buffer, the input
stays in BufferedOutputStream. The close method tries to write it again,
and IOException is thrown.
Since we ignore IOException in StreamGobbler, we also ignore it when
we close the stream.
Shawn Pearce [Fri, 19 Aug 2016 18:51:40 +0000 (11:51 -0700)]
DfsObjDatabase: clear PackList dirty bit if no new packs
If a reference was updated more recently than a pack was written
(typical) the PackList was perpetually dirty until the next GC
was completed for the repository.
Detect this condition by observing no changes to the PackList
membership and resetting the dirty bit.
David Pursehouse [Sat, 13 Aug 2016 07:56:20 +0000 (16:56 +0900)]
LfsProtocolServlet: Always set the Content-Type header on response
If the Content-Type is not set on error responses, the git-lfs client
does not read the body which contains the error message, and instead
just displays a generic error message.
Also set the charset on the Content-Type header.
Change-Id: I88e6f07f20b622a670e7c5063145dffb8b630aee Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Jonathan Nieder [Tue, 9 Aug 2016 01:50:01 +0000 (18:50 -0700)]
RepoCommand: Avoid group lists shadowing groups strings
Reported-by: David Pursehouse <david.pursehouse@gmail.com>
Change-Id: I9e9b021d335bda4d58b6bcc30f59b81ac5b37724 Signed-off-by: Jonathan Nieder <jrn@google.com>
Matthias Sohn [Mon, 8 Aug 2016 15:10:53 +0000 (17:10 +0200)]
Silence API errors in LfsProtocolServlet
bb9988c2 changed the signature of getLargeFileRepository() which is only
breaking implementors which is ok according to OSGi semantic versioning
rules.
Change-Id: I68bda7900b72e217571f74aee53705167f8100a2 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Jonathan Nieder [Mon, 8 Aug 2016 19:35:36 +0000 (12:35 -0700)]
Shallow fetch: Pass along "shallow"s in unparsed-wants case, too
Since 84d2738ff21c (Don't skip want validation when the client sends no
haves, 2013-06-21), this branch is not taken. Process the
"shallow"s anyway as a defensive measure in case the code path gets
revived.
Change-Id: Idfb834825d77f51e17191c1635c9d78c78738cfd Signed-off-by: Jonathan Nieder <jrn@google.com>
Jonathan Nieder [Mon, 8 Aug 2016 19:31:39 +0000 (12:31 -0700)]
Shallow fetch: Pass a DepthWalk to PackWriter
d385a7a5e5ca (Shallow fetch: Respect "shallow" lines, 2016-08-03) forgot
that UploadPack wasn't passing a DepthWalk to PackWriter in the first
place. As a result, shallow clones fail:
java.lang.IllegalArgumentException: Shallow packs require a DepthWalk
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:756)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1497)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1381)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:774)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:667)
at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:191)
Jonathan Nieder [Sat, 6 Aug 2016 00:36:08 +0000 (20:36 -0400)]
Merge changes I27961679,I91be6165,If0dbd562
* changes:
LfsProtocolServlet: Allow access to objects in request
LfsProtocolServlet: Allow getLargeFileRepository to raise exceptions
Remove references to org.eclipse.jgit.java7
Terry Parker [Thu, 4 Aug 2016 18:14:33 +0000 (11:14 -0700)]
Shallow fetch/clone: Make --depth mean the total history depth
cgit changed the --depth parameter to mean the total depth of history
rather than the depth of ancestors to be returned [1]. JGit still uses
the latter meaning, so update it to match cgit.
depth=0 still means a non-shallow clone. depth=1 now means only the
wants rather than the wants and their direct parents.
This is accomplished by changing the semantic meaning of "depth" in
UploadPack and PackWriter to mean the total depth of history desired,
while keeping "depth" in DepthWalk.{RevWalk,ObjectWalk} to mean
the depth of traversal. Thus UploadPack and PackWriter always
initialize their DepthWalks with "depth-1".
Terry Parker [Wed, 3 Aug 2016 16:01:22 +0000 (09:01 -0700)]
Shallow fetch: Respect "shallow" lines
When fetching from a shallow clone, the client sends "have" lines
to tell the server about objects it already has and "shallow" lines
to tell where its local history terminates. In some circumstances,
the server fails to honor the shallow lines and fails to return
objects that the client needs.
UploadPack passes the "have" lines to PackWriter so PackWriter can
omit them from the generated pack. UploadPack processes "shallow"
lines by calling RevWalk.assumeShallow() with the set of shallow
commits. RevWalk creates and caches RevCommits for these shallow
commits, clearing out their parents. That way, walks correctly
terminate at the shallow commits instead of assuming the client has
history going back behind them. UploadPack converts its RevWalk to an
ObjectWalk, maintaining the cached RevCommits, and passes it to
PackWriter.
Unfortunately, to support shallow fetches the PackWriter does the
following:
if (shallowPack && !(walk instanceof DepthWalk.ObjectWalk))
walk = new DepthWalk.ObjectWalk(reader, depth);
That is, when the client sends a "deepen" line (fetch --depth=<n>)
and the caller has not passed in a DepthWalk.ObjectWalk, PackWriter
throws away the RevWalk that was passed in and makes a new one. The
cleared parent lists prepared by RevWalk.assumeShallow() are lost.
Fortunately UploadPack intends to pass in a DepthWalk.ObjectWalk.
It tries to create it by calling toObjectWalkWithSameObjects() on
a DepthWalk.RevWalk. But it doesn't work: because DepthWalk.RevWalk
does not override the standard RevWalk#toObjectWalkWithSameObjects
implementation, the result is a plain ObjectWalk instead of an
instance of DepthWalk.ObjectWalk.
The result is that the "shallow" information is thrown away and
objects reachable from the shallow commits can be omitted from the
pack sent when fetching with --depth from a shallow clone.
Multiple factors collude to limit the circumstances under which this
bug can be observed:
1. Commits with depth != 0 don't enter DepthGenerator's pending queue.
That means a "have" cannot have any effect on DepthGenerator unless
it is also a "want".
2. DepthGenerator#next() doesn't call carryFlagsImpl(), so the
uninteresting flag is not propagated to ancestors there even if a
"have" is also a "want".
3. JGit treats a depth of 1 as "1 past the wants".
Because of (2), the only place the UNINTERESTING flag can leak to a
shallow commit's parents is in the carryFlags() call from
markUninteresting(). carryFlags() only traverses commits that have
already been parsed: commits yet to be parsed are supposed to inherit
correct flags from their parent in PendingGenerator#next (which
doesn't happen here --- that is (2)). So the list of commits that have
already been parsed becomes relevant.
When we hit the markUninteresting() call, all "want"s, "have"s, and
commits to be unshallowed have been parsed. carryFlags() only
affects the parsed commits. If the "want" is a direct parent of a
"have", then it carryFlags() marks it as uninteresting. If the "have"
was also a "shallow", then its parent pointer should have been null
and the "want" shouldn't have been marked, so we see the bug. If the
"want" is a more distant ancestor then (2) keeps the uninteresting
state from propagating to the "want" and we don't see the bug. If the
"shallow" is not also a "have" then the shallow commit isn't parsed
so (2) keeps the uninteresting state from propagating to the "want
so we don't see the bug.
Here is a reproduction case (time flowing left to right, arrows
pointing to parents). "C" must be a commit that the client
reports as a "have" during negotiation. That can only happen if the
server reports it as an existing branch or tag in the first round of
negotiation:
A <-- B <-- C <-- D
First do
git clone --depth 1 <repo>
which yields D as a "have" and C as a "shallow" commit. Then try
git fetch --depth 1 <repo> B:refs/heads/B
Negotiation sets up: have D, shallow C, have C, want B.
But due to this bug B is marked as uninteresting and is not sent.
Change-Id: I6e14b57b2f85e52d28cdcf356df647870f475440 Signed-off-by: Terry Parker <tparker@google.com>
David Pursehouse [Fri, 29 Jul 2016 03:37:48 +0000 (12:37 +0900)]
LfsProtocolServlet: Allow getLargeFileRepository to raise exceptions
According to the specification [1] the server may return the following
HTTP error responses:
- 403: The user has read, but not write access.
- 404: The repository does not exist for the user.
- 422: Validation error with one or more of the objects in the request.
In the current implementation, however, getLargeFileRepository can only
return null to indicate an error. This results in the error code:
- 503: Service Unavailable
being returned to the client regardless of what the actual reason was.
Add exception classes to cover these cases, derived from a common base
exception, and change the specification of getLargeFileRepository to throw
the base exception.
In LfsProtocolServlet#post, handle the new exceptions and send back the
appropriate HTTP responses as mentioned above.
The specification also mentions several other optional response codes (406,
429, 501, and 509) but these are not implemented in this commit. It should
be trivial to implement them in follow-up commits.
Terry Parker [Wed, 3 Aug 2016 15:36:55 +0000 (08:36 -0700)]
RevWalk: Make fields available to DepthWalk
DepthWalk needs to override toObjectWalkWithSameObjects() and thus
needs to be able to directly set the objects and freeFlags fields, so
make them package private.
Change-Id: I24561b82c54ba3d6522582ca25105b204d777074 Signed-off-by: Terry Parker <tparker@google.com>
Terry Parker [Tue, 2 Aug 2016 15:53:06 +0000 (08:53 -0700)]
Shallow fetch: avoid sending unneeded blobs
When doing an incremental fetch from JGit, "have" commits are marked
as "uninteresting". In a non-shallow fetch, when the RevWalk hits an
"uninteresting" commit it marks the commit's corresponding tree as
uninteresting. That has the effect of dropping those trees and all the
trees and blobs they reference out of the thin pack returned to the
client.
However, shallow fetches use a DepthWalk to limit the RevWalk, which
nearly always causes the RevWalk to terminate before encountering the
"have" commits. As a result the pack created for the incremental fetch
never encounters "uninteresting" tree objects and thus includes
duplicate objects that it knows the client already has.
Change-Id: I7b1f7c3b0d83e04d34cd2fa676f1ad4fec904c05 Signed-off-by: Terry Parker <tparker@google.com>
Skip cleaning inner repositories by default in CleanCommand
Previously jgit would attempt to clean git repositories that had not
been committed by calling a non-recursive delete on them, which would
fail as they are directories. This commit addresses that issue in the
following ways.
Repositories are skipped in a default clean, similarly to cgit and only
cleaned when the force flag is applied. When the force flag is applied
repositories are deleted using a recursive delete call. The force flag
and setForce method are added here to CleanCommand to support this
change.
David Pursehouse [Fri, 22 Jul 2016 08:21:20 +0000 (17:21 +0900)]
FileLfsServlet: Return HTTP 422 instead of 400
According to the specification [1], the error response status code
should be 422 when there is a validation error with one or more of
the objects in the request
David Pursehouse [Fri, 22 Jul 2016 05:05:19 +0000 (14:05 +0900)]
Repository: Log negative useCnt message together with stack trace
The message "close() called when useCnt is already zero" is logged with
level warning, and then if debug logging is enabled, the stack trace is
logged separately with level debug.
Log the message and the stack trace in the same call, so that they always
appear together in the output rather than potentially interleaved with
other log statements.
Change-Id: I1b5c1557ddc2d19f3f5b29baec96e62bc467d88a Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
David Pursehouse [Tue, 26 Jul 2016 01:16:18 +0000 (10:16 +0900)]
MergeFormatter: Suppress warning about unchecked conversion
The warning can be fixed by adding a type to the argument, but doing so
breaks the API and previous attempts to fix it in that way [1, 2] were
reverted [3, 4].
Dan Wang [Fri, 3 Jun 2016 23:39:45 +0000 (16:39 -0700)]
Push implementation of option strings
Example usage:
$ ./jgit push \
--push-option "Reviewer=j.doe@example.org" \
--push-option "<arbitrary string>" \
origin HEAD:refs/for/master
Stefan Beller has also made an equivalent change to CGit:
http://thread.gmane.org/gmane.comp.version-control.git/299872
Change-Id: I6797e50681054dce3bd179e80b731aef5e200d77 Signed-off-by: Dan Wang <dwwang@google.com>
Dave Borowitz [Thu, 14 Jul 2016 16:11:51 +0000 (12:11 -0400)]
DfsObjectDatabase: Expose PackList and move markDirty there
What's invalidated when an object database is "dirty" is not the whole
database, but rather a specific list of packs. If there is a race
between getting the pack list and setting the volatile dirty flag
where the packs are rescanned, we don't need to mark the new pack list
as dirty.
This is a fine point that only really applies if the decision of
whether or not to mark dirty actually requires introspecting the pack
list (say, its timestamps). The general operation of "take whatever
is the current pack list and mark it dirty" may still be inherently
racy, but the cost is not so high.
This variable has been populated and never used since it was
introduced in commit 5cf53fdacf28d5cabe7ad1ed154fe7f4971225a9
(Speed up clone/fetch with large number of refs, 2013-02-18).
Noted by FindBugs:
"BatchRefUpdate.java:359, UC_USELESS_OBJECT, Priority: Normal"
Change-Id: I7aacb49540aaee4a83db3d38b15633bb6c4773d0 Signed-off-by: Dan Wang <dwwang@google.com>
Matthias Sohn [Thu, 14 Jul 2016 20:59:14 +0000 (22:59 +0200)]
Fix AppServer build errors in Eclipse with <4.6 target platforms
9aa3748 added dummy implementations for loadRoleInfo() and
loadUserInfo() to class MappedLoginService to fix compile errors in
Eclipse when using 4.6 target platform which brings Jetty 9.3 adding
these two methods. Unfortunately this causes errors when using non 4.6
target platform coming with an older Jetty version. Fix this by
extracting the anonymous subclass of MappedLoginService which allows to
suppress the unused private method errors in Eclipse.
Change-Id: I75baeea7ff4502ce9ef2b541b3c0555da5535d79 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Dave Borowitz [Thu, 14 Jul 2016 15:40:55 +0000 (11:40 -0400)]
Invalidate DfsObjDatabase pack list when refs are updated
Currently, there is a race where a user of a DfsRepository in a single
thread may get unexpected MissingObjectExceptions trying to look up an
object that appears as the current value of a ref:
1. Thread A scans packs before scanning refs, for example by reading
an object by SHA-1.
2. Thread B flushes an object and updates a ref to point to that
object.
3. Thread A looks up the ref updated in (2). Since it is scanning refs
for the first time, it sees the new object SHA-1.
4. Thread A tries to read the object it found in (3), using the cached
pack list it got from (1). The object appears missing.
Allow implementations to work around this by marking the object
database's current pack list as "dirty." A dirty pack list means that
DfsReader will rescan packs and try again if a requested object is
missing. Implementations should mark objects as dirty any time the ref
database reads or scans refs that might be newer than a previously
cached pack list.
Matthias Sohn [Tue, 12 Jul 2016 15:12:41 +0000 (17:12 +0200)]
Merge branch 'stable-4.4'
* stable-4.4:
Log if Repository.useCnt becomes negative
Time based eviction strategy for repository cache
Add method to read time unit from config
Align include.path max depth with native git
Config load should not fail on unsupported or nonexistent include path
Allow using JDK 7 bootclasspath when compiling JGit using Java 8
Extract work queue to allow reusing it
Change-Id: I6aeedb1cb8b0c3068af344a719c80a03ae68fc23 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log