Masaya Suzuki [Fri, 18 May 2018 17:06:56 +0000 (10:06 -0700)]
Introduce ProtocolV2Hook
In Git protocol v2, UploadPack and ReceivePack have the same
capabilities and can process any protocol v2 request. For example, a
client can sent a "fetch" command to the "/git-receive-pack" endpoint.
This makes it difficult for existing hook interfaces. For example,
PreUploadHook takes UploadPack, but a "fetch" command may be received by
ReceivePack.
To resolve this skew, this change introduce a different hook interface
for the protocol v2. The hook takes a request that is independent to the
handlers (UploadPack, ReceivePack). Also this makes it clear what
parameters the hook is counting on, instead of keep track of the hook
using getters from UploadPack / ReceivePack.
Hugo Arès [Wed, 15 Aug 2018 13:54:29 +0000 (09:54 -0400)]
Fix GC run in foreground to not use executor
Since I3870cadb4, GC task was always delegated to an executor even when
background option was set to false. This was an issue because if more
than one GC object was instantiated and executed in parallel, only one GC
was actually running because of the single thread executor.
Change-Id: I8c587d22d63c1601b7d75914692644a385cd86d6 Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Matthias Sohn [Tue, 14 Aug 2018 22:31:37 +0000 (00:31 +0200)]
Declare ConfigConstants and GitProtocolConstants final
This avoids that we have to suppress API errors whenever we add a new
constant in a minor release. This change affects implementors only which
is ok to do in a minor release following OSGi semantic versioning rules.
Change-Id: Iece841886fbe00f1ba567c5ff68093c542ba265e Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Jonathan Tan [Mon, 13 Aug 2018 23:57:25 +0000 (16:57 -0700)]
Respond with shallow-info before wanted-refs
When fetching with protocol v2, git expects the shallow-info section to
appear before wanted-refs if both appear in the response. Teach
UploadPack to do this.
Change-Id: Ie26a91edcce5d27a1d727d7fba5c30e1144e118b Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Tan [Tue, 24 Jul 2018 15:41:17 +0000 (08:41 -0700)]
Support protocol v2 want-ref in UploadPack
UploadPack already allows the client to send wanted OIDs as "want"
lines. Extend UploadPack to also allow the client to send wanted ref
names as "want-ref" lines when the fetch is done using protocol v2.
The corresponding Git commit is 516e2b76bd ("upload-pack: implement
ref-in-want", 2018-06-28).
To support a two-stage rollout, two configuration variables are
provided: uploadpack.allowrefinwant (default "false") allows clients to
specify "want-ref" in their requests, and uploadpack.advertiserefinwant
(default "true") makes UploadPack advertise this capability. If
uploadpack.allowrefinwant is true but uploadpack.advertiserefinwant is
false, UploadPack will not advertise that it supports "want-ref", but it
will support it.
Change-Id: I3c24077949640d453af90d81a7f48ce4b8ac9833 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Thomas Wolf [Wed, 8 Aug 2018 11:36:35 +0000 (13:36 +0200)]
Ensure that JSch knows HostKeyAlgorithms ssh-rsa and ssh-dss
Without these registrations, JSch's up-front checks which algorithms
are available at all fail if the ssh config explicitly sets only these
algorithms.
Bug: 537790
Change-Id: Idb0431190a7f101913363ee95af6c8fcbda6c923 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Use org.eclipse.jgit.errors.CancelledException which is a subclass of
IOException instead of org.eclipse.jgit.api.errors.CanceledException in
order to avoid breaking API. We can reconsider this with the next major
version 6.0.
Bug: 536324
Change-Id: Ia6f84f59aa6b7d78b8fccaba24ade320a54f7458 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Jonathan Nieder [Sat, 4 Aug 2018 21:00:26 +0000 (14:00 -0700)]
RebaseCommand: Remove unnecessary fall through comments
It's easier to read a list of cases
case A:
case B:
case C:
... do something ...;
break;
without intervening "// fall through" lines separating the cases.
This also makes it clearer that JGit uses "$FALL-THROUGH$", not "fall
through", to suppress fallthrough warnings in switch statements.
Eclipse does not pay attention to the second comment style.
Change-Id: I5279a727aee2868854d06bfcaac8cb8186b4299e Signed-off-by: Jonathan Nieder <jrn@google.com>
Use project names instead of paths for the submodule name
Two submodules at the same path on different branches need not represent
the same repository, and two submodules at different paths can represent
the same one.
The C Git implementation uses the submodule name to internally manage
the submodule repositories under .git/modules. When a submodule
represents different repositories in different branches, it makes a
conflict inside .git/modules.
The current RepoCommand implementation uses submodule paths as the
submodule names. When the manifest file mounts different repositories to
the same path in different branches, this makes a situation described
above. To solve this issue, we can use the project name instead of
the path as the submodule name.
On the other hand, since repo v1.12.8~3^2 (repo: Support multiple
branches for the same project., 2013-10-11), a manifest file can mount
the same project to different paths. If we naively use the project
name as the submodule name, it makes a conflict in .git/modules, too.
This patch uses the project name as the submodule name basically, but
when the same project is mounted to different paths, it uses the project
name and path as the submodule name.
Matthias Sohn [Fri, 27 Jul 2018 12:16:42 +0000 (14:16 +0200)]
Merge branch 'stable-5.0'
* stable-5.0:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: I9717444c42ea89195dfcde8bb10121b9bb178e87 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 27 Jul 2018 12:06:14 +0000 (14:06 +0200)]
Merge branch 'stable-4.11' into stable-5.0
* stable-4.11:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: I780d4a1624fbb97a2acdf5321b9c3244b0626444 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 27 Jul 2018 11:38:21 +0000 (13:38 +0200)]
Merge branch 'stable-4.10' into stable-4.11
* stable-4.10:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: I2b44d862869d4453c57db668fc7c925da591f671 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 27 Jul 2018 09:03:11 +0000 (11:03 +0200)]
Merge branch 'stable-4.9' into stable-4.10
* stable-4.9:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: Ie7029bc91621af32e7bfd2e0d76a424b991b1995 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 27 Jul 2018 08:51:58 +0000 (10:51 +0200)]
Merge branch 'stable-4.8' into stable-4.9
* stable-4.8:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: I6ab2b951dd94a9fc1c4f5283847a3e2ec37d0895 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Fri, 27 Jul 2018 07:22:51 +0000 (08:22 +0100)]
Merge branch 'stable-4.7' into stable-4.8
* stable-4.7:
Prepare 4.7.3-SNAPSHOT builds
JGit v4.7.2.201807261330-r
Delete all loose refs empty directories
Use java.nio to delete path to get detailed errors
GC: Remove empty references folders
Do not ignore path deletion errors
Change-Id: Iadc8275fbaa3d6f7d08a96ab66d49f392f6aab78 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Jonathan Tan [Tue, 24 Jul 2018 22:35:16 +0000 (15:35 -0700)]
Send only 1 flush pkt at the end of v2 fetch
When processing a fetch using protocol v2, UploadPack#fetchV2 sends an
extraneous flush pkt when also sending a packfile (#sendPack sending its
own flush pkt). Update that method to only send the flush pkt if the
packfile is not being sent.
Change-Id: I7117a264bccd2d7f3a048645fcb8425a9d78d526 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
David Pursehouse [Tue, 17 Jul 2018 01:43:04 +0000 (10:43 +0900)]
ErrorProne: Enable check for misuse of ExpectedException
Any additional statements after the statement that is expected to
throw will never be executed in a passing test. This can lead to
inappropriately passing tests where later incorrect assertions are
skipped by the thrown exception. See [1] for examples.
There are no cases of this in the code base, but by enabling the
checker with severity ERROR we can ensure that we don't overlook
any future misuse of ExpectedException.
Jonathan Tan [Thu, 12 Jul 2018 17:58:28 +0000 (10:58 -0700)]
Always send refs' objects despite "filter" in pack
In a0c9016abd ("upload-pack: send refs' objects despite "filter"",
2018-07-09), Git updated the "filter" option in the fetch-pack
upload-pack protocol to not filter objects explicitly specified in
"want" lines, even if they match the criterion of the filter. Update
JGit to match that behavior.
Change-Id: Ia4d74326edb89e61062e397e05483298c50f9232 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Jonathan Nieder [Fri, 20 Jul 2018 07:48:22 +0000 (00:48 -0700)]
RepoCommand: record remote branch, group, shallow bit by default
Propagating more information from the manifest to the superproject
tends to work out well. These particular settings mean:
Recording the remote branch in .gitmodules allows
- "git submodule update --remote" to fetch the right branch
- Gerrit's submodule subscription feature to subscribe from the
right branch, if requested
Recording groups in .gitattributes allows commands like "git clone
--recurse-submodules=:(attr:platform-linux) $superproject" to clone
the correct set of projects.
Recording the shallow bit in .gitmodules means that "git clone
--recurse-submodules" will use shallow clone in submodules where that
was requested in the manifest.
Gerrit's supermanifest plugin records the remote branch and shallow
bit by default already, and most users of that plugin configure it to
record groups in .gitattributes as well, so for most users this will
be a no-op.
Change-Id: Id2ed47cbca5ce822bde517494673c86ab8c58da9 Signed-off-by: Jonathan Nieder <jrn@google.com>
Thomas Wolf [Wed, 18 Jul 2018 07:47:32 +0000 (09:47 +0200)]
ResolveMerger: don't try needlessly to delete directories
Don't try to delete folders if the merger chooses THEIRS, but all of
BASE, OURS, and THEIRS contain the folder.
Add a test for rebase with auto-stash and subdirectories that
verifies this case. The needless directory deletion and reporting
such directories in getModifiedFiles() was the root cause of bug
536880.
Note even with this fix, bug 536880 will not be fixed in all cases
yet. There may still be cases where the set of modified files ends
up containing directories. This will be dealt with in EGit where
this set is used. (See https://git.eclipse.org/r/#/c/126242/ .)
Bug: 536880
Change-Id: I62b4571a1c1d4415934a6cb4270e0c8036deb2e9 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
RacyGitTests depend on filesystem timer resolution. We wait for a file
system timer tick, remember that time as t1, modify a file and assume
that this file has a lastmodified of t1.
If this assumption is not fulfilled then ignore the test result.
Terry Parker [Mon, 16 Jul 2018 20:58:30 +0000 (13:58 -0700)]
GC: Trim more EWAHCompressedBitmaps to free unused memory
04b9f4436 fixed places where compressed bitmaps were holding on to their
full buffers, but missed this StoredBitmap.getBitmap() case where a
bitmap is resonstituted from an xor chain.
Change-Id: I7cf75d9e49c18a1a8a880a4df7e821502edc68a4 Signed-off-by: Terry Parker <tparker@google.com>
Currently SubmoduleAddCommand always uses the path as submodule name.
This patch lets the caller specify a submodule name.
SubmoduleUpdateCommand still does not make use of the submodule name
(see bug 535027) but Git does. To avoid triggering CVE-2018-11235,
do some validation on the name to avoid '..' path components.
[jn: fleshed out commit message, mostly to work around flaky CI]
Change-Id: I6879c043c6d7973556e2080387f23c246e3d76a5 Signed-off-by: Masaya Suzuki <masayasuzuki@google.com> Signed-off-by: Jonathan Nieder <jrn@google.com>
Make Reftable seek* and has* method names more consistent
Make the method names more consistent and their semantics simpler:
hasRef and seekRef to look up a single exact reference by name and
hasRefsByPrefix and seekRefsByPrefix to look up multiple references by
name prefix.
In particular, splitting hasRef into two separate methods for its
different uses makes DfsReftableDatabase.isNameConflicting easier to
follow.
[jn: fleshed out commit message]
Change-Id: I71106068ff3ec4f7e14dd9eb6ee6b5fab8d14d0b Signed-off-by: Minh Thai <mthai@google.com> Signed-off-by: Jonathan Nieder <jrn@google.com>
Reftable implementation of RefDatabase.getRefsByPrefix() should be
more performant, as references are filtered directly by prefix;
instead of fetching the whole subtree then filter by prefix.
Terry Parker [Tue, 10 Jul 2018 00:42:21 +0000 (17:42 -0700)]
GC: Trim EWAHCompressedBitmaps to free unused memory
The "Building bitmaps" GC phase fails for large repositories (repos with
10M objects use 1.25MB per uncompressed bitmap, and those with long
histories may build >25k bitmaps). Since these bitmaps xor well against
each other, the actual space needed for each compressed bitmap is
usually no more than a few KB. Calling trim() will ensure we aren't
holding on to excess memory.
Change-Id: I40bf78c730b9f6051da6025f9777ce27220a5b0a Signed-off-by: Terry Parker <tparker@google.com>
Dave Borowitz [Wed, 20 Jun 2018 12:48:37 +0000 (08:48 -0400)]
Make DfsConfig public
This may be convenient for downstream implementers who require a dummy
StoredConfig implementation, rather than making them reimplement the two
abstract StoredConfig methods.
After packaging references, the folders containing these references are
not deleted. In a busy repository, this causes operations to slow down
as traversing the references tree becomes longer.
Delete empty reference folders after the loose references have been
packed.
To avoid deleting a folder that was just created by another concurrent
operation, only delete folders that were not modified in the last 30
seconds.
Thomas Wolf [Tue, 3 Jul 2018 06:46:12 +0000 (08:46 +0200)]
Add response message, if any, on HTTP status 404
Try to give as much information as possible. The connection's
response message might contain additional hints as to why the
connection could not be established.
Bug: 536541
Change-Id: I7230e4e0be9417be8cedeb8aaab35186fcbf00a5 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Terry Parker [Wed, 27 Jun 2018 00:22:07 +0000 (17:22 -0700)]
Return parsed objects from TestRepository.commit/tree/blob()
It is convenient for TestRepository to return fully parsed
objects from its commit()/tree()/blob() methods, so that test
code doesn't have to remember to parse them before making
assertions about them.
Update TestRepostiory to return fully parsed objects.
Adjust the tests that are affected by this change in behavior.
Change-Id: I09d03d0c80ad22cb7092f4a2eaed99d40a10af63 Signed-off-by: Terry Parker <tparker@google.com>
Terry Parker [Tue, 26 Jun 2018 23:44:01 +0000 (16:44 -0700)]
Correctly handle initialization of shallow commits
In a new RevWalk, if the first object parsed is one of the
shallow commits, the following happens:
1) RevCommit.parseCanonical() is called on a new "r1" RevCommit.
2) RevCommit.parseCanonical() immediately calls
RevWalk.initializeShallowCommits().
3) RevWalk.initializeShallowCommits() calls lookupCommit(id),
creating and adding a new "r2" version of this same object and
marking its parents empty.
4) RevCommit.parseCanonical() initializes the "r1" RevCommit's
fields, including the parents.
5) RevCommit.parseCanonical()'s caller uses the "r1" commit that
has parents, losing the fact that it is a shallow commit.
This change passes the current RevCommit as an argument to
RevWalk.initializeShallowCommits() so that method can set its
parents empty rather than creating the duplicate "r2" commit.
Change-Id: I67b79aa2927dd71ac7b0d8f8917f423dcaf08c8a Signed-off-by: Terry Parker <tparker@google.com>
tparker [Mon, 25 Jun 2018 03:00:55 +0000 (20:00 -0700)]
Fix a GC scalability issue when selecting commit bitmaps
The previous algorithm selected commits by creating bitmaps at
each branch tip, doing a revwalk to populate each bitmap, and
looping in this way:
1) Select the remaining branch with the most commits (the branch
whose bitmap has the highest cardinality)
2) Select well-spaced bitmaps in that branch
3) Remove commits in the selected branch from the remaining
branch-tip bitmaps
4) Repeat at #1
This algorithm gave good commit selection on all branches but
a more uniform selection on "important" branches, where branch
length is the proxy for "important". However the algorithm
required N bitmaps of size M solely for the purpose of commit
selection, where N is the number of branch tips in the primary
GC pack, and M is the number of objects in the pack.
This new algorithm uses branch modification date as the proxy for
"important" branches, replacing the N*M memory allocation with a
single M-sized bitmap and N revwalks from new branch tips to
shared history (which will be short when there is a lot of shared
history).
GcCommitSelectionTest.testDistributionOnMultipleBranches verifies
that this algorithm still yields good coverage on all branches.
Change-Id: Ib6019b102b67eabb379e6b85623e4b5549590e6e Signed-off-by: Terry Parker <tparker@google.com>
Marco Miller [Thu, 21 Jun 2018 18:18:48 +0000 (14:18 -0400)]
ResolveMerger: Fix encoding with string; use bytes
This change fixes the issue [1]. Before this fix, a merge involving
the caching of consecutive yet similar filenames with Norwegian
characters [2] used to throw an IllegalStateException: Duplicate
stages not allowed. This was caused by inaccurate decoding of the
filenames, using string values assuming default encoding. In the
toString method of DirCacheEntry, used before through getPathString,
UTF-8 encoding is used, but the end result becomes default encoding,
through Object's default toString usage. The special characters in
those two consecutive (particular) filenames [2] were becoming the
very same decoded /single character, lending consecutive -but then
identical- filenames. Thus the perceived duplicate 0-staging of the
file(s).
Replace getPathString usage with getRawPath for this specific case,
or use byte array representations of cached entries instead of string.
Adding a test for this change is not possible, as there is no known
way to change the default encoding for filenames such as [2] (e.g.).
JGitTestUtil does write file contents through UTF-8, but encoding like
so does not apply to the actual file name. Hence there is no way to
create files with names properly made of special characters such as
[2]'s. And the test that is necessary for this case assumes such
Norwegian (or similar characters) filenames. Changing the default
locale programmatically in a test has no effect either. And changing
the LANG value passed to the JVM is only possible upon starting it.
then I expect to have a full history, just as though I had fetched
without --depth in the first place. Instead, it reports success
but does not fetch enough objects:
The false success indicates problems in the client and the server.
Git 2.18-rc2 (the client) ought to have been more defensive, noticing
the incomplete history. The greater error is in JGit (the server),
which neglects to send the objects requested.
When serving protocol v0 requests, JGit sends the correct objects by
taking unshallowCommits into account when generating the pack to send
to the client. Do the same in the protocol v2 code path. I forgot to
do this in v5.0.0.201806050710-rc3~6 (Teach UploadPack shallow fetch
in protocol v2, 2018-03-15).
Reported-by: Russ Cox <rsc@golang.org>
Change-Id: I282b45f47616a641b9e8d6210b4a070d3efdbb9b Signed-off-by: Jonathan Nieder <jrn@google.com>
projects / jgit.git / log