Use a new RevWalk for validating not advertised wants
Shadow commits in the RevWalk in the UploadPack object may cause the
UNINTERESTING flag not being carried over to their parents commits since
they were marked NO_PARENTS during the assumeShallow or
initializeShallowCommits call.
A new RevWalk needs to be created for this reason, but instead of
creating a new RevWalk from Repository, we can reuse the ObjectReader in
the RevWalk of UploadPack to load objects.
Change-Id: Ic3fee0512d35b4f555c60e696a880f8b192e4439
Signed-off-by: Zhen Chen <czhen@google.com>
ReflogWriter: Align auto-creation defaults with C git
Per git-config(1), core.logAllRefUpdates auto-creates reflogs for HEAD
and for refs under heads, notes, tags, and for HEAD. Add notes and
remove stash from ReflogWriter#shouldAutoCreateLog. Explicitly force
writing reflogs for refs/stash at call sites, now that this is
supported.
Change-Id: I3a46d2c2703b7c243e0ee2bbf6948279800c485c
Support force writing reflog on a per-update basis
Even if a repository has core.logAllRefUpdates=true, ReflogWriter does
not create reflog files unless the refs are under a hard-coded list of
prefixes, or unless the forceWrite bit is set. Expose the forceWrite bit
on a per-update basis in RefUpdate/BatchRefUpdate/ReceiveCommand,
creating RefLogWriters as necessary.
Change-Id: Ifc851fba00f76bf56d4134f821d0576b37810f80
Ensure ReflogWriter only works with a RefDirectory
The ReflogWriter constructor just took a Repository and called
getDirectory() on it to figure out the reflog dirs, but not all
Repository instances use this storage format for reflogs, so it's
incorrect to attempt to use ReflogWriter when there is not a
RefDirectory directly involved. In practice, ReflogWriter was mostly
only used by the implementation of RefDirectory, so enforcing this is
mostly just shuffling around calls in the same internal package.
The one exception is StashDropCommand, which writes to a reflog lock
file directly. This was a reasonable implementation decision, because
there is no general reflog interface in JGit beyond using
(Batch)RefUpdate to write new entries to the reflog. So to implement
"git stash drop <N>", which removes an arbitrary element from the
reflog, it's fair to fall back to the RefDirectory implementation.
Creating and using a more general interface is well beyond the scope of
this change.
That said, the old behavior of writing out the reflog file even if
that's not the reflog format used by the given Repository is clearly
wrong. Fail fast in this case instead.
Change-Id: I9bd4b047bc3e28a5607fd346ec2400dde9151730
HttpConfig: load user config before reading values from it
Same problem as in commit c227268: openUserConfig() just creates the
FileBasedConfig object, but doesn't read the file yet. An explicit
load() is needed.
As HttpConfig is read-only this omission did not cause any bad effects,
but it simply ignored values from the user config. Most uses of
HttpConfig go through the two-argument constructor, though, where
HttpConfig is given an already loaded repo config.
Change-Id: Ibe7c562c17d6ef37de8b661ab7f6fa0246db01a2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
SystemReader.openUserConfig() does not load the config yet; an
explicit StoredConfig.load() is needed.
Bug: 374703
Change-Id: I1c397e2fb1a07ac4d9de3675d996417734ff90e9
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
When the submodule already exists, it is fetched instead of
cloned.
Use the fetch callback instead of clone callback in this case.
Change-Id: I170c21ab92b4117f25fdf940fe6807f214b04d39
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
By default, this is turned off unless cmd.setFetch(true) is given. It
will default to true in a future release to mimic c-git behaviour.
This is needed to prevent Eclipse from crashing with "Missing unknown
[REF]" when cloning a repo with submodules.
Bug: 470318
Change-Id: I8ae37c7c5bd2408cead8d57dd13e93e01e0e9dc1
Signed-off-by: Michael FIG <michael@fig.org>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
When a https connection could not be established because the SSL
handshake was unsuccessful, TransportHttp would unconditionally
throw a TransportException.
Other https clients like web browsers or also some SVN clients
handle this more gracefully. If there's a problem with the server
certificate, they inform the user and give him a possibility to
connect to the server all the same.
In git, this would correspond to dynamically setting http.sslVerify
to false for the server.
Implement this using the CredentialsProvider to inform and ask the
user. We offer three choices:
1. skip SSL verification for the current git operation, or
2. skip SSL verification for the server always from now on for
requests originating from the current repository, or
3. always skip SSL verification for the server from now on.
For (1), we just suppress SSL verification for the current instance of
TransportHttp.
For (2), we store a http.<uri>.sslVerify = false setting for the
original URI in the repo config.
For (3), we store the http.<uri>.sslVerify setting in the git user
config.
Adapt the SmartClientSmartServerSslTest such that it uses this
mechanism instead of setting http.sslVerify up front.
Improve SimpleHttpServer to enable setting it up also with HTTPS
support in anticipation of an EGit SWTbot UI test verifying that
cloning via HTTPS from a server that has a certificate that doesn't
validate pops up the correct dialog, and that cloning subsequently
proceeds successfully if the user decides to skip SSL verification.
Bug: 374703
Change-Id: Ie1abada9a3d389ad4d8d52c2d5265d2764e3fb0e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Git has a rather elaborate mechanism to specify HTTP configuration
options per URL, based on pattern matching the URL against "http"
subsection names.[1] The URLs used for this matching are always the
original URLs; redirected URLs do not participate.
* Scheme and host must match exactly case-insensitively.
* An optional user name must match exactly.
* Ports must match exactly after default ports have been filled in.
* The path of a subsection, if any, must match a segment prefix of
the path of the URL.
* Matches with user name take precedence over equal-length path
matches without, but longer path matches are preferred over
shorter matches with user name.
Implement this for JGit. Factor out the HttpConfig from TransportHttp
and implement the matching and override mechanism.
The set of supported settings is still the same; JGit currently
supports only followRedirects, postBuffer, and sslVerify, plus the
JGit-specific maxRedirects key.
Add tests for path normalization and prefix matching only on segment
separators, and use the new mechanism in SmartClientSmartServerSslTest
to disable sslVerify selectively for only the test server URLs.
Compare also bug 374703 and bug 465492. With this commit it would be
possible to set sslVerify to false for only the git server using a
self-signed certificate instead of having to switch it off globally
via http.sslVerify.
[1] https://git-scm.com/docs/git-config
Change-Id: I42a3c2399cb937cd7884116a2a32fcaa7a418fcb
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Add PushConfig class with PushRecurseSubmodulesMode
This will be used later when adding for support for recursing
submodules on push.
Change-Id: Ie2a183e5404a32046de9f6524e6ceeec37919671
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Fix missing RefsChangedEvent when packed refs are used
With atomic ref updates using packed refs, JGit did not fire a
RefsChangedEvent. This resulted in a user-visible regression in
EGit: the UI would not update after a "Fetch from upstream...".
Presumably it would also make Gerrit miss out on ref changes?
Strengthen the BatchRefUpdateTest by also asserting the expected
number of RefsChangedEvents, and ensure modCnt is incremented in
RefDirectory.commitPackedRefs() when refs really changed (as opposed
to some internal housekeeping operation, such as packing loose refs).
Bug: 521296
Change-Id: Ia985bda1d99f45a5f89c8020ca4845e7a66e743e
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Much of the time the caller can specify a RefSpec succinctly using a
string, and doesn't care about calling setters. Add a convenience method
for this case, and use it where applicable in JGit core.
Change-Id: Ic3fac7fc568eee4759236a5264d2e7e5f9b9716d
ReceivePack: clear advertised .haves if application changes refs
An application can choose to invoke setAdvertisedRefs multiple times,
for example several AdvertiseRefsHook installed in a chain. Each of
these invocations populates the advertisedHaves collection with the
unique set of ObjectIds.
This can lead to a server over-advertising with ".have" lines if the
first hook pushes in a lot of references, and the second hook filters
this to a subset. ReceivePack will advertise the unique objects from
the first hook using ".have" lines, which may lead to a huge
advertisement sent to the client.
This can also contribute to a very slow connectivity check after the
pack is parsed as ReceivePack calls markUninteresting on every commit
in advertisedHaves. This may require expanding a lot of subtrees to
mark all trees as uninteresting as well. On a very big repository
this can lead to a many-second stall.
Clear the advertisedHaves collection any time the refs are updated.
Add a test to verify the correct set of objects was sent.
Change-Id: I97f6998d0597251444a2e846a3ea1f461bae96f9
DfsGarbageCollector: support disabling conversion to reftable
When a repository is initially created using only reftable but doesn't
yet have a GC pack, the garbage collector shouldn't scan the ref
database. Support disabling the reftable conversion path.
Change-Id: Iaaf3a4375cd43760b7181c1bd10244de8b2c5d9e
Remove dead warning about minUpdaeIndex and maxUpdateIndex affecting refresh
DfsGarbageCollector always performs refreshes. This warning was from
a prior iteration of the patch set and should have been removed before
the change was merged.
Change-Id: Id4750bb6c1c177e1ef040fd22912f434bbb5e992
Just return configValue instead of transforming name().
Change-Id: I7f94ab2e206e93d1370467e187c5e68e8f6a8836
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
If a ReftableConfig has been supplied by the caller, write out a
reftable as a sibling of the the GC pack, alongside the heads.
To bootstrap from a non-reftable system, the refs are read from the
DfsRefDatabase if no GC reftables are present. Its assumed the
references are fully current, and do not need to be merged with any
other reftables. Any non-GC reftables will be pruned at the end of
the GC cycle, just like any packs that were replaced.
If a GC reftable is present, all existing reftables are compacted, and
references from DfsRefDatabase are only used to seed the packer. Its
assumed these are consistent with each other.
Change-Id: Ie397eb58aaaefb6865c816d9b39de3ac12998019
Fix Daemon.stop() to actually stop the listener thread
ServerSocket.accept() is not interruptible: a thread busy in accept()
may not react to Thread.interrupt() and may not return from accept()
via an InterruptedException. Close the socket instead to make the
daemon's listener thread terminate.
* Close the listening socket to get the listening thread to exit
instead of interrupting it.
* Add a stopAndWait() method that stops the listening thread and
then waits until it has indeed finished.
* Set SO_REUSE_ADDRESS on the listening socket.
Bug: 376369
Change-Id: I9d6014103e6dcb0173daea134feb44dc52c5c69a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Remove workaround for bug in Java's ReferenceQueue
Sun's Java 5, 6, 7 implementation had a bug [1] where a Reference can be
enqueued and dequeued twice on the same reference queue due to a race
condition within ReferenceQueue.enqueue(Reference).
This bug was fixed for Java 8 [2] hence remove the workaround.
[1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6837858
[2] http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/858c75eb83b5
Change-Id: I2deeb607e3d237f9f825a207533acdee305c7e73
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Use Config.getEnum() to read the IgnoreSubmoduleMode
Doing so goes through the TypedConfigGetter and thus allows library
clients (for instance EGit) to warn about invalid configurations.
Change-Id: If1080ad90b8aff54a903d4d75637614faad6469b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
While parsing .gitmodules, the name of the submodule subsection is
purely arbitrary: it frequently is the path of the submodule, but
there's no requirement for it to be. By building a map of paths to
the section name in .gitmodules, we can more accurately return
the submodule URL.
Bug: 508801
Change-Id: I8399ccada1834d4cc5d023344b97dcf8d5869b16
Also-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
FsckPackParser: Fix "Empty block should be documented" warnings
Add "Do nothing" comments, consistent with other empty methods in
the same class.
Change-Id: I27a13a402e94104af617be0e14d8982e75fa73bd
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Don't convert a lot of ObjectId to String stored in generic
java.util.HashSet. This is a very expensive way to store objects.
Instead rely on "this" from the FsckPackParser to lookup information
about the objects in this pack file, which lets the verify code avoid
sorting the object list.
Use ObjectIdOwnerMap, which is the most efficient format JGit has
for storing lots of objects.
Change-Id: Ib68f93acb4d91b96d0a44c0612f704500d332ac1
This simplifies the logic about allocation of the DfsReader, and
clarifies the code considerably by using smaller scopes with less
indentation.
A few static imports from PackExt and slightly shorter variable names
make for a more understandable-at-glance implementation.
Change-Id: Iaf5a0e14fe0349215d9e44446f68d1129ad3bb3d
The simpler algorithm is to load all branch tips into an ObjectWalk
and run that walk exactly once. This avoids redoing work related to
parsing and considering trees reused across side branches.
Move the connectivity check into its own helper method. This moves it
left one level of identation, and makes it easier to fit the method's
logic with less line wrapping.
Add a "Counting objects..." progress monitor around this phase. Its
what is used when a server receives a push and is also trying to
verify the client sent all required objects.
Change-Id: I4d53d75d0cdd1a13fff7d513a6ae0b2d14ea4090
Pass along the original exception when an ssh connection fails
Otherwise, the stack trace doesn't really tell anything.
See for instance [1].
[1] https://www.eclipse.org/forums/index.php/t/1088535/
Change-Id: If22f2c63c36fec6b32818d2c2acecf20531b4185
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
ReftableStack maintains multiple open reftables
in an AutoCloseable format, making it easier for
higher level code to handle multiple files.
Change-Id: I7ac35c18e67b7e771fb3de29169d1ee50fab62ca
Reftable storage in DFS is related to pack storage. Reftables are
stored in the same namespace, but with PackExt.REFTABLE. Include
the set of DfsReftable instances in the PackList and export some
helpers to access the tables.
Change-Id: I6a4f5f953ed6b0ff80a7780f4c6cbcc5eda0da3e
dfs: support reading reftables through DfsBlockCache
DfsBlockCache directly shares its internal byte[] with ReftableReader,
avoding copying between the DfsBlockCache and the BlockReader
instances used by ReftableReader.
Change-Id: Icaa4f40052b26f952681414653a8b5314b7c2c23
Some repositories can have a policy that do not accept certain blobs. To
check if the incoming pack file contains such blobs, ObjectChecker can
be used. However, this ObjectChecker is not called by PackParser if the
blob is stored as a whole. This is because the object can be so large
that it doesn't fit in memory.
This change introduces BlobObjectChecker. This interface takes chunks of
a blob instead of the entire object. ObjectChecker can optionally return
a BlobObjectChecker. This won't change existing ObjectChecker
implementation; existing implementation continues to receive deltified
blob objects only.
Change-Id: Ic33a92c2de42bd7a89786a4da26b7a648b25218d
Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
FetchCommand: pass on CredentialsProvider to submodule fetches
When a JGit API command is implemented in terms of other API
commands, the child command must "inherit" all relevant settings.
Calling configure() ensures that the CredentialsProvider and the
connection timeout are propagated correctly.
Bug: 515325
Change-Id: I948e306693a9edb7b199a735877413b6eddcfba4
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Exclude file matching: fix backtracking on match failures after **
** matching always tries the empty match first. If a mismatch occurs
later, the ** must be extended by exactly one segment and matching must
resume with the matcher following the ** matcher.
Bug: 520920
Change-Id: Id019ad1c773bd645ae92e398021952f8e961f45c
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Fix path pattern matching to work also for gitattributes
Path pattern matching for attribute rules is different than matching
for excluded files.
The first difference concerns patterns without slashes. For
gitattributes those must match on the last component only, not on
any earlier segment. This is true also for directory-only patterns.
The second difference concerns directory-only patterns. Those also
must not match on a prefix or segment except the last one. They do
not apply recursively to all files beneath.
And third, matches only on a prefix must match for gitattributes
only if the last matcher was "/**".
Add a new parameter for such path matching to IMatcher.matches() and
pass it through as appropriate (false for gitignore, true for
gitattributes). As far as gitignore is concerned, there is no change.
New tests have been added, and some existing attribute matching tests
have been fixed since they operated on wrong assumptions.
Bug: 508568
Change-Id: Ie825dc2cac8a85a72a7eeb0abb888f3193d21dd2
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
C git silently ignores invalid tagopt values; so make JGit behave the
same way.
Bug: 429625
Change-Id: I99587cc46c7e0c19348bcc63f602038fa9a7f378
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Reading RefSpecs from a Config can be seen as another typed value
conversion, so add a getter to Config and to TypedConfigGetter. Use
it in RemoteConfig.
Doing this allows clients of the JGit library to customize the
handling of invalid RefSpecs in git config files by installing a
custom TypedConfigGetter.
Bug: 517314
Change-Id: I0ebc0f073fabc85c2a693b43f5ba5962d8a795ff
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Make the handling of typed values somewhat configurable by using
a separate converter. The default converter is the same as before;
just the implementations of the getters were moved. They also still
raise IllegalArgumentException on invalid values as before.
The converter can be set globally via Config.setTypedConfigGetter(),
which EGit can use in its core Activator to plug in a variant that
catches the IllegalArgumentException, logs the problem, and then
returns the default value.
In this way the behavior for other users of the JGit library is
unchanged, while EGit can deal gracefully with invalid git configs.
Bug: 520978
Change-Id: Ie8f81d206e358b6cc57aa29b9d7ad2a5d34b86a1
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
In Eclipse Oxygen, the following warning is emitted:
At least one of the problems in category 'synthetic-access' is not
analysed due to a compiler option being ignored
Removing the suppression gets rid of the warning.
Change-Id: Ibfe5cc1e347150b699f54e2f204ab5ee770da202
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Except for %p and %r and partially %C, we can do token substitutions
as defined by OpenSSH inside the config file parser. %p and %r can
be replaced only if specified in the config; if not, it would be the
caller's responsibility to replace them with values obtained from the
URI to connect to.
Jsch doesn't know about token substitutions at all. By doing the
replacements as good as we can in the config file parser, we can
make Jsch support most of these tokens.
%i is not handled at all as Java has no concept of a "user ID".
Includes unit tests.
Bug: 496170
Change-Id: If9d324090707de5d50c740b0d4455aefa8db46ee
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Ensure the Jsch instance used knows about ~/.ssh/config. This
enables Jsch to honor more user configurations (see
com.jcraft.jsch.Session.applyConfig()), in particular also the
UserKnownHostsFile configuration, or additional identities given
via multiple IdentityFile entries.
Turn JGit's OpenSshConfig into a full parser that can be a
Jsch-compliant ConfigRepository. This avoids a few bugs
in Jsch's OpenSSHConfig and keeps the JGit-facing interface
unchanged. At the same time we can supply a JGit OpenSshConfig
instance as a ConfigRepository to Jsch. And since they'll both
work from the same object, we can also be sure that the parsing
behavior is identical.
The parser does not handle the "Match" and "Include" keys, and it
doesn't do %-token substitutions (yet).
Note that Jsch doesn't handle multi-valued UserKnownHostFile
entries as known by modern OpenSSH.[1]
[1] http://man.openbsd.org/OpenBSD-current/man5/ssh_config.5
Additional tests for new features are provided in OpenSshConfigTest.
Bug: 490939
Change-Id: Ic683bd412fa8c5632142aebba4a07fad4c64c637
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Kenneth Endfinger
Till Brychcy
Zhen Chen
Dmitry Neverov
Dave Borowitz
Thomas Wolf
David Pursehouse
Michael FIG
Shawn Pearce
Matthias Sohn
Masaya Suzuki