Keep track of the original cause for a packfile invalidation.
It is needed for the sysadmin to understand if there is a real
underlying filesystem problem and repository corruption or if it is
simply a consequence of a concurrency of Git operations (e.g. repack
or GC).
Change-Id: I06ddda9ec847844ec31616ab6d17f153a5a34e33
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Due to finite filesystem timestamp resolution the last modified
timestamp of files cannot detect file changes which happened in the
immediate past (less than one filesystem timer tick ago).
Read and consider file size also, so that differing file size can help
to more accurately detect file changes without reading the file content.
Use bulk read to avoid multiple stat calls to retrieve file attributes.
Change-Id: I974288fff78ac78c52245d9218b5639603f67a46
Signed-off-by: Luca Milanesio <luca.milanesio@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
ObjectChecker: Report .gitmodules files found in the pack
In order to validate .gitmodules files, we first need to find them
in the incoming pack.
Do it in the ObjectChecker stage. Check in the tree objects if they
point to a .gitmodules file and report the tree id and the .gitmodules
blob id.
This can be used later to check if the file is in the root of the
project and if the contents are good.
While we're here, make isMacHFSGit more accurate by detecting variants
of filenames that vary in case.
[jn: tweaked NTFS and HFS+ checking; added more tests]
Change-Id: I70802e7d2c1374116149de4f89836b9498f39582
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Jonathan Nieder <jrn@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
SubmoduleAddCommand: Reject submodule URIs that look like cli options
In C git versions before 2.19.1, the submodule is fetched by running
"git clone <uri> <path>". A URI starting with "-" would be interpreted
as an option, causing security problems. See CVE-2018-17456.
Refuse to add submodules with URIs, names or paths starting with "-",
that could be confused with command line arguments.
[jn: backported to JGit 4.7.y, bringing portions of Masaya Suzuki's
dotdot check code in v5.1.0.201808281540-m3~57 (Add API to specify
the submodule name, 2018-07-12) along for the ride]
Change-Id: I2607c3acc480b75ab2b13386fe2cac435839f017
Signed-off-by: Ivan Frade <ifrade@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The following commits introduced in stable-4.5 and stable-4.9
introduced some minor API additions in service releases.
f7ceeaa2 FileRepository: Add pack-based inserter implementation
085d1f95 Make PackInserter public
10e65cb4 Fix LockFile semantics when running on NFS
Change-Id: I4afed7e0395cf93d828e671080e3ec9ddf20987d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
These methods were introduced for 4.11.1 so we have to silence the API
error adding API in a service release raises.
Change-Id: Ic847cebbed439912d3979ec2ec1809f77a28f61e
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
LFS: support merge/rebase/cherry-pick/diff/compare with LFS files
Respect merge=lfs and diff=lfs attributes where required to replace (in
memory) the content of LFS pointers with the actual blob content from
the LFS storage (and vice versa when staging/merging).
Does not implement general support for merge/diff attributes for any
other use case apart from LFS.
Change-Id: Ibad8875de1e0bee8fe3a1dffb1add93111534cae
Signed-off-by: Markus Duft <markus.duft@ssi-schaefer.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Merges are performed using the raw text as stored in the git
repository. When we write the merge result, we must apply the
correct CRLF settings. Otherwise the line endings in the result
will be wrong.
Bug: 499615
Change-Id: I37a9b987e9404c97645d2720cd1c7c04c076a96b
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Silence API warnings for reintroduced ResolveMerger#processEntry
This was silenced before but suppression was unintentionally lost in
merge commit 6858339c1e.
This method was removed in 4.9.0 and reintroduced in 4.9.1 to avoid
breaking EMF compare versions which were built against older versions.
See: abf420302b
Change-Id: I152d58ac885e044bcab682b9423f6cc83b667989
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When running on NFS there was a chance that JGits LockFile
semantic is broken because File#createNewFile() may allow
multiple clients to create the same file in parallel. This
change provides a fix which is only used when the new config
option core.supportsAtomicCreateNewFile is set to false. The
default for this option is true. This option can only be set in the
global or the system config file. The repository config file is not
taken into account in this case.
If the config option core.supportsAtomicCreateNewFile is true
then File#createNewFile() is trusted and the behaviour doesn't
change.
But if core.supportsAtomicCreateNewFile is set to false then after
successful creation of the lock file a hardlink to that lock file is
created and the attribute nlink of the lock file is checked to be 2. If
multiple clients manage to create the same lock file nlink would be
greater than 2 showing the error.
This expensive workaround is described in
https://www.time-travellers.org/shane/papers/NFS_considered_harmful.html
section III.d) "Exclusive File Creation"
Change-Id: I3d2cc48d8eb280d5f7039eb94da37804f903be6a
Silence invalid @since 3.5 API warning on ResolveMerge#processEntry
This method was removed in 4.9 and reintroduced in
I48ba4308dee73925fa32d6c2fd6b5fd89632c571 as deprecated in 4.9.1 in
order to help EMF Compare to avoid breakage.
Change-Id: Ia638517178313da42ae13ebcf88ad535d9a02723
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Reintroduce protected method which removal broke EMF Compare
So far we follow OSGi semantic versioning [1] which says the following:
"A change in the second (minor) part of the version signals that the
change is backward compatible with consumers of the API package but not
with the providers of that API. That is, when the API package goes from
version 1.5 to 1.6 it is no longer compatible with a provider of that
API but consumers of that API are backward compatible with that API
package."
The change Ib5fbf17bdaf727bc5d0e106ce88f2620d9f87a6f broke EMF Compare
which subclasses ResolveMerger since we added a new parameter to the
protected ResolveMerger.processEntry() method. According to the above
cited OSGi semantic versioning this is ok, implementers should expect
that they break on minor version changes of the API they implement.
This change reintroduces the old processEntry() method in order to help
avoid breakage for existing EMF Compare versions which expect breakage
also for the implementer case only for major version change (in this
case from JGit 4.x to 5.x).
[1] http://www.osgi.org/wp-content/uploads/SemanticVersioning1.pdf
See: https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03431.html
Change-Id: I48ba4308dee73925fa32d6c2fd6b5fd89632c571
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
git-core follows HTTP redirects so JGit should also provide this.
Implement config setting http.followRedirects with possible values
"false" (= never), "true" (= always), and "initial" (only on GET, but
not on POST).[1]
We must do our own redirect handling and cannot rely on the support
that the underlying real connection may offer. At least the JDK's
HttpURLConnection has two features that get in the way:
* it does not allow cross-protocol redirects and thus fails on
http->https redirects (for instance, on Github).
* it translates a redirect after a POST to a GET unless the system
property "http.strictPostRedirect" is set to true. We don't want
to manipulate that system setting nor require it.
Additionally, git has its own rules about what redirects it accepts;[2]
for instance, it does not allow a redirect that adds query arguments.
We handle response codes 301, 302, 303, and 307 as per RFC 2616.[3]
On POST we do not handle 303, and we follow redirects only if
http.followRedirects == true.
Redirects are followed only a certain number of times. There are two
ways to control that limit:
* by default, the limit is given by the http.maxRedirects system
property that is also used by the JDK. If the system property is
not set, the default is 5. (This is much lower than the JDK default
of 20, but I don't see the value of following so many redirects.)
* this can be overwritten by a http.maxRedirects git config setting.
The JGit http.* git config settings are currently all global; JGit has
no support yet for URI-specific settings "http.<pattern>.name". Adding
support for that is well beyond the scope of this change.
Like git-core, we log every redirect attempt (LOG.info) so that users
may know about the redirection having occurred.
Extends the test framework to configure an AppServer with HTTPS support
so that we can test cloning via HTTPS and redirections involving HTTPS.
[1] https://git-scm.com/docs/git-config
[2] 6628eb41db
[3] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
CQ: 13987
Bug: 465167
Change-Id: I86518cb76842f7d326b51f8715e3bbf8ada89859
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
The merger is now able to react to the use of the merge attribute.
The value unset and the custom value 'binary' are handled (-merge
and merge=binary)
Since the specification of the merge attribute states that when the
attribute is unset, ours version must be kept in case of a conflict, we
don't overwrite the file but keep the local version.
Bug: 517128
Change-Id: Ib5fbf17bdaf727bc5d0e106ce88f2620d9f87a6f
Signed-off-by: Mathieu Cartaud <mathieu.cartaud@obeo.fr>
When running an automatic GC on a FileRepository, when the caller
passes a NullProgressMonitor, run the GC in a background thread. Use a
thread pool of size 1 to limit the number of background threads spawned
for background gc in the same application. In the next minor release we
can make the thread pool configurable.
In some cases, the auto GC limit is lower than the true number of
unreachable loose objects, so auto GC will run after every (e.g) fetch
operation. This leads to the appearance of poor fetch performance.
Since these GCs will never make progress (until either the objects
become referenced, or the two week timeout expires), blocking on them
simply reduces throughput.
In the event that an auto GC would make progress, it's still OK if it
runs in the background. The progress will still happen.
This matches the behavior of regular git.
Git (and now jgit) uses the lock file for gc.log to prevent simultaneous
runs of background gc. Further, it writes errors to gc.log, and won't
run background gc if that file is present and recent. If gc.log is too
old (according to the config gc.logexpiry), it will be ignored.
Change-Id: I3870cadb4a0a6763feff252e6eaef99f4aa8d0df
Signed-off-by: David Turner <dturner@twosigma.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Silence API error for new method added to abstract MergeStrategy
OSGi semantic versioning rules allow to break implementors of an API in
a minor version.
Change-Id: I4ada3e6455e8e8e1bb8fb71affa0a1b36bd46fc4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Only mark packfile invalid if exception signals permanent problem
Add NoPackSignatureException and UnsupportedPackVersionException to
explicitly mark permanent unrecoverable problems with a pack
Assume problem with a pack is permanent only if we are sure the
exception signals a non-transient problem we can't recover from:
- AccessDeniedException: we lack permissions
- CorruptObjectException: we detected corruption
- EOFException: file ended unexpectedly
- NoPackSignatureException: pack has no pack signature
- NoSuchFileException: file has gone missing
- PackMismatchException: pack no longer matches its index
- UnpackException: unpacking failed
- UnsupportedPackIndexVersionException: unsupported pack index version
- UnsupportedPackVersionException: unsupported pack version
Do not attempt to handle Errors since they are thrown for serious
problems applications should not try to recover from.
Change-Id: I2c416ce2b0e23255c4fb03a3f9a0ee237f7a484a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Archived zip files for a same commit have different MD5 hash because
mdate and mdate in the header of zip entries are not specified. In
this case, Commons Compress sets an archived time.
In the original git implementation, it's set a commit time:
e2b2d6a172/archive.c (L378)
By this fix, archive command sets the commit time to ZipArchiveEntry
when RevCommit is given as an archiving target.
Change-Id: I30dd8710e910cdf42d57742f8709e9803930a123
Signed-off-by: Naoki Takezoe <takezoe@gmail.com>
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
TreeWalk provides the new method getEolStreamType. This new method can
be used with EolStreamTypeUtil in order to create a wrapped InputStream
or OutputStream when reading / writing files. The implementation
implements support for the git configuration options core.crlf, core.eol
and the .gitattributes "text", "eol" and "binary"
CQ: 10896
Bug: 486563
Change-Id: Ie4f6367afc2a6aec1de56faf95120fff0339a358
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Attributes MacroExpander implements macros used in git attributes. This
is implemented inside the TreeWalk using a lazy created MacroExpander.
In addition, the macro expander caches the global and info attributes
node in order to provide fast merge of attributes.
Change-Id: I2e69c9fc84e9d7fb8df0a05817d688fc456d8f00
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Add the new class Attributes holding multiple Attribute(s)
Attributes represents a semantic collector of Attribute(s) and replaces
the anonymous Map<String,Attribute>. This class will be returned by
TreeWalk.getAttributes(). It offers convenient access to the attributes
wrapped in the Attributes object. Adds preparations for a future
Attribute Macro Expansion
Change-Id: I8348c8c457a2a7f1f0c48050e10399b0fa1cdbe1
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Remove no longer needed outdated API warning filter
This fixes a warning saying this filter isn't needed anymore.
Change-Id: If77056378befe86c1773950dbe48a82c833fd532
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add an attribute accessor to CanonicalTreeParser and use it in Treewalk
When checking out a branch we need to access the attributes stored
in the tree to be checked out. E.g. directly after a clone we checkout
the remote HEAD. In this case index and workingtree are still empty.
So we have to search the tree to be checked out for attributes.
Change-Id: I6d96f5d095ed2e3c259d4b12124e404f5215bd9f
Adds the git attributes computation on the treewalk
Adds the getAttributes feature to the tree walk. The computation of
attributes needs to be done by the TreeWalk since it needs both a
WorkingTreeIterator and a DirCacheIterator.
Bug: 342372
CQ: 9120
Change-Id: I5e33257fd8c9895869a128bad3fd1e720409d361
Signed-off-by: Arthur Daussy <arthur.daussy@obeo.fr>
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Silence API errors for new methods added to BitmapBuilder
These API changes were introduced in f523f21e and 73474466.
BitmapBuilder is an interface to be implemented by implementors of this
interface. According to OSGi semantic versioning rules breaking API
changes require update of the minor version only if implementors of the
API have to be adapted and the changes do not affect clients of the API.
Change-Id: If45d204181ea9bc788b6b57693ca17b1847564c7
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Delete non empty directories before checkout a path
If the checkout path is currently a non-empty directory (and was a link
or a regular file before), this directory will be removed before
performing checkout, but only if the checkout path is specified.
Bug: 474973
Change-Id: Ifc6c61592d9b54d26c66367163acdebea369145c
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Luca Milanesio
Matthias Sohn
Ivan Frade
Markus Duft
Thomas Wolf
David Pursehouse
Christian Halstrick
Mathieu Cartaud
David Turner
Naoki Takezoe
Bo Zhang
Hugo Arès
Ivan Motsch
Shawn Pearce
Arthur Daussy
Andrey Loskutov