David Pursehouse [Fri, 22 Jul 2016 08:21:20 +0000 (17:21 +0900)]
FileLfsServlet: Return HTTP 422 instead of 400
According to the specification [1], the error response status code
should be 422 when there is a validation error with one or more of
the objects in the request
David Pursehouse [Tue, 26 Jul 2016 01:16:18 +0000 (10:16 +0900)]
MergeFormatter: Suppress warning about unchecked conversion
The warning can be fixed by adding a type to the argument, but doing so
breaks the API and previous attempts to fix it in that way [1, 2] were
reverted [3, 4].
David Pursehouse [Fri, 22 Jul 2016 05:05:19 +0000 (14:05 +0900)]
Repository: Log negative useCnt message together with stack trace
The message "close() called when useCnt is already zero" is logged with
level warning, and then if debug logging is enabled, the stack trace is
logged separately with level debug.
Log the message and the stack trace in the same call, so that they always
appear together in the output rather than potentially interleaved with
other log statements.
Change-Id: I1b5c1557ddc2d19f3f5b29baec96e62bc467d88a Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Dan Wang [Fri, 3 Jun 2016 23:39:45 +0000 (16:39 -0700)]
Push implementation of option strings
Example usage:
$ ./jgit push \
--push-option "Reviewer=j.doe@example.org" \
--push-option "<arbitrary string>" \
origin HEAD:refs/for/master
Stefan Beller has also made an equivalent change to CGit:
http://thread.gmane.org/gmane.comp.version-control.git/299872
Change-Id: I6797e50681054dce3bd179e80b731aef5e200d77 Signed-off-by: Dan Wang <dwwang@google.com>
Dave Borowitz [Thu, 14 Jul 2016 16:11:51 +0000 (12:11 -0400)]
DfsObjectDatabase: Expose PackList and move markDirty there
What's invalidated when an object database is "dirty" is not the whole
database, but rather a specific list of packs. If there is a race
between getting the pack list and setting the volatile dirty flag
where the packs are rescanned, we don't need to mark the new pack list
as dirty.
This is a fine point that only really applies if the decision of
whether or not to mark dirty actually requires introspecting the pack
list (say, its timestamps). The general operation of "take whatever
is the current pack list and mark it dirty" may still be inherently
racy, but the cost is not so high.
This variable has been populated and never used since it was
introduced in commit 5cf53fdacf28d5cabe7ad1ed154fe7f4971225a9
(Speed up clone/fetch with large number of refs, 2013-02-18).
Noted by FindBugs:
"BatchRefUpdate.java:359, UC_USELESS_OBJECT, Priority: Normal"
Change-Id: I7aacb49540aaee4a83db3d38b15633bb6c4773d0 Signed-off-by: Dan Wang <dwwang@google.com>
Matthias Sohn [Thu, 14 Jul 2016 20:59:14 +0000 (22:59 +0200)]
Fix AppServer build errors in Eclipse with <4.6 target platforms
9aa3748 added dummy implementations for loadRoleInfo() and
loadUserInfo() to class MappedLoginService to fix compile errors in
Eclipse when using 4.6 target platform which brings Jetty 9.3 adding
these two methods. Unfortunately this causes errors when using non 4.6
target platform coming with an older Jetty version. Fix this by
extracting the anonymous subclass of MappedLoginService which allows to
suppress the unused private method errors in Eclipse.
Change-Id: I75baeea7ff4502ce9ef2b541b3c0555da5535d79 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Dave Borowitz [Thu, 14 Jul 2016 15:40:55 +0000 (11:40 -0400)]
Invalidate DfsObjDatabase pack list when refs are updated
Currently, there is a race where a user of a DfsRepository in a single
thread may get unexpected MissingObjectExceptions trying to look up an
object that appears as the current value of a ref:
1. Thread A scans packs before scanning refs, for example by reading
an object by SHA-1.
2. Thread B flushes an object and updates a ref to point to that
object.
3. Thread A looks up the ref updated in (2). Since it is scanning refs
for the first time, it sees the new object SHA-1.
4. Thread A tries to read the object it found in (3), using the cached
pack list it got from (1). The object appears missing.
Allow implementations to work around this by marking the object
database's current pack list as "dirty." A dirty pack list means that
DfsReader will rescan packs and try again if a requested object is
missing. Implementations should mark objects as dirty any time the ref
database reads or scans refs that might be newer than a previously
cached pack list.
Matthias Sohn [Tue, 12 Jul 2016 15:12:41 +0000 (17:12 +0200)]
Merge branch 'stable-4.4'
* stable-4.4:
Log if Repository.useCnt becomes negative
Time based eviction strategy for repository cache
Add method to read time unit from config
Align include.path max depth with native git
Config load should not fail on unsupported or nonexistent include path
Allow using JDK 7 bootclasspath when compiling JGit using Java 8
Extract work queue to allow reusing it
Change-Id: I6aeedb1cb8b0c3068af344a719c80a03ae68fc23 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When Repository.close() decrements the useCount to 0 currently the cache
immediately evicts the repository from WindowCache and RepositoryCache.
This leads to I/O overhead on busy repositories because pack files and
references are inserted and deleted from the cache frequently.
This commit defers the eviction of a repository from the caches until
last use of the repository is older than time to live. The eviction is
handled by a background task running periodically.
Add two new configuration parameters:
* core.repositoryCacheExpireAfter: cache entries are evicted if the
cache entry wasn't accessed longer than this time in milliseconds
* core.repositoryCacheCleanupDelay: defines the interval in milliseconds
for running a background task evicting expired cache entries. If set to
-1 the delay is set to min(repositoryCacheExpireAfter, 10 minutes). If
set to 0 the time based eviction is switched off and no background task
is started. If time based eviction is switched off the JVM can still
evict cache entries if heap memory is running low.
Change-Id: I4a0214ad8b4a193985dda6a0ade63b70bdb948d7 Also-by: Matthias Sohn <matthias.sohn@sap.com> Also-by: Hugo Arès <hugo.ares@ericsson.com> Also-by: Sasa Zivkov <sasa.zivkov@sap.com>
This functionality is implemented in Gerrit ConfigUtil class. Add it to
JGit so it can eventually be remove from Gerrit.
Change-Id: I2d6564ff656b6ab9424a9360624061c94fd5f413 Signed-off-by: Hugo Arès <hugo.ares@ericsson.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sun, 10 Jul 2016 22:27:56 +0000 (00:27 +0200)]
Implement new abstract MappedLoginService methods added in Jetty 9.3
Eclipse Neon comes with Jetty 9.3 which is causing unimplemented
abstract method errors in test class AppServer when using the JGit or
EGit Neon target platform. Fix this by adding dummy implementations.
Change-Id: Ie49107d814a846997de95f149e91fe1ec2fbe4d8 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Jonathan Nieder [Thu, 7 Jul 2016 23:08:02 +0000 (16:08 -0700)]
DfsGarbageCollector: avoid closing idx and bitmap streams twice
These try-with-resources blocks close the underlying output stream
twice: once when closing the CountingOutputStream wrapper, then again
when closing the DfsOutputStream out.
Simplify by only closing the CountingOutputStream.
In practice this shouldn't matter because the close() method of a
Closable is required to be idempotent, but avoiding the redundant
extra close makes the code simpler to read and understand.
ReceivePack: report protocol parsing failures on channel 3
If the client sent a well-formed enough request to see it wants to use
side-band-64k for status reporting (meaning its a modern client), but
any other command record was somehow invalid (e.g. corrupt SHA-1)
report the parsing exception using channel 3. This allows clients to
see the failure and know the server will not be continuing.
git-core and JGit clients send all commands and then start a sideband
demux before sending the pack. By consuming all commands first we get
the client into a state where it can see and respond to the channel 3
server failure.
This behavior is useful on HTTPS connections when the client is buggy
and sent a corrupt command, but still managed to request side-band-64k
in the first line.
ReceivePack: catch InvalidObjectIdException while parsing shallow
The "shallow $id" parsing can also throw InvalidObjectIdException,
just like parseCommand. Move it into its own method with a proper
try-catch block to convert to the checked PackProtocolException.
ReceivePack: enable capabilities immediately on first line
Instead of deferring until after command parsing, enable the
capabilities after the first pkt-line has been read from the client.
This allows the server to setup the side-band-64k channel immediately.
push: Report fatal server errors during pack writing
If the push client has requested side-band support the server can
signal a fatal error parsing the pack using the error channel (3)
and then hang up. This may cause the PackWriter to fail to write to
data onto the network socket, which throws a misleading error back
up to the application and the user.
During a write failure poll the input to see if the side band system
can parse out an error message off channel 3. This should be fast as
there will either be an error present in the buffer, or the remote will
also have hung-up on the side band channel. In the case of a hang-up
just rethrow the original IOException as its a network error.
This roughly matches what C git does; once commands are sent and the
packer is started a new thread runs in the background to decode any
possible server error during unpacking on the remote peer
ReceivePack: Catch InvalidObjectIdException instead of IAE
The more specific type InvalidObjectIdException is thrown by
ObjectId.fromString(). Use it here in ReceivePack as the more
generic IAE is never thrown by the body of the try-catch block.
A RefAdvertiser writing to the network includes both the reference's
ObjectId and its peeled ObjectId in the advertised set. In smart HTTP
negotiation requests may bypass the RefAdvertiser and quickly build
the set based on current refs; include the peeled ObjectIds to match
behavior with the normal bidirectional protocols on git:// and SSH.
This field was being set twice within the block. Setting it just once
is sufficient. writeString() does not examine the field so it is fine
to set it after the call.
Shawn Pearce [Mon, 27 Jun 2016 15:52:00 +0000 (11:52 -0400)]
Merge changes from topic 'dfs-gc'
* changes:
Prune UNREACHABLE_GARBAGE packs when they expire
Use try-with-resources in DfsGarbageCollector.writePack
Fix lastModified to be consistent in DfsGarbageCollector
Add GC_REST PackSource to better order DFS packs
Mike Williams [Fri, 17 Jun 2016 15:34:36 +0000 (11:34 -0400)]
Prune UNREACHABLE_GARBAGE packs when they expire
DfsGarbageCollector will now enforce a maximum time to live (TTL) for
UNREACHABLE_GARBAGE packs. The default TTL is 1 day, which should be
enough time to avoid races with other processes that are inserting
data into the repository.
Change-Id: Id719e6e2a03cfc9a0c0aef8ed71d261dda14bd0c Signed-off-by: Mike Williams <miwilliams@google.com>
Hugo Arès [Wed, 15 Jun 2016 18:40:05 +0000 (14:40 -0400)]
Config load should not fail on unsupported or nonexistent include path
1f86350 added initial support for include.path. Relative path and path
with tilde are not yet supported but config load was failing if one of
those 2 unsupported options was encountered. Another problem was that
config load was failing if the include.path file did not exist.
Change the behavior to be consistent with native git. Ignore unsupported
or nonexistent include.path.
Bug: 495505
Bug: 496732
Change-Id: I7285d0e7abb6389ba6983e9c46021bea4344af68 Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Shawn Pearce [Sun, 26 Jun 2016 18:18:59 +0000 (11:18 -0700)]
Fix lastModified to be consistent in DfsGarbageCollector
Set all packs written by the DfsGarbageCollector to use the same
starting timestamp as lastModified. This makes it easier to see
which packs came from the same DfsGarbageCollector run, as they
share the same timestamp.
This has provided decent performance for object lookups. Starting
from an arbitrary reference may find the content in a newer pack
created by DfsObjectInserter or a ReceivePack server. Compaction of
recent packs also contains newer content, and then most interesting
data is in the "main" GC pack. As the GC pack is self-contained (has
no edges leading outside) readers typically do not need to go further.
Adding a new GC_REST PackSource allows the DfsGarbageCollector to
identify to the pack ordering code which pack is which, so the
non-heads are scanned second during reads. This removes a hack that
was unique to Google's implementation that enforced this behavior by
fixing up the lastModified timestamp.
Renumber the PackSource's categories to reflect this search ordering.
Fix TreeWalk to reset attributes cache for each entry
Treewalk has a member 'attr' which caches the attributes for the current
entry. We did not reset the cache always when moving to next entry. The
effect was that when there are no attributes for an entry 'a' but 'a'
was skipped by a Treewalk filter then Treewalk stopped looking for
attributes until TreeWalk.next() was called again.
Fix DirCacheCheckout to return CheckoutConflictException
Problem occurs when the checkout wants to create a file 'd/f' but
the workingtree contains a dirty file 'd'. In order to create d/f the
file 'd' would have to be deleted and since the file is dirty that
content would be lost. This should lead to a CheckoutConflictException
for d/f when failOnConflict was set to true.
This fix also changes jgit checkout semantics to be more like native
gits checkout semantics. If during a checkout jgit wants to delete a
folder but finds that the working tree contains a dirty file at this
path then JGit will now throw an exception instead of silently keeping
the dirty file. Like in this example:
git init
touch b
git add b
git commit -m addB
mkdir a
touch a/c
git add a/c
git commit -m addAC
rm -fr a
touch a
git checkout HEAD~
Matthias Sohn [Sun, 19 Jun 2016 10:45:37 +0000 (12:45 +0200)]
Allow using JDK 7 bootclasspath when compiling JGit using Java 8
When compiling jgit using Java 8 set system property JDK_HOME to
JAVA_HOME path of JDK7 installation to compile against JDK 7 class
libraries. Otherwise jgit may hit runtime exceptions when running on
Java 7 (e.g. return type of ConcurrentHashMap.keySet() in JDK 8 class
library doesn't exist in JDK 7).
Shawn Pearce [Sun, 19 Jun 2016 04:40:35 +0000 (21:40 -0700)]
Optimize RefAdvertiser for wire protocol
The native wire protocol sends ref advertisements in the pkt-line
format, which requires encoding the ObjectId and ref name onto a byte
sequence. Busy servers show this is a very high source of garbage,
which pushes the garbage collector harder when there are many refs in
the repository (e.g. 70k, in a Gerrit managed repository).
Optimize the side band advertiser by retaining the CharsetEncoder,
minimizing the amount of temporary garbage built during encoding.
Marco Miller [Wed, 25 May 2016 20:45:13 +0000 (16:45 -0400)]
RefDirectory: remove ref lock file for following ref dir removal
Before this fix, ref directory removal did not work. That was because
the ref lock file was still in the leaf directory at deletion time.
Hence no deep ref directories were ever deleted, which negatively
impacted performance under large directory structure circumstances.
This fix removes the ref lock file before attempting to delete the ref
directory (which includes it). The other deep parent directories are
therefore now successfully deleted in turn, since leaf's content
(lock file) gets removed first.
So, given a structure such as refs/any/directory[/**], this fix now
deletes all empty directories up to -and including- 'directory'. The
'any' directory (e.g.) does not get deleted even if empty, as before.
The ref lock file is still also removed in the calling block's finally
clause, just in case, as before. Such double-unlock brought by this
fix is harmless (a no-op).
A new (private) RefDirectory#delete method is introduced to support
this #pack-specific case; other RefDirectory#delete callers remain
untouched.
Change-Id: I47ba1eeb9bcf0cb93d2ed105d84fea2dac756a5a Signed-off-by: Marco Miller <marco.miller@ericsson.com>
http transport does not use authentication fallback
Git servers supporting HTTP transport can send multiple WWW-Authenticate
challenges [1] for different authentication schemes the server supports.
If authentication fails now retry all authentication types proposed by
the server.
[1] https://tools.ietf.org/html/rfc2617#page-3
Bug: 492057
Change-Id: I01d438a5896f9b1008bd6b751ad9c7cbf780af1a Signed-off-by: Christian Pontesegger <christian.pontesegger@web.de> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn Pearce [Thu, 2 Jun 2016 04:48:21 +0000 (21:48 -0700)]
DfsBlock: throw DataFormatException on 0 bytes
setInput should always push at least 1 byte into the Inflater. If 0
bytes (or negative!) are being sent the DfsBlock is inconsistent with
the position passed in. This indicates a severe programming problem
in the caller, and may cause an infinite loop in DfsReader.
Today we saw a handful of live examples of this but don't know what
the cause is. Guard against this error condition and throw with a
more verbose failure, which may prevent an infinite loop. Callers
will eventually catch DataFormatException and rethrow with more detail
about the object that cannot be inflated, with the DFE in the chain.
When using a DfsInserter for high-throughput insertion of many
objects (analogous to git-fast-import), we don't necessarily want to
do a random object lookup for each. It'll be faster from the
inserter's perspective to insert the duplicate objects and let a later
GC handle the deduplication.
Matthias Sohn [Wed, 1 Jun 2016 14:09:40 +0000 (16:09 +0200)]
Merge branch 'master' into stable-4.4
* master:
Fix javadoc errors and unused imports introduced by ddd0fe25
RepoCommand: record manifest shallow recommendation in .gitmodules
RepoCommand: record manifest groups as submodule labels
Remove the deprecated TestRepository.getClock() method
Replace use of deprecated method Repository.getRef()
[findBugs] Prevent potential NPE in
FileLfsRepository.getOutputStream()
Better report on client side if push failed due to too large object
[findBugs] Prevent potential NPE in CloneCommand.init()
RepoCommand: remove --record-remote-branches
RepoCommandTest: Improve assertion message for remote branch recording
Change-Id: I4fbce4f84925a933fcc9a48058ed6793f5821b97 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Stefan Beller [Tue, 31 May 2016 22:19:52 +0000 (15:19 -0700)]
RepoCommand: record manifest shallow recommendation in .gitmodules
Git core learned about the submodule.<name>.shallow option in
.gitmodules files, which is a recommendation to clone a submodule
shallow. A repo manifest may record a clone depth recommendation as
an optional field, which contains more information than a binary
shallow/nonshallow recommendation, so any attempted conversion may be
lossy. In practice the clone depth recommendation is either '1' or doesn't
exist, which is the binary behavior we have in Git core.
Change-Id: I51aa9cb6d1d9660dae6ab6d21ad7bae9bc5325e6 Signed-off-by: Stefan Beller <sbeller@google.com>
Stefan Beller [Tue, 31 May 2016 22:18:20 +0000 (15:18 -0700)]
RepoCommand: record manifest groups as submodule labels
Git core learned about attributes in pathspecs:
pathspec: allow querying for attributes
The pathspec mechanism is extended via the new
":(attr:eol=input)pattern/to/match" syntax to filter paths so that it
requires paths to not just match the given pattern but also have the
specified attrs attached for them to be chosen.
We intend to use these pathspec attribute patterns for submodule
grouping, similar to the grouping in repo. So the RepoCommand which
translates repo manifest files into submodules should propagate this
information along. This requires writing information to the
.gitattributes file instead of the .gitmodules file. For now we just
overwrite any existing .gitattributes file and do not care about prior
attributes set. If this becomes an issue we need to figure out how to
correctly amend the grouping information to an existing .gitattributes
file.
Change-Id: I0f55b45786b6b8fc3d5be62d7f6aab9ac00ed60e Signed-off-by: Stefan Beller <sbeller@google.com>
Matthias Sohn [Mon, 23 May 2016 23:12:20 +0000 (01:12 +0200)]
Better report on client side if push failed due to too large object
JGits PushCommand and BasePackPushConnection were throwing a generic
exception when the pushed pack-file was rejected by the server since it
contained too large objects. Teach JGit to better analyze the server's
response to detect this situation and throw a more specific exception.
Detect this situation by parsing the status line sent by the server.
This change only recognizes the response sent by a JGit based server.
All other servers which report such problems in a different way still
lead to a generic TransportExceptions.
Also see https://git.eclipse.org/r/#/c/46348/
Change-Id: I8d6d65e4585ebb3846f7207e7d1a2f82fa9cbd86 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 24 May 2016 14:59:15 +0000 (16:59 +0200)]
Merge branch 'master' into stable-4.4
* master:
JGit CLI: allow to call git init with specific directory
Redirect all Show output to outs
Support git config [include] section with absolute path(s)
Added filter for merge and non-merges commits.
[findBugs] Prevent potential NPE in FS_POSIX.readUmask()
[findBugs] Fix calculation of host header in SignerV4
Update Orbit repository to S20160518051658 for Neon RC2
Fix StashApply regarding handling of untracked files
GC should not pack objects only referenced by ORIG_HEAD,...
Make sure to overwrite files when "reset --hard" detects conflicts
Allow setting FileMode to executable when applying patches in
ApplyCommand
Fix config value get to return last instead of 1st just like git
Remove UTF-8 checking duplication in Config lib subclasses
Update Maven plugins
Fix type parameter in javadoc in TestRepository.delete(String ref)
TestRepository: Add delete() method
Make BaseReceivePack.setAtomic public
ReceivePack: Pass atomic setting from client to BatchRefUpdate
Change-Id: I5c9c5b7ccb23fb48b44b3da10b2c5d876d043d24 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Tue, 24 May 2016 14:44:03 +0000 (16:44 +0200)]
Merge branch 'stable-4.3' into stable-4.4
* stable-4.3:
Fix computation of id in WorkingTreeIterator with autocrlf and
smudging
Prepare 4.3.2-SNAPSHOT builds
JGit v4.3.1.201605051710-r
Scan loose ref before packed in case gc about to remove the loose
Fix possible NPEs when reporting transport errors
Fix calling of clean/smudge filters from Checkout,MergeCommands
Fix ApplyCommand when result of patch is an empty file
Change-Id: I829f06699f6670e519d04c927bdba4b82df29199 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log