Shawn Pearce [Sun, 13 Nov 2016 19:52:12 +0000 (11:52 -0800)]
Extract insecure Cipher factory
Bazel runs ErrorProne by default and ErrorProne rightly complains that
allowing the user to specify any Cipher can lead to insecure code
(in particular, getCipher("AES") operates in ECB mode). Unfortunately
this is required to support existing repositories insecurely stored
on S3.
Extract the insecure factory code to its own class so this can be built
as a java_library() with this check disabled.
Jonathan Nieder [Sun, 13 Nov 2016 21:16:48 +0000 (13:16 -0800)]
StreamCopyThread: Remove unnecessary flushCount
StreamCopyThread#run consistently interrupts itself whenever it
discovers it has been interrupted by StreamCopyThread#flush while not
reading. The flushCount is not needed to avoid lost flushes.
All in-tree users of StreamCopyThread never flush. As a nice side
benefit, this avoids the expense of atomic operations that have no
purpose for those users.
Hugo Arès [Wed, 9 Nov 2016 13:49:41 +0000 (08:49 -0500)]
Get rid of SoftReference in RepositoryCache
Now that RepositoryCache have a time based eviction strategy, get rid
of the strategy to evict cache entries if heap memory is running low,
i.e. soft references. Main reason why time based eviction was
implemented was to offer an alternative to the unpredictable soft
references.
Relying on soft references is not working, especially in large heap. The
JVM GC will consider collecting soft references as last resort before
throwing an out of memory error. For example, an application like Gerrit
configured with a 128GB heap, GC will wait until all 128GB is filled
before collecting the soft references so the application will be
suffering long pauses caused by GC for a long time already. In other
words, you will have to restart application because it's unusable before
JVM eviction kicks in.
Keeping the SoftReference in RepositoryCache is causing more harm than
good. If you use the time based eviction (which is the default strategy)
and want to tune JVM to release soft references more aggressively, it
will release repositories from the cache even though they are not
expired which defeats the purpose of the repository cache.
Gerrit uses Lucene library which uses soft references and this is
causing a "memory leak" except if you configure JVM to release soft
references more aggressively which have the nasty side effect of
evicting non expired repositories from the cache.
Change-Id: I9940bd800464c7f007696d0ccde52ea617b2ebce Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Shawn Pearce [Sun, 13 Nov 2016 01:06:39 +0000 (17:06 -0800)]
Switch JSchSession to simple isolated OutputStream
Work around issues with JSch not handling interrupts by
isolating the JSch interactions onto another thread.
Run write and flush on a single threaded Executor using
simple Callable operations wrapping the method calls,
waiting on the future to determine the outcome before
allowing the caller to continue.
If any operation was interrupted the state of the stream
becomes fuzzy at close time. The implementation tries to
interrupt the pending write or flush, but this is very
likely to corrupt the stream object, so exceptions are
ignored during such a dirty close.
Philipp Marx [Fri, 11 Nov 2016 09:43:09 +0000 (10:43 +0100)]
Check that DfsBlockCache#blockSize is a power of 2
In case a value is used which isn’t a power of 2 there will be a high
chance of java.lang.ArrayIndexOutBoundsException and
org.eclipse.jgit.errors.CorruptObjectException due to a mismatching
assumption for the DfsBlockCache#blockSizeShift parameter.
Change-Id: Ib348b3704edf10b5f93a3ffab4fa6f09cbbae231 Signed-off-by: Philipp Marx <smigfu@googlemail.com>
Matthias Sohn [Mon, 7 Nov 2016 21:31:10 +0000 (22:31 +0100)]
Fix loop in auto gc
* GC.tooManyLooseObjects() always responded true since the loop missed
to advance the iterator so it always incremented until the threshold was
exceeded.
* Also fix loop exit criterion which was off by 1.
* Add some tests.
Change-Id: I70976dfaa026efbcf3c46bd45941f37277a18e04 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Jonathan Nieder [Fri, 4 Nov 2016 22:59:32 +0000 (15:59 -0700)]
StreamCopyThread: Do not drop data when flush is observed before writing
StreamCopyThread.flush was introduced in 61645b938bc934fda3b0624c5bac1e3495634750 (Add timeouts to smart
transport clients, 2009-06-19) to support timeouts on write in JSch.
The commit message from that change explains:
JSch made a timeout on write difficult because they explicitly do
a catch for InterruptedException inside of their OutputStream. We
have to work around that by creating an additional thread that just
shuttles data between our own OutputStream and the real JSch stream.
The code that runs on that thread is structured as follows:
while (!done) {
int n = src.read(buf);
dst.write(buf, 0, n);
}
with src being a PipedInputStream representing the data to be written
to JSch. To add flush support, that change wanted to add an extra step
if (wantFlush)
dst.flush();
but to handle the case where the thread is blocked in the read() call
waiting for new input, it needs to interrupt the read. So that is how
it works: the caller runs
to write some data and force it to flush by interrupting the read.
After the pipeOut.flush(), the StreamCopyThread reads the data that was
written and prepares to copy it out. If the copyThread.flush() call
interrupts the copyThread before it acquires writeLock and starts
writing, we throw away the data we just read to fulfill the flush.
Oops.
Noticed during the review of e67d59df3fb8ae1efe94a54efce36784f7cc83e8
(StreamCopyThread: Do not let flush interrupt a write, 2016-11-04),
which introduced this bug.
Jonathan Nieder [Fri, 4 Nov 2016 18:48:38 +0000 (11:48 -0700)]
StreamCopyThread: Do not let flush interrupt a write
flush calls interrupt() to interrupt a pending read and trigger a
flush. Unfortunately that interrupt() call can also interrupt a
pending write, putting Jsch in a bad state and triggering "Short read
of block" errors. Add locking to ensure the flush only interrupts
reads as intended.
Zhen Chen [Mon, 31 Oct 2016 21:27:36 +0000 (14:27 -0700)]
Fix flush call race condition in StreamCopyThread
If there was a new flush() call during flush previous bytes, we need to
catch it in order to process the new bytes between the two flush()
calls instead of going to last catch IOException clause and end the
thread.
Matthias Sohn [Mon, 24 Oct 2016 11:26:58 +0000 (13:26 +0200)]
Speedup CleanFilter by transferring data in chunks of 8k
Transferring data byte per byte is slow, running add with CleanFilter on
a 2.9MB file takes 20 seconds. Using a buffer of 8k shrinks this time to
70ms.
Change-Id: I3bc2d8c11fe6cfaffcc99dc2a00643e01ac4e9cc Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Kevin Corcoran [Wed, 13 Apr 2016 22:55:08 +0000 (15:55 -0700)]
Make streamFileThreshold configurable
Previously, the streamFileThreshold, the threshold at which a file
would be streamed rather than loaded entirely into memory, was only
configurable on a global basis.
This commit makes this threshold configurable on a per-loader basis.
Bug: 490404
Change-Id: I492c18c3155dbf56eedda9044a61d76120fd75f9 Signed-off-by: Kevin Corcoran <kevin.corcoran@puppetlabs.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Matthias Sohn [Wed, 13 Apr 2016 12:54:09 +0000 (14:54 +0200)]
Implement auto gc
With the auto option, gc checks whether any housekeeping is required; if
not, it exits without performing any work. Some JGit commands run gc
--auto after performing operations that could create many loose objects.
Housekeeping is required if there are too many loose objects or too many
packs in the repository.
If the number of loose objects exceeds the value of the gc.auto option
jgit's GC consolidates all existing packs into a single pack (equivalent
to -A option), whereas git-core would combine all loose objects into a
single pack using repack -d -l. Setting the value of gc.auto to 0
disables automatic packing of loose objects.
If the number of packs exceeds the value of gc.autoPackLimit, then
existing packs (except those marked with a .keep file) are consolidated
into a single pack by using the -A option of repack. Setting
gc.autoPackLimit to 0 disables automatic consolidation of packs.
Like git the following jgit commands run auto gc:
- fetch
- merge
- rebase
- receive-pack
The auto gc for receive-pack can be suppressed by setting the config
option receive.autogc = false
Change-Id: I68a2a051b39ec2c53cb7c4b8f6c596ba65eeba5d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Ned Twigg [Fri, 18 Mar 2016 09:46:38 +0000 (02:46 -0700)]
Checkout: Add the ability to checkout all paths.
Change-Id: Ie1e59c566b63d0dfac231e44e7ebd7f3f08f3e9f Signed-off-by: Ned Twigg <ned.twigg@diffplug.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Marc Strapetz [Mon, 22 Aug 2016 07:53:18 +0000 (09:53 +0200)]
Fix possible SIOOBE in RefDirectory.parsePackedRefs
This SIOOBE happens reproducibly when trying to access
a repository containing Cygwin symlinks
Change-Id: I25f103fcc723bac7bfaaeee333a86f11627a92c7 Signed-off-by: Marc Strapetz <marc.strapetz@syntevo.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Remove the assumption that the local repository is a file based one.
Change-Id: I8f10fe7a54e9fc07f2a23d7901e52b65aa570d45 Signed-off-by: Thomas Meyer <thomas.mey@web.de> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Turner [Mon, 26 Sep 2016 20:41:19 +0000 (16:41 -0400)]
TreeFormatter: disallow empty filenames in trees
Git barfs on these (and they don't make any sense), so we certainly
shouldn't write them.
Change-Id: I3faf8554a05f0fd147be2e63fbe55987d3f88099 Signed-off-by: David Turner <dturner@twosigma.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
David Pursehouse [Wed, 19 Oct 2016 03:54:46 +0000 (12:54 +0900)]
ArchiveTest: Don't use string concatenation in loop
According to FindBugs:
In each iteration, the String is converted to a StringBuffer/
StringBuilder, appended to, and converted back to a String. This
can lead to a cost quadratic in the number of iterations, as the
growing string is recopied in each iteration.
Replace string concatenation with StringBuffer.
Change-Id: I60e09f274bed6722f4e0e4d096b0f2b1b31ec1b4 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
David Turner [Fri, 14 Oct 2016 20:44:51 +0000 (16:44 -0400)]
Config: do not add spaces before units
Adding a space before the unit ('g', 'm', 'k) causes git to fail with
the error:
fatal: bad numeric config value
Change-Id: I57f11d3a1cdcca4549858e773af1a2a80fc0369f Signed-off-by: David Turner <dturner@twosigma.com> Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
David Pursehouse [Tue, 18 Oct 2016 04:51:27 +0000 (13:51 +0900)]
FS: Fix lazy initialization of non-volatile static field
The 'factory' field is lazy initialized in the detect() method.
According to FindBugs:
Because the compiler or processor may reorder instructions, threads
are not guaranteed to see a completely initialized object, if the
method can be called by multiple threads.
Fix this by declaring the member as 'volatile'.
Change-Id: Ib32663bb28c9564584256e01f625b4e7875e6223 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>
Fix JGit CLI to follow native git's interpretation of http_proxy...
Native git (as many other tools) interprets the environment variables
http_proxy, HTTP_PROXY, ... in a specific way. "http_proxy" has to be
lowercase while "https_proxy" can be lowercase or uppercase (means:
"HTTPS_PROXY"). Lowercase has precedence. This can be looked up in
"ENVIRONMENT" section of [1]. Teach JGit CLI to behave similar.
Additionally teach JGit not to interpret the environment variables if
the java process was explicitly started with the system properties
telling JVM which proxy to use. A call like "http_proxy=proxy1 java
-Dhttp.proxyHost=proxy2 ..." should use proxy2 as proxy.
Hugo Arès [Wed, 12 Oct 2016 10:54:52 +0000 (06:54 -0400)]
Fix eviction of repositories with negative usage count
If the repository close method was called twice (or more) for one open,
the usage count became negative and the repository was never be evicted
from the cache because the method checking if repository is expired was
not considering negative usage count.
Change-Id: I18a80c415c54c37d1b9def2b311ff2d0afa455ca Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Matthias Sohn [Mon, 10 Oct 2016 22:34:42 +0000 (00:34 +0200)]
Merge branch 'stable-4.5'
* stable-4.5:
pgm: Fix misspelled key of an externalized string
Add missing online help for Ketch server type option in CLI daemon
Remove wrong junit dependencies in org.eclipse.jgit.pgm
Change-Id: I9cac024c0b488101b539c713b0f5ffc8c01b55bd Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Thomas Wolf [Mon, 21 Dec 2015 18:43:56 +0000 (19:43 +0100)]
Fix symlink content comparison on MacOS in tree walk
Symlinks on MacOS are written as UTF-8 NFD, but
readSymbolicLink().toString() converts to NFC with potentially fewer
bytes. May occur in particular if the link target has non-ASCII
characters for which the NFC and NFD encodings differ. This may lead
to an EOFException: Short read of block.
This causes all kinds of weird effects in EGit, ranging from failing
rebases (which report the exception to the user) to EGit decorations in
the navigator silently disappearing (and never coming back).
* Rename readContentAsNormalizedString() to readSymlinkTarget() as it's
called only for symlinks. Also make it protected.
* Fix by allowing the read to succeed even if less than the expected
number of bytes are returned by the entry's input stream.
* Override in FileTreeIterator to use fs.readSymlink() directly.
Includes a new MacOS-only test.
Change-Id: I264c5972d67b1cbb1ed690580f5706e671b9affd Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix CheckoutCommand to return updated files even on NONDELETED status
CheckoutCommand was not returning updated and removed files in case of
an overall status of NONDELETED. That's status which occurs especially
on the Windows platform when Checkout wanted to delete files but the
filesystem doesn't allow this. The situation is more seldom on linux/mac
because open filehandles don't stop a deletion attempt and checkout
succeeds more often.
There was a bug when carrying over flags from a merge commit to its
non-first parents. The first parent of a merge commit was handled
differently and correct but the non-first parents are handled by a
recursive algorithm. Flags should be copied from the root merge commit
to parent-2, to grandparent-2, ... up to the limit of STACK_DEPTH==500
parents-levels. But the recursive algorithm was always copying only to
the direct parents of the merge commit and not the grand*-parents.
This seems to be no problem when commits are handled in a strict date
order because then copying only one level is no problem if children are
handled before parents. But when commits are not seperated anymore by
distinctive correct dates (e.g. because all commits have the same date)
then it may happen that a merge-parent is handled before the merge
commit and when dealing later with the merge commit one has to copy
flags down to more than one level
Thomas Wolf [Fri, 23 Sep 2016 05:37:21 +0000 (07:37 +0200)]
[releng] maven: compile against the Java 1.8 libraries
Remove the JDK_HOME settings that were introduced to build against the
Java 1.7 libraries. Now that the minimum requirement is Java 1.8, this
is wrong and prevents use of Java 1.8 features.
Change-Id: I91a194c9449d7810ef02b038907dbbc708e600a5 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
David Pursehouse [Thu, 22 Sep 2016 09:30:14 +0000 (11:30 +0200)]
pgm, test: Add missing dependency for buck build
Change-Id: I26d69fb6d35c6fb120360ef143d1b1f565d4014c Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
David Pursehouse [Tue, 20 Sep 2016 19:19:44 +0000 (21:19 +0200)]
HttpClientConnection: Don't use deprecated HttpClient classes
- raise minimum version for HttpClient packages to 4.3 since some of the
used classes aren't available in older versions
- recompute OSGi uses clauses
Change-Id: I8f0bff1433762561e02f7439db27a6a9e846c290 Signed-off-by: David Pursehouse <david.pursehouse@gmail.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Wed, 21 Sep 2016 16:08:00 +0000 (18:08 +0200)]
Merge branch 'stable-4.5'
* stable-4.5:
Turn off doclint also during Maven site generation
Prepare 4.5.1-SNAPSHOT builds
JGit v4.5.0.201609210915-r
Unconditionally close repository in unregisterAndCloseRepository
Change-Id: Ibfd11669cd74d2e62b014c18fd39b646b200c8c5 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Unconditionally close repository in unregisterAndCloseRepository
Repository.close() method is used when reference counting and expiration
needs to be honored. The RepositoryCache.unregisterAndCloseRepository
method should close the repository unconditionally. This is also indicated
from its javadoc.
The class AtomicObjectOutputStream should be available to all lfs
related classes, not only to the server side. Move the class from
org.eclipse.jgit.lfs.server.fs to org.eclipse.jgit.lfs.internal to
achieve that.
Adds a JGit built-in implementation of the "git lfs smudge" filter. This
filter should do the same as the one described in [1] besides that it
only supports the local case when the lfs objects are already present in
the media directory. Remote cases where download of LFS objects from an
LFS server is needed will be done in a later commit.
Adds a JGit built-in implementation of the "git lfs clean" filter. This
filter should do the same as the one described in [1]. But since this
filter is written in Java and can be called by JGit without forking new
processes it should be much faster
Add configuration parameter to enable built-in hooks/filters
If the configuration parameter filter.<filterDriverName>.useJGitBuiltin
is set to true then for all corresponding filters JGit will try to
execute the built-in filter instead of the filter-command which is
defined in git configuration. It will fallback to the non-built-in
filters if no built-in filters are registered or if constructing them
leads to exceptions. If set to false JGit will not try to execute
built-in filters for the specified filter driver.
Example: The configuration contains the following lines
Addtionally the .gitattributes file in the root of the working tree
contains:
*.bin filter=lfs
In this case when new content is added similar to "git add 1.bin" then
the following will happen:
- jgit will check whether a built-in command factory was registered
for the command "jgit://builtin/lfs/clean". If that is true the
factory is used to create a built-in filter command and that
command is used to filter the content
- Otherwise jgit will call the external program "git lfs clean ..."
to do the filtering
JGit supports smudge filters defined in repository configuration. The
filters are implemented as external programs filtering content by
accepting the original content (as seen in git's object database) on
stdin and which emit the filtered content on stdout. This content is
then written to the file in the working tree. To run such a filter JGit
has to start an external process and pump data into/from this process.
This commit adds support for built-in smudge filters which are
implemented in Java and which are executed by jgit's main thread. When a
filter is defined in the configuration as
"jgit://builtin/<filterDriverName>/smudge" then JGit will lookup in a
static map whether a builtin filter is registered under this name. If
found such a filter is called to do the filtering.
The functionality in this commit requires that a program using JGit
explicitly calls the JGit API to register built-in implementations for
specific smudge filters. In follow-up commits configuration parameters
will be added which trigger such registrations.
JGit supports clean filters defined in repository configuration. The
filters are implemented as external programs filtering content by
accepting the original content (as seen in the working tree) on stdin
and which emit the filtered content on stdout. To run such a filter JGit
has to start an external process and pump data into/from this process.
This commit adds support for clean filters which are implemented
in Java and which are executed by jgit's main thread. When a filter is
defined in the configuration as
"jgit://builtin/<filterDriverName>/clean" then JGit will lookup in a
static map whether a filter is registered under this name. If found
such a filter is called to do the filtering.
The functionality in this commit requires that a program using JGit
explicitly calls the JGit API to register built-in implementations for
specific clean filters. In follow-up commits configuration parameters
will be added which trigger such registrations. Other commits will add
implementations for lfs filters.
Thomas Wolf [Mon, 15 Aug 2016 05:55:44 +0000 (07:55 +0200)]
Handle all values of branch.[name].rebase
BranchConfig treated this config property as a boolean, but git also
allows the values "preserve" and "interactive". Config property
pull.rebase also allows the same values.
Replace private enum PullCommand.PullRebaseMode by new public enum
BranchConfig.BranchRebaseMode and adapt all uses. Add a new setter to
PullCommand.
Note: PullCommand will treat "interactive" like "true", i.e., as a
non-interactive rebase. Not sure how "interactive" should be handled.
At least it won't balk on it.
Bug: 499482
Change-Id: I7309360f5662b2c2efa1bd8ea6f112c63cf064af Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
Martin Goellnitz [Sat, 10 Sep 2016 09:29:30 +0000 (11:29 +0200)]
Add support for post-commit hooks
Change-Id: I6691b454404dd4db3c690ecfc7515de765bc2ef7 Signed-off-by: Martin Goellnitz <m.goellnitz@outlook.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Sun, 4 Sep 2016 10:07:37 +0000 (12:07 +0200)]
Don't log error if system git config does not exist
- enhance FS.readPipe to throw an exception if the external command
fails to enable the caller to handle the command failure
- reduce log level to warning if system git config does not exist
- improve log message
Bug: 476639
Change-Id: I94ae3caec22150dde81f1ea8e1e665df55290d42 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
projects / jgit.git / log