RepositoryCache has 2 methods to remove a repository from the cache but
they are never called when a repository is closed. Users of the cache
were expected to call one of those 2 methods but how could they have
called them at proper time without having visibility of the repository
usage count.
Ideally, I would have reworked the RepositoryCache to wrap any
repository it opens in a class that would be responsible to unregister
them from the cache when it's really closed, i.e. when usage counter
reaches 0. The problem preventing the wrapping solution is the
RepositoryCache.register method that allows to register an already
opened repository in the cache. Such repositories cannot be wrapped
because callers are still holding a reference on the unwrapped
repository.
Document that RepositoryCache.close method is removing the repository
from the cache as well as closing it and rework
RepositoryCache.unregister method to only remove the repository from the
cache. Use the latter to unregister repository when Repository.doClose
is getting executed.
Change-Id: Ia364816e4da8d7b6cfa72f10758ca31aa8a1f9db
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When repositories are opened using the RepositoryCache, they are kept in
memory and when the repository usage counter reaches 0, the
Repository.close method is called which then calls close method on its
reference and object databases.
The problem is that RefDirectory.close method was a no-op and the
reference database was kept in memory. This problem is only happening
when opening a repository using the RepositoryCache because it never
evicts repositories, it's just calling the close method.
Change-Id: Iacb961de8e8b1f5b37824bf0d1a4caf4c6f1233f
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Repository has a usage counter that is initialized to 1 at
instantiation and this counter is decremented when Repository.close
method is called. There is also a Repository.incrementOpen method that
RepositoryCache uses to increment the usage count when it's returning a
repository that is already opened.
The problem was that RepositoryCache was incrementing the usage count
for repositories that it just opened or registered. The usage count was
2 when it should have been 1.
Incrementing usage count is now only be done for repository that are
served from the cache.
This bug is causing slow memory increase of our Gerrit server until the
server become slow. Even if the RepositoryCache is using SoftReference,
it seems that the JVM is not garbage collecting the repositories because
it's not yet on the edge of being out of memory.
To test this change, I replicated all repositories(11k) from Gerrit
master to one slave. The Gerrit master used memory after this test was
10GB without this change and 3.5GB with.
Change-Id: I86c7b36174e384f106b51fe92f306018fd1dbdf0
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Enable calling of smudge filters when checking out paths
When checking out commits/branches JGit was triggering correctly
configured smudge filters. But when checking out paths (either from
index or from commits) JGit was not triggering smudge filters. Fix
CheckoutCommand to properly call filters.
Bug: 486560
Also-by: Pascal Krause <pascal.krausek@sap.com>
Change-Id: I5ff893054defe57ab12e201d901fe74e1376efea
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Implement the DIR_NO_GITLINKS setting with the same functionality
it provides in cGit.
Bug: 436200
Change-Id: I8304e42df2d7e8d7925f515805e075a92ff6ce28
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>
Add config parameter gc.prunePackExpire for packfile expiration
JGit's Garbage Collector is repacking relevant objects into new
packfiles and is afterwards deleting the now obsolete packfiles. But to
prevent problems caused by race conditions JGit was not deleting
packfiles when they are too young. The same mechanism as for loose
objects and the config parameter gc.pruneExpire was used.
But JGit was reusing the parameter gc.pruneExpire also for packfiles
which may cause a lot of filesystem consumption if gc.pruneExpire was
set to the default of 2 weeks. Only two weeks after packfile creation gc
was allowed to delete this packfile.
This change introduces a new config paramter gc.prunePackExpire with a
default of "1.hour". This parameter is used when packfiles are deleted.
Only packfiles older than the specified time can be deleted.
For loose objects the behaviour is not changed and only the old
parameter gc.pruneExpire is relevant.
Change-Id: I6209efb05678b15153bd22479dc13486907a44f8
This change replaced some tabs with spaces introduced in
change I8b3765713599e34f1411f9bbc7f575ec7c2384e0.
Change-Id: Ia5c23b38c9fbbb46f150e527347b61c64c8d9e87
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
With ignoreRemoteFailures set to true, we can ignore remote failures
(e.g. the branch of a project described in the manifest file does not
exist), skip that project and continue to the next one, instead of fail
the whole operation.
Change-Id: I8b3765713599e34f1411f9bbc7f575ec7c2384e0
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
This commit introduces a FileModeStrategy to
the FileTreeIterator class. This provides a way to
allow different modes of traversing a file tree;
for example, to control whether or not a nested
.git directory should be treated as a gitlink.
Bug: 436200
Change-Id: Ibf85defee28cdeec1e1463e596d0dcd03090dddd
Signed-off-by: Preben Ingvaldsen <preben@puppetlabs.com>
Change visibility of DirCacheTree#getObjectId to public
Allow access to the ObjectId of a DirCacheTree if known for low level
integration code.
Change-Id: I6f05b10c9ac781f5e8b38af4a19e653313c91fa8
Signed-off-by: Philipp Marx <smigfu@googlemail.com>
TreeWalk provides the new method getEolStreamType. This new method can
be used with EolStreamTypeUtil in order to create a wrapped InputStream
or OutputStream when reading / writing files. The implementation
implements support for the git configuration options core.crlf, core.eol
and the .gitattributes "text", "eol" and "binary"
CQ: 10896
Bug: 486563
Change-Id: Ie4f6367afc2a6aec1de56faf95120fff0339a358
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The reason is that URIish(URL) and URIish(String) make different parsing
of path / rawPath with regard to drive letters. /C:/... for URL and
C:/... for String. This patch fixes the issue.
Change-Id: I8e2013fff30b7bb198ff733c038e21366667b8a0
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Introduce specific WantNotValidException for servers
Capture the internal "want X not valid" state as a specific subclass
of PackProtocolException, allowing this to be more easily identified
in server stack traces and wrapper application code.
Change-Id: I4b1adb7497f396432da420b0f600ad25a261f912
smart HTTP server: Pass along "want X not valid" to client
If the client sends a SHA-1 that the server does not recognize echo
this back to the client with an explicit error message instead of
the generic "internal server error".
This was always the intent of the implementation but it was being
dropped on smart HTTP due to the UploadPackServlet catching the
PackProtocolException, discarding the buffered message UploadPack
meant to send, and sending along a generic message instead.
Change-Id: I8d96b064ec655aef64ac2ef3e01853625af32cd1
InMemoryRepository: Remove unused RevWalk from batch method signature
The RevWalk given in the arguments is not used. According to the
comment at the top of the method, a new RevWalk is intentionally
used in the implementation.
Remove the unused argument.
Change-Id: Iec81a1341d5bf377801475845b96a465753096ef
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Attributes MacroExpander implements macros used in git attributes. This
is implemented inside the TreeWalk using a lazy created MacroExpander.
In addition, the macro expander caches the global and info attributes
node in order to provide fast merge of attributes.
Change-Id: I2e69c9fc84e9d7fb8df0a05817d688fc456d8f00
Signed-off-by: Ivan Motsch <ivan.motsch@bsiag.com>
Method parameter names were hiding class members of the same
name.
Change-Id: I182f2715894ac4259b09a371cb4e0eb24f52518a
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Allow to reuse disableSslVerify method, move it to HttpSupport
The disableSslVerify method will be used in the follow up change.
Change-Id: Ie00b5e14244a9a036cbdef94768007f1c25aa8d3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
The Large File Storage extension specified by GitHub [1] uses SHA-256 to
compute the ID of large files stored by the extension. Hence implement a
SHA-256 abstraction similar to the SHA-1 abstraction used by JGit.
[1] https://git-lfs.github.com/
Bug: 470333
Change-Id: I3a95954543c8570d73929e55f4a884b55dbf1b7a
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Since If13f7b406, submodule difference are shown as a hunk. The issue
was that added and removed submodule were considered as Edit.REPLACE
instead of Edit.INSERT and Edit.DELETE in the DiffFormatter result.
Change-Id: I4330c2aa3f10e29d7d6b0b2e5286e59293a06239
Signed-off-by: Hugo Arès <hugo.ares@ericsson.com>
Repo manifest file allows revision attribute in <remote> tag. This change
teaches JGit to read that information.
Change-Id: I1c878a2505b9d09fa09fbd404a119b71f2fb8fdb
Signed-off-by: Yuxuan 'fishy' Wang <fishywang@google.com>
Monitoring progress of merges can be useful for users for large
repositories or complex merge processes that take some time.
This enables setting a monitor. Existing merge implementations in jgit
do not yet report progress if a monitor is set. This will be added in a
later change.
Change-Id: I17b978b3fc91750dd88649638b90a46820a0877c
Signed-off-by: Laurent Delaigue <laurent.delaigue@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When reading a bundle file, commit messages who's oneline format is
longer than 982 characters caused JGit to treat subsequent text in
the commit as a SHA, then throw a TransportException because it's
not a valid SHA.
Now the readLine method will read all the way to the end of the
line, not just the first 1024 characters of it.
Change-Id: If15b491aa9a1e4fd9b8bbed2dd9e6be47a64ccb7
Signed-off-by: Chris Gavin <chris@chrisgavin.me>
Monitoring progress of merges can be useful for users for large
repositories or complex merge processes that take some time.
This enables setting a monitor. Existing merge implementations in jgit
do not yet report progress if a monitor is set. This will be added in a
later change.
Change-Id: I17b978b3fc91750dd88649638b90a46820a0877c
Signed-off-by: Laurent Delaigue <laurent.delaigue@obeo.fr>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
When reading a bundle file, commit messages who's oneline format is
longer than 982 characters caused JGit to treat subsequent text in
the commit as a SHA, then throw a TransportException because it's
not a valid SHA.
Now the readLine method will read all the way to the end of the
line, not just the first 1024 characters of it.
Change-Id: If15b491aa9a1e4fd9b8bbed2dd9e6be47a64ccb7
Signed-off-by: Chris Gavin <chris@chrisgavin.me>
TreeWalk: Remove CorruptObjectException from addTree(AbstractTreeIterator)
This form of addTree() does not parse any objects and cannot throw the
declared checked exception. Callers are being forced to try-catch
CorruptObjectException that cannot occur when the iterator instance
has already been constructed.
Change-Id: Id338035302903bab81569d1576eab063eee0885a
ResetCommand was not returning the updated ref as a result of the call()
method. Since the ResetCommand is always updating the same ref (HEAD)
this should always be the HEAD ref.
Bug: 440750
Change-Id: I7974975c3ab05e68c208384e69cf0692ded6e8db
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add option to allow empty commits to CommitCommand
CommitCommand should allow to specify whether empty commits (commits
having the same tree as the sole predecessor commit) are allowed or not.
Similar to native git's "--allow-empty" flag.
The defaults differ between JGit and native git even after this change.
When not specifying paths then by default JGit allows to create empty
commits while native git does not. It would be API breaking to change
this now.
Bug: 460301
Change-Id: I88feb0c3ffb2c686b1d0594e669729b065cda4cb
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Experimental flag to turn on the KetchLeader within this daemon JVM.
This is a manually elected leader process, set from the command line.
Remote followers for each repository are configured per-repository
using remote sections with ketch-type = FULL. For example:
Manually elected leader's $GIT_DIR/config:
[ketch]
name = A
[remote "A"]
ketch-type = FULL
[remote "B"]
url = git://127.0.0.1:9421/sample.git
ketch-type = FULL
[remote "C"]
url = git://127.0.0.1:9422/sample.git
ketch-type = FULL
Replica B and C daemons:
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9421 \
--base-path=$HOME/ketch_test/follower_one \
$HOME/ketch_test/follower_one &
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9422 \
--base-path=$HOME/ketch_test/follower_two \
$HOME/ketch_test/follower_two &
Change-Id: I165f85970a77e16b5263115290d685d8a00566f5
Git Ketch is a multi-master Git repository management system. Writes
are successful only if a majority of participant servers agree. Acked
writes are durable against server failures as a majority of the
participants store all required objects.
Git Ketch is modeled on the Raft Consensus Algorithm[1]. A ketch
sailing vessel is faster and more nimble than a raft. It can also
carry more source codes.
Git Ketch front-loads replication costs, which vaguely resembles a
ketch sailing vessel's distinguishing feature of the main mast on the
front of the ship.
[1] https://raft.github.io/
Change-Id: Ib378dab068961fc7de624cd96030266660b64fb4
This could have only happened during the getBytes call. Instead, use
Constants.encode, which is a non-throwing implementation.
This change is binary compatible with existing code compiled against
older versions of JGit, although it might break compilation of
previously compiling code due to dead catch blocks.
Change-Id: I191fec5cac718657407230de141440e86d0151fb
After creating a Transport instance callers should always call
its close() method. Use AutoCloseable to document this idiom
and allow use of try-with-resources.
Change-Id: I0c6ff3e39ebecdd7a028dbcae1856a818937b186
ReceiveCommand.abort(): Utility to mark batch of commands as failed
If one or more commands is failing the entire group usually has to
also fail with "transaction aborted". Pull this loop into a helper
so the idiom can be easily reused in several places throughout JGit.
Change-Id: I3b9399b7e26ce2b0dc5f7baa85d585a433b4eaed
RefTreeDatabase: Allow ORIG_HEAD, etc. on non-bare repositories
Store these in the bootstrap layer where they are using $GIT_DIR
as the storage directory for any reference that does not contain '/'.
Change-Id: I5595bf514e4475b7c7e799c2c79446597a3abb4a
RefTreeDatabase: Expose bootstrap refs in getAdditionalRefs
By showing the bootstrap layer in getAdditionalRefs() garbage
collector code can be more RefDatabase agnostic and not care about
the special case of RefTree and RefTreeNames for the purposes of
building up the roots to GC. Instead they can combine getRefs(ALL)
and getAdditionalRefs() and have a clean set of roots.
Change-Id: I665cd2456e9316640215b6a08bc728d1356f36d8
Hugo Arès
Christian Halstrick
Preben Ingvaldsen
Jonathan Nieder
Yuxuan 'fishy' Wang
Philipp Marx
Ivan Motsch
XinTong Wang
Shawn Pearce
David Pursehouse
Saša Živkov
Matthias Sohn
Mike Williams
Rüdiger Herrmann
Thomas Wolf
Terry Parker
Laurent Delaigue
Chris Gavin
Dave Borowitz
Andrey Loskutov