Fix RebuildRefTree trying to add HEAD twice to RefTree
14dfa70520 fixed the problem that HEAD wasn't added to the reftree when
rebuilding the reftree in an empty repository where HEAD isn't yet
resolvable. Since non-resolvable refs are filtered out by
RefDatabase.getRefs(ALL) we have to add HEAD to the reftree explicitly
in this special case.
This fix resulted in another bug: rebuilding the reftree in a repository
which has a resolvable HEAD failed with a DirCacheNameConflictException
in RefTree.apply(). If HEAD is resolvable RefDatabase.getRefs(ALL) does
not filter out HEAD. This results in two identical CREATE commands for
HEAD which RefTree.apply() refuses to execute.
Fix this by no longer creating a duplicate CREATE command for HEAD.
See: I46cbc2611b9ae683ef7319dc46af277925dfaee5
Change-Id: I58dd6bcdef88820aa7de29761d43e2edfa18fcbe
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Fix help text for option --enable of command debug-rebuild-ref-tree
This fixes a MissingResourceException thrown when executing
jgit debug-rebuild-ref-tree --help
Change-Id: I637ea55084a913f5105ebf4cf2baef8b81877938
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Add a storage implementation storing large objects in Amazon S3.
The AmazonS3Repository pre-signs download and upload requests.
AWS access and secret key are expected to be in the
$HOME/.aws/credentials file in the following format:
[default]
accessKey = ...
secretKey = ...
Use AWS version 4 request signing [1] because it is more secure and
supported by all regions. The version 3 signing is not supported in
newer regions.
In follow up changes we should:
- implement getVerifyAction() and do actual verification. Subclasses of
S3Repository can implement caching for object meta data (size) in order
to avoid extra roundtrips to S3. Verification should ensure that meta
data store and content of S3 storage are in sync
- HEAD request used in S3Repository.getSize() seems to always return
Content-length 0 in contrast to the documentation [2]. So getSize() does
detect if the object exists in S3 or not but in case the object exists
it always returns size 0
[1] http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
[2] https://forums.aws.amazon.com/thread.jspa?threadID=223616
Change-Id: Ic47f094928a259e5264c92b3aacf6d90210907a8
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
DiffAlgorithms: Fix warnings about variable hiding
Local variables/parameters named 'db' and 'cmp' were hiding class
member variables of the same name.
Change-Id: I98b770587aaf73744a93e6a3ee33d131a9fa91e9
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
TextHashFunctions: Fix warnings about variable hiding
The local variables/arguments named 'db' were hiding the member
variable of the same name in the superclass TextBuiltin.
Change-Id: Ic49dcb7cc2a1b3fb46ad78450042c8c8fb464a80
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Support LFS protocol and a file system based LFS storage
Implement LfsProtocolServlet handling the "Git LFS v1 Batch API"
protocol [1]. Add a simple file system based LFS content store and the
debug-lfs-store command to simplify testing.
Introduce a LargeFileRepository interface to enable additional storage
implementation while reusing the same protocol implementation.
At the client side we have to configure the lfs.url, specify that
we use the batch API and we don't use authentication:
[lfs]
url = http://host:port/lfs
batch = true
[lfs "http://host:port/lfs"]
access = none
the git-lfs client appends the "objects/batch" to the lfs.url.
Hard code an Authorization header in the FileLfsRepository.getAction
because then git-lfs client will skip asking for credentials. It will
just forward the Authorization header from the response to the
download/upload request.
The FileLfsServlet supports file content storage for "Large File
Storage" (LFS) server as defined by the Github LFS API [2].
- upload and download of large files is probably network bound hence use
an asynchronous servlet for good scalability
- simple object storage in file system with 2 level fan-out
- use LockFile to protect writing large objects against multiple
concurrent uploads of the same object
- to prevent corrupt uploads the uploaded file is rejected if its hash
doesn't match id given in URL
The debug-lfs-store command is used to run the LfsProtocolServlet and,
optionally, the FileLfsServlet which makes it easier to setup a
local test server.
[1]
https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md
[2] https://github.com/github/git-lfs/tree/master/docs/api
Bug: 472961
Change-Id: I7378da5575159d2195138d799704880c5c82d5f3
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
The Branch class has a member named 'branch', which was being
hidden by the local variable of the same name used in a for-loop.
Change-Id: I334092010a9c80686fb79713852d4bfa166ce12f
Signed-off-by: David Pursehouse <david.pursehouse@sonymobile.com>
Experimental flag to turn on the KetchLeader within this daemon JVM.
This is a manually elected leader process, set from the command line.
Remote followers for each repository are configured per-repository
using remote sections with ketch-type = FULL. For example:
Manually elected leader's $GIT_DIR/config:
[ketch]
name = A
[remote "A"]
ketch-type = FULL
[remote "B"]
url = git://127.0.0.1:9421/sample.git
ketch-type = FULL
[remote "C"]
url = git://127.0.0.1:9422/sample.git
ketch-type = FULL
Replica B and C daemons:
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9421 \
--base-path=$HOME/ketch_test/follower_one \
$HOME/ketch_test/follower_one &
git daemon \
--export-all \
--enable=receive-pack \
--listen=127.0.0.1 --port=9422 \
--base-path=$HOME/ketch_test/follower_two \
$HOME/ketch_test/follower_two &
Change-Id: I165f85970a77e16b5263115290d685d8a00566f5
HEAD is not part of getRefs(ALL) and must be copied explicitly.
This allows `jgit debug-rebuild-ref-tree --enable` to convert an
existing repository to use RefTree with a local working tree:
mkdir testRepo
cd testRepo
jgit init
jgit debug-rebuild-ref-tree --enable
touch a
jgit add a
jgit commit -m initial
Change-Id: I46cbc2611b9ae683ef7319dc46af277925dfaee5
FileRepository: Support extensions.refsBackendType = RefTree
This experimental code can be enabled in $GIT_DIR/config:
[core]
repositoryformatversion = 1
[extensions]
refsBackendType = RefTree
When these are set the repository will read references from the
RefTree rooted by the $GIT_DIR/refs/txn/committed reference.
Update debug-rebuild-ref-tree to rebuild refs/txn/committed only from
the bootstrap layer. This avoids misuse by rebuilding using packed-refs
and $GIT_DIR/refs tree.
Change-Id: Icf600e4a36b2f7867822a7ab1f1617d73c710a4b
debug-rebuild-ref-tree: Simple program to build a RefTree
This tool scans all references in the repository and writes out a new
reference pointing to a single commit whose root tree is a RefTree
containing the current refs of this repository.
It alway skips storing the reference it will write to, avoiding the
obvious cycle.
Change-Id: I20b1eeb81c55dc49dd600eac3bf8f90297394113
Make sure CLIGitCommand and Main produce (almost) same results
Currently execution of tests in pgm uses CLIGitCommand which
re-implements few things from Main. Unfortunately this can results in a
different test behavior compared to the real CLI runtime.
The change let CLIGitCommand extend Main and only slightly modifies the
runtime (stream redirection and undesired exit() termination).
Change-Id: I87b7b61d1c84a89e5917610d84409f01be90b70b
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
branch command: print help if requested, even if arguments are wrong
git branch -d -h reports an error (because of missing -d option value)
but does not print the help as expected.
To fix this, CmdLineParser must catch, print but do not propagate
exceptions if help is requested.
Bug: 484951
Change-Id: I51265ebe295f22da540792c6a1980b8bdb295a02
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
In different places (Main, TextBuiltin, CLIGitCommand) we report fatal
errors and at same time want to check for fatal errors in the tests.
Using common API simplifies the error testing and helps to navigate to
the actual error check implementation.
Change-Id: Iecde79beb33ea595171f168f46b0b10ab2f339bb
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Run git describe during the build to determine the lineage of this
working directory and stamp this information into the binary.
Change-Id: I0ad24125c31e4280ccf900bac4065924087b05aa
Compile each test in its own java_test() target so they can run in
parallel, reducing total time spent testing on large machines.
$ buck test --all
[-] PROCESSING BUCK FILES...FINISHED 0.3s [100%]
[-] BUILDING...FINISHED 2.9s [100%] (351/383 JOBS, 351 UPDATED, 0.0% CACHE MISS)
[-] TESTING...FINISHED 98.1s (3360 PASS/15 SKIP/0 FAIL)
Change-Id: I8d6541268315089299f933ed23d785b1b3431133
Construct the java_application JAR wrapped with the shell script
header. This is enough to clone a repository over HTTPs:
$ buck build :jgit_bin
$ buck-out/gen/jgit_bin/jgit_bin clone https://...
Change-Id: I4aceb4e77b2ec9be76a32ec93d94f2dafe9acce6
Sort "eager" path-like options to the end of the help
The "--" path option (and all other similar options consuming all
remaining arguments) should be placed at the end of the command line
help.
Currently jgit reset -h shows this:
jgit reset [commit-ish] [path ... ...] [-- path ... ...] [--hard]
[--help (-h)] [--mixed] [--soft]
After the patch the help shows this:
jgit reset [commit-ish] [path ... ...] [--hard] [--help (-h)] [--mixed]
[--soft] [-- path ... ...]
Bug: 484951
Change-Id: I3db332bf293ca8d6bfaab0d546cd35af689bd46e
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
reset command: provide convenient and meaningful options help
This commit changes the jgit "reset" command line options help from
this:
jgit reset name [VAL ...] [-- path ... ...] [--hard] [--help (-h)]
[--mixed] [--soft]
name : Reset current HEAD to the specified state
[...]
to this:
jgit reset [commit-ish] [path ... ...] [-- path ... ...] [--hard]
[--help (-h)] [--mixed] [--soft]
commit-ish : Reset to given reference name
[...]
Bug: 484951
Change-Id: I614e71101b4f9f46ef8f02379d1a9d135f3292d2
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
status command: consume more then one argument after --
See bug 484951 comment 4: "jgit status -- a b" doesn't work and
complains that "b" is not an allowed argument
Bug: 484951
Change-Id: I86b81e7f2bab6e928bb8e973bd50c8f4b9c6fecf
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
repo command: properly name the required 'path' argument
Fixes point 4 in bug 484951, where "jgit repo" or "jgit repo -h" dumps a
stack trace.
Bug: 484951
Change-Id: Ic8b362e07a40ad923dc9acde0c0983a1e7932a02
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Don't treat command termination due '-h' option as a fatal error
Signal early command termination due '-h' or '--help' option via
TerminatedByHelpException. This allows tests using
CLIGitCommand differentiate between unexpected command parsing errors
and expected command cancellation "on help" (which also allows
validation of expected/unexpected help messages).
Additional side-effect: jgit supports now git style of handling help
option: any unexpected command line options before help are reported as
errors, but after help ignored.
Bug: 484951
Change-Id: If45c41c0d32895ab6822a7ff9d851877dcef5771
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Allow checkout paths without specifying branch name
JGit CLI should allow to do this: checkout -- <path>
Currently, even if "a" is a valid path in the git repo, jgit CLI can't
checkout it:
$jgit checkout -- a
error: pathspec 'a' did not match any file(s) known to git.
The fix also fixes at same time "unnamed" zombie "[VAL ...]" argument
shown on the command line.
Before fix:
$jgit -h
jgit checkout name [VAL ...] [-- path ... ...] [--force (-f)] [--help
(-h)] [--orphan] [-b]
After fix:
$jgit -h
jgit checkout [name] [-- path ... ...] [--force (-f)] [--help (-h)]
[--orphan] [-b]
Bug: 475765
Change-Id: I2b0e77959a72e4aac68452dc3846adaa745b0831
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Fix push with jgit pgm failing with "unauthorized"
Pushing with JGit commandline to e.g. Github failed with "unauthorized"
since HttpUrlConnection calls the configured authenticator implicitly.
The problem is that during a push two requests are sent to the server,
first a GET and then a POST (containing the pack data). The first GET
request sent anonymously is rejected with 401 (unauthorized). When an
Authenticator is installed the java.net classes will use the
Authenticator to ask the user for credentials and retry the request.
But this happens under the hood and JGit level code doesn't see that
this happens.
The next request is the POST but since JGit thinks the first GET request
went through anonymously it doesn't add authentication headers to the
POST request. This POST of course also fails with 401 but since this
request contains a lot of body-data streamed from JGit (the pack file!)
the java.net classes can't simply retry the request with authorization
headers. The whole process fails.
Fix this by using Apache httpclient which doesn't use Authenticator to
retrieve credentials. Instead initialize TransportCommand to use the
default credential provider if no other credentials provider was set
explicitly. org.eclipse.jgit.pgm.Main sets this default for the JGit
command line client.
Change-Id: Ic4e0f8b60d4bd6e69d91eae0c7e1b44cdf851b00
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Null-annotated Ref class and fixed related compiler errors
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Change-Id: I24c8a600ec962d61d5f40abf73eac4203e115240
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
This should mirror the behavior of `git push --atomic` where the
client asks the server to apply all-or-nothing. Some JGit servers
already support this based on a custom DFS backend. InMemoryRepository
is extended to support atomic push for unit testing purposes.
Local disk server side support inside of JGit is a more complex animal
due to the excessive amount of file locking required to protect every
reference as a loose reference.
Change-Id: I15083fbe48447678e034afeffb4639572a32f50c
Null-annotated Repository class and fixed related compiler errors
org.eclipse.jgit.lib.Repository class is an example of the API which
should be written with Java 8 java.util.Optional<T> type. Unfortunately
this API is already released and widely used. The good clients are
currently doing their best with checking return values for null and bad
clients do not know how bad their code is.
I've tried not to change any logic and to be as less intrusive as
possible. Most of the JGit code was well prepared to this, only few
classes needed some smaller fixes.
This change fixes all compiler errors in JGit and replaces possible
NPE's with either appropriate exceptions, avoiding multiple "Nullable
return" method calls or early returning from the method.
Because annotating getDirectory() and getFS() as Nullable would cause
lot of additional changes in JGit and EGit they are postponed.
Change-Id: Ie8369d2c9c5fac5ce83b3b1b9bc217d7b55502a3
Signed-off-by: Andrey Loskutov <loskutov@gmx.de>
Matthias Sohn
David Pursehouse
Shawn Pearce
Andrey Loskutov
Shawn Pearce
Rüdiger Herrmann
Kaloyan Raev