Shawn Pearce [Mon, 18 Jan 2016 18:33:31 +0000 (10:33 -0800)]
Ketch: Basic replication system
Git Ketch is a multi-master Git repository management system. Writes
are successful only if a majority of participant servers agree. Acked
writes are durable against server failures as a majority of the
participants store all required objects.
Git Ketch is modeled on the Raft Consensus Algorithm[1]. A ketch
sailing vessel is faster and more nimble than a raft. It can also
carry more source codes.
Git Ketch front-loads replication costs, which vaguely resembles a
ketch sailing vessel's distinguishing feature of the main mast on the
front of the ship.
Dave Borowitz [Tue, 19 Jan 2016 16:13:40 +0000 (11:13 -0500)]
ChangeIdUtil: Don't throw IOException
This could have only happened during the getBytes call. Instead, use
Constants.encode, which is a non-throwing implementation.
This change is binary compatible with existing code compiled against
older versions of JGit, although it might break compilation of
previously compiling code due to dead catch blocks.
Shawn Pearce [Tue, 19 Jan 2016 21:08:38 +0000 (13:08 -0800)]
Transport: Implement AutoCloseable
After creating a Transport instance callers should always call
its close() method. Use AutoCloseable to document this idiom
and allow use of try-with-resources.
Shawn Pearce [Tue, 19 Jan 2016 22:08:53 +0000 (17:08 -0500)]
Merge changes from topic 'deadtree'
* changes:
Ignore API errors about Tree, TreeEntry, FileTreeEntry and friends being removed
Revert "Revert "Remove deprecated Tree, TreeEntry, FileTreeEntry and friends""
Shawn Pearce [Mon, 18 Jan 2016 18:42:00 +0000 (10:42 -0800)]
ReceiveCommand.abort(): Utility to mark batch of commands as failed
If one or more commands is failing the entire group usually has to
also fail with "transaction aborted". Pull this loop into a helper
so the idiom can be easily reused in several places throughout JGit.
Shawn Pearce [Wed, 13 Jan 2016 00:11:36 +0000 (16:11 -0800)]
PackWriter: Declare preparePack object sets as @NonNull
Require callers to pass in valid sets for both want and have
collections. Offer PackWriter.NONE as a handy constant for an
empty collection for the have part of preparePack instead of null.
Shawn Pearce [Tue, 12 Jan 2016 18:50:36 +0000 (10:50 -0800)]
GC: Pack RefTrees in their own pack
The RefTree graph needs to be quickly accessed to read references.
It is also distinct graph disconnected from the rest of the
repository. Store the commit and tree objects in their own pack.
Shawn Pearce [Sun, 10 Jan 2016 01:27:25 +0000 (17:27 -0800)]
RefTree: Change peel suffix to " ^" (space carrot)
Using ^{} as the peel suffix has caused problems when projects used
tags like v2.1 and then v2.1.1, v2.2.2, etc. The peeled value for
v2.1 was stored very far away in the tree relative to v2.1 itself as
^ sorts in the ASCII/UTF-8 encoding after all other common tag
characters like digits and dots.
Use " ^" instead as space is not valid in a reference name, sorts
before all other valid reference characters (thus forcing next entry
locality) and this looks like a peeled marker for the prior tag.
Shawn Pearce [Sat, 9 Jan 2016 20:51:14 +0000 (12:51 -0800)]
FileRepository: Support extensions.refsBackendType = RefTree
This experimental code can be enabled in $GIT_DIR/config:
[core]
repositoryformatversion = 1
[extensions]
refsBackendType = RefTree
When these are set the repository will read references from the
RefTree rooted by the $GIT_DIR/refs/txn/committed reference.
Update debug-rebuild-ref-tree to rebuild refs/txn/committed only from
the bootstrap layer. This avoids misuse by rebuilding using packed-refs
and $GIT_DIR/refs tree.
Shawn Pearce [Sat, 28 Nov 2015 07:21:43 +0000 (23:21 -0800)]
RefTreeDatabase: Ref database using refs/txn/committed
Instead of storing references in the local filesystem rely on the
RefTree rooted at refs/txn/committed. This avoids needing to store
references in the packed-refs file by keeping all data rooted under
a single refs/txn/committed ref.
Performance to scan all references from a well packed RefTree is very
close to reading the packed-refs file from local disk.
Storing a packed RefTree is smaller due to pack file compression,
about 49.39 bytes/ref (on average) compared to packed-refs using
~65.49 bytes/ref.
Shawn Pearce [Mon, 11 Jan 2016 20:30:35 +0000 (12:30 -0800)]
RevCommit: Better support invalid encoding headers
With this support we no longer need the 'utf-8' alias. UTF-8 will be
automatically tried when the encoding header is not recognized and used
if the character sequence cleanly decodes as UTF-8.
Modernize some of the references to use StandardCharsets.
Shawn Pearce [Sat, 28 Nov 2015 03:34:36 +0000 (19:34 -0800)]
debug-rebuild-ref-tree: Simple program to build a RefTree
This tool scans all references in the repository and writes out a new
reference pointing to a single commit whose root tree is a RefTree
containing the current refs of this repository.
It alway skips storing the reference it will write to, avoiding the
obvious cycle.
Shawn Pearce [Wed, 18 Nov 2015 00:22:18 +0000 (16:22 -0800)]
RefTree: Store references in a Git tree
A group of updates can be applied by updating the tree in one step,
writing out a new root tree, and storing its SHA-1. If references
are stored in RefTrees, comparing two repositories is a matter of
checking if two SHA-1s are identical. Without RefTrees comparing two
repositories requires listing all references and comparing the sets.
Track the "refs/" directory as a root tree by storing references
that point directly at an object as a GITLINK entry in the tree.
For example "refs/heads/master" is written as "heads/master".
Annotated tags also store their peeled value with ^{} suffix, using
"tags/v1.0" and "tags/v1.0^{}" GITLINK entries.
Symbolic references are written as SYMLINK entries with the blob of
the symlink carrying the name of the symbolic reference target.
HEAD is outside of "refs/" namespace so it is stored as a special
"..HEAD" entry. This name is chosen because ".." is not valid in
a reference name and it almost looks like "../HEAD" which names
HEAD if the reader was inside of the "refs/" directory.
A new Command type is required to handle symbolic references and
peeled references.
Andrey Loskutov [Sun, 3 Jan 2016 11:29:55 +0000 (12:29 +0100)]
Make sure CLIGitCommand and Main produce (almost) same results
Currently execution of tests in pgm uses CLIGitCommand which
re-implements few things from Main. Unfortunately this can results in a
different test behavior compared to the real CLI runtime.
The change let CLIGitCommand extend Main and only slightly modifies the
runtime (stream redirection and undesired exit() termination).
Andrey Loskutov [Sun, 3 Jan 2016 14:27:01 +0000 (15:27 +0100)]
Added CLIText.fatalError(String) API for tests
In different places (Main, TextBuiltin, CLIGitCommand) we report fatal
errors and at same time want to check for fatal errors in the tests.
Using common API simplifies the error testing and helps to navigate to
the actual error check implementation.
The recent ObjectChecker changes to pass in AnyObjectId as part
of the checkCommit method signature meant the override here was no
longer throwing an exception as expected.
David Ostrovsky [Sun, 3 Jan 2016 23:44:07 +0000 (00:44 +0100)]
buck: Make :jgit_src target work in cross-cell environment
This artifact is used from unzip utility in Gerrit Code Review
build toolchain and thus the file must exist on the file system.
Moreover, trying to use java_binary() didn't work either, as the
zip layout was wrong: all files contained 'org.eclipse.jgit/src/'
prefix.
Change-Id: I00e3269a7a1a6c6d1fe7e60d1bf1c69b8e57d79d Signed-off-by: David Ostrovsky <david@ostrovsky.org>
Matthias Sohn [Sat, 2 Jan 2016 01:09:16 +0000 (02:09 +0100)]
Ensure all http tests are run and fix broken tests
HttpClientTests were broken. This wasn't discovered since
maven-surefire-plugin's by default only executes test classes
matching **/*Test.java. Fix this by also including **/*.Tests.java
and fix the failing tests.
Change-Id: I487a30fb333de993a9f8d8fff491d3b0e7fb02cc Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Matthias Sohn [Fri, 1 Jan 2016 22:54:15 +0000 (23:54 +0100)]
buck: run http tests
Running tests using buck reveals that HttpClientTests are broken
and weren't executed by Maven since these test classes don't match the
maven-surefire-plugin's default for test classes **/*Test.java.
Will be fixed in a follow-up change.
Change-Id: I82a01b5fd3f0a930bec2423a29a256601dadc248 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Shawn Pearce [Fri, 1 Jan 2016 19:04:11 +0000 (11:04 -0800)]
buck: set vm_args for tests
Maven pom files force the local encoding to UTF-8 to ensure there are
no differences between machines. They also set the JVM max heap to
256m. Match both in Buck so that results are consistent.
Shawn Pearce [Fri, 1 Jan 2016 04:07:03 +0000 (20:07 -0800)]
buck: pin to stable version
Like with Gerrit, pin JGit to a single version of Buck that is known
to work with current Buck files and JUnit tests. Notably a more recent
version of Buck used by Gerrit (01a0c54d827) fails WalkEncryptionTest.
Shawn Pearce [Fri, 1 Jan 2016 17:58:34 +0000 (12:58 -0500)]
Merge changes from topic 'add-df'
* changes:
DirCache: Do not create duplicate tree entries
DirCacheEditor: Replace file-with-tree and tree-with-file
AddCommand: Use NameConflictTreeWalk to identify file-dir changes
Shawn Pearce [Fri, 1 Jan 2016 00:12:51 +0000 (16:12 -0800)]
Fix "remote: Counting objects: ..." formatting
Trailing whitespace is usually removed in properties files so
JGitText did not supply a space between : and the remote message.
Ensure the space exists at runtime by reading the localized string
and appending a space if it is missing.
Messages should be dynamically fetched and not held in a static
class variable, as they can be changed using thread locals.
The attributes, that all these build tools have in common, are:
* reliable
* correct
* very fast
* reproducible
It must not always be the other build tool, this project is currently
using. Or, quoting Gerrit Code Review maintainer here:
"Friends, don't let friends use <the other build tool system>!"
This change is non-complete implementation of JGit build in Buck,
needed by Gerrit Code Review to replace its dependency with standlone
JGit cell. This is very useful when a developer is working on both
projects and is trying to integrate changes made in JGit in Gerrit.
The supported workflow is:
$ cd jgit
$ emacs <hack>
$ cd ../gerrit
$ buck build --config repositories.jgit=../jgit gerrit
With --config repositories.jgit=../jgit jgit cell is routed through
JGit development tree.
Eryk Szymanski [Wed, 17 Jun 2015 15:17:17 +0000 (17:17 +0200)]
Fix encoding problem from curl repostory on github
Pushing curl repository to gerrit fails with a message:
remote: error: internal error while processing changes
java.nio.charset.IllegalCharsetNameException: 'utf8'
The zeroPaddedFilemode = ignore is a synonym for the JGit specific
allowLeadingZeroFileMode = true. Only accept the JGit key if git-core
key was not specified.
Shawn Pearce [Tue, 29 Dec 2015 23:52:16 +0000 (15:52 -0800)]
ObjectChecker: allow some objects to skip errors
Some ancient objects may be broken, but in a relatively harmless way.
Allow the ObjectChecker caller to whitelist specific objects that are
going to fail checks, but that have been reviewed by a human and decided
the objects are OK enough to permit continued use of.
This avoids needing to rewrite history to scrub the broken objects out.
Honor the git-core fsck.skipList configuration setting when receiving a
push or fetching from a remote repository.
Shawn Pearce [Wed, 30 Dec 2015 20:23:06 +0000 (12:23 -0800)]
ObjectChecker: use java.text.Normalizer directly
Base Java version for JGit is now Java 7. The java.text.Normalizer
class was available in Java 6. Reflection is no longer required to
normalize strings for Mac OS X.
* changes:
Sort "eager" path-like options to the end of the help
reset command: provide convenient and meaningful options help
commit command: allow to specify path(s) argument(s)
status command: consume more then one argument after --
repo command: properly name the required 'path' argument
Un-ignored existing CLI tests which run just fine on Java 7+
Don't treat command termination due '-h' option as a fatal error
Shawn Pearce [Wed, 30 Dec 2015 00:53:56 +0000 (16:53 -0800)]
Unify fetch and receive ObjectChecker setup
This avoids duplication of code between receive-pack and fetch-pack paths.
Separate methods are still required to check use of receive.fsckobjects vs.
fetch.fsckobjects, both of which default to transfer.fsckobjects.
Shawn Pearce [Tue, 29 Dec 2015 23:11:21 +0000 (15:11 -0800)]
PackWriter: use lib.ObjectIdSet to avoid wrapper
Hoist ObjectIdSet up to lib as part of the public API and add
the interface to some common types like PackIndex and JGit custom
ObjectId map types. This cleans up wrapper code in a number of
places by allowing direct use of the types as an ObjectIdSet.
Future commits can now rely on ObjectIdSet as a simple read-only
type to check a set of objects from a number of storage options.
Shawn Pearce [Fri, 18 Dec 2015 22:31:20 +0000 (14:31 -0800)]
DirCache: Do not create duplicate tree entries
If a file (e.g. "A") and a subtree file (e.g. "A/foo.c") both appear
in the DirCache this cache should not be written out as a tree object.
The "A" file and "A" subtree conflict with each other in the same tree
and will fail fsck.
Detect this condition during DirCacheBuilder and DirCacheEditor
finish() so the application can be halted early before it updates a
DirCache that might later write an invalid tree structure.
Shawn Pearce [Mon, 28 Dec 2015 19:43:07 +0000 (11:43 -0800)]
DirCacheEditor: Replace file-with-tree and tree-with-file
If a PathEdit tries to store a file where a subtree was, or a subtree
where a file was, replace the entry in the DirCache with the new
name(s). This supports switching between file and tree entry types
using a DirCacheEditor.
Add new unit tests to cover the conditions where these can happen.
Shawn Pearce [Thu, 24 Dec 2015 22:27:50 +0000 (14:27 -0800)]
AddCommand: Use NameConflictTreeWalk to identify file-dir changes
Adding a path that already exists but is changing type such as
from symlink to subdirectory requires a NameConflictTreeWalk to
match up the two different entry types that share the same name.
NameConflictTreeWalk needs a bug fix to pop conflicting entries
when PathFilterGroup aborts the walk early so that it does not
allow DirCacheBuilderIterator to copy conflicting entries into
the output cache.
Andrey Loskutov [Mon, 28 Dec 2015 22:27:09 +0000 (23:27 +0100)]
Don't treat command termination due '-h' option as a fatal error
Signal early command termination due '-h' or '--help' option via
TerminatedByHelpException. This allows tests using
CLIGitCommand differentiate between unexpected command parsing errors
and expected command cancellation "on help" (which also allows
validation of expected/unexpected help messages).
Additional side-effect: jgit supports now git style of handling help
option: any unexpected command line options before help are reported as
errors, but after help ignored.
Andrey Loskutov [Mon, 28 Dec 2015 14:42:04 +0000 (15:42 +0100)]
Simplify development of commands: added main() to CLIGitCommand
This will execute git commands (with arguments) specified on the command
line, handy for developing/debugging a sequence of arbitrary git
commands working on same repository.
The git working dir path can be specified via Java system property
"git_work_tree". If not specified, current directory will be used.
Shawn Pearce [Thu, 24 Dec 2015 23:46:19 +0000 (15:46 -0800)]
DirCacheEditor: Cleanup DeleteTree constructor
Neaten up formatting and avoid strings, which prevents the need for
NLS comment tags. Instead check the last character using char
literal, and append a char literal instead of a string.
Andrey Loskutov [Wed, 16 Dec 2015 23:12:04 +0000 (00:12 +0100)]
Allow checkout paths without specifying branch name
JGit CLI should allow to do this: checkout -- <path>
Currently, even if "a" is a valid path in the git repo, jgit CLI can't
checkout it:
$jgit checkout -- a
error: pathspec 'a' did not match any file(s) known to git.
The fix also fixes at same time "unnamed" zombie "[VAL ...]" argument
shown on the command line.
Before fix:
$jgit -h
jgit checkout name [VAL ...] [-- path ... ...] [--force (-f)] [--help
(-h)] [--orphan] [-b]
projects / jgit.git / log