jgit.storage.dht is a storage provider implementation for JGit that
permits storing the Git repository in a distributed hashtable, NoSQL
system, or other database. The actual underlying storage system is
undefined, and can be plugged in by implementing 7 small interfaces:
* Database
* RepositoryIndexTable
* RepositoryTable
* RefTable
* ChunkTable
* ObjectIndexTable
* WriteBuffer
The storage provider interface tries to assume very little about the
underlying storage system, and requires only three key features:
* key -> value lookup (a hashtable is suitable)
* atomic updates on single rows
* asynchronous operations (Java's ExecutorService is easy to use)
Most NoSQL database products offer all 3 of these features in their
clients, and so does any decent network based cache system like the
open source memcache product. Relying only on key equality for data
retrevial makes it simple for the storage engine to distribute across
multiple machines. Traditional SQL systems could also be used with a
JDBC based spi implementation.
Before submitting this change I have implemented six storage systems
for the spi layer:
* Apache HBase[1]
* Apache Cassandra[2]
* Google Bigtable[3]
* an in-memory implementation for unit testing
* a JDBC implementation for SQL
* a generic cache provider that can ride on top of memcache
All six systems came in with an spi layer around 1000 lines of code to
implement the above 7 interfaces. This is a huge reduction in size
compared to prior attempts to implement a new JGit storage layer. As
this package shows, a complete JGit storage implementation is more
than 17,000 lines of fairly complex code.
A simple cache is provided in storage.dht.spi.cache. Implementers can
use CacheDatabase to wrap any other type of Database and perform fast
reads against a network based cache service, such as the open source
memcached[4]. An implementation of CacheService must be provided to
glue this spi onto the network cache.
[1] https://github.com/spearce/jgit_hbase
[2] https://github.com/spearce/jgit_cassandra
[3] http://labs.google.com/papers/bigtable.html
[4] http://memcached.org/
Change-Id: I0aa4072781f5ccc019ca421c036adff2c40c4295
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Revert "Hide Maven target directories from Eclipse"
This reverts commit db4c516f67 since
it breaks compatibility with Eclipse 3.5 which can no longer import
the projects
Bug: 323390
Change-Id: I3cc91364a6747cfcb4c611a9be5258f81562f726
This is a simple HTTP server that provides the minimum server side
support required for dumb (non-git aware) transport clients.
We produce the info/refs and objects/info/packs file on the fly
from the local repository state, but otherwise serve data as raw
files from the on-disk structure.
In the future we could better optimize the FileSender class and the
servlets that use it to take advantage of direct file to network
APIs in more advanced servlet containers like Jetty.
Our glue package borrows the idea of a micro embedded DSL from
Google Guice and uses it to configure a collection of Filters
and HttpServlets, all of which are matched against requests using
regular expressions. If a subgroup exists in the pattern, it is
extracted and used for the path info component of the request.
Change-Id: Ia0f1a425d07d035e344ae54faf8aeb04763e7487
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Partial revert "Switch build to Apache Felix maven-bundle-plugin"
This restores the ability to build using just Eclipse without
strange procedures, extra plugins and it is again possible to
work on both JGit and EGit in the same Eclipse workspace with
ease.
Change-Id: I0af08127d507fbce186f428f1cdeff280f0ddcda
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
Tycho isn't production ready for projects like JGit to be using as
their primary build driver. Some problems we ran into with Tycho
0.6.0 that are preventing us from using it are:
* Tycho can't run offline
The P2 artifact resolver cannot perform its work offline. If the
build system has no network connection, it cannot compile a
project through Tycho. This is insane for a distributed version
control system where developers are used to being offline during
development and local testing.
* Magic state in ~/.m2/repository/.meta/p2-metadata.properties
Earlier iterations of this patch tried to use a hybrid build,
where Tycho was only used for the Eclipse specific feature and P2
update site, and maven-bundle-plugin was used for the other code.
This build seemed to work, but only due to magic Tycho specific
state held in my local home directory. This means builds are not
consistently repeatable across systems, and lead me to believe
I had a valid build, when in fact I did not.
* Manifest-first build produces incomplete POMs
The POM created by the manifest-first build format does not
contain the dependency chain, leading a downstream consumer to
not import the runtime dependencies necessary to execute the
bundle it has imported. In JGit's case, this means JSch isn't
included in our dependency chain.
* Manifest-first build produces POMs unreadable by Maven 2.x
JGit has existing application consumers who are relying on
Maven 2.x builds. Forcing them to step up to an alpha release
of Maven 3 is simply unacceptable.
* OSGi bundle export data management is tedious
Editing each of our pom.xml files to mark a new release is
difficult enough as it is. Editing every MANIFEST.MF file to
list our exported packages and their current version number is
something a machine should do, not a human. Yet the Tycho OSGi
way unfortunately demands that a human do this work.
* OSGi bundle import data management is tedious
There isn't a way in the MANIFEST.MF file format to reuse the
same version tags across all of our imports, but we want to have
a consistent view of our dependencies when we compile JGit.
After wasting more than 2 full days trying to get Tycho to work,
I've decided its a lost cause right now. We need to be chasing down
bugs and critical features, not trying to bridge the gap between
the stable Maven repository format and the undocumented P2 format
used only by Eclipse.
So, switch the build to use Apache Felix's maven-bundle-plugin.
This is the same plugin Jetty uses to produce their OSGi bundle
manifests, and is the same plugin used by the Apache Felix project,
which is an open-source OSGi runtime. It has a reasonable number
of folks using it for production builds, and is running on top of
the stable Maven 2.x code base.
With this switch we get automatically generated MANIFEST.MF files
based on reasonably sane default rules, which reduces the amount
of things we have to maintain by hand. When necessary, we can add
a few lines of XML to our POMs to tweak the output.
Our build artifacts are still fully compatible with Maven 2.x, so
any downstream consumers are still able to use our build products,
without stepping up to Maven 3.x. Our artifacts are also valid as
OSGi bundles, provided they are organized on disk into a repository
that the runtime can read.
With maven-bundle-plugin the build runs offline, as much as Maven
2.x is able to run offline anyway, so we're able to return to a
distributed development environment again.
By generating MANIFEST.MF at the top level of each project (and
therefore outside of the target directory), we're still compatible
with Eclipse's PDE tooling. Our projects can be imported as standard
Maven projects using the m2eclipse plugin, but the PDE will think
they are vaild plugins and make them available for plugin builds,
or while debugging another workbench.
This change also completely removes Tycho from the build.
Unfortunately, Tycho 0.6.0's pom-first dependency resolver is broken
when resolving a pom-first plugin bundle through a manifest-first
feature package, so bundle org.eclipse.jgit can't be resolved,
even though it might actually exist in the local Maven repository.
Rather than fight with Tycho any further, I'm just declaring it
plugina-non-grata and ripping it out of the build.
Since there are very few tools to build a P2 format repository, and
no documentation on how to create one without running the Eclipse
UI manually by poking buttons, I'm declaring that we are not going
to produce a P2 update site from our automated builds.
Change-Id: If7938a86fb0cc8e25099028d832dbd38110b9124
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This way dependencies are described by the MANIFEST.MF,
and the same build tools can be used to compile the tests.
Change-Id: I4dc926148410ecbadcf71b9474aeeb509691aa32
Per CQ 3448 this is the initial contribution of the JGit project
to eclipse.org. It is derived from the historical JGit repository
at commit 3a2dd9921c.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>