mirror of
https://github.com/eclipse-jgit/jgit.git
synced 2024-09-13 15:46:45 +02:00
de8946c0c2
jgit.storage.dht is a storage provider implementation for JGit that permits storing the Git repository in a distributed hashtable, NoSQL system, or other database. The actual underlying storage system is undefined, and can be plugged in by implementing 7 small interfaces: * Database * RepositoryIndexTable * RepositoryTable * RefTable * ChunkTable * ObjectIndexTable * WriteBuffer The storage provider interface tries to assume very little about the underlying storage system, and requires only three key features: * key -> value lookup (a hashtable is suitable) * atomic updates on single rows * asynchronous operations (Java's ExecutorService is easy to use) Most NoSQL database products offer all 3 of these features in their clients, and so does any decent network based cache system like the open source memcache product. Relying only on key equality for data retrevial makes it simple for the storage engine to distribute across multiple machines. Traditional SQL systems could also be used with a JDBC based spi implementation. Before submitting this change I have implemented six storage systems for the spi layer: * Apache HBase[1] * Apache Cassandra[2] * Google Bigtable[3] * an in-memory implementation for unit testing * a JDBC implementation for SQL * a generic cache provider that can ride on top of memcache All six systems came in with an spi layer around 1000 lines of code to implement the above 7 interfaces. This is a huge reduction in size compared to prior attempts to implement a new JGit storage layer. As this package shows, a complete JGit storage implementation is more than 17,000 lines of fairly complex code. A simple cache is provided in storage.dht.spi.cache. Implementers can use CacheDatabase to wrap any other type of Database and perform fast reads against a network based cache service, such as the open source memcached[4]. An implementation of CacheService must be provided to glue this spi onto the network cache. [1] https://github.com/spearce/jgit_hbase [2] https://github.com/spearce/jgit_cassandra [3] http://labs.google.com/papers/bigtable.html [4] http://memcached.org/ Change-Id: I0aa4072781f5ccc019ca421c036adff2c40c4295 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
90 lines
4.2 KiB
Plaintext
90 lines
4.2 KiB
Plaintext
JGit Storage on DHT
|
|
-------------------
|
|
|
|
This implementation still has some pending issues:
|
|
|
|
* DhtInserter must skip existing objects
|
|
|
|
DirCache writes all trees to the ObjectInserter, letting the
|
|
inserter figure out which trees we already have, and which are new.
|
|
DhtInserter should buffer trees into a chunk, then before writing
|
|
the chunk to the DHT do a batch lookup to find the existing
|
|
ObjectInfo (if any). If any exist, the chunk should be compacted to
|
|
eliminate these objects, and if there is room in the chunk for more
|
|
objects, it should go back to the DhtInserter to be filled further
|
|
before flushing.
|
|
|
|
This implies the DhtInserter needs to work on multiple chunks at
|
|
once, and may need to combine chunks together when there is more
|
|
than one partial chunk.
|
|
|
|
* DhtPackParser must check for collisions
|
|
|
|
Because ChunkCache blindly assumes any copy of an object is an OK
|
|
copy of an object, DhtPackParser needs to validate all new objects
|
|
at the end of its importing phase, before it links the objects into
|
|
the ObjectIndexTable. Most objects won't already exist, but some
|
|
may, and those that do must either be removed from their chunk, or
|
|
have their content byte-for-byte validated.
|
|
|
|
Removal from a chunk just means deleting it from the chunk's local
|
|
index, and not writing it to the global ObjectIndexTable. This
|
|
creates a hole in the chunk which is wasted space, and that isn't
|
|
very useful. Fortunately objects that fit fully within one chunk
|
|
may be easy to inflate and double check, as they are small. Objects
|
|
that are big span multiple chunks, and the new chunks can simply be
|
|
deleted from the ChunkTable, leaving the original chunks.
|
|
|
|
Deltas can be checked quickly by inflating the delta and checking
|
|
only the insertion point text, comparing that to the existing data
|
|
in the repository. Unfortunately the repository is likely to use a
|
|
different delta representation, which means at least one of them
|
|
will need to be fully inflated to check the delta against.
|
|
|
|
* DhtPackParser should handle small-huge-small-huge
|
|
|
|
Multiple chunks need to be open at once, in case we get a bad
|
|
pattern of small-object, huge-object, small-object, huge-object. In
|
|
this case the small-objects should be put together into the same
|
|
chunk, to prevent having too many tiny chunks. This is tricky to do
|
|
with OFS_DELTA. A long OFS_DELTA requires all prior chunks to be
|
|
closed out so we know their lengths.
|
|
|
|
* RepresentationSelector performance bad on Cassandra
|
|
|
|
The 1.8 million batch lookups done for linux-2.6 kills Cassandra, it
|
|
cannot handle this read load.
|
|
|
|
* READ_REPAIR isn't fully accurate
|
|
|
|
There are a lot of places where the generic DHT code should be
|
|
helping to validate the local replica is consistent, and where it is
|
|
not, help the underlying storage system to heal the local replica by
|
|
reading from a remote replica and putting it back to the local one.
|
|
Most of this should be handled in the DHT SPI layer, but the generic
|
|
DHT code should be giving better hints during get() method calls.
|
|
|
|
* LOCAL / WORLD writes
|
|
|
|
Many writes should be done locally first, before they replicate to
|
|
the other replicas, as they might be backed out on an abort.
|
|
|
|
Likewise some writes must take place across sufficient replicas to
|
|
ensure the write is not lost... and this may include ensuring that
|
|
earlier local-only writes have actually been committed to all
|
|
replicas. This committing to replicas might be happening in the
|
|
background automatically after the local write (e.g. Cassandra will
|
|
start to send writes made by one node to other nodes, but doesn't
|
|
promise they finish). But parts of the code may need to force this
|
|
replication to complete before the higher level git operation ends.
|
|
|
|
* Forks/alternates
|
|
|
|
Forking is common, but we should avoid duplicating content into the
|
|
fork if the base repository has it. This requires some sort of
|
|
change to the key structure so that chunks are owned by an object
|
|
pool, and the object pool owns the repositories that use it. GC
|
|
proceeds at the object pool level, rather than the repository level,
|
|
but might want to take some of the reference namespace into account
|
|
to avoid placing forked less-common content near primary content.
|