You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

DhtRepository.java 5.4KB

Store Git on any DHT jgit.storage.dht is a storage provider implementation for JGit that permits storing the Git repository in a distributed hashtable, NoSQL system, or other database. The actual underlying storage system is undefined, and can be plugged in by implementing 7 small interfaces: * Database * RepositoryIndexTable * RepositoryTable * RefTable * ChunkTable * ObjectIndexTable * WriteBuffer The storage provider interface tries to assume very little about the underlying storage system, and requires only three key features: * key -> value lookup (a hashtable is suitable) * atomic updates on single rows * asynchronous operations (Java's ExecutorService is easy to use) Most NoSQL database products offer all 3 of these features in their clients, and so does any decent network based cache system like the open source memcache product. Relying only on key equality for data retrevial makes it simple for the storage engine to distribute across multiple machines. Traditional SQL systems could also be used with a JDBC based spi implementation. Before submitting this change I have implemented six storage systems for the spi layer: * Apache HBase[1] * Apache Cassandra[2] * Google Bigtable[3] * an in-memory implementation for unit testing * a JDBC implementation for SQL * a generic cache provider that can ride on top of memcache All six systems came in with an spi layer around 1000 lines of code to implement the above 7 interfaces. This is a huge reduction in size compared to prior attempts to implement a new JGit storage layer. As this package shows, a complete JGit storage implementation is more than 17,000 lines of fairly complex code. A simple cache is provided in storage.dht.spi.cache. Implementers can use CacheDatabase to wrap any other type of Database and perform fast reads against a network based cache service, such as the open source memcached[4]. An implementation of CacheService must be provided to glue this spi onto the network cache. [1] https://github.com/spearce/jgit_hbase [2] https://github.com/spearce/jgit_cassandra [3] http://labs.google.com/papers/bigtable.html [4] http://memcached.org/ Change-Id: I0aa4072781f5ccc019ca421c036adff2c40c4295 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
13 years ago
Store Git on any DHT jgit.storage.dht is a storage provider implementation for JGit that permits storing the Git repository in a distributed hashtable, NoSQL system, or other database. The actual underlying storage system is undefined, and can be plugged in by implementing 7 small interfaces: * Database * RepositoryIndexTable * RepositoryTable * RefTable * ChunkTable * ObjectIndexTable * WriteBuffer The storage provider interface tries to assume very little about the underlying storage system, and requires only three key features: * key -> value lookup (a hashtable is suitable) * atomic updates on single rows * asynchronous operations (Java's ExecutorService is easy to use) Most NoSQL database products offer all 3 of these features in their clients, and so does any decent network based cache system like the open source memcache product. Relying only on key equality for data retrevial makes it simple for the storage engine to distribute across multiple machines. Traditional SQL systems could also be used with a JDBC based spi implementation. Before submitting this change I have implemented six storage systems for the spi layer: * Apache HBase[1] * Apache Cassandra[2] * Google Bigtable[3] * an in-memory implementation for unit testing * a JDBC implementation for SQL * a generic cache provider that can ride on top of memcache All six systems came in with an spi layer around 1000 lines of code to implement the above 7 interfaces. This is a huge reduction in size compared to prior attempts to implement a new JGit storage layer. As this package shows, a complete JGit storage implementation is more than 17,000 lines of fairly complex code. A simple cache is provided in storage.dht.spi.cache. Implementers can use CacheDatabase to wrap any other type of Database and perform fast reads against a network based cache service, such as the open source memcached[4]. An implementation of CacheService must be provided to glue this spi onto the network cache. [1] https://github.com/spearce/jgit_hbase [2] https://github.com/spearce/jgit_cassandra [3] http://labs.google.com/papers/bigtable.html [4] http://memcached.org/ Change-Id: I0aa4072781f5ccc019ca421c036adff2c40c4295 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
13 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171
  1. /*
  2. * Copyright (C) 2011, Google Inc.
  3. * and other copyright owners as documented in the project's IP log.
  4. *
  5. * This program and the accompanying materials are made available
  6. * under the terms of the Eclipse Distribution License v1.0 which
  7. * accompanies this distribution, is reproduced below, and is
  8. * available at http://www.eclipse.org/org/documents/edl-v10.php
  9. *
  10. * All rights reserved.
  11. *
  12. * Redistribution and use in source and binary forms, with or
  13. * without modification, are permitted provided that the following
  14. * conditions are met:
  15. *
  16. * - Redistributions of source code must retain the above copyright
  17. * notice, this list of conditions and the following disclaimer.
  18. *
  19. * - Redistributions in binary form must reproduce the above
  20. * copyright notice, this list of conditions and the following
  21. * disclaimer in the documentation and/or other materials provided
  22. * with the distribution.
  23. *
  24. * - Neither the name of the Eclipse Foundation, Inc. nor the
  25. * names of its contributors may be used to endorse or promote
  26. * products derived from this software without specific prior
  27. * written permission.
  28. *
  29. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
  30. * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
  31. * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
  32. * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  33. * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
  34. * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  35. * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
  36. * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
  37. * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
  38. * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
  39. * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
  40. * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
  41. * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  42. */
  43. package org.eclipse.jgit.storage.dht;
  44. import java.io.IOException;
  45. import java.text.MessageFormat;
  46. import java.util.concurrent.TimeoutException;
  47. import org.eclipse.jgit.lib.Constants;
  48. import org.eclipse.jgit.lib.RefUpdate;
  49. import org.eclipse.jgit.lib.Repository;
  50. import org.eclipse.jgit.lib.StoredConfig;
  51. import org.eclipse.jgit.storage.dht.spi.Database;
  52. import org.eclipse.jgit.storage.file.ReflogReader;
  53. /**
  54. * A Git repository storing its objects and references in a DHT.
  55. * <p>
  56. * With the exception of repository creation, this class is thread-safe, but
  57. * readers created from it are not. When creating a new repository using the
  58. * {@link #create(boolean)} method, the newly constructed repository object does
  59. * not ensure the assigned {@link #getRepositoryKey()} will be visible to all
  60. * threads. Applications are encouraged to use their own synchronization when
  61. * sharing a Repository instance that was used to create a new repository.
  62. */
  63. public class DhtRepository extends Repository {
  64. private final RepositoryName name;
  65. private final Database db;
  66. private final DhtRefDatabase refdb;
  67. private final DhtObjDatabase objdb;
  68. private final DhtConfig config;
  69. private RepositoryKey key;
  70. /**
  71. * Initialize an in-memory representation of a DHT backed repository.
  72. *
  73. * @param builder
  74. * description of the repository and its data storage.
  75. */
  76. public DhtRepository(DhtRepositoryBuilder builder) {
  77. super(builder);
  78. this.name = RepositoryName.create(builder.getRepositoryName());
  79. this.key = builder.getRepositoryKey();
  80. this.db = builder.getDatabase();
  81. this.refdb = new DhtRefDatabase(this, db);
  82. this.objdb = new DhtObjDatabase(this, builder);
  83. this.config = new DhtConfig();
  84. }
  85. /** @return database cluster that houses this repository (among others). */
  86. public Database getDatabase() {
  87. return db;
  88. }
  89. /** @return human readable name used to open this repository. */
  90. public RepositoryName getRepositoryName() {
  91. return name;
  92. }
  93. /** @return unique identity of the repository in the {@link #getDatabase()}. */
  94. public RepositoryKey getRepositoryKey() {
  95. return key;
  96. }
  97. @Override
  98. public StoredConfig getConfig() {
  99. return config;
  100. }
  101. @Override
  102. public DhtRefDatabase getRefDatabase() {
  103. return refdb;
  104. }
  105. @Override
  106. public DhtObjDatabase getObjectDatabase() {
  107. return objdb;
  108. }
  109. @Override
  110. public void create(boolean bare) throws IOException {
  111. if (!bare)
  112. throw new IllegalArgumentException(
  113. DhtText.get().repositoryMustBeBare);
  114. if (getObjectDatabase().exists())
  115. throw new DhtException(MessageFormat.format(
  116. DhtText.get().repositoryAlreadyExists, name.asString()));
  117. try {
  118. key = db.repository().nextKey();
  119. db.repositoryIndex().putUnique(name, key);
  120. } catch (TimeoutException err) {
  121. throw new DhtTimeoutException(MessageFormat.format(
  122. DhtText.get().timeoutLocatingRepository, name), err);
  123. }
  124. String master = Constants.R_HEADS + Constants.MASTER;
  125. RefUpdate.Result result = updateRef(Constants.HEAD, true).link(master);
  126. if (result != RefUpdate.Result.NEW)
  127. throw new IOException(result.name());
  128. }
  129. @Override
  130. public void scanForRepoChanges() {
  131. refdb.clearCache();
  132. }
  133. @Override
  134. public void notifyIndexChanged() {
  135. // we do not support non-bare repositories yet
  136. }
  137. @Override
  138. public String toString() {
  139. return "DhtRepostitory[" + key + " / " + name + "]";
  140. }
  141. // TODO This method should be removed from the JGit API.
  142. @Override
  143. public ReflogReader getReflogReader(String refName) {
  144. throw new UnsupportedOperationException();
  145. }
  146. }