![Shawn O. Pearce](https://secure.gravatar.com/avatar/a4611f1fb34714fc54ceec3859c490f7?d=identicon) DFS: A storage layer for JGit
In practice the DHT storage layer has not been performing as well as
large scale server environments want to see from a Git server.
The performance of the DHT schema degrades rapidly as small changes
are pushed into the repository due to the chunk size being less than
1/3 of the pushed pack size. Small chunks cause poor prefetch
performance during reading, and require significantly longer prefetch
lists inside of the chunk meta field to work around the small size.
The DHT code is very complex (>17,000 lines of code) and is very
sensitive to the underlying database round-trip time, as well as the
way objects were written into the pack stream that was chunked and
stored on the database. A poor pack layout (from any version of C Git
prior to Junio reworking it) can cause the DHT code to be unable to
enumerate the objects of the linux-2.6 repository in a completable
time scale.
Performing a clone from a DHT stored repository of 2 million objects
takes 2 million row lookups in the DHT to locate the OBJECT_INDEX row
for each object being cloned. This is very difficult for some DHTs to
scale, even at 5000 rows/second the lookup stage alone takes 6 minutes
(on local filesystem, this is almost too fast to bother measuring).
Some servers like Apache Cassandra just fall over and cannot complete
the 2 million lookups in rapid fire.
On a ~400 MiB repository, the DHT schema has an extra 25 MiB of
redundant data that gets downloaded to the JGit process, and that is
before you consider the cost of the OBJECT_INDEX table also being
fully loaded, which is at least 223 MiB of data for the linux kernel
repository. In the DHT schema answering a `git clone` of the ~400 MiB
linux kernel needs to load 248 MiB of "index" data from the DHT, in
addition to the ~400 MiB of pack data that gets sent to the client.
This is 193 MiB more data to be accessed than the native filesystem
format, but it needs to come over a much smaller pipe (local Ethernet
typically) than the local SATA disk drive.
I also never got around to writing the "repack" support for the DHT
schema, as it turns out to be fairly complex to safely repack data in
the repository while also trying to minimize the amount of changes
made to the database, due to very common limitations on database
mutation rates..
This new DFS storage layer fixes a lot of those issues by taking the
simple approach for storing relatively standard Git pack and index
files on an abstract filesystem. Packs are accessed by an in-process
buffer cache, similar to the WindowCache used by the local filesystem
storage layer. Unlike the local file IO, there are some assumptions
that the storage system has relatively high latency and no concept of
"file handles". Instead it looks at the file more like HTTP byte range
requests, where a read channel is a simply a thunk to trigger a read
request over the network.
The DFS code in this change is still abstract, it does not store on
any particular filesystem, but is fairly well suited to the Amazon S3
or Apache Hadoop HDFS. Storing packs directly on HDFS rather than
HBase removes a layer of abstraction, as most HBase row reads turn
into an HDFS read.
Most of the DFS code in this change was blatently copied from the
local filesystem code. Most parts should be refactored to be shared
between the two storage systems, but right now I am hesistent to do
this due to how well tuned the local filesystem code currently is.
Change-Id: Iec524abdf172e9ec5485d6c88ca6512cd8a6eafb
13 years ago ![Shawn O. Pearce](https://secure.gravatar.com/avatar/a4611f1fb34714fc54ceec3859c490f7?d=identicon) DFS: A storage layer for JGit
In practice the DHT storage layer has not been performing as well as
large scale server environments want to see from a Git server.
The performance of the DHT schema degrades rapidly as small changes
are pushed into the repository due to the chunk size being less than
1/3 of the pushed pack size. Small chunks cause poor prefetch
performance during reading, and require significantly longer prefetch
lists inside of the chunk meta field to work around the small size.
The DHT code is very complex (>17,000 lines of code) and is very
sensitive to the underlying database round-trip time, as well as the
way objects were written into the pack stream that was chunked and
stored on the database. A poor pack layout (from any version of C Git
prior to Junio reworking it) can cause the DHT code to be unable to
enumerate the objects of the linux-2.6 repository in a completable
time scale.
Performing a clone from a DHT stored repository of 2 million objects
takes 2 million row lookups in the DHT to locate the OBJECT_INDEX row
for each object being cloned. This is very difficult for some DHTs to
scale, even at 5000 rows/second the lookup stage alone takes 6 minutes
(on local filesystem, this is almost too fast to bother measuring).
Some servers like Apache Cassandra just fall over and cannot complete
the 2 million lookups in rapid fire.
On a ~400 MiB repository, the DHT schema has an extra 25 MiB of
redundant data that gets downloaded to the JGit process, and that is
before you consider the cost of the OBJECT_INDEX table also being
fully loaded, which is at least 223 MiB of data for the linux kernel
repository. In the DHT schema answering a `git clone` of the ~400 MiB
linux kernel needs to load 248 MiB of "index" data from the DHT, in
addition to the ~400 MiB of pack data that gets sent to the client.
This is 193 MiB more data to be accessed than the native filesystem
format, but it needs to come over a much smaller pipe (local Ethernet
typically) than the local SATA disk drive.
I also never got around to writing the "repack" support for the DHT
schema, as it turns out to be fairly complex to safely repack data in
the repository while also trying to minimize the amount of changes
made to the database, due to very common limitations on database
mutation rates..
This new DFS storage layer fixes a lot of those issues by taking the
simple approach for storing relatively standard Git pack and index
files on an abstract filesystem. Packs are accessed by an in-process
buffer cache, similar to the WindowCache used by the local filesystem
storage layer. Unlike the local file IO, there are some assumptions
that the storage system has relatively high latency and no concept of
"file handles". Instead it looks at the file more like HTTP byte range
requests, where a read channel is a simply a thunk to trigger a read
request over the network.
The DFS code in this change is still abstract, it does not store on
any particular filesystem, but is fairly well suited to the Amazon S3
or Apache Hadoop HDFS. Storing packs directly on HDFS rather than
HBase removes a layer of abstraction, as most HBase row reads turn
into an HDFS read.
Most of the DFS code in this change was blatently copied from the
local filesystem code. Most parts should be refactored to be shared
between the two storage systems, but right now I am hesistent to do
this due to how well tuned the local filesystem code currently is.
Change-Id: Iec524abdf172e9ec5485d6c88ca6512cd8a6eafb
13 years ago ![Shawn O. Pearce](https://secure.gravatar.com/avatar/a4611f1fb34714fc54ceec3859c490f7?d=identicon) DFS: A storage layer for JGit
In practice the DHT storage layer has not been performing as well as
large scale server environments want to see from a Git server.
The performance of the DHT schema degrades rapidly as small changes
are pushed into the repository due to the chunk size being less than
1/3 of the pushed pack size. Small chunks cause poor prefetch
performance during reading, and require significantly longer prefetch
lists inside of the chunk meta field to work around the small size.
The DHT code is very complex (>17,000 lines of code) and is very
sensitive to the underlying database round-trip time, as well as the
way objects were written into the pack stream that was chunked and
stored on the database. A poor pack layout (from any version of C Git
prior to Junio reworking it) can cause the DHT code to be unable to
enumerate the objects of the linux-2.6 repository in a completable
time scale.
Performing a clone from a DHT stored repository of 2 million objects
takes 2 million row lookups in the DHT to locate the OBJECT_INDEX row
for each object being cloned. This is very difficult for some DHTs to
scale, even at 5000 rows/second the lookup stage alone takes 6 minutes
(on local filesystem, this is almost too fast to bother measuring).
Some servers like Apache Cassandra just fall over and cannot complete
the 2 million lookups in rapid fire.
On a ~400 MiB repository, the DHT schema has an extra 25 MiB of
redundant data that gets downloaded to the JGit process, and that is
before you consider the cost of the OBJECT_INDEX table also being
fully loaded, which is at least 223 MiB of data for the linux kernel
repository. In the DHT schema answering a `git clone` of the ~400 MiB
linux kernel needs to load 248 MiB of "index" data from the DHT, in
addition to the ~400 MiB of pack data that gets sent to the client.
This is 193 MiB more data to be accessed than the native filesystem
format, but it needs to come over a much smaller pipe (local Ethernet
typically) than the local SATA disk drive.
I also never got around to writing the "repack" support for the DHT
schema, as it turns out to be fairly complex to safely repack data in
the repository while also trying to minimize the amount of changes
made to the database, due to very common limitations on database
mutation rates..
This new DFS storage layer fixes a lot of those issues by taking the
simple approach for storing relatively standard Git pack and index
files on an abstract filesystem. Packs are accessed by an in-process
buffer cache, similar to the WindowCache used by the local filesystem
storage layer. Unlike the local file IO, there are some assumptions
that the storage system has relatively high latency and no concept of
"file handles". Instead it looks at the file more like HTTP byte range
requests, where a read channel is a simply a thunk to trigger a read
request over the network.
The DFS code in this change is still abstract, it does not store on
any particular filesystem, but is fairly well suited to the Amazon S3
or Apache Hadoop HDFS. Storing packs directly on HDFS rather than
HBase removes a layer of abstraction, as most HBase row reads turn
into an HDFS read.
Most of the DFS code in this change was blatently copied from the
local filesystem code. Most parts should be refactored to be shared
between the two storage systems, but right now I am hesistent to do
this due to how well tuned the local filesystem code currently is.
Change-Id: Iec524abdf172e9ec5485d6c88ca6512cd8a6eafb
13 years ago |
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277 |
- /*
- * Copyright (C) 2010, Mathias Kinzler <mathias.kinzler@sap.com>
- * Copyright (C) 2010, Chris Aniszczyk <caniszczyk@gmail.com>
- * Copyright (C) 2012-2013, Robin Rosenberg
- * and other copyright owners as documented in the project's IP log.
- *
- * This program and the accompanying materials are made available
- * under the terms of the Eclipse Distribution License v1.0 which
- * accompanies this distribution, is reproduced below, and is
- * available at http://www.eclipse.org/org/documents/edl-v10.php
- *
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials provided
- * with the distribution.
- *
- * - Neither the name of the Eclipse Foundation, Inc. nor the
- * names of its contributors may be used to endorse or promote
- * products derived from this software without specific prior
- * written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
- * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
- * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
- * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
- * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
- * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
- * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
- * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
- package org.eclipse.jgit.lib;
-
- /**
- * Constants for use with the Configuration classes: section names,
- * configuration keys
- */
- @SuppressWarnings("nls")
- public class ConfigConstants {
- /** The "core" section */
- public static final String CONFIG_CORE_SECTION = "core";
-
- /** The "branch" section */
- public static final String CONFIG_BRANCH_SECTION = "branch";
-
- /** The "remote" section */
- public static final String CONFIG_REMOTE_SECTION = "remote";
-
- /** The "diff" section */
- public static final String CONFIG_DIFF_SECTION = "diff";
-
- /** The "dfs" section */
- public static final String CONFIG_DFS_SECTION = "dfs";
-
- /** The "user" section */
- public static final String CONFIG_USER_SECTION = "user";
-
- /** The "gerrit" section */
- public static final String CONFIG_GERRIT_SECTION = "gerrit";
-
- /** The "workflow" section */
- public static final String CONFIG_WORKFLOW_SECTION = "workflow";
-
- /** The "submodule" section */
- public static final String CONFIG_SUBMODULE_SECTION = "submodule";
-
- /**
- * The "rebase" section
- *
- * @since 3.2
- */
- public static final String CONFIG_REBASE_SECTION = "rebase";
-
- /** The "gc" section */
- public static final String CONFIG_GC_SECTION = "gc";
-
- /** The "pack" section */
- public static final String CONFIG_PACK_SECTION = "pack";
-
- /**
- * The "fetch" section
- *
- * @since 3.3
- */
- public static final String CONFIG_FETCH_SECTION = "fetch";
-
- /**
- * The "pull" section
- *
- * @since 3.5
- */
- public static final String CONFIG_PULL_SECTION = "pull";
-
- /** The "algorithm" key */
- public static final String CONFIG_KEY_ALGORITHM = "algorithm";
-
- /** The "autocrlf" key */
- public static final String CONFIG_KEY_AUTOCRLF = "autocrlf";
-
- /** The "bare" key */
- public static final String CONFIG_KEY_BARE = "bare";
-
- /** The "excludesfile" key */
- public static final String CONFIG_KEY_EXCLUDESFILE = "excludesfile";
-
- /** The "filemode" key */
- public static final String CONFIG_KEY_FILEMODE = "filemode";
-
- /** The "logallrefupdates" key */
- public static final String CONFIG_KEY_LOGALLREFUPDATES = "logallrefupdates";
-
- /** The "repositoryformatversion" key */
- public static final String CONFIG_KEY_REPO_FORMAT_VERSION = "repositoryformatversion";
-
- /** The "worktree" key */
- public static final String CONFIG_KEY_WORKTREE = "worktree";
-
- /** The "blockLimit" key */
- public static final String CONFIG_KEY_BLOCK_LIMIT = "blockLimit";
-
- /** The "blockSize" key */
- public static final String CONFIG_KEY_BLOCK_SIZE = "blockSize";
-
- /** The "deltaBaseCacheLimit" key */
- public static final String CONFIG_KEY_DELTA_BASE_CACHE_LIMIT = "deltaBaseCacheLimit";
-
- /**
- * The "symlinks" key
- *
- * @since 3.3
- */
- public static final String CONFIG_KEY_SYMLINKS = "symlinks";
-
- /** The "streamFileThreshold" key */
- public static final String CONFIG_KEY_STREAM_FILE_TRESHOLD = "streamFileThreshold";
-
- /** The "remote" key */
- public static final String CONFIG_KEY_REMOTE = "remote";
-
- /** The "merge" key */
- public static final String CONFIG_KEY_MERGE = "merge";
-
- /** The "rebase" key */
- public static final String CONFIG_KEY_REBASE = "rebase";
-
- /** The "url" key */
- public static final String CONFIG_KEY_URL = "url";
-
- /** The "autosetupmerge" key */
- public static final String CONFIG_KEY_AUTOSETUPMERGE = "autosetupmerge";
-
- /** The "autosetuprebase" key */
- public static final String CONFIG_KEY_AUTOSETUPREBASE = "autosetuprebase";
-
- /**
- * The "autostash" key
- *
- * @since 3.2
- */
- public static final String CONFIG_KEY_AUTOSTASH = "autostash";
-
- /** The "name" key */
- public static final String CONFIG_KEY_NAME = "name";
-
- /** The "email" key */
- public static final String CONFIG_KEY_EMAIL = "email";
-
- /** The "false" key (used to configure {@link #CONFIG_KEY_AUTOSETUPMERGE} */
- public static final String CONFIG_KEY_FALSE = "false";
-
- /** The "true" key (used to configure {@link #CONFIG_KEY_AUTOSETUPMERGE} */
- public static final String CONFIG_KEY_TRUE = "true";
-
- /**
- * The "always" key (used to configure {@link #CONFIG_KEY_AUTOSETUPREBASE}
- * and {@link #CONFIG_KEY_AUTOSETUPMERGE}
- */
- public static final String CONFIG_KEY_ALWAYS = "always";
-
- /** The "never" key (used to configure {@link #CONFIG_KEY_AUTOSETUPREBASE} */
- public static final String CONFIG_KEY_NEVER = "never";
-
- /** The "local" key (used to configure {@link #CONFIG_KEY_AUTOSETUPREBASE} */
- public static final String CONFIG_KEY_LOCAL = "local";
-
- /** The "createchangeid" key */
- public static final String CONFIG_KEY_CREATECHANGEID = "createchangeid";
-
- /** The "defaultsourceref" key */
- public static final String CONFIG_KEY_DEFBRANCHSTARTPOINT = "defbranchstartpoint";
-
- /** The "path" key */
- public static final String CONFIG_KEY_PATH = "path";
-
- /** The "update" key */
- public static final String CONFIG_KEY_UPDATE = "update";
-
- /** The "compression" key */
- public static final String CONFIG_KEY_COMPRESSION = "compression";
-
- /** The "indexversion" key */
- public static final String CONFIG_KEY_INDEXVERSION = "indexversion";
-
- /**
- * The "hidedotfiles" key
- * @since 3.5
- */
- public static final String CONFIG_KEY_HIDEDOTFILES = "hidedotfiles";
-
- /** The "precomposeunicode" key */
- public static final String CONFIG_KEY_PRECOMPOSEUNICODE = "precomposeunicode";
-
- /** The "pruneexpire" key */
- public static final String CONFIG_KEY_PRUNEEXPIRE = "pruneexpire";
-
- /** The "mergeoptions" key */
- public static final String CONFIG_KEY_MERGEOPTIONS = "mergeoptions";
-
- /** The "ff" key */
- public static final String CONFIG_KEY_FF = "ff";
-
- /**
- * The "checkstat" key
- * @since 3.0
- */
- public static final String CONFIG_KEY_CHECKSTAT = "checkstat";
-
- /**
- * The "renamelimit" key in the "diff section"
- * @since 3.0
- */
- public static final String CONFIG_KEY_RENAMELIMIT = "renamelimit";
-
- /**
- * The "noprefix" key in the "diff section"
- * @since 3.0
- */
- public static final String CONFIG_KEY_NOPREFIX = "noprefix";
-
- /**
- * A "renamelimit" value in the "diff section"
- * @since 3.0
- */
- public static final String CONFIG_RENAMELIMIT_COPY = "copy";
-
- /**
- * A "renamelimit" value in the "diff section"
- * @since 3.0
- */
- public static final String CONFIG_RENAMELIMIT_COPIES = "copies";
-
- /**
- * The "renames" key in the "diff section"
- * @since 3.0
- */
- public static final String CONFIG_KEY_RENAMES = "renames";
-
- /**
- * The "prune" key
- *
- * @since 3.3
- */
- public static final String CONFIG_KEY_PRUNE = "prune";
- }
|