Introduce core.packedIndexGitUseStrongRefs config key

Introduce a core.packedIndexGitUseStrongRefs configuration key, which
defaults to true so that the current behavior does not change. However,
setting it to false allows soft references to be used for Pack indices
instead of strong references so that they can be garbage collected when
there is memory pressure.

Pack objects can be large when associated with pack files with large
object counts, and this memory is not really accounted for or tracked by
the WindowCache and it can be very substantial at times, especially with
many large object count projects. A particularly problematic use case is
Gerrit's ls-projects command which loads very little data in the
WindowCache via ByteWindows, but ends up loading and holding many entire
indices in memory, sometimes even after the ByteWindows for their Pack
objects have already been garbage collected since they won't get cleared
until after a new ByteWindow is loaded. By using SoftReferences, single
use indices can get cleared when there is memory pressure and OOMs can
be easily avoided, drastically reducing the amount of memory required to
perform an ls-projects on large sites with many projects and large
object counts.

On one of our test sites, an ls-projects command with strong index
references requires more than 66GB of heap to complete successfully,
with soft index references it requires less than 23GB.

Change-Id: I3cb3df52f4ce1b8c554d378807218f199077d80b
Signed-off-by: Martin Fick <quic_mfick@quicinc.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

1 files changed, 1 insertions, 0 deletions

diff --git a/Documentation/config-options.md b/Documentation/config-options.md
index 3495813e76..5651001bce 100644
--- a/Documentation/config-options.md
+++ b/Documentation/config-options.md
@@ -38,6 +38,7 @@ For details on native git options see also the official [git config documentatio
 | `core.packedGitMmap` | `false` | ✅ | Whether to use Java NIO virtual memory mapping for JGit buffer cache. When set to `true` enables use of Java NIO virtual memory mapping for cache windows, `false` reads entire window into a `byte[]` with standard read calls. `true` is experimental and may cause instabilities and crashes since Java doesn't support explicit unmapping of file regions mapped to virtual memory. |
 | `core.packedGitOpenFiles` | `128` | ⃞ | Maximum number of streams to open at a time. Open packs count against the process limits. |
 | `core.packedGitUseStrongRefs` | `false` | ⃞ | Whether the window cache should use strong references (`true`) or SoftReferences (`false`). When `false` the JVM will drop data cached in the JGit block cache when heap usage comes close to the maximum heap size. |
+| `core.packedIndexGitUseStrongRefs` | `true` | ⃞ | Whether pack indices should use strong references (`true`) or SoftReferences (`false`). When `false` the JVM will drop data cached in the JGit pack indices when heap usage comes close to the maximum heap size. |
 | `core.packedGitWindowSize` | `8 kiB` | ✅ | Number of bytes of a pack file to load into memory in a single read operation. This is the "page size" of the JGit buffer cache, used for all pack access operations. All disk IO occurs as single window reads. Setting this too large may cause the process to load more data than is required; setting this too small may increase the frequency of read() system calls. |
 | `core.precomposeUnicode` | `true` on Mac OS | ✅ | MacOS only. When `true`, JGit reverts the unicode decomposition of filenames done by Mac OS. |
 | `core.quotePath` | `true` | ✅ | Commands that output paths (e.g. ls-files, diff), will quote "unusual" characters in the pathname by enclosing the pathname in double-quotes and escaping those characters with backslashes in the same way C escapes control characters (e.g. `\t` for TAB, `\n` for LF, `\\` for backslash) or bytes with values larger than `0x80` (e.g. octal `\302\265` for "micro" in UTF-8). |