In a distributed setting, one can have multiple datacenters use
reftables for serving, while the ground truth for the Ref database is
administered centrally. In this setting, replication delays combined
with compaction can cause update-index ranges to overlap.

Such a setting is used at Google, and the JGit code already handles
this correctly (modulo a bugfix that applied in change I8f8215b99a).

Remove the restriction that was applied at FileReftableDatabase.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I6f9ed0fbd7fbc5220083ab808b22a909215f13a9

As reported by Sonar Lint:

Array designators should always be located on the type for better code
readability. Otherwise, developers must look both at the type and the
variable name to know whether or not a variable is an array.

Change-Id: If6b41fed3483d0992d402d8680552ab4bef89ffb
Signed-off-by: David Pursehouse <david.pursehouse@gmail.com>

The ReftableCompactor supported a byteLimit, but this is currently
unused. The FileReftableStack has a more sophisticated strategy that
amortizes compaction costs.

Rename min/maxUpdateIndex to reflogExpire{Min,Max}UpdateIndex to
reflect their purpose  more accurately.

Since reflogs are generally pruned chronologically (oldest entries are
expired first), one can only prune entries on full compaction, so they
should not be set by default.

Rephrase the function Reader#minUpdateIndex and maxUpdateIndex. These
vars are documented to affect log entries, but semantically, they are
about ref entries. Since ref entries have their timestamps
delta-compressed, it is important for the min/maxUpdateIndex values to
be coherent between different tables.

The logical timestamps for log entries do not have to be coherent in
different tables, as the timestamps of a log entry is part of the key.
For example, a table written at update index 20 may contain a tombstone
log entry at timestamp 1.

Therefore, we set ReftableWriter's min/maxUpdateIndex from the merged
tables we are compacting, rather than from the compaction settings
(which should only control reflog expiry.)

The previous behavior could drop log entries erroneously, especially
in the presence of tombstone log entries. Unfortunately, testing this
properly requires both an API for adding log tombstones, and a more
refined API for controlling automatic compaction. Hence, no test.

Change-Id: I2f4eb7866f607fddd0629809e8e61f0b9097717f
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

This helps debug stepping.

Change-Id: I020dafab4ffac75e6df0dbcde0ed4805c7867f72
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>

Change-Id: I16e32aeb325b604eb31f84db18a214f792395941
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I41a7d9934d53553b5f81f40ff9d663378676ac5d

Change-Id: I28b6a6476df185f3f98f8020408b4b120086cb0b
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

This is the format given by the Eclipse legal doc generator [1].

[1] https://www.eclipse.org/projects/tools/documentation.php?id=technology.jgit

Bug: 548298
Change-Id: I8d8cabc998ba1b083e3f0906a8d558d391ffb6c4
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>

Reftable is a binary, block-based storage format for the ref-database.
It provides several advantages over the traditional packed + loose
storage format:

 * O(1) write performance, even for deletions and transactions.

 * atomic updates to the ref database.

 * O(log N) lookup and prefix scans

 * free from restrictions imposed by the file system: it is
   case-sensitive even on case-insensitive file systems, and has
   no inherent limitations for directory/file conflicts

 * prefix compression reduces space usage for repetitive ref names,
   such as gerrit's refs/changes/xx/xxxxx format.

FileReftableDatabase is based on FileReftableStack, which does
compactions inline. This is simple, and has good median performance,
but every so often it will rewrite the entire ref database.

For testing, a FileReftableTest (mirroring RefUpdateTest) is added to
check for Reftable specific behavior. This must be done separately, as
reflogs have different semantics.

Add a reftable flavor of BatchRefUpdateTest.

Add a FileReftableStackTest to exercise compaction.

Add FileRepository#convertToReftable so existing testdata can be
reused.

CQ: 21007
Change-Id: I1837f268e91c6b446cb0155061727dbaccb714b8
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>