| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\
| |
| |
| |
| |
| |
| | |
* stable-7.3:
Lock reftable auto-refresh to ensure consistency
Change-Id: I43df38f98f0b03dec2e64d4c52257ddc3affd64f
|
| | |\
| | |
| | |
| | |
| | |
| | |
| | | |
* stable-7.2:
Lock reftable auto-refresh to ensure consistency
Change-Id: I9219f0fd606b2eb6b9a0632375934f51f1f4c547
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Ensure that reftable auto-refresh operations, clearing the database
cache and reloading the reftable stack are executed in an exclusive
critical section under lock. Previously, these steps were performed
without an exclusive critical section, creating a window where
concurrent threads could interfere with each other.
In a race condition, one thread might clear the cache and before it had
a chance of reloading the stack, another thread could repopulate the
cache with stale data, keeping a reference to the open BlockSource
channel to the underlying tables that are subsequently removed when the
first thread reloads the stack.
The above race condition resulted in attempts to access closed resources
and lead to ClosedChannelException errors.
As an example, consider the following scenario:
* T0 - Thread-1 is executing auto-refresh and it clears the database
cache
* T1 - The master branch moves forward (for any reason):
- A new refTable (`R_new`) file is created
- An existing refTable (`R_old`) file is deleted due to
auto-compaction.
* T3 - Thread-2 repopulates the database cache before Thread-1 has had a
chance to reload the refTable stack.
* T4 - Thread-1 finally reloads the refTable stack, causing the closing
of the BlockSource wrapping the removed `R_old` refTable file.
* T5 - Thread-2 attempts to read from the already-closed `R_old`
BlockSource and the `j.n.c.ClosedChannelException` is thrown
To reproduce this problem, you can run a script created to craft this
racing condition: I1e78e175cff.
While such errors during concurrent execution might be expected and
tolerable in isolation, the situation becomes more severe when the
`RepositoryCache` is involved, as is the case with Gerrit.
The `FileReftableDatabase` instance is cached within the `Repository`
object. When a `BlockSource` is closed prematurely due to this race
condition, the dangling reference remains in memory until the cached
Repository expires, which is one hour by default.
This means that, once the race occurs, the repository may be unable to
perform any ref lookups for up to an hour, effectively causing a
repository outage.
By introducing a ReentrantLock around both operations, the refresh logic
now guarantees that concurrent readers and writers maintain a consistent
view of the reftable state, eliminating the race condition.
Verified by executing the script provided at I1e78e175cff and no
exceptions are raised anymore.
Bug: jgit-130
Change-Id: I6153528a7b2695115b670bda04d4d4228c1731e1
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
PackList is an abstract class with two implementations, the default
PackListImpl (never dirty) and NO_PACKS (always dirty).
Simplify this to a class (PackList) with an special implementation for
NO_PACKS.
Change-Id: Ie97e1084417bbfa15c46bd570121a83883ed183c
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
There are no callers for markDirty(), so in NO_PACKS dirty is always
true and in the PackListImpl is always false. clearDirty() is a noop
in both cases.
Delete these methods to avoid confusions.
Change-Id: I74d02be7dc287bef27fb7b9d6588c8d8c74105a8
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This method is inneficient, as callers lookup the object in the
primary index twice: once to find it is in the pack (before calling),
another for the position (in the method). Now we have
\#getIndexedObjectSize(ctx, indexPosition) variant, so callers find the
pack with \#findIdxPosition() and reuse the idxPosition here.
Remove this variant with objectId. It doesn't do anything that the
combination above cannot do.
Change-Id: Ia913f511338b44ab5c45c55d47754d299946f03c
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Reported as warning in the mvn build.
Change-Id: I981947c5717b74ef9547322b25a7211337b32cf6
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
mvn build was warning about these messages lacking the NLS
annotation.
As the method has 3 lines with 3 string, just suppress the
warning in the whole method.
Change-Id: I0dbd4055266517edd21418dc12528776078c34cc
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
To get the indexed size of an object, we do two lookups in the primary
index. First to find the object, then to find its position in the
index. This hits performance when checking many object sizes on big
packs.
The first lookup of the object returns the index position. Reuse it
for the object size index.
Change-Id: If875d1b5d0b70f0cf4c448e0f9da09db65caac5e
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Now findPack does the same than findPackWithObject, but it also returns
the index position of the object, which we wil use in the next change.
In getObjectSize and isNotLargerThan, use "findPack" to lookup the
object. There are no more usages of findPackWithObject.
Change-Id: I9d26a1bf4ac4e8f87148be90e25d17efa4b70e69
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This is a preparatory change to avoid double lookup in the primary
index when using the object size index. If we get the index position
during the object lookup, we dont need to query the index again when
looking up the size.
Make "hasImpl" return the index position of the object in the
pack (besides leaving "last" pointing to the pack). Move the "has"
logic that checks "last" before other packs to a method we can reuse.
Change-Id: I36d2b348971a642c54da3d3727c04d0134ca9eba
|
| |/ /
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
safeGetIndexedObjectSize returns an Optional to differentiate
between "object not found in index" and "index not found". This is
wrong for two reasons: the IOException indicating "index not found" is only thrown
once (on first loading failure) and the first loading happens in
the "hasObjectSizeIndex" just before.
Rethrow the IOException in getIndexesObjectSize as
IllegalStateException (as it shouldn't happen) and return a plain long
in the call.
As a side, this saves instantiating an Optional per call and
simplifies further optimizations.
Change-Id: Ic728483f0e1ec7f98495fdbdb15dc0b24f30dbfd
|
| |\|
| |
| |
| |
| |
| |
| | |
* stable-7.2:
Fix: Close the "preserved" PackDirectory
Change-Id: If9f2cd1278aad72f6ca7cae72fbd6b9f5ec66bc8
|
| | |\
| | |
| | |
| | |
| | |
| | |
| | | |
* stable-7.1:
Fix: Close the "preserved" PackDirectory
Change-Id: I82f138f134fe09717e2e024b3c87971140f01b29
|
| | | |\
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* stable-7.0:
Fix: Close the "preserved" PackDirectory
Change-Id: Icd3f79322f8c021e18fd5c881cd9f2a406230fa8
|
| | | | |\
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-6.10:
Fix: Close the "preserved" PackDirectory
Change-Id: Ie0ecfd8178ef4e2eef6a29d46be5645648fe88f3
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This has been missing since the feature was first added in commit
6167641834e28f8ad322f8fde60866b339bfb7fe.
It's possible we could be more aggressive and close soon after
attempting to get an object from the preserved packs, but for concurrent
misses that might cause thrashing. More likely it would be safe to
attempt closing after successfully restoring a preserved pack. A follow
up change should attempt that.
Change-Id: I87d61007bcc3d03fc86bd18465ca66a2e6f697a1
|
| |\| | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-7.2:
Use the same ordering/locking in delete() as C git
Change-Id: I0b583cd218c39b1dc0726ae82da86eca58cc81eb
|
| | |\| | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-7.1:
Use the same ordering/locking in delete() as C git
Change-Id: Id52c938b041604162dca9162726bfb594e96f5d1
|
| | | |\| |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-7.0:
Use the same ordering/locking in delete() as C git
Change-Id: I2c38321ee410d9ec60481d56315710beaebd393a
|
| | | | |\|
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-6.10:
Use the same ordering/locking in delete() as C git
Change-Id: I0d06e39d06315e0b9e770bdf79164779d98f9f50
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Following the examples of cgit, lock packed-refs *before* checking for
existance of refs in it [1] and *keep the lock* until the loose ref (if
any) is removed [2]. The packed-refs lock is kept even when no
packed-refs update is required [3] so that somebody else doesn't pack a
reference that we are trying to delete.
This fixes a concurrency issue that happens on projects with a
substantial amount of refs(>~500k) where packing takes long enough for a
ref deletion to be triggered half way through it. Not locking the
packed-refs file before checking if the refs exists is not safe, as it
opens up situations where loose refs are repacked in memory and locked
on disk, but before the lock is released and packed-refs is flushed to
disk, a ref is deleted.
As packed-refs was NOT locked while checking wether a ref existed in it,
the current content on disk was read, which was about to be overwritten
and did not contain the ref about to be deleted. As the delete doesn't
see the ref in the current, on-disk, version of packed refs, it skips
processing altogether and moves on, correctly, deleting only the
associated loose ref and leaving the packed one behind.
Once the new packed-refs, containing the ref that was just deleted, was
commited to disk, the ref would come back to life.
Therefore, the packed-refs needs to be locked before checking if it
contains a ref or not in the same way the C implementation of Git does
at [1].
There are tradeoffs, though, in this decision, which will reduce the
parallelism of deleting loose refs and performing the refs repacking,
which happens very often in certain JGit implementations like Gerrit
Code Review. Before this change, repacking of refs and removal of loose
refs unrelated to the in-flight repacking was possible without involving
any locking; after this change, all loose refs removals have to wait for
the packing of refs to be completed, even though the repacking and the
refs removals were completely unrelated and their namespaces disjoint.
See more details on the test's performance results and the associated
tradeoffs in the Issue jgit-152.
NOTE: This delete ref locking logic was incorrect regardless of how the
packing of the refs is implemented. Making decisions if the pack
transaction is needed or not on an unlocked resource is racy and also
flagged as bug at [1].
[1]https://github.com/git/git/blob/master/refs/packed-backend.c#L1590
[2]https://github.com/git/git/blob/master/refs/files-backend.c#L3261
[3]https://github.com/git/git/blob/master/refs/files-backend.c#L2943
Bug: jgit-152
Change-Id: I158ec837904617c5fdf667e295ae667b2f037945
|
| | | |\| |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-7.0:
Prepare 7.0.2-SNAPSHOT builds
JGit v7.0.1.202505221510-r
Prepare 6.10.2-SNAPSHOT builds
JGit v6.10.1.202505221210-r
AmazonS3: Do not accept DOCTYPE and entities
ManifestParser: Do not accept DOCTYPE and entities
Change-Id: I4506e4bf51225000418b15bf09df3287be26242a
|
| | | | |\|
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-6.10:
Prepare 6.10.2-SNAPSHOT builds
JGit v6.10.1.202505221210-r
AmazonS3: Do not accept DOCTYPE and entities
ManifestParser: Do not accept DOCTYPE and entities
Change-Id: I699d57974d9ef2428355c59194c6becbc16828b7
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This follows OWASP recommendations in
https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html
Change-Id: I3d47debf14d95c8189d51256b4eb2ba991279452
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
These open the door for XXE attacks [1] and manifest do not need them.
[1] https://en.wikipedia.org/wiki/XML_external_entity_attack
Change-Id: Ia79971e1c34afaf287584ae4a7f71baebcb48b6a
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
to avoid unnecessary computations if the task getting
FileStoreAttributes asynchronously timed out, cancelled or failed with
another exceptions.
Change-Id: I127a3e2f3710fc5a8742c61f513576ee2d84baed
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
The reader iterates the packs looking for the best representation of
an object. It does so using the primary and reverse indexes of the
pack. This direct use of the indexes prevent the pack to be more
clever and use the multipack index in these operations.
Move finding objects in the pack and the representation to the pack
itself. Now the pack can decide how to implement it (e.g. using the
multipack index).
Change-Id: Ief0f2384ab395557c2533990f7b3c532a88d7ac9
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This method doesn't seem to be used.
Change-Id: I9b7a25aa7bd99b461bb79a0d25c61a1fd6d40e64
|
| |\| | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-7.2:
Prepare 7.2.2-SNAPSHOT builds
JGit v7.2.1.202505142326-r
AmazonS3: Do not accept DOCTYPE and entities
ManifestParser: Do not accept DOCTYPE and entities
Change-Id: I2a56d92735c003d7d61147743409a24e1539ad55
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This follows OWASP recommendations in
https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html
Change-Id: I3d47debf14d95c8189d51256b4eb2ba991279452
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
These open the door for XXE attacks [1] and manifest do not need them.
[1] https://en.wikipedia.org/wiki/XML_external_entity_attack
Change-Id: Ia79971e1c34afaf287584ae4a7f71baebcb48b6a
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
If "core.trustTablesListStat" is set to "after_open" to ensure file
attributes are refreshed on NFS refreshing only the attributes of the
tables.list file is not sufficient since then NFS may not detect new
reftable files (stored in the same refs/reftable/ directory) which may
cause FileNotFoundExceptions. Fix this by refreshing attributes of the
refs/reftable/ directory.
Change-Id: I7e07834fd7628a07ed644cb2740101a749d433bb
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
The filesystem layout of reftables is fixed in the reftable
specification hence the constructor
FileReftableStack(File tablesListFile, File reftableDir,
@Nullable Runnable onChange, Supplier<Config> configSupplier)
should not allow to pass the path of the tables.list file independently
from the refs/reftable/ directory containing both the tables.list file
and the reftable files. Hence remove the path of the tables.list file
from its argument list and instead set it inside the constructor.
Use reftableDir instead of tablesListFile.getParentFile().
Also rename FileReftableStack.stackPath to tablesListFile which is
easier to understand and remove the useless return value of
FileReftableDatabase#convertFrom which was unused and always returned
null.
Change-Id: If1efba5d49d979d99a86871996abe550422d8945
|
| |\ \ \ \ \ |
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
The #timeof method returned
- the commit time in seconds since the epoch for RevCommits
- the tagger time in milliseconds since the epoch for annotated tags
Fix this by converting commit time to milliseconds to ensure consistent
timestamp comparison of tags and commits.
Change-Id: I91621da19c234fe823788d85e4077750cb5cba1a
|
| |\ \ \ \ \ \
| |/ / / / /
|/| | | | | |
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
We turned this off to debug something else and it created
significantly bigger packs that spammed the storage. There is no
reason to allow duplicates while inserting in a pack, so lets remove
completely the option.
The setter is not used and the variable is always true.
Change-Id: I978049dce30c33b7aa650d3ddef9430cbef764fa
|
| |/ / / / /
| | | | |
| | | | |
| | | | | |
Change-Id: I0972121aaba6fd05f733d3ec62515e00c30530c7
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This code was introduced recently and the license header says Google
Inc. where it should be Google LLC.
Change-Id: If3d4f992bdcd165cfba356b4f9721c4605017dda
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
The code establishing the PackExt from the file name uses endsWith, i
think to cover for the x.<ext> and x.old-<ext> cases. This doesn't
work if we add the "midx" case, as "midx".endsWith("idx").
Check explicitely that the extension maches itself or with the old-
prefix. We rather keep the "midx" as file extension for upstream
compatibility.
Change-Id: I25d0e850b9df17ed99d776b397560774bc402d39
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
The reader offers the resolve functionality to find object ids with
certain prefix. The basic implementation iterates through packs and
calls resolve in their indexes. With multipack index, we can answer
the resolve directly for all the packs included.
Offer a resolve() method and implement it in the multipack index.
Change-Id: If5679652f149a41afe568c719ba40b291ae1b917
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
If a pack doesn't have data, its offsets are null, not empty. A pack
without objects is probably a pathological case, but I bumped into
this while writing other tests.
Check if the offsets are null (instead of empty) before trying to
write the ridx.
Change-Id: I8cadea086b302b2ead02a5a8d4e9e8adf85bc07b
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This is useful later, in GC, to write the size of the stream in the
pack description.
Change-Id: Ic3d759c1120ef2fde77b432d33c46ba1791d954b
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Errorprone pointed out that adding with int and casting to long at the
end could overflow.
Mark one of the operands as long to do the addition directly as long
and avoid the overflow risk.
Change-Id: I12449af90c84044a04e6dbd0978ff48d0ea33674
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Cosmetic changes.
Change-Id: I5f4d463cbb4e256d068369efc798c1c00f17c18c
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
We need a size estimation to store this ref in the DfsBlockCache.
Change-Id: I974099cf66a66d09320a795638abfeac506da2c1
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Reader and java API to load and query a multipack index.
Note: the `multi-pack-index.v1` file used in the tests
has been created on a local repository with the
"git multi-pack-index" command [1].
[1]: https://git-scm.com/docs/git-multi-pack-index
Change-Id: I09def882e1e77e9f22c6236b0a035a1e80bc1ace
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
DfsPackFile.PackIndexes interface doesn't cache anything and assumes
that the caller will keep the loaded references. When the DFS cache
implementation needs a primary index to create the reverse
index, it invokes its own getIndex method, skipping the reference hold
by the caller and potentially triggering reloads.
Pass the primary index as parameter in #getReverseIndex(), so the
caller can reuse its reference.
Change-Id: I1f310eecea73d8f1afc1fb94d40cd94cccdb5996
|
| |\ \ \ \ \ |
|
