aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* ReadChangedPathFilter: fix Non-externalized string literal warningJörg Kubitz2023-08-281-2/+2
| | | | Change-Id: I78161a2dbc08918267bc59a1ed267c5108f5c391
* Introduce core.packedIndexGitUseStrongRefs config keyMartin Fick2023-08-2610-81/+274
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Introduce a core.packedIndexGitUseStrongRefs configuration key, which defaults to true so that the current behavior does not change. However, setting it to false allows soft references to be used for Pack indices instead of strong references so that they can be garbage collected when there is memory pressure. Pack objects can be large when associated with pack files with large object counts, and this memory is not really accounted for or tracked by the WindowCache and it can be very substantial at times, especially with many large object count projects. A particularly problematic use case is Gerrit's ls-projects command which loads very little data in the WindowCache via ByteWindows, but ends up loading and holding many entire indices in memory, sometimes even after the ByteWindows for their Pack objects have already been garbage collected since they won't get cleared until after a new ByteWindow is loaded. By using SoftReferences, single use indices can get cleared when there is memory pressure and OOMs can be easily avoided, drastically reducing the amount of memory required to perform an ls-projects on large sites with many projects and large object counts. On one of our test sites, an ls-projects command with strong index references requires more than 66GB of heap to complete successfully, with soft index references it requires less than 23GB. Change-Id: I3cb3df52f4ce1b8c554d378807218f199077d80b Signed-off-by: Martin Fick <quic_mfick@quicinc.com> Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
* Merge "DfsGarbageCollector: provide commit graph stats"Jonathan Tan2023-08-213-2/+41
|\
| * DfsGarbageCollector: provide commit graph statsJonathan Tan2023-08-173-2/+41
| | | | | | | | | | | | | | | | Provide commit graph stats in the same way that we provide reftable stats. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Change-Id: Ib80c892a26f9b552bc90f3cbe7da83b02ffebdfd
* | DfsReader: Make PackLoadListener interface visible to subclassesIvan Frade2023-08-181-4/+4
|/ | | | | | | | | | A subclass cannot implement a listener with the default access. Make the interface protected. Not public because so far only subclasses are interested in this interface. We can widen the visibility later if needed. Change-Id: I54e5c0ef1312dfe2fa660bc8fb54e2be35c0f6df
* DfsGarbageCollector: put only GC commits into the commit graphIvan Frade2023-08-162-14/+13
| | | | | | | | | | | | | GC puts all commits reachable from heads and tags into the GC pack, and commits reachable only from other refs (e.g. refs/changes) into GC_REST. The commit-graph contains all commits in GC and GC_REST. This produces too big commit graphs in some repos, beating the purpose of loading the index. Limit the commit graph to commits reachable from heads and tags (i.e. commits in the GC pack). Change-Id: I4962faea5a726d2ea3e548af0aeae370a6cc8588
* DfsReader: Expose when indices are loadedIvan Frade2023-08-035-25/+321
| | | | | | | | | | | | | | We want to measure the data used to serve a request. As a first step, we want to know how many indices are accessed during the request and their sizes. Expose an interface in DfsReader to announce when an index is loaded into the reader, i.e. when its reference is set. The interface is more flexible to implementors (what/how to collect) than the existing DfsReaderIOStats object. Change-Id: I56f7658fde1758efaf869fa779d11b533a81a0a7
* Merge branch 'stable-6.7'Matthias Sohn2023-08-034-6/+35
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.7: Update to Tycho 4.0.1 Prepare 6.7.0-SNAPSHOT builds JGit v6.7.0.202308011830-m2 Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: I197d6116a4b5badc58b57d309982d2f6b3e45825
| * Merge branch 'stable-6.6' into stable-6.7Matthias Sohn2023-08-034-6/+35
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.6: Update to Tycho 4.0.1 Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: I7294c21748897eb3f94eeffbda944b62e3206c0d
| | * Merge branch 'stable-6.5' into stable-6.6Matthias Sohn2023-08-033-6/+35
| | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.5: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: Id2e49252a9dc268210c9439848e77604885371aa
| | | * Merge branch 'stable-6.4' into stable-6.5Matthias Sohn2023-08-033-6/+35
| | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.4: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: Idb6dd6160e023673e3650653a15f6b1c540de96e
| | | | * Merge branch 'stable-6.3' into stable-6.4Matthias Sohn2023-08-033-6/+35
| | | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.3: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: I0bccc36d9cc9a36f1be9b1562df35ce3a0e95eee
| | | | | * Merge branch 'stable-6.2' into stable-6.3Matthias Sohn2023-08-033-6/+35
| | | | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.2: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: I589ed444b5cbfc5b073cac91323e2cc97ab98087
| | | | | | * Merge branch 'stable-6.1' into stable-6.2Matthias Sohn2023-08-033-6/+35
| | | | | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.1: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: I5b16c3b613a95b7f28c8f6ac0b20c4c593759cea
| | | | | | | * Merge branch 'stable-6.0' into stable-6.1Matthias Sohn2023-08-033-6/+35
| | | | | | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-6.0: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: Ib08037f6055dac1776e38cfb4ff8c88a50ad3e60
| | | | | | | | * Merge branch 'stable-5.13' into stable-6.0Matthias Sohn2023-08-033-6/+35
| | | | | | | | |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * stable-5.13: Add verification in GcKeepFilesTest that bitmaps are generated Express the explicit intention of creating bitmaps in GC GC: prune all packfiles after the loosen phase Prepare 5.13.3-SNAPSHOT builds JGit v5.13.2.202306221912-r Change-Id: I1f50995d9d9c592ec0e02a04e0e409440b49f9f3
| | | | | | | | | * Add verification in GcKeepFilesTest that bitmaps are generatedLuca Milanesio2023-07-051-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The packfiles with the .keep extensions are meant to prevent a packfile from being processed or removed during GC. From the point of view of the GC process then, the associated packfile should be completely transparent: - it should not included in the repacked file - it should not pruned - its objects should be left untouched, even if unreachable - the GC process, including the bitmap generation should continue as usual, as the the packfiles with .keep file did not exist Add one explicit test for making sure that the management of .keep file is also transparent to the generation of bitmaps, which are still generated if a .keep file exists. Bug: 582039 Change-Id: I14f6adc3f961c606fbc617e51ea6ed6e2ef8604f
| | | | | | | | | * Express the explicit intention of creating bitmaps in GCLuca Milanesio2023-07-052-5/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add an explicit flag to PackWriter for allowing the GC.repack() phase to explicitly generate bitmaps only for the heads packfile and not for the others. Previously the bitmap generation was conditioned to the presence of object ids exclusion from the PackWriter. The introduction of the bitmap generation in the PackWriter done in Icdb0cdd66 has accidentally made the .keep files not completely transparent, because their presence have disabled the generation of the bitmap index, even if the generation of bitmaps is enabled. This bug has been an accidental consequence of the intention of the bitmap generator to avoid generating bitmaps for the non-heads packfile, however the implementation done by Colby decided to use the excludeInPacks variable (see [1]) which is unfortunately also used for excluding the packfiles having an associated .keep file (see [2]). [1] https://git.eclipse.org/r/c/jgit/jgit/+/7940/18/org.eclipse.jgit/src/org/eclipse/jgit/storage/pack/PackWriter.java#1617 [2] https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/dafcb8f6db82b899c917832768f1c240d273190c/org.eclipse.jgit/src/org/eclipse/jgit/storage/file/GC.java#506 Bug: 582039 Change-Id: Id722e68d9ff4ac24e73bf765ab11017586b6766e
| | | | | | | | | * GC: prune all packfiles after the loosen phaseLuca Milanesio2023-07-051-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When loosening the objects inside the packfiles to be pruned, make sure that the packfile list is stable and prune all the files after the loosening is done. This prevents a series of exceptions previously thrown when loosening the packfiles, due to the too early pruning of the packfiles that were still in the pack list. Bug: 581532 Change-Id: I776776e2e083f1fa749d53f965bf50f919823b4f
| | | | | | | | | * Prepare 5.13.3-SNAPSHOT buildsMatthias Sohn2023-06-2290-503/+561
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Change-Id: I02b9388c8bc1c266bb29b4502504d137dd42142f
| | | | | | | | | * JGit v5.13.2.202306221912-rv5.13.2.202306221912-rMatthias Sohn2023-06-2288-121/+121
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Change-Id: Id0ee779fba85a6d5557f6319969adb2c74feebcf
| | * | | | | | | | Update to Tycho 4.0.1Matthias Sohn2023-08-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Tycho 4.0.0-SNAPSHOT is no longer available and it's a bad practice to depend on any snapshot version (we had to since this was the only way to get gpg signing to work in time for releasing 6.6.0). Change-Id: I1d4af5f69965b4cad50b379fd81f6f442b38c8d0
| * | | | | | | | | Prepare 6.7.0-SNAPSHOT buildsMatthias Sohn2023-08-0391-125/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Change-Id: I936d2d9106a1e3b7a98ec89fec8ae8a92ec765f2
| * | | | | | | | | JGit v6.7.0.202308011830-m2v6.7.0.202308011830-m2Matthias Sohn2023-08-0291-125/+125
|/ / / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Change-Id: I255a979e9f48f60a251ef7b74ced3f720f012706
* | | | | | | | | Add missing @since tagsMatthias Sohn2023-08-022-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was missed in c353645a09b6 Change-Id: I4ae5b13bd7bfd09c113d91ece727a26706660826
* | | | | | | | | Merge "Merge: Add diff3 style merge conflict formatter."Han-Wen NIenhuys2023-08-015-48/+258
|\ \ \ \ \ \ \ \ \
| * | | | | | | | | Merge: Add diff3 style merge conflict formatter.Haamed Gheibi2023-07-315-48/+258
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add base section to the merge conflict hunks. Bug: 442284 Change-Id: I977b43e7dd8119d6b72d11f09c4e8ec241750383
* | | | | | | | | | Fix errorprone warning about precedenceHan-Wen Nienhuys2023-08-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The condition looks suspicious, as in case of (hasElement==null && hasNext()) the check will generate a NPE Change-Id: I267f9df6746393d72f5102bd5271441422550968
* | | | | | | | | | Move footer-line parsing methods from RevCommit to FooterLineNitzan Gur-Furman2023-08-014-163/+228
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows extracting footers from a messages not associated with a commit. The public API of RevCommit is kept intact. Change-Id: I5809c23df7b7d49641a4be3a26d6f987d3d57c9b Bug: Google b/287891316
* | | | | | | | | | Merge changes I8c60d970,I09bdd4b8,I87ff3933Jonathan Tan2023-07-2612-14/+198
|\ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * changes: Pack: open reverse index from file if present PackReverseIndex: open file if present otherwise compute PackReverseIndex: verify checksums
| * | | | | | | | | | Pack: open reverse index from file if presentAnna Papitto2023-07-181-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The reverse index for a pack is still always computed if needed, which is slower than parsing it from a file. Supply the file path where the reverse index file might be so that it parsed instead of computed if the file is present. Change-Id: I8c60d970fd587341dfb2763fb87f1c586279f2a5 Signed-off-by: Anna Papitto <annapapitto@google.com>
| * | | | | | | | | | PackReverseIndex: open file if present otherwise computeAnna Papitto2023-07-184-0/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The existing #read and #computeFromIndex static builder methods require the caller to choose whether to supply an input stream of a reverse index file or a forward index to compute the reverse index from, which is slower. Allow a caller to provide a file path where the pack's reverse index might be and the pack's forward index index and simply get some reverse index instance back. Prefer opening and parsing the file if it is present, to save computation time. Otherwise, fall back onto computing the reverse index from the pack's forward index. Change-Id: I09bdd4b813ad62c86add586417b2ab86e9331aec Signed-off-by: Anna Papitto <annapapitto@google.com>
| * | | | | | | | | | PackReverseIndex: verify checksumsAnna Papitto2023-07-188-12/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new version 1 file-based reverse index has a footer with the checksum of the corresponding pack file and a checksum of its own contents. The initial implementation doesn't enforce that the pack checksum matches the checksum found in the forward index nor that the self checksum matches the contents of the file just read in. Offer a method for reverse index users to verify the checksums in a way appropriate to the version being used. For the pre-existing computed version, always succeed since it is not based on a file so there is no possibility of corruption. Check for corruption of the file itself during parsing the checksum footer, by comparing the self checksum with the digest of the file contents read. Change-Id: I87ff3933cf1afa76663350400b616695e4966cb6 Signed-off-by: Anna Papitto <annapapitto@google.com>
* | | | | | | | | | | RevCommitCG: Read changed-path-filters directly from commit graphIvan Frade2023-07-265-28/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | RevCommit and RevCommitCG were designed like "pointers" to data that load the content on demand, not on construction. This saves memory. Make the loading of changed-path-filter follow the same pattern. The ChangedPathFilters are only pointers to locations in the commit-graph (not the actual data), so the memory saving is not that big, but this is more consistent with the rest of the API. As 6.7 is not released, we can still change the RevWalk API. Change-Id: Id4186ea744b8a2418d0329facae69f785108d356
* | | | | | | | | | | Update commons-codec to 1.16.0Matthias Sohn2023-07-2515-28/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Change-Id: I64617b17a168da1966b93c283c150d549477f3e1
* | | | | | | | | | | Add missing @since tags for new API methodsMatthias Sohn2023-07-251-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was missed in d3b40e72acd3. Change-Id: I6e90157c6be34ae6618e246b02cf77631c8e9732
* | | | | | | | | | | Add missing package import needed to use MurmurHash3Matthias Sohn2023-07-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was missed in 49beb5ae519e and broke the OSGi classpath. Change-Id: I08a307e9e3aade4ed8a5b5e2cc5e5d03c57dfa56
* | | | | | | | | | | Merge "Identify a commit that generates a diffEntry on a rename Event."Jonathan Tan2023-07-253-3/+51
|\ \ \ \ \ \ \ \ \ \ \ | |_|/ / / / / / / / / |/| | | | | | | | | |
| * | | | | | | | | | Identify a commit that generates a diffEntry on a rename Event.Ronald Bhuleskar2023-07-243-3/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using FollowFilter's rename callback, a callback is generated with the diff. The caller that is interested in the renames knows what the diff's are but have no idea what commit generated that diff. This will allow FollowFilter's rename callback to track diffEntry for a given commit. Change-Id: If1e63ccd19fdcb9c58c59137110fe24e0ce023d2
* | | | | | | | | | | Merge changes I60a92463,Ic3b68220Jonathan Tan2023-07-2111-59/+747
|\ \ \ \ \ \ \ \ \ \ \ | |/ / / / / / / / / / |/| / / / / / / / / / | |/ / / / / / / / / | | | | | | | | | | | | | | | | | | | | * changes: PackReverseIndexV1: reverse index parsed from version 1 file ComputedPackReverseIndex: Clarify custom bucket sort algorithm
| * | | | | | | | | PackReverseIndexV1: reverse index parsed from version 1 fileAnna Papitto2023-07-1810-12/+676
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The reverse index for a pack is used to quickly find an object's position in the pack's forward index based on that object's pack offset. It is currently computed from the forward index by sorting the index entries by the corresponding pack offset. This computation uses insertion sort, which has an average runtime of O(n^2). Cgit persists a pack reverse index file to avoid recomputing the reverse index ordering. Instead they write a file with format https://git-scm.com/docs/pack-format#_pack_rev_files_have_the_format which can later be read and parsed into the in-memory reverse index each time it is needed. PackReverseIndexV1 parses a reverse index file with the official version 1 format into an in-memory representation of the reverse index which implements methods to find an object's forward index position from its offset in logorithmic time. Change-Id: I60a92463fbd6a8cc9c1c7451df1c14d0a21a0f64 Signed-off-by: Anna Papitto <annapapitto@google.com>
| * | | | | | | | | ComputedPackReverseIndex: Clarify custom bucket sort algorithmAnna Papitto2023-07-181-47/+71
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ComputedPackReverseIndex uses a custom sorting algorithm, based on bucket sort with insertion sort but with the data managed as a linked list across two int arrays. This custom algorithm relies on the set of values being sorted being exactly 0, ..., n-1; so that they can serve a second purpose of being indexes into a second equally sized list. This custom algorithm was introduced ~10 years ago in https://eclipse.googlesource.com/jgit/jgit/+/6cc532a43cf28403cb623d3df8600a2542a40a43. The original author is no longer an active contributor, so it is valuable for the code to be readable, especially as there is currently active work on reverse indexes. Rename variables and add comments to clarify the algorithm and improve readability. There are no functional changes to the algorithm. Change-Id: Ic3b682203f20e06f9f865f81259e034230f9720a Signed-off-by: Anna Papitto <annapapitto@google.com>
* | | | | | | | | | CommitGraphWriter: add option for writing/using bloom filtersRonald Bhuleskar2023-07-189-20/+238
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, bloom filters are written and used without any way to turn them off. Add a per-repo config variable to control whether bloom filters are written. As for reading, add a JGit option to control this. (A JGit option is used instead of a per-repo config variable as there is usually no reason not to use the bloom filters if they are present, but a global control to disable them is useful if there turns out to be an issue with the implementation of bloom filters.) The config that controls reading is the same as C Git, but the config for writing is not: C Git has no config to control writing, but whether bloom filters are written depends on whether bloom filters are already present and what arguments are passed to "git commit-graph write". See the manpage of "git commit-graph" for more information. Change-Id: I1b7b25340387673506252b9260b22bfe147bde58
* | | | | | | | | | CommitGraphWriter: reuse changed path filtersJonathan Tan2023-07-184-14/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Teach CommitGraphWriter to reuse changed path filters that have been read from the commit graph file whenever possible. Change-Id: I1acbfa1613ca7198386a49209028886af360ddb6 Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
* | | | | | | | | | RevWalk: use changed path filtersJonathan Tan2023-07-188-13/+211
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Teach RevWalk, TreeRevFilter, PathFilter, and FollowFilter to use changed path filters, whenever available, to speed revision walks by skipping commits that fail the changed path filter. This work is based on earlier work by Kyle Zhao (I441be984b609669cff77617ecfc838b080ce0816). Change-Id: I7396f70241e571c63aabe337f6de1b8b9800f7ed Signed-off-by: kylezhao <kylezhao@tencent.com> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
* | | | | | | | | | CommitGraphLoader: read changed-path filtersJonathan Tan2023-07-187-2/+186
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As described in the parent commit, add support for reading the BIDX and BDAT chunks of the commit graph file, as described in man gitformat- commit-graph(5). This work is based on earlier work by Kyle Zhao (I160f6b022afaa842c331fb9a086974e49dced7b2). Change-Id: I82e02e6a3a3b758e6bf9d7bbd2198f0ffe3a331b Signed-off-by: kylezhao <kylezhao@tencent.com> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
* | | | | | | | | | CommitGraphWriter: write changed-path filtersJonathan Tan2023-07-1811-13/+570
|/ / / / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add support for writing the BIDX and BDAT chunks of the commit graph file, as described in man gitformat-commit-graph(5). The ability to read such chunks will be added in a subsequent commit. This work is based on earlier work by Kyle Zhao (Ib863782af209f26381e3ca0a2c119b99e84b679c). Change-Id: Ic18e6f0eeec7da1e1ff31751aabda5e6952dbe6e Signed-off-by: kylezhao <kylezhao@tencent.com> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
* | | | | | | | | Merge "ssh: PKCS#11 support"Matthias Sohn2023-07-1712-33/+974
|\ \ \ \ \ \ \ \ \
| * | | | | | | | | ssh: PKCS#11 supportThomas Wolf2023-07-1712-33/+974
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Support PKCS#11 HSMs (like YubiKey PIV) for SSH authentication. Use the SunPKCS11 provider as described at [1]. This provider dynamically loads the library from the PKCS11Provider SSH configuration and creates a Java KeyStore with that provider. A Java CallbackHandler is needed to feed PIN prompts from the KeyStore into the JGit CredentialsProvider framework. Because the JGit CredentialsProvider may be specific to a SSH session but the PKCS11Provider may be used by several sessions, the CallbackHandler needs to be configurable per session. PIN prompts respect the NumberOfPasswordPrompts SSH configuration. As long as the library asks only for a PIN, we use the KeyPasswordProvider to prompt for it. This gives automatic integration in Eclipse with the Eclipse secure storage, so a user has even the option to store the PIN there. (Eclipse will then ask for the secure storage master password on first access, so the usefulness of this is debatable.) By default the provider uses the first PKCS#11 token (slot list index zero). This can be overridden by a non-standard PKCS11SlotListIndex ssh configuration entry. (For OpenSSH interoperability, also set "IgnoreUnknown PKCS11SlotListIndex" in the SSH config file then.) Once loaded, the provider and its shared library and the keys contained remain available until the application exits. Manually tested using SoftHSM. See file manual_tests.txt. Kudos to Christopher Lamb for additional manual testing with a real YubiKey, also on Windows.[2] [1] https://docs.oracle.com/en/java/javase/11/security/pkcs11-reference-guide1.html [2] https://www.eclipse.org/forums/index.php/t/1113295/ Change-Id: I544c97e1e24d05e28a9f0e803fd4b9151a76ed11 Signed-off-by: Thomas Wolf <twolf@apache.org>
* | | | | | | | | | GC: Remove handling of extra pack for RefTreeMatthias Sohn2023-07-171-7/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | RefTree was packed in its own packfile, see Icbb735be8fa91ccbf0708ca3a219b364e11a6b83. RefTree was deleted in Ia3da7f2b82d9e365cec2ccf9397cbc47439cd150, since it was experimental and never used productively. This change missed to remove the extra pack handling for RefTree. Change-Id: I8c0d0a66440c331c3d03d0e07d5629682af2a7a9
generated by cgit v1.2.3 (git 2.39.1) at 2025-04-03 06:13:40 +0000