diff options
| author | Peter Gardfjäll <peter.gardfjall.work@gmail.com> | 2022-03-31 14:30:40 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2022-03-31 14:30:40 +0200 |
| commit | e28cc79c92c851251eaaecb0584f8bd4259d290c (patch) | |
| tree | 1f73ef8a360f7a67d6cf8fad4535fd5156ae6f49 /services/auth | |
| parent | b877504b03a43dc5ed19109f1a00f08a343ebaba (diff) | |
| download | gitea-e28cc79c92c851251eaaecb0584f8bd4259d290c.tar.gz gitea-e28cc79c92c851251eaaecb0584f8bd4259d290c.zip | |
Improve sync performance for pull-mirrors (#19125)
This addresses https://github.com/go-gitea/gitea/issues/18352
It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors.
For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold:
1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date.
2. the procedure for getting _each tag_ involves a series of git operations
```bash
git show-ref --tags -- v8.2.4477
git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
```
of which the `git rev-list --count` can be particularly heavy.
This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go.
For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine:
- https://github.com/spring-projects/spring-boot (223 tags)
- before: `0m28,673s`
- after: `0m2,244s`
- https://github.com/kubernetes/kubernetes (890 tags)
- before: `8m00s`
- after: `0m8,520s`
- https://github.com/vim/vim (13954 tags)
- before: `14m20,383s`
- after: `0m35,467s`
I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.
Diffstat (limited to 'services/auth')
0 files changed, 0 insertions, 0 deletions