summaryrefslogtreecommitdiffstats
path: root/services/mirror/mirror.go
Commit message (Collapse)AuthorAgeFilesLines
* More `db.DefaultContext` refactor (#27265) (#27347)Giteabot2023-09-291-2/+2
| | | | | | | | | | | Backport #27265 by @JakobDev Part of #27065 This PR touches functions used in templates. As templates are not static typed, errors are harder to find, but I hope I catch it all. I think some tests from other persons do not hurt. Co-authored-by: JakobDev <jakobdev@gmx.de>
* Move `modules/mirror` to `services` (#26737)Chongyi Zheng2023-08-271-11/+10
| | | | | To solve the cyclic imports in a better way Closes #20261
* Replace `interface{}` with `any` (#25686)silverwind2023-07-041-3/+3
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Rewrite queue (#24505)wxiaoguang2023-05-081-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # ⚠️ Breaking Many deprecated queue config options are removed (actually, they should have been removed in 1.18/1.19). If you see the fatal message when starting Gitea: "Please update your app.ini to remove deprecated config options", please follow the error messages to remove these options from your app.ini. Example: ``` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options ``` Many options in `[queue]` are are dropped, including: `WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`, `BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed from app.ini. # The problem The old queue package has some legacy problems: * complexity: I doubt few people could tell how it works. * maintainability: Too many channels and mutex/cond are mixed together, too many different structs/interfaces depends each other. * stability: due to the complexity & maintainability, sometimes there are strange bugs and difficult to debug, and some code doesn't have test (indeed some code is difficult to test because a lot of things are mixed together). * general applicability: although it is called "queue", its behavior is not a well-known queue. * scalability: it doesn't seem easy to make it work with a cluster without breaking its behaviors. It came from some very old code to "avoid breaking", however, its technical debt is too heavy now. It's a good time to introduce a better "queue" package. # The new queue package It keeps using old config and concept as much as possible. * It only contains two major kinds of concepts: * The "base queue": channel, levelqueue, redis * They have the same abstraction, the same interface, and they are tested by the same testing code. * The "WokerPoolQueue", it uses the "base queue" to provide "worker pool" function, calls the "handler" to process the data in the base queue. * The new code doesn't do "PushBack" * Think about a queue with many workers, the "PushBack" can't guarantee the order for re-queued unhandled items, so in new code it just does "normal push" * The new code doesn't do "pause/resume" * The "pause/resume" was designed to handle some handler's failure: eg: document indexer (elasticsearch) is down * If a queue is paused for long time, either the producers blocks or the new items are dropped. * The new code doesn't do such "pause/resume" trick, it's not a common queue's behavior and it doesn't help much. * If there are unhandled items, the "push" function just blocks for a few seconds and then re-queue them and retry. * The new code doesn't do "worker booster" * Gitea's queue's handlers are light functions, the cost is only the go-routine, so it doesn't make sense to "boost" them. * The new code only use "max worker number" to limit the concurrent workers. * The new "Push" never blocks forever * Instead of creating more and more blocking goroutines, return an error is more friendly to the server and to the end user. There are more details in code comments: eg: the "Flush" problem, the strange "code.index" hanging problem, the "immediate" queue problem. Almost ready for review. TODO: * [x] add some necessary comments during review * [x] add some more tests if necessary * [x] update documents and config options * [x] test max worker / active worker * [x] re-run the CI tasks to see whether any test is flaky * [x] improve the `handleOldLengthConfiguration` to provide more friendly messages * [x] fine tune default config values (eg: length?) ## Code coverage: ![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-271-2/+1
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Add new API endpoints for push mirrors management (#19841)Mohamed Sekour2022-07-301-1/+1
| | | | | | | | | | | - Add a new push mirror to specific repository - Sync now ( send all the changes to the configured push mirrors ) - Get list of all push mirrors of a repository - Get a push mirror by ID - Delete push mirror by ID Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com> Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Implement sync push mirror on commit (#19411)Chongyi Zheng2022-07-081-72/+15
| | | | | | | | | Support synchronizing with the push mirrors whenever new commits are pushed or synced from pull mirror. Related Issues: #18220 Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Always try to fetch repo for mirrors (#19975)Gusted2022-06-151-2/+2
| | | | | | | - Always give a best-effort to fetching the repositories, if even that fails indeed give a disconnected mirror found error. - *Partially* resolves #19928 Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor mirror code & fix `StartToMirror` (#18904)Gusted2022-03-021-14/+19
| | | | | | | | | | | | | | | | * Use MirrorID instead of RepoID - Use the MirrorID as index(SQL uses `id` column not the `repo_id`). Passes the Mirror ID's into the Sync functions. * Check for MirrorID == 0 * Fix `StartToMirror` + refactor * Update services/mirror/mirror.go Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve mirror iterator (#18928)Lunny Xiao2022-02-281-16/+14
| | | | | * Improve mirror iterator * fix test
* Restart zero worker if there is still work to do (#18658)zeripath2022-02-081-10/+18
| | | | | | | | | | | | | | * Restart zero worker if there is still work to do It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push. Fix #18607 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Pause queues (#15928)zeripath2022-01-221-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Start adding mechanism to return unhandled data Signed-off-by: Andrew Thornton <art27@cantab.net> * Create pushback interface Signed-off-by: Andrew Thornton <art27@cantab.net> * Add Pausable interface to WorkerPool and Manager Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and PushBack for the bytefifos Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues Signed-off-by: Andrew Thornton <art27@cantab.net> * Wire in UI for pausing Signed-off-by: Andrew Thornton <art27@cantab.net> * add testcases and fix a few issues Signed-off-by: Andrew Thornton <art27@cantab.net> * fix build Signed-off-by: Andrew Thornton <art27@cantab.net> * prevent "race" in the test Signed-off-by: Andrew Thornton <art27@cantab.net> * fix jsoniter mismerge Signed-off-by: Andrew Thornton <art27@cantab.net> * fix conflicts Signed-off-by: Andrew Thornton <art27@cantab.net> * fix format Signed-off-by: Andrew Thornton <art27@cantab.net> * Add warnings for no worker configurations and prevent data-loss with redis/levelqueue Signed-off-by: Andrew Thornton <art27@cantab.net> * Use StopTimer Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-101-5/+5
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Add `PULL_LIMIT` and `PUSH_LIMIT` to cron.update_mirror task (#17568)zeripath2021-11-221-10/+42
|
* Make the Mirror Queue a queue (#17326)zeripath2021-10-171-33/+65
| | | | | | | | | Convert the old mirror syncing queue to the more modern queue format. Fix a bug in the from the repo-archive queue PR - the assumption was made that uniqueness could be enforced with by checking equality in a map in channel unique queues - however this only works for primitive types - which was the initial intention but is an imperfect. This is fixed by marshalling the data and placing the martialled data in the unique map instead. The documentation is also updated to add information about the deprecated configuration values. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Make mirror feature more configurable (#16957)65432021-09-071-0/+13
| | | | | | | | | | | Rename`[repository]` `DISABLE_MIRRORS` to `[mirror]` `DISABLE_NEW_PULL` and add `ENABLED` and `DISABLE_NEW_PUSH` with the below meanings: - `ENABLED`: **true**: Enables the mirror functionality. Set to **false** to disable all mirrors. - `DISABLE_NEW_PULL`: **false**: Disable the creation of **new** mirrors. Pre-existing mirrors remain valid. - `DISABLE_NEW_PUSH`: **false**: Disable the creation of **new** push mirrors. Pre-existing mirrors remain valid. Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: delvh <dev.lh@web.de>
* Add push to remote mirror repository (#15157)KN4CK3R2021-06-141-533/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added push mirror model. * Integrated push mirror into queue. * Moved methods into own file. * Added basic implementation. * Mirror wiki too. * Removed duplicated method. * Get url for different remotes. * Added migration. * Unified remote url access. * Add/Remove push mirror remotes. * Prevent hangs with missing credentials. * Moved code between files. * Changed sanitizer interface. * Added push mirror backend methods. * Only update the mirror remote. * Limit refs on push. * Added UI part. * Added missing table. * Delete mirror if repository gets removed. * Changed signature. Handle object errors. * Added upload method. * Added "upload" unit tests. * Added transfer adapter unit tests. * Send correct headers. * Added pushing of LFS objects. * Added more logging. * Simpler body handling. * Process files in batches to reduce HTTP calls. * Added created timestamp. * Fixed invalid column name. * Changed name to prevent xorm auto setting. * Remove table header im empty. * Strip exit code from error message. * Added docs page about mirroring. * Fixed date. * Fixed merge errors. * Moved test to integrations. * Added push mirror test. * Added test.
* Fixed sanitize errors. (#15240)KN4CK3R2021-06-131-15/+7
| | | | | Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Add LFS Migration and Mirror (#14726)KN4CK3R2021-04-081-6/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* [API] Add pagination to ListBranches (#14524)65432021-02-031-1/+1
| | | | | | | | | | | | | | | | | | * make PaginateUserSlice generic -> PaginateSlice * Add pagination to ListBranches * add skip, limit to Repository.GetBranches() * Move routers/api/v1/utils/utils PaginateSlice -> modules/util/paginate.go * repo_module.GetBranches paginate * fix & rename & more logging * better description Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: a1012112796 <1012112796@qq.com>
* Check for 'main' as potential default branch name (#14193)Chester Liu2020-12-301-0/+4
|
* migrate from com.* to alternatives (#14103)65432020-12-251-4/+3
| | | | | | | | | | | | | | | * remove github.com/unknwon/com from models * dont use "com.ToStr()" * replace "com.ToStr" with "fmt.Sprint" where its easy to do * more refactor * fix test * just "proxy" Copy func for now * as per @lunny
* Fix branch/tag notifications in mirror sync (#13855)manuelluis2020-12-051-0/+20
| | | | | | | | * Fix branch/tag notifications in mirror sync * Fix gofmt Co-authored-by: Gitea <gitea@fake.local> Co-authored-by: Lauris BH <lauris@nix.lv>
* Refactor: Move PushUpdateOptions (#13363)Lunny Xiao2020-10-301-1/+5
| | | Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
* Update Mirror IsEmpty status on synchronize (#13185)zeripath2020-10-251-2/+67
| | | | | | | | Fix #9630 Fix #13183 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* go-version constraints ignore pre-releases (#13234)zeripath2020-10-211-1/+1
| | | | | | | | | Go-version constraints ignore pre-releases. Rather than change the library further this PR simply changes the git version comparison to use simple version compare ignoring the issue of pre-releases. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Mirror: Update DB on Address-Update too (#12964)65432020-09-281-3/+5
| | | | | * Mirror: Update DB on Address-Update too * new name for function to better describe
* [Vendor] Switch go-version lib (#12719)65432020-09-051-3/+2
| | | | | | | | | | | | | | | | | | | | | * vendor: switch from "mcuadros/go-version" to "hashicorp/go-version" * Adapt P1 * simplify * fix lint * adapt * fix lint & rm old code * no deadlock * rm RWMutex and check GoVersion only 1-time * Copyright header Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Support Force-update in Mirror and improve Tracing in mirror (#12242)zeripath2020-08-201-1/+33
| | | | | | | | | | | | | | | | | * Remove double indirect in NewColoredIDValue Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle forced-update in mirror.go Signed-off-by: Andrew Thornton <art27@cantab.net> * Add tracing Signed-off-by: Andrew Thornton <art27@cantab.net> * As per @lafriks Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update the wiki repository remote origin while update the mirror ↵yisiliang2020-07-051-1/+19
| | | | | | | | | | | | | | repository's Clone From URL (#12053) * update the wiki repository remote origin #12050 * wikiRemoteURL is under repo_module * export WikiRemoteURL func * remove redundant space and empty line Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactor Cron and merge dashboard tasks (#10745)zeripath2020-05-161-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Refactor Cron and merge dashboard tasks * Merge Cron and Dashboard tasks * Make every cron task report a system notice on completion * Refactor the creation of these tasks * Ensure that execution counts of tasks is correct * Allow cron tasks to be started from the cron page * golangci-lint fixes * Enforce that only one task with the same name can be registered Signed-off-by: Andrew Thornton <art27@cantab.net> * fix name check Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @guillep2k * as per @lafriks Signed-off-by: Andrew Thornton <art27@cantab.net> * Add git.CommandContext variants Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Handle panics that percolate up to the graceful module (#11291)zeripath2020-05-141-0/+8
| | | | | | | | | | | | | | | | | | * Handle panics in graceful goroutines Adds a some deferred functions to handle panics in graceful goroutines Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle panic in webhook.Deliver Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle panic in mirror.syncMirror Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Move newbranch to standalone package (#9627)Lunny Xiao2020-01-141-6/+5
| | | | | | | | | | | | * Move newbranch to standalone package * move branch functions to modules to avoid dependencies cycles * fix tests * fix lint * fix lint
* Move create/fork repository from models to modules/repository (#9489)Lunny Xiao2020-01-121-1/+1
| | | | | | | | | | | | | | | | | | | | * Move create/fork repository from models to modules/repository * fix wrong reference * fix test * fix test * fix lint * Fix DBContext * remove duplicated TestMain * fix lint * fix conflicts
* Move push commits from models to modules/repository (#9370)Lunny Xiao2020-01-101-1/+1
| | | | | | | | | | | | | | | | * Move push commits from models to modules/repository * fix test * fix test * fix test * fix test * fix test Co-authored-by: zeripath <art27@cantab.net>
* Use limited commits (#9572)John Olheiser2020-01-011-1/+1
| | | Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Graceful: Xorm, RepoIndexer, Cron and Others (#9282)zeripath2019-12-151-10/+22
| | | | | | | | | | | | * Change graceful to use a singleton obtained through GetManager instead of a global. * Graceful: Make TestPullRequests shutdownable * Graceful: Make the cron tasks graceful * Graceful: AddTestPullRequest run in graceful ctx * Graceful: SyncMirrors shutdown * Graceful: SetDefaultContext for Xorm to be HammerContext * Avoid starting graceful for migrate commands and checkout * Graceful: DeliverHooks now can be shutdown * Fix multiple syncing errors in modules/sync/UniqueQueue & Make UniqueQueue closable * Begin the process of making the repo indexer shutdown gracefully
* Move some repository methods from models to modules/repository (#9353)Lunny Xiao2019-12-141-1/+2
| | | | | | * Move some repository methods from models to modules/repository * fix test
* Graceful: Cancel Process on monitor pages & HammerTime (#9213)zeripath2019-11-301-17/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Graceful: Create callbacks to with contexts * Graceful: Say when Gitea is completely finished * Graceful: Git and Process within HammerTime Force all git commands to terminate at HammerTime Force all process commands to terminate at HammerTime Move almost all git processes to run as git Commands * Graceful: Always Hammer after Shutdown * ProcessManager: Add cancel functionality * Fix tests * Make sure that process.Manager.Kill() cancels * Make threadsafe access to Processes and remove own unused Kill * Remove cmd from the process manager as it is no longer used * the default context is the correct context * get rid of double till
* Move mirror sync actions to notification (#9022)Lunny Xiao2019-11-241-14/+13
| | | | | | * Move mirror sync actions to notification * fix lint
* Add Close() method to gogitRepository (#8901)zeripath2019-11-131-72/+80
| | | | | | | | | In investigating #7947 it has become clear that the storage component of go-git repositories needs closing. This PR adds this Close function and adds the Close functions as necessary. In TransferOwnership the ctx.Repo.GitRepo is closed if it is open to help prevent the risk of multiple open files. Fixes #7947
* Move sync mirror actions to mirror service package (#8518)Lunny Xiao2019-10-161-6/+6
| | | | | | | | | | * move sync mirror actions to mirror service * fix go.mod * fix lint * fix lint
* Restore functionality for early gits (#7775)zeripath2019-10-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | * Change tests to make it possible to run TestGit with 1.7.2 * Make merge run on 1.7.2 * Fix tracking and staging branch name problem * Ensure that git 1.7.2 works on tests * ensure that there is no chance for conflicts * Fix-up missing merge issues * Final rm * Ensure LFS filters run on the tests * Do not sign commits from temp repo * Restore tracking fetch change * Apply suggestions from code review * Update modules/repofiles/temp_repo.go
* Move mirror to a standalone package from models (#7486)Lunny Xiao2019-10-011-0/+395
* move mirror to a standalone package * fix mirror address in template * fix tests * fix lint * fix comment * fix tests * fix tests * fix vendor * fix fmt * fix lint * remove wrong file submitted * fix conflict * remove unrelated changes * fix go mod * fix tests * clean go mod * make vendor work * make vendor work * fix tests * remove duplicated test
generated by cgit v1.2.3 (git 2.39.1) at 2025-03-24 02:25:21 +0000