summaryrefslogtreecommitdiffstats
path: root/services/mirror
Commit message (Collapse)AuthorAgeFilesLines
* Add new API endpoints for push mirrors management (#19841)Mohamed Sekour2022-07-302-3/+3
| | | | | | | | | | | - Add a new push mirror to specific repository - Sync now ( send all the changes to the configured push mirrors ) - Get list of all push mirrors of a repository - Get a push mirror by ID - Delete push mirror by ID Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com> Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Implement sync push mirror on commit (#19411)Chongyi Zheng2022-07-081-72/+15
| | | | | | | | | Support synchronizing with the push mirrors whenever new commits are pushed or synced from pull mirror. Related Issues: #18220 Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Always try to fetch repo for mirrors (#19975)Gusted2022-06-151-2/+2
| | | | | | | - Always give a best-effort to fetching the repositories, if even that fails indeed give a disconnected mirror found error. - *Partially* resolves #19928 Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Fix some mirror bugs (#18649)Lunny Xiao2022-06-112-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix some mirror bugs * Remove unnecessary code * Fix lint * rename stdard url * Allow more charactors in git ssh protocol url * improve the detection * support ipv6 for git url parse * Fix bug * Fix template * Fix bug * fix template * Fix tmpl * Fix tmpl * Fix parse ssh with interface * Rename functions name Co-authored-by: zeripath <art27@cantab.net>
* Move some repository related code into sub package (#19711)Lunny Xiao2022-06-061-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move some repository related code into sub package * Move more repository functions out of models * Fix lint * Some performance optimization for webhooks and others * some refactors * Fix lint * Fix * Update modules/repository/delete.go Co-authored-by: delvh <dev.lh@web.de> * Fix test * Merge * Fix test * Fix test * Fix test * Fix test Co-authored-by: delvh <dev.lh@web.de>
* Prevent NPE on update mirror settings (#19864)zeripath2022-06-021-1/+1
| | | | | | | | | | | | | A `repo_model.Mirror` repository field (`.Repo`) will not automatically be set, but is used without checking in mirror_pull.go:UpdateAddress. This will cause an NPE. This PR changes UpdateAddress to use the helper function GetRepository() helping prevent future NPEs but also changes modules/context/repo.go to ensure that the Mirror.Repo is set. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Move almost all functions' parameter db.Engine to context.Context (#19748)Lunny Xiao2022-05-202-4/+9
| | | | * Move almost all functions' parameter db.Engine to context.Context * remove some unnecessary wrap functions
* Remove `git.Command.Run` and `git.Command.RunInDir*` (#19280)wxiaoguang2022-04-012-14/+14
| | | | | | Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions. * before: `stdout, err := RunInDir(path)` * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
* Use a more general (and faster) method to sanitize URLs with credentials ↵wxiaoguang2022-03-312-32/+19
| | | | | (#19239) Use a more general method to sanitize URLs with credentials: Simple and intuitive / Faster / Remove all credentials in all URLs
* Make git.OpenRepository accept Context (#19260)65432022-03-302-3/+3
| | | | | * OpenRepositoryCtx -> OpenRepository * OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
* Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235)zeripath2022-03-291-0/+9
| | | | | | | | | | | | | | | | | Strangely #19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added. I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. Fix #19038 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Touch mirrors on even on fail to update (#19217)zeripath2022-03-271-0/+3
| | | | | | | | | | * Touch mirrors on even on fail to update If a mirror fails to be synchronised it should be pushed to the bottom of the queue of the awaiting mirrors to be synchronised. At present if there LIMIT number of broken mirrors they can effectively prevent all other mirrors from being synchronized as their last_updated time will remain earlier than other mirrors. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Hide sensitive content on admin panel progress monitor (#19218)Lunny Xiao2022-03-272-3/+22
| | | | | | Sanitize urls within git process descriptions. Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Use `ctx` instead of `db.DefaultContext` in some ↵wxiaoguang2022-03-221-2/+1
| | | | | | | | | | | | packages(routers/services/modules) (#19163) * Remove `db.DefaultContext` usage in routers, use `ctx` directly * Use `ctx` directly if there is one, remove some `db.DefaultContext` in `services` * Use ctx instead of db.DefaultContext for `cmd` and some `modules` packages * fix incorrect context usage
* Improve SyncMirrors logging (#19045)zeripath2022-03-101-21/+21
| | | | | | | | | Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related #19038 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Refactor mirror code & fix `StartToMirror` (#18904)Gusted2022-03-021-14/+19
| | | | | | | | | | | | | | | | * Use MirrorID instead of RepoID - Use the MirrorID as index(SQL uses `id` column not the `repo_id`). Passes the Mirror ID's into the Sync functions. * Check for MirrorID == 0 * Fix `StartToMirror` + refactor * Update services/mirror/mirror.go Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve mirror iterator (#18928)Lunny Xiao2022-02-281-16/+14
| | | | | * Improve mirror iterator * fix test
* Change git.cmd to RunWithContext (#18693)Martin Scholz2022-02-111-5/+30
| | | | | | | Change all `cmd...Pipeline` commands to `cmd.RunWithContext`. #18553 Co-authored-by: Martin Scholz <martin.scholz@versasec.com>
* Restart zero worker if there is still work to do (#18658)zeripath2022-02-081-10/+18
| | | | | | | | | | | | | | * Restart zero worker if there is still work to do It is possible for the zero worker to timeout before all the work is finished. This may mean that work may take a long time to complete because a worker will only be induced on repushing. Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push. Fix #18607 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552)65432022-02-062-13/+13
|
* Pause queues (#15928)zeripath2022-01-221-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Start adding mechanism to return unhandled data Signed-off-by: Andrew Thornton <art27@cantab.net> * Create pushback interface Signed-off-by: Andrew Thornton <art27@cantab.net> * Add Pausable interface to WorkerPool and Manager Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and PushBack for the bytefifos Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues Signed-off-by: Andrew Thornton <art27@cantab.net> * Wire in UI for pausing Signed-off-by: Andrew Thornton <art27@cantab.net> * add testcases and fix a few issues Signed-off-by: Andrew Thornton <art27@cantab.net> * fix build Signed-off-by: Andrew Thornton <art27@cantab.net> * prevent "race" in the test Signed-off-by: Andrew Thornton <art27@cantab.net> * fix jsoniter mismerge Signed-off-by: Andrew Thornton <art27@cantab.net> * fix conflicts Signed-off-by: Andrew Thornton <art27@cantab.net> * fix format Signed-off-by: Andrew Thornton <art27@cantab.net> * Add warnings for no worker configurations and prevent data-loss with redis/levelqueue Signed-off-by: Andrew Thornton <art27@cantab.net> * Use StopTimer Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* format with gofumpt (#18184)65432022-01-201-2/+2
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Propagate context and ensure git commands run in request context (#17868)zeripath2022-01-192-19/+19
| | | | | | | | | This PR continues the work in #17125 by progressively ensuring that git commands run within the request context. This now means that the if there is a git repo already open in the context it will be used instead of reopening it. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Some repository refactors (#17950)Lunny Xiao2021-12-121-3/+3
| | | | | | | | | * some repository refactors * remove unnecessary code * Fix test * Remove unnecessary banner
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-103-18/+19
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Refactor various strings (#17784)mscherer2021-12-021-2/+2
| | | | | | | | Fixes #16478 Co-authored-by: Gusted <williamzijl7@hotmail.com> Co-authored-by: Gusted <williamzijl7@hotmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Make Requests Processes and create process hierarchy. Associate ↵zeripath2021-11-302-7/+16
| | | | | | | | | OpenRepository with context. (#17125) This PR registers requests with the process manager and manages hierarchy within the processes. Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add `PULL_LIMIT` and `PUSH_LIMIT` to cron.update_mirror task (#17568)zeripath2021-11-221-10/+42
|
* Use `hostmatcher` to replace `matchlist`, improve security (#17605)wxiaoguang2021-11-202-9/+9
| | | | | Use hostmacher to replace matchlist. And we introduce a better DialContext to do a full host/IP check, otherwise the attackers can still bypass the allow/block list by a 302 redirection.
* Move user functions into user.go (#17659)Lunny Xiao2021-11-181-5/+6
| | | | | * Move user functions into user.go * Fix test
* Move some functions into services/repository (#17677)Lunny Xiao2021-11-171-1/+1
|
* Handle broken references in mirror sync (#17013)zeripath2021-10-211-13/+105
| | | | | | | | | | | * Handle broken references in mirror sync If there are broken references during a mirror attempt to fix using `git remote prune`. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Make the Mirror Queue a queue (#17326)zeripath2021-10-171-33/+65
| | | | | | | | | Convert the old mirror syncing queue to the more modern queue format. Fix a bug in the from the repo-archive queue PR - the assumption was made that uniqueness could be enforced with by checking equality in a map in channel unique queues - however this only works for primitive types - which was the initial intention but is an imperfect. This is fixed by marshalling the data and placing the martialled data in the unique map instead. The documentation is also updated to add information about the deprecated configuration values. Signed-off-by: Andrew Thornton <art27@cantab.net>
* DBContext is just a Context (#17100)zeripath2021-09-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | * DBContext is just a Context This PR removes some of the specialness from the DBContext and makes it context This allows us to simplify the GetEngine code to wrap around any context in future and means that we can change our loadRepo(e Engine) functions to simply take contexts. Signed-off-by: Andrew Thornton <art27@cantab.net> * fix unit tests Signed-off-by: Andrew Thornton <art27@cantab.net> * another place that needs to set the initial context Signed-off-by: Andrew Thornton <art27@cantab.net> * avoid race Signed-off-by: Andrew Thornton <art27@cantab.net> * change attachment error Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move db related basic functions to models/db (#17075)Lunny Xiao2021-09-191-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | * Move db related basic functions to models/db * Fix lint * Fix lint * Fix test * Fix lint * Fix lint * revert unnecessary change * Fix test * Fix wrong replace string * Use *Context * Correct committer spelling and fix wrong replaced words Co-authored-by: zeripath <art27@cantab.net>
* Make mirror feature more configurable (#16957)65432021-09-071-0/+13
| | | | | | | | | | | Rename`[repository]` `DISABLE_MIRRORS` to `[mirror]` `DISABLE_NEW_PULL` and add `ENABLED` and `DISABLE_NEW_PUSH` with the below meanings: - `ENABLED`: **true**: Enables the mirror functionality. Set to **false** to disable all mirrors. - `DISABLE_NEW_PULL`: **false**: Disable the creation of **new** mirrors. Pre-existing mirrors remain valid. - `DISABLE_NEW_PUSH`: **false**: Disable the creation of **new** push mirrors. Pre-existing mirrors remain valid. Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: delvh <dev.lh@web.de>
* Add proxy settings and support for migration and webhook (#16704)Lunny Xiao2021-08-182-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | * Add proxy settings and support for migration and webhook * Fix default value * Add newline for example ini * Add lfs proxy support * Fix lint * Follow @zeripath's review * Fix git clone * Fix test * missgin http requests for proxy * use empty Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: zeripath <art27@cantab.net>
* Replace `list.List` with slices (#16311)KN4CK3R2021-08-091-1/+1
| | | | | | | | | | | | | | | | | | | * Replaced list with slice. * Fixed usage of pointer to temporary variable. * Replaced LIFO list with slice. * Lint * Removed type check. * Removed duplicated code. * Lint * Fixed merge. Co-authored-by: 6543 <6543@obermui.de>
* Add push to remote mirror repository (#15157)KN4CK3R2021-06-144-629/+739
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added push mirror model. * Integrated push mirror into queue. * Moved methods into own file. * Added basic implementation. * Mirror wiki too. * Removed duplicated method. * Get url for different remotes. * Added migration. * Unified remote url access. * Add/Remove push mirror remotes. * Prevent hangs with missing credentials. * Moved code between files. * Changed sanitizer interface. * Added push mirror backend methods. * Only update the mirror remote. * Limit refs on push. * Added UI part. * Added missing table. * Delete mirror if repository gets removed. * Changed signature. Handle object errors. * Added upload method. * Added "upload" unit tests. * Added transfer adapter unit tests. * Send correct headers. * Added pushing of LFS objects. * Added more logging. * Simpler body handling. * Process files in batches to reduce HTTP calls. * Added created timestamp. * Fixed invalid column name. * Changed name to prevent xorm auto setting. * Remove table header im empty. * Strip exit code from error message. * Added docs page about mirroring. * Fixed date. * Fixed merge errors. * Moved test to integrations. * Added push mirror test. * Added test.
* Fixed sanitize errors. (#15240)KN4CK3R2021-06-131-15/+7
| | | | | Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Add LFS Migration and Mirror (#14726)KN4CK3R2021-04-082-9/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Create tag on ui (#13467)a10121127962021-02-281-1/+1
| | | | | | | | | | Support create single tag directly support create tag with message from create release ui Signed-off-by: a1012112796 <1012112796@qq.com> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: zeripath <art27@cantab.net>
* [API] Add pagination to ListBranches (#14524)65432021-02-031-1/+1
| | | | | | | | | | | | | | | | | | * make PaginateUserSlice generic -> PaginateSlice * Add pagination to ListBranches * add skip, limit to Repository.GetBranches() * Move routers/api/v1/utils/utils PaginateSlice -> modules/util/paginate.go * repo_module.GetBranches paginate * fix & rename & more logging * better description Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: a1012112796 <1012112796@qq.com>
* Check for 'main' as potential default branch name (#14193)Chester Liu2020-12-301-0/+4
|
* migrate from com.* to alternatives (#14103)65432020-12-251-4/+3
| | | | | | | | | | | | | | | * remove github.com/unknwon/com from models * dont use "com.ToStr()" * replace "com.ToStr" with "fmt.Sprint" where its easy to do * more refactor * fix test * just "proxy" Copy func for now * as per @lunny
* Fix branch/tag notifications in mirror sync (#13855)manuelluis2020-12-051-0/+20
| | | | | | | | * Fix branch/tag notifications in mirror sync * Fix gofmt Co-authored-by: Gitea <gitea@fake.local> Co-authored-by: Lauris BH <lauris@nix.lv>
* Migrations: Use Process Manager to create own Context (#13792)65432020-12-021-1/+2
|
* Refactor: Move PushUpdateOptions (#13363)Lunny Xiao2020-10-301-1/+5
| | | Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
* Update Mirror IsEmpty status on synchronize (#13185)zeripath2020-10-251-2/+67
| | | | | | | | Fix #9630 Fix #13183 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* go-version constraints ignore pre-releases (#13234)zeripath2020-10-211-1/+1
| | | | | | | | | Go-version constraints ignore pre-releases. Rather than change the library further this PR simply changes the git version comparison to use simple version compare ignoring the issue of pre-releases. Signed-off-by: Andrew Thornton <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-27 22:23:20 +0000