summaryrefslogtreecommitdiffstats
path: root/modules/migrations/migrate.go
Commit message (Collapse)AuthorAgeFilesLines
* Move migrations into services and base into modules/migration (#17663)Lunny Xiao2021-11-161-476/+0
| | | | | | | * Move migrtions into services and base into modules/migration * Fix imports * Fix lint
* Only allow webhook to send requests to allowed hosts (#17482)wxiaoguang2021-11-011-11/+1
|
* Resolve TODO: Enable pagination on GiteaDownloader.GetComments() & update ↵65432021-09-061-4/+1
| | | | | | | another TODO (#16963) * Update TODO in migrations * Resolve TODO: enable pagination on GiteaDownloader.GetComments()
* Add migrate from OneDev (#16356)KN4CK3R2021-08-221-15/+3
| | | | | | | | | | | | | | * Use context to simplify logic. * Added migration from OneDev. This PR adds [OneDev](https://code.onedev.io/) as migration source. Supported: - [x] Milestones - [x] Issues - [x] Pull Requests - [x] Comments - [x] Reviews - [x] Labels
* Make the github migration less rate limit waiting to get comment per page ↵Lunny Xiao2021-06-301-21/+50
| | | | | | | | | | | | | | | from repository but not per issue (#16070) * Make the github migration less rate limit waiting to get comment per page from repository but not per issue * Fix lint * adjust Downloader interface * Fix missed reviews * Fix test * Remove unused struct
* Add Status Updates whilst Gitea migrations are occurring (#15076)zeripath2021-06-161-3/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Add migrating message Signed-off-by: Andrew Thornton <art27@cantab.net> * simplify messenger Signed-off-by: Andrew Thornton <art27@cantab.net> * make messenger an interface Signed-off-by: Andrew Thornton <art27@cantab.net> * rename Signed-off-by: Andrew Thornton <art27@cantab.net> * prepare for merge Signed-off-by: Andrew Thornton <art27@cantab.net> * as per tech Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Copy git data from disk when restore repository (#16066)Lunny Xiao2021-06-041-1/+1
|
* Add LFS Migration and Mirror (#14726)KN4CK3R2021-04-081-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Do not convert file path to lowercase (#15023)KN4CK3R2021-03-181-3/+4
| | | | | | | | * Do not convert file path to lowercase. * lint * Check against lowercase hostname.
* Ensure validation occurs on clone addresses too (#14994)zeripath2021-03-151-18/+41
| | | | | | | | | | | | | | | | | | | | | * Ensure validation occurs on clone addresses too Fix #14984 Signed-off-by: Andrew Thornton <art27@cantab.net> * fix lint Signed-off-by: Andrew Thornton <art27@cantab.net> * fix test Signed-off-by: Andrew Thornton <art27@cantab.net> * Fix api tests Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Re-enable import local paths after reversion from #13610 (#14925)zeripath2021-03-081-0/+7
| | | | | | | | | PR #13610 unfortunately disabled importing repositories from local paths. This PR restores this functionality. Fix #14700 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add support to migrate from gogs (#14342)65432021-01-211-23/+60
| | | | | | | | | | | | | Add support to migrate gogs: * issues * comments * labels * milestones * wiki Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Dump github/gitlab/gitea repository data to a local directory and restore to ↵Lunny Xiao2020-12-271-83/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | gitea (#12244) * Dump github/gitlab repository data to a local directory * Fix lint * Adjust directory structure * Allow migration special units * Allow migration ignore release assets * Fix lint * Add restore repository * stage the changes * Merge * Fix lint * Update the interface * Add some restore methods * Finish restore * Add comments * Fix restore * Add a token flag * Fix bug * Fix test * Fix test * Fix bug * Fix bug * Fix lint * Fix restore * refactor downloader * fmt * Fix bug isEnd detection on getIssues * Refactor maxPerPage * Remove unused codes * Remove unused codes * Fix bug * Fix restore * Fix dump * Uploader should not depend downloader * use release attachment name but not id * Fix restore bug * Fix lint * Fix restore bug * Add a method of DownloadFunc for base.Release to make uploader not depend on downloader * fix Release yml marshal * Fix trace information * Fix bug when dump & restore * Save relative path on yml file * Fix bug * Use relative path * Update docs * Use git service string but not int * Recognize clone addr to service type
* Add Allow-/Block-List for Migrate & Mirrors (#13610)65432020-11-281-1/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add black list and white list support for migrating repositories * fix fmt * fix lint * fix vendor * fix modules.txt * clean diff * specify log message * use blocklist/allowlist * allways use lowercase to match url * Apply allow/block * Settings: use existing "migrations" section * convert domains lower case * dont store unused value * Block private addresses for migration by default * fix lint * use proposed-upstream func to detect private IP addr * a nit * add own error for blocked migration, add tests, imprufe api * fix test * fix-if-localhost-is-ipv4 * rename error & error message * rename setting options * Apply suggestions from code review Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Migration failure during reaction migration from gitea (#13344)zeripath2020-10-281-1/+1
| | | | | | | | | | | | | | * Migrating reactions is just not that important A failure during migrating reactions should not cause failure of migration. Signed-off-by: Andrew Thornton <art27@cantab.net> * When checking issue reactions check the correct permission Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Gitea 2 Gitea migration (#12657)65432020-10-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * first draft * update gitea sdk to 9e280adb4da * adapt feat of updated sdk * releases now works * break the Reactions loop * use convertGiteaLabel * fix endless loop because paggination is not supported there !!! * rename gitea local uploader files * pagination can bite you in the ass * Version Checks * lint * docs * rename gitea sdk import to miss future conficts * go-swagger: dont scan the sdk structs * make sure gitea can shutdown gracefully * make GetPullRequests and GetIssues similar * rm useles * Add Test: started ... * ... add tests ... * Add tests and Fixing things * Workaround missing SHA * Adapt: Ensure that all migration requests are cancellable (714ab71ddc4260937b1480519d453d2dc4e77dd6) * LINT: fix misspells in test set * adapt ListMergeRequestAwardEmoji * update sdk * Return error when creating giteadownloader failed * update sdk * adapt new sdk * adopt new features * check version before err * adapt: 'migrate service type switch page' * optimize * Fix DefaultBranch * impruve * handle subPath * fix test * Fix ReviewCommentPosition * test GetReviews * add DefaultBranch int test set * rm unused * Update SDK to v0.13.0 * addopt sdk changes * found better link * format template * Update Docs * Update Gitea SDK (v0.13.1)
* Ensure that all migration requests are cancellable (#12669)zeripath2020-09-021-4/+2
| | | | | | | | | | | * Ensure that all migration requests are cancellable Signed-off-by: Andrew Thornton <art27@cantab.net> * Use WithContext as RequestWithContext is go 1.14 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Git migration UX (#12619)John Olheiser2020-08-281-10/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Initial work Signed-off-by: jolheiser <john.olheiser@gmail.com> * Implementation Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix gitlab and token cloning Signed-off-by: jolheiser <john.olheiser@gmail.com> * Imports and JS Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix test Signed-off-by: jolheiser <john.olheiser@gmail.com> * Linting Signed-off-by: jolheiser <john.olheiser@gmail.com> * Generate swagger Signed-off-by: jolheiser <john.olheiser@gmail.com> * Move mirror toggle and rename options Signed-off-by: jolheiser <john.olheiser@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* When migrating from Gitlab map Approvals to approving Reviews (#11147)65432020-04-201-2/+14
| | | | | | When migrating from Gitlab map Gitlab Approvals to approving Reviews Co-Authored-By: zeripath <art27@cantab.net>
* Migrate reviews when migrating repository from github (#9463)Lunny Xiao2020-01-231-1/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * fix typo * Migrate reviews when migrating repository from github * fix lint * Added test and migration when external user login * fix test * fix commented state * Some improvements * fix bug when get pull request and ref original author on code comments * Fix migrated line; Added comment for review * Don't load all pull requests attributes * Fix typo * wrong change copy head * fix tests * fix reactions * Fix test * fix fmt * fix review comment reactions
* Use OriginalURL insead of CloneAddr in migration logging (#9418)mrsdizzie2019-12-181-2/+2
| | | | | CloneAddr will contain username and password credentials and they will get stored in system notices about failed migrations (and logs if trace is set). Replace with OriginalURL that doesn't have those.
* Sleep longer if request speed is over github limitation (#9335)Lunny Xiao2019-12-161-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Sleep longer if request speed is over github limitation * improve code * remove unused code * fix lint * Use github's rate limit remain value to determine how long to sleep * Save reset time when finished github api request * fix bug * fix lint * Add context.Context for sleep * fix test * improve code * fix bug and lint * fix import order
* Add SyncTags to uploader interface (#9326)John Olheiser2019-12-121-7/+8
| | | | | | | | | | * Add sync tags to interface Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix revive Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Only sync tags after all batches (#9319)John Olheiser2019-12-111-3/+7
| | | Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Add retry for migration http/https requests (#9019)Lunny Xiao2019-11-161-0/+6
| | | | | | | | | | * Add retry for migration http/https requests * give the more suitable name for retry configuraion items * fix docs and lint * Only use retryDownloader when setting > 1
* Add Close() method to gogitRepository (#8901)zeripath2019-11-131-0/+1
| | | | | | | | | In investigating #7947 it has become clear that the storage component of go-git repositories needs closing. This PR adds this Close function and adds the Close functions as necessary. In TransferOwnership the ctx.Repo.GitRepo is closed if it is open to help prevent the risk of multiple open files. Fixes #7947
* Update migrated repositories' issues/comments/prs poster id if user has a ↵Lunny Xiao2019-10-141-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | github external user saved (#7751) * update migrated issues/comments when login as github * add get userid when migrating or login with github oauth2 * fix lint * add migrations for repository service type * fix build * remove unnecessary dependencies on migrations * add cron task to update migrations poster ids and fix posterid when migrating * fix lint * fix lint * improve code * fix lint * improve code * replace releases publish id to actual author id * fix import * fix bug * fix lint * fix rawdata definition * fix some bugs * fix error message
* Move migrating repository from frontend to backend (#6200)Lunny Xiao2019-10-131-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * move migrating to backend * add loading image when migrating and fix tests * fix format * fix lint * add redis task queue support and improve docs * add redis vendor * fix vet * add database migrations and fix app.ini sample * add comments for task section on app.ini.sample * Update models/migrations/v84.go Co-Authored-By: lunny <xiaolunwen@gmail.com> * Update models/repo.go Co-Authored-By: lunny <xiaolunwen@gmail.com> * move migrating to backend * add loading image when migrating and fix tests * fix fmt * add redis task queue support and improve docs * fix fixtures * fix fixtures * fix duplicate function on index.js * fix tests * rename repository statuses * check if repository is being create when SSH request * fix lint * fix template * some improvements * fix template * unified migrate options * fix lint * fix loading page * refactor * When gitea restart, don't restart the running tasks because we may have servel gitea instances, that may break the migration * fix js * Update models/repo.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update docs/content/doc/advanced/config-cheat-sheet.en-us.md Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * fix tests * rename ErrTaskIsNotExist to ErrTaskDoesNotExist * delete release after add one on tests to make it run happy * fix tests * fix tests * improve codes * fix lint * fix lint * fix migrations
* Import topics during migration (#7851)Antoine GIRARD2019-08-141-0/+11
| | | | | | | | * add GetTopics interface * CreateTopics * remove un-needed comment
* Display original author and URL information when showing migrated ↵mrsdizzie2019-07-081-22/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | issues/comments (#7352) * Store original author info for migrated issues and comments Keep original author name for displaying in Gitea interface and also store original author user ID for potential future use in linking accounts from old location. * Add original_url for repo Store the original URL for a migrated repo Clean up migrations/tests * fix migration * fix golangci-lint * make 'make revive' happy also * Modify templates to use OriginalAuthor if set Use the original author name in templates if it is set rather than the user who migrated/currently owns the issues * formatting fixes * make generate-swagger * Use default avatar for imported comments * Remove no longer used IgnoreIssueAuthor option * Add OriginalAuthorID to swagger also
* Fix typo in PR migration check (#7368)mrsdizzie2019-07-061-1/+1
|
* Detect migrating batch size (#7353)Lunny Xiao2019-07-061-17/+47
| | | | | | | | * Make migrating batch size as configurable * detect different table batch insert size and remove config item * remove unrelated changes
* Fix bug conflict between SyncReleasesWithTags and InsertReleases (#7337)Lunny Xiao2019-07-011-1/+1
| | | | | | | | * fix bug conflict between SyncReleasesWithTags and InsertReleases * fix tests * fix fmt
* Use batch insert on migrating repository to make the process faster (#7050)Lunny Xiao2019-06-291-26/+50
| | | | | | | | | | * Use batch insert on migrating repository to make the process faster * fix lint * fix tests * fix comments
* improve github downloader on migrations (#7049)Lunny Xiao2019-05-301-5/+5
| | | | | | | | * improve github downloader on migrations * fix tests * fix uppercase function parameters
* fix missing repo description when migrating (#7000)Lunny Xiao2019-05-201-0/+3
|
* Improve migrations to support migrating ↵Lunny Xiao2019-05-071-0/+205
milestones/labels/issues/comments/pullrequests (#6290) * add migrations * fix package dependency * fix lints * implements migrations except pull requests * add releases * migrating releases * fix bug * fix lint * fix migrate releases * fix tests * add rollback * pull request migtations * fix import * fix go module vendor * add tests for upload to gitea * more migrate options * fix swagger-check * fix misspell * add options on migration UI * fix log error * improve UI options on migrating * add support for username password when migrating from github * fix tests * remove comments and fix migrate limitation * improve error handles * migrate API will also support migrate milestones/labels/issues/pulls/releases * fix tests and remove unused codes * add DownloaderFactory and docs about how to create a new Downloader * fix misspell * fix migration docs * Add hints about migrate options on migration page * fix tests
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-16 23:00:09 +0000