aboutsummaryrefslogtreecommitdiffstats
path: root/services/migrations
Commit message (Collapse)AuthorAgeFilesLines
* remove context from retry downloader (#33871)TheFox0x72025-03-141-1/+1
| | | | follow up to https://github.com/go-gitea/gitea/pull/33399 as I apparently missed this one.
* Remove context from git struct (#33793)TheFox0x72025-03-043-11/+11
| | | | Argument is moved from struct init in command run, which lets us remove context from struct.
* Use test context in tests and new loop system in benchmarks (#33648)TheFox0x72025-02-207-19/+12
| | | | | | | | Replace all contexts in tests with go1.24 t.Context() --------- Co-authored-by: Giteabot <teabot@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* refactor: decouple context from migration structs (#33399)TheFox0x72025-02-0719-448/+403
| | | | | | | Use context as much as possible. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor older tests to use testify (#33140)TheFox0x72025-01-091-6/+3
| | | | | Refactor checks to use assert/require Use require.Eventually for waiting in elastic and meilisearch tests Use require to exit early instead of assert
* Fix typo in gitea downloader test and add missing codebase in ↵yp053272025-01-081-2/+2
| | | | `ToGitServiceType` (#33146)
* Convert github.com/xanzy/go-gitlab into gitlab.com/gitlab-org/api/client-go ↵yp053272025-01-082-2/+2
| | | | | (#33126) Fix #32985
* Remove aws go sdk package dependency (#33029)Lunny Xiao2024-12-301-5/+5
| | | | | This PR removed the dependency of `github.com/aws/aws-sdk-go/aws` Patially fix for #33023
* Refactor "string truncate" (#32984)wxiaoguang2024-12-261-2/+1
|
* Move some errors to their own sub packages (#32880)Lunny Xiao2024-12-201-9/+9
|
* Enable tenv and testifylint rules (#32852)TheFox0x72024-12-151-1/+1
| | | | Enables tenv and testifylint linters closes: https://github.com/go-gitea/gitea/issues/32842
* Support migrating GitHub/GitLab PR draft status (#32242)Zettat1232024-10-133-1/+8
| | | | Resolve #32196
* Support repo license (#24872)yp053272024-10-011-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Close #278 Close #24076 ## Solutions: - Use [google/licenseclassifier](https://github.com/google/licenseclassifier/) Test result between [google/licensecheck](https://github.com/google/licensecheck) and [go-license-detector](https://github.com/go-enry/go-license-detector): https://github.com/go-gitea/gitea/pull/24872#issuecomment-1560361167 Test result between [google/licensecheck](https://github.com/google/licensecheck) and [google/licenseclassifier](https://github.com/google/licenseclassifier/): https://github.com/go-gitea/gitea/pull/24872#issuecomment-1576092178 - Generate License Convert Name List to avoid import license templates with same contents Gitea automatically get latest license data from[ spdx/license-list-data](https://github.com/spdx/license-list-data). But unfortunately, some license templates have same contents. #20915 [click here to see the list](https://github.com/go-gitea/gitea/pull/24872#issuecomment-1584141684) So we will generate a list of these license templates with same contents and create a new file to save the result when using `make generate-license`. (Need to decide the save path) - Save License info into a new table `repo_license` Can easily support searching repo by license in the future. ## Screen shot Single License: ![image](https://github.com/go-gitea/gitea/assets/18380374/41260bd7-0b4c-4038-8592-508706cffa9f) Multiple Licenses: ![image](https://github.com/go-gitea/gitea/assets/18380374/34ce2f73-7e18-446b-9b96-ecc4fb61bd70) Triggers: - [x] Push commit to default branch - [x] Create repo - [x] Mirror repo - [x] When Default Branch is changed, licenses should be updated Todo: - [x] Save Licenses info in to DB when there's a change to license file in the commit - [x] DB Migration - [x] A nominal test? - [x] Select which library to use(https://github.com/go-gitea/gitea/pull/24872#issuecomment-1560361167) - [x] API Support - [x] Add repo license table - ~Select license in settings if there are several licenses(Not recommended)~ - License board(later, not in this PR) ![image](https://github.com/go-gitea/gitea/assets/18380374/2c3c3bf8-bcc2-4c6d-8ce0-81d1a9733878) --------- Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Denys Konovalov <kontakt@denyskon.de> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: 6543 <m.huber@kithara.com> Co-authored-by: a1012112796 <1012112796@qq.com> Co-authored-by: techknowlogick <techknowlogick@gitea.com>
* Support allowed hosts for migrations to work with proxy (#32025)Jason Song2024-09-112-5/+1
|
* Support migration from AWS CodeCommit (#31981)Zettat1232024-09-111-0/+269
| | | | | | | | | | | | | | | | | | | | This PR adds support for migrating repos from [AWS CodeCommit](https://docs.aws.amazon.com/codecommit/latest/userguide/welcome.html). The access key ID and secret access key are required to get repository information and pull requests. And [HTTPS Git credentials](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html) are required to clone the repository. <img src="https://github.com/user-attachments/assets/82ecb2d0-8d43-42b0-b5af-f5347a13b9d0" width="680" /> The AWS CodeCommit icon is from [AWS Architecture Icons](https://aws.amazon.com/architecture/icons/). <img src="https://github.com/user-attachments/assets/3c44d21f-d753-40f5-9eae-5d3589e0d50d" width="320" />
* Add `lint-go-gopls` (#30729)silverwind2024-06-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | Uses `gopls check <files>` as a linter. Tested locally and brings up 149 errors currently for me. I don't think I want to fix them in this PR, but I would like at least to get this analysis running on CI. List of errors: ``` modules/indexer/code/indexer.go:181:11: impossible condition: nil != nil routers/private/hook_post_receive.go:120:15: tautological condition: nil == nil services/auth/source/oauth2/providers.go:185:9: tautological condition: nil == nil services/convert/issue.go:216:11: tautological condition: non-nil != nil tests/integration/git_test.go:332:9: impossible condition: nil != nil services/migrations/migrate.go:179:24-43: unused parameter: ctx services/repository/transfer.go:288:48-69: unused parameter: doer tests/integration/api_repo_tags_test.go:75:41-61: unused parameter: session tests/integration/git_test.go:696:64-74: unused parameter: baseBranch tests/integration/gpg_git_test.go:265:27-39: unused parameter: t tests/integration/gpg_git_test.go:284:23-29: unused parameter: tmpDir tests/integration/gpg_git_test.go:284:31-35: unused parameter: name tests/integration/gpg_git_test.go:284:37-42: unused parameter: email ```
* Fix "force private" logic (#31012)wxiaoguang2024-05-201-1/+1
| | | | | When creating a repo, the "FORCE_PRIVATE" config option should be respected, `readonly` doesn't work for checkbox, so it should use `disabled` attribute.
* Resolve lint for unused parameter and unnecessary type arguments (#30750)Chongyi Zheng2024-04-291-8/+7
| | | | | | | | | | Resolve all cases for `unused parameter` and `unnecessary type arguments` Related: #30729 --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Bump `github.com/google/go-github` to v61 (#30738)Chongyi Zheng2024-04-282-2/+2
|
* Enable more `revive` linter rules (#30608)silverwind2024-04-222-2/+0
| | | | | | | | | | | Noteable additions: - `redefines-builtin-id` forbid variable names that shadow go builtins - `empty-lines` remove unnecessary empty lines that `gofumpt` does not remove for some reason - `superfluous-else` eliminate more superfluous `else` branches Rules are also sorted alphabetically and I cleaned up various parts of `.golangci.yml`.
* Change the default maxPerPage for gitbucket (#30392)Kazushi (Jam) Marukawa2024-04-111-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | This patch improves the migration from gitbucket to gitea. The gitbucket uses it's own internal perPage value (= 25) for paging and ignore per_page arguments in the requested URL. This cause gitea to migrate only 25 issues and 25 PRs from gitbucket repository. This may not happens on old gitbucket. But recent gitbucket 4.40 or 4.38.4 has this problem. This patch change to use this internally hardcoded perPage of gitbucket as gitea's maxPerPage numer when migrating from gitbucket. There are several perPage values in gitbucket like 25 for Isseus/PRs and 10 for Releases. Some of those API doesn't support paging yet. It sounds difficult to implement, but using the minimum number among them worked out very well. So, I use 10 in this patch. Brief descriptions of problems and this patch are also available in https://github.com/go-gitea/gitea/issues/30316. In addition, I'm not sure what kind of test cases are possible to write here. It's a test for migration, so it requires testing gitbucket server and gitea server, I guess. Please let me know if it is possible to write such test cases here. Thanks!
* Fix duplicate migrated milestones (#30102)yp053272024-03-261-2/+1
| | | Fix #17567
* remove util.OptionalBool and related functions (#29513)65432024-03-021-4/+4
| | | | | | and migrate affected code _last refactoring bits to replace **util.OptionalBool** with **optional.Option[bool]**_
* Move migration functions to services layer (#29497)Lunny Xiao2024-03-011-1/+1
|
* Include resource state events in Gitlab downloads (#29382)Sebastian Brückner2024-02-262-0/+60
| | | | | | | | | | | Some specific events on Gitlab issues and merge requests are stored separately from comments as "resource state events". With this change, all relevant resource state events are downloaded during issue and merge request migration, and converted to comments. This PR also updates the template used to render comments to add support for migrated comments of these types. ref: https://docs.gitlab.com/ee/api/resource_state_events.html
* Properly migrate target branch change GitLab comment (#29340)Sebastian Brückner2024-02-243-4/+35
| | | | | | | | | | | | | | | | | | GitLab generates "system notes" whenever an event happens within the platform. Unlike Gitea, those events are stored and retrieved as text comments with no semantic details. The only way to tell whether a comment was generated in this manner is the `system` flag on the note type. This PR adds detection for a new specific kind of event: Changing the target branch of a PR. When detected, it is downloaded using Gitea's type for this event, and eventually uploaded into Gitea in the expected format, i.e. with no text content in the comment. This PR also updates the template used to render comments to add support for migrated comments of this type. ref: https://gitlab.com/gitlab-org/gitlab/-/blob/11bd6dc826e0bea2832324a1d7356949a9398884/app/services/system_notes/merge_requests_service.rb#L102
* Use the database object format name but not read from git repoisitory ↵Lunny Xiao2024-02-241-3/+13
| | | | | | | | | | | | | everytime and fix possible migration wrong objectformat when migrating a sha256 repository (#29294) Now we can get object format name from git command line or from the database repository table. Assume the column is right, we don't need to read from git command line every time. This also fixed a possible bug that the object format is wrong when migrating a sha256 repository from external. <img width="658" alt="image" src="https://github.com/go-gitea/gitea/assets/81045/6e9a9dcf-13bf-4267-928b-6bf2c2560423">
* Properly migrate automatic merge GitLab comments (#27873)Sebastian Brückner2024-02-223-24/+93
| | | | | | | | | | | | | | | | | | | | | | | | GitLab generates "system notes" whenever an event happens within the platform. Unlike Gitea, those events are stored and retrieved as text comments with no semantic details. The only way to tell whether a comment was generated in this manner is the `system` flag on the note type. This PR adds detection for two specific kinds of events: Scheduling and un-scheduling of automatic merges on a PR. When detected, they are downloaded using Gitea's type for these events, and eventually uploaded into Gitea in the expected format, i.e. with no text content in the comment. This PR also updates the template used to render comments to add support for migrated comments of these two types. ref: https://gitlab.com/gitlab-org/gitlab/-/blob/11bd6dc826e0bea2832324a1d7356949a9398884/app/services/system_notes/merge_requests_service.rb#L6-L17 --------- Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Simplify how git repositories are opened (#28937)Lunny Xiao2024-01-272-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## Purpose This is a refactor toward building an abstraction over managing git repositories. Afterwards, it does not matter anymore if they are stored on the local disk or somewhere remote. ## What this PR changes We used `git.OpenRepository` everywhere previously. Now, we should split them into two distinct functions: Firstly, there are temporary repositories which do not change: ```go git.OpenRepository(ctx, diskPath) ``` Gitea managed repositories having a record in the database in the `repository` table are moved into the new package `gitrepo`: ```go gitrepo.OpenRepository(ctx, repo_model.Repo) ``` Why is `repo_model.Repository` the second parameter instead of file path? Because then we can easily adapt our repository storage strategy. The repositories can be stored locally, however, they could just as well be stored on a remote server. ## Further changes in other PRs - A Git Command wrapper on package `gitrepo` could be created. i.e. `NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir: repo.RepoPath()}`, the directory should be empty before invoking this method and it can be filled in the function only. #28940 - Remove the `RepoPath()`/`WikiPath()` functions to reduce the possibility of mistakes. --------- Co-authored-by: delvh <dev.lh@web.de>
* Only migrate the first 255 chars of a Github issue title (#28902)JakobDev2024-01-241-1/+2
| | | Fixes #28846
* Move more functions to db.Find (#28419)Lunny Xiao2024-01-151-2/+4
| | | | | | | | | Following #28220 This PR move more functions to use `db.Find`. --------- Co-authored-by: delvh <dev.lh@web.de>
* Use known issue IID to generate new PR index number when migrating from ↵wxiaoguang2023-12-262-11/+45
| | | | | GitLab (#28616) Fix #13884
* Bump google/go-github to v57 (#28514)Yevhen Pavlov2023-12-182-5/+5
|
* Adjust object format interface (#28469)Lunny Xiao2023-12-172-2/+2
| | | | | | | - Remove `ObjectFormatID` - Remove function `ObjectFormatFromID`. - Use `Sha1ObjectFormat` directly but not a pointer because it's an empty struct. - Store `ObjectFormatName` in `repository` struct
* Abstract hash function usage (#28138)Adam Majer2023-12-133-5/+8
| | | | | | Refactor Hash interfaces and centralize hash function. This will allow easier introduction of different hash function later on. This forms the "no-op" part of the SHA256 enablement patch.
* Second part of refactor `db.Find` (#28194)Lunny Xiao2023-12-111-6/+6
| | | Continue of #27798 and move more functions to `db.Find` and `db.Count`.
* Fix migration panic due to an empty review comment diff (#28334)Nanguan Lin2023-12-051-1/+1
| | | | | | | | | | | | | | | | | Fix #28328 ``` func (p *PullRequestComment) GetDiffHunk() string { if p == nil || p.DiffHunk == nil { return "" } return *p.DiffHunk } ``` This function in the package `go-github` may return an empty diff. When it's empty, the following code will panic because it access `ss[1]` https://github.com/go-gitea/gitea/blob/ec1feedbf582b05b6a5e8c59fb2457f25d053ba2/services/migrations/gitea_uploader.go#L861-L867 https://github.com/go-gitea/gitea/blob/ec1feedbf582b05b6a5e8c59fb2457f25d053ba2/modules/git/diff.go#L97-L101
* Use db.Find instead of writing methods for every object (#28084)Lunny Xiao2023-11-241-6/+7
| | | | For those simple objects, it's unnecessary to write the find and count methods again and again.
* Fix DownloadFunc when migrating releases (#27887)Zettat1232023-11-032-6/+9
| | | | | | | | | We should not use `asset.ID` in DownloadFunc because DownloadFunc is a closure. https://github.com/go-gitea/gitea/blob/1bf5527eac6b947010c8faf408f6747de2a2384f/services/migrations/gitea_downloader.go#L284-L295 A similar bug when migrating from GitHub has been fixed in #14703. This PR fixes the bug when migrating from Gitea and GitLab.
* Fix merge base commit for fast-forwarded GitLab PRs (#27825)Sebastian Brückner2023-10-291-1/+9
| | | | | | | | | | | | | | | | | | | | | | | Due to a bug in the GitLab API, the diff_refs field is populated in the response when fetching an individual merge request, but not when fetching a list of them. That field is used to populate the merge base commit SHA. While there is detection for the merge base even when not populated by the downloader, that detection is not flawless. Specifically, when a GitLab merge request has a single commit, and gets merged with the squash strategy, the base branch will be fast-forwarded instead of a separate squash or merge commit being created. The merge base detection attempts to find the last commit on the base branch that is also on the PR branch, but in the fast-forward case that is the PR's only commit. Assuming the head commit is also the merge base results in the import of a PR with 0 commits and no diff. This PR uses the individual merge request endpoint to fetch merge request data with the diff_refs field. With its data, the base merge commit can be properly set, which—by not relying on the detection mentioned above—correctly imports PRs that were "merged" by fast-forwarding the base branch. ref: https://gitlab.com/gitlab-org/gitlab/-/issues/29620
* Use GitLab's squash_commit_sha when available (#27824)Sebastian Brückner2023-10-291-1/+6
| | | | | | | | | | Before this PR, the PR migration code populates Gitea's MergedCommitID field by using GitLab's merge_commit_sha field. However, that field is only populated when the PR was merged using a merge strategy. When a squash strategy is used, squash_commit_sha is populated instead. Given that Gitea does not keep track of merge and squash commits separately, this PR simply populates Gitea's MergedCommitID by using whichever field is present in the GitLab API response.
* Final round of `db.DefaultContext` refactor (#27587)JakobDev2023-10-143-3/+3
| | | Last part of #27065
* Penultimate round of `db.DefaultContext` refactor (#27414)JakobDev2023-10-112-3/+3
| | | | | | | Part of #27065 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Even more `db.DefaultContext` refactor (#27352)JakobDev2023-10-031-1/+1
| | | | | | | | Part of #27065 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: delvh <dev.lh@web.de>
* More `db.DefaultContext` refactor (#27265)JakobDev2023-09-291-2/+2
| | | | | | | Part of #27065 This PR touches functions used in templates. As templates are not static typed, errors are harder to find, but I hope I catch it all. I think some tests from other persons do not hurt.
* make writing main test easier (#27270)Lunny Xiao2023-09-281-4/+1
| | | | | | | | | This PR removed `unittest.MainTest` the second parameter `TestOptions.GiteaRoot`. Now it detects the root directory by current working directory. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Another round of `db.DefaultContext` refactor (#27103)JakobDev2023-09-252-4/+4
| | | | | | | Part of #27065 --------- Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* Next round of `db.DefaultContext` refactor (#27089)JakobDev2023-09-164-11/+10
| | | Part of #27065
* More refactoring of `db.DefaultContext` (#27083)JakobDev2023-09-151-1/+1
| | | Next step of #27065
* Move some functions to service layer (#26969)Lunny Xiao2023-09-082-7/+7
|
generated by cgit v1.2.3 (git 2.39.1) at 2025-03-29 09:35:44 +0000