aboutsummaryrefslogtreecommitdiffstats
path: root/modules
Commit message (Collapse)AuthorAgeFilesLines
* Never use /api/v1 from Gitea UI Pages (#19318)Lunny Xiao2022-04-073-16/+92
| | | | | | | | | | | | | Reusing `/api/v1` from Gitea UI Pages have pros and cons. Pros: 1) Less code copy Cons: 1) API/v1 have to support shared session with page requests. 2) You need to consider for each other when you want to change something about api/v1 or page. This PR moves all dependencies to API/v1 from UI Pages. Partially replace #16052
* Allow custom redirect for landing page (#19324)techknowlogick2022-04-051-2/+7
| | | | | | | | | | | | | * Allow custom redirect for landing page * Update modules/setting/setting.go Co-authored-by: delvh <dev.lh@web.de> * fix lint * one option Co-authored-by: delvh <dev.lh@web.de>
* Remove legacy unmaintained packages, refactor to support change default ↵wxiaoguang2022-04-0310-19/+738
| | | | | | | | | | locale (#19308) Remove two unmaintained vendor packages `i18n` and `paginater`. Changes: * Rewrite `i18n` package with a more clear fallback mechanism. Fix an unstable `Tr` behavior, add more tests. * Refactor the legacy `Paginater` to `Paginator`, test cases are kept unchanged. Trivial enhancement (no breaking for end users): * Use the first locale in LANGS setting option as the default, add a log to prevent from surprising users.
* Prevent intermittent NPE in queue tests (#19301)zeripath2022-04-022-28/+97
| | | | | | | | | | There appears to be an intermittent NPE in queue tests relating to the deferred shutdown/terminate functions. This PR more formally asserts that shutdown and termination occurs before starting and finishing the tests but leaves the defer in place to ensure that if there is an issue shutdown/termination will occur. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove legacy `unknwon/com` package (#19298)wxiaoguang2022-04-028-47/+236
| | | | | | | Follows: #19284 * The `CopyDir` is only used inside test code * Rewrite `ToSnakeCase` with more test cases * The `RedisCacher` only put strings into cache, here we use internal `toStr` to replace the legacy `ToStr` * The `UniqueQueue` can use string as ID directly, no need to call `ToStr`
* A pull-mirror repo should be marked as such on creation (#19295)Peter Gardfjäll2022-04-011-0/+1
| | | | | Right now, a pull-mirror repo does not get marked as such until *after* the mirroring completes. In the meantime, it will show up (in API and UI) as a regular repo.
* Refactor legacy `unknwon/com` package, improve golangci lint (#19284)wxiaoguang2022-04-0117-56/+359
| | | | | | | | The main purpose is to refactor the legacy `unknwon/com` package. 1. Remove most imports of `unknwon/com`, only `util/legacy.go` imports the legacy `unknwon/com` 2. Use golangci's depguard to process denied packages 3. Fix some incorrect values in golangci.yml, eg, the version should be quoted string `"1.18"` 4. Use correctly escaped content for `go-import` and `go-source` meta tags 5. Refactor `com.Expand` to our stable (and the same fast) `vars.Expand`, our `vars.Expand` can still return partially rendered content even if the template is not good (eg: key mistach).
* Remove `git.Command.Run` and `git.Command.RunInDir*` (#19280)wxiaoguang2022-04-0145-399/+353
| | | | | | Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions. * before: `stdout, err := RunInDir(path)` * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
* Update JS dependencies (#19281)silverwind2022-04-011-1/+1
| | | | | | - Upgrade all JS dependencies minus vue and vue-loader - Adapt to breaking change of octicons - Update eslint rules - Tested Swagger UI, sortablejs and prod build
* Fix global packages enabled avaiable (#19276)Lunny Xiao2022-03-311-0/+2
| | | Fix #19275
* Add Goroutine stack inspector to admin/monitor (#19207)zeripath2022-03-3127-232/+732
| | | | | | | | | | | | | | | | | | | Continues on from #19202. Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile. The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines. If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound. In this way we should be able to more easily examine goroutines that have been stuck. A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Improve sync performance for pull-mirrors (#19125)Peter Gardfjäll2022-03-317-19/+834
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.
* Refactor `git.Command.Run*`, introduce `RunWithContextString` and ↵wxiaoguang2022-03-315-114/+122
| | | | | | | | | | | | | | | | | | `RunWithContextBytes` (#19266) This follows * https://github.com/go-gitea/gitea/issues/18553 Introduce `RunWithContextString` and `RunWithContextBytes` to help the refactoring. Add related unit tests. They keep the same behavior to save stderr into err.Error() as `RunInXxx` before. Remove `RunInDirTimeoutPipeline` `RunInDirTimeoutFullPipeline` `RunInDirTimeout` `RunInDirTimeoutEnv` `RunInDirPipeline` `RunInDirFullPipeline` `RunTimeout`, `RunInDirTimeoutEnvPipeline`, `RunInDirTimeoutEnvFullPipeline`, `RunInDirTimeoutEnvFullPipelineFunc`. Then remaining `RunInDir` `RunInDirBytes` `RunInDirWithEnv` can be easily refactored in next PR with a simple search & replace: * before: `stdout, err := RunInDir(path)` * next: `stdout, _, err := RunWithContextString(&git.RunContext{Dir:path})` Other changes: 1. When `timeout <= 0`, use default. Because `timeout==0` is meaningless and could cause bugs. And now many functions becomes more simple, eg: `GitGcRepos` 9 lines to 1 line. `Fsck` 6 lines to 1 line. 2. Only set defaultCommandExecutionTimeout when the option `setting.Git.Timeout.Default > 0`
* Move reaction to models/issues/ (#19264)Lunny Xiao2022-03-311-0/+14
| | | | | | | | | | | | | | | * Move reaction to models/issues/ * Fix test * move the function * improve code * Update models/issues/reaction.go Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Use a more general (and faster) method to sanitize URLs with credentials ↵wxiaoguang2022-03-314-155/+78
| | | | | (#19239) Use a more general method to sanitize URLs with credentials: Simple and intuitive / Faster / Remove all credentials in all URLs
* Add Redis Sentinel Authentication Support (#19213)Justin Sievenpiper2022-03-302-62/+159
| | | | | Gitea was not able to supply any authentication parameters to it. So this brings support to do that, along with some light extraction of a couple of bits into some separate functions for easier testing. I looked at other libraries supporting similar RedisUri-style connection strings (e.g. Lettuce), but it looks like this type of configuration is beyond what would typically be done in a connection string. Since gitea doesn't have configuration options for manually specifying all this redis connection detail, I went ahead and just chose straightforward names for these new parameters.
* Add Package Registry (#16510)KN4CK3R2022-03-3048-24/+4369
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added package store settings. * Added models. * Added generic package registry. * Added tests. * Added NuGet package registry. * Moved service index to api file. * Added NPM package registry. * Added Maven package registry. * Added PyPI package registry. * Summary is deprecated. * Changed npm name. * Sanitize project url. * Allow only scoped packages. * Added user interface. * Changed method name. * Added missing migration file. * Set page info. * Added documentation. * Added documentation links. * Fixed wrong error message. * Lint template files. * Fixed merge errors. * Fixed unit test storage path. * Switch to json module. * Added suggestions. * Added package webhook. * Add package api. * Fixed swagger file. * Fixed enum and comments. * Fixed NuGet pagination. * Print test names. * Added api tests. * Fixed access level. * Fix User unmarshal. * Added RubyGems package registry. * Fix lint. * Implemented io.Writer. * Added support for sha256/sha512 checksum files. * Improved maven-metadata.xml support. * Added support for symbol package uploads. * Added tests. * Added overview docs. * Added npm dependencies and keywords. * Added no-packages information. * Display file size. * Display asset count. * Fixed filter alignment. * Added package icons. * Formatted instructions. * Allow anonymous package downloads. * Fixed comments. * Fixed postgres test. * Moved file. * Moved models to models/packages. * Use correct error response format per client. * Use simpler search form. * Fixed IsProd. * Restructured data model. * Prevent empty filename. * Fix swagger. * Implemented user/org registry. * Implemented UI. * Use GetUserByIDCtx. * Use table for dependencies. * make svg * Added support for unscoped npm packages. * Add support for npm dist tags. * Added tests for npm tags. * Unlink packages if repository gets deleted. * Prevent user/org delete if a packages exist. * Use package unlink in repository service. * Added support for composer packages. * Restructured package docs. * Added missing tests. * Fixed generic content page. * Fixed docs. * Fixed swagger. * Added missing type. * Fixed ambiguous column. * Organize content store by sha256 hash. * Added admin package management. * Added support for sorting. * Add support for multiple identical versions/files. * Added missing repository unlink. * Added file properties. * make fmt * lint * Added Conan package registry. * Updated docs. * Unify package names. * Added swagger enum. * Use longer TEXT column type. * Removed version composite key. * Merged package and container registry. * Removed index. * Use dedicated package router. * Moved files to new location. * Updated docs. * Fixed JOIN order. * Fixed GROUP BY statement. * Fixed GROUP BY #2. * Added symbol server support. * Added more tests. * Set NOT NULL. * Added setting to disable package registries. * Moved auth into service. * refactor * Use ctx everywhere. * Added package cleanup task. * Changed packages path. * Added container registry. * Refactoring * Updated comparison. * Fix swagger. * Fixed table order. * Use token auth for npm routes. * Enabled ReverseProxy auth. * Added packages link for orgs. * Fixed anonymous org access. * Enable copy button for setup instructions. * Merge error * Added suggestions. * Fixed merge. * Handle "generic". * Added link for TODO. * Added suggestions. * Changed temporary buffer filename. * Added suggestions. * Apply suggestions from code review Co-authored-by: Thomas Boerger <thomas@webhippie.de> * Update docs/content/doc/packages/nuget.en-us.md Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Thomas Boerger <thomas@webhippie.de>
* Make git.OpenRepository accept Context (#19260)65432022-03-3031-64/+64
| | | | | * OpenRepositoryCtx -> OpenRepository * OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
* Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235)zeripath2022-03-297-101/+313
| | | | | | | | | | | | | | | | | Strangely #19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added. I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. Fix #19038 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* More commit info from API (#19252)John Olheiser2022-03-292-5/+28
| | | | | Signed-off-by: jolheiser <john.olheiser@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Provide configuration to allow camo-media proxying (#12802)zeripath2022-03-294-0/+108
| | | | | | | | | * Provide configuration to allow camo-media proxying Fix #916 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move init repository related functions to modules (#19159)Lunny Xiao2022-03-292-5/+222
| | | | | | | | | * Move init repository related functions to modules * Fix lint * Use ctx but db.DefaultContext Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move organization related structs into sub package (#18518)Lunny Xiao2022-03-296-27/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move organization related structs into sub package * Fix test * Fix lint * Move more functions into sub packages * Fix bug * Fix test * Update models/organization/team_repo.go Co-authored-by: KN4CK3R <admin@oldschoolhack.me> * Apply suggestions from code review Co-authored-by: KN4CK3R <admin@oldschoolhack.me> * Fix fmt * Follow suggestion from @Gusted * Fix test * Fix test * Fix bug * Use ctx but db.DefaultContext on routers * Fix bug * Fix bug * fix bug * Update models/organization/team_user.go * Fix bug Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor repo clone button and repo clone links, fix JS error on empty repo ↵wxiaoguang2022-03-291-5/+12
| | | | | | | | | | | | | | | | | page (#19208) The last PR about clone buttons introduced an JS error when visiting an empty repo page: * https://github.com/go-gitea/gitea/pull/19028 * `Uncaught ReferenceError: isSSH is not defined`, because the variables are scoped and doesn't share between sub templates. This: 1. Simplify `templates/repo/clone_buttons.tmpl` and make code clear 2. Move most JS code into `initRepoCloneLink` 3. Remove unused `CloneLink.Git` 4. Remove `ctx.Data["DisableSSH"] / ctx.Data["ExposeAnonSSH"] / ctx.Data["DisableHTTP"]`, and only set them when is is needed (eg: deploy keys / ssh keys) 5. Introduce `Data["CloneButton*"]` to provide data for clone buttons and links 6. Introduce `Data["RepoCloneLink"]` for the repo clone link (not the wiki) 7. Remove most `ctx.Data["PageIsWiki"]` because it has been set in the `/wiki` middleware 8. Remove incorrect `quickstart` class in `migrating.tmpl`
* Prevent intermittent failures in RepoIndexerTest (2) (#19229)zeripath2022-03-271-0/+5
| | | | | | | | | | | So whilst #19225 fixes one issue it caused another. We need to initialise the Git module first. Related #19225 Fix #19162 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Hide sensitive content on admin panel progress monitor (#19218)Lunny Xiao2022-03-272-1/+28
| | | | | | Sanitize urls within git process descriptions. Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Do not include global arguments in process manager (#19226)zeripath2022-03-271-8/+10
| | | | | | | | The git command by default adds a number of global arguments. These are not helpful to be displayed in the process manager and so should be skipped for default process descriptions. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent intermittent failures in RepoIndexerTest (#19225)zeripath2022-03-271-2/+8
| | | | | | | | | | | The RepoIndexerTest is failing with considerable frequency due to a race inherrent in its design. This PR adjust this test to avoid the reliance on waiting for the populate repo indexer to run and forcibly adds the repo to the queue. It then flushes the queue. It may be worth separating out the tests somewhat by testing the Index function directly away from the queue however, this forceful method should solve the current problem. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set the default branch for repositories generated from templates (#19136)Abheek Dhawan2022-03-272-2/+15
| | | | | | | | | | | | * Set the default branch for repositories generated from templates * Allows default branch to be set through the API for repos generated from templates * Update swagger API template * Only set default branch to the one from the template if not specified * Use specified default branch if it exists while generating git commits Fix #19082 Co-authored-by: John Olheiser <john.olheiser@gmail.com> Co-authored-by: zeripath <art27@cantab.net>
* Add auto logging of goroutine pid label (#19212)zeripath2022-03-263-0/+61
| | | | | | | | | | | | | | * Add auto logging of goroutine pid label This PR uses unsafe to export the hidden runtime_getProfLabel function from the runtime package and then casts the result to a map[string]string. We can then interrogate this map to get the pid label from the goroutine allowing us to log it with any logging request. Reference #19202 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add `ContextUser` to http request context (#18798)KN4CK3R2022-03-263-3/+6
| | | | | This PR adds a middleware which sets a ContextUser (like GetUserByParams before) in a single place which can be used by other methods. For routes which represent a repo or org the respective middlewares set the field too. Also fix a bug in modules/context/org.go during refactoring.
* Add pprof labels in processes and for lifecycles (#19202)zeripath2022-03-254-48/+55
| | | | | | Use pprof labelling to help identify goroutines with stacks. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Bump goldmark to v1.4.11 (#19201)Robert Kaussow2022-03-241-0/+13
| | | | | | | | | * Bump goldmark to v1.4.11 * add testcase Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Prevent redirect to Host (2) (#19175)zeripath2022-03-231-0/+6
| | | | | | | | | | | | | | | Unhelpfully Locations starting with `/\` will be converted by the browser to `//` because ... well I do not fully understand. Certainly the RFCs and MDN do not indicate that this would be expected. Providing "compatibility" with the (mis)behaviour of a certain proprietary OS is my suspicion. However, we clearly have to protect against this. Therefore we should reject redirection locations that match the regular expression: `^/[\\\\/]+` Reference #9678 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent start panic due to missing DotEscape functionAndrew Thornton2022-03-231-0/+1
| | | | | | | | | | Unfortunately #19169 causing a panic at startup in prod mode. This was hidden by dev mode because the templates are compiled dynamically there. The issue is that DotEscape is not in the original FuncMap at the time of compilation which causes a panic. Ref #19169 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Redirect .wiki/* ui link to /wiki (#18831)a10121127962022-03-231-0/+20
| | | | | | | | | | Redirect .wiki/* ui link to /wiki fix #18590 Signed-off-by: a1012112796 <1012112796@qq.com> Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Try to prevent autolinking of displaynames by email readers (#19169)zeripath2022-03-231-0/+5
| | | | | | | | | | | | | | Unfortunately many email readers will (helpfully) detect url or url-like names and automatically create links to them, even in HTML emails. This is not ideal when usernames can have dots in them. This PR tries to prevent this behaviour by sticking ZWJ characters between dots and also set the meta tag to prevent format detection. Not every email template has been changed in this way - just the activation emails but it may be that we should be setting the above meta tag in all of our emails too. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update HTTP status codes to modern codes (#18063)KN4CK3R2022-03-236-9/+9
| | | | | | * 2xx/3xx/4xx/5xx -> http.Status... * http.StatusFound -> http.StatusTemporaryRedirect * http.StatusMovedPermanently -> http.StatusPermanentRedirect
* Clean paths when looking in Storage (#19124)zeripath2022-03-223-40/+30
| | | | | | | | | | | | | * Clean paths when looking in Storage Ensure paths are clean for minio aswell as local storage. Use url.Path not RequestURI/EscapedPath in storageHandler. Signed-off-by: Andrew Thornton <art27@cantab.net> * Apply suggestions from code review Co-authored-by: Lauris BH <lauris@nix.lv>
* Ensure that setting.LocalURL always has a trailing slash (#19171)zeripath2022-03-221-1/+4
| | | | | Fix #19166 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Use `ctx` instead of `db.DefaultContext` in some ↵wxiaoguang2022-03-227-19/+17
| | | | | | | | | | | | packages(routers/services/modules) (#19163) * Remove `db.DefaultContext` usage in routers, use `ctx` directly * Use `ctx` directly if there is one, remove some `db.DefaultContext` in `services` * Use ctx instead of db.DefaultContext for `cmd` and some `modules` packages * fix incorrect context usage
* Fix the bug: deploy key with write access can not push (#19010)wxiaoguang2022-03-222-4/+4
| | | Use DeployKeyID to replace the IsDeployKey, then CanWriteCode uses the DeployKeyID to check the write permission.
* Renamed ctx.User to ctx.Doer. (#19161)KN4CK3R2022-03-227-56/+56
| | | | Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Cleanup protected branches when deleting users & teams (#19158)Norwin2022-03-221-0/+18
| | | | | | | | | | | * Clean up protected_branches when deleting user fixes #19094 * Clean up protected_branches when deleting teams * fix issue Co-authored-by: Lauris BH <lauris@nix.lv>
* Make migrations SKIP_TLS_VERIFY apply to git too (#19132)zeripath2022-03-192-16/+22
| | | | | | | | Make SKIP_TLS_VERIFY apply to git data migrations too through adding the `-c http.sslVerify=false` option to the git clone command. Fix #18998 Signed-off-by: Andrew Thornton <art27@cantab.net>
* remove not needed (#19128)65432022-03-183-73/+2
|
* nit fix (#19116)65432022-03-171-2/+1
|
* Store the foreign ID of issues during migration (#18446)Aravinth Manivannan2022-03-177-53/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
* rm .sample hooks which aren't used (#19101)techknowlogick2022-03-1634-1950/+0
|
* use go1.18 to build gitea (#19099)techknowlogick2022-03-162-7/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | * use go1.18 to build gitea& update min go version to 1.17 * bump in a few more places * add a few simple tests for isipprivate * update go.mod * update URL to https://go.dev/dl/ * golangci-lint * attempt golangci-lint workaround * change version * bump fumpt version * skip strings.title test * go mod tidy * update tests as some aren't private?? * update tests
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-27 05:10:29 +0000