aboutsummaryrefslogtreecommitdiffstats
path: root/models/db
Commit message (Collapse)AuthorAgeFilesLines
* Refactor context flash msg and global variables (#33375)wxiaoguang11 days1-3/+1
| | | | | | 1. add `GetSiteCookieFlashMessage` to help to parse flash message 2. clarify `handleRepoHomeFeed` logic 3. remove unnecessary global variables, use `sync.OnceValue` instead 4. add some tests for `IsUsableUsername` and `IsUsableRepoName`
* Support performance trace (#32973)wxiaoguang2025-01-212-6/+19
| | | | | | | | | | | | | 1. Add a OpenTelemetry-like shim-layer to collect traces 2. Add a simple builtin trace collector and exporter, end users could download the diagnosis report to get the traces. This PR's design is quite lightweight, no hard-dependency, and it is easy to improve or remove. We can try it on gitea.com first to see whether it works well, and fine tune the details. --------- Co-authored-by: silverwind <me@silverwind.io>
* Refactor older tests to use testify (#33140)TheFox0x72025-01-091-6/+3
| | | | | Refactor checks to use assert/require Use require.Eventually for waiting in elastic and meilisearch tests Use require to exit early instead of assert
* Fix unittest and repo create bug (#33061)wxiaoguang2024-12-311-13/+6
| | | | | | | 1. `StatDir` was not right, fix the FIXME 2. Clarify the test cases for `IsUsableRepoName` 3. Fix regression bug in `repo-new.ts` Fix #33060
* Refactor db package and remove unnecessary `DumpTables` (#32930)wxiaoguang2024-12-209-219/+242
|
* Enable tenv and testifylint rules (#32852)TheFox0x72024-12-151-2/+0
| | | | Enables tenv and testifylint linters closes: https://github.com/go-gitea/gitea/issues/32842
* chore: use errors.New to replace fmt.Errorf with no parameters (#32800)RiceChuan2024-12-121-3/+3
| | | | | use errors.New to replace fmt.Errorf with no parameters Signed-off-by: RiceChuan <lc582041246@gmail.com>
* Add label/author/assignee filters to the user/org home issue list (#32779)wxiaoguang2024-12-111-5/+7
| | | | | | | | | | Replace #26661, fix #25979 Not perfect, but usable and much better than before. Since it is quite complex, I am not quite sure whether there would be any regression, if any, I will fix in first time. I have tested the related pages many times: issue list, milestone issue list, project view, user issue list, org issue list.
* Fix: passkey login not working anymore (#32623)hiifong2024-11-261-0/+3
| | | | | | | Quick fix #32595, use authenticator auth flags to login --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor push mirror find and add check for updating push mirror (#32539)Lunny Xiao2024-11-181-1/+2
| | | Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Fix db engine (#32351)wxiaoguang2024-10-275-53/+114
| | | Fix #32349
* bump to go 1.23 (#31855)techknowlogick2024-09-101-2/+2
|
* [Refactor] Unify repo search order by logic (#30876)65432024-06-131-6/+0
| | | | have repo OrderBy definitions defined in one place and use a single type for OrderBy database options
* Fix various problems around projects board view (#30696)Lunny Xiao2024-05-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | # The problem The previous implementation will start multiple POST requests from the frontend when moving a column and another bug is moving the default column will never be remembered in fact. # What's changed - [x] This PR will allow the default column to move to a non-first position - [x] And it also uses one request instead of multiple requests when moving the columns - [x] Use a star instead of a pin as the icon for setting the default column action - [x] Inserted new column will be append to the end - [x] Fix #30701 the newly added issue will be append to the end of the default column - [x] Fix when deleting a column, all issues in it will be displayed from UI but database records exist. - [x] Add a limitation for columns in a project to 20. So the sorting will not be overflow because it's int8. --------- Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Enable more `revive` linter rules (#30608)silverwind2024-04-221-1/+0
| | | | | | | | | | | Noteable additions: - `redefines-builtin-id` forbid variable names that shadow go builtins - `empty-lines` remove unnecessary empty lines that `gofumpt` does not remove for some reason - `superfluous-else` eliminate more superfluous `else` branches Rules are also sorted alphabetically and I cleaned up various parts of `.golangci.yml`.
* Improve "must-change-password" logic and document (#30472)wxiaoguang2024-04-141-2/+2
| | | | | Unify the behaviors of "user create" and "user change-password". Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* Replace MSSQL driver with a better maintained version (#30390)Lunny Xiao2024-04-111-3/+3
| | | | | | | | As the latest tag of `github.com/denisenkom/go-mssqldb` is in 2022, but as a fork of it, `github.com/microsoft/go-mssqldb` has more activities than the original repository. We can convert the driver to the fork. Since the interface of Go database driver are the same, it should have no any affect for the end users.
* Fix oauth2 builtin application logic (#30304)wxiaoguang2024-04-081-6/+6
| | | | | | Fix #29074 (allow to disable all builtin apps) and don't make the doctor command remove the builtin apps. By the way, rename refobject and joincond to camel case.
* Fix misuse of `TxContext` (#30061)Jason Song2024-03-251-0/+10
| | | | | | | | | | Help #29999, or its tests cannot pass. Also, add some comments to clarify the usage of `TxContext`. I don't check all usages of `TxContext` because there are too many (almost 140+). It's a better idea to replace them with `WithTx` instead of checking them one by one. However, that may be another refactoring PR.
* Partially enable MSSQL case-sensitive collation support (#29238)wxiaoguang2024-03-081-2/+1
| | | Follow #28662
* Add slow SQL query warning (#27545)Earl Warren2024-02-231-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Databases are one of the most important parts of Forgejo, every interaction uses the database in one way or another. Therefore, it is important to maintain the database and recognize when the server is not doing well with the database. There already is the option to log *every* SQL query along with its execution time, but monitoring becomes impractical for larger instances and takes up unnecessary storage in the logs. - Add a QoL enhancement that allows instance administrators to specify a threshold value beyond which query execution time is logged as a warning in the xorm logger. The default value is a conservative five seconds to avoid this becoming a source of spam in the logs. - The use case for this patch is that with an instance the size of Codeberg, monitoring SQL logs is not very fruitful and most of them are uninteresting. Recently, in the context of persistent deadlock issues (https://codeberg.org/forgejo/forgejo/issues/220), I have noticed that certain queries hold locks on tables like comment and issue for several seconds. This patch helps to identify which queries these are and when they happen. - Added unit test. (cherry picked from commit 9cf501f1af4cd870221cef6af489618785b71186) --------- Co-authored-by: Gusted <postmaster@gusted.xyz> Co-authored-by: Giteabot <teabot@gitea.io> Co-authored-by: 6543 <6543@obermui.de>
* Fix joins in `db.Find(AndCount)` (#28978)KN4CK3R2024-01-301-11/+21
|
* Move more functions to db.Find (#28419)Lunny Xiao2024-01-152-33/+31
| | | | | | | | | Following #28220 This PR move more functions to use `db.Find`. --------- Co-authored-by: delvh <dev.lh@web.de>
* Recommend/convert to use case-sensitive collation for MySQL/MSSQL (#28662)wxiaoguang2024-01-103-5/+203
| | | | | | | | | | | | | | | | | | Mainly for MySQL/MSSQL. It is important for Gitea to use case-sensitive database charset collation. If the database is using a case-insensitive collation, Gitea will show startup error/warning messages, and show the errors/warnings on the admin panel's Self-Check page. Make `gitea doctor convert` work for MySQL to convert the collations of database & tables & columns. * Fix #28131 ## :warning: BREAKING :warning: It is not quite breaking, but it's highly recommended to convert the database&table&column to a consistent and case-sensitive collation.
* Avoid cycle-redirecting user/login page (#28636)wxiaoguang2023-12-301-3/+0
| | | | Fix #28231, and remove some unused code. The `db.HasEngine` doesn't seem useful because the db engine is always initialized before web route.
* Refactor deletion (#28610)delvh2023-12-252-25/+27
| | | | | | | | | | | | | | | | | | Introduce the new generic deletion methods - `func DeleteByID[T any](ctx context.Context, id int64) (int64, error)` - `func DeleteByIDs[T any](ctx context.Context, ids ...int64) error` - `func Delete[T any](ctx context.Context, opts FindOptions) (int64, error)` So, we no longer need any specific deletion method and can just use the generic ones instead. Replacement of #28450 Closes #28450 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Remove GetByBean method because sometimes it's danger when query condition ↵Lunny Xiao2023-12-073-11/+59
| | | | | | | | | | | | | | | | | | | | parameter is zero and also introduce new generic methods (#28220) The function `GetByBean` has an obvious defect that when the fields are empty values, it will be ignored. Then users will get a wrong result which is possibly used to make a security problem. To avoid the possibility, this PR removed function `GetByBean` and all references. And some new generic functions have been introduced to be used. The recommand usage like below. ```go // if query an object according id obj, err := db.GetByID[Object](ctx, id) // query with other conditions obj, err := db.Get[Object](ctx, builder.Eq{"a": a, "b":b}) ```
* Use db.Find instead of writing methods for every object (#28084)Lunny Xiao2023-11-243-21/+77
| | | | For those simple objects, it's unnecessary to write the find and count methods again and again.
* refactor: make db iterate context aware (#27710)65432023-10-211-14/+19
| | | | | | | | | the iteration will run until finished atm. this changes it by checking if if the context got canceled before each run of a loop sequence is executed [View this pull with now whitespace](https://github.com/go-gitea/gitea/pull/27710/files?diff=unified&w=1)
* make writing main test easier (#27270)Lunny Xiao2023-09-282-8/+2
| | | | | | | | | This PR removed `unittest.MainTest` the second parameter `TestOptions.GiteaRoot`. Now it detects the root directory by current working directory. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Improve DeleteByID (#26904)Lunny Xiao2023-09-051-1/+1
|
* Fix NuGet search endpoints (#25613)KN4CK3R2023-08-141-0/+16
| | | | | | | | | Fixes #25564 Fixes #23191 - Api v2 search endpoint should return only the latest version matching the query - Api v3 search endpoint should return `take` packages not package versions
* Rename `Sync2` -> `Sync` (#26479)delvh2023-08-132-3/+3
| | | | | | | | | The xorm `Sync2` has already been deprecated in favor of `Sync`, so let's do the same inside the Gitea codebase. Command used to replace everything: ```sh for i in $(ag Sync2 --files-with-matches); do vim $i -c ':%sno/Sync2/Sync/g' -c ':wq'; done ```
* Refactor and enhance issue indexer to support both searching, filtering and ↵Jason Song2023-07-311-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Replace `interface{}` with `any` (#25686)silverwind2023-07-044-52/+52
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Split lfs size from repository size (#22900)a10121127962023-06-281-0/+4
| | | | | | | | | | | | | | | | | | | | releated to #21820 - Split `Size` in repository table as two new colunms, one is `GitSize` for git size, the other is `LFSSize` for lfs data. still store full size in `Size` colunm. - Show full size on ui, but show each of them by a `title`; example: ![image](https://user-images.githubusercontent.com/25342410/218636251-e200f085-d7e7-4a25-9ff1-b586a63e07a9.png) - Return full size in api response. --------- Signed-off-by: a1012112796 <1012112796@qq.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: DmitryFrolovTri <23313323+DmitryFrolovTri@users.noreply.github.com> Co-authored-by: Giteabot <teabot@gitea.io>
* Fix wrong warn messages in migration steps (#25475)Lunny Xiao2023-06-241-1/+4
| | | | | | | | | | | | | | | | | | | The recent change on xorm for `Sync` is it will not warn when database have columns which is not listed on struct. So we just need this warn logs when `Sync` the whole database but not in the migrations Sync. This PR will remove almost unnecessary warning logs on migrations. Now below logs in CI will disappear. ```log 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column creator_id but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column is_closed but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column board_type but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column type but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column closed_date_unix but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column created_unix but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column updated_unix but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column card_type but struct has not related field ```
* Fix index generation parallelly failure (#25235)Lunny Xiao2023-06-151-0/+29
| | | | | | | Fix #22109 --------- Co-authored-by: silverwind <me@silverwind.io>
* Fix parallelly generating index failure with Mysql (#24567)Lunny Xiao2023-06-051-1/+22
|
* Use a separate admin page to show global stats, remove `actions` stat (#25062)wxiaoguang2023-06-031-25/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Before, Gitea shows the database table stats on the `admin dashboard` page. It has some problems: * `count(*)` is quite heavy. If tables have many records, this blocks loading the admin page blocks for a long time * Some users had even reported issues that they can't visit their admin page because this page causes blocking or `50x error (reverse proxy timeout)` * The `actions` stat is not useful. The table is simply too large. Does it really matter if it contains 1,000,000 rows or 9,999,999 rows? * The translation `admin.dashboard.statistic_info` is difficult to maintain. So, this PR uses a separate page to show the stats and removes the `actions` stat. ![image](https://github.com/go-gitea/gitea/assets/2114189/babf7c61-b93b-4a62-bfaa-22983636427e) ## :warning: BREAKING The `actions` Prometheus metrics collector has been removed for the reasons mentioned beforehand. Please do not rely on its output anymore.
* Rewrite logger system (#24726)wxiaoguang2023-05-211-29/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>
* Improve milestone filter on issues page (#22423)Lunny Xiao2023-04-301-1/+1
| | | | | | | | | | Now we have `All milestones`, `No milestones`, `Open milestones` and `Closed milestones`. Fix #11924 Fix #22411 <img width="1166" alt="image" src="https://user-images.githubusercontent.com/81045/212243375-95eea035-a972-44b8-8088-53db614cb07e.png">
* Use more specific test methods (#24265)KN4CK3R2023-04-221-1/+1
| | | | Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Giteabot <teabot@gitea.io>
* Allow adding new files to an empty repo (#24164)wxiaoguang2023-04-192-7/+15
| | | ![image](https://user-images.githubusercontent.com/2114189/232561612-2bfcfd0a-fc04-47ba-965f-5d0bcea46c54.png)
* fix calReleaseNumCommitsBehind (#24148)oliverpool2023-04-182-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | `repoCtx.CommitsCount` is not reliably the commit count of the default branch (Repository.GetCommitsCount depends on what is currently displayed). For instance on the releases page the commit count is correct: https://codeberg.org/Codeberg/pages-server/releases ![2023-04-15-215027](https://user-images.githubusercontent.com/3864879/232250500-6c05dc00-7030-4ec9-87f1-18c7797d36bf.png) However it is not on the single page: https://codeberg.org/Codeberg/pages-server/releases/tag/v4.6.2 ![2023-04-15-215036](https://user-images.githubusercontent.com/3864879/232250503-620c8038-7c2c-45a1-b99d-cb994ef955a6.png) This PR fixes this by removing a "fast branch" which was using this field (I think this field should be removed, since it is a bit unpredictable - but this would mean a larger refactoring PR). _contributed in the context of @forgejo_ --------- Co-authored-by: Giteabot <teabot@gitea.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Support converting varchar to nvarchar for mssql database (#24105)Lunny Xiao2023-04-171-0/+27
|
* Test renderReadmeFile (#23185)Nick2023-03-092-3/+3
| | | | | | | | | | | | | Add test coverage to the important features of [`routers.web.repo.renderReadmeFile`](https://github.com/go-gitea/gitea/blob/067b0c2664d127c552ccdfd264257caca4907a77/routers/web/repo/view.go#L273); namely that: - it can handle looking in docs/, .gitea/, and .github/ - it can handle choosing between multiple competing READMEs - it prefers the localized README to the markdown README to the plaintext README - it can handle broken symlinks when processing all the options - it uses the name of the symlink, not the name of the target of the symlink
* Refactor `setting.Database.UseXXX` to methods (#23354)Jason Song2023-03-074-6/+6
| | | | | | | | | | | Replace #23350. Refactor `setting.Database.UseMySQL` to `setting.Database.Type.IsMySQL()`. To avoid mismatching between `Type` and `UseXXX`. This refactor can fix the bug mentioned in #23350, so it should be backported.
* Fix db.Find bug (#23115)Lunny Xiao2023-02-242-2/+50
| | | | | Caused by #20821 Fix #23110
* Remove all package data after tests (#22984)KN4CK3R2023-02-232-1/+13
| | | | | | | | Fixes #21020 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: zeripath <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2025-02-05 05:54:22 +0000