aboutsummaryrefslogtreecommitdiffstats
path: root/models/db
Commit message (Collapse)AuthorAgeFilesLines
* Fix joins in `db.Find(AndCount)` (#28978)KN4CK3R2024-01-301-11/+21
|
* Move more functions to db.Find (#28419)Lunny Xiao2024-01-152-33/+31
| | | | | | | | | Following #28220 This PR move more functions to use `db.Find`. --------- Co-authored-by: delvh <dev.lh@web.de>
* Recommend/convert to use case-sensitive collation for MySQL/MSSQL (#28662)wxiaoguang2024-01-103-5/+203
| | | | | | | | | | | | | | | | | | Mainly for MySQL/MSSQL. It is important for Gitea to use case-sensitive database charset collation. If the database is using a case-insensitive collation, Gitea will show startup error/warning messages, and show the errors/warnings on the admin panel's Self-Check page. Make `gitea doctor convert` work for MySQL to convert the collations of database & tables & columns. * Fix #28131 ## :warning: BREAKING :warning: It is not quite breaking, but it's highly recommended to convert the database&table&column to a consistent and case-sensitive collation.
* Avoid cycle-redirecting user/login page (#28636)wxiaoguang2023-12-301-3/+0
| | | | Fix #28231, and remove some unused code. The `db.HasEngine` doesn't seem useful because the db engine is always initialized before web route.
* Refactor deletion (#28610)delvh2023-12-252-25/+27
| | | | | | | | | | | | | | | | | | Introduce the new generic deletion methods - `func DeleteByID[T any](ctx context.Context, id int64) (int64, error)` - `func DeleteByIDs[T any](ctx context.Context, ids ...int64) error` - `func Delete[T any](ctx context.Context, opts FindOptions) (int64, error)` So, we no longer need any specific deletion method and can just use the generic ones instead. Replacement of #28450 Closes #28450 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Remove GetByBean method because sometimes it's danger when query condition ↵Lunny Xiao2023-12-073-11/+59
| | | | | | | | | | | | | | | | | | | | parameter is zero and also introduce new generic methods (#28220) The function `GetByBean` has an obvious defect that when the fields are empty values, it will be ignored. Then users will get a wrong result which is possibly used to make a security problem. To avoid the possibility, this PR removed function `GetByBean` and all references. And some new generic functions have been introduced to be used. The recommand usage like below. ```go // if query an object according id obj, err := db.GetByID[Object](ctx, id) // query with other conditions obj, err := db.Get[Object](ctx, builder.Eq{"a": a, "b":b}) ```
* Use db.Find instead of writing methods for every object (#28084)Lunny Xiao2023-11-243-21/+77
| | | | For those simple objects, it's unnecessary to write the find and count methods again and again.
* refactor: make db iterate context aware (#27710)65432023-10-211-14/+19
| | | | | | | | | the iteration will run until finished atm. this changes it by checking if if the context got canceled before each run of a loop sequence is executed [View this pull with now whitespace](https://github.com/go-gitea/gitea/pull/27710/files?diff=unified&w=1)
* make writing main test easier (#27270)Lunny Xiao2023-09-282-8/+2
| | | | | | | | | This PR removed `unittest.MainTest` the second parameter `TestOptions.GiteaRoot`. Now it detects the root directory by current working directory. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Improve DeleteByID (#26904)Lunny Xiao2023-09-051-1/+1
|
* Fix NuGet search endpoints (#25613)KN4CK3R2023-08-141-0/+16
| | | | | | | | | Fixes #25564 Fixes #23191 - Api v2 search endpoint should return only the latest version matching the query - Api v3 search endpoint should return `take` packages not package versions
* Rename `Sync2` -> `Sync` (#26479)delvh2023-08-132-3/+3
| | | | | | | | | The xorm `Sync2` has already been deprecated in favor of `Sync`, so let's do the same inside the Gitea codebase. Command used to replace everything: ```sh for i in $(ag Sync2 --files-with-matches); do vim $i -c ':%sno/Sync2/Sync/g' -c ':wq'; done ```
* Refactor and enhance issue indexer to support both searching, filtering and ↵Jason Song2023-07-311-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Replace `interface{}` with `any` (#25686)silverwind2023-07-044-52/+52
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Split lfs size from repository size (#22900)a10121127962023-06-281-0/+4
| | | | | | | | | | | | | | | | | | | | releated to #21820 - Split `Size` in repository table as two new colunms, one is `GitSize` for git size, the other is `LFSSize` for lfs data. still store full size in `Size` colunm. - Show full size on ui, but show each of them by a `title`; example: ![image](https://user-images.githubusercontent.com/25342410/218636251-e200f085-d7e7-4a25-9ff1-b586a63e07a9.png) - Return full size in api response. --------- Signed-off-by: a1012112796 <1012112796@qq.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: DmitryFrolovTri <23313323+DmitryFrolovTri@users.noreply.github.com> Co-authored-by: Giteabot <teabot@gitea.io>
* Fix wrong warn messages in migration steps (#25475)Lunny Xiao2023-06-241-1/+4
| | | | | | | | | | | | | | | | | | | The recent change on xorm for `Sync` is it will not warn when database have columns which is not listed on struct. So we just need this warn logs when `Sync` the whole database but not in the migrations Sync. This PR will remove almost unnecessary warning logs on migrations. Now below logs in CI will disappear. ```log 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column creator_id but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column is_closed but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column board_type but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column type but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column closed_date_unix but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column created_unix but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column updated_unix but struct has not related field 2023/06/23 17:51:32 models/db/engine.go:191:InitEngineWithMigration() [W] Table gtestschema.project has column card_type but struct has not related field ```
* Fix index generation parallelly failure (#25235)Lunny Xiao2023-06-151-0/+29
| | | | | | | Fix #22109 --------- Co-authored-by: silverwind <me@silverwind.io>
* Fix parallelly generating index failure with Mysql (#24567)Lunny Xiao2023-06-051-1/+22
|
* Use a separate admin page to show global stats, remove `actions` stat (#25062)wxiaoguang2023-06-031-25/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Before, Gitea shows the database table stats on the `admin dashboard` page. It has some problems: * `count(*)` is quite heavy. If tables have many records, this blocks loading the admin page blocks for a long time * Some users had even reported issues that they can't visit their admin page because this page causes blocking or `50x error (reverse proxy timeout)` * The `actions` stat is not useful. The table is simply too large. Does it really matter if it contains 1,000,000 rows or 9,999,999 rows? * The translation `admin.dashboard.statistic_info` is difficult to maintain. So, this PR uses a separate page to show the stats and removes the `actions` stat. ![image](https://github.com/go-gitea/gitea/assets/2114189/babf7c61-b93b-4a62-bfaa-22983636427e) ## :warning: BREAKING The `actions` Prometheus metrics collector has been removed for the reasons mentioned beforehand. Please do not rely on its output anymore.
* Rewrite logger system (#24726)wxiaoguang2023-05-211-29/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>
* Improve milestone filter on issues page (#22423)Lunny Xiao2023-04-301-1/+1
| | | | | | | | | | Now we have `All milestones`, `No milestones`, `Open milestones` and `Closed milestones`. Fix #11924 Fix #22411 <img width="1166" alt="image" src="https://user-images.githubusercontent.com/81045/212243375-95eea035-a972-44b8-8088-53db614cb07e.png">
* Use more specific test methods (#24265)KN4CK3R2023-04-221-1/+1
| | | | Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Giteabot <teabot@gitea.io>
* Allow adding new files to an empty repo (#24164)wxiaoguang2023-04-192-7/+15
| | | ![image](https://user-images.githubusercontent.com/2114189/232561612-2bfcfd0a-fc04-47ba-965f-5d0bcea46c54.png)
* fix calReleaseNumCommitsBehind (#24148)oliverpool2023-04-182-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | `repoCtx.CommitsCount` is not reliably the commit count of the default branch (Repository.GetCommitsCount depends on what is currently displayed). For instance on the releases page the commit count is correct: https://codeberg.org/Codeberg/pages-server/releases ![2023-04-15-215027](https://user-images.githubusercontent.com/3864879/232250500-6c05dc00-7030-4ec9-87f1-18c7797d36bf.png) However it is not on the single page: https://codeberg.org/Codeberg/pages-server/releases/tag/v4.6.2 ![2023-04-15-215036](https://user-images.githubusercontent.com/3864879/232250503-620c8038-7c2c-45a1-b99d-cb994ef955a6.png) This PR fixes this by removing a "fast branch" which was using this field (I think this field should be removed, since it is a bit unpredictable - but this would mean a larger refactoring PR). _contributed in the context of @forgejo_ --------- Co-authored-by: Giteabot <teabot@gitea.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Support converting varchar to nvarchar for mssql database (#24105)Lunny Xiao2023-04-171-0/+27
|
* Test renderReadmeFile (#23185)Nick2023-03-092-3/+3
| | | | | | | | | | | | | Add test coverage to the important features of [`routers.web.repo.renderReadmeFile`](https://github.com/go-gitea/gitea/blob/067b0c2664d127c552ccdfd264257caca4907a77/routers/web/repo/view.go#L273); namely that: - it can handle looking in docs/, .gitea/, and .github/ - it can handle choosing between multiple competing READMEs - it prefers the localized README to the markdown README to the plaintext README - it can handle broken symlinks when processing all the options - it uses the name of the symlink, not the name of the target of the symlink
* Refactor `setting.Database.UseXXX` to methods (#23354)Jason Song2023-03-074-6/+6
| | | | | | | | | | | Replace #23350. Refactor `setting.Database.UseMySQL` to `setting.Database.Type.IsMySQL()`. To avoid mismatching between `Type` and `UseXXX`. This refactor can fix the bug mentioned in #23350, so it should be backported.
* Fix db.Find bug (#23115)Lunny Xiao2023-02-242-2/+50
| | | | | Caused by #20821 Fix #23110
* Remove all package data after tests (#22984)KN4CK3R2023-02-232-1/+13
| | | | | | | | Fixes #21020 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: zeripath <art27@cantab.net>
* Use beforeCommit instead of baseCommit (#22949)Kyle D2023-02-201-1/+1
| | | | | | | | | | | | | | | | | Replaces: https://github.com/go-gitea/gitea/pull/22947 Fixes https://github.com/go-gitea/gitea/issues/22946 Probably related to https://github.com/go-gitea/gitea/issues/19530 Basically, many of the diffs were broken because they were comparing to the base commit, where a 3-dot diff should be comparing to the [last common ancestor](https://matthew-brett.github.io/pydagogue/git_diff_dots.html). This should have an integration test so that we don’t run into this issue again. --------- Co-authored-by: Jonathan Tran <jonnytran@gmail.com>
* Move delete user to service (#22478)Lunny Xiao2023-02-131-0/+26
| | | | | | Move delete user to service Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Jason Song <i@wolfogre.com>
* Fix .golangci.yml (#22868)zeripath2023-02-111-2/+5
| | | | | | | | | When we updated the .golangci.yml for 1.20 we should have used a string as 1.20 is not a valid number. In doing so we need to restore the nolint markings within the pq driver. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Show all projects, not just repo projects and open/closed projects (#22640)Lunny Xiao2023-02-041-0/+6
| | | | | | | | | | | | | | | | | This PR fixes two problems. One is when filter repository issues, only repository level projects are listed. Another is if you list open issues, only open projects will be displayed in filter options and if you list closed issues, only closed projects will be displayed in filter options. In this PR, both repository level and org/user level projects will be displayed in filter, and both open and closed projects will be listed as filter items. --------- Co-authored-by: John Olheiser <john.olheiser@gmail.com> Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: delvh <dev.lh@web.de>
* update to build with go1.20 (#22732)techknowlogick2023-02-031-5/+2
| | | | | | | as title --------- Co-authored-by: Lauris BH <lauris@nix.lv>
* some refactor about code comments (#20821)Lunny Xiao2023-01-171-1/+49
|
* Fix halfCommitter and WithTx (#22366)Jason Song2023-01-092-5/+124
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Related to #22362. I overlooked that there's always `committer.Close()`, like: ```go ctx, committer, err := db.TxContext(db.DefaultContext) if err != nil { return nil } defer committer.Close() // ... if err != nil { return nil } // ... return committer.Commit() ``` So the `Close` of `halfCommitter` should ignore `commit and close`, it's not a rollback. See: [Why `halfCommitter` and `WithTx` should rollback IMMEDIATELY or commit LATER](https://github.com/go-gitea/gitea/pull/22366#issuecomment-1374778612). Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Always reuse transaction (#22362)Jason Song2023-01-083-34/+96
|
* Support estimated count with multiple schemas (#22276)Jason Song2022-12-301-1/+4
| | | | The `EstimateCount` could be incorrect when the table lives in multiple schemas. Related to #19775.
* Check primary keys for all tables and drop ForeignReference (#21721)Jason Song2022-12-231-0/+33
| | | | | | | | | | | | | | | | | Some dbs require that all tables have primary keys, see - #16802 - #21086 We can add a test to keep it from being broken again. Edit: ~Added missing primary key for `ForeignReference`~ Dropped the `ForeignReference` table to satisfy the check, so it closes #21086. More context can be found in comments. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Test views of LFS files (#22196)Nick2022-12-231-1/+1
|
* Add a simple test for external renderer (#20033)Lunny Xiao2022-12-121-1/+1
| | | Fix #16402
* Update xorm (#22094)KN4CK3R2022-12-101-1/+1
|
* Update go dev dependencies (#22064)silverwind2022-12-081-2/+4
| | | | | `golangci-lint` [deprecated](https://github.com/golangci/golangci-lint/issues/1841) a bunch of linters, removed them.
* Fix generate index failure possibility on postgres (#21998)Lunny Xiao2022-12-021-0/+20
| | | | | @wxiaoguang Please review Co-authored-by: silverwind <me@silverwind.io>
* Fix parallel creating commit status bug with tests (#21911)Lunny Xiao2022-12-011-5/+0
| | | | | This PR is a follow up of #21469 Co-authored-by: Lauris BH <lauris@nix.lv>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-2723-46/+23
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Add `context.Context` to more methods (#21546)KN4CK3R2022-11-193-9/+15
| | | | | | | This PR adds a context parameter to a bunch of methods. Some helper `xxxCtx()` methods got replaced with the normal name now. Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Allow detect whether it's in a database transaction for a context.Context ↵Lunny Xiao2022-11-124-11/+87
| | | | | | | | | | | | | | | | (#21756) Fix #19513 This PR introduce a new db method `InTransaction(context.Context)`, and also builtin check on `db.TxContext` and `db.WithTx`. There is also a new method `db.AutoTx` has been introduced but could be used by other PRs. `WithTx` will always open a new transaction, if a transaction exist in context, return an error. `AutoTx` will try to open a new transaction if no transaction exist in context. That means it will always enter a transaction if there is no error. Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: 6543 <6543@obermui.de>
* Merge db.Iterate and IterateObjects (#21641)Lunny Xiao2022-10-313-18/+57
| | | These two functions are similiar, merge them.
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-241-4/+4
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-23 09:08:28 +0000