summaryrefslogtreecommitdiffstats
path: root/modules/indexer
Commit message (Collapse)AuthorAgeFilesLines
* Fix bug filtering issues which have no project (#31337) (#31367)Lunny Xiao2024-06-171-1/+6
|
* Fix bleve fuzziness (#30799) (#30804)Giteabot2024-05-013-9/+15
| | | | | | | | Backport #30799 by wxiaoguang Fix #30797 Fix #30317 Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Perform Newest sort type correctly when sorting issues (#30644)Kemal Zebari2024-04-231-2/+2
| | | | | | | | | | | | | Should resolve #30642. Before this commit, we were treating an empty `?sort=` query parameter as the correct sorting type (which is to sort issues in descending order by their created UNIX time). But when we perform `sort=latest`, we did not include this as a type so we would sort by the most recently updated when reaching the `default` switch statement block. This commit fixes this by considering the empty string, "latest", and just any other string that is not mentioned in the switch statement as sorting by newest.
* Enable more `revive` linter rules (#30608)silverwind2024-04-222-3/+0
| | | | | | | | | | | Noteable additions: - `redefines-builtin-id` forbid variable names that shadow go builtins - `empty-lines` remove unnecessary empty lines that `gofumpt` does not remove for some reason - `superfluous-else` eliminate more superfluous `else` branches Rules are also sorted alphabetically and I cleaned up various parts of `.golangci.yml`.
* Render embedded code preview by permlink in markdown (#30234)wxiaoguang2024-04-021-7/+9
| | | | | The permlink in markdown will be rendered as a code preview block, like GitHub Co-authored-by: silverwind <me@silverwind.io>
* Use db.ListOptions directly instead of Paginator interface to make it easier ↵Lunny Xiao2024-03-246-16/+39
| | | | | | | | | | | | | to use and fix performance of /pulls and /issues (#29990) This PR uses `db.ListOptions` instead of `Paginor` to make the code simpler. And it also fixed the performance problem when viewing /pulls or /issues. Before the counting in fact will also do the search. --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: silverwind <me@silverwind.io>
* Support repo code search without setting up an indexer (#29998)wxiaoguang2024-03-241-17/+18
| | | | | | | | | | | | | | | | | By using git's ability, end users (especially small instance users) do not need to enable the indexer, they could also benefit from the code searching feature. Fix #29996 ![image](https://github.com/go-gitea/gitea/assets/2114189/11b7e458-88a4-480d-b4d7-72ee59406dd1) ![image](https://github.com/go-gitea/gitea/assets/2114189/0fe777d5-c95c-4288-a818-0427680805b6) --------- Co-authored-by: silverwind <me@silverwind.io>
* Determine fuzziness of bleve indexer by keyword length (#29706)65432024-03-233-28/+22
| | | also bleve did match on fuzzy search and the other way around. this also fix that bug.
* Use db.ListOptionsAll instead of db.ListOptions{ListAll: true} (#29995)Lunny Xiao2024-03-222-35/+17
|
* Meilisearch double quote on "match" query (#29740)65432024-03-162-70/+37
| | | | | make `nonFuzzyWorkaround` unessesary cc @Kerollmops
* Refactor code_indexer to use an SearchOptions struct for PerformSearch (#29724)65432024-03-166-35/+53
| | | | | | | | similar to how it's already done for the issue_indexer --- *Sponsored by Kithara Software GmbH*
* Refactor to use optional.Option for issue index search option (#29739)65432024-03-139-187/+140
| | | Signed-off-by: 6543 <6543@obermui.de>
* Use repo object format name instead of detecting from git repository (#29702)Lunny Xiao2024-03-101-8/+4
| | | | It's unnecessary to detect the repository object format from git repository. Just use the repository's object format name.
* Patch in exact search for meilisearch (#29671)65432024-03-0911-27/+178
| | | | | | | | | | | | | | | | | | | meilisearch does not have an search option to contorl fuzzynes per query right now: - https://github.com/meilisearch/meilisearch/issues/1192 - https://github.com/orgs/meilisearch/discussions/377 - https://github.com/meilisearch/meilisearch/discussions/1096 so we have to create a workaround by post-filter the search result in gitea until this is addressed. For future works I added an option in backend only atm, to enable fuzzynes for issue indexer too. And also refactored the code so the fuzzy option is equal in logic to code indexer --- *Sponsored by Kithara Software GmbH*
* Fix wrong line number in code search result (#29260)yp053272024-03-061-19/+31
| | | | | | | | | | | Fix #29136 Before: The result is a table and all line numbers are all in one row. After: Use a separate table column for the line numbers. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* remove util.OptionalBool and related functions (#29513)65432024-03-027-26/+27
| | | | | | and migrate affected code _last refactoring bits to replace **util.OptionalBool** with **optional.Option[bool]**_
* migrate some more "OptionalBool" to "Option[bool]" (#29479)65432024-02-291-2/+2
| | | | | | | | just some refactoring bits towards replacing **util.OptionalBool** with **optional.Option[bool]** --------- Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* Add `io.Closer` guidelines (#29387)KN4CK3R2024-02-252-4/+1
| | | Co-authored-by: Yarden Shoham <git@yardenshoham.com>
* Allow non-admin users to delete review requests (#29057)Zettat1232024-02-241-7/+7
| | | | | | | | Fix #14459 The following users can add/remove review requests of a PR - the poster of the PR - the owner or collaborators of the repository - members with read permission on the pull requests unit
* fix: Elasticsearch: Request Entity Too Large #28117 (#29062)dark-angel2024-02-071-5/+11
| | | | | | | | | | | | | | | | Fix for gitea putting everything into one request without batching and sending it to Elasticsearch for indexing as issued in #28117 This issue occured in large repositories while Gitea tries to index the code using ElasticSearch. I've applied necessary changes that takes batch length from below config (app.ini) ``` [queue.code_indexer] BATCH_LENGTH=<length_int> ``` and batches all requests to Elasticsearch in chunks as configured in the above config
* Simplify how git repositories are opened (#28937)Lunny Xiao2024-01-271-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## Purpose This is a refactor toward building an abstraction over managing git repositories. Afterwards, it does not matter anymore if they are stored on the local disk or somewhere remote. ## What this PR changes We used `git.OpenRepository` everywhere previously. Now, we should split them into two distinct functions: Firstly, there are temporary repositories which do not change: ```go git.OpenRepository(ctx, diskPath) ``` Gitea managed repositories having a record in the database in the `repository` table are moved into the new package `gitrepo`: ```go gitrepo.OpenRepository(ctx, repo_model.Repo) ``` Why is `repo_model.Repository` the second parameter instead of file path? Because then we can easily adapt our repository storage strategy. The repositories can be stored locally, however, they could just as well be stored on a remote server. ## Further changes in other PRs - A Git Command wrapper on package `gitrepo` could be created. i.e. `NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir: repo.RepoPath()}`, the directory should be empty before invoking this method and it can be filled in the function only. #28940 - Remove the `RepoPath()`/`WikiPath()` functions to reduce the possibility of mistakes. --------- Co-authored-by: delvh <dev.lh@web.de>
* Preserve BOM in web editor (#28935)silverwind2024-01-272-2/+2
| | | | | | | | | | | | | | | | | | | The `ToUTF8*` functions were stripping BOM, while BOM is actually valid in UTF8, so the stripping must be optional depending on use case. This does: - Add a options struct to all `ToUTF8*` functions, that by default will strip BOM to preserve existing behaviour - Remove `ToUTF8` function, it was dead code - Rename `ToUTF8WithErr` to `ToUTF8` - Preserve BOM in Monaco Editor - Remove a unnecessary newline in the textarea value. Browsers did ignore it, it seems but it's better not to rely on this behaviour. Fixes: https://github.com/go-gitea/gitea/issues/28743 Related: https://github.com/go-gitea/gitea/issues/6716 which seems to have once introduced a mechanism that strips and re-adds the BOM, but from what I can tell, this mechanism was removed at some point after that PR.
* Fix sort bug on repository issues list (#28897)Lunny Xiao2024-01-231-2/+2
| | | Fix #28896
* Add option to disable ambiguous unicode characters detection (#28454)wxiaoguang2023-12-171-1/+2
| | | | | | | | * Close #24483 * Close #28123 * Close #23682 * Close #23149 (maybe more)
* Abstract hash function usage (#28138)Adam Majer2023-12-131-4/+13
| | | | | | Refactor Hash interfaces and centralize hash function. This will allow easier introduction of different hash function later on. This forms the "no-op" part of the SHA256 enablement patch.
* Include public repos in doer's dashboard for issue search (#28304)Jason Song2023-12-073-29/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It will fix #28268 . <img width="1313" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/cb1e07d5-7a12-4691-a054-8278ba255bfc"> <img width="1318" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/4fd60820-97f1-4c2c-a233-d3671a5039e9"> ## :warning: BREAKING :warning: But need to give up some features: <img width="1312" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/281c0d51-0e7d-473f-bbed-216e2f645610"> However, such abandonment may fix #28055 . ## Backgroud When the user switches the dashboard context to an org, it means they want to search issues in the repos that belong to the org. However, when they switch to themselves, it means all repos they can access because they may have created an issue in a public repo that they don't own. <img width="286" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/182dcd5b-1c20-4725-93af-96e8dfae5b97"> It's a confusing design. Think about this: What does "In your repositories" mean when the user switches to an org? Repos belong to the user or the org? Whatever, it has been broken by #26012 and its following PRs. After the PR, it searches for issues in repos that the dashboard context user owns or has been explicitly granted access to, so it causes #28268. ## How to fix it It's not really difficult to fix it. Just extend the repo scope to search issues when the dashboard context user is the doer. Since the user may create issues or be mentioned in any public repo, we can just set `AllPublic` to true, which is already supported by indexers. The DB condition will also support it in this PR. But the real difficulty is how to count the search results grouped by repos. It's something like "search issues with this keyword and those filters, and return the total number and the top results. **Then, group all of them by repo and return the counts of each group.**" <img width="314" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/5206eb20-f8f5-49b9-b45a-1be2fcf679f4"> Before #26012, it was being done in the DB, but it caused the results to be incomplete (see the description of #26012). And to keep this, #26012 implement it in an inefficient way, just count the issues by repo one by one, so it cannot work when `AllPublic` is true because it's almost impossible to do this for all public repos. https://github.com/go-gitea/gitea/blob/1bfcdeef4cca0f5509476358e5931c13d37ed1ca/modules/indexer/issues/indexer.go#L318-L338 ## Give up unnecessary features We may can resovle `TODO: use "group by" of the indexer engines to implement it`, I'm sure it can be done with Elasticsearch, but IIRC, Bleve and Meilisearch don't support "group by". And the real question is, does it worth it? Why should we need to know the counts grouped by repos? Let me show you my search dashboard on gitea.com. <img width="1304" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/2bca2d46-6c71-4de1-94cb-0c9af27c62ff"> I never think the long repo list helps anything. And if we agree to abandon it, things will be much easier. That is this PR. ## TODO I know it's important to filter by repos when searching issues. However, it shouldn't be the way we have it now. It could be implemented like this. <img width="1316" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/99ee5f21-cbb5-4dfe-914d-cb796cb79fbe"> The indexers support it well now, but it requires some frontend work, which I'm not good at. So, I think someone could help do that in another PR and merge this one to fix the bug first. Or please block this PR and help to complete it. Finally, "Switch dashboard context" is also a design that needs improvement. In my opinion, it can be accomplished by adding filtering conditions instead of "switching".
* Meilisearch: require all query terms to be matched (#28293)Brecht Van Lommel2023-11-291-4/+5
| | | | | | | | | | | | | Previously only the first term had to be matched. That default Meilisearch behavior makes sense for e.g. some kind of autocomplete to find and select a single result. But for filtering issues it means you can't narrow down results by adding more terms. This is also more consistent with other indexers and GitHub. --- Reference: https://www.meilisearch.com/docs/reference/api/search#matching-strategy
* Fix issue not showing on default board and add test (#27720)Nanguan Lin2023-10-251-0/+6
| | | | | | | | | | | | See https://github.com/go-gitea/gitea/pull/27718#issuecomment-1773743014 . Add a test to ensure its behavior. Why this test uses `ProjectBoardID=0`? Because in `SearchOptions`, `ProjectBoardID=0` means what it is. But in `IssueOptions`, `ProjectBoardID=0` means there is no condition, and `ProjectBoardID=db.NoConditionID` means the board ID = 0. It's really confusing. Probably it's better to separate the db search engine and the other issue search code. It's really two different systems. As far as I can see, `IssueOptions` is not necessary for most of the code, which has very simple issue search conditions.
* Clean some functions about project issue (#27705)Nanguan Lin2023-10-201-1/+0
| | | | | | | | | 1. remove unused function `MoveIssueAcrossProjectBoards` 2. extract the project board condition into a function 3. use db.NoCondition instead of -1. (BTW, the usage of db.NoCondition is too confusing. Is there any way to avoid that?) 4. remove the unnecessary comment since the ctx refactor is completed. 5. Change `b.ID != 0` to `b.ID > 0`. It's more intuitive but I think they're the same since board ID can't be negative.
* Improve retrying index issues (#27554)Jason Song2023-10-152-30/+41
| | | Fix #27540
* Replace assert.Fail with assert.FailNow (#27578)Nanguan Lin2023-10-111-4/+2
| | | | | | | | | assert.Fail() will continue to execute the code while assert.FailNow() not. I thought those uses of assert.Fail() should exit immediately. PS: perhaps it's a good idea to use [require](https://pkg.go.dev/github.com/stretchr/testify/require) somewhere because the assert package's default behavior does not exit when an error occurs, which makes it difficult to find the root error reason.
* Penultimate round of `db.DefaultContext` refactor (#27414)JakobDev2023-10-115-7/+9
| | | | | | | Part of #27065 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* make writing main test easier (#27270)Lunny Xiao2023-09-283-12/+3
| | | | | | | | | This PR removed `unittest.MainTest` the second parameter `TestOptions.GiteaRoot`. Now it detects the root directory by current working directory. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Fix review request number and add more tests (#27104)Nanguan Lin2023-09-211-6/+21
| | | | | | | | | | | | | | | | | | | | | fix #27019 ## testfixture yml 1. add issue20(a pr issue) in repo 23, org 17 2. add user15 to team 9 3. add four reviews about issue20 ## test case add two tests that are described with code comments the code before pr #26784 failed the first test <img width="479" alt="image" src="https://github.com/go-gitea/gitea/assets/70063547/1d9b5787-11b4-4c4d-931f-6a9869547f35"> current code failed the second test(as mentioned in #27019) <img width="484" alt="image" src="https://github.com/go-gitea/gitea/assets/70063547/05608055-7587-43d1-bae1-92c688270819"> Any advice is appreciated. --------- Co-authored-by: CaiCandong <50507092+CaiCandong@users.noreply.github.com> Co-authored-by: Giteabot <teabot@gitea.io>
* Next round of `db.DefaultContext` refactor (#27089)JakobDev2023-09-161-1/+1
| | | Part of #27065
* Add tests for db indexer in indexer_test.go (#27087)Nanguan Lin2023-09-161-131/+324
| | | | | | | | | | | | | | | | | | | | As described in the title. Some points: 1. Why need those tests? Because `buildIssueOverview` is not well tested, there are several continuous bugs in the issue overview webpage. 2. Why in indexer_test.go? It's hard to put those tests in `./modules/indexer/issue/db/db_test.go` because those tests need 'real' data in db mocked by fixtures instead of random data in `./modules/indexer/issue/internal/tests`. When using 'real' data(`unittest.PrepareTestDatabase`), `InitIssueIndexer` and the package `init()` function of `indexer` are required to init indexer. 3. Why only db? The other three indexer engines are well tested by random data and it's okay to also test them with 'real' data in db mocked by fixtures. Any follow-up PR is welcome. 4. Those tests are really basic, any more complicated tests are welcome. 5. I think it's also necessary to add tests in `TestAPISearchIssues` in`api_test_issue.go` and `TestIssues` in `home_test.go`
* Remove the useless function `GetUserIssueStats` and move relevant tests to ↵Nanguan Lin2023-09-141-1/+82
| | | | | | | | | | | | | | | | | | | `indexer_test.go` (#27067) Since the issue indexer has been refactored, the issue overview webpage is built by the `buildIssueOverview` function and underlying `indexer.Search` function and `GetIssueStats` instead of `GetUserIssueStats`. So the function is no longer used. I moved the relevant tests to `indexer_test.go` and since the search option changed from `IssueOptions` to `SearchOptions`, most of the tests are useless now. We need more tests about the db indexer because those tests are highly connected with the issue overview webpage and now this page has several bugs. Any advice about those test cases is appreciated. --------- Co-authored-by: CaiCandong <50507092+CaiCandong@users.noreply.github.com>
* Update status and code index after changing the default branch (#27018)Nanguan Lin2023-09-131-1/+8
| | | | | | | | | | | Fix #26723 Add `ChangeDefaultBranch` to the `notifier` interface and implement it in `indexerNotifier`. So when changing the default branch, `indexerNotifier` sends a message to the `indexer queue` to update the index. --------- Co-authored-by: techknowlogick <matti@mdranta.net>
* Clarify the git command Stdin hanging problem (#26967)wxiaoguang2023-09-081-15/+0
|
* move repository deletion to service layer (#26948)Lunny Xiao2023-09-083-0/+6
| | | Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Replace `util.SliceXxx` with `slices.Xxx` (#26958)CaiCandong2023-09-072-9/+10
|
* Use Go 1.21 and update dependencies (#26878)wxiaoguang2023-09-031-3/+2
| | | | | | To make sure Gitea's next release's lifecycle could have active Golang support. And min/max are builtin now.
* Fix counting and filtering on the dashboard page for issues (#26657)Jason Song2023-08-232-3/+51
| | | | | | | | | | | | | | | | | | This PR has multiple parts, and I didn't split them because it's not easy to test them separately since they are all about the dashboard page for issues. 1. Support counting issues via indexer to fix #26361 2. Fix repo selection so it also fixes #26653 3. Keep keywords in filter links. The first two are regressions of #26012. After: https://github.com/go-gitea/gitea/assets/9418365/71dfea7e-d9e2-42b6-851a-cc081435c946 Thanks to @CaiCandong for helping with some tests.
* Support rebuilding issue indexer manually (#26546)Jason Song2023-08-171-7/+15
| | | | | | | | | | | Provide a way to rebuild issue indexer manually. So if the indexer get outdated because of some bugs like #26539, we can rebuild it. <img width="1104" alt="image" src="https://github.com/go-gitea/gitea/assets/9418365/ac242e29-6f04-47ca-b3d0-801a796448d3"> Co-authored-by: Giteabot <teabot@gitea.io>
* Explain SearchOptions and fix ToSearchOptions (#26542)Jason Song2023-08-163-22/+35
| | | | | Follow #26012 #26490. A detailed description has been added to the comment.
* Fix project filter bugs (#26490)CaiCandong2023-08-151-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | related: #26012 ### Bugs 1. missing project filter on the issue page. https://github.com/go-gitea/gitea/blob/1e76a824bcd71acd59cdfb2c4547806bc34b3d86/modules/indexer/issues/dboptions.go#L11-L15 3. incorrect SQL condition: some issue does not belong to a project but exists on the project_issue table. https://github.com/go-gitea/gitea/blob/f5dbac9d36f1678b928bee04e85fbd045c725698/models/issues/issue_search.go#L233 ### Before: ![before](https://github.com/go-gitea/gitea/assets/50507092/1dcde39e-3e2f-4151-b2c6-4d67bf493c2f) ### After: ![after](https://github.com/go-gitea/gitea/assets/50507092/badfb81f-056d-4a2f-9838-1cba9c15768d) --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Refactor and enhance issue indexer to support both searching, filtering and ↵Jason Song2023-07-3121-352/+2301
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add open/closed field support for issue index (#25708)techknowlogick2023-07-079-30/+45
| | | | | | A couple of notes: * Future changes should refactor arguments into a struct * This filtering only is supported by meilisearch right now * Issue index number is bumped which will cause a re-index
* Replace `interface{}` with `any` (#25686)silverwind2023-07-046-12/+12
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Fix issues indexer document mapping (#25619)Jason Song2023-07-041-20/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix regression of #5363 (so long ago). The old code definded a document mapping for `issueIndexerDocType`, and assigned it to `BleveIndexerData` as its type. (`BleveIndexerData` has been renamed to `IndexerData` in #25174, but nothing more.) But the old code never used `BleveIndexerData`, it wrote the index with an anonymous struct type. Nonetheless, bleve would use the default auto-mapping for struct it didn't know, so the indexer still worked. This means the custom document mapping was always dead code. The custom document mapping is not useless, it can reduce index storage, this PR brings it back and disable default mapping to prevent it from happening again. Since `IndexerData`(`BleveIndexerData`) has JSON tags, and bleve uses them first, so we should use `repo_id` as the field name instead of `RepoID`. I did a test to compare the storage size before and after this, with about 3k real comments that were migrated from some public repos. Before: ```text [ 160] . ├── [ 42] index_meta.json ├── [ 13] rupture_meta.json └── [ 128] store ├── [6.9M] 00000000005d.zap └── [256K] root.bolt ``` After: ```text [ 160] . ├── [ 42] index_meta.json ├── [ 13] rupture_meta.json └── [ 128] store ├── [3.5M] 000000000065.zap └── [256K] root.bolt ``` It saves about half the storage space. --------- Co-authored-by: Giteabot <teabot@gitea.io>