aboutsummaryrefslogtreecommitdiffstats
path: root/modules/indexer/code
Commit message (Collapse)AuthorAgeFilesLines
* Run `gopls modernize` on codebase (#34751)silverwind2025-06-183-4/+4
| | | | Recent modernize fixes: https://github.com/golang/tools/commits/master/gopls/internal/analysis/modernize
* update go&js dependencies (#34262)wxiaoguang2025-04-231-1/+5
|
* Enable addtional linters (#34085)TheFox0x72025-04-012-6/+6
| | | | | | | | enable mirror, usestdlibbars and perfsprint part of: https://github.com/go-gitea/gitea/issues/34083 --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Enable testifylint rules (#34075)TheFox0x72025-03-312-3/+3
| | | | enable testifylint rules disabled in: https://github.com/go-gitea/gitea/pull/34054
* enable staticcheck QFxxxx rules (#34064)TheFox0x72025-03-291-2/+3
|
* Decouple Batch from git.Repository to simplify usage without requiring the ↵Lunny Xiao2025-03-272-14/+2
| | | | | creation of a Repository struct. (#34001) No logic change
* Fix incorrect code search indexer options (#33992)wxiaoguang2025-03-241-3/+2
| | | | | Fix #33798 Co-authored-by: Giteabot <teabot@gitea.io>
* Fix file name could not be searched if the file was not a text file when ↵charles2025-03-211-1/+2
| | | | | | | | | using the Bleve indexer (#33959) Close #33828 --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Make SearchMode have default value and add comments (#33863)wxiaoguang2025-03-143-4/+13
| | | | | | | * Make `SearchMode` have default value if it is empty * Add some comments for the "match" queries * Fix a copy-paste mistake in `buildMatchQuery` (`db.go`) * Add missing `q.Analyzer = repoIndexerAnalyzer`, it is in old code, although I do not see real difference ....
* Improve issue & code search (#33860)wxiaoguang2025-03-139-80/+66
| | | | Each "indexer" should provide the "search modes" they support by themselves. And we need to remove the "fuzzy" search for code.
* Remove context from git struct (#33793)TheFox0x72025-03-043-10/+10
| | | | Argument is moved from struct init in command run, which lets us remove context from struct.
* Use test context in tests and new loop system in benchmarks (#33648)TheFox0x72025-02-201-8/+7
| | | | | | | | Replace all contexts in tests with go1.24 t.Context() --------- Co-authored-by: Giteabot <teabot@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* enable literal string for code search (#33590)Darren Hoo2025-02-167-15/+147
| | | | | | | | Close: #33588 --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Giteabot <teabot@gitea.io>
* Fix bleve fuzziness search (#33078)wxiaoguang2025-01-032-3/+4
| | | Close #31565
* Fix markup render regression and fix some tests (#32640)wxiaoguang2024-11-261-2/+0
| | | | | | | Fix #32639, https://github.com/go-gitea/gitea/issues/32608#issuecomment-2497918210 By the way, fix some incorrect SQLs (use single quote but not double quote)
* Reduce integration test overhead (#32475)Rowan Bohde2024-11-141-7/+4
| | | | | | | | | | | | | In profiling integration tests, I found a couple places where per-test overhead could be reduced: * Avoiding disk IO by synchronizing instead of deleting & copying test Git repository data. This saves ~100ms per test on my machine * When flushing queues in `PrintCurrentTest`, invoke `FlushWithContext` in a parallel. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Updated tokenizer to better matching when search for code snippets (#32261)Bruno Sofiato2024-11-063-4/+63
| | | | | | | | | | | | | | | | | | | | This PR improves the accuracy of Gitea's code search. Currently, Gitea does not consider statements such as `onsole.log("hello")` as hits when the user searches for `log`. The culprit is how both ES and Bleve are tokenizing the file contents (in both cases, `console.log` is a whole token). In ES' case, we changed the tokenizer to [simple_pattern_split](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simplepatternsplit-tokenizer.html#:~:text=The%20simple_pattern_split%20tokenizer%20uses%20a,the%20tokenization%20is%20generally%20faster.). In such a case, tokens are words formed by digits and letters. In Bleve's case, it employs a [letter](https://blevesearch.com/docs/Tokenizers/) tokenizer. Resolves #32220 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
* Allow code search by filename (#32210)Bruno Sofiato2024-10-117-39/+463
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a large and complex PR, so let me explain in detail its changes. First, I had to create new index mappings for Bleve and ElasticSerach as the current ones do not support search by filename. This requires Gitea to recreate the code search indexes (I do not know if this is a breaking change, but I feel it deserves a heads-up). I've used [this approach](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/analysis-pathhierarchy-tokenizer.html) to model the filename index. It allows us to efficiently search for both the full path and the name of a file. Bleve, however, does not support this out-of-box, so I had to code a brand new [token filter](https://blevesearch.com/docs/Token-Filters/) to generate the search terms. I also did an overhaul in the `indexer_test.go` file. It now asserts the order of the expected results (this is important since matches based on the name of a file are more relevant than those based on its content). I've added new test scenarios that deal with searching by filename. They use a new repo included in the Gitea fixture. The screenshot below depicts how Gitea shows the search results. It shows results based on content in the same way as the current version does. In matches based on the filename, the first seven lines of the file contents are shown (BTW, this is how GitHub does it). ![image](https://github.com/user-attachments/assets/9d938d86-1a8d-4f89-8644-1921a473e858) Resolves #32096 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
* Fixed race condition when deleting documents by repoId in ElasticSearch (#32185)Bruno Sofiato2024-10-031-1/+27
| | | | | | | Resolves #32184 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
* Change the code search to sort results by relevance (#32134)Bruno Sofiato2024-09-282-2/+6
| | | | | Resolves #32129 Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
* Fix index too many file names bug (#31903)Lunny Xiao2024-09-011-7/+31
| | | | Try to fix #31884 Fix #28584
* Refactor the usage of batch catfile (#31754)Lunny Xiao2024-08-202-18/+22
| | | | | | | | | | When opening a repository, it will call `ensureValidRepository` and also `CatFileBatch`. But sometimes these will not be used until repository closed. So it's a waste of CPU to invoke 3 times git command for every open repository. This PR removed all of these from `OpenRepository` but only kept checking whether the folder exists. When a batch is necessary, the necessary functions will be invoked.
* Fix bleve fuzziness (#30799)wxiaoguang2024-05-011-3/+1
| | | | Fix #30797 Fix #30317
* Fix tautological conditions (#30735)silverwind2024-04-301-6/+0
| | | | | | | As discovered by https://github.com/go-gitea/gitea/pull/30729. --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Resolve lint for unused parameter and unnecessary type arguments (#30750)Chongyi Zheng2024-04-291-8/+4
| | | | | | | | | | Resolve all cases for `unused parameter` and `unnecessary type arguments` Related: #30729 --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Enable more `revive` linter rules (#30608)silverwind2024-04-221-2/+0
| | | | | | | | | | | Noteable additions: - `redefines-builtin-id` forbid variable names that shadow go builtins - `empty-lines` remove unnecessary empty lines that `gofumpt` does not remove for some reason - `superfluous-else` eliminate more superfluous `else` branches Rules are also sorted alphabetically and I cleaned up various parts of `.golangci.yml`.
* Render embedded code preview by permlink in markdown (#30234)wxiaoguang2024-04-021-7/+9
| | | | | The permlink in markdown will be rendered as a code preview block, like GitHub Co-authored-by: silverwind <me@silverwind.io>
* Support repo code search without setting up an indexer (#29998)wxiaoguang2024-03-241-17/+18
| | | | | | | | | | | | | | | | | By using git's ability, end users (especially small instance users) do not need to enable the indexer, they could also benefit from the code searching feature. Fix #29996 ![image](https://github.com/go-gitea/gitea/assets/2114189/11b7e458-88a4-480d-b4d7-72ee59406dd1) ![image](https://github.com/go-gitea/gitea/assets/2114189/0fe777d5-c95c-4288-a818-0427680805b6) --------- Co-authored-by: silverwind <me@silverwind.io>
* Determine fuzziness of bleve indexer by keyword length (#29706)65432024-03-231-8/+7
| | | also bleve did match on fuzzy search and the other way around. this also fix that bug.
* Refactor code_indexer to use an SearchOptions struct for PerformSearch (#29724)65432024-03-166-35/+53
| | | | | | | | similar to how it's already done for the issue_indexer --- *Sponsored by Kithara Software GmbH*
* Use repo object format name instead of detecting from git repository (#29702)Lunny Xiao2024-03-101-8/+4
| | | | It's unnecessary to detect the repository object format from git repository. Just use the repository's object format name.
* Patch in exact search for meilisearch (#29671)65432024-03-095-15/+16
| | | | | | | | | | | | | | | | | | | meilisearch does not have an search option to contorl fuzzynes per query right now: - https://github.com/meilisearch/meilisearch/issues/1192 - https://github.com/orgs/meilisearch/discussions/377 - https://github.com/meilisearch/meilisearch/discussions/1096 so we have to create a workaround by post-filter the search result in gitea until this is addressed. For future works I added an option in backend only atm, to enable fuzzynes for issue indexer too. And also refactored the code so the fuzzy option is equal in logic to code indexer --- *Sponsored by Kithara Software GmbH*
* Fix wrong line number in code search result (#29260)yp053272024-03-061-19/+31
| | | | | | | | | | | Fix #29136 Before: The result is a table and all line numbers are all in one row. After: Use a separate table column for the line numbers. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* fix: Elasticsearch: Request Entity Too Large #28117 (#29062)dark-angel2024-02-071-5/+11
| | | | | | | | | | | | | | | | Fix for gitea putting everything into one request without batching and sending it to Elasticsearch for indexing as issued in #28117 This issue occured in large repositories while Gitea tries to index the code using ElasticSearch. I've applied necessary changes that takes batch length from below config (app.ini) ``` [queue.code_indexer] BATCH_LENGTH=<length_int> ``` and batches all requests to Elasticsearch in chunks as configured in the above config
* Preserve BOM in web editor (#28935)silverwind2024-01-272-2/+2
| | | | | | | | | | | | | | | | | | | The `ToUTF8*` functions were stripping BOM, while BOM is actually valid in UTF8, so the stripping must be optional depending on use case. This does: - Add a options struct to all `ToUTF8*` functions, that by default will strip BOM to preserve existing behaviour - Remove `ToUTF8` function, it was dead code - Rename `ToUTF8WithErr` to `ToUTF8` - Preserve BOM in Monaco Editor - Remove a unnecessary newline in the textarea value. Browsers did ignore it, it seems but it's better not to rely on this behaviour. Fixes: https://github.com/go-gitea/gitea/issues/28743 Related: https://github.com/go-gitea/gitea/issues/6716 which seems to have once introduced a mechanism that strips and re-adds the BOM, but from what I can tell, this mechanism was removed at some point after that PR.
* Add option to disable ambiguous unicode characters detection (#28454)wxiaoguang2023-12-171-1/+2
| | | | | | | | * Close #24483 * Close #28123 * Close #23682 * Close #23149 (maybe more)
* Abstract hash function usage (#28138)Adam Majer2023-12-131-4/+13
| | | | | | Refactor Hash interfaces and centralize hash function. This will allow easier introduction of different hash function later on. This forms the "no-op" part of the SHA256 enablement patch.
* Replace assert.Fail with assert.FailNow (#27578)Nanguan Lin2023-10-111-4/+2
| | | | | | | | | assert.Fail() will continue to execute the code while assert.FailNow() not. I thought those uses of assert.Fail() should exit immediately. PS: perhaps it's a good idea to use [require](https://pkg.go.dev/github.com/stretchr/testify/require) somewhere because the assert package's default behavior does not exit when an error occurs, which makes it difficult to find the root error reason.
* Penultimate round of `db.DefaultContext` refactor (#27414)JakobDev2023-10-111-1/+1
| | | | | | | Part of #27065 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* make writing main test easier (#27270)Lunny Xiao2023-09-281-4/+1
| | | | | | | | | This PR removed `unittest.MainTest` the second parameter `TestOptions.GiteaRoot`. Now it detects the root directory by current working directory. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Update status and code index after changing the default branch (#27018)Nanguan Lin2023-09-131-1/+8
| | | | | | | | | | | Fix #26723 Add `ChangeDefaultBranch` to the `notifier` interface and implement it in `indexerNotifier`. So when changing the default branch, `indexerNotifier` sends a message to the `indexer queue` to update the index. --------- Co-authored-by: techknowlogick <matti@mdranta.net>
* Clarify the git command Stdin hanging problem (#26967)wxiaoguang2023-09-081-15/+0
|
* move repository deletion to service layer (#26948)Lunny Xiao2023-09-081-0/+2
| | | Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Replace `util.SliceXxx` with `slices.Xxx` (#26958)CaiCandong2023-09-071-5/+5
|
* Use Go 1.21 and update dependencies (#26878)wxiaoguang2023-09-031-3/+2
| | | | | | To make sure Gitea's next release's lifecycle could have active Golang support. And min/max are builtin now.
* Refactor and enhance issue indexer to support both searching, filtering and ↵Jason Song2023-07-311-11/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Replace `interface{}` with `any` (#25686)silverwind2023-07-042-5/+5
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Refactor indexer (#25174)Jason Song2023-06-2313-623/+331
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactor `modules/indexer` to make it more maintainable. And it can be easier to support more features. I'm trying to solve some of issue searching, this is a precursor to making functional changes. Current supported engines and the index versions: | engines | issues | code | | - | - | - | | db | Just a wrapper for database queries, doesn't need version | - | | bleve | The version of index is **2** | The version of index is **6** | | elasticsearch | The old index has no version, will be treated as version **0** in this PR | The version of index is **1** | | meilisearch | The old index has no version, will be treated as version **0** in this PR | - | ## Changes ### Split Splited it into mutiple packages ```text indexer ├── internal │   ├── bleve │   ├── db │   ├── elasticsearch │   └── meilisearch ├── code │   ├── bleve │   ├── elasticsearch │   └── internal └── issues ├── bleve ├── db ├── elasticsearch ├── internal └── meilisearch ``` - `indexer/interanal`: Internal shared package for indexer. - `indexer/interanal/[engine]`: Internal shared package for each engine (bleve/db/elasticsearch/meilisearch). - `indexer/code`: Implementations for code indexer. - `indexer/code/internal`: Internal shared package for code indexer. - `indexer/code/[engine]`: Implementation via each engine for code indexer. - `indexer/issues`: Implementations for issues indexer. ### Deduplication - Combine `Init/Ping/Close` for code indexer and issues indexer. - ~Combine `issues.indexerHolder` and `code.wrappedIndexer` to `internal.IndexHolder`.~ Remove it, use dummy indexer instead when the indexer is not ready. - Duplicate two copies of creating ES clients. - Duplicate two copies of `indexerID()`. ### Enhancement - [x] Support index version for elasticsearch issues indexer, the old index without version will be treated as version 0. - [x] Fix spell of `elastic_search/ElasticSearch`, it should be `Elasticsearch`. - [x] Improve versioning of ES index. We don't need `Aliases`: - Gitea does't need aliases for "Zero Downtime" because it never delete old indexes. - The old code of issues indexer uses the orignal name to create issue index, so it's tricky to convert it to an alias. - [x] Support index version for meilisearch issues indexer, the old index without version will be treated as version 0. - [x] Do "ping" only when `Ping` has been called, don't ping periodically and cache the status. - [x] Support the context parameter whenever possible. - [x] Fix outdated example config. - [x] Give up the requeue logic of issues indexer: When indexing fails, call Ping to check if it was caused by the engine being unavailable, and only requeue the task if the engine is unavailable. - It is fragile and tricky, could cause data losing (It did happen when I was doing some tests for this PR). And it works for ES only. - Just always requeue the failed task, if it caused by bad data, it's a bug of Gitea which should be fixed. --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Improve queue and logger context (#24924)wxiaoguang2023-05-261-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | Before there was a "graceful function": RunWithShutdownFns, it's mainly for some modules which doesn't support context. The old queue system doesn't work well with context, so the old queues need it. After the queue refactoring, the new queue works with context well, so, use Golang context as much as possible, the `RunWithShutdownFns` could be removed (replaced by RunWithCancel for context cancel mechanism), the related code could be simplified. This PR also fixes some legacy queue-init problems, eg: * typo : archiver: "unable to create codes indexer queue" => "unable to create repo-archive queue" * no nil check for failed queues, which causes unfriendly panic After this PR, many goroutines could have better display name: ![image](https://github.com/go-gitea/gitea/assets/2114189/701b2a9b-8065-4137-aeaa-0bda2b34604a) ![image](https://github.com/go-gitea/gitea/assets/2114189/f1d5f50f-0534-40f0-b0be-f2c9daa5fe92)
* Allow skipping forks and mirrors from being indexed (#23187)techknowlogick2023-05-251-0/+27
| | | | | | This PR adds two new options to disable repo/code search indexing of both forks and mirrors. Related: #22842
generated by cgit v1.2.3 (git 2.39.1) at 2025-08-08 04:30:30 +0000