summaryrefslogtreecommitdiffstats
path: root/modules/indexer/code/indexer.go
Commit message (Collapse)AuthorAgeFilesLines
* Move almost all functions' parameter db.Engine to context.Context (#19748)Lunny Xiao2022-05-201-1/+1
| | | | * Move almost all functions' parameter db.Engine to context.Context * remove some unnecessary wrap functions
* Prevent dangling archiver goroutine (#19516)zeripath2022-04-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | Within doArchive there is a service goroutine that performs the archiving function. This goroutine reports its error using a `chan error` called `done`. Prior to this PR this channel had 0 capacity meaning that the goroutine would block until the `done` channel was cleared - however there are a couple of ways in which this channel might not be read. The simplest solution is to add a single space of capacity to the goroutine which will mean that the goroutine will always complete and even if the `done` channel is not read it will be simply garbage collected away. (The PR also contains two other places when setting up the indexers which do not leak but where the blocking of the sending goroutine is also unnecessary and so we should just add a small amount of capacity and let the sending goroutine complete as soon as it can.) Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Add Goroutine stack inspector to admin/monitor (#19207)zeripath2022-03-311-1/+6
| | | | | | | | | | | | | | | | | | | Continues on from #19202. Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile. The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines. If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound. In this way we should be able to more easily examine goroutines that have been stuck. A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Automatically pause queue if index service is unavailable (#15066)Lauris BH2022-01-271-3/+33
| | | | | | * Handle keyword search error when issue indexer service is not available * Implement automatic disabling and resume of code indexer queue
* Pause queues (#15928)zeripath2022-01-221-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Start adding mechanism to return unhandled data Signed-off-by: Andrew Thornton <art27@cantab.net> * Create pushback interface Signed-off-by: Andrew Thornton <art27@cantab.net> * Add Pausable interface to WorkerPool and Manager Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and PushBack for the bytefifos Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues Signed-off-by: Andrew Thornton <art27@cantab.net> * Wire in UI for pausing Signed-off-by: Andrew Thornton <art27@cantab.net> * add testcases and fix a few issues Signed-off-by: Andrew Thornton <art27@cantab.net> * fix build Signed-off-by: Andrew Thornton <art27@cantab.net> * prevent "race" in the test Signed-off-by: Andrew Thornton <art27@cantab.net> * fix jsoniter mismerge Signed-off-by: Andrew Thornton <art27@cantab.net> * fix conflicts Signed-off-by: Andrew Thornton <art27@cantab.net> * fix format Signed-off-by: Andrew Thornton <art27@cantab.net> * Add warnings for no worker configurations and prevent data-loss with redis/levelqueue Signed-off-by: Andrew Thornton <art27@cantab.net> * Use StopTimer Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* format with gofumpt (#18184)65432022-01-201-3/+1
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Propagate context and ensure git commands run in request context (#17868)zeripath2022-01-191-6/+6
| | | | | | | | | This PR continues the work in #17125 by progressively ensuring that git commands run within the request context. This now means that the if there is a git repo already open in the context it will be used instead of reopening it. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-101-7/+7
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Fix nil checking on typed interface (#17598)Gusted2021-11-151-6/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | * Fix nil checking on typed interface - Partially resoles #17596 - Resolves SA4023 errors. - Ensure correctly that typed interface are nil. * Remove unnecessary code `NewBleveIndexer` will never return nil, even on errors. * Patch `NewBleveIndexer` * Fix low-level functions * Remove deadcode * Fix GetSession * Close Elastic search when err isn't nil * Update elastic_search.go Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Make Repo Code Indexer an Unique Queue (#17515)zeripath2021-11-021-23/+11
| | | | | | | The functioning of the code indexer queue really only makes sense as an unique queue and doing this allows use to simplify the indexer data to simply delete the data if the repo is no longer in the db. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move db related basic functions to models/db (#17075)Lunny Xiao2021-09-191-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | * Move db related basic functions to models/db * Fix lint * Fix lint * Fix test * Fix lint * Fix lint * revert unnecessary change * Fix test * Fix wrong replace string * Use *Context * Correct committer spelling and fix wrong replaced words Co-authored-by: zeripath <art27@cantab.net>
* Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow ↵zeripath2021-05-151-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | level queue, reduce goroutines etc (#15693) * move shutdownfns, terminatefns and hammerfns out of separate goroutines Coalesce the shutdownfns etc into a list of functions that get run at shutdown rather then have them run at goroutines blocked on selects. This may help reduce the background select/poll load in certain configurations. * The LevelDB queues can actually wait on empty instead of polling Slight refactor to cause leveldb queues to wait on empty instead of polling. * Shutdown the shadow level queue once it is empty * Remove bytefifo additional goroutine for readToChan as it can just be run in run * Remove additional removeWorkers goroutine for workers * Simplify the AtShutdown and AtTerminate functions and add Channel Flusher * Add shutdown flusher to CUQ * move persistable channel shutdown stuff to Shutdown Fn * Ensure that UPCQ has the correct config * handle shutdown during the flushing * reduce risk of race between zeroBoost and addWorkers * prevent double shutdown Signed-off-by: Andrew Thornton <art27@cantab.net>
* [Feature] add precise search type for Elastic Search (#12869)Jui-Nan Lin2021-01-271-1/+1
| | | | | | | | * feat: add type query parameters for specifying precise search * feat: add select dropdown in search box Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* fix: use Base36 for all code indexers (#12830)Jui-Nan Lin2020-09-141-1/+5
|
* Add queue for code indexer (#10332)Lunny Xiao2020-09-071-5/+153
| | | | | | | | | | | | | | | | | | | | | * Add queue for code indexer * Fix lint * Fix test * Fix lint * Fix bug * Fix bug * Fix lint * Add noqueue * Fix tests * Rename noqueue to immediate
* Support elastic search for code search (#10273)Lunny Xiao2020-08-301-19/+71
| | | | | | | | | | | | | | | | | | | | | | | * Support elastic search for code search * Finished elastic search implementation and add some tests * Enable test on drone and added docs * Add new fields to elastic search * Fix bug * remove unused changes * Use indexer alias to keep the gitea indexer version * Improve codes * Some code improvements * The real indexer name changed to xxx.v1 Co-authored-by: zeripath <art27@cantab.net>
* Log the indexer path on failure (#11172)zeripath2020-04-221-1/+1
| | | | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
* Handle panic in indexer initialisation better (#10534)zeripath2020-02-281-0/+11
| | | | | * Handle panic in indexer initialisation better * as per @guillep2k
* Add detected file language to code search (#10256)Lauris BH2020-02-201-6/+18
| | | | | | | Move langauge detection to separate module to be more reusable Add option to disable vendored file exclusion from file search Allways show all language stats for search
* fix datarace on issue indexer queue (#9490)Lunny Xiao2019-12-251-0/+2
|
* Wrap the code indexer (#9476)zeripath2019-12-241-11/+38
| | | | | | | | | | | | | | | | | | | | * Wrap the code indexer In order to prevent a data race in the code indexer it must be wrapped with a holder otherwise it is possible to Search/Index on an incompletely initialised indexer, and search will fail with a nil pointer until the repository indexer is initialised. Further a completely initialised repository indexer should not be closed until Termination otherwise actions in Hammer/Shutdown phases could block or be lost. Finally, there is a complex dance of shutdown etiquette should the index initialisation fail. This PR restores that. * Always return err if closed whilst waiting Co-authored-by: techknowlogick <matti@mdranta.net>
* Refactor code indexer (#9313)Lunny Xiao2019-12-231-53/+54
| | | | | | | | | | | | | | | | | | | | * Refactor code indexer * fix test * fix test * refactor code indexer * fix import * improve code * fix typo * fix test and make code clean * fix lint
* Move modules/indexer to modules/indexer/code (#9301)Lunny Xiao2019-12-101-0/+76
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-19 01:15:35 +0000