summaryrefslogtreecommitdiffstats
path: root/modules/indexer/code
Commit message (Collapse)AuthorAgeFilesLines
* Make issue and code search support camel case (#22829)sillyguodong2023-02-121-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #22714 ### Changes: 1. Add a token filter which named "camelCase" between custom unicode token filter and "to_lower" token filter when add custom analyzer. ### Notice: If users want this feature to work, they should delete folder under {giteaPath}/data/indexers and restart application. Then application will create a new IndexMapping. ### Screenshots: ![image](https://user-images.githubusercontent.com/33891828/217715692-c18c41f2-57a1-4727-861c-470935c8e0c8.png) ### Others: I originally attempted to give users the ability to configure the "token_filters" in the "app.ini" file. But I found that if users does not strictly follow a right order to register "token_filters", they won't get the expected results. I think it is difficult to ask users to do this. So I finally give up this idea. --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* refactor some functions to support ctx as first parameter (#21878)Lunny Xiao2022-12-031-1/+1
| | | | Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: Lauris BH <lauris@nix.lv>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-279-18/+9
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Show syntax lexer name in file view/blame (#21814)silverwind2022-11-191-1/+4
| | | | | | | | | | | | | | | | | | | | | | Show which Chroma Lexer is used to highlight the file in the file header. It's useful for development to see what was detected, and I think it's not bad info to have for the user: <img width="233" alt="Screenshot 2022-11-14 at 22 31 16" src="https://user-images.githubusercontent.com/115237/201770854-44933dfc-70a4-487c-8457-1bb3cc43ea62.png"> <img width="226" alt="Screenshot 2022-11-14 at 22 36 06" src="https://user-images.githubusercontent.com/115237/201770856-9260ce6f-6c0f-442c-92b5-201e5b113188.png"> <img width="194" alt="Screenshot 2022-11-14 at 22 36 26" src="https://user-images.githubusercontent.com/115237/201770857-6f56591b-80ea-42cc-8ea5-21b9156c018b.png"> Also, I improved the way this header overflows on small screens: <img width="354" alt="Screenshot 2022-11-14 at 22 44 36" src="https://user-images.githubusercontent.com/115237/201774828-2ddbcde1-da15-403f-bf7a-6248449fa2c5.png"> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-242-2/+2
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor git command arguments and make all arguments to be safe to be used ↵wxiaoguang2022-10-233-7/+7
| | | | | | | (#21535) Follow #21464 Make all git command arguments strictly safe. Most changes are one-to-one replacing, keep all existing logic.
* test: use `T.TempDir` to create temporary test directory (#21043)Eng Zer Jun2022-09-041-9/+1
| | | | | | | | | | | | | | | | | | | | | | | | A testing cleanup. This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `T.TempDir` function from the `testing` package to create temporary directory. The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot. Reference: https://pkg.go.dev/testing#T.TempDir ```go func TestFoo(t *testing.T) { // before tmpDir, err := os.MkdirTemp("", "") require.NoError(t, err) defer os.RemoveAll(tmpDir) // now tmpDir := t.TempDir() } ``` Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Fix `dump-repo` git init, fix wrong error type for NullDownloader (#20182)wxiaoguang2022-07-011-1/+1
| | | | | * Fix `dump-repo` git init * fix wrong error type for NullDownloader
* Add more linters to improve code readability (#19989)Wim2022-06-202-2/+2
| | | | | | | | | | Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability - nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length. - unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions - wastedassign - https://github.com/sanposhiho/wastedassign - wastedassign finds wasted assignment statements. - notlintlint - Reports ill-formed or insufficient nolint directives - stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
* Move almost all functions' parameter db.Engine to context.Context (#19748)Lunny Xiao2022-05-202-2/+2
| | | | * Move almost all functions' parameter db.Engine to context.Context * remove some unnecessary wrap functions
* Prevent dangling archiver goroutine (#19516)zeripath2022-04-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | Within doArchive there is a service goroutine that performs the archiving function. This goroutine reports its error using a `chan error` called `done`. Prior to this PR this channel had 0 capacity meaning that the goroutine would block until the `done` channel was cleared - however there are a couple of ways in which this channel might not be read. The simplest solution is to add a single space of capacity to the goroutine which will mean that the goroutine will always complete and even if the `done` channel is not read it will be simply garbage collected away. (The PR also contains two other places when setting up the indexers which do not leak but where the blocking of the sending goroutine is also unnecessary and so we should just add a small amount of capacity and let the sending goroutine complete as soon as it can.) Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Use a struct as test options (#19393)Lunny Xiao2022-04-141-1/+3
| | | | | | | * Use a struct as test options * Fix name * Fix test
* Remove `git.Command.Run` and `git.Command.RunInDir*` (#19280)wxiaoguang2022-04-013-19/+22
| | | | | | Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions. * before: `stdout, err := RunInDir(path)` * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
* Add Goroutine stack inspector to admin/monitor (#19207)zeripath2022-03-311-1/+6
| | | | | | | | | | | | | | | | | | | Continues on from #19202. Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile. The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines. If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound. In this way we should be able to more easily examine goroutines that have been stuck. A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Lock gofumpt to v0.3.0 and run it (#18866)silverwind2022-02-231-1/+2
| | | | | We can't depend on `latest` version of gofumpt because the output will not be stable across versions. Lock it down to the latest version released yesterday and run it again.
* Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552)65432022-02-063-6/+6
|
* Automatically pause queue if index service is unavailable (#15066)Lauris BH2022-01-276-33/+180
| | | | | | * Handle keyword search error when issue indexer service is not available * Implement automatic disabling and resume of code indexer queue
* Pause queues (#15928)zeripath2022-01-221-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Start adding mechanism to return unhandled data Signed-off-by: Andrew Thornton <art27@cantab.net> * Create pushback interface Signed-off-by: Andrew Thornton <art27@cantab.net> * Add Pausable interface to WorkerPool and Manager Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and PushBack for the bytefifos Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues Signed-off-by: Andrew Thornton <art27@cantab.net> * Wire in UI for pausing Signed-off-by: Andrew Thornton <art27@cantab.net> * add testcases and fix a few issues Signed-off-by: Andrew Thornton <art27@cantab.net> * fix build Signed-off-by: Andrew Thornton <art27@cantab.net> * prevent "race" in the test Signed-off-by: Andrew Thornton <art27@cantab.net> * fix jsoniter mismerge Signed-off-by: Andrew Thornton <art27@cantab.net> * fix conflicts Signed-off-by: Andrew Thornton <art27@cantab.net> * fix format Signed-off-by: Andrew Thornton <art27@cantab.net> * Add warnings for no worker configurations and prevent data-loss with redis/levelqueue Signed-off-by: Andrew Thornton <art27@cantab.net> * Use StopTimer Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* format with gofumpt (#18184)65432022-01-206-60/+51
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Propagate context and ensure git commands run in request context (#17868)zeripath2022-01-196-31/+35
| | | | | | | | | This PR continues the work in #17125 by progressively ensuring that git commands run within the request context. This now means that the if there is a git repo already open in the context it will be used instead of reopening it. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Upgrade bleve from v2.0.6 to v2.3.0 (#18132)Lunny Xiao2022-01-011-1/+1
|
* Prevent hang in git cat-file if repository is not a valid repository and ↵zeripath2021-12-162-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | other fixes (#17991) This PR contains multiple fixes. The most important of which is: * Prevent hang in git cat-file if the repository is not a valid repository Unfortunately it appears that if git cat-file is run in an invalid repository it will hang until stdin is closed. This will result in deadlocked /pulls pages and dangling git cat-file calls if a broken repository is tried to be reviewed or pulls exists for a broken repository. Fix #14734 Fix #9271 Fix #16113 Otherwise there are a few small other fixes included which this PR was initially intending to fix: * Fix panic on partial compares due to missing PullRequestWorkInProgressPrefixes * Fix links on pulls pages due to regression from #17551 - by making most /issues routes match /pulls too - Fix #17983 * Fix links on feeds pages due to another regression from #17551 but also fix issue with syncing tags - Fix #17943 * Add missing locale entries for oauth group claims * Prevent NPEs if ColorFormat is called on nil users, repos or teams.
* Add missing `X-Total-Count` and fix some related bugs (#17968)qwerty2872021-12-151-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add missing `X-Total-Count` and fix some related bugs Adds `X-Total-Count` header to APIs that return a list but doesn't have it yet. Fixed bugs: * not returned after reporting error (https://github.com/qwerty287/gitea/blob/39eb82446c6fe5da3d79124e1f701f3795625b69/routers/api/v1/user/star.go#L70) * crash with index out of bounds, API issue/issueSubscriptions I also found various endpoints that return lists but do not apply/support pagination yet: ``` /repos/{owner}/{repo}/issues/{index}/labels /repos/{owner}/{repo}/issues/comments/{id}/reactions /repos/{owner}/{repo}/branch_protections /repos/{owner}/{repo}/contents /repos/{owner}/{repo}/hooks/git /repos/{owner}/{repo}/issue_templates /repos/{owner}/{repo}/releases/{id}/assets /repos/{owner}/{repo}/reviewers /repos/{owner}/{repo}/teams /user/emails /users/{username}/heatmap ``` If this is not expected, an new issue should be opened. Closes #13043 * fmt * Update routers/api/v1/repo/issue_subscription.go Co-authored-by: KN4CK3R <admin@oldschoolhack.me> * Use FindAndCount Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: 6543 <6543@obermui.de>
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-106-23/+24
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Make Requests Processes and create process hierarchy. Associate ↵zeripath2021-11-302-2/+2
| | | | | | | | | OpenRepository with context. (#17125) This PR registers requests with the process manager and manages hierarchy within the processes. Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add .gitattribute assisted language detection to blame, diff and render (#17590)zeripath2021-11-171-1/+1
| | | | | | | Use check attribute code to check the assigned language of a file and send that in to chroma as a hint for the language of the file. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix nil checking on typed interface (#17598)Gusted2021-11-153-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | * Fix nil checking on typed interface - Partially resoles #17596 - Resolves SA4023 errors. - Ensure correctly that typed interface are nil. * Remove unnecessary code `NewBleveIndexer` will never return nil, even on errors. * Patch `NewBleveIndexer` * Fix low-level functions * Remove deadcode * Fix GetSession * Close Elastic search when err isn't nil * Update elastic_search.go Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Decouple unit test code from business code (#17623)wxiaoguang2021-11-123-6/+8
|
* Make Repo Code Indexer an Unique Queue (#17515)zeripath2021-11-021-23/+11
| | | | | | | The functioning of the code indexer queue really only makes sense as an unique queue and doing this allows use to simplify the indexer data to simply delete the data if the repo is no longer in the db. Signed-off-by: Andrew Thornton <art27@cantab.net>
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-223-6/+4
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Move db related basic functions to models/db (#17075)Lunny Xiao2021-09-194-11/+10
| | | | | | | | | | | | | | | | | | | | | | | | | * Move db related basic functions to models/db * Fix lint * Fix lint * Fix test * Fix lint * Fix lint * revert unnecessary change * Fix test * Fix wrong replace string * Use *Context * Correct committer spelling and fix wrong replaced words Co-authored-by: zeripath <art27@cantab.net>
* Add an abstract json layout to make it's easier to change json library (#16528)Lunny Xiao2021-07-241-2/+1
| | | | | | | | | | | * Add an abstract json layout to make it's easier to change json library * Fix import * Fix import sequence * Fix blank lines * Fix blank lines
* Fix data race in bleve indexer (#16474)Lunny Xiao2021-07-181-4/+6
| | | * Fix data race in bleve indexer
* Fix various documentation, user-facing, and source comment typos (#16367)luzpaz2021-07-081-3/+3
| | | | | * Fix various doc, user-facing, and source comment typos Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
* Use git log name-status in get last commit (#16059)zeripath2021-06-212-0/+6
| | | | | | | | | | | | | | | | | * Improve get last commit using git log --name-status git log --name-status -c provides information about the diff between a commit and its parents. Using this and adjusting the algorithm to use the first change to a path allows for a much faster generation of commit info. There is a subtle change in the results generated but this will cause the results to more closely match those from elsewhere. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fixed assert statements. (#16089)KN4CK3R2021-06-071-1/+1
|
* Add Image Diff for SVG files (#14867)KN4CK3R2021-06-052-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added type sniffer. * Switched content detection from base to typesniffer. * Added GuessContentType to Blob. * Moved image info logic to client. Added support for SVG images in diff. * Restore old blocked svg behaviour. * Added missing image formats. * Execute image diff only when container is visible. * add margin to spinner * improve BIN tag on image diffs * Default to render view. * Show image diff on incomplete diff. Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
* Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow ↵zeripath2021-05-151-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | level queue, reduce goroutines etc (#15693) * move shutdownfns, terminatefns and hammerfns out of separate goroutines Coalesce the shutdownfns etc into a list of functions that get run at shutdown rather then have them run at goroutines blocked on selects. This may help reduce the background select/poll load in certain configurations. * The LevelDB queues can actually wait on empty instead of polling Slight refactor to cause leveldb queues to wait on empty instead of polling. * Shutdown the shadow level queue once it is empty * Remove bytefifo additional goroutine for readToChan as it can just be run in run * Remove additional removeWorkers goroutine for workers * Simplify the AtShutdown and AtTerminate functions and add Channel Flusher * Add shutdown flusher to CUQ * move persistable channel shutdown stuff to Shutdown Fn * Ensure that UPCQ has the correct config * handle shutdown during the flushing * reduce risk of race between zeroBoost and addWorkers * prevent double shutdown Signed-off-by: Andrew Thornton <art27@cantab.net>
* On open repository open common cat file batch and batch-check (#15667)zeripath2021-05-102-2/+2
| | | | | | | Use common git cat-file --batch and git cat-file --batch-check to significantly reduce calls to git. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Speed up `enry.IsVendor` (#15213)zeripath2021-04-012-2/+2
| | | | | | | | | `enry.IsVendor` is kinda slow as it simply iterates across all regexps. This PR ajdusts the regexps to combine them to make this process a little quicker. Related #15143 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move Bleve and Elastic code indexers to use a common cat-file --batch (#14781)zeripath2021-03-032-16/+52
| | | | | | | | | | | | | | | | * Extract out the common cat-file batch calls Signed-off-by: Andrew Thornton <art27@cantab.net> * Move bleve and elastic indexers to use a common cat-file --batch when indexing Signed-off-by: Andrew Thornton <art27@cantab.net> * move catfilebatch to batch_reader and rename to batch_reader.go Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv>
* Migrate to use jsoniter instead of encoding/json (#14841)zeripath2021-03-011-1/+2
| | | | | | | | | | * Migrate to use jsoniter * fix tests * update gitea.com/go-chi/binding Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Reduce calls to git cat-file -s (#14682)zeripath2021-02-173-16/+32
| | | | | | | | | | | | | * Reduce calls to git cat-file -s There are multiple places where there are repeated calls to git cat-file -s due to the blobs not being created with their size. Through judicious use of git ls-tree -l and slight adjustments to the indexer code we can avoid a lot of these calls. * simplify by always expecting the long format * Also always set the sized field and tell the indexer the update is sized
* [Feature] add precise search type for Elastic Search (#12869)Jui-Nan Lin2021-01-276-15/+35
| | | | | | | | * feat: add type query parameters for specifying precise search * feat: add select dropdown in search box Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Upgrade blevesearch dependency to v2.0.1 (#14346)Lauris BH2021-01-171-9/+9
| | | | | | | * Upgrade blevesearch dependency to v2.0.1 * Update rupture to v1.0.0 * Fix test
* Refactor Logger (#13294)zeripath2020-10-311-1/+1
| | | | | | Refactor Logger to make a logger interface and make it possible to wrap loggers for specific purposes. Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* fix: use Base36 for all code indexers (#12830)Jui-Nan Lin2020-09-142-6/+5
|
* Fix bug on elastic search (#12811)Lunny Xiao2020-09-122-15/+42
| | | | | | | | | | | * Fix bug on elastic search * Add more comments for elastic search result startIndex and endIndex * refactor indexPos * refactor indexPos * Fix bug
* Add queue for code indexer (#10332)Lunny Xiao2020-09-073-159/+158
| | | | | | | | | | | | | | | | | | | | | * Add queue for code indexer * Fix lint * Fix test * Fix lint * Fix bug * Fix bug * Fix lint * Add noqueue * Fix tests * Rename noqueue to immediate
* Support elastic search for code search (#10273)Lunny Xiao2020-08-308-157/+658
| | | | | | | | | | | | | | | | | | | | | | | * Support elastic search for code search * Finished elastic search implementation and add some tests * Enable test on drone and added docs * Add new fields to elastic search * Fix bug * remove unused changes * Use indexer alias to keep the gitea indexer version * Improve codes * Some code improvements * The real indexer name changed to xxx.v1 Co-authored-by: zeripath <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-16 15:51:49 +0000