aboutsummaryrefslogtreecommitdiffstats
path: root/modules/indexer
Commit message (Collapse)AuthorAgeFilesLines
* Add meilisearch support (#23136)techknowlogick2023-03-282-1/+199
| | | | | Add meilisearch support Fixes #20665
* Refactor the setting to make unit test easier (#22405)Lunny Xiao2023-02-202-5/+5
| | | | | | | | | | | | | | | | | | | | | | Some bugs caused by less unit tests in fundamental packages. This PR refactor `setting` package so that create a unit test will be easier than before. - All `LoadFromXXX` files has been splited as two functions, one is `InitProviderFromXXX` and `LoadCommonSettings`. The first functions will only include the code to create or new a ini file. The second function will load common settings. - It also renames all functions in setting from `newXXXService` to `loadXXXSetting` or `loadXXXFrom` to make the function name less confusing. - Move `XORMLog` to `SQLLog` because it's a better name for that. Maybe we should finally move these `loadXXXSetting` into the `XXXInit` function? Any idea? --------- Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: delvh <dev.lh@web.de>
* Make issue and code search support camel case (#22829)sillyguodong2023-02-122-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #22714 ### Changes: 1. Add a token filter which named "camelCase" between custom unicode token filter and "to_lower" token filter when add custom analyzer. ### Notice: If users want this feature to work, they should delete folder under {giteaPath}/data/indexers and restart application. Then application will create a new IndexMapping. ### Screenshots: ![image](https://user-images.githubusercontent.com/33891828/217715692-c18c41f2-57a1-4727-861c-470935c8e0c8.png) ### Others: I originally attempted to give users the ability to configure the "token_filters" in the "app.ini" file. But I found that if users does not strictly follow a right order to register "token_filters", they won't get the expected results. I think it is difficult to ask users to do this. So I finally give up this idea. --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* refactor some functions to support ctx as first parameter (#21878)Lunny Xiao2022-12-033-3/+3
| | | | Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: Lauris BH <lauris@nix.lv>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-2720-40/+20
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Show syntax lexer name in file view/blame (#21814)silverwind2022-11-191-1/+4
| | | | | | | | | | | | | | | | | | | | | | Show which Chroma Lexer is used to highlight the file in the file header. It's useful for development to see what was detected, and I think it's not bad info to have for the user: <img width="233" alt="Screenshot 2022-11-14 at 22 31 16" src="https://user-images.githubusercontent.com/115237/201770854-44933dfc-70a4-487c-8457-1bb3cc43ea62.png"> <img width="226" alt="Screenshot 2022-11-14 at 22 36 06" src="https://user-images.githubusercontent.com/115237/201770856-9260ce6f-6c0f-442c-92b5-201e5b113188.png"> <img width="194" alt="Screenshot 2022-11-14 at 22 36 26" src="https://user-images.githubusercontent.com/115237/201770857-6f56591b-80ea-42cc-8ea5-21b9156c018b.png"> Also, I improved the way this header overflows on small screens: <img width="354" alt="Screenshot 2022-11-14 at 22 44 36" src="https://user-images.githubusercontent.com/115237/201774828-2ddbcde1-da15-403f-bf7a-6248449fa2c5.png"> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Add `context.Context` to more methods (#21546)KN4CK3R2022-11-191-9/+9
| | | | | | | This PR adds a context parameter to a bunch of methods. Some helper `xxxCtx()` methods got replaced with the normal name now. Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-243-3/+3
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor git command arguments and make all arguments to be safe to be used ↵wxiaoguang2022-10-233-7/+7
| | | | | | | (#21535) Follow #21464 Make all git command arguments strictly safe. Most changes are one-to-one replacing, keep all existing logic.
* test: use `T.TempDir` to create temporary test directory (#21043)Eng Zer Jun2022-09-043-28/+4
| | | | | | | | | | | | | | | | | | | | | | | | A testing cleanup. This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `T.TempDir` function from the `testing` package to create temporary directory. The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot. Reference: https://pkg.go.dev/testing#T.TempDir ```go func TestFoo(t *testing.T) { // before tmpDir, err := os.MkdirTemp("", "") require.NoError(t, err) defer os.RemoveAll(tmpDir) // now tmpDir := t.TempDir() } ``` Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Fix `dump-repo` git init, fix wrong error type for NullDownloader (#20182)wxiaoguang2022-07-011-1/+1
| | | | | * Fix `dump-repo` git init * fix wrong error type for NullDownloader
* Add more linters to improve code readability (#19989)Wim2022-06-202-2/+2
| | | | | | | | | | Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability - nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length. - unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions - wastedassign - https://github.com/sanposhiho/wastedassign - wastedassign finds wasted assignment statements. - notlintlint - Reports ill-formed or insufficient nolint directives - stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
* Move issues related files into models/issues (#19931)Lunny Xiao2022-06-132-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Move access and repo permission to models/perm/access * fix test * fix git test * Move functions sequence * Some improvements per @KN4CK3R and @delvh * Move issues related code to models/issues * Move some issues related sub package * Merge * Fix test * Fix test * Fix test * Fix test * Rename some files
* fix: some typos (#19956)yutotnh2022-06-131-1/+1
|
* Fix data-race problems in git module (quick patch) (#19934)wxiaoguang2022-06-111-5/+0
| | | | | | | * Fix data-race problems in git module * use HomeDir instead of setting.RepoRootPath Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Refactor git module, make Gitea use internal git config (#19732)wxiaoguang2022-06-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Refactor git module, make Gitea use internal git config, add safe.directory config * introduce git.InitSimple and git.InitWithConfigSync, make serv cmd use gitconfig * use HOME instead of GIT_CONFIG_GLOBAL, because git always needs a correct HOME * fix cmd env in cmd/serv.go * fine tune error message * Fix a incorrect test case * fix configAddNonExist * fix configAddNonExist logic, add `--fixed-value` flag, add tests * add configSetNonExist function in case it's needed. * use configSetNonExist for `user.name` and `user.email` * add some comments * Update cmd/serv.go Co-authored-by: zeripath <art27@cantab.net> * Update cmd/serv.go Co-authored-by: zeripath <art27@cantab.net> * Update modules/git/git.go Co-authored-by: zeripath <art27@cantab.net> * Update modules/setting/setting.go Co-authored-by: zeripath <art27@cantab.net> * Update modules/git/repo_attribute.go Co-authored-by: zeripath <art27@cantab.net> * fix spaces in messages * use `configSet("core.protectNTFS", ...)` instead of `globalCommandArgs` * remove GIT_CONFIG_NOSYSTEM, continue to use system's git config * Update cmd/serv.go Co-authored-by: zeripath <art27@cantab.net> * fix merge * remove code for safe.directory * separate git.CommonEnvs to CommonGitCmdEnvs and CommonCmdServEnvs * avoid Golang's data race error Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Move some repository related code into sub package (#19711)Lunny Xiao2022-06-061-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move some repository related code into sub package * Move more repository functions out of models * Fix lint * Some performance optimization for webhooks and others * some refactors * Fix lint * Fix * Update modules/repository/delete.go Co-authored-by: delvh <dev.lh@web.de> * Fix test * Merge * Fix test * Fix test * Fix test * Fix test Co-authored-by: delvh <dev.lh@web.de>
* Move almost all functions' parameter db.Engine to context.Context (#19748)Lunny Xiao2022-05-205-5/+6
| | | | * Move almost all functions' parameter db.Engine to context.Context * remove some unnecessary wrap functions
* Prevent dangling archiver goroutine (#19516)zeripath2022-04-262-2/+2
| | | | | | | | | | | | | | | | | | | | | | Within doArchive there is a service goroutine that performs the archiving function. This goroutine reports its error using a `chan error` called `done`. Prior to this PR this channel had 0 capacity meaning that the goroutine would block until the `done` channel was cleared - however there are a couple of ways in which this channel might not be read. The simplest solution is to add a single space of capacity to the goroutine which will mean that the goroutine will always complete and even if the `done` channel is not read it will be simply garbage collected away. (The PR also contains two other places when setting up the indexers which do not leak but where the blocking of the sending goroutine is also unnecessary and so we should just add a small amount of capacity and let the sending goroutine complete as soon as it can.) Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* User specific repoID or xorm builder conditions for issue search (#19475)65432022-04-251-1/+1
| | | | | | | | | * extend models.IssuesOptions to have more specific repo filter options * use new options * unrelated refactor * rm RepoIDs
* Use a struct as test options (#19393)Lunny Xiao2022-04-143-3/+9
| | | | | | | * Use a struct as test options * Fix name * Fix test
* Remove `git.Command.Run` and `git.Command.RunInDir*` (#19280)wxiaoguang2022-04-013-19/+22
| | | | | | Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions. * before: `stdout, err := RunInDir(path)` * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
* Add Goroutine stack inspector to admin/monitor (#19207)zeripath2022-03-312-1/+18
| | | | | | | | | | | | | | | | | | | Continues on from #19202. Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile. The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines. If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound. In this way we should be able to more easily examine goroutines that have been stuck. A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Make git.OpenRepository accept Context (#19260)65432022-03-301-1/+1
| | | | | * OpenRepositoryCtx -> OpenRepository * OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
* Prevent intermittent failures in RepoIndexerTest (2) (#19229)zeripath2022-03-271-0/+5
| | | | | | | | | | | So whilst #19225 fixes one issue it caused another. We need to initialise the Git module first. Related #19225 Fix #19162 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Prevent intermittent failures in RepoIndexerTest (#19225)zeripath2022-03-271-2/+8
| | | | | | | | | | | The RepoIndexerTest is failing with considerable frequency due to a race inherrent in its design. This PR adjust this test to avoid the reliance on waiting for the populate repo indexer to run and forcibly adds the repo to the queue. It then flushes the queue. It may be worth separating out the tests somewhat by testing the Index function directly away from the queue however, this forceful method should solve the current problem. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent Stats Indexer reporting error if repo dir missing (#18870)zeripath2022-02-241-0/+3
| | | | | | | | | | Repositories missing their directory should not report an error from the stats indexer. Close #18847 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Lock gofumpt to v0.3.0 and run it (#18866)silverwind2022-02-231-1/+2
| | | | | We can't depend on `latest` version of gofumpt because the output will not be stable across versions. Lock it down to the latest version released yesterday and run it again.
* Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552)65432022-02-063-6/+6
|
* Automatically pause queue if index service is unavailable (#15066)Lauris BH2022-01-2712-74/+379
| | | | | | * Handle keyword search error when issue indexer service is not available * Implement automatic disabling and resume of code indexer queue
* Pause queues (#15928)zeripath2022-01-223-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Start adding mechanism to return unhandled data Signed-off-by: Andrew Thornton <art27@cantab.net> * Create pushback interface Signed-off-by: Andrew Thornton <art27@cantab.net> * Add Pausable interface to WorkerPool and Manager Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and PushBack for the bytefifos Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues Signed-off-by: Andrew Thornton <art27@cantab.net> * Wire in UI for pausing Signed-off-by: Andrew Thornton <art27@cantab.net> * add testcases and fix a few issues Signed-off-by: Andrew Thornton <art27@cantab.net> * fix build Signed-off-by: Andrew Thornton <art27@cantab.net> * prevent "race" in the test Signed-off-by: Andrew Thornton <art27@cantab.net> * fix jsoniter mismerge Signed-off-by: Andrew Thornton <art27@cantab.net> * fix conflicts Signed-off-by: Andrew Thornton <art27@cantab.net> * fix format Signed-off-by: Andrew Thornton <art27@cantab.net> * Add warnings for no worker configurations and prevent data-loss with redis/levelqueue Signed-off-by: Andrew Thornton <art27@cantab.net> * Use StopTimer Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* format with gofumpt (#18184)65432022-01-2012-107/+89
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Enable deprecation error for v1.17.0 (#18341)Gusted2022-01-202-4/+4
| | | Co-authored-by: Andrew Thornton <art27@cantab.net>
* Propagate context and ensure git commands run in request context (#17868)zeripath2022-01-196-31/+35
| | | | | | | | | This PR continues the work in #17125 by progressively ensuring that git commands run within the request context. This now means that the if there is a git repo already open in the context it will be used instead of reopening it. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Upgrade bleve from v2.0.6 to v2.3.0 (#18132)Lunny Xiao2022-01-011-1/+1
|
* Prevent hang in git cat-file if repository is not a valid repository and ↵zeripath2021-12-162-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | other fixes (#17991) This PR contains multiple fixes. The most important of which is: * Prevent hang in git cat-file if the repository is not a valid repository Unfortunately it appears that if git cat-file is run in an invalid repository it will hang until stdin is closed. This will result in deadlocked /pulls pages and dangling git cat-file calls if a broken repository is tried to be reviewed or pulls exists for a broken repository. Fix #14734 Fix #9271 Fix #16113 Otherwise there are a few small other fixes included which this PR was initially intending to fix: * Fix panic on partial compares due to missing PullRequestWorkInProgressPrefixes * Fix links on pulls pages due to regression from #17551 - by making most /issues routes match /pulls too - Fix #17983 * Fix links on feeds pages due to another regression from #17551 but also fix issue with syncing tags - Fix #17943 * Add missing locale entries for oauth group claims * Prevent NPEs if ColorFormat is called on nil users, repos or teams.
* Add missing `X-Total-Count` and fix some related bugs (#17968)qwerty2872021-12-153-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add missing `X-Total-Count` and fix some related bugs Adds `X-Total-Count` header to APIs that return a list but doesn't have it yet. Fixed bugs: * not returned after reporting error (https://github.com/qwerty287/gitea/blob/39eb82446c6fe5da3d79124e1f701f3795625b69/routers/api/v1/user/star.go#L70) * crash with index out of bounds, API issue/issueSubscriptions I also found various endpoints that return lists but do not apply/support pagination yet: ``` /repos/{owner}/{repo}/issues/{index}/labels /repos/{owner}/{repo}/issues/comments/{id}/reactions /repos/{owner}/{repo}/branch_protections /repos/{owner}/{repo}/contents /repos/{owner}/{repo}/hooks/git /repos/{owner}/{repo}/issue_templates /repos/{owner}/{repo}/releases/{id}/assets /repos/{owner}/{repo}/reviewers /repos/{owner}/{repo}/teams /user/emails /users/{username}/heatmap ``` If this is not expected, an new issue should be opened. Closes #13043 * fmt * Update routers/api/v1/repo/issue_subscription.go Co-authored-by: KN4CK3R <admin@oldschoolhack.me> * Use FindAndCount Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: 6543 <6543@obermui.de>
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-1012-37/+41
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Make Requests Processes and create process hierarchy. Associate ↵zeripath2021-11-303-3/+10
| | | | | | | | | OpenRepository with context. (#17125) This PR registers requests with the process manager and manages hierarchy within the processes. Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move user related model into models/user (#17781)Lunny Xiao2021-11-241-1/+1
| | | | | | | | | | | | | * Move user related model into models/user * Fix lint for windows * Fix windows lint * Fix windows lint * Move some tests in models * Merge
* Add .gitattribute assisted language detection to blame, diff and render (#17590)zeripath2021-11-171-1/+1
| | | | | | | Use check attribute code to check the assigned language of a file and send that in to chroma as a hint for the language of the file. Signed-off-by: Andrew Thornton <art27@cantab.net>
* A better go code formatter, and now `make fmt` can run in Windows (#17684)wxiaoguang2021-11-173-4/+3
| | | | * go build / format tools * re-format imports
* Fix nil checking on typed interface (#17598)Gusted2021-11-153-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | * Fix nil checking on typed interface - Partially resoles #17596 - Resolves SA4023 errors. - Ensure correctly that typed interface are nil. * Remove unnecessary code `NewBleveIndexer` will never return nil, even on errors. * Patch `NewBleveIndexer` * Fix low-level functions * Remove deadcode * Fix GetSession * Close Elastic search when err isn't nil * Update elastic_search.go Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Decouple unit test code from business code (#17623)wxiaoguang2021-11-125-13/+15
|
* Make Repo Code Indexer an Unique Queue (#17515)zeripath2021-11-021-23/+11
| | | | | | | The functioning of the code indexer queue really only makes sense as an unique queue and doing this allows use to simplify the indexer data to simply delete the data if the repo is no longer in the db. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix some lints (#17337)Lunny Xiao2021-10-171-1/+1
| | | Fix some linting problems.
* Move login related structs and functions to models/login (#17093)Lunny Xiao2021-09-241-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move login related structs and functions to models/login * Fix test * Fix lint * Fix lint * Fix lint of windows * Fix lint * Fix test * Fix test * Only load necessary fixtures when preparing unit tests envs * Fix lint * Fix test * Fix test * Fix error log * Fix error log * Fix error log * remove unnecessary change * fix error log * merge main branch
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-225-10/+8
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Ignore Sync errors on pipes when doing `CheckAttributeReader.CheckPath`, fix ↵wxiaoguang2021-09-201-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | the hang of `git cat-file` (#17096) * Ignore Sync errors on pipes when doing `CheckAttributeReader.CheckPath` * apply env patch * Drop the Sync and fix a number of issues with the Close function Signed-off-by: Andrew Thornton <art27@cantab.net> * add logs for DBIndexer and CheckPath * Fix some more closing bugs Signed-off-by: Andrew Thornton <art27@cantab.net> * Add test case for language_stats Signed-off-by: Andrew Thornton <art27@cantab.net> * Update modules/indexer/stats/db.go Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de>
* Move db related basic functions to models/db (#17075)Lunny Xiao2021-09-197-19/+20
| | | | | | | | | | | | | | | | | | | | | | | | | * Move db related basic functions to models/db * Fix lint * Fix lint * Fix test * Fix lint * Fix lint * revert unnecessary change * Fix test * Fix wrong replace string * Use *Context * Correct committer spelling and fix wrong replaced words Co-authored-by: zeripath <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-28 15:01:29 +0000