aboutsummaryrefslogtreecommitdiffstats
path: root/modules/indexer
Commit message (Collapse)AuthorAgeFilesLines
* Fix some lints (#17337)Lunny Xiao2021-10-171-1/+1
| | | Fix some linting problems.
* Move login related structs and functions to models/login (#17093)Lunny Xiao2021-09-241-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move login related structs and functions to models/login * Fix test * Fix lint * Fix lint * Fix lint of windows * Fix lint * Fix test * Fix test * Only load necessary fixtures when preparing unit tests envs * Fix lint * Fix test * Fix test * Fix error log * Fix error log * Fix error log * remove unnecessary change * fix error log * merge main branch
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-225-10/+8
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Ignore Sync errors on pipes when doing `CheckAttributeReader.CheckPath`, fix ↵wxiaoguang2021-09-201-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | the hang of `git cat-file` (#17096) * Ignore Sync errors on pipes when doing `CheckAttributeReader.CheckPath` * apply env patch * Drop the Sync and fix a number of issues with the Close function Signed-off-by: Andrew Thornton <art27@cantab.net> * add logs for DBIndexer and CheckPath * Fix some more closing bugs Signed-off-by: Andrew Thornton <art27@cantab.net> * Add test case for language_stats Signed-off-by: Andrew Thornton <art27@cantab.net> * Update modules/indexer/stats/db.go Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de>
* Move db related basic functions to models/db (#17075)Lunny Xiao2021-09-197-19/+20
| | | | | | | | | | | | | | | | | | | | | | | | | * Move db related basic functions to models/db * Fix lint * Fix lint * Fix test * Fix lint * Fix lint * revert unnecessary change * Fix test * Fix wrong replace string * Use *Context * Correct committer spelling and fix wrong replaced words Co-authored-by: zeripath <art27@cantab.net>
* Add an abstract json layout to make it's easier to change json library (#16528)Lunny Xiao2021-07-241-2/+1
| | | | | | | | | | | * Add an abstract json layout to make it's easier to change json library * Fix import * Fix import sequence * Fix blank lines * Fix blank lines
* Fix data race in bleve indexer (#16474)Lunny Xiao2021-07-183-6/+69
| | | * Fix data race in bleve indexer
* Fix various documentation, user-facing, and source comment typos (#16367)luzpaz2021-07-081-3/+3
| | | | | * Fix various doc, user-facing, and source comment typos Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
* Use git log name-status in get last commit (#16059)zeripath2021-06-212-0/+6
| | | | | | | | | | | | | | | | | * Improve get last commit using git log --name-status git log --name-status -c provides information about the diff between a commit and its parents. Using this and adjusting the algorithm to use the first change to a path allows for a much faster generation of commit info. There is a subtle change in the results generated but this will cause the results to more closely match those from elsewhere. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fixed assert statements. (#16089)KN4CK3R2021-06-071-1/+1
|
* Add Image Diff for SVG files (#14867)KN4CK3R2021-06-052-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added type sniffer. * Switched content detection from base to typesniffer. * Added GuessContentType to Blob. * Moved image info logic to client. Added support for SVG images in diff. * Restore old blocked svg behaviour. * Added missing image formats. * Execute image diff only when container is visible. * add margin to spinner * improve BIN tag on image diffs * Default to render view. * Show image diff on incomplete diff. Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
* Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow ↵zeripath2021-05-152-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | level queue, reduce goroutines etc (#15693) * move shutdownfns, terminatefns and hammerfns out of separate goroutines Coalesce the shutdownfns etc into a list of functions that get run at shutdown rather then have them run at goroutines blocked on selects. This may help reduce the background select/poll load in certain configurations. * The LevelDB queues can actually wait on empty instead of polling Slight refactor to cause leveldb queues to wait on empty instead of polling. * Shutdown the shadow level queue once it is empty * Remove bytefifo additional goroutine for readToChan as it can just be run in run * Remove additional removeWorkers goroutine for workers * Simplify the AtShutdown and AtTerminate functions and add Channel Flusher * Add shutdown flusher to CUQ * move persistable channel shutdown stuff to Shutdown Fn * Ensure that UPCQ has the correct config * handle shutdown during the flushing * reduce risk of race between zeroBoost and addWorkers * prevent double shutdown Signed-off-by: Andrew Thornton <art27@cantab.net>
* On open repository open common cat file batch and batch-check (#15667)zeripath2021-05-102-2/+2
| | | | | | | Use common git cat-file --batch and git cat-file --batch-check to significantly reduce calls to git. Signed-off-by: Andrew Thornton <art27@cantab.net>
* If the default branch is not present do not report error on stats indexing ↵zeripath2021-04-221-1/+1
| | | | | | | | | | | | (follow-up of #15546) (#15583) #15546 doesn't completely fix this problem because the error returned is an ObjectNotExist error not a BranchNotExist error. Add test for ErrObjectNotExist too Fix #15257 Signed-off-by: Andrew Thornton <art27@cantab.net>
* If the default branch is not present do not report error on stats indexing ↵zeripath2021-04-221-1/+5
| | | | | | | | | | | | | | | (#15546) * If the default branch is not present do not report error on stats indexing Fix #15257 Signed-off-by: Andrew Thornton <art27@cantab.net> * as per lunny Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Speed up `enry.IsVendor` (#15213)zeripath2021-04-012-2/+2
| | | | | | | | | `enry.IsVendor` is kinda slow as it simply iterates across all regexps. This PR ajdusts the regexps to combine them to make this process a little quicker. Related #15143 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move Bleve and Elastic code indexers to use a common cat-file --batch (#14781)zeripath2021-03-032-16/+52
| | | | | | | | | | | | | | | | * Extract out the common cat-file batch calls Signed-off-by: Andrew Thornton <art27@cantab.net> * Move bleve and elastic indexers to use a common cat-file --batch when indexing Signed-off-by: Andrew Thornton <art27@cantab.net> * move catfilebatch to batch_reader and rename to batch_reader.go Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv>
* Migrate to use jsoniter instead of encoding/json (#14841)zeripath2021-03-011-1/+2
| | | | | | | | | | * Migrate to use jsoniter * fix tests * update gitea.com/go-chi/binding Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Reduce calls to git cat-file -s (#14682)zeripath2021-02-173-16/+32
| | | | | | | | | | | | | * Reduce calls to git cat-file -s There are multiple places where there are repeated calls to git cat-file -s due to the blobs not being created with their size. Through judicious use of git ls-tree -l and slight adjustments to the indexer code we can avoid a lot of these calls. * simplify by always expecting the long format * Also always set the sized field and tell the indexer the update is sized
* [Feature] add precise search type for Elastic Search (#12869)Jui-Nan Lin2021-01-276-15/+35
| | | | | | | | * feat: add type query parameters for specifying precise search * feat: add select dropdown in search box Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Upgrade blevesearch dependency to v2.0.1 (#14346)Lauris BH2021-01-173-18/+18
| | | | | | | * Upgrade blevesearch dependency to v2.0.1 * Update rupture to v1.0.0 * Fix test
* Sort issue search results by revelance (#14353)Lauris BH2021-01-163-3/+4
|
* Use native git variants by default with go-git variants as build tag (#13673)zeripath2020-12-171-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move last commit cache back into modules/git Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove go-git from the interface for last commit cache Signed-off-by: Andrew Thornton <art27@cantab.net> * move cacheref to last_commit_cache Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove go-git from routers/private/hook Signed-off-by: Andrew Thornton <art27@cantab.net> * Move FindLFSFiles to pipeline Signed-off-by: Andrew Thornton <art27@cantab.net> * Make no-go-git variants Signed-off-by: Andrew Thornton <art27@cantab.net> * Submodule RefID Signed-off-by: Andrew Thornton <art27@cantab.net> * fix issue with GetCommitsInfo Signed-off-by: Andrew Thornton <art27@cantab.net> * fix GetLastCommitForPaths Signed-off-by: Andrew Thornton <art27@cantab.net> * Improve efficiency Signed-off-by: Andrew Thornton <art27@cantab.net> * More efficiency Signed-off-by: Andrew Thornton <art27@cantab.net> * even faster Signed-off-by: Andrew Thornton <art27@cantab.net> * Reduce duplication * As per @lunny Signed-off-by: Andrew Thornton <art27@cantab.net> * attempt to fix drone Signed-off-by: Andrew Thornton <art27@cantab.net> * fix test-tags Signed-off-by: Andrew Thornton <art27@cantab.net> * default to use no-go-git variants and add gogit build tag Signed-off-by: Andrew Thornton <art27@cantab.net> * placate lint Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @6543 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactor Logger (#13294)zeripath2020-10-312-3/+3
| | | | | | Refactor Logger to make a logger interface and make it possible to wrap loggers for specific purposes. Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* fix: use Base36 for all code indexers (#12830)Jui-Nan Lin2020-09-142-6/+5
|
* Fix bug on elastic search (#12811)Lunny Xiao2020-09-122-15/+42
| | | | | | | | | | | * Fix bug on elastic search * Add more comments for elastic search result startIndex and endIndex * refactor indexPos * refactor indexPos * Fix bug
* Add queue for code indexer (#10332)Lunny Xiao2020-09-073-159/+158
| | | | | | | | | | | | | | | | | | | | | * Add queue for code indexer * Fix lint * Fix test * Fix lint * Fix bug * Fix bug * Fix lint * Add noqueue * Fix tests * Rename noqueue to immediate
* Support elastic search for code search (#10273)Lunny Xiao2020-08-308-157/+658
| | | | | | | | | | | | | | | | | | | | | | | * Support elastic search for code search * Finished elastic search implementation and add some tests * Enable test on drone and added docs * Add new fields to elastic search * Fix bug * remove unused changes * Use indexer alias to keep the gitea indexer version * Improve codes * Some code improvements * The real indexer name changed to xxx.v1 Co-authored-by: zeripath <art27@cantab.net>
* Remove hardcoded ES indexername (#12521)Wim2020-08-181-1/+1
|
* Re-attempt to delete temporary upload if the file is locked by another ↵zeripath2020-08-115-10/+12
| | | | | | | | | | | process (#12447) Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util. Fix #12339 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: silverwind <me@silverwind.io>
* Fix incorrect error logging in Stats indexer and OAuth2 (#12387)zeripath2020-08-011-2/+8
| | | | | | | | | | | | | | | | | | | | * Fix incorrect logging in oauth2.go Fix #11945 Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle ErrAlreadyInQueue in stats indexer Fix #12380 Signed-off-by: Andrew Thornton <art27@cantab.net> * Fixes type in error message of indexer Add the missing character in the error message. Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lieven Hollevoet <hollie@lika.be>
* Server-side syntax highlighting for all code (#12047)mrsdizzie2020-07-011-16/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Server-side syntax hilighting for all code This PR does a few things: * Remove all traces of highlight.js * Use chroma library to provide fast syntax hilighting directly on the server * Provide syntax hilighting for diffs * Re-style both unified and split diffs views * Add custom syntax hilighting styling for both regular and arc-green Fixes #7729 Fixes #10157 Fixes #11825 Fixes #7728 Fixes #3872 Fixes #3682 And perhaps gets closer to #9553 * fix line marker * fix repo search * Fix single line select * properly load settings * npm uninstall highlight.js * review suggestion * code review * forgot to call function * fix test * Apply suggestions from code review suggestions from @silverwind thanks Co-authored-by: silverwind <me@silverwind.io> * code review * copy/paste error * Use const for highlight size limit * Update web_src/less/_repository.less Co-authored-by: Lauris BH <lauris@nix.lv> * update size limit to 1MB and other styling tweaks * fix highlighting for certain diff sections * fix test * add worker back as suggested Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fix language stat calculation (#11692)Cirno the Strongest2020-05-311-3/+1
| | | | | | | * Fix language stat calculation * Group languages and ignore 0 size files * remove unneeded code
* Change language statistics to save size instead of percentage (#11681)Lauris BH2020-05-301-0/+3
| | | | | | | | | | | * Change language statistics to save size instead of percentage in database Co-Authored-By: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com> * Do not exclude if only language * Fix edge cases with special langauges Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Log the indexer path on failure (#11172)zeripath2020-04-222-4/+5
| | | | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
* Migrate to go-enry new version (#10906)Lauris BH2020-04-151-1/+1
|
* Handle panic in indexer initialisation better (#10534)zeripath2020-02-282-0/+19
| | | | | * Handle panic in indexer initialisation better * as per @guillep2k
* Add detected file language to code search (#10256)Lauris BH2020-02-205-34/+140
| | | | | | | Move langauge detection to separate module to be more reusable Add option to disable vendored file exclusion from file search Allways show all language stats for search
* Index code and stats only for non-empty repositories (#10251)Lauris BH2020-02-141-2/+2
| | | | | Fix test and switch to unique queue Fix MySQL support when deleting old statistics
* Issue search support elasticsearch (#9428)Lunny Xiao2020-02-133-10/+254
| | | | | | | | | | | | * Issue search support elasticsearch * Fix lint * Add indexer name on app.ini * add a warnning on SearchIssuesByKeyword * improve code
* Fix queue pop error and stat empty repository error (#10248)Lunny Xiao2020-02-121-0/+4
| | | | | | * Fix queue pop error and stat empty repository error * Fix error
* Stop hanging issue indexer initialisation from preventing shutdown (#10243)zeripath2020-02-121-17/+15
| | | | | Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
* Language statistics bar for repositories (#8037)Lauris BH2020-02-117-5/+230
| | | | | | | | | | | | | | | | | | * Implementation for calculating language statistics Impement saving code language statistics to database Implement rendering langauge stats Add primary laguage to show in repository list Implement repository stats indexer queue Add indexer test Refactor to use queue module * Do not timeout for queues
* Fix file rename/copy not supported by indexer (#9965)guillep2k2020-01-241-2/+27
| | | | Co-authored-by: zeripath <art27@cantab.net>
* API add/generalize pagination (#9452)SpaWn2KiLl2020-01-241-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * paginate results * fixed deadlock * prevented breaking change * updated swagger * go fmt * fixed find topic * go mod tidy * go mod vendor with go1.13.5 * fixed repo find topics * fixed unit test * added Limit method to Engine struct; use engine variable when provided; fixed gitignore * use ItemsPerPage for default pagesize; fix GetWatchers, getOrgUsersByOrgID and GetStargazers; fix GetAllCommits headers; reverted some changed behaviors * set Page value on Home route * improved memory allocations * fixed response headers * removed logfiles * fixed import order * import order * improved swagger * added function to get models.ListOptions from context * removed pagesize diff on unit test * fixed imports * removed unnecessary struct field * fixed go fmt * scoped PR * code improvements * code improvements * go mod tidy * fixed import order * fixed commit statuses session * fixed files headers * fixed headers; added pagination for notifications * go mod tidy * go fmt * removed Private from user search options; added setting.UI.IssuePagingNum as default valeu on repo's issues list * Apply suggestions from code review Co-Authored-By: 6543 <6543@obermui.de> Co-Authored-By: zeripath <art27@cantab.net> * fixed build error * CI.restart() * fixed merge conflicts resolve * fixed conflicts resolve * improved FindTrackedTimesOptions.ToOptions() method * added backwards compatibility on ListReleases request; fixed issue tracked time ToSession * fixed build error; fixed swagger template * fixed swagger template * fixed ListReleases backwards compatibility * added page to user search route Co-authored-by: techknowlogick <matti@mdranta.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: zeripath <art27@cantab.net>
* Add "Update Branch" button to Pull Requests (#9784)65432020-01-171-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add Divergence * add Update Button * first working version * re-use code * split raw merge commands and db-change functions (notify, cache, ...) * use rawMerge (remove redundant code) * own function to get Diverging of PRs * use FlashError * correct Error Msg * hook is triggerd ... so remove comment * add "branch2" to "user2/repo1" because it unit-test "TestPullView_ReviewerMissed" use it but dont exist jet :/ * move GetPerm to IsUserAllowedToUpdate * add Flash Success MSG * imprufe code - remove useless js chage * fix-lint * TEST: add PullRequest ID:5 Repo: user2/repo1 Base: branch1 Head: pr-to-update * correct comments * make PR5 outdated * fix Tests * WIP: add pull update test * update revs * update locales * working TEST * update UI * misspell * change style * add 1s delay so rev exist * move row up (before merge row) * fix lint nit * UI remove divider * Update style * nits * do it right * introduce IsSameRepo * remove useless check Co-authored-by: Lauris BH <lauris@nix.lv>
* Graceful Queues: Issue Indexing and Tasks (#9363)zeripath2020-01-077-411/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Queue: Add generic graceful queues with settings * Queue & Setting: Add worker pool implementation * Queue: Add worker settings * Queue: Make resizing worker pools * Queue: Add name variable to queues * Queue: Add monitoring * Queue: Improve logging * Issues: Gracefulise the issues indexer Remove the old now unused specific queues * Task: Move to generic queue and gracefulise * Issues: Standardise the issues indexer queue settings * Fix test * Queue: Allow Redis to connect to unix * Prevent deadlock during early shutdown of issue indexer * Add MaxWorker settings to queues * Merge branch 'master' into graceful-queues * Update modules/indexer/issues/indexer.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/indexer/issues/indexer.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/queue/queue_channel.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/queue/queue_disk.go * Update modules/queue/queue_disk_channel.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Rename queue.Description to queue.ManagedQueue as per @guillep2k * Cancel pool workers when removed * Remove dependency on queue from setting * Update modules/queue/queue_redis.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * As per @guillep2k add mutex locks on shutdown/terminate * move unlocking out of setInternal * Add warning if number of workers < 0 * Small changes as per @guillep2k * No redis host specified not found * Clean up documentation for queues * Update docs/content/doc/advanced/config-cheat-sheet.en-us.md * Update modules/indexer/issues/indexer_test.go * Ensure that persistable channel queue is added to manager * Rename QUEUE_NAME REDIS_QUEUE_NAME * Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME" This reverts commit 1f83b4fc9b9dabda186257b38c265fe7012f90df. Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <matti@mdranta.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Stop various tests from adding to the source tree (#9515)zeripath2019-12-284-11/+61
| | | | | | | Instead of just adding test generated files to .gitignore prevent them from being produced in the first place. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* fix datarace on issue indexer queue (#9490)Lunny Xiao2019-12-252-1/+6
|
* Wrap the code indexer (#9476)zeripath2019-12-243-13/+132
| | | | | | | | | | | | | | | | | | | | * Wrap the code indexer In order to prevent a data race in the code indexer it must be wrapped with a holder otherwise it is possible to Search/Index on an incompletely initialised indexer, and search will fail with a nil pointer until the repository indexer is initialised. Further a completely initialised repository indexer should not be closed until Termination otherwise actions in Hammer/Shutdown phases could block or be lost. Finally, there is a complex dance of shutdown etiquette should the index initialisation fail. This PR restores that. * Always return err if closed whilst waiting Co-authored-by: techknowlogick <matti@mdranta.net>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-29 06:36:39 +0000