summaryrefslogtreecommitdiffstats
path: root/modules/indexer
Commit message (Collapse)AuthorAgeFilesLines
* Use git log name-status in get last commit (#16059)zeripath2021-06-212-0/+6
| | | | | | | | | | | | | | | | | * Improve get last commit using git log --name-status git log --name-status -c provides information about the diff between a commit and its parents. Using this and adjusting the algorithm to use the first change to a path allows for a much faster generation of commit info. There is a subtle change in the results generated but this will cause the results to more closely match those from elsewhere. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fixed assert statements. (#16089)KN4CK3R2021-06-071-1/+1
|
* Add Image Diff for SVG files (#14867)KN4CK3R2021-06-052-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added type sniffer. * Switched content detection from base to typesniffer. * Added GuessContentType to Blob. * Moved image info logic to client. Added support for SVG images in diff. * Restore old blocked svg behaviour. * Added missing image formats. * Execute image diff only when container is visible. * add margin to spinner * improve BIN tag on image diffs * Default to render view. * Show image diff on incomplete diff. Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
* Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow ↵zeripath2021-05-152-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | level queue, reduce goroutines etc (#15693) * move shutdownfns, terminatefns and hammerfns out of separate goroutines Coalesce the shutdownfns etc into a list of functions that get run at shutdown rather then have them run at goroutines blocked on selects. This may help reduce the background select/poll load in certain configurations. * The LevelDB queues can actually wait on empty instead of polling Slight refactor to cause leveldb queues to wait on empty instead of polling. * Shutdown the shadow level queue once it is empty * Remove bytefifo additional goroutine for readToChan as it can just be run in run * Remove additional removeWorkers goroutine for workers * Simplify the AtShutdown and AtTerminate functions and add Channel Flusher * Add shutdown flusher to CUQ * move persistable channel shutdown stuff to Shutdown Fn * Ensure that UPCQ has the correct config * handle shutdown during the flushing * reduce risk of race between zeroBoost and addWorkers * prevent double shutdown Signed-off-by: Andrew Thornton <art27@cantab.net>
* On open repository open common cat file batch and batch-check (#15667)zeripath2021-05-102-2/+2
| | | | | | | Use common git cat-file --batch and git cat-file --batch-check to significantly reduce calls to git. Signed-off-by: Andrew Thornton <art27@cantab.net>
* If the default branch is not present do not report error on stats indexing ↵zeripath2021-04-221-1/+1
| | | | | | | | | | | | (follow-up of #15546) (#15583) #15546 doesn't completely fix this problem because the error returned is an ObjectNotExist error not a BranchNotExist error. Add test for ErrObjectNotExist too Fix #15257 Signed-off-by: Andrew Thornton <art27@cantab.net>
* If the default branch is not present do not report error on stats indexing ↵zeripath2021-04-221-1/+5
| | | | | | | | | | | | | | | (#15546) * If the default branch is not present do not report error on stats indexing Fix #15257 Signed-off-by: Andrew Thornton <art27@cantab.net> * as per lunny Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Speed up `enry.IsVendor` (#15213)zeripath2021-04-012-2/+2
| | | | | | | | | `enry.IsVendor` is kinda slow as it simply iterates across all regexps. This PR ajdusts the regexps to combine them to make this process a little quicker. Related #15143 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move Bleve and Elastic code indexers to use a common cat-file --batch (#14781)zeripath2021-03-032-16/+52
| | | | | | | | | | | | | | | | * Extract out the common cat-file batch calls Signed-off-by: Andrew Thornton <art27@cantab.net> * Move bleve and elastic indexers to use a common cat-file --batch when indexing Signed-off-by: Andrew Thornton <art27@cantab.net> * move catfilebatch to batch_reader and rename to batch_reader.go Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv>
* Migrate to use jsoniter instead of encoding/json (#14841)zeripath2021-03-011-1/+2
| | | | | | | | | | * Migrate to use jsoniter * fix tests * update gitea.com/go-chi/binding Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Reduce calls to git cat-file -s (#14682)zeripath2021-02-173-16/+32
| | | | | | | | | | | | | * Reduce calls to git cat-file -s There are multiple places where there are repeated calls to git cat-file -s due to the blobs not being created with their size. Through judicious use of git ls-tree -l and slight adjustments to the indexer code we can avoid a lot of these calls. * simplify by always expecting the long format * Also always set the sized field and tell the indexer the update is sized
* [Feature] add precise search type for Elastic Search (#12869)Jui-Nan Lin2021-01-276-15/+35
| | | | | | | | * feat: add type query parameters for specifying precise search * feat: add select dropdown in search box Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Upgrade blevesearch dependency to v2.0.1 (#14346)Lauris BH2021-01-173-18/+18
| | | | | | | * Upgrade blevesearch dependency to v2.0.1 * Update rupture to v1.0.0 * Fix test
* Sort issue search results by revelance (#14353)Lauris BH2021-01-163-3/+4
|
* Use native git variants by default with go-git variants as build tag (#13673)zeripath2020-12-171-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move last commit cache back into modules/git Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove go-git from the interface for last commit cache Signed-off-by: Andrew Thornton <art27@cantab.net> * move cacheref to last_commit_cache Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove go-git from routers/private/hook Signed-off-by: Andrew Thornton <art27@cantab.net> * Move FindLFSFiles to pipeline Signed-off-by: Andrew Thornton <art27@cantab.net> * Make no-go-git variants Signed-off-by: Andrew Thornton <art27@cantab.net> * Submodule RefID Signed-off-by: Andrew Thornton <art27@cantab.net> * fix issue with GetCommitsInfo Signed-off-by: Andrew Thornton <art27@cantab.net> * fix GetLastCommitForPaths Signed-off-by: Andrew Thornton <art27@cantab.net> * Improve efficiency Signed-off-by: Andrew Thornton <art27@cantab.net> * More efficiency Signed-off-by: Andrew Thornton <art27@cantab.net> * even faster Signed-off-by: Andrew Thornton <art27@cantab.net> * Reduce duplication * As per @lunny Signed-off-by: Andrew Thornton <art27@cantab.net> * attempt to fix drone Signed-off-by: Andrew Thornton <art27@cantab.net> * fix test-tags Signed-off-by: Andrew Thornton <art27@cantab.net> * default to use no-go-git variants and add gogit build tag Signed-off-by: Andrew Thornton <art27@cantab.net> * placate lint Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @6543 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactor Logger (#13294)zeripath2020-10-312-3/+3
| | | | | | Refactor Logger to make a logger interface and make it possible to wrap loggers for specific purposes. Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* fix: use Base36 for all code indexers (#12830)Jui-Nan Lin2020-09-142-6/+5
|
* Fix bug on elastic search (#12811)Lunny Xiao2020-09-122-15/+42
| | | | | | | | | | | * Fix bug on elastic search * Add more comments for elastic search result startIndex and endIndex * refactor indexPos * refactor indexPos * Fix bug
* Add queue for code indexer (#10332)Lunny Xiao2020-09-073-159/+158
| | | | | | | | | | | | | | | | | | | | | * Add queue for code indexer * Fix lint * Fix test * Fix lint * Fix bug * Fix bug * Fix lint * Add noqueue * Fix tests * Rename noqueue to immediate
* Support elastic search for code search (#10273)Lunny Xiao2020-08-308-157/+658
| | | | | | | | | | | | | | | | | | | | | | | * Support elastic search for code search * Finished elastic search implementation and add some tests * Enable test on drone and added docs * Add new fields to elastic search * Fix bug * remove unused changes * Use indexer alias to keep the gitea indexer version * Improve codes * Some code improvements * The real indexer name changed to xxx.v1 Co-authored-by: zeripath <art27@cantab.net>
* Remove hardcoded ES indexername (#12521)Wim2020-08-181-1/+1
|
* Re-attempt to delete temporary upload if the file is locked by another ↵zeripath2020-08-115-10/+12
| | | | | | | | | | | process (#12447) Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util. Fix #12339 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: silverwind <me@silverwind.io>
* Fix incorrect error logging in Stats indexer and OAuth2 (#12387)zeripath2020-08-011-2/+8
| | | | | | | | | | | | | | | | | | | | * Fix incorrect logging in oauth2.go Fix #11945 Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle ErrAlreadyInQueue in stats indexer Fix #12380 Signed-off-by: Andrew Thornton <art27@cantab.net> * Fixes type in error message of indexer Add the missing character in the error message. Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lieven Hollevoet <hollie@lika.be>
* Server-side syntax highlighting for all code (#12047)mrsdizzie2020-07-011-16/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Server-side syntax hilighting for all code This PR does a few things: * Remove all traces of highlight.js * Use chroma library to provide fast syntax hilighting directly on the server * Provide syntax hilighting for diffs * Re-style both unified and split diffs views * Add custom syntax hilighting styling for both regular and arc-green Fixes #7729 Fixes #10157 Fixes #11825 Fixes #7728 Fixes #3872 Fixes #3682 And perhaps gets closer to #9553 * fix line marker * fix repo search * Fix single line select * properly load settings * npm uninstall highlight.js * review suggestion * code review * forgot to call function * fix test * Apply suggestions from code review suggestions from @silverwind thanks Co-authored-by: silverwind <me@silverwind.io> * code review * copy/paste error * Use const for highlight size limit * Update web_src/less/_repository.less Co-authored-by: Lauris BH <lauris@nix.lv> * update size limit to 1MB and other styling tweaks * fix highlighting for certain diff sections * fix test * add worker back as suggested Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fix language stat calculation (#11692)Cirno the Strongest2020-05-311-3/+1
| | | | | | | * Fix language stat calculation * Group languages and ignore 0 size files * remove unneeded code
* Change language statistics to save size instead of percentage (#11681)Lauris BH2020-05-301-0/+3
| | | | | | | | | | | * Change language statistics to save size instead of percentage in database Co-Authored-By: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com> * Do not exclude if only language * Fix edge cases with special langauges Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Log the indexer path on failure (#11172)zeripath2020-04-222-4/+5
| | | | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: Lauris BH <lauris@nix.lv>
* Migrate to go-enry new version (#10906)Lauris BH2020-04-151-1/+1
|
* Handle panic in indexer initialisation better (#10534)zeripath2020-02-282-0/+19
| | | | | * Handle panic in indexer initialisation better * as per @guillep2k
* Add detected file language to code search (#10256)Lauris BH2020-02-205-34/+140
| | | | | | | Move langauge detection to separate module to be more reusable Add option to disable vendored file exclusion from file search Allways show all language stats for search
* Index code and stats only for non-empty repositories (#10251)Lauris BH2020-02-141-2/+2
| | | | | Fix test and switch to unique queue Fix MySQL support when deleting old statistics
* Issue search support elasticsearch (#9428)Lunny Xiao2020-02-133-10/+254
| | | | | | | | | | | | * Issue search support elasticsearch * Fix lint * Add indexer name on app.ini * add a warnning on SearchIssuesByKeyword * improve code
* Fix queue pop error and stat empty repository error (#10248)Lunny Xiao2020-02-121-0/+4
| | | | | | * Fix queue pop error and stat empty repository error * Fix error
* Stop hanging issue indexer initialisation from preventing shutdown (#10243)zeripath2020-02-121-17/+15
| | | | | Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
* Language statistics bar for repositories (#8037)Lauris BH2020-02-117-5/+230
| | | | | | | | | | | | | | | | | | * Implementation for calculating language statistics Impement saving code language statistics to database Implement rendering langauge stats Add primary laguage to show in repository list Implement repository stats indexer queue Add indexer test Refactor to use queue module * Do not timeout for queues
* Fix file rename/copy not supported by indexer (#9965)guillep2k2020-01-241-2/+27
| | | | Co-authored-by: zeripath <art27@cantab.net>
* API add/generalize pagination (#9452)SpaWn2KiLl2020-01-241-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * paginate results * fixed deadlock * prevented breaking change * updated swagger * go fmt * fixed find topic * go mod tidy * go mod vendor with go1.13.5 * fixed repo find topics * fixed unit test * added Limit method to Engine struct; use engine variable when provided; fixed gitignore * use ItemsPerPage for default pagesize; fix GetWatchers, getOrgUsersByOrgID and GetStargazers; fix GetAllCommits headers; reverted some changed behaviors * set Page value on Home route * improved memory allocations * fixed response headers * removed logfiles * fixed import order * import order * improved swagger * added function to get models.ListOptions from context * removed pagesize diff on unit test * fixed imports * removed unnecessary struct field * fixed go fmt * scoped PR * code improvements * code improvements * go mod tidy * fixed import order * fixed commit statuses session * fixed files headers * fixed headers; added pagination for notifications * go mod tidy * go fmt * removed Private from user search options; added setting.UI.IssuePagingNum as default valeu on repo's issues list * Apply suggestions from code review Co-Authored-By: 6543 <6543@obermui.de> Co-Authored-By: zeripath <art27@cantab.net> * fixed build error * CI.restart() * fixed merge conflicts resolve * fixed conflicts resolve * improved FindTrackedTimesOptions.ToOptions() method * added backwards compatibility on ListReleases request; fixed issue tracked time ToSession * fixed build error; fixed swagger template * fixed swagger template * fixed ListReleases backwards compatibility * added page to user search route Co-authored-by: techknowlogick <matti@mdranta.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: zeripath <art27@cantab.net>
* Add "Update Branch" button to Pull Requests (#9784)65432020-01-171-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add Divergence * add Update Button * first working version * re-use code * split raw merge commands and db-change functions (notify, cache, ...) * use rawMerge (remove redundant code) * own function to get Diverging of PRs * use FlashError * correct Error Msg * hook is triggerd ... so remove comment * add "branch2" to "user2/repo1" because it unit-test "TestPullView_ReviewerMissed" use it but dont exist jet :/ * move GetPerm to IsUserAllowedToUpdate * add Flash Success MSG * imprufe code - remove useless js chage * fix-lint * TEST: add PullRequest ID:5 Repo: user2/repo1 Base: branch1 Head: pr-to-update * correct comments * make PR5 outdated * fix Tests * WIP: add pull update test * update revs * update locales * working TEST * update UI * misspell * change style * add 1s delay so rev exist * move row up (before merge row) * fix lint nit * UI remove divider * Update style * nits * do it right * introduce IsSameRepo * remove useless check Co-authored-by: Lauris BH <lauris@nix.lv>
* Graceful Queues: Issue Indexing and Tasks (#9363)zeripath2020-01-077-411/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Queue: Add generic graceful queues with settings * Queue & Setting: Add worker pool implementation * Queue: Add worker settings * Queue: Make resizing worker pools * Queue: Add name variable to queues * Queue: Add monitoring * Queue: Improve logging * Issues: Gracefulise the issues indexer Remove the old now unused specific queues * Task: Move to generic queue and gracefulise * Issues: Standardise the issues indexer queue settings * Fix test * Queue: Allow Redis to connect to unix * Prevent deadlock during early shutdown of issue indexer * Add MaxWorker settings to queues * Merge branch 'master' into graceful-queues * Update modules/indexer/issues/indexer.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/indexer/issues/indexer.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/queue/queue_channel.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/queue/queue_disk.go * Update modules/queue/queue_disk_channel.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Rename queue.Description to queue.ManagedQueue as per @guillep2k * Cancel pool workers when removed * Remove dependency on queue from setting * Update modules/queue/queue_redis.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * As per @guillep2k add mutex locks on shutdown/terminate * move unlocking out of setInternal * Add warning if number of workers < 0 * Small changes as per @guillep2k * No redis host specified not found * Clean up documentation for queues * Update docs/content/doc/advanced/config-cheat-sheet.en-us.md * Update modules/indexer/issues/indexer_test.go * Ensure that persistable channel queue is added to manager * Rename QUEUE_NAME REDIS_QUEUE_NAME * Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME" This reverts commit 1f83b4fc9b9dabda186257b38c265fe7012f90df. Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <matti@mdranta.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Stop various tests from adding to the source tree (#9515)zeripath2019-12-284-11/+61
| | | | | | | Instead of just adding test generated files to .gitignore prevent them from being produced in the first place. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* fix datarace on issue indexer queue (#9490)Lunny Xiao2019-12-252-1/+6
|
* Wrap the code indexer (#9476)zeripath2019-12-243-13/+132
| | | | | | | | | | | | | | | | | | | | * Wrap the code indexer In order to prevent a data race in the code indexer it must be wrapped with a holder otherwise it is possible to Search/Index on an incompletely initialised indexer, and search will fail with a nil pointer until the repository indexer is initialised. Further a completely initialised repository indexer should not be closed until Termination otherwise actions in Hammer/Shutdown phases could block or be lost. Finally, there is a complex dance of shutdown etiquette should the index initialisation fail. This PR restores that. * Always return err if closed whilst waiting Co-authored-by: techknowlogick <matti@mdranta.net>
* Refactor code indexer (#9313)Lunny Xiao2019-12-239-627/+767
| | | | | | | | | | | | | | | | | | | | * Refactor code indexer * fix test * fix test * refactor code indexer * fix import * improve code * fix typo * fix test and make code clean * fix lint
* Graceful: Xorm, RepoIndexer, Cron and Others (#9282)zeripath2019-12-153-35/+84
| | | | | | | | | | | | * Change graceful to use a singleton obtained through GetManager instead of a global. * Graceful: Make TestPullRequests shutdownable * Graceful: Make the cron tasks graceful * Graceful: AddTestPullRequest run in graceful ctx * Graceful: SyncMirrors shutdown * Graceful: SetDefaultContext for Xorm to be HammerContext * Avoid starting graceful for migrate commands and checkout * Graceful: DeliverHooks now can be shutdown * Fix multiple syncing errors in modules/sync/UniqueQueue & Make UniqueQueue closable * Begin the process of making the repo indexer shutdown gracefully
* DefaultBranch needs to be prefixed by BranchPrefix (#9356)zeripath2019-12-141-1/+1
|
* fix issue indexer not triggered when migrating a repository (#9332)Lunny Xiao2019-12-121-16/+21
|
* Move modules/indexer to modules/indexer/code (#9301)Lunny Xiao2019-12-103-18/+17
|
* Move code indexer related code to a new package (#9191)Lunny Xiao2019-12-082-0/+381
| | | | | | | | | | | | * move code indexer related code to a new package * fix lint * fix tests * fix fmt * GetMaxID support interface parameter
* upgrade levelqueue to 0.1.0 (#9192)Lunny Xiao2019-11-281-1/+1
|
* Add Graceful shutdown for Windows and hooks for shutdown of goroutines (#8964)zeripath2019-11-211-1/+1
| | | | | | | | | | | | | | | | | | * Graceful Shutdown for windows and others Restructures modules/graceful, adding shutdown for windows, removing and replacing the old minwinsvc code. Creates a new waitGroup - terminate which allows for goroutines to finish up after the shutdown of the servers. Shutdown and terminate hooks are added for goroutines. * Remove unused functions - these can be added in a different PR * Add startup timeout functionality * Document STARTUP_TIMEOUT
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-17 01:36:00 +0000