aboutsummaryrefslogtreecommitdiffstats
path: root/modules/queue
Commit message (Collapse)AuthorAgeFilesLines
* Prevent race in PersistableChannelUniqueQueue.Has (#14651)zeripath2021-02-131-0/+5
| | | | | | | | | | | There is potentially a race with a slow starting internal queue causing a NPE if Has is checked before the internal queue has been setup. This PR adds a lock on the Has() fn. Fix #14311 Signed-off-by: Andrew Thornton <art27@cantab.net>
* [Vendor] Update go-redis to v8.5.0 (#13749)65432021-02-103-18/+43
| | | | | | | | | | | | | | | | | | | | | * Update go-redis to v8.4.0 * github.com/go-redis/redis/v8 v8.4.0 -> v8.5.0 * Apply suggestions from code review Co-authored-by: zeripath <art27@cantab.net> * TODO * Use the Queue termination channel as the default context for pushes Signed-off-by: Andrew Thornton <art27@cantab.net> * missed one Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Slightly simplify the queue settings code to help reduce the risk of ↵zeripath2020-10-152-25/+39
| | | | | | | problems (#12976) Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Fix the issue reported on #12385 (#12969)zeripath2020-09-281-0/+1
| | | | | Missed setting ConnectionString on queuesettings Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow common redis and leveldb connections (#12385)zeripath2020-09-284-39/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Allow common redis and leveldb connections Prevents multiple reopening of redis and leveldb connections to the same place by sharing connections. Further allows for more configurable redis connection type using the redisURI and a leveldbURI scheme. Signed-off-by: Andrew Thornton <art27@cantab.net> * add unit-test Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @lunny Signed-off-by: Andrew Thornton <art27@cantab.net> * add test Signed-off-by: Andrew Thornton <art27@cantab.net> * Update modules/cache/cache_redis.go * Update modules/queue/queue_disk.go * Update modules/cache/cache_redis.go * Update modules/cache/cache_redis.go * Update modules/queue/unique_queue_disk.go * Update modules/queue/queue_disk.go * Update modules/queue/unique_queue_disk.go * Update modules/session/redis.go Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Add queue for code indexer (#10332)Lunny Xiao2020-09-071-1/+58
| | | | | | | | | | | | | | | | | | | | | * Add queue for code indexer * Fix lint * Fix test * Fix lint * Fix bug * Fix bug * Fix lint * Add noqueue * Fix tests * Rename noqueue to immediate
* use assignment in tests (#12734)zeripath2020-09-062-2/+2
| | | Signed-off-by: Andrew Thornton <art27@cantab.net>
* Re-attempt to delete temporary upload if the file is locked by another ↵zeripath2020-08-112-4/+4
| | | | | | | | | | | process (#12447) Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util. Fix #12339 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: silverwind <me@silverwind.io>
* Multiple small admin dashboard fixes (#12153)zeripath2020-07-051-0/+1
| | | | | | | | | * Remove spurious spacing between Maintenance Operations and its table on dashboard * Prevent (EXTRA string) comments in Task headers * Redirect tasks started from monitor page back to monitor * Fix #12107 - redirects from process cancel should use AppSubUrl * When wrapping queues set the name correctly Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent timer leaks in Workerpool and others (#11333)zeripath2020-05-082-19/+6
| | | | | | | | | | | | | | | | There is a potential memory leak in `Workerpool` due to the intricacies of `time.Timer` stopping. Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a `select` if the result of the `Stop()` is `false`. Unfortunately in `Workerpool` these were checked the wrong way round. However, there were a few other places that were not being checked. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve config logging when WrappedQueue times out (#11174)Lunny Xiao2020-04-221-1/+5
| | | | | | | | | | | | Before ```sh Unable to set the internal queue for -wrapper Error: Timedout creating queue redis with cfg []byte{0x7b, 0x22, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x22, 0x3a, 0x22, 0x31, 0x32, 0x37, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x31, 0x3a, 0x36, 0x33, 0x37, 0x39, 0x22, 0x2c, 0x22, 0x42, 0x61, 0x74, 0x63, 0x68, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x22, 0x3a, 0x32, 0x30, 0x2c, 0x22, 0x42, 0x6c, 0x6f, 0x63, 0x6b, 0x54, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x22, 0x3a, 0x31, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x2c, 0x22, 0x42, 0x6f, 0x6f, 0x73, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x22, 0x3a, 0x33, 0x30, 0x30, 0x30, 0x30, 0x30 ...... ``` After ```sh Unable to set the internal queue for -wrapper Error: Timedout creating queue redis with cfg "{\"Addresses\":\"127.0.0.1:6379\",\"BatchLength\":20,\"BlockTimeout\":1000000000,\"BoostTimeout\":300000000000,\"BoostWorkers\":5,\"DBIndex\":0,\"DataDir\":\".../data/queues/mail\",\"MaxWorkers\":10,\"Name\":\"mail\",\"Network\":\"\",\"Password\":\"\",\"QueueLength\":20,\"QueueName\":\"mail_queue\",\"SetName\":\"\",\"Workers\":1}" in ```
* Avoid logging []byte in queue failures - convert to string first (#10865)zeripath2020-03-291-1/+8
| | | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix queue log param (#10733)Lunny Xiao2020-03-161-3/+3
|
* Fix Workerpool deadlock (#10283)zeripath2020-02-152-3/+6
| | | | | | * Prevent deadlock on boost * Force a boost in testchannelqueue
* Fix queue pop error and stat empty repository error (#10248)Lunny Xiao2020-02-122-2/+4
| | | | | | * Fix queue pop error and stat empty repository error * Fix error
* Add Unique Queue infrastructure and move TestPullRequests to this (#9856)zeripath2020-02-0215-403/+1371
| | | | | | | | | | | | | | | | | | | | | * Upgrade levelqueue to version 0.2.0 This adds functionality for Unique Queues * Add UniqueQueue interface and functions to create them * Add UniqueQueue implementations * Move TestPullRequests over to use UniqueQueue * Reduce code duplication * Add bytefifos * Ensure invalid types are logged * Fix close race in PersistableChannelQueue Shutdown
* Fix broken FlushAll (#10101)zeripath2020-02-011-4/+4
| | | | | | | | | | * go function contexting is not what you expect * Apply suggestions from code review Co-Authored-By: Lauris BH <lauris@nix.lv> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fix leveldb test race (#10054)Lunny Xiao2020-01-301-0/+18
| | | | | Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Queue: Make WorkerPools and Queues flushable (#10001)zeripath2020-01-2812-292/+628
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make WorkerPools and Queues flushable Adds Flush methods to Queues and the WorkerPool Further abstracts the WorkerPool Adds a final step to Flush the queues in the defer from PrintCurrentTest Fixes an issue with Settings inheritance in queues Signed-off-by: Andrew Thornton <art27@cantab.net> * Change to for loop * Add IsEmpty and begin just making the queues composed WorkerPools * subsume workerpool into the queues and create a flushable interface * Add manager command * Move flushall to queue.Manager and add to testlogger * As per @guillep2k * as per @guillep2k * Just make queues all implement flushable and clean up the wrapped queue flushes * cope with no timeout Co-authored-by: Lauris BH <lauris@nix.lv>
* Update queue_redis.go (#9910)zeripath2020-01-211-5/+6
| | | | Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Remove unused lock (#9709)zeripath2020-01-112-1/+3
|
* Missed q.lock.Unlock() will cause panic (#9705)zeripath2020-01-111-1/+0
|
* Graceful Queues: Issue Indexing and Tasks (#9363)zeripath2020-01-0713-0/+2132
* Queue: Add generic graceful queues with settings * Queue & Setting: Add worker pool implementation * Queue: Add worker settings * Queue: Make resizing worker pools * Queue: Add name variable to queues * Queue: Add monitoring * Queue: Improve logging * Issues: Gracefulise the issues indexer Remove the old now unused specific queues * Task: Move to generic queue and gracefulise * Issues: Standardise the issues indexer queue settings * Fix test * Queue: Allow Redis to connect to unix * Prevent deadlock during early shutdown of issue indexer * Add MaxWorker settings to queues * Merge branch 'master' into graceful-queues * Update modules/indexer/issues/indexer.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/indexer/issues/indexer.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/queue/queue_channel.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Update modules/queue/queue_disk.go * Update modules/queue/queue_disk_channel.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * Rename queue.Description to queue.ManagedQueue as per @guillep2k * Cancel pool workers when removed * Remove dependency on queue from setting * Update modules/queue/queue_redis.go Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> * As per @guillep2k add mutex locks on shutdown/terminate * move unlocking out of setInternal * Add warning if number of workers < 0 * Small changes as per @guillep2k * No redis host specified not found * Clean up documentation for queues * Update docs/content/doc/advanced/config-cheat-sheet.en-us.md * Update modules/indexer/issues/indexer_test.go * Ensure that persistable channel queue is added to manager * Rename QUEUE_NAME REDIS_QUEUE_NAME * Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME" This reverts commit 1f83b4fc9b9dabda186257b38c265fe7012f90df. Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <matti@mdranta.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-17 07:59:22 +0000