aboutsummaryrefslogtreecommitdiffstats
path: root/modules/graceful
Commit message (Collapse)AuthorAgeFilesLines
* Improve queue and logger context (#24924)wxiaoguang2023-05-264-96/+11
| | | | | | | | | | | | | | | | | | | | | | | | Before there was a "graceful function": RunWithShutdownFns, it's mainly for some modules which doesn't support context. The old queue system doesn't work well with context, so the old queues need it. After the queue refactoring, the new queue works with context well, so, use Golang context as much as possible, the `RunWithShutdownFns` could be removed (replaced by RunWithCancel for context cancel mechanism), the related code could be simplified. This PR also fixes some legacy queue-init problems, eg: * typo : archiver: "unable to create codes indexer queue" => "unable to create repo-archive queue" * no nil check for failed queues, which causes unfriendly panic After this PR, many goroutines could have better display name: ![image](https://github.com/go-gitea/gitea/assets/2114189/701b2a9b-8065-4137-aeaa-0bda2b34604a) ![image](https://github.com/go-gitea/gitea/assets/2114189/f1d5f50f-0534-40f0-b0be-f2c9daa5fe92)
* Fix regression: access log template, gitea manager cli command (#24838)wxiaoguang2023-05-222-1/+7
| | | | | | | Close #24836 ![image](https://github.com/go-gitea/gitea/assets/2114189/95b025d2-f25f-4246-a08a-fe44ecb787a9) ![image](https://github.com/go-gitea/gitea/assets/2114189/c3afe1fa-2a23-420d-a016-3b67dcd04cd5)
* Rewrite logger system (#24726)wxiaoguang2023-05-214-2/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>
* Implement systemd-notify protocol (#21151)zeripath2023-05-153-12/+159
| | | | | | | | | | | | | | | This PR adds support for the systemd notify protocol. Several status messagess are provided. We should likely add a common notify/status message for graceful. Replaces #21140 Signed-off-by: Andrew Thornton <art27@cantab.net> --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: ltdk <usr@ltdk.xyz> Co-authored-by: Giteabot <teabot@gitea.io>
* Remove most path-based golangci exclusions (#24214)silverwind2023-04-191-1/+1
| | | | | They are non-obvious and do not survive refactor. Will replace with `//nolint` comments after CI results are in.
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-2710-20/+14
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-241-5/+5
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Support Proxy protocol (#12527)zeripath2022-08-212-11/+45
| | | | | | | | | This PR adds functionality to allow Gitea to sit behind an HAProxy and HAProxy protocolled connections directly. Fix #7508 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix graceful doc (#20883)wxiaoguang2022-08-211-4/+6
|
* call builtinUnused() if internal SSH is disabled (#20877)JonRB2022-08-201-2/+2
| | | | | | | | | | | | | The graceful manager waits for 4 listeners to be created or to be told that they are not needed. If it is not told about them it will indefinitely and timeout. This leads to SVC hosts not being told of being in the readyState but on Unix would lead to the termination of the process. There was an unfortunate regression in #20299 which missed this subtly and in the case whereby SSH is disabled the `builtinUnused()` is not called. This PR adds a call to `builtinUnused()` when not using the builtin ssh to allow `createServerWaitGroup.Done()` to be called. In addition it was noted that the if/else clauses for timeout informing of the SVC host were in the wrong order. These have been swapped. Fix #20609
* Preserve unix socket file (#20499)Frank2022-08-132-3/+50
| | | | | | | | | | By default Gitea will always unlink any sockets that are provided using the `LISTEN_FDS` environment variable. This is because it uses this variable to handle passing when it is doing a graceful restart. However, this same mechanism is used by systemd - which explicitly expects that passed in sockets should not be unlinked by the receiving process. This PR adjusts Gitea's graceful restart mechanism to use an additional environment variable which tracks if a listening socket was opened by Gitea - and therefore should be unlinked on shutdown by Gitea. Fix #20490 Co-authored-by: zeripath <art27@cantab.net>
* Add more linters to improve code readability (#19989)Wim2022-06-201-1/+1
| | | | | | | | | | Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability - nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length. - unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions - wastedassign - https://github.com/sanposhiho/wastedassign - wastedassign finds wasted assignment statements. - notlintlint - Reports ill-formed or insufficient nolint directives - stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
* Fix signal loop in graceful manager (#19943)wxiaoguang2022-06-121-0/+1
|
* Remove legacy `+build:` constraint (#19582)wxiaoguang2022-05-025-5/+0
| | | Go 1.17 and later use modern `//go:build` constraints, the old `// +build:` constraints should be removed.
* Add Goroutine stack inspector to admin/monitor (#19207)zeripath2022-03-311-1/+5
| | | | | | | | | | | | | | | | | | | Continues on from #19202. Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile. The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines. If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound. In this way we should be able to more easily examine goroutines that have been stuck. A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add pprof labels in processes and for lifecycles (#19202)zeripath2022-03-253-40/+44
| | | | | | Use pprof labelling to help identify goroutines with stacks. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Immediately Hammer if second kill is sent (#18823)zeripath2022-02-192-2/+7
| | | | | | | Currently Gitea will wait for HammerTime or nice shutdown if kill -1 or kill -2 is sent. We should just immediately hammer if there is a second kill. Signed-off-by: Andrew Thornton <art27@cantab.net>
* format with gofumpt (#18184)65432022-01-201-0/+1
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Make SSL cipher suite configurable (#17440)zeripath2021-11-202-44/+3
|
* Fix some lints (#17337)Lunny Xiao2021-10-171-1/+3
| | | Fix some linting problems.
* Use pointer for wrappedConn methods (#17295)zeripath2021-10-121-3/+3
| | | | | Fix #17294 Signed-off-by: Andrew Thornton <art27@cantab.net>
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-221-3/+2
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Add bundle download for repository (#14538)John Olheiser2021-08-245-10/+15
| | | | | | | | | | | | | | | | | * Add bundle download Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix build tags Signed-off-by: jolheiser <john.olheiser@gmail.com> * Download specific commit Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Support HTTP/2 in Let's Encrypt (#16371)Stanley Hu2021-07-131-1/+1
| | | | | Modify the tlsConfig.NextProtos for Let's Encrypt and built-in HTTPS server in order to support HTTP/2. Co-authored-by: 6543 <6543@obermui.de>
* Set self-adjusting deadline for connection writing (#16068)zeripath2021-06-111-23/+41
| | | | | | | | | | | | | | | | | | | | | | | | | * Set self-adjusting deadline for connection writing In #16055 it appears that the simple 5s deadline doesn't work for large file writes. Now we can't - or at least shouldn't just set no deadline as go will happily let these connections block indefinitely. However, what seems reasonable is to set some minimum rate we expect for writing. This PR suggests the following algorithm: * Every write has a minimum timeout of 5s (adjustable at compile time.) * If there has been a previous write - then consider its previous deadline, add half of the minimum timeout + 2s per kb about to written. * If that new deadline is after the minimum timeout use that. Fix #16055 * Linearly increase timeout * Make PerWriteTimeout, PerWritePerKbTimeouts configurable Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Make modules/context.Context a context.Context (#16031)zeripath2021-05-311-0/+3
| | | | | | | | | | | | | | | * Make modules/context.Context a context.Context Signed-off-by: Andrew Thornton <art27@cantab.net> * Simplify context calls Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the base context for requests to the HammerContext Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow ↵zeripath2021-05-154-110/+121
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | level queue, reduce goroutines etc (#15693) * move shutdownfns, terminatefns and hammerfns out of separate goroutines Coalesce the shutdownfns etc into a list of functions that get run at shutdown rather then have them run at goroutines blocked on selects. This may help reduce the background select/poll load in certain configurations. * The LevelDB queues can actually wait on empty instead of polling Slight refactor to cause leveldb queues to wait on empty instead of polling. * Shutdown the shadow level queue once it is empty * Remove bytefifo additional goroutine for readToChan as it can just be run in run * Remove additional removeWorkers goroutine for workers * Simplify the AtShutdown and AtTerminate functions and add Channel Flusher * Add shutdown flusher to CUQ * move persistable channel shutdown stuff to Shutdown Fn * Ensure that UPCQ has the correct config * handle shutdown during the flushing * reduce risk of race between zeroBoost and addWorkers * prevent double shutdown Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add timeout to writing to responses (#15831)zeripath2021-05-141-0/+10
| | | | | | | | | | In #15826 it has become apparent that there are a few occasions when a response can hang during writing, and because there is no timeout go will happily just block interminably. This PR adds a fixed 5 second timeout to all writes to a connection. Fix #15826 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Drop back to use IsAnInteractiveSession for SVC (#15749)zeripath2021-05-071-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | * Drop back to use IsAnInteractiveSession for SVC There is an apparent permission change problem when using IsWindowsService to determine if the SVC manager should be used. This PR simply drops back to using IsAnInteractiveSession as this does not change behaviour. Fix #15454 Signed-off-by: Andrew Thornton <art27@cantab.net> * Yes staticcheck I know this is deprecated Signed-off-by: Andrew Thornton <art27@cantab.net> * Just leave me alone lint Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Fix regression from #14623 - use debug SVC handler only on interactive ↵zeripath2021-03-311-3/+22
| | | | | | | | | | | | | | | | sessions (#15210) Unfortunately #14623 changed from the deprecated IsInteractiveSession to IsWindowsService without recognising that they are the complement of each other. This means that Windows SVC control is not working correctly. This PR adds some Tracing statements but also fixes the bug. Fix #15159 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Make internal SSH server host key path configurable (#14918)zeripath2021-03-082-11/+11
| | | | | | | | | | | * Make SSH server host key path configurable * make it possible to have multiple keys * Make gitea.rsa the default key * Add some more logging Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix windows build error (#14263)Lunny Xiao2021-01-061-2/+4
| | | | | | | | | | | * fix build * take flash error message back and fix more windows lint error * performance optimization * own step to check lint for windows Co-authored-by: 6543 <6543@obermui.de>
* Move install pages out of main macaron routes (#13195)zeripath2020-10-192-2/+2
| | | | | | | | | | | | | * Move install pages out of main macaron loop Signed-off-by: Andrew Thornton <art27@cantab.net> * Update templates/post-install.tmpl Co-authored-by: Lauris BH <lauris@nix.lv> * remove prefetch Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set TLS minimum version to 1.2 (#12689)zeripath2020-09-021-0/+2
| | | Signed-off-by: Andrew Thornton <art27@cantab.net>
* Re-attempt to delete temporary upload if the file is locked by another ↵zeripath2020-08-111-1/+2
| | | | | | | | | | | process (#12447) Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util. Fix #12339 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: silverwind <me@silverwind.io>
* Pause, Resume, Release&Reopen, Add and Remove Logging from command line (#11777)zeripath2020-07-051-1/+4
| | | | | | | | | | | | | | | | | | | | | | * Make LogDescriptions race safe * Add manager commands for pausing, resuming, adding and removing loggers Signed-off-by: Andrew Thornton <art27@cantab.net> * Placate lint * Ensure that file logger is run! * Add support for smtp and conn Signed-off-by: Andrew Thornton <art27@cantab.net> * Add release-and-reopen Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Handle panics that percolate up to the graceful module (#11291)zeripath2020-05-141-1/+34
| | | | | | | | | | | | | | | | | | * Handle panics in graceful goroutines Adds a some deferred functions to handle panics in graceful goroutines Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle panic in webhook.Deliver Signed-off-by: Andrew Thornton <art27@cantab.net> * Handle panic in mirror.syncMirror Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Prevent panic during wrappedConn close at hammertime (#11219)zeripath2020-04-261-0/+11
| | | | | | | | | | | * Prevent panic during wrappedConn close at hammertime Signed-off-by: Andrew Thornton <art27@cantab.net> * Update modules/graceful/server.go * Fix extraneous debug in goldmark.go Signed-off-by: Andrew Thornton <art27@cantab.net>
* Restore graceful restart on SIGHUP (#10224)zeripath2020-02-111-2/+2
| | | | Co-authored-by: Lauris BH <lauris@nix.lv>
* Prevent double waitgroup decrement (#10170)zeripath2020-02-071-3/+7
| | | | * Prevent double waitgroup decrement
* [UI] Show pull icon on pull (#10061)65432020-01-291-1/+1
| | | | | * make fmt * show pull icon on pull
* Queue: Make WorkerPools and Queues flushable (#10001)zeripath2020-01-282-20/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make WorkerPools and Queues flushable Adds Flush methods to Queues and the WorkerPool Further abstracts the WorkerPool Adds a final step to Flush the queues in the defer from PrintCurrentTest Fixes an issue with Settings inheritance in queues Signed-off-by: Andrew Thornton <art27@cantab.net> * Change to for loop * Add IsEmpty and begin just making the queues composed WorkerPools * subsume workerpool into the queues and create a flushable interface * Add manager command * Move flushall to queue.Manager and add to testlogger * As per @guillep2k * as per @guillep2k * Just make queues all implement flushable and clean up the wrapped queue flushes * cope with no timeout Co-authored-by: Lauris BH <lauris@nix.lv>
* Make CertFile and KeyFile relative to CustomPath (#9868)zeripath2020-01-191-2/+16
| | | | | | | | | | | | | | | | * Make CertFile and KeyFile relative to CustomPath The current code will absolute CertFile and KeyFile against the current working directory. This is quite unexpected for users. This code makes relative paths absolute against the CustomPath. Fix #4196 * Improve error reporting when reading certificates * Apply suggestions from code review Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix missed change to GetManager() (#9361)zeripath2019-12-151-1/+1
|
* Graceful: Xorm, RepoIndexer, Cron and Others (#9282)zeripath2019-12-158-95/+143
| | | | | | | | | | | | * Change graceful to use a singleton obtained through GetManager instead of a global. * Graceful: Make TestPullRequests shutdownable * Graceful: Make the cron tasks graceful * Graceful: AddTestPullRequest run in graceful ctx * Graceful: SyncMirrors shutdown * Graceful: SetDefaultContext for Xorm to be HammerContext * Avoid starting graceful for migrate commands and checkout * Graceful: DeliverHooks now can be shutdown * Fix multiple syncing errors in modules/sync/UniqueQueue & Make UniqueQueue closable * Begin the process of making the repo indexer shutdown gracefully
* Add comment to exported function WindowsServiceName (make revive) (#9241)David Svantesson2019-12-031-0/+1
|
* Graceful: Cancel Process on monitor pages & HammerTime (#9213)zeripath2019-11-304-46/+255
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Graceful: Create callbacks to with contexts * Graceful: Say when Gitea is completely finished * Graceful: Git and Process within HammerTime Force all git commands to terminate at HammerTime Force all process commands to terminate at HammerTime Move almost all git processes to run as git Commands * Graceful: Always Hammer after Shutdown * ProcessManager: Add cancel functionality * Fix tests * Make sure that process.Manager.Kill() cancels * Make threadsafe access to Processes and remove own unused Kill * Remove cmd from the process manager as it is no longer used * the default context is the correct context * get rid of double till
* Graceful: Allow graceful restart for unix sockets (#9113)zeripath2019-11-242-1/+19
| | | | Previously we could not handle graceful restarts for http over unix sockets. These can now be handled.
* Run make fmt on master (#9129)David Svantesson2019-11-221-5/+4
|
* Add Graceful shutdown for Windows and hooks for shutdown of goroutines (#8964)zeripath2019-11-2112-281/+550
| | | | | | | | | | | | | | | | | | * Graceful Shutdown for windows and others Restructures modules/graceful, adding shutdown for windows, removing and replacing the old minwinsvc code. Creates a new waitGroup - terminate which allows for goroutines to finish up after the shutdown of the servers. Shutdown and terminate hooks are added for goroutines. * Remove unused functions - these can be added in a different PR * Add startup timeout functionality * Document STARTUP_TIMEOUT
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-09 12:19:11 +0000