summaryrefslogtreecommitdiffstats
path: root/services/webhook/webhook.go
Commit message (Collapse)AuthorAgeFilesLines
* Warn instead of reporting an error when a webhook cannot be found (#26039)puni98692023-07-281-1/+6
| | | | | | | | | Attemp fix: #25744 Fixing the log level when we delete any repo then we get error hook not found by id. That should be warn level to reduce the noise in the logs. --------- Co-authored-by: delvh <dev.lh@web.de>
* Rewrite queue (#24505)wxiaoguang2023-05-081-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # ⚠️ Breaking Many deprecated queue config options are removed (actually, they should have been removed in 1.18/1.19). If you see the fatal message when starting Gitea: "Please update your app.ini to remove deprecated config options", please follow the error messages to remove these options from your app.ini. Example: ``` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options ``` Many options in `[queue]` are are dropped, including: `WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`, `BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed from app.ini. # The problem The old queue package has some legacy problems: * complexity: I doubt few people could tell how it works. * maintainability: Too many channels and mutex/cond are mixed together, too many different structs/interfaces depends each other. * stability: due to the complexity & maintainability, sometimes there are strange bugs and difficult to debug, and some code doesn't have test (indeed some code is difficult to test because a lot of things are mixed together). * general applicability: although it is called "queue", its behavior is not a well-known queue. * scalability: it doesn't seem easy to make it work with a cluster without breaking its behaviors. It came from some very old code to "avoid breaking", however, its technical debt is too heavy now. It's a good time to introduce a better "queue" package. # The new queue package It keeps using old config and concept as much as possible. * It only contains two major kinds of concepts: * The "base queue": channel, levelqueue, redis * They have the same abstraction, the same interface, and they are tested by the same testing code. * The "WokerPoolQueue", it uses the "base queue" to provide "worker pool" function, calls the "handler" to process the data in the base queue. * The new code doesn't do "PushBack" * Think about a queue with many workers, the "PushBack" can't guarantee the order for re-queued unhandled items, so in new code it just does "normal push" * The new code doesn't do "pause/resume" * The "pause/resume" was designed to handle some handler's failure: eg: document indexer (elasticsearch) is down * If a queue is paused for long time, either the producers blocks or the new items are dropped. * The new code doesn't do such "pause/resume" trick, it's not a common queue's behavior and it doesn't help much. * If there are unhandled items, the "push" function just blocks for a few seconds and then re-queue them and retry. * The new code doesn't do "worker booster" * Gitea's queue's handlers are light functions, the cost is only the go-routine, so it doesn't make sense to "boost" them. * The new code only use "max worker number" to limit the concurrent workers. * The new "Push" never blocks forever * Instead of creating more and more blocking goroutines, return an error is more friendly to the server and to the end user. There are more details in code comments: eg: the "Flush" problem, the strange "code.index" hanging problem, the "immediate" queue problem. Almost ready for review. TODO: * [x] add some necessary comments during review * [x] add some more tests if necessary * [x] update documents and config options * [x] test max worker / active worker * [x] re-run the CI tasks to see whether any test is flaky * [x] improve the `handleOldLengthConfiguration` to provide more friendly messages * [x] fine tune default config values (eg: length?) ## Code coverage: ![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
* Add user webhooks (#21563)KN4CK3R2023-03-101-5/+5
| | | | | | | Currently we can add webhooks for organizations but not for users. This PR adds the latter. You can access it from the current users settings. ![grafik](https://user-images.githubusercontent.com/1666336/197391408-15dfdc23-b476-4d0c-82f7-9bc9b065988f.png)
* Restructure `webhook` module (#22256)delvh2023-01-011-30/+26
| | | | | | | | | | | | Previously, there was an `import services/webhooks` inside `modules/notification/webhook`. This import was removed (after fighting against many import cycles). Additionally, `modules/notification/webhook` was moved to `modules/webhook`, and a few structs/constants were extracted from `models/webhooks` to `modules/webhook`. Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-271-2/+1
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Ensure that Webhook tasks are not double delivered (#21558)zeripath2022-11-231-9/+16
| | | | | | | | | | | | | | | | | | When re-retrieving hook tasks from the DB double check if they have not been delivered in the meantime. Further ensure that tasks are marked as delivered when they are being delivered. In addition: * Improve the error reporting and make sure that the webhook task population script runs in a separate goroutine. * Only get hook task IDs out of the DB instead of the whole task when repopulating the queue * When repopulating the queue make the DB request paged Ref #17940 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add `context.Context` to more methods (#21546)KN4CK3R2022-11-191-1/+1
| | | | | | | This PR adds a context parameter to a bunch of methods. Some helper `xxxCtx()` methods got replaced with the normal name now. Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-241-5/+5
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Decouple HookTask from Repository (#17940)KN4CK3R2022-10-211-57/+47
| | | | | | | | | | | | | At the moment a repository reference is needed for webhooks. With the upcoming package PR we need to send webhooks without a repository reference. For example a package is uploaded to an organization. In theory this enables the usage of webhooks for future user actions. This PR removes the repository id from `HookTask` and changes how the hooks are processed (see `services/webhook/deliver.go`). In a follow up PR I want to remove the usage of the `UniqueQueue´ and replace it with a normal queue because there is no reason to be unique. Co-authored-by: 6543 <6543@obermui.de>
* Add more linters to improve code readability (#19989)Wim2022-06-201-2/+2
| | | | | | | | | | Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability - nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length. - unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions - wastedassign - https://github.com/sanposhiho/wastedassign - wastedassign finds wasted assignment statements. - notlintlint - Reports ill-formed or insufficient nolint directives - stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
* Move almost all functions' parameter db.Engine to context.Context (#19748)Lunny Xiao2022-05-201-5/+7
| | | | * Move almost all functions' parameter db.Engine to context.Context * remove some unnecessary wrap functions
* Use queue instead of memory queue in webhook send service (#19390)Lunny Xiao2022-04-251-9/+40
|
* Remove legacy `unknwon/com` package (#19298)wxiaoguang2022-04-021-3/+4
| | | | | | | Follows: #19284 * The `CopyDir` is only used inside test code * Rewrite `ToSnakeCase` with more test cases * The `RedisCacher` only put strings into cache, here we use internal `toStr` to replace the legacy `ToStr` * The `UniqueQueue` can use string as ID directly, no need to call `ToStr`
* Only send webhook events to active system webhooks and only deliver to ↵zeripath2022-03-281-1/+1
| | | | | | | | | | | | | | | | | active hooks (#19234) There is a bug in the system webhooks whereby the active state is not checked when webhooks are prepared and there is a bug that deactivating webhooks do not prevent queued deliveries. * Only add SystemWebhooks to the prepareWebhooks list if they are active * At the time of delivery if the underlying webhook is not active mark it as "delivered" but with a failed delivery so it does not get delivered. Fix #19220 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add packagist webhook (#18224)Dr. Tobias Quathamer2022-01-231-0/+4
| | | | Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* format with gofumpt (#18184)65432022-01-201-36/+34
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Add replay of webhooks. (#18191)KN4CK3R2022-01-051-0/+12
|
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-101-5/+5
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Move webhook into models/webhook/ (#17579)Lunny Xiao2021-11-101-32/+33
|
* [API] generalize list header (#16551)65432021-08-121-2/+10
| | | | | | | | | | | | | * Add info about list endpoints to CONTRIBUTING.md * Let all list endpoints return X-Total-Count header * Add TODOs for GetCombinedCommitStatusByRef * Fix models/issue_stopwatch.go * Rrefactor models.ListDeployKeys * Introduce helper func and use them for SetLinkHeader related func
* Add support for corporate WeChat webhooks (#15910)lengyuqu2021-07-231-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * 企业微信webhook * 企业微信webhook * 企业微信webhook * Update templates/admin/hook_new.tmpl Co-authored-by: a1012112796 <1012112796@qq.com> * Update services/webhook/wechatwork.go Co-authored-by: a1012112796 <1012112796@qq.com> * 修善wechatwork * 修善wechatwork * fix * Update locale_cs-CZ.ini fix * fix build * fix * fix build * make webhooks.zh-cn.md * delet unnecessary blank line * delet unnecessary blank line * 企业微信webhook * 企业微信webhook * 企业微信webhook * Update templates/admin/hook_new.tmpl Co-authored-by: a1012112796 <1012112796@qq.com> * Update services/webhook/wechatwork.go Co-authored-by: a1012112796 <1012112796@qq.com> * 修善wechatwork * 修善wechatwork * fix * fix build * fix * fix build * make webhooks.zh-cn.md * delet unnecessary blank line * delet unnecessary blank line * 企业微信webhook * 企业微信webhook * 企业微信webhook * 企业微信webhook * 企业微信webhook * fix * fix * 企业微信webhook * 企业微信webhook * 企业微信webhook * fix wechat * fix wechat * fix wechat * fix wechat * Fix invalid params and typo of email templates (#16394) Signed-off-by: Meano <meanocat@gmail.com> * Add LRU mem cache implementation (#16226) The current default memory cache implementation is unbounded in size and number of objects cached. This is hardly ideal. This PR proposes creating a TwoQueue LRU cache as the underlying cache for Gitea. The cache is limited by the number of objects stored in the cache (rather than size) for simplicity. The default number of objects is 50000 - which is perhaps too small as most of our objects cached are going to be much less than 1kB. It may be worth considering using a different LRU implementation that actively limits sizes or avoids GC - however, this is just a beginning implementation. Signed-off-by: Andrew Thornton <art27@cantab.net> * [skip ci] Updated translations via Crowdin * Replace `plugins/docker` with `techknowlogick/drone-docker`in ci (#16407) * plugins/docker -> techknowlogick/drone-docker * It is multi-arch * docs: rewrite email setup (#16404) * Add intro for both the docs page and mailer methods * Fix numbering level in SMTP section * Recommends implicit TLS Signed-off-by: Bagas Sanjaya <bagasdotme@gmail.com> * Validate Issue Index before querying DB (#16406) * Fix external renderer (#16401) * fix external renderer * use GBackground context as fallback * no fallback, return error Co-authored-by: Lauris BH <lauris@nix.lv> * Add checkbox to delete pull branch after successful merge (#16049) * Add checkbox to delete pull branch after successful merge * Omit DeleteBranchAfterMerge field in json * Log a warning instead of error when PR head branch deleted * Add DefaultDeleteBranchAfterMerge to PullRequestConfig * Add support for delete_branch_after_merge via API * Fix for API: the branch should be deleted from the HEAD repo If head and base repo are the same, reuse the already opened ctx.Repo.GitRepo * Don't delegate to CleanupBranch, only reuse branch deletion code CleanupBranch contains too much logic that has already been performed by the Merge * Reuse gitrepo in MergePullRequest Co-authored-by: Andrew Thornton <art27@cantab.net> * [skip ci] Updated translations via Crowdin * Detect encoding changes while parsing diff (#16330) * Detect encoding changes while parsing diff * Let branch/tag name be a valid ref to get CI status (#16400) * fix #16384# * refactor: move shared helper func to utils package * extend Tests * use ctx.Repo.GitRepo if not nil * fix * fix * 企业微信webhook * 企业微信webhook * 企业微信webhook * fix build * fix build * Apply suggestions from code review Co-authored-by: a1012112796 <1012112796@qq.com> Co-authored-by: myheavily <myheavily> Co-authored-by: zhaoxin <gitea@fake.local> Co-authored-by: Meano <Meano@foxmail.com> Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: GiteaBot <teabot@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Bagas Sanjaya <bagasdotme@gmail.com> Co-authored-by: Norwin <noerw@users.noreply.github.com> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Jimmy Praet <jimmy.praet@telenet.be> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Refactor Webhook + Add X-Hub-Signature (#16176)KN4CK3R2021-06-271-32/+8
| | | | | | | | | | | | | | This PR removes multiple unneeded fields from the `HookTask` struct and adds the two headers `X-Hub-Signature` and `X-Hub-Signature-256`. ## :warning: BREAKING :warning: * The `Secret` field is no longer passed as part of the payload. * "Breaking" change (or fix?): The webhook history shows the real called url and not the url registered in the webhook (`deliver.go`@129). Close #16115 Fixes #7788 Fixes #11755 Co-authored-by: zeripath <art27@cantab.net>
* fix a small nit (#14769)a10121127962021-02-221-0/+2
|
* Added option to disable webhooks (#13176)Paweł Bogusławski2021-02-111-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Added option to disable web hooks This mod introduces DISABLE_WEB_HOOKS parameter in [security] section of app.ini (by default set to false). If set to true it disables web hooks feature. Any existing undelivered web hook tasks will be cancelled. Any existing web hook definitions will be left untouched in db but its delivery tasks will be ignored. Author-Change-Id: IB#1105130 * Webhook spelling fixed Webhook spelling fixed. Fixes: 07df6614dc84cdd2e9f39c57577fa1062bd70012 Related: https://github.com/go-gitea/gitea/pull/13176#pullrequestreview-510868421 Author-Change-Id: IB#1105174 * Parameter description fixed Parameter description fixed. Fixes: 07df6614dc84cdd2e9f39c57577fa1062bd70012 Related: https://github.com/go-gitea/gitea/pull/13176#pullrequestreview-514086107 Author-Change-Id: IB#1105174
* Fix wrong type on hooktask to convert typ from char(16) to varchar(16) (#14148)Lunny Xiao2021-01-061-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | * Fix wrong type on hooktask to convert typ from char(16) to varchar(16) * Fix bugs * Improve code * Use different trim function for MSSQL * Fix bug * Removed wrong changed line * Removed wrong changed line * Fix nullable * Fix lint * Ignore sqlite on migration * Fix mssql modify column failure * Move modifyColumn to migrations.go so that other migrate function could use it
* Possible fix the webhook API creation (#13960)Lunny Xiao2020-12-121-2/+5
| | | | | * Possible fix the webhook API creation * Fix api create webhook bug
* Fix bug for webhook and feishu caused by API changed (#13937)Lunny Xiao2020-12-111-1/+1
| | | | | * Fix bug for webhook * Fix bug for feishu
* Fix broken migration on webhook (#13911)Lunny Xiao2020-12-091-4/+4
| | | | | | | * Fix broken migration on webhook * Fix lint Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Move webhook type from int to string (#13664)Lunny Xiao2020-12-081-0/+232
* Move webhook type from int to string * rename webhook_services * finish refactor * Fix merge * Ignore unnecessary ci Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: zeripath <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2025-04-18 11:17:05 +0000