summaryrefslogtreecommitdiffstats
path: root/services/mailer/mailer.go
Commit message (Collapse)AuthorAgeFilesLines
* Improve queue and logger context (#24924)wxiaoguang2023-05-261-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | Before there was a "graceful function": RunWithShutdownFns, it's mainly for some modules which doesn't support context. The old queue system doesn't work well with context, so the old queues need it. After the queue refactoring, the new queue works with context well, so, use Golang context as much as possible, the `RunWithShutdownFns` could be removed (replaced by RunWithCancel for context cancel mechanism), the related code could be simplified. This PR also fixes some legacy queue-init problems, eg: * typo : archiver: "unable to create codes indexer queue" => "unable to create repo-archive queue" * no nil check for failed queues, which causes unfriendly panic After this PR, many goroutines could have better display name: ![image](https://github.com/go-gitea/gitea/assets/2114189/701b2a9b-8065-4137-aeaa-0bda2b34604a) ![image](https://github.com/go-gitea/gitea/assets/2114189/f1d5f50f-0534-40f0-b0be-f2c9daa5fe92)
* Rewrite queue (#24505)wxiaoguang2023-05-081-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # ⚠️ Breaking Many deprecated queue config options are removed (actually, they should have been removed in 1.18/1.19). If you see the fatal message when starting Gitea: "Please update your app.ini to remove deprecated config options", please follow the error messages to remove these options from your app.ini. Example: ``` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options ``` Many options in `[queue]` are are dropped, including: `WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`, `BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed from app.ini. # The problem The old queue package has some legacy problems: * complexity: I doubt few people could tell how it works. * maintainability: Too many channels and mutex/cond are mixed together, too many different structs/interfaces depends each other. * stability: due to the complexity & maintainability, sometimes there are strange bugs and difficult to debug, and some code doesn't have test (indeed some code is difficult to test because a lot of things are mixed together). * general applicability: although it is called "queue", its behavior is not a well-known queue. * scalability: it doesn't seem easy to make it work with a cluster without breaking its behaviors. It came from some very old code to "avoid breaking", however, its technical debt is too heavy now. It's a good time to introduce a better "queue" package. # The new queue package It keeps using old config and concept as much as possible. * It only contains two major kinds of concepts: * The "base queue": channel, levelqueue, redis * They have the same abstraction, the same interface, and they are tested by the same testing code. * The "WokerPoolQueue", it uses the "base queue" to provide "worker pool" function, calls the "handler" to process the data in the base queue. * The new code doesn't do "PushBack" * Think about a queue with many workers, the "PushBack" can't guarantee the order for re-queued unhandled items, so in new code it just does "normal push" * The new code doesn't do "pause/resume" * The "pause/resume" was designed to handle some handler's failure: eg: document indexer (elasticsearch) is down * If a queue is paused for long time, either the producers blocks or the new items are dropped. * The new code doesn't do such "pause/resume" trick, it's not a common queue's behavior and it doesn't help much. * If there are unhandled items, the "push" function just blocks for a few seconds and then re-queue them and retry. * The new code doesn't do "worker booster" * Gitea's queue's handlers are light functions, the cost is only the go-routine, so it doesn't make sense to "boost" them. * The new code only use "max worker number" to limit the concurrent workers. * The new "Push" never blocks forever * Instead of creating more and more blocking goroutines, return an error is more friendly to the server and to the end user. There are more details in code comments: eg: the "Flush" problem, the strange "code.index" hanging problem, the "immediate" queue problem. Almost ready for review. TODO: * [x] add some necessary comments during review * [x] add some more tests if necessary * [x] update documents and config options * [x] test max worker / active worker * [x] re-run the CI tasks to see whether any test is flaky * [x] improve the `handleOldLengthConfiguration` to provide more friendly messages * [x] fine tune default config values (eg: length?) ## Code coverage: ![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
* Add ntlm authentication support for mail (#23811)木木田2023-05-021-0/+32
| | | | | | | | | Add ntlm authentication support for mail use "github.com/Azure/go-ntlmssp" --------- Co-authored-by: yangtan_win <YangTan@Fitsco.com.cn> Co-authored-by: silverwind <me@silverwind.io>
* Prevent multiple `To` recipients (#22566)KN4CK3R2023-01-221-5/+5
| | | | | | Change the mailer interface to prevent leaking of possible hidden email addresses when sending to multiple recipients. Co-authored-by: Gusted <williamzijl7@hotmail.com>
* Add support for incoming emails (#22056)KN4CK3R2023-01-141-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | closes #13585 fixes #9067 fixes #2386 ref #6226 ref #6219 fixes #745 This PR adds support to process incoming emails to perform actions. Currently I added handling of replies and unsubscribing from issues/pulls. In contrast to #13585 the IMAP IDLE command is used instead of polling which results (in my opinion 😉) in cleaner code. Procedure: - When sending an issue/pull reply email, a token is generated which is present in the Reply-To and References header. - IMAP IDLE waits until a new email arrives - The token tells which action should be performed A possible signature and/or reply gets stripped from the content. I added a new service to the drone pipeline to test the receiving of incoming mails. If we keep this in, we may test our outgoing emails too in future. Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-271-2/+1
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Correct the fallbacks for mailer configuration (#21945)zeripath2022-11-271-2/+2
| | | | | | | | | | | | | | | Unfortunately the fallback configuration code for [mailer] that were added in #18982 are incorrect. When you read a value from an ini section that key is added. This leads to a failure of the fallback mechanism. Further there is also a spelling mistake in the startTLS configuration. This PR restructures the mailer code to first map the deprecated settings on to the new ones - and then use ini.MapTo to map those on to the struct with additional validation as necessary. Ref #21744 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-241-13/+13
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Share HTML template renderers and create a watcher framework (#20218)zeripath2022-08-281-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | The recovery, API, Web and package frameworks all create their own HTML Renderers. This increases the memory requirements of Gitea unnecessarily with duplicate templates being kept in memory. Further the reloading framework in dev mode for these involves locking and recompiling all of the templates on each load. This will potentially hide concurrency issues and it is inefficient. This PR stores the templates renderer in the context and stores this context in the NormalRoutes, it then creates a fsnotify.Watcher framework to watch files. The watching framework is then extended to the mailer templates which were previously not being reloaded in dev. Then the locales are simplified to a similar structure. Fix #20210 Fix #20211 Fix #20217 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix init mail render logic (#20704)KN4CK3R2022-08-081-0/+3
| | | This bug affects tests which are sending emails (#20307). Some tests reinitialise the web routes (like `TestNodeinfo`) which messed up the mail templates. There is no reason why the templates should be loaded in the routes method.
* Rework mailer settings (#18982)Clar Fon2022-08-021-39/+56
| | | | | | | | | | | * `PROTOCOL`: can be smtp, smtps, smtp+startls, smtp+unix, sendmail, dummy * `SMTP_ADDR`: domain for SMTP, or path to unix socket * `SMTP_PORT`: port for SMTP; defaults to 25 for `smtp`, 465 for `smtps`, and 587 for `smtp+startls` * `ENABLE_HELO`, `HELO_HOSTNAME`: reverse `DISABLE_HELO` to `ENABLE_HELO`; default to false + system hostname * `FORCE_TRUST_SERVER_CERT`: replace the unclear `SKIP_VERIFY` * `CLIENT_CERT_FILE`, `CLIENT_KEY_FILE`, `USE_CLIENT_CERT`: clarify client certificates here Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Set Setpgid on child git processes (#19865)zeripath2022-06-031-0/+1
| | | | | | | | | | When Gitea is running as PID 1 git will occassionally orphan child processes leading to (defunct) processes. This PR simply sets Setpgid to true on these child processes meaning that these defunct processes will also be correctly reaped. Fix #19077 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Pause queues (#15928)zeripath2022-01-221-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Start adding mechanism to return unhandled data Signed-off-by: Andrew Thornton <art27@cantab.net> * Create pushback interface Signed-off-by: Andrew Thornton <art27@cantab.net> * Add Pausable interface to WorkerPool and Manager Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and PushBack for the bytefifos Signed-off-by: Andrew Thornton <art27@cantab.net> * Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues Signed-off-by: Andrew Thornton <art27@cantab.net> * Wire in UI for pausing Signed-off-by: Andrew Thornton <art27@cantab.net> * add testcases and fix a few issues Signed-off-by: Andrew Thornton <art27@cantab.net> * fix build Signed-off-by: Andrew Thornton <art27@cantab.net> * prevent "race" in the test Signed-off-by: Andrew Thornton <art27@cantab.net> * fix jsoniter mismerge Signed-off-by: Andrew Thornton <art27@cantab.net> * fix conflicts Signed-off-by: Andrew Thornton <art27@cantab.net> * fix format Signed-off-by: Andrew Thornton <art27@cantab.net> * Add warnings for no worker configurations and prevent data-loss with redis/levelqueue Signed-off-by: Andrew Thornton <art27@cantab.net> * Use StopTimer Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* format with gofumpt (#18184)65432022-01-201-7/+3
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Add option to convert CRLF to LF line endings for sendmail (#18075)zeripath2022-01-061-2/+9
| | | | | | | | | | | | | It appears that several versions of sendmail require that the mail is sent to them with LF line endings instead of CRLF endings - which of course they will then convert back to CRLF line endings to comply with the SMTP standard. This PR adds another setting SENDMAIL_CONVERT_CRLF which will pass the message writer through a filter. This will filter out and convert CRLFs to LFs before writing them out to sendmail. Fix #18024 Signed-off-by: Andrew Thornton <art27@cantab.net>
* allways set a message-id on mails (#17900)Garionion2021-12-081-0/+16
| | | | | | | * allways set a message-id on mails * Add unit tests for mailer & Message-ID Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Make Requests Processes and create process hierarchy. Associate ↵zeripath2021-11-301-7/+4
| | | | | | | | | OpenRepository with context. (#17125) This PR registers requests with the process manager and manages hierarchy within the processes. Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add settings to allow different SMTP envelope from address (#17479)zeripath2021-11-191-3/+14
| | | | | | | | | | | * Add settings to allow different SMTP envelope from address Sometimes it may be advisable to hide or alias the from address on an SMTP mail envelope. This PR adds two new options to the mailer to allow setting of an overriding from address. Fix #17477 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix various documentation, user-facing, and source comment typos (#16367)luzpaz2021-07-081-1/+1
| | | | | * Fix various doc, user-facing, and source comment typos Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
* [refactor] mailer service (#15072)65432021-04-021-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Unexport SendUserMail * Instead of "[]*models.User" or "[]string" lists infent "[]*MailRecipient" for mailer * adopt * code format * TODOs for "i18n" * clean * no fallback for lang -> just use english * lint * exec testComposeIssueCommentMessage per lang and use only emails * rm MailRecipient * Dont reload from users from db if you alredy have in ram * nits * minimize diff Signed-off-by: 6543 <6543@obermui.de> * localize subjects * linter ... * Tr extend * start tmpl edit ... * Apply suggestions from code review * use translation.Locale * improve mailIssueCommentBatch Signed-off-by: Andrew Thornton <art27@cantab.net> * add i18n to datas Signed-off-by: Andrew Thornton <art27@cantab.net> * a comment Co-authored-by: Andrew Thornton <art27@cantab.net>
* Sendmail should create a process on the gitea system and have a default ↵zeripath2020-05-031-1/+13
| | | | | | | | | | | | timeout (#11256) * Make sure that sendmail processes register with the process manager * Provide a timeout for these (initially of 5 minutes) * Add configurable value and tie in to documentation * Tie in to the admin config page. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Switch to use -f instead of -F for sendmail (#9961)zeripath2020-01-241-1/+1
| | | Fix #9385
* Move mailer to use a queue (#9789)zeripath2020-01-161-31/+59
| | | | | | | | * Move mailer to use a queue * Make sectionMap map[string]bool * Ensure that Message is json encodable
* Rewrite delivery of issue and comment mails (#9009)guillep2k2019-11-181-1/+10
| | | | | | | | | | | | | | | | * Mail issue subscribers, rework the function * Simplify a little more * Fix unused variable * Refactor mail delivery to avoid heavy load on server * Avoid splitting into too many goroutines * Fix comments and optimize GetMaileableUsersByIDs() * Fix return on errors
* Move all mail related codes from models to services/mailer (#7200)Lunny Xiao2019-09-241-0/+303
* move all mail related codes from models to modules/mailer * fix lint * use DBContext instead Engine * use WithContext not WithEngine * Use DBContext instead of Engine * don't use defer when sess.Close() * move DBContext to context.go and add some methods * move mailer from modules/ to services * fix lint * fix tests * fix fmt * add gitea copyright * fix tests * don't expose db functions * make code clear * add DefaultDBContext * fix build * fix bug
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-14 05:40:58 +0000