Rewrite logger system (#24726)

## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>

1 files changed, 25 insertions, 67 deletions

diff --git a/custom/conf/app.example.ini b/custom/conf/app.example.ini
index 91d829c0ca..27e56542ee 100644
--- a/custom/conf/app.example.ini
+++ b/custom/conf/app.example.ini
@@ -230,7 +230,6 @@ RUN_MODE = ; prod
 ;;
 ;; Disable CDN even in "prod" mode
 ;OFFLINE_MODE = false
-;DISABLE_ROUTER_LOG = false
 ;;
 ;; TLS Settings: Either ACME or manual
 ;; (Other common TLS configuration are found before)
@@ -387,7 +386,7 @@ USER = root
 ;ITERATE_BUFFER_SIZE = 50
 ;;
 ;; Show the database generated SQL
-LOG_SQL = false ; if unset defaults to true
+;LOG_SQL = false
 ;;
 ;; Maximum number of DB Connect retries
 ;DB_RETRIES = 10
@@ -550,34 +549,32 @@ ENABLE = true
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Main Logger
 ;;
-;; Either "console", "file", "conn", "smtp" or "database", default is "console"
+;; Either "console", "file" or "conn", default is "console"
 ;; Use comma to separate multiple modes, e.g. "console, file"
 MODE = console
 ;;
-;; Either "Trace", "Debug", "Info", "Warn", "Error", "Critical" or "None", default is "Info"
+;; Either "Trace", "Debug", "Info", "Warn", "Error" or "None", default is "Info"
 LEVEL = Info
 ;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; Router Logger
-;;
-;; Switch off the router log
-;DISABLE_ROUTER_LOG=false
+;; Print Stacktrace with logs (rarely helpful, do not set) Either "Trace", "Debug", "Info", "Warn", "Error", default is "None"
+;STACKTRACE_LEVEL = None
 ;;
-;; Set the log "modes" for the router log (if file is set the log file will default to router.log)
-ROUTER = console
+;; Buffer length of the channel, keep it as it is if you don't know what it is.
+;BUFFER_LEN = 10000
 ;;
-;; The router will log different things at different levels.
+;; Sub logger modes, a single comma means use default MODE above, empty means disable it
+;logger.access.MODE=
+;logger.router.MODE=,
+;logger.xorm.MODE=,
 ;;
-;; * started messages will be logged at TRACE level
-;; * polling/completed routers will be logged at INFO
-;; * slow routers will be logged at WARN
-;; * failed routers will be logged at WARN
+;; Collect SSH logs (Creates log from ssh git request)
 ;;
-;; The routing level will default to that of the system but individual router level can be set in
-;; [log.<mode>.router] LEVEL
+;ENABLE_SSH_LOG = false
 ;;
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;
+;; Access Logger (Creates log in NCSA common log format)
+;;
 ;; Print request id which parsed from request headers in access log, when access log is enabled.
 ;; * E.g:
 ;; * In request Header:         X-Request-ID: test-id-123
@@ -587,57 +584,32 @@ ROUTER = console
 ;; If you configure more than one in the .ini file, it will match in the order of configuration,
 ;; and the first match will be finally printed in the log.
 ;; * E.g:
-;; * In reuqest Header:         X-Trace-ID: trace-id-1q2w3e4r
+;; * In request Header:         X-Trace-ID: trace-id-1q2w3e4r
 ;; * Configuration in app.ini:  REQUEST_ID_HEADERS = X-Request-ID, X-Trace-ID, X-Req-ID
 ;; * Print in log:              127.0.0.1:58384 - - [14/Feb/2023:16:33:51 +0800] "trace-id-1q2w3e4r"
 ;;
-;; REQUEST_ID_HEADERS =
-
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;
-;; Access Logger (Creates log in NCSA common log format)
-;;
-;ENABLE_ACCESS_LOG = false
-;;
-;; Set the log "modes" for the access log (if file is set the log file will default to access.log)
-;ACCESS = file
+;REQUEST_ID_HEADERS =
 ;;
 ;; Sets the template used to create the access log.
 ;ACCESS_LOG_TEMPLATE = {{.Ctx.RemoteHost}} - {{.Identity}} {{.Start.Format "[02/Jan/2006:15:04:05 -0700]" }} "{{.Ctx.Req.Method}} {{.Ctx.Req.URL.RequestURI}} {{.Ctx.Req.Proto}}" {{.ResponseWriter.Status}} {{.ResponseWriter.Size}} "{{.Ctx.Req.Referer}}" "{{.Ctx.Req.UserAgent}}"
-;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;
-;; SSH log (Creates log from ssh git request)
-;;
-;ENABLE_SSH_LOG = false
-;;
-;; Other Settings
-;;
-;; Print Stacktraces with logs. (Rarely helpful.) Either "Trace", "Debug", "Info", "Warn", "Error", "Critical", default is "None"
-;STACKTRACE_LEVEL = None
-;;
-;; Buffer length of the channel, keep it as it is if you don't know what it is.
-;BUFFER_LEN = 10000
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; Creating specific log configuration
 ;;
-;; You can set specific configuration for individual modes and subloggers
+;; Log modes (aka log writers)
 ;;
-;; Configuration available to all log modes/subloggers
+;[log.%(WriterMode)]
+;MODE=console/file/conn/...
 ;LEVEL=
 ;FLAGS = stdflags
 ;EXPRESSION =
 ;PREFIX =
 ;COLORIZE = false
 ;;
-;; For "console" mode only
+;[log.console]
 ;STDERR = false
 ;;
-;; For "file" mode only
-;LEVEL =
-;; Set the file_name for the logger. If this is a relative path this
-;; will be relative to ROOT_PATH
+;[log.file]
+;; Set the file_name for the logger. If this is a relative path this will be relative to ROOT_PATH
 ;FILE_NAME =
 ;; This enables automated log rotate(switch of following options), default is true
 ;LOG_ROTATE = true
@@ -651,9 +623,8 @@ ROUTER = console
 ;COMPRESS = true
 ;; compression level see godoc for compress/gzip
 ;COMPRESSION_LEVEL = -1
-;
-;; For "conn" mode only
-;LEVEL =
+;;
+;[log.conn]
 ;; Reconnect host for every single message, default is false
 ;RECONNECT_ON_MSG = false
 ;; Try to reconnect when connection is lost, default is false
@@ -662,19 +633,6 @@ ROUTER = console
 ;PROTOCOL = tcp
 ;; Host address
 ;ADDR =
-;
-;; For "smtp" mode only
-;LEVEL =
-;; Name displayed in mail title, default is "Diagnostic message from server"
-;SUBJECT = Diagnostic message from server
-;; Mail server
-;HOST =
-;; Mailer user name and password
-;USER =
-;; Use PASSWD = `your password` for quoting if you use special characters in the password.
-;PASSWD =
-;; Receivers, can be one or more, e.g. 1@example.com,2@example.com
-;RECEIVERS =
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;