aboutsummaryrefslogtreecommitdiffstats
path: root/cmd
Commit message (Collapse)AuthorAgeFilesLines
* clarify aspects of the dump command (#26887)JonRB2023-09-031-1/+1
| | | | | | | | | clarify aspects of the dump command Possibly closes #26862 --------- Co-authored-by: delvh <dev.lh@web.de>
* Expanded minimum RSA Keylength to 3072 (#26604)mainboarder2023-08-281-1/+1
| | | | | | | | | | | | | | | German Federal Office for Information Security requests in its technical guideline BSI TR-02102-1 RSA Keylength not shorter than 3000bits starting 2024, in the year 2023 3000bits as a recommendation. Gitea should request longer RSA Keys by default in favor of security and drop old clients which do not support longer keys. https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR02102/BSI-TR-02102.pdf?__blob=publicationFile&v=9 - Page 19, Table 1.2 --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Handle base64 decoding correctly to avoid panic (#26483)wxiaoguang2023-08-141-2/+2
| | | Fix the panic if the "base64 secret" is too long.
* Fix stderr usages (#26477)wxiaoguang2023-08-131-2/+1
|
* Refactor tests (#26464)wxiaoguang2023-08-121-2/+2
| | | | 1. Give the global variable clear names 2. Use generic parameter for `onGiteaRun`
* Call git.InitSimple for runRepoSyncReleases (#26396)wxiaoguang2023-08-101-0/+4
| | | | | | Fix #26394 Otherwise, the git module is not initialized and it doesn't respect the "timeout" config in app.ini
* Fix incorrect CLI exit code and duplicate error message (#26346)wxiaoguang2023-08-052-16/+85
| | | Follow the CLI refactoring, and add tests.
* Improve CLI and messages (#26341)wxiaoguang2023-08-052-52/+17
| | | | | | | Follow the CLI refactoring 1. Remove the "checkCommandFlags" helper 2. Unify the web startup message, make them have consistent names as `./gitea help` 3. Fine tune some other messages (see the diff)
* Fix CLI allowing creation of access tokens with existing name (#26071)Yarden Shoham2023-07-251-5/+16
| | | | | | | | | | | | | We are now: - Making sure there is no existing access token with the same name - Making sure the given scopes are valid (we already did this before but now we have a message) The logic is mostly taken from https://github.com/go-gitea/gitea/blob/a12a5f3652c339b17b187ff424a480631a3c1e1e/routers/api/v1/user/app.go#L101-L123 Closes #26044 Signed-off-by: Yarden Shoham <git@yardenshoham.com>
* Improve "gitea doctor" sub-command and fix "help" commands (#26072)wxiaoguang2023-07-253-27/+36
| | | | | | | | | | | | | | | | | | | | | | | Replace #21790 And close #25965 by the way (it needs a separate fix for 1.20) Major changes: 1. Move "gitea convert" to "gitea doctor conver". The old "gitea doctor" still works as a hidden sub-command (to avoid breaking) 2. Do not write "doctor.log" by default, it's not useful in most cases and causes bugs like 25965 3. Improve documents 4. Fix the "help" commands. Before, the "./gitea doctor" can't show the sub-command help correctly (regression of the last cli/v2 refactoring) After this PR: ``` ./gitea help # show all sub-commands for the app ./gitea doctor # show the sub-commands for the "doctor" ./gitea doctor help # show the sub-commands for the "doctor", as above ```
* Add missing default value for some Bool cli flags (#26082)wxiaoguang2023-07-242-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In #25959 I forgot to add default values to some Bool flags (which were BoolT in cli/v1, BoolT means default to be true) This PR adds the default "Value" for them. ``` ./cmd/manager_logging.go: }, cli.BoolTFlag{ ./cmd/manager_logging.go- Name: "rotate, r", ./cmd/manager_logging.go- Usage: "Rotate logs", -- ./cmd/manager_logging.go: }, cli.BoolTFlag{ ./cmd/manager_logging.go- Name: "daily, d", ./cmd/manager_logging.go- Usage: "Rotate logs daily", -- ./cmd/manager_logging.go: }, cli.BoolTFlag{ ./cmd/manager_logging.go- Name: "compress, z", ./cmd/manager_logging.go- Usage: "Compress rotated logs", -- ./cmd/admin.go: cli.BoolTFlag{ ./cmd/admin.go- Name: "force-smtps", ./cmd/admin.go- Usage: "SMTPS is always used on port 465. Set this to force SMTPS on other ports.", -- ./cmd/admin.go: cli.BoolTFlag{ ./cmd/admin.go- Name: "skip-verify", ./cmd/admin.go- Usage: "Skip TLS verify.", -- ./cmd/admin.go: cli.BoolTFlag{ ./cmd/admin.go- Name: "disable-helo", ./cmd/admin.go- Usage: "Disable SMTP helo.", -- ./cmd/admin.go: cli.BoolTFlag{ ./cmd/admin.go- Name: "skip-local-2fa", ./cmd/admin.go- Usage: "Skip 2FA to log on.", -- ./cmd/admin.go: cli.BoolTFlag{ ./cmd/admin.go- Name: "active", ./cmd/admin.go- Usage: "This Authentication Source is Activated.", ```
* Remove `db.DefaultContext` in `routers/` and `cmd/` (#26076)delvh2023-07-231-4/+5
| | | | | | Now, the only remaining usages of `models.db.DefaultContext` are in - `modules` - `models` - `services`
* Serve pre-defined files in "public", add "security.txt", add CORS header for ↵wxiaoguang2023-07-211-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ".well-known" (#25974) Replace #25892 Close #21942 Close #25464 Major changes: 1. Serve "robots.txt" and ".well-known/security.txt" in the "public" custom path * All files in "public/.well-known" can be served, just like "public/assets" 3. Add a test for ".well-known/security.txt" 4. Simplify the "FileHandlerFunc" logic, now the paths are consistent so the code can be simpler 5. Add CORS header for ".well-known" endpoints 6. Add logs to tell users they should move some of their legacy custom public files ``` 2023/07/19 13:00:37 cmd/web.go:178:serveInstalled() [E] Found legacy public asset "img" in CustomPath. Please move it to /work/gitea/custom/public/assets/img 2023/07/19 13:00:37 cmd/web.go:182:serveInstalled() [E] Found legacy public asset "robots.txt" in CustomPath. Please move it to /work/gitea/custom/public/robots.txt ``` This PR is not breaking. --------- Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Giteabot <teabot@gitea.io>
* Refactor to use urfave/cli/v2 (#25959)wxiaoguang2023-07-2132-444/+832
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace #10912 And there are many new tests to cover the CLI behavior There were some concerns about the "option order in hook scripts" (https://github.com/go-gitea/gitea/pull/10912#issuecomment-1137543314), it's not a problem now. Because the hook script uses `/gitea hook --config=/app.ini pre-receive` format. The "config" is a global option, it can appear anywhere. ---- ## ⚠️ BREAKING ⚠️ This PR does it best to avoid breaking anything. The major changes are: * `gitea` itself won't accept web's options: `--install-port` / `--pid` / `--port` / `--quiet` / `--verbose` .... They are `web` sub-command's options. * Use `./gitea web --pid ....` instead * `./gitea` can still run the `web` sub-command as shorthand, with default options * The sub-command's options must follow the sub-command * Before: `./gitea --sub-opt subcmd` might equal to `./gitea subcmd --sub-opt` (well, might not ...) * After: only `./gitea subcmd --sub-opt` could be used * The global options like `--config` are not affected
* Bump github.com/golang-jwt/jwt to v5 (#25975)harryzcy2023-07-191-1/+1
| | | | | | | | | | | | | | | | Bumping `github.com/golang-jwt/jwt` from v4 to v5. `github.com/golang-jwt/jwt` v5 is bringing some breaking changes: - standard `Valid()` method on claims is removed. It's replaced by `ClaimsValidator` interface implementing `Validator()` method instead, which is called after standard validation. Gitea doesn't seem to be using this logic. - `jwt.Token` has a field `Valid`, so it's checked in `ParseToken` function in `services/auth/source/oauth2/token.go` --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Replace `interface{}` with `any` (#25686)silverwind2023-07-044-9/+9
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Do not prepare oauth2 config if it is not enabled, do not write config in ↵wxiaoguang2023-06-281-2/+2
| | | | | | | | | | some sub-commands (#25567) Ref: * https://github.com/go-gitea/gitea/issues/25377#issuecomment-1609757289 And some sub-commands like "generate" / "docs", they do not need to use the ini config
* Fix sub-command log level (#25537)wxiaoguang2023-06-286-7/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | More fix for #24981 * #24981 Close #22361 * #22361 There were many patches for Gitea's sub-commands to satisfy the facts: * Some sub-commands shouldn't output any log, otherwise the git protocol would be broken * Sometimes the users want to see "verbose" or "quiet" outputs That's a longstanding problem, and very fragile. This PR is only a quick patch for the problem. In the future, the sub-command system should be refactored to a clear solution. ---- Other changes: * Use `ReplaceAllWriters` to replace `RemoveAllWriters().AddWriters(writer)`, then it's an atomic operation. * Remove unnecessary `syncLevelInternal` calls, because `AddWriters/addWritersInternal` already calls it. Co-authored-by: Giteabot <teabot@gitea.io>
* Refactor path & config system (#25330)wxiaoguang2023-06-219-72/+115
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # The problem There were many "path tricks": * By default, Gitea uses its program directory as its work path * Gitea tries to use the "work path" to guess its "custom path" and "custom conf (app.ini)" * Users might want to use other directories as work path * The non-default work path should be passed to Gitea by GITEA_WORK_DIR or "--work-path" * But some Gitea processes are started without these values * The "serv" process started by OpenSSH server * The CLI sub-commands started by site admin * The paths are guessed by SetCustomPathAndConf again and again * The default values of "work path / custom path / custom conf" can be changed when compiling # The solution * Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use test code to cover its behaviors. * When Gitea's web server runs, write the WORK_PATH to "app.ini", this value must be the most correct one, because if this value is not right, users would find that the web UI doesn't work and then they should be able to fix it. * Then all other sub-commands can use the WORK_PATH in app.ini to initialize their paths. * By the way, when Gitea starts for git protocol, it shouldn't output any log, otherwise the git protocol gets broken and client blocks forever. The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path > env var GITEA_WORK_DIR > builtin default The "app.ini" searching order is: cmd arg --config > cmd arg "work path / custom path" > env var "work path / custom path" > builtin default ## ⚠️ BREAKING If your instance's "work path / custom path / custom conf" doesn't meet the requirements (eg: work path must be absolute), Gitea will report a fatal error and exit. You need to set these values according to the error log. ---- Close #24818 Close #24222 Close #21606 Close #21498 Close #25107 Close #24981 Maybe close #24503 Replace #23301 Replace #22754 And maybe more
* Avoid polluting config file when "save" (#25395)wxiaoguang2023-06-211-3/+9
| | | | | | | | | | | | | That's a longstanding INI package problem: the "MustXxx" calls change the option values, and the following "Save" will save a lot of garbage options into the user's config file. Ideally we should refactor the INI package to a clear solution, but it's a huge work. A clear workaround is what this PR does: when "Save", load a clear INI instance and save it. Partially fix #25377, the "install" page needs more fine tunes.
* Refactor web package and context package (#25298)wxiaoguang2023-06-181-1/+1
| | | | | | | | | | | | | 1. The "web" package shouldn't depends on "modules/context" package, instead, let each "web context" register themselves to the "web" package. 2. The old Init/Free doesn't make sense, so simplify it * The ctx in "Init(ctx)" is never used, and shouldn't be used that way * The "Free" is never called and shouldn't be called because the SSPI instance is shared --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Fix all possible setting error related storages and added some tests (#23911)Lunny Xiao2023-06-143-17/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | Follow up #22405 Fix #20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Use the type RefName for all the needed places and fix pull mirror sync bugs ↵Lunny Xiao2023-05-261-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#24634) This PR replaces all string refName as a type `git.RefName` to make the code more maintainable. Fix #15367 Replaces #23070 It also fixed a bug that tags are not sync because `git remote --prune origin` will not remove local tags if remote removed. We in fact should use `git fetch --prune --tags origin` but not `git remote update origin` to do the sync. Some answer from ChatGPT as ref. > If the git fetch --prune --tags command is not working as expected, there could be a few reasons why. Here are a few things to check: > >Make sure that you have the latest version of Git installed on your system. You can check the version by running git --version in your terminal. If you have an outdated version, try updating Git and see if that resolves the issue. > >Check that your Git repository is properly configured to track the remote repository's tags. You can check this by running git config --get-all remote.origin.fetch and verifying that it includes +refs/tags/*:refs/tags/*. If it does not, you can add it by running git config --add remote.origin.fetch "+refs/tags/*:refs/tags/*". > >Verify that the tags you are trying to prune actually exist on the remote repository. You can do this by running git ls-remote --tags origin to list all the tags on the remote repository. > >Check if any local tags have been created that match the names of tags on the remote repository. If so, these local tags may be preventing the git fetch --prune --tags command from working properly. You can delete local tags using the git tag -d command. --------- Co-authored-by: delvh <dev.lh@web.de>
* Fix install page context, make the install page tests really test (#24858)wxiaoguang2023-05-231-3/+1
| | | | | | | | | | | | Fix #24856 Rename "context.contextKey" to "context.WebContextKey", this context is for web context only. But the Context itself is not renamed, otherwise it would cause a lot of changes (if we really want to rename it, there could be a separate PR). The old test code doesn't really test, the "install page" gets broken not only one time, so use new test code to make sure the "install page" could work.
* Add gitea manager reload-templates command (#24843)Brecht Van Lommel2023-05-221-0/+20
| | | | | This can be useful to update custom templates in production mode, when they are updated frequently and a full Gitea restart each time is disruptive.
* Rewrite logger system (#24726)wxiaoguang2023-05-217-179/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>
* Support migrating storage for actions log via command line (#24679)Jason Song2023-05-121-1/+19
| | | | | | | Close #24677 --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Rewrite queue (#24505)wxiaoguang2023-05-081-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # ⚠️ Breaking Many deprecated queue config options are removed (actually, they should have been removed in 1.18/1.19). If you see the fatal message when starting Gitea: "Please update your app.ini to remove deprecated config options", please follow the error messages to remove these options from your app.ini. Example: ``` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options ``` Many options in `[queue]` are are dropped, including: `WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`, `BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed from app.ini. # The problem The old queue package has some legacy problems: * complexity: I doubt few people could tell how it works. * maintainability: Too many channels and mutex/cond are mixed together, too many different structs/interfaces depends each other. * stability: due to the complexity & maintainability, sometimes there are strange bugs and difficult to debug, and some code doesn't have test (indeed some code is difficult to test because a lot of things are mixed together). * general applicability: although it is called "queue", its behavior is not a well-known queue. * scalability: it doesn't seem easy to make it work with a cluster without breaking its behaviors. It came from some very old code to "avoid breaking", however, its technical debt is too heavy now. It's a good time to introduce a better "queue" package. # The new queue package It keeps using old config and concept as much as possible. * It only contains two major kinds of concepts: * The "base queue": channel, levelqueue, redis * They have the same abstraction, the same interface, and they are tested by the same testing code. * The "WokerPoolQueue", it uses the "base queue" to provide "worker pool" function, calls the "handler" to process the data in the base queue. * The new code doesn't do "PushBack" * Think about a queue with many workers, the "PushBack" can't guarantee the order for re-queued unhandled items, so in new code it just does "normal push" * The new code doesn't do "pause/resume" * The "pause/resume" was designed to handle some handler's failure: eg: document indexer (elasticsearch) is down * If a queue is paused for long time, either the producers blocks or the new items are dropped. * The new code doesn't do such "pause/resume" trick, it's not a common queue's behavior and it doesn't help much. * If there are unhandled items, the "push" function just blocks for a few seconds and then re-queue them and retry. * The new code doesn't do "worker booster" * Gitea's queue's handlers are light functions, the cost is only the go-routine, so it doesn't make sense to "boost" them. * The new code only use "max worker number" to limit the concurrent workers. * The new "Push" never blocks forever * Instead of creating more and more blocking goroutines, return an error is more friendly to the server and to the end user. There are more details in code comments: eg: the "Flush" problem, the strange "code.index" hanging problem, the "immediate" queue problem. Almost ready for review. TODO: * [x] add some necessary comments during review * [x] add some more tests if necessary * [x] update documents and config options * [x] test max worker / active worker * [x] re-run the CI tasks to see whether any test is flaky * [x] improve the `handleOldLengthConfiguration` to provide more friendly messages * [x] fine tune default config values (eg: length?) ## Code coverage: ![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
* Merge setting.InitXXX into one function with options (#24389)Lunny Xiao2023-05-0410-24/+11
| | | | This PR will merge 3 Init functions on setting packages as 1 and introduce an options struct.
* Add Debian package registry (#24426)KN4CK3R2023-05-021-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: @awkwardbunny This PR adds a Debian package registry. You can follow [this tutorial](https://www.baeldung.com/linux/create-debian-package) to build a *.deb package for testing. Source packages are not supported at the moment and I did not find documentation of the architecture "all" and how these packages should be treated. ![grafik](https://user-images.githubusercontent.com/1666336/218126879-eb80a866-775c-4c8e-8529-5797203a64e6.png) Part of #20751. Revised copy of #22854. --------- Co-authored-by: Brian Hong <brian@hongs.me> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Giteabot <teabot@gitea.io>
* Revert "Add Debian package registry" (#24412)Yarden Shoham2023-04-281-1/+1
| | | Reverts go-gitea/gitea#22854
* Add Debian package registry (#22854)KN4CK3R2023-04-281-1/+1
| | | | | | | | | | | | | | Co-authored-by: @awkwardbunny This PR adds a Debian package registry. You can follow [this tutorial](https://www.baeldung.com/linux/create-debian-package) to build a *.deb package for testing. Source packages are not supported at the moment and I did not find documentation of the architecture "all" and how these packages should be treated. --------- Co-authored-by: Brian Hong <brian@hongs.me> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactor config provider (#24245)Lunny Xiao2023-04-251-4/+4
| | | | | | | This PR introduces more abstract about `ConfigProvider` and hides more `ini` references. --------- Co-authored-by: delvh <dev.lh@web.de>
* Use more specific test methods (#24265)KN4CK3R2023-04-221-1/+1
| | | | Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Giteabot <teabot@gitea.io>
* add CLI command to register runner tokens (#23762)techknowlogick2023-04-171-0/+56
| | | | | | | | | | This is a CLI command to generate new tokens for the runners to register with Fix https://github.com/go-gitea/gitea/issues/23643 --------- Co-authored-by: delvh <dev.lh@web.de>
* Support converting varchar to nvarchar for mssql database (#24105)Lunny Xiao2023-04-171-11/+16
|
* Use a general approach to access custom/static/builtin assets (#24022)wxiaoguang2023-04-122-99/+52
| | | | | | | | | | | | | | | | | | | The idea is to use a Layered Asset File-system (modules/assetfs/layered.go) For example: when there are 2 layers: "custom", "builtin", when access to asset "my/page.tmpl", the Layered Asset File-system will first try to use "custom" assets, if not found, then use "builtin" assets. This approach will hugely simplify a lot of code, make them testable. Other changes: * Simplify the AssetsHandlerFunc code * Simplify the `gitea embedded` sub-command code --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add --quiet option to gitea dump (#22969)Fredrik Eriksson2023-04-101-1/+17
| | | | | | | | | | | | Fixes: #19687 The --quiet options to gitea dump silences informational and less important messages, but will still log warnings and errors to console. Very useful in combination with cron backups and '-f -'. Since --verbose and --quiet are incompatible with each other I made it an error to specify both. To get the error message to be printed to stderr I had to make this test after the NewServices()-call, which is why there are three new blocks of code instead of two.
* Refactor internal API for git commands, use meaningful messages instead of ↵wxiaoguang2023-03-298-242/+184
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "Internal Server Error" (#23687) # Why this PR comes At first, I'd like to help users like #23636 (there are a lot) The unclear "Internal Server Error" is quite anonying, scare users, frustrate contributors, nobody knows what happens. So, it's always good to provide meaningful messages to end users (of course, do not leak sensitive information). When I started working on the "response message to end users", I found that the related code has a lot of technical debt. A lot of copy&paste code, unclear fields and usages. So I think it's good to make everything clear. # Tech Backgrounds Gitea has many sub-commands, some are used by admins, some are used by SSH servers or Git Hooks. Many sub-commands use "internal API" to communicate with Gitea web server. Before, Gitea server always use `StatusCode + Json "err" field` to return messages. * The CLI sub-commands: they expect to show all error related messages to site admin * The Serv/Hook sub-commands (for git clients): they could only show safe messages to end users, the error log could only be recorded by "SSHLog" to Gitea web server. In the old design, it assumes that: * If the StatusCode is 500 (in some functions), then the "err" field is error log, shouldn't be exposed to git client. * If the StatusCode is 40x, then the "err" field could be exposed. And some functions always read the "err" no matter what the StatusCode is. The old code is not strict, and it's difficult to distinguish the messages clearly and then output them correctly. # This PR To help to remove duplicate code and make everything clear, this PR introduces `ResponseExtra` and `requestJSONResp`. * `ResponseExtra` is a struct which contains "extra" information of a internal API response, including StatusCode, UserMsg, Error * `requestJSONResp` is a generic function which can be used for all cases to help to simplify the calls. * Remove all `map["err"]`, always use `private.Response{Err}` to construct error messages. * User messages and error messages are separated clearly, the `fail` and `handleCliResponseExtra` will output correct messages. * Replace all `Internal Server Error` messages with meaningful (still safe) messages. This PR saves more than 300 lines, while makes the git client messages more clear. Many gitea-serv/git-hook related essential functions are covered by tests. --------- Co-authored-by: delvh <dev.lh@web.de>
* Make minio package support legacy MD5 checksum (#23768)wxiaoguang2023-03-281-8/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | A feedback from discord: https://discord.com/channels/322538954119184384/561007778139734027/1090185427115319386 Some storages like: * https://developers.cloudflare.com/r2/api/s3/api/ * https://www.backblaze.com/b2/docs/s3_compatible_api.html They do not support "x-amz-checksum-algorithm" header But minio recently uses that header with CRC32C by default. So we have to tell minio to use legacy MD5 checksum. I guess this needs to be backported because IIRC we 1.19 and 1.20 are using similar minio package. The minio package code for SendContentMD5 looks like this: <details> <img width="755" alt="image" src="https://user-images.githubusercontent.com/2114189/228186768-4f2f6f67-62b9-4aee-9251-5af714ad9674.png"> </details>
* Check LFS/Packages settings in dump and doctor command (#23631)Zettat1232023-03-231-0/+4
| | | | | | | | | | Close #23622 As described in the issue, disabling the LFS/Package settings will cause errors when running `gitea dump` or `gitea doctor`. We need to check the settings and the related operations should be skipped if the settings are disabled. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Move pidfile creation from setting to web cmd package (#23285)Lunny Xiao2023-03-161-3/+23
| | | | | | | | | | | | Creating pid file should not belong to setting package and only web command needs that. So this PR moves pidfile creation from setting package to web command package to keep setting package more readable. I marked this as `break` because the PIDFile path moved. For those who have used the pid build argument, it has to be changed. --------- Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* add path prefix to ObjectStorage.Iterator (#23332)FuXiaoHei2023-03-131-3/+3
| | | | | | | | | | | | Support to iterator subdirectory in ObjectStorage for ObjectStorage.Iterator method. It's required for https://github.com/go-gitea/gitea/pull/22738 to make artifact files cleanable. --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Handle OpenID discovery URL errors a little nicer when creating/editing ↵zeripath2023-03-101-1/+10
| | | | | | | | | | | sources (#23397) When there is an error creating a new openIDConnect authentication source try to handle the error a little better. Close #23283 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactor `setting.Database.UseXXX` to methods (#23354)Jason Song2023-03-072-2/+2
| | | | | | | | | | | Replace #23350. Refactor `setting.Database.UseMySQL` to `setting.Database.Type.IsMySQL()`. To avoid mismatching between `Type` and `UseXXX`. This refactor can fix the bug mentioned in #23350, so it should be backported.
* Close the temp file when dumping database to make the temp file can be ↵wxiaoguang2023-03-021-0/+1
| | | | | | deleted on Windows (#23249) There was no `dbDump.Close()` before, Windows doesn't like to delete opened files.
* Make `gitea serv` respect git binary home (#23138)wxiaoguang2023-02-281-10/+15
| | | | | | | | | | | | | Close #23137 The old code is too old (8-9 years ago) Let's try to execute the git commands from git bin home directly. The verb has been checked above, it could only be: * git-upload-pack * git-upload-archive * git-receive-pack * git-lfs-authenticate
* Refactor the setting to make unit test easier (#22405)Lunny Xiao2023-02-2013-27/+37
| | | | | | | | | | | | | | | | | | | | | | Some bugs caused by less unit tests in fundamental packages. This PR refactor `setting` package so that create a unit test will be easier than before. - All `LoadFromXXX` files has been splited as two functions, one is `InitProviderFromXXX` and `LoadCommonSettings`. The first functions will only include the code to create or new a ini file. The second function will load common settings. - It also renames all functions in setting from `newXXXService` to `loadXXXSetting` or `loadXXXFrom` to make the function name less confusing. - Move `XORMLog` to `SQLLog` because it's a better name for that. Maybe we should finally move these `loadXXXSetting` into the `XXXInit` function? Any idea? --------- Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: delvh <dev.lh@web.de>
* Provide the ability to set password hash algorithm parameters (#22942)zeripath2023-02-192-2/+2
| | | | | | | | | | | | | | | | | | | | | | | This PR refactors and improves the password hashing code within gitea and makes it possible for server administrators to set the password hashing parameters In addition it takes the opportunity to adjust the settings for `pbkdf2` in order to make the hashing a little stronger. The majority of this work was inspired by PR #14751 and I would like to thank @boppy for their work on this. Thanks to @gusted for the suggestion to adjust the `pbkdf2` hashing parameters. Close #14751 --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: John Olheiser <john.olheiser@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add context cache as a request level cache (#22294)Lunny Xiao2023-02-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To avoid duplicated load of the same data in an HTTP request, we can set a context cache to do that. i.e. Some pages may load a user from a database with the same id in different areas on the same page. But the code is hidden in two different deep logic. How should we share the user? As a result of this PR, now if both entry functions accept `context.Context` as the first parameter and we just need to refactor `GetUserByID` to reuse the user from the context cache. Then it will not be loaded twice on an HTTP request. But of course, sometimes we would like to reload an object from the database, that's why `RemoveContextData` is also exposed. The core context cache is here. It defines a new context ```go type cacheContext struct { ctx context.Context data map[any]map[any]any lock sync.RWMutex } var cacheContextKey = struct{}{} func WithCacheContext(ctx context.Context) context.Context { return context.WithValue(ctx, cacheContextKey, &cacheContext{ ctx: ctx, data: make(map[any]map[any]any), }) } ``` Then you can use the below 4 methods to read/write/del the data within the same context. ```go func GetContextData(ctx context.Context, tp, key any) any func SetContextData(ctx context.Context, tp, key, value any) func RemoveContextData(ctx context.Context, tp, key any) func GetWithContextCache[T any](ctx context.Context, cacheGroupKey string, cacheTargetID any, f func() (T, error)) (T, error) ``` Then let's take a look at how `system.GetString` implement it. ```go func GetSetting(ctx context.Context, key string) (string, error) { return cache.GetWithContextCache(ctx, contextCacheKey, key, func() (string, error) { return cache.GetString(genSettingCacheKey(key), func() (string, error) { res, err := GetSettingNoCache(ctx, key) if err != nil { return "", err } return res.SettingValue, nil }) }) } ``` First, it will check if context data include the setting object with the key. If not, it will query from the global cache which may be memory or a Redis cache. If not, it will get the object from the database. In the end, if the object gets from the global cache or database, it will be set into the context cache. An object stored in the context cache will only be destroyed after the context disappeared.
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-29 12:06:13 +0000