aboutsummaryrefslogtreecommitdiffstats
path: root/cmd/dump.go
Commit message (Collapse)AuthorAgeFilesLines
* Refactor path & config system (#25330)wxiaoguang2023-06-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # The problem There were many "path tricks": * By default, Gitea uses its program directory as its work path * Gitea tries to use the "work path" to guess its "custom path" and "custom conf (app.ini)" * Users might want to use other directories as work path * The non-default work path should be passed to Gitea by GITEA_WORK_DIR or "--work-path" * But some Gitea processes are started without these values * The "serv" process started by OpenSSH server * The CLI sub-commands started by site admin * The paths are guessed by SetCustomPathAndConf again and again * The default values of "work path / custom path / custom conf" can be changed when compiling # The solution * Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use test code to cover its behaviors. * When Gitea's web server runs, write the WORK_PATH to "app.ini", this value must be the most correct one, because if this value is not right, users would find that the web UI doesn't work and then they should be able to fix it. * Then all other sub-commands can use the WORK_PATH in app.ini to initialize their paths. * By the way, when Gitea starts for git protocol, it shouldn't output any log, otherwise the git protocol gets broken and client blocks forever. The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path > env var GITEA_WORK_DIR > builtin default The "app.ini" searching order is: cmd arg --config > cmd arg "work path / custom path" > env var "work path / custom path" > builtin default ## ⚠️ BREAKING If your instance's "work path / custom path / custom conf" doesn't meet the requirements (eg: work path must be absolute), Gitea will report a fatal error and exit. You need to set these values according to the error log. ---- Close #24818 Close #24222 Close #21606 Close #21498 Close #25107 Close #24981 Maybe close #24503 Replace #23301 Replace #22754 And maybe more
* Fix all possible setting error related storages and added some tests (#23911)Lunny Xiao2023-06-141-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | Follow up #22405 Fix #20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Rewrite logger system (#24726)wxiaoguang2023-05-211-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>
* Merge setting.InitXXX into one function with options (#24389)Lunny Xiao2023-05-041-2/+1
| | | | This PR will merge 3 Init functions on setting packages as 1 and introduce an options struct.
* Add --quiet option to gitea dump (#22969)Fredrik Eriksson2023-04-101-1/+17
| | | | | | | | | | | | Fixes: #19687 The --quiet options to gitea dump silences informational and less important messages, but will still log warnings and errors to console. Very useful in combination with cron backups and '-f -'. Since --verbose and --quiet are incompatible with each other I made it an error to specify both. To get the error message to be printed to stderr I had to make this test after the NewServices()-call, which is why there are three new blocks of code instead of two.
* Check LFS/Packages settings in dump and doctor command (#23631)Zettat1232023-03-231-0/+4
| | | | | | | | | | Close #23622 As described in the issue, disabling the LFS/Package settings will cause errors when running `gitea dump` or `gitea doctor`. We need to check the settings and the related operations should be skipped if the settings are disabled. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* add path prefix to ObjectStorage.Iterator (#23332)FuXiaoHei2023-03-131-3/+3
| | | | | | | | | | | | Support to iterator subdirectory in ObjectStorage for ObjectStorage.Iterator method. It's required for https://github.com/go-gitea/gitea/pull/22738 to make artifact files cleanable. --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Refactor `setting.Database.UseXXX` to methods (#23354)Jason Song2023-03-071-1/+1
| | | | | | | | | | | Replace #23350. Refactor `setting.Database.UseMySQL` to `setting.Database.Type.IsMySQL()`. To avoid mismatching between `Type` and `UseXXX`. This refactor can fix the bug mentioned in #23350, so it should be backported.
* Close the temp file when dumping database to make the temp file can be ↵wxiaoguang2023-03-021-0/+1
| | | | | | deleted on Windows (#23249) There was no `dbDump.Close()` before, Windows doesn't like to delete opened files.
* Refactor the setting to make unit test easier (#22405)Lunny Xiao2023-02-201-9/+11
| | | | | | | | | | | | | | | | | | | | | | Some bugs caused by less unit tests in fundamental packages. This PR refactor `setting` package so that create a unit test will be easier than before. - All `LoadFromXXX` files has been splited as two functions, one is `InitProviderFromXXX` and `LoadCommonSettings`. The first functions will only include the code to create or new a ini file. The second function will load common settings. - It also renames all functions in setting from `newXXXService` to `loadXXXSetting` or `loadXXXFrom` to make the function name less confusing. - Move `XORMLog` to `SQLLog` because it's a better name for that. Maybe we should finally move these `loadXXXSetting` into the `XXXInit` function? Any idea? --------- Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: delvh <dev.lh@web.de>
* Improve utils of slices (#22379)Jason Song2023-01-111-10/+1
| | | | | | | | | | | | | | | | | | | - Move the file `compare.go` and `slice.go` to `slice.go`. - Fix `ExistsInSlice`, it's buggy - It uses `sort.Search`, so it assumes that the input slice is sorted. - It passes `func(i int) bool { return slice[i] == target })` to `sort.Search`, that's incorrect, check the doc of `sort.Search`. - Conbine `IsInt64InSlice(int64, []int64)` and `ExistsInSlice(string, []string)` to `SliceContains[T]([]T, T)`. - Conbine `IsSliceInt64Eq([]int64, []int64)` and `IsEqualSlice([]string, []string)` to `SliceSortedEqual[T]([]T, T)`. - Add `SliceEqual[T]([]T, T)` as a distinction from `SliceSortedEqual[T]([]T, T)`. - Redesign `RemoveIDFromList([]int64, int64) ([]int64, bool)` to `SliceRemoveAll[T]([]T, T) []T`. - Add `SliceContainsFunc[T]([]T, func(T) bool)` and `SliceRemoveAllFunc[T]([]T, func(T) bool)` for general use. - Add comments to explain why not `golang.org/x/exp/slices`. - Add unit tests.
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-271-2/+1
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* dump: Add option to skip index dirs (#21501)Johan Van de Wauw2022-10-241-0/+9
| | | | | | | | | | closes #20683 Add an option to gitea dump to skip the bleve indexes, which can become quite large (in my case the same size as the repo's) and can be regenerated after restore. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: zeripath <art27@cantab.net>
* Add Tar ZSTD support (#20493)Gusted2022-07-271-1/+1
| | | | | - Add `.tar.zst` as supported output type. - Resolves #14290
* Dump should only copy regular files and symlink regular files (#20015)v1.18.0-devwxiaoguang2022-06-181-3/+18
|
* Allow package dump skipping (#19506)KN4CK3R2022-04-261-23/+35
| | | | | | | | | | | | | | * Added addReader to support verbose. * Allow skipping packages. * Updated docs. * Update cmd/dump.go Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* When dumping trim the standard suffices instead of a random suffix (#19440)zeripath2022-04-201-2/+7
| | | | | | | | | | | | | | * When dumping trim the standard suffices instead of a random suffix Instead of using the `path.Ext()` to trim the last "extension" suffix, just iterate through the supported suffices and trim those. Fix #19424 Signed-off-by: Andrew Thornton <art27@cantab.net> * fix enum with to have correct supported types only Co-authored-by: 6543 <6543@obermui.de>
* format with gofumpt (#18184)65432022-01-201-1/+1
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Simplify parameter types (#18006)Gusted2021-12-201-2/+2
| | | Remove repeated type declarations in function definitions.
* Fix outType on gitea dump (#18000)Gusted2021-12-171-2/+5
| | | | | | - Force to output the dump file to use the given `--type`. - Resolves #17959
* Improve install code to avoid low-level mistakes. (#17779)wxiaoguang2021-12-011-1/+2
| | | | | | | | | | | | | | * Improve install code to avoid low-level mistakes. If a user tries to do a re-install in a Gitea database, they gets a warning and double check. When Gitea runs, it never create empty app.ini automatically. Also some small (related) refactoring: * Refactor db.InitEngine related logic make it more clean (especially for the install code) * Move some i18n strings out from setting.go to make the setting.go can be easily maintained. * Show errors in CLI code if an incorrect app.ini is used. * APP_DATA_PATH is created when installing, and checked when starting (no empty directory is created any more).
* Fix bug on admin subcommand (#17533)Lunny Xiao2021-11-071-1/+4
| | | | | | | * Fix bug on admin subcommand * Add signals for all initDB Co-authored-by: Lauris BH <lauris@nix.lv>
* Rename db Engines related functions (#17481)Lunny Xiao2021-10-301-1/+1
| | | | | * Rename db Engines related functions Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-221-2/+1
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Move db related basic functions to models/db (#17075)Lunny Xiao2021-09-191-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | * Move db related basic functions to models/db * Fix lint * Fix lint * Fix test * Fix lint * Fix lint * revert unnecessary change * Fix test * Fix wrong replace string * Use *Context * Correct committer spelling and fix wrong replaced words Co-authored-by: zeripath <art27@cantab.net>
* Add an abstract json layout to make it's easier to change json library (#16528)Lunny Xiao2021-07-241-2/+1
| | | | | | | | | | | * Add an abstract json layout to make it's easier to change json library * Fix import * Fix import sequence * Fix blank lines * Fix blank lines
* Fix various documentation, user-facing, and source comment typos (#16367)luzpaz2021-07-081-1/+1
| | | | | * Fix various doc, user-facing, and source comment typos Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
* dump: Add option to skip LFS/attachment files (#15407)Johan Van de Wauw2021-04-121-2/+14
| | | | | | | * Add option to skip dumping LFS/attachment files * Fix fmt issues Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Migrate to use jsoniter instead of encoding/json (#14841)zeripath2021-03-011-1/+2
| | | | | | | | | | * Migrate to use jsoniter * fix tests * update gitea.com/go-chi/binding Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Exclude the current dump file from the dump (#14606)zeripath2021-02-081-42/+24
| | | | | | | | | | | | | | | | | | | * Exclude the current dump file from the dump Always prevent the current file from being added to the dump. Fix #13618 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add skip custom directory option Signed-off-by: Andrew Thornton <art27@cantab.net> * placate lint Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Move macaron to chi (#14293)Lunny Xiao2021-01-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR. - [x] Define `context.ResponseWriter` interface with an implementation `context.Response`. - [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before. - [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic . - [x] Use https://github.com/unrolled/render instead of macaron's internal render - [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip - [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK** - [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha - [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache - [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding - [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors - [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation` - [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle. - [x] Removed macaron log service because it's not need any more. **BREAK** - [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition. - [x] Move Git HTTP protocol implementation to use routers directly. - [x] Fix the problem that chi routes don't support trailing slash but macaron did. - [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render. Notices: - Chi router don't support request with trailing slash - Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI. Co-authored-by: 6543 <6543@obermui.de>
* Handle and propagate errors when checking if paths are Dirs, Files or Exist ↵zeripath2020-11-271-5/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | (#13186) * Ensure errors from IsDir propagate * Handle errors when checking IsFile * Handle and propagate errors from IsExist * Update modules/templates/static.go * Update modules/templates/static.go * Return after ctx.ServerError * Apply suggestions from code review * Fix tests The previous merge managed to break repo_form.go Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Lauris BH <lauris@nix.lv>
* Add default storage configurations (#12813)Lunny Xiao2020-09-291-5/+40
| | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* gitea dump: include version & Check InstallLock (#12760)65432020-09-071-0/+4
| | | | | | | | | * gitea dump: include version * Check InstallLock (close #12759) * fix test * fix lint
* Re-attempt to delete temporary upload if the file is locked by another ↵zeripath2020-08-111-2/+7
| | | | | | | | | | | process (#12447) Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util. Fix #12339 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: silverwind <me@silverwind.io>
* Dump: add output format tar and output to stdout (#10376)PhilippHomann2020-06-051-57/+212
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Dump: Use mholt/archive/v3 to support tar including many compressions Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * Dump: Allow dump output to stdout Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * Dump: Fixed bug present since #6677 where SessionConfig.Provider is never "file" Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * Dump: never pack RepoRootPath, LFS.ContentPath and LogRootPath when they are below AppDataPath Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * Dump: also dump LFS (fixes #10058) Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * Dump: never dump CustomPath if CustomPath is a subdir of or equal to AppDataPath (fixes #10365) Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * Use log.Info instead of fmt.Fprintf Signed-off-by: Philipp Homann <homann.philipp@googlemail.com> * import ordering * make fmt Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: Matti R <matti@mdranta.net>
* cmd: dump: check value of skip-repository flag (#11254)Kyle Evans2020-05-031-1/+1
| | | | | | This is a boolean flag; simply checking if it's set isn't enough, we must check the value as well. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* cmd: dump: add an -L/--skip-log option (#11253)Kyle Evans2020-04-301-1/+10
| | | | | | | | | | Not all dumps need to include the logs, in a similar vain to not all dumps needing to include repositories; these may be subject to different backup mechanisms/constraints. Add a simple option to let them be excluded from the dump to simplify workflows that need to exclude them or not collect in the first place. Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: zeripath <art27@cantab.net>
* fix dump non-exist log directory (#9818)Lunny Xiao2020-01-161-2/+4
|
* Fixed errors logging in dump.go (#9218)Rychu2019-12-171-26/+32
| | | | | | | | | | | | * Fixed errors logging in dump.go * Replaced overlooked log.Fatal() with fatal() * Run make fmt on dump.go * Added missing new line to the fatal log * Run make fmt on dump.go
* Move database settings from models to setting (#7806)Lunny Xiao2019-08-241-3/+2
| | | | | | | | | | | | | | | | | | * move database settings from models to setting * update docs * fix checkout pr * fix tests * fix lint * remove unsupported tidb options * correct wrong variable name * remove tidb totally
* Use gitea forked macaron (#7933)Tamal Saha2019-08-231-2/+2
| | | Signed-off-by: Tamal Saha <tamal@appscode.com>
* Change verbose flag in dump command to avoid colliding with global version ↵John Olheiser2019-05-011-1/+1
| | | | | | | | | flag (#6822) * Change verbose flag to avoid colliding with version flag * Update docs Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Make CustomPath, CustomConf and AppWorkPath configurable at build (#6631)zeripath2019-04-291-8/+0
|
* Include custom configuration file in dump (#6516)mrsdizzie2019-04-051-0/+8
| | | | | | | When running the dump command, include the specific configuration file provided as part of the backup. Currently we only zip up custom/ and this misses the config file if it is somewhere else, like /etc/gitea/app.ini Fixes #6034
* support custom file name in `gitea dump` command (#6474)glaszig2019-04-011-1/+6
| | | | | | | * support custom file name in `gitea dump` command * simpler approach to handle default dump file name in `gitea dump` command
* Add flag to skip repository dumping (#5695)Zsombor2019-01-131-14/+22
|
* General refactor of the cmd package (#3328)Morgan Bazalgette2018-01-121-8/+8
| | | | | | | | * General refactor of the cmd package * Address breakage in runCreateUser * Place "common" functions into cmd.go
* Set session and indexers' data files rel to AppDataPath (#2192)Guillaume Dube2017-11-031-4/+1
| | | | | | | | | | * Set session and indexers' data files rel to AppDataPath The setting AppDataPath is now relative to the working directory. The session svc's PROVIDER_CONFIG now defaults to AppDataPath/data/sessions. The issue indexer's IssuePath now defaults to AppDataPath/indexers/issues.bleves. * fix bug
* Set TMPDIR enviroment variable for dump command (#1915)Jonas Östanbäck2017-06-081-0/+5
|
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-02 17:16:18 +0000