aboutsummaryrefslogtreecommitdiffstats
path: root/modules/storage
Commit message (Collapse)AuthorAgeFilesLines
* Implement actions (#21937)Jason Song2023-01-311-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Close #13539. Co-authored by: @lunny @appleboy @fuxiaohei and others. Related projects: - https://gitea.com/gitea/actions-proto-def - https://gitea.com/gitea/actions-proto-go - https://gitea.com/gitea/act - https://gitea.com/gitea/act_runner ### Summary The target of this PR is to bring a basic implementation of "Actions", an internal CI/CD system of Gitea. That means even though it has been merged, the state of the feature is **EXPERIMENTAL**, and please note that: - It is disabled by default; - It shouldn't be used in a production environment currently; - It shouldn't be used in a public Gitea instance currently; - Breaking changes may be made before it's stable. **Please comment on #13539 if you have any different product design ideas**, all decisions reached there will be adopted here. But in this PR, we don't talk about **naming, feature-creep or alternatives**. ### โš ๏ธ Breaking `gitea-actions` will become a reserved user name. If a user with the name already exists in the database, it is recommended to rename it. ### Some important reviews - What is `DEFAULT_ACTIONS_URL` in `app.ini` for? - https://github.com/go-gitea/gitea/pull/21937#discussion_r1055954954 - Why the api for runners is not under the normal `/api/v1` prefix? - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061173592 - Why DBFS? - https://github.com/go-gitea/gitea/pull/21937#discussion_r1061301178 - Why ignore events triggered by `gitea-actions` bot? - https://github.com/go-gitea/gitea/pull/21937#discussion_r1063254103 - Why there's no permission control for actions? - https://github.com/go-gitea/gitea/pull/21937#discussion_r1090229868 ### What it looks like <details> #### Manage runners <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205870657-c72f590e-2e08-4cd4-be7f-2e0abb299bbf.png"> #### List runs <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205872794-50fde990-2b45-48c1-a178-908e4ec5b627.png"> #### View logs <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205872501-9b7b9000-9542-4991-8f55-18ccdada77c3.png"> </details> ### How to try it <details> #### 1. Start Gitea Clone this branch and [install from source](https://docs.gitea.io/en-us/install-from-source). Add additional configurations in `app.ini` to enable Actions: ```ini [actions] ENABLED = true ``` Start it. If all is well, you'll see the management page of runners: <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205877365-8e30a780-9b10-4154-b3e8-ee6c3cb35a59.png"> #### 2. Start runner Clone the [act_runner](https://gitea.com/gitea/act_runner), and follow the [README](https://gitea.com/gitea/act_runner/src/branch/main/README.md) to start it. If all is well, you'll see a new runner has been added: <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205878000-216f5937-e696-470d-b66c-8473987d91c3.png"> #### 3. Enable actions for a repo Create a new repo or open an existing one, check the `Actions` checkbox in settings and submit. <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205879705-53e09208-73c0-4b3e-a123-2dcf9aba4b9c.png"> <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205879383-23f3d08f-1a85-41dd-a8b3-54e2ee6453e8.png"> If all is well, you'll see a new tab "Actions": <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205881648-a8072d8c-5803-4d76-b8a8-9b2fb49516c1.png"> #### 4. Upload workflow files Upload some workflow files to `.gitea/workflows/xxx.yaml`, you can follow the [quickstart](https://docs.github.com/en/actions/quickstart) of GitHub Actions. Yes, Gitea Actions is compatible with GitHub Actions in most cases, you can use the same demo: ```yaml name: GitHub Actions Demo run-name: ${{ github.actor }} is testing out GitHub Actions ๐Ÿš€ on: [push] jobs: Explore-GitHub-Actions: runs-on: ubuntu-latest steps: - run: echo "๐ŸŽ‰ The job was automatically triggered by a ${{ github.event_name }} event." - run: echo "๐Ÿง This job is now running on a ${{ runner.os }} server hosted by GitHub!" - run: echo "๐Ÿ”Ž The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}." - name: Check out repository code uses: actions/checkout@v3 - run: echo "๐Ÿ’ก The ${{ github.repository }} repository has been cloned to the runner." - run: echo "๐Ÿ–ฅ๏ธ The workflow is now ready to test your code on the runner." - name: List files in the repository run: | ls ${{ github.workspace }} - run: echo "๐Ÿ This job's status is ${{ job.status }}." ``` If all is well, you'll see a new run in `Actions` tab: <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205884473-79a874bc-171b-4aaf-acd5-0241a45c3b53.png"> #### 5. Check the logs of jobs Click a run and you'll see the logs: <img width="1792" alt="image" src="https://user-images.githubusercontent.com/9418365/205884800-994b0374-67f7-48ff-be9a-4c53f3141547.png"> #### 6. Go on You can try more examples in [the documents](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions) of GitHub Actions, then you might find a lot of bugs. Come on, PRs are welcome. </details> See also: [Feature Preview: Gitea Actions](https://blog.gitea.io/2022/12/feature-preview-gitea-actions/) --------- Co-authored-by: a1012112796 <1012112796@qq.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: ChristopherHX <christopher.homberger@web.de> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Change use of Walk to WalkDir to improve disk performance (#22462)Felipe Leopoldo Sologuren Gutiรฉrrez2023-01-161-2/+2
| | | | | | | | | | | | | As suggest by Go developers, use `filepath.WalkDir` instead of `filepath.Walk` because [*Walk is less efficient than WalkDir, introduced in Go 1.16, which avoids calling `os.Lstat` on every file or directory visited](https://pkg.go.dev/path/filepath#Walk). This proposition address that, in a similar way as https://github.com/go-gitea/gitea/pull/22392 did. Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Local storage should not store files as executable (#22162)zeripath2022-12-191-1/+2
| | | | | | | | | | The PR #21198 introduced a probable security vulnerability which resulted in making all storage files be marked as executable. This PR ensures that these are forcibly marked as non-executable. Fix #22161 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Skip initing LFS storage if disabled (#21996)Jason Song2022-12-011-0/+4
| | | | | | | A complement to #21985. I overlooked it because the name of the switch is `StartServer`, not `Enabled`. I believe the weird name is a legacy, but renaming is out of scope.
* Skip initing disabled storages (#21985)Jason Song2022-11-303-26/+108
| | | | If `Attachment` or `Packages` are disabled, we don't have to init the storages for them.
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-275-10/+5
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Save files in local storage as umask (#21198)Nulo (he/him)2022-09-241-0/+4
| | | | | | | | Go creates temporary files as 600, but sometimes we want the group to be able to read them (for example, for another user to back up the storage) This PR applies the umask to the renamed tmp files in local storage. Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Add more linters to improve code readability (#19989)Wim2022-06-201-6/+6
| | | | | | | | | | Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability - nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length. - unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions - wastedassign - https://github.com/sanposhiho/wastedassign - wastedassign finds wasted assignment statements. - notlintlint - Reports ill-formed or insufficient nolint directives - stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
* Add Package Registry (#16510)KN4CK3R2022-03-301-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added package store settings. * Added models. * Added generic package registry. * Added tests. * Added NuGet package registry. * Moved service index to api file. * Added NPM package registry. * Added Maven package registry. * Added PyPI package registry. * Summary is deprecated. * Changed npm name. * Sanitize project url. * Allow only scoped packages. * Added user interface. * Changed method name. * Added missing migration file. * Set page info. * Added documentation. * Added documentation links. * Fixed wrong error message. * Lint template files. * Fixed merge errors. * Fixed unit test storage path. * Switch to json module. * Added suggestions. * Added package webhook. * Add package api. * Fixed swagger file. * Fixed enum and comments. * Fixed NuGet pagination. * Print test names. * Added api tests. * Fixed access level. * Fix User unmarshal. * Added RubyGems package registry. * Fix lint. * Implemented io.Writer. * Added support for sha256/sha512 checksum files. * Improved maven-metadata.xml support. * Added support for symbol package uploads. * Added tests. * Added overview docs. * Added npm dependencies and keywords. * Added no-packages information. * Display file size. * Display asset count. * Fixed filter alignment. * Added package icons. * Formatted instructions. * Allow anonymous package downloads. * Fixed comments. * Fixed postgres test. * Moved file. * Moved models to models/packages. * Use correct error response format per client. * Use simpler search form. * Fixed IsProd. * Restructured data model. * Prevent empty filename. * Fix swagger. * Implemented user/org registry. * Implemented UI. * Use GetUserByIDCtx. * Use table for dependencies. * make svg * Added support for unscoped npm packages. * Add support for npm dist tags. * Added tests for npm tags. * Unlink packages if repository gets deleted. * Prevent user/org delete if a packages exist. * Use package unlink in repository service. * Added support for composer packages. * Restructured package docs. * Added missing tests. * Fixed generic content page. * Fixed docs. * Fixed swagger. * Added missing type. * Fixed ambiguous column. * Organize content store by sha256 hash. * Added admin package management. * Added support for sorting. * Add support for multiple identical versions/files. * Added missing repository unlink. * Added file properties. * make fmt * lint * Added Conan package registry. * Updated docs. * Unify package names. * Added swagger enum. * Use longer TEXT column type. * Removed version composite key. * Merged package and container registry. * Removed index. * Use dedicated package router. * Moved files to new location. * Updated docs. * Fixed JOIN order. * Fixed GROUP BY statement. * Fixed GROUP BY #2. * Added symbol server support. * Added more tests. * Set NOT NULL. * Added setting to disable package registries. * Moved auth into service. * refactor * Use ctx everywhere. * Added package cleanup task. * Changed packages path. * Added container registry. * Refactoring * Updated comparison. * Fix swagger. * Fixed table order. * Use token auth for npm routes. * Enabled ReverseProxy auth. * Added packages link for orgs. * Fixed anonymous org access. * Enable copy button for setup instructions. * Merge error * Added suggestions. * Fixed merge. * Handle "generic". * Added link for TODO. * Added suggestions. * Changed temporary buffer filename. * Added suggestions. * Apply suggestions from code review Co-authored-by: Thomas Boerger <thomas@webhippie.de> * Update docs/content/doc/packages/nuget.en-us.md Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: Thomas Boerger <thomas@webhippie.de>
* Clean paths when looking in Storage (#19124)zeripath2022-03-223-40/+30
| | | | | | | | | | | | | * Clean paths when looking in Storage Ensure paths are clean for minio aswell as local storage. Use url.Path not RequestURI/EscapedPath in storageHandler. Signed-off-by: Andrew Thornton <art27@cantab.net> * Apply suggestions from code review Co-authored-by: Lauris BH <lauris@nix.lv>
* Fix lfs bug (#19072)Lunny Xiao2022-03-142-0/+68
| | | * Fix lfs bug
* format with gofumpt (#18184)65432022-01-203-9/+5
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-221-2/+1
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Fix storage Iterate bug and Add storage doctor to delete garbage attachments โ†ตLunny Xiao2021-09-061-2/+2
| | | | | | | (#16971) * Fix storage Iterate bug and Add storage doctor to delete garbage attachments * Close object when used
* Close storage objects before cleaning (#16934)zeripath2021-09-031-0/+1
| | | | | | | | | | | | Storage.Iterate provides the path and an open object. On windows using local storage means that the objects will be locked thus preventing clean from deleting them. This PR simply closes the objects early. Fix #16932 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add an abstract json layout to make it's easier to change json library (#16528)Lunny Xiao2021-07-241-4/+1
| | | | | | | | | | | * Add an abstract json layout to make it's easier to change json library * Fix import * Fix import sequence * Fix blank lines * Fix blank lines
* Retry rename on lock induced failures (#16435)zeripath2021-07-151-1/+1
| | | | | | | | | | | | | | | | | | * Retry rename on lock induced failures Due to external locking on Windows it is possible for an os.Rename to fail if the files or directories are being used elsewhere. This PR simply suggests retrying the rename again similar to how we handle the os.Remove problems. Fix #16427 Signed-off-by: Andrew Thornton <art27@cantab.net> * resolve CI fail Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Fix archive error when rename repo or user (#16399)Lunny Xiao2021-07-131-0/+7
| | | Use repo id instead of full name to generate archive path
* Fix various documentation, user-facing, and source comment typos (#16367)luzpaz2021-07-081-1/+1
| | | | | * Fix various doc, user-facing, and source comment typos Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
* Rework repository archive (#14723)Lunny Xiao2021-06-231-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Use storage to store archive files * Fix backend lint * Add archiver table on database * Finish archive download * Fix test * Add database migrations * Add status for archiver * Fix lint * Add queue * Add doctor to check and delete old archives * Improve archive queue * Fix tests * improve archive storage * Delete repo archives * Add missing fixture * fix fixture * Fix fixture * Fix test * Fix archiver cleaning * Fix bug * Add docs for repository archive storage * remove repo-archive configuration * Fix test * Fix test * Fix lint Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Code Formats, Nits & Unused Func/Var deletions (#15286)65432021-04-092-3/+1
| | | | | | | | | | | | | | | * _ to unused func options * rm useless brakets * rm trifial non used models functions * rm dead code * rm dead global vars * fix routers/api/v1/repo/issue.go * dont overload import module
* Add size to Save function (#15264)zeripath2021-04-033-6/+13
| | | | | Fix #15253 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix race in local storage (#14888)zeripath2021-03-051-10/+38
| | | | | LocalStorage should only put completed files in position Signed-off-by: Andrew Thornton <art27@cantab.net>
* Migrate to use jsoniter instead of encoding/json (#14841)zeripath2021-03-011-1/+3
| | | | | | | | | | * Migrate to use jsoniter * fix tests * update gitea.com/go-chi/binding Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>
* Fix Storage mapping (#13297)zeripath2020-10-251-4/+4
| | | | | | | | | | | | | This PR fixes several bugs in setting storage * The default STORAGE_TYPE should be the provided type. * The Storage config should be passed in to NewStorage as a pointer - otherwise the Mappable interface function MapTo will not be found * There was a bug in the MapTo function. Fix #13286 Signed-off-by: Andrew Thornton <art27@cantab.net>
* When handling errors in storageHandler check underlying error (#13178)zeripath2020-10-171-1/+1
| | | | | | | | Unfortunately there was a mistake in #13164 which fails to handle os.PathError wrapping an os.ErrNotExist Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Prevent panics with missing storage (#13164)zeripath2020-10-153-15/+44
| | | | | | | | | | | | | | | | * The `.Use` of storageHandler before setting up the template renderer causes a panic if there is an error to log. * The error passed to `ctx.Error` in that case may contain sensitive information and should not be rendered to the end user. We should instead log the error and render a simple error message. * There is no handling of missing avatars and this needs a 404. Minio errors need to be mapped to standard golang errors such as os.ErrNotExist. * There is no logging when storage is set up. Related #13159 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Avatars and Repo avatars support storing in minio (#12516)Lunny Xiao2020-10-141-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Avatar support minio * Support repo avatar minio storage * Add missing migration * Fix bug * Fix test * Add test for minio store type on avatars and repo avatars; Add documents * Fix bug * Fix bug * Add back missed avatar link method * refactor codes * Simplify the codes * Code improvements * Fix lint * Fix test mysql * Fix test mysql * Fix test mysql * Fix settings * Fix test * fix test * Fix bug
* Provide self-registering storage system (#12978)zeripath2020-10-124-40/+172
| | | | | | | | | | | | | | | | | | | * Provide self-registering storage system Signed-off-by: Andrew Thornton <art27@cantab.net> * More simplification Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove old strings from setting Signed-off-by: Andrew Thornton <art27@cantab.net> * oops attachments not attachment Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add default storage configurations (#12813)Lunny Xiao2020-09-293-47/+100
| | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* LFS support to be stored on minio (#12518)Lunny Xiao2020-09-083-3/+98
| | | | | | | | | | | | | | | | | * LFS support to be stored on minio * Fix test * Fix lint * Fix lint * Fix check * Fix test * Update documents and add migration for LFS * Fix some bugs
* Add a storage layer for attachments (#11387)Lunny Xiao2020-08-183-0/+244
* Add a storage layer for attachments * Fix some bug * fix test * Fix copyright head and lint * Fix bug * Add setting for minio and flags for migrate-storage * Add documents * fix lint * Add test for minio store type on attachments * fix test * fix test * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Add warning when storage migrated successfully * Fix drone * fix test * rebase * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * remove log on xorm * Fi download bug * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * Add URL function to serve attachments directly from S3/Minio * Add ability to enable/disable redirection in attachment configuration * Fix typo * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * don't change unrelated files * Fix lint * Fix build * update go.mod and go.sum * Use github.com/minio/minio-go/v6 * Remove unused function * Upgrade minio to v7 and some other improvements * fix lint * Fix go mod Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Tyler <tystuyfzand@gmail.com>
generated by cgit v1.2.3 (git 2.39.1) at 2025-01-27 02:39:31 +0000