aboutsummaryrefslogtreecommitdiffstats
path: root/modules/storage/minio.go
Commit message (Collapse)AuthorAgeFilesLines
* Supports forced use of S3 virtual-hosted style (#30969)dicarne2024-05-151-4/+16
| | | | | Add a configuration item to enable S3 virtual-hosted style (V2) to solve the problem caused by some S3 service providers not supporting path style (V1).
* Fix object storage path handling (#27024)wxiaoguang2023-09-131-14/+14
| | | | | | Object storage path rules: * No single `/` or `.`, use empty string for root path * Need to use trailing `/` for a list prefix to distinguish a "dir/"
* S3: log human readable error on connection failure (#26856)Earl Warren2023-09-121-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | Should BucketExists (HeadBucket) fail because of an error related to the connection rather than the existence of the bucket, no information is available and the admin is left guessing. https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html > This action is useful to determine if a bucket exists and you have > permission to access it. The action returns a 200 OK if the bucket > exists and you have permission to access it. > > If the bucket does not exist or you do not have permission to access > it, the HEAD request returns a generic 400 Bad Request, 403 > Forbidden or 404 Not Found code. A message body is not included, so > you cannot determine the exception beyond these error codes. GetBucketVersioning is used instead and exclusively dedicated to asserting if using the connection does not return a BadRequest. If it does the NewMinioStorage logs an error and returns. Otherwise it keeps going knowing that BucketExists is not going to fail for reasons unrelated to the existence of the bucket and the permissions to access it. (cherry picked from commit d1df4b3bc62e5e61893a923f1c4b58f084eb03af) Refs: https://codeberg.org/forgejo/forgejo/issues/1338
* Use correct minio error (#26634)delvh2023-08-211-2/+2
| | | | | | | | | | | Previously, `err` was defined above, checked for `err == nil` and used nowhere else. Hence, the result of `convertMinioErr` would always be `nil`. This leads to a NPE further down the line. That is not intentional, it should convert the error of the most recent operation, not one of its predecessors. Found through https://discord.com/channels/322538954119184384/322538954119184384/1143185780206993550.
* Adjust minio new sequence, now it will check whether bucket exist first and ↵Lunny Xiao2023-08-121-6/+10
| | | | | | | | | | | then create one if it doesn't exist (#26420) For some reason, the permission of the client_id and secret may cannot create bucket, so now we will check whether bucket does exist first and then try to create a bucket if it doesn't exist. Try to fix #25984 Co-authored-by: silverwind <me@silverwind.io>
* minio: add missing region on client initialization (#26412)nekrondev2023-08-101-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The MinIO client isn't redirecting to the correct AWS endpoint if a non-default data center is used. In my use case I created an AWS bucket at `eu-central-1` region. Because of the missing region initialization of the client the default `us-east-1` API endpoint is used returning a `301 Moved Permanently` response that's not handled properly by MinIO client. This in return aborts using S3 storage on AWS as the `BucketExists()` call will fail with the http moved error. MinIO client trace shows the issue: ```text ---------START-HTTP--------- HEAD / HTTP/1.1 Host: xxxxxxxxxxx-prod-gitea-data.s3.dualstack.us-east-1.amazonaws.com User-Agent: MinIO (windows; amd64) minio-go/v7.0.61 Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230809/accesspoint.eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20230809T141143Z HTTP/1.1 301 Moved Permanently Connection: close Content-Type: application/xml Date: Wed, 09 Aug 2023 14:11:43 GMT Server: AmazonS3 X-Amz-Bucket-Region: eu-central-1 X-Amz-Id-2: UK7wfeYi0HcTcytNvQ3wTAZ5ZP1mOSMnvRZ9Fz4xXzeNsS47NB/KfFx2unFxo3L7XckHpMNPPVo= X-Amz-Request-Id: S1V2MJV8SZ11GEVN ---------END-HTTP--------- ``` Co-authored-by: Heiko Besemann <heiko.besemann@qbeyond.de>
* Replace `interface{}` with `any` (#25686)silverwind2023-07-041-1/+1
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Fix all possible setting error related storages and added some tests (#23911)Lunny Xiao2023-06-141-25/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | Follow up #22405 Fix #20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* fix minio storage iterator path (#24691)FuXiaoHei2023-05-131-5/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | minio storage iterator shows different behavior with local fs iterator. in local fs storage: ``` go s.IterateObjects("prefix", func(path,obj) println(path) // show "prefix/xxx.file" }) ``` in minio storage: ```go s.IterateObjects("prefix", func(path,obj) println(path) // show "xxx.file" }) ``` I think local fs is correct, minio use wrong `basePath` to trim storage path prefix. --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Make minio package support legacy MD5 checksum (#23768)wxiaoguang2023-03-281-3/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | A feedback from discord: https://discord.com/channels/322538954119184384/561007778139734027/1090185427115319386 Some storages like: * https://developers.cloudflare.com/r2/api/s3/api/ * https://www.backblaze.com/b2/docs/s3_compatible_api.html They do not support "x-amz-checksum-algorithm" header But minio recently uses that header with CRC32C by default. So we have to tell minio to use legacy MD5 checksum. I guess this needs to be backported because IIRC we 1.19 and 1.20 are using similar minio package. The minio package code for SendContentMD5 looks like this: <details> <img width="755" alt="image" src="https://user-images.githubusercontent.com/2114189/228186768-4f2f6f67-62b9-4aee-9251-5af714ad9674.png"> </details>
* Introduce path Clean/Join helper functions (#23495)wxiaoguang2023-03-211-1/+1
| | | | | | | | | | | | | | | Since #23493 has conflicts with latest commits, this PR is my proposal for fixing #23371 Details are in the comments And refactor the `modules/options` module, to make it always use "filepath" to access local files. Benefits: * No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))), "/")` any more (not only one before) * The function behaviors are clearly defined
* add path prefix to ObjectStorage.Iterator (#23332)FuXiaoHei2023-03-131-3/+9
| | | | | | | | | | | | Support to iterator subdirectory in ObjectStorage for ObjectStorage.Iterator method. It's required for https://github.com/go-gitea/gitea/pull/22738 to make artifact files cleanable. --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Use CleanPath instead of path.Clean (#23371)Lunny Xiao2023-03-081-1/+2
| | | As title.
* Add InsecureSkipVerify to Minio Client for Storage (#23166)Yarden Shoham2023-02-271-9/+13
| | | | | | | Allows using Minio with untrusted certificates Closes #23128 Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-271-2/+1
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Clean paths when looking in Storage (#19124)zeripath2022-03-221-1/+1
| | | | | | | | | | | | | * Clean paths when looking in Storage Ensure paths are clean for minio aswell as local storage. Use url.Path not RequestURI/EscapedPath in storageHandler. Signed-off-by: Andrew Thornton <art27@cantab.net> * Apply suggestions from code review Co-authored-by: Lauris BH <lauris@nix.lv>
* format with gofumpt (#18184)65432022-01-201-2/+2
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Fix storage Iterate bug and Add storage doctor to delete garbage attachments ↵Lunny Xiao2021-09-061-2/+2
| | | | | | | (#16971) * Fix storage Iterate bug and Add storage doctor to delete garbage attachments * Close object when used
* Add size to Save function (#15264)zeripath2021-04-031-2/+2
| | | | | Fix #15253 Signed-off-by: Andrew Thornton <art27@cantab.net>
* When handling errors in storageHandler check underlying error (#13178)zeripath2020-10-171-1/+1
| | | | | | | | Unfortunately there was a mistake in #13164 which fails to handle os.PathError wrapping an os.ErrNotExist Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Prevent panics with missing storage (#13164)zeripath2020-10-151-15/+37
| | | | | | | | | | | | | | | | * The `.Use` of storageHandler before setting up the template renderer causes a panic if there is an error to log. * The error passed to `ctx.Error` in that case may contain sensitive information and should not be rendered to the end user. We should instead log the error and render a simple error message. * There is no handling of missing avatars and this needs a 404. Minio errors need to be mapped to standard golang errors such as os.ErrNotExist. * There is no logging when storage is set up. Related #13159 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Provide self-registering storage system (#12978)zeripath2020-10-121-11/+36
| | | | | | | | | | | | | | | | | | | * Provide self-registering storage system Signed-off-by: Andrew Thornton <art27@cantab.net> * More simplification Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove old strings from setting Signed-off-by: Andrew Thornton <art27@cantab.net> * oops attachments not attachment Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add default storage configurations (#12813)Lunny Xiao2020-09-291-2/+50
| | | | Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* LFS support to be stored on minio (#12518)Lunny Xiao2020-09-081-1/+37
| | | | | | | | | | | | | | | | | * LFS support to be stored on minio * Fix test * Fix lint * Fix lint * Fix check * Fix test * Update documents and add migration for LFS * Fix some bugs
* Add a storage layer for attachments (#11387)Lunny Xiao2020-08-181-0/+101
* Add a storage layer for attachments * Fix some bug * fix test * Fix copyright head and lint * Fix bug * Add setting for minio and flags for migrate-storage * Add documents * fix lint * Add test for minio store type on attachments * fix test * fix test * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Add warning when storage migrated successfully * Fix drone * fix test * rebase * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * remove log on xorm * Fi download bug * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * refactor the codes * add trace * Fix test * Add URL function to serve attachments directly from S3/Minio * Add ability to enable/disable redirection in attachment configuration * Fix typo * Add a storage layer for attachments * Add setting for minio and flags for migrate-storage * fix lint * Add test for minio store type on attachments * Apply suggestions from code review Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> * Fix drone * fix test * Fix test * display the error on console * Move minio test to amd64 since minio docker don't support arm64 * don't change unrelated files * Fix lint * Fix build * update go.mod and go.sum * Use github.com/minio/minio-go/v6 * Remove unused function * Upgrade minio to v7 and some other improvements * fix lint * Fix go mod Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: Tyler <tystuyfzand@gmail.com>