summaryrefslogtreecommitdiffstats
path: root/services/lfs/server.go
Commit message (Collapse)AuthorAgeFilesLines
* Move context from modules to services (#29440)Lunny Xiao2024-02-271-1/+1
| | | | | | | | | | | | | | | Since `modules/context` has to depend on `models` and many other packages, it should be moved from `modules/context` to `services/context` according to design principles. There is no logic code change on this PR, only move packages. - Move `code.gitea.io/gitea/modules/context` to `code.gitea.io/gitea/services/context` - Move `code.gitea.io/gitea/modules/contexttest` to `code.gitea.io/gitea/services/contexttest` because of depending on context - Move `code.gitea.io/gitea/modules/upload` to `code.gitea.io/gitea/services/context/upload` because of depending on context
* Use `crypto/sha256` (#29386)KN4CK3R2024-02-251-1/+1
| | | | | | | | Go 1.21 improved the performance of `crypto/sha256`. It's now similar to `minio/sha256-simd`, so we should just use the standard libs. https://go.dev/doc/go1.21#crypto/sha256 https://go-review.googlesource.com/c/go/+/408795 https://github.com/multiformats/go-multihash/pull/173
* Remove GetByBean method because sometimes it's danger when query condition ↵Lunny Xiao2023-12-071-2/+2
| | | | | | | | | | | | | | | | | | | | parameter is zero and also introduce new generic methods (#28220) The function `GetByBean` has an obvious defect that when the fields are empty values, it will be ignored. Then users will get a wrong result which is possibly used to make a security problem. To avoid the possibility, this PR removed function `GetByBean` and all references. And some new generic functions have been introduced to be used. The recommand usage like below. ```go // if query an object according id obj, err := db.GetByID[Object](ctx, id) // query with other conditions obj, err := db.Get[Object](ctx, builder.Eq{"a": a, "b":b}) ```
* Bump github.com/golang-jwt/jwt to v5 (#25975)harryzcy2023-07-191-1/+1
| | | | | | | | | | | | | | | | Bumping `github.com/golang-jwt/jwt` from v4 to v5. `github.com/golang-jwt/jwt` v5 is bringing some breaking changes: - standard `Valid()` method on claims is removed. It's replaced by `ClaimsValidator` interface implementing `Validator()` method instead, which is called after standard validation. Gitea doesn't seem to be using this logic. - `jwt.Token` has a field `Valid`, so it's checked in `ParseToken` function in `services/auth/source/oauth2/token.go` --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Replace `interface{}` with `any` (#25686)silverwind2023-07-041-2/+2
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Move some regexp out of functions (#25430)John Olheiser2023-06-211-2/+3
| | | | | | | | | | /cc @KN4CK3R https://github.com/go-gitea/gitea/pull/25294#discussion_r1237425343 I also searched the codebase and found a few more. --------- Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Fix all possible setting error related storages and added some tests (#23911)Lunny Xiao2023-06-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Follow up #22405 Fix #20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Redesign Scoped Access Tokens (#24767)Jack Hay2023-06-041-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## Changes - Adds the following high level access scopes, each with `read` and `write` levels: - `activitypub` - `admin` (hidden if user is not a site admin) - `misc` - `notification` - `organization` - `package` - `issue` - `repository` - `user` - Adds new middleware function `tokenRequiresScopes()` in addition to `reqToken()` - `tokenRequiresScopes()` is used for each high-level api section - _if_ a scoped token is present, checks that the required scope is included based on the section and HTTP method - `reqToken()` is used for individual routes - checks that required authentication is present (but does not check scope levels as this will already have been handled by `tokenRequiresScopes()` - Adds migration to convert old scoped access tokens to the new set of scopes - Updates the user interface for scope selection ### User interface example <img width="903" alt="Screen Shot 2023-05-31 at 1 56 55 PM" src="https://github.com/go-gitea/gitea/assets/23248839/654766ec-2143-4f59-9037-3b51600e32f3"> <img width="917" alt="Screen Shot 2023-05-31 at 1 56 43 PM" src="https://github.com/go-gitea/gitea/assets/23248839/1ad64081-012c-4a73-b393-66b30352654c"> ## tokenRequiresScopes Design Decision - `tokenRequiresScopes()` was added to more reliably cover api routes. For an incoming request, this function uses the given scope category (say `AccessTokenScopeCategoryOrganization`) and the HTTP method (say `DELETE`) and verifies that any scoped tokens in use include `delete:organization`. - `reqToken()` is used to enforce auth for individual routes that require it. If a scoped token is not present for a request, `tokenRequiresScopes()` will not return an error ## TODO - [x] Alphabetize scope categories - [x] Change 'public repos only' to a radio button (private vs public). Also expand this to organizations - [X] Disable token creation if no scopes selected. Alternatively, show warning - [x] `reqToken()` is missing from many `POST/DELETE` routes in the api. `tokenRequiresScopes()` only checks that a given token has the correct scope, `reqToken()` must be used to check that a token (or some other auth) is present. - _This should be addressed in this PR_ - [x] The migration should be reviewed very carefully in order to minimize access changes to existing user tokens. - _This should be addressed in this PR_ - [x] Link to api to swagger documentation, clarify what read/write/delete levels correspond to - [x] Review cases where more than one scope is needed as this directly deviates from the api definition. - _This should be addressed in this PR_ - For example: ```go m.Group("/users/{username}/orgs", func() { m.Get("", reqToken(), org.ListUserOrgs) m.Get("/{org}/permissions", reqToken(), org.GetUserOrgsPermissions) }, tokenRequiresScopes(auth_model.AccessTokenScopeCategoryUser, auth_model.AccessTokenScopeCategoryOrganization), context_service.UserAssignmentAPI()) ``` ## Future improvements - [ ] Add required scopes to swagger documentation - [ ] Redesign `reqToken()` to be opt-out rather than opt-in - [ ] Subdivide scopes like `repository` - [ ] Once a token is created, if it has no scopes, we should display text instead of an empty bullet point - [ ] If the 'public repos only' option is selected, should read categories be selected by default Closes #24501 Closes #24799 Co-authored-by: Jonathan Tran <jon@allspice.io> Co-authored-by: Kyle D <kdumontnu@gmail.com> Co-authored-by: silverwind <me@silverwind.io>
* Fix auth check bug (#24382)Lunny Xiao2023-04-271-10/+0
| | | | | | | Fix https://github.com/go-gitea/gitea/pull/24362/files#r1179095324 `getAuthenticatedMeta` has checked them, these code are duplicated one. And the first invokation has a wrong permission check. `DownloadHandle` should require read permission but not write.
* Require repo scope for PATs for private repos and basic authentication (#24362)John Olheiser2023-04-261-0/+15
| | | | | | | | | | > The scoped token PR just checked all API routes but in fact, some web routes like `LFS`, git `HTTP`, container, and attachments supports basic auth. This PR added scoped token check for them. --------- Signed-off-by: jolheiser <john.olheiser@gmail.com> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve LFS error logs (#24072)wxiaoguang2023-04-121-2/+3
| | | | | The error logs were not clear. Help (but not fix) #24053
* Check `IsActionsToken` for LFS authentication (#23841)Zettat1232023-04-021-1/+19
| | | | | | | Close #23824 Actions cannot fetch LFS objects from private repos because we don't check if the user is the `ActionUser`.
* Use minio/sha256-simd for accelerated SHA256 (#23052)zeripath2023-02-221-1/+1
| | | | | | | | | | | | minio/sha256-simd provides additional acceleration for SHA256 using AVX512, SHA Extensions for x86 and ARM64 for ARM. It provides a drop-in replacement for crypto/sha256 and if the extensions are not available it falls back to standard crypto/sha256. --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Use context parameter in models/git (#22367)Jason Song2023-01-091-7/+7
| | | | | | | | | After #22362, we can feel free to use transactions without `db.DefaultContext`. And there are still lots of models using `db.DefaultContext`, I think we should refactor them carefully and one by one. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* refactor some functions to support ctx as first parameter (#21878)Lunny Xiao2022-12-031-6/+7
| | | | Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: Lauris BH <lauris@nix.lv>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-271-2/+1
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Prevent Authorization header for presigned LFS urls (#21531)KN4CK3R2022-10-221-2/+9
| | | | | Fixes #21525 Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Move some code into models/git (#19879)Lunny Xiao2022-06-121-10/+10
| | | | | | | | | | | | | | | | | | | * Move access and repo permission to models/perm/access * fix test * Move some git related files into sub package models/git * Fix build * fix git test * move lfs to sub package * move more git related functions to models/git * Move functions sequence * Some improvements per @KN4CK3R and @delvh
* Move access and repo permission to models/perm/access (#19350)Lunny Xiao2022-05-111-1/+2
| | | | | * Move access and repo permission to models/perm/access * Remove unnecessary code
* more context for models (#19511)65432022-04-281-1/+1
| | | | | make more usage of context, to have more db transaction in one session (make diff of #9307 smaller)
* Renamed ctx.User to ctx.Doer. (#19161)KN4CK3R2022-03-221-5/+5
| | | | Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor jwt.StandardClaims to RegisteredClaims (#18344)zeripath2022-01-201-2/+1
| | | | | | | | | | | | | | | * Refactor jwt.StandardClaims to RegisteredClaims go-jwt/jwt has deprecated the StandardClaims interface to use RegisteredClaims instead. This PR migrates to use this new format. Signed-off-by: Andrew Thornton <art27@cantab.net> * Apply suggestions from code review Co-authored-by: Gusted <williamzijl7@hotmail.com> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Gusted <williamzijl7@hotmail.com>
* format with gofumpt (#18184)65432022-01-201-1/+1
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Support webauthn (#17957)Lunny Xiao2022-01-141-1/+2
| | | | | | | Migrate from U2F to Webauthn Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move repository model into models/repo (#17933)Lunny Xiao2021-12-101-8/+9
| | | | | | | | | | | | | | | * Some refactors related repository model * Move more methods out of repository * Move repository into models/repo * Fix test * Fix test * some improvements * Remove unnecessary function
* Move accessmode into models/perm (#17828)Lunny Xiao2021-11-281-5/+6
|
* Move user related model into models/user (#17781)Lunny Xiao2021-11-241-3/+4
| | | | | | | | | | | | | * Move user related model into models/user * Fix lint for windows * Fix windows lint * Fix windows lint * Move some tests in models * Merge
* Multiple Escaping Improvements (#17551)zeripath2021-11-161-3/+4
| | | | | | | | | | | | | | There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this. This is an extensive PR attempting to fix these issues. 1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping. 2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand. 3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future. 4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.) 5. The title generation in feeds is now properly escaped. 6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net> Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move unit into models/unit/ (#17576)Lunny Xiao2021-11-091-1/+2
| | | | | * Move unit into models/unit/ * Rename unit.UnitType as unit.Type
* Test if LFS object is accessible (#16865)KN4CK3R2021-08-311-14/+45
| | | | | | * Test if object is accessible. * Added more logging.
* Download lfs in git and web workflow from minio/s3 directly (#16731)Abner2021-08-211-0/+8
|
* Add an abstract json layout to make it's easier to change json library (#16528)Lunny Xiao2021-07-241-4/+4
| | | | | | | | | | | * Add an abstract json layout to make it's easier to change json library * Fix import * Fix import sequence * Fix blank lines * Fix blank lines
* switch to maintained lib (#16532)techknowlogick2021-07-241-1/+1
| | | | Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Rewrite of the LFS server (#15523)KN4CK3R2021-06-061-341/+217
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Restructured code. Moved static checks out of loop. * Restructured batch api. Add support for individual errors. * Let router decide if LFS is enabled. * Renamed methods. * Return correct status from verify handler. * Unified media type check in router. * Changed error code according to spec. * Moved checks into router. * Removed invalid v1 api methods. * Unified methods. * Display better error messages. * Added size parameter. Create meta object on upload. * Use object error on invalid size. * Skip upload if object exists. * Moved methods. * Suppress fields in response. * Changed error on accept. * Added tests. * Use ErrorResponse object. * Test against message property. * Add support for the old invalid lfs client. * Fixed the check because MinIO wraps the error. * Use individual repositories. Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv>
* Prevent double-login for Git HTTP and LFS and simplify login (#15303)zeripath2021-05-151-69/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Prevent double-login for Git HTTP and LFS and simplify login There are a number of inconsistencies with our current methods for logging in for git and lfs. The first is that there is a double login process. This is particularly evident in 1.13 where there are no less than 4 hash checks for basic authentication due to the previous IsPasswordSet behaviour. This duplicated code had individual inconsistencies that were not helpful and caused confusion. This PR does the following: * Remove the specific login code from the git and lfs handlers except for the lfs special bearer token * Simplify the meaning of DisableBasicAuthentication to allow Token and Oauth2 sign-in. * The removal of the specific code from git and lfs means that these both now have the same login semantics and can - if not DisableBasicAuthentication - login from external services. Further it allows Oauth2 token authentication as per our standard mechanisms. * The change in the recovery handler prevents the service from re-attempting to login - primarily because this could easily cause a further panic and it is wasteful. * add test Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Add LFS Migration and Mirror (#14726)KN4CK3R2021-04-081-0/+666
* Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-26 15:17:09 +0000