summaryrefslogtreecommitdiffstats
path: root/modules/lfs
Commit message (Collapse)AuthorAgeFilesLines
* Enable tenv and testifylint rules (#32852)TheFox0x72024-12-151-3/+3
| | | | Enables tenv and testifylint linters closes: https://github.com/go-gitea/gitea/issues/32842
* In some lfs server implementations, they require the ref attribute. (#32838)hiifong2024-12-151-1/+4
| | | | | | | | | Fix: #32611 In some lfs server implementations, they require the ref attribute. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Add User-Agent for gitea's self-implemented lfs client. (#32832)hiifong2024-12-141-3/+6
|
* Fix lfs migration (#32812)hiifong2024-12-122-1/+3
| | | Fix: #32803
* Use 8 as default value for git lfs concurrency (#32421)wxiaoguang2024-11-052-2/+5
|
* Make LFS http_client parallel within a batch. (#32369)Royce Remer2024-11-042-133/+137
| | | | Signed-off-by: Royce Remer <royceremer@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Add new [lfs_client].BATCH_SIZE and [server].LFS_MAX_BATCH_SIZE config ↵Royce Remer2024-10-301-3/+2
| | | | | | | | | | | | | | | | | | settings. (#32307) This contains two backwards-compatible changes: * in the lfs http_client, the number of lfs oids requested per batch is loaded from lfs_client#BATCH_SIZE and defaulted to the previous value of 20 * in the lfs server/service, the max number of lfs oids allowed in a batch api request is loaded from server#LFS_MAX_BATCH_SIZE and defaults to 'nil' which equates to the previous behavior of 'infinite' This fixes #32306 --------- Signed-off-by: Royce Remer <royceremer@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Distinguish LFS object errors to ignore missing objects during migration ↵Jason Song2024-07-312-4/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#31702) Fix #31137. Replace #31623 #31697. When migrating LFS objects, if there's any object that failed (like some objects are losted, which is not really critical), Gitea will stop migrating LFS immediately but treat the migration as successful. This PR checks the error according to the [LFS api doc](https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md#successful-responses). > LFS object error codes should match HTTP status codes where possible: > > - 404 - The object does not exist on the server. > - 409 - The specified hash algorithm disagrees with the server's acceptable options. > - 410 - The object was removed by the owner. > - 422 - Validation error. If the error is `404`, it's safe to ignore it and continue migration. Otherwise, stop the migration and mark it as failed to ensure data integrity of LFS objects. And maybe we should also ignore others errors (maybe `410`? I'm not sure what's the difference between "does not exist" and "removed by the owner".), we can add it later when some users report that they have failed to migrate LFS because of an error which should be ignored.
* Support legacy _links LFS batch responses (#31513)Royce Remer2024-06-283-0/+21
| | | | | | | | | | | | | | | | | | | | | Support legacy _links LFS batch response. Fixes #31512. This is backwards-compatible change to the LFS client so that, upon mirroring from an upstream which has a batch api, it can download objects whether the responses contain the `_links` field or its successor the `actions` field. When Gitea must fallback to the legacy `_links` field a logline is emitted at INFO level which looks like this: ``` ...s/lfs/http_client.go:188:performOperation() [I] <LFSPointer ee95d0a27ccdfc7c12516d4f80dcf144a5eaf10d0461d282a7206390635cdbee:160> is using a deprecated batch schema response! ``` I've only run `test-backend` with this code, but added a new test to cover this case. Additionally I have a fork with this change deployed which I've confirmed syncs LFS from Gitea<-Artifactory (which has legacy `_links`) as well as from Gitea<-Gitea (which has the modern `actions`). Signed-off-by: Royce Remer <royceremer@gmail.com>
* Fix #31185 try fix lfs download from bitbucket failed (#31201)Zoupers Zou2024-06-115-5/+8
| | | Fix #31185
* Make sure git version&feature are always prepared (#30877)wxiaoguang2024-05-061-1/+1
| | | Otherwise there would be more similar issues like #29287
* fix: close file in the Upload func (#30262)guangwu2024-04-031-2/+2
|
* Use `crypto/sha256` (#29386)KN4CK3R2024-02-252-4/+2
| | | | | | | | Go 1.21 improved the performance of `crypto/sha256`. It's now similar to `minio/sha256-simd`, so we should just use the standard libs. https://go.dev/doc/go1.21#crypto/sha256 https://go-review.googlesource.com/c/go/+/408795 https://github.com/multiformats/go-multihash/pull/173
* Update tool dependencies (#29030)silverwind2024-02-021-1/+0
|
* Do not set `Accept` header twice (#28598)KN4CK3R2023-12-281-4/+1
| | | | | Revert #28550 Don't add the `Accept` header twice.
* Add missing head of lfs client batch (#28550)Lunny Xiao2023-12-201-1/+4
| | | ref https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md#git-lfs-batch-api
* Upgrade to golangci-lint@v1.55.0 (#27756)silverwind2023-10-241-3/+2
| | | https://github.com/golangci/golangci-lint/releases/tag/v1.55.0
* Refactor lfs requests (#26783)Chongyi Zheng2023-09-185-135/+123
| | | | | | | | | | | | | | - Refactor lfs request code - The original code uses `performRequest` function to create the request, uses a callback to modify the request, and then send the request. - Now it's replaced with `createRequest` that only creates request and `performRequest` that only sends the request. - Reuse `createRequest` and `performRequest` in `http_client.go` and `transferadapter.go` --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Rewrite logger system (#24726)wxiaoguang2023-05-211-9/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## ⚠️ Breaking The `log.<mode>.<logger>` style config has been dropped. If you used it, please check the new config manual & app.example.ini to make your instance output logs as expected. Although many legacy options still work, it's encouraged to upgrade to the new options. The SMTP logger is deleted because SMTP is not suitable to collect logs. If you have manually configured Gitea log options, please confirm the logger system works as expected after upgrading. ## Description Close #12082 and maybe more log-related issues, resolve some related FIXMEs in old code (which seems unfixable before) Just like rewriting queue #24505 : make code maintainable, clear legacy bugs, and add the ability to support more writers (eg: JSON, structured log) There is a new document (with examples): `logging-config.en-us.md` This PR is safer than the queue rewriting, because it's just for logging, it won't break other logic. ## The old problems The logging system is quite old and difficult to maintain: * Unclear concepts: Logger, NamedLogger, MultiChannelledLogger, SubLogger, EventLogger, WriterLogger etc * Some code is diffuclt to konw whether it is right: `log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs `log.DelLogger("console")` * The old system heavily depends on ini config system, it's difficult to create new logger for different purpose, and it's very fragile. * The "color" trick is difficult to use and read, many colors are unnecessary, and in the future structured log could help * It's difficult to add other log formats, eg: JSON format * The log outputer doesn't have full control of its goroutine, it's difficult to make outputer have advanced behaviors * The logs could be lost in some cases: eg: no Fatal error when using CLI. * Config options are passed by JSON, which is quite fragile. * INI package makes the KEY in `[log]` section visible in `[log.sub1]` and `[log.sub1.subA]`, this behavior is quite fragile and would cause more unclear problems, and there is no strong requirement to support `log.<mode>.<logger>` syntax. ## The new design See `logger.go` for documents. ## Screenshot <details> ![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff) ![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9) ![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee) </details> ## TODO * [x] add some new tests * [x] fix some tests * [x] test some sub-commands (manually ....) --------- Co-authored-by: Jason Song <i@wolfogre.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Giteabot <teabot@gitea.io>
* Make repository response support HTTP range request (#24592)wxiaoguang2023-05-091-3/+3
| | | | | | Replace #20480 Replace #18448 Close #16414
* Use more specific test methods (#24265)KN4CK3R2023-04-221-2/+2
| | | | Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Giteabot <teabot@gitea.io>
* Make minio package support legacy MD5 checksum (#23768)wxiaoguang2023-03-281-11/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | A feedback from discord: https://discord.com/channels/322538954119184384/561007778139734027/1090185427115319386 Some storages like: * https://developers.cloudflare.com/r2/api/s3/api/ * https://www.backblaze.com/b2/docs/s3_compatible_api.html They do not support "x-amz-checksum-algorithm" header But minio recently uses that header with CRC32C by default. So we have to tell minio to use legacy MD5 checksum. I guess this needs to be backported because IIRC we 1.19 and 1.20 are using similar minio package. The minio package code for SendContentMD5 looks like this: <details> <img width="755" alt="image" src="https://user-images.githubusercontent.com/2114189/228186768-4f2f6f67-62b9-4aee-9251-5af714ad9674.png"> </details>
* Use minio/sha256-simd for accelerated SHA256 (#23052)zeripath2023-02-222-2/+4
| | | | | | | | | | | | minio/sha256-simd provides additional acceleration for SHA256 using AVX512, SHA Extensions for x86 and ARM64 for ARM. It provides a drop-in replacement for crypto/sha256 and if the extensions are not available it falls back to standard crypto/sha256. --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* escape filename when assemble URL (#22850)sillyguodong2023-02-121-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #22843 ### Cause: https://github.com/go-gitea/gitea/blob/affdd40296960a08a4223330ccbd1fb88c96ea1a/services/repository/files/content.go#L161 Previously, we did not escape the **"%"** that might be in "treePath" when call "url.parse()". ![image](https://user-images.githubusercontent.com/33891828/218066318-5a909e50-2a17-46e6-b32f-684b2aa4b91f.png) This function will check whether "%" is the beginning of an escape character. Obviously, the "%" in the example (hello%mother.txt) is not that. So, the function will return a error. ### Solution: We can escape "treePath" by call "url.PathEscape()" function firstly. ### Screenshot: ![image](https://user-images.githubusercontent.com/33891828/218069781-1a030f8b-18d0-4804-b0f8-73997849ef43.png) --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-2715-30/+15
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Removed some vestigial code related to Range bounds checks (#20312)Philip Peterson2022-07-281-16/+0
|
* Move some code into models/git (#19879)Lunny Xiao2022-06-121-1/+1
| | | | | | | | | | | | | | | | | | | * Move access and repo permission to models/perm/access * fix test * Move some git related files into sub package models/git * Fix build * fix git test * move lfs to sub package * move more git related functions to models/git * Move functions sequence * Some improvements per @KN4CK3R and @delvh
* Remove legacy `+build:` constraint (#19582)wxiaoguang2022-05-022-2/+0
| | | Go 1.17 and later use modern `//go:build` constraints, the old `// +build:` constraints should be removed.
* Update HTTP status codes to modern codes (#18063)KN4CK3R2022-03-231-1/+1
| | | | | | * 2xx/3xx/4xx/5xx -> http.Status... * http.StatusFound -> http.StatusTemporaryRedirect * http.StatusMovedPermanently -> http.StatusPermanentRedirect
* Improve SyncMirrors logging (#19045)zeripath2022-03-101-0/+13
| | | | | | | | | Yet another issue has come up where the logging from SyncMirrors does not provide enough context. This PR adds more context to these logging events. Related #19038 Signed-off-by: Andrew Thornton <art27@cantab.net>
* format with gofumpt (#18184)65432022-01-203-6/+4
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Propagate context and ensure git commands run in request context (#17868)zeripath2022-01-191-4/+4
| | | | | | | | | This PR continues the work in #17125 by progressively ensuring that git commands run within the request context. This now means that the if there is a git repo already open in the context it will be used instead of reopening it. Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix source typos (#18227)luzpaz2022-01-101-2/+2
| | | Follow-up to #18219
* Use `hostmatcher` to replace `matchlist`, improve security (#17605)wxiaoguang2021-11-203-10/+13
| | | | | Use hostmacher to replace matchlist. And we introduce a better DialContext to do a full host/IP check, otherwise the attackers can still bypass the allow/block list by a 302 redirection.
* Fix some lints (#17337)Lunny Xiao2021-10-171-3/+1
| | | Fix some linting problems.
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-222-8/+6
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Add bundle download for repository (#14538)John Olheiser2021-08-242-0/+2
| | | | | | | | | | | | | | | | | * Add bundle download Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix build tags Signed-off-by: jolheiser <john.olheiser@gmail.com> * Download specific commit Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Add proxy settings and support for migration and webhook (#16704)Lunny Xiao2021-08-183-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | * Add proxy settings and support for migration and webhook * Fix default value * Add newline for example ini * Add lfs proxy support * Fix lint * Follow @zeripath's review * Fix git clone * Fix test * missgin http requests for proxy * use empty Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: zeripath <art27@cantab.net>
* Add an abstract json layout to make it's easier to change json library (#16528)Lunny Xiao2021-07-244-15/+15
| | | | | | | | | | | * Add an abstract json layout to make it's easier to change json library * Fix import * Fix import sequence * Fix blank lines * Fix blank lines
* Fix various documentation, user-facing, and source comment typos (#16367)luzpaz2021-07-081-1/+1
| | | | | * Fix various doc, user-facing, and source comment typos Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
* Add push to remote mirror repository (#15157)KN4CK3R2021-06-148-138/+662
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Added push mirror model. * Integrated push mirror into queue. * Moved methods into own file. * Added basic implementation. * Mirror wiki too. * Removed duplicated method. * Get url for different remotes. * Added migration. * Unified remote url access. * Add/Remove push mirror remotes. * Prevent hangs with missing credentials. * Moved code between files. * Changed sanitizer interface. * Added push mirror backend methods. * Only update the mirror remote. * Limit refs on push. * Added UI part. * Added missing table. * Delete mirror if repository gets removed. * Changed signature. Handle object errors. * Added upload method. * Added "upload" unit tests. * Added transfer adapter unit tests. * Send correct headers. * Added pushing of LFS objects. * Added more logging. * Simpler body handling. * Process files in batches to reduce HTTP calls. * Added created timestamp. * Fixed invalid column name. * Changed name to prevent xorm auto setting. * Remove table header im empty. * Strip exit code from error message. * Added docs page about mirroring. * Fixed date. * Fixed merge errors. * Moved test to integrations. * Added push mirror test. * Added test.
* Fixed assert statements. (#16089)KN4CK3R2021-06-072-12/+12
|
* Rewrite of the LFS server (#15523)KN4CK3R2021-06-061-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Restructured code. Moved static checks out of loop. * Restructured batch api. Add support for individual errors. * Let router decide if LFS is enabled. * Renamed methods. * Return correct status from verify handler. * Unified media type check in router. * Changed error code according to spec. * Moved checks into router. * Removed invalid v1 api methods. * Unified methods. * Display better error messages. * Added size parameter. Create meta object on upload. * Use object error on invalid size. * Skip upload if object exists. * Moved methods. * Suppress fields in response. * Changed error on accept. * Added tests. * Use ErrorResponse object. * Test against message property. * Add support for the old invalid lfs client. * Fixed the check because MinIO wraps the error. * Use individual repositories. Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv>
* Fix lfs management find (#15537)Lunny Xiao2021-04-211-0/+2
| | | | | | | | | Fix #15236 * Do not do 40byte conversion within ParseTreeLine * Missed a to40ByteSHA Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Add LFS Migration and Mirror (#14726)KN4CK3R2021-04-0818-1157/+1195
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
* Close file on invalid range (Addition to #15166) (#15268)KN4CK3R2021-04-063-29/+21
| | | | | | | | | | | | | | | * Close file on invalid range. * Close on seek error Signed-off-by: Andrew Thornton <art27@cantab.net> * Moved 'Seek' into server. * io.ReadSeekCloser is only available in Go 1.16 Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* [refactor] replace int with httpStatusCodes (#15282)65432021-04-051-23/+24
| | | | | | | | | | | * replace "200" (int) with "http.StatusOK" (const) * ctx.Error & ctx.HTML * ctx.JSON Part1 * ctx.JSON Part2 * ctx.JSON Part3
* Add size to Save function (#15264)zeripath2021-04-031-1/+1
| | | | | Fix #15253 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Close file on invalid range (#15166)KN4CK3R2021-04-031-0/+8
| | | | | | | | | * Close file on invalid range. * Close on seek error Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Fix race in LFS ContentStore.Put(...) (#14895)zeripath2021-03-061-11/+51
| | | | | | | | | | | Continuing on from #14888 The previous implementation has race whereby an incomplete upload or hash mismatch upload can end up in the ContentStore. This PR moves the validation into the reader so that if there is a hash error or size mismatch the reader will return with an error instead of an io.EOF causing the storage to abort the storage. Signed-off-by: Andrew Thornton <art27@cantab.net>
generated by cgit v1.2.3 (git 2.39.1) at 2025-03-21 04:11:07 +0000