summaryrefslogtreecommitdiffstats
path: root/services/repository/files
Commit message (Collapse)AuthorAgeFilesLines
* Make sure git version&feature are always prepared (#30877) (#30879)wxiaoguang2024-05-071-1/+1
| | | Backport #30877
* Enable more `revive` linter rules (#30608)silverwind2024-04-221-2/+0
| | | | | | | | | | | Noteable additions: - `redefines-builtin-id` forbid variable names that shadow go builtins - `empty-lines` remove unnecessary empty lines that `gofumpt` does not remove for some reason - `superfluous-else` eliminate more superfluous `else` branches Rules are also sorted alphabetically and I cleaned up various parts of `.golangci.yml`.
* Use correct hash for "git update-index" (#30626)wxiaoguang2024-04-221-5/+8
|
* Clean up log messages (#30313)wxiaoguang2024-04-073-3/+3
| | | | | | `log.Xxx("%v")` is not ideal, this PR adds necessary context messages. Remove some unnecessary logs. Co-authored-by: Giteabot <teabot@gitea.io>
* Add download URL for executable files (#28260)Earl Warren2024-03-061-1/+1
| | | | | | Consider executable files as a valid case when returning a DownloadURL for them. They are just regular files with the difference being the executable permission bit being set. Co-authored-by: Gusted <postmaster@gusted.xyz>
* Cache repository default branch commit status to reduce query on commit ↵Lunny Xiao2024-03-061-48/+0
| | | | | | | | | | | | | status table (#29444) After repository commit status has been introduced on dashaboard, the most top SQL comes from `GetLatestCommitStatusForPairs`. This PR adds a cache for the repository's default branch's latest combined commit status. When a new commit status updated, the cache will be marked as invalid. <img width="998" alt="image" src="https://github.com/go-gitea/gitea/assets/81045/76759de7-3a83-4d54-8571-278f5422aed3">
* Move context from modules to services (#29440)Lunny Xiao2024-02-274-4/+4
| | | | | | | | | | | | | | | Since `modules/context` has to depend on `models` and many other packages, it should be moved from `modules/context` to `services/context` according to design principles. There is no logic code change on this PR, only move packages. - Move `code.gitea.io/gitea/modules/context` to `code.gitea.io/gitea/services/context` - Move `code.gitea.io/gitea/modules/contexttest` to `code.gitea.io/gitea/services/contexttest` because of depending on context - Move `code.gitea.io/gitea/modules/upload` to `code.gitea.io/gitea/services/context/upload` because of depending on context
* Refactor git attributes (#29356)KN4CK3R2024-02-241-9/+3
|
* Use the database object format name but not read from git repoisitory ↵Lunny Xiao2024-02-242-5/+3
| | | | | | | | | | | | | everytime and fix possible migration wrong objectformat when migrating a sha256 repository (#29294) Now we can get object format name from git command line or from the database repository table. Assume the column is right, we don't need to read from git command line every time. This also fixed a possible bug that the object format is wrong when migrating a sha256 repository from external. <img width="658" alt="image" src="https://github.com/go-gitea/gitea/assets/81045/6e9a9dcf-13bf-4267-928b-6bf2c2560423">
* Fix content size does not match error when uploading lfs file (#29259)vincent2024-02-191-1/+5
| | | | | | | | | | | | | | | | | | | | ![image](https://github.com/go-gitea/gitea/assets/38434877/cd726b4d-4771-4547-8aee-ae4e4b56b1d1) When we update an lfs file by API `api/v1/repos/{owner}/{repo}/contents/{filepath}`, there will show an error ```json { "message": "Put \"http://localhost:9000/gitea/lfs/38/92/05904d6c7bb83fc676513911226f2be25bf1465616bb9b29587100ab1414\": readfrom tcp [::1]:57300->[::1]:9000: content size does not match", "url": "http://localhost:3000/api/swagger" } ``` The reason of this error is https://github.com/go-gitea/gitea/blob/main/services/repository/files/update.go, in this file, the `file.ContentReader` been used twice. So when use `file.ContentReader` in the second time, the `i` of this Reader has been updated to the length of the content. it will return 0 and an `io.EOF` error when we try to read cotent from this Reader.
* Extract linguist code to method (#29168)KN4CK3R2024-02-141-0/+31
|
* Simplify how git repositories are opened (#28937)Lunny Xiao2024-01-276-10/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## Purpose This is a refactor toward building an abstraction over managing git repositories. Afterwards, it does not matter anymore if they are stored on the local disk or somewhere remote. ## What this PR changes We used `git.OpenRepository` everywhere previously. Now, we should split them into two distinct functions: Firstly, there are temporary repositories which do not change: ```go git.OpenRepository(ctx, diskPath) ``` Gitea managed repositories having a record in the database in the `repository` table are moved into the new package `gitrepo`: ```go gitrepo.OpenRepository(ctx, repo_model.Repo) ``` Why is `repo_model.Repository` the second parameter instead of file path? Because then we can easily adapt our repository storage strategy. The repositories can be stored locally, however, they could just as well be stored on a remote server. ## Further changes in other PRs - A Git Command wrapper on package `gitrepo` could be created. i.e. `NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir: repo.RepoPath()}`, the directory should be empty before invoking this method and it can be filled in the function only. #28940 - Remove the `RepoPath()`/`WikiPath()` functions to reduce the possibility of mistakes. --------- Co-authored-by: delvh <dev.lh@web.de>
* Fix reverting a merge commit failing (#28794)Mihir Joshi2024-01-166-7/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #22236 --- Error occurring currently while trying to revert commit using read-tree -m approach: > 2022/12/26 16:04:43 ...rvices/pull/patch.go:240:AttemptThreeWayMerge() [E] [63a9c61a] Unable to run read-tree -m! Error: exit status 128 - fatal: this operation must be run in a work tree > - fatal: this operation must be run in a work tree We need to clone a non-bare repository for `git read-tree -m` to work. https://github.com/go-gitea/gitea/commit/bb371aee6ecf5e570cdf7b5f7f0d6f47a607a325 adds support to create a non-bare cloned temporary upload repository. After cloning a non-bare temporary upload repository, we [set default index](https://github.com/go-gitea/gitea/blob/main/services/repository/files/cherry_pick.go#L37) (`git read-tree HEAD`). This operation ends up resetting the git index file (see investigation details below), due to which, we need to call `git update-index --refresh` afterward. Here's the diff of the index file before and after we execute SetDefaultIndex: https://www.diffchecker.com/hyOP3eJy/ Notice the **ctime**, **mtime** are set to 0 after SetDefaultIndex. You can reproduce the same behavior using these steps: ```bash $ git clone https://try.gitea.io/me-heer/test.git -s -b main $ cd test $ git read-tree HEAD $ git read-tree -m 1f085d7ed8 1f085d7ed8 9933caed00 error: Entry '1' not uptodate. Cannot merge. ``` After which, we can fix like this: ``` $ git update-index --refresh $ git read-tree -m 1f085d7ed8 1f085d7ed8 9933caed00 ```
* Adjust object format interface (#28469)Lunny Xiao2023-12-174-6/+6
| | | | | | | - Remove `ObjectFormatID` - Remove function `ObjectFormatFromID`. - Use `Sha1ObjectFormat` directly but not a pointer because it's an empty struct. - Store `ObjectFormatName` in `repository` struct
* Abstract hash function usage (#28138)Adam Majer2023-12-137-20/+27
| | | | | | Refactor Hash interfaces and centralize hash function. This will allow easier introduction of different hash function later on. This forms the "no-op" part of the SHA256 enablement patch.
* Remove GetByBean method because sometimes it's danger when query condition ↵Lunny Xiao2023-12-072-2/+2
| | | | | | | | | | | | | | | | | | | | parameter is zero and also introduce new generic methods (#28220) The function `GetByBean` has an obvious defect that when the fields are empty values, it will be ignored. Then users will get a wrong result which is possibly used to make a security problem. To avoid the possibility, this PR removed function `GetByBean` and all references. And some new generic functions have been introduced to be used. The recommand usage like below. ```go // if query an object according id obj, err := db.GetByID[Object](ctx, id) // query with other conditions obj, err := db.Get[Object](ctx, builder.Eq{"a": a, "b":b}) ```
* Upgrade to golangci-lint@v1.55.0 (#27756)silverwind2023-10-241-2/+1
| | | https://github.com/golangci/golangci-lint/releases/tag/v1.55.0
* Even more `db.DefaultContext` refactor (#27352)JakobDev2023-10-031-1/+1
| | | | | | | | Part of #27065 --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: delvh <dev.lh@web.de>
* make writing main test easier (#27270)Lunny Xiao2023-09-281-4/+1
| | | | | | | | | This PR removed `unittest.MainTest` the second parameter `TestOptions.GiteaRoot`. Now it detects the root directory by current working directory. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Updates to the API for archived repos (#27149)JakobDev2023-09-212-0/+10
|
* More refactoring of `db.DefaultContext` (#27083)JakobDev2023-09-151-7/+6
| | | Next step of #27065
* move repository deletion to service layer (#26948)Lunny Xiao2023-09-081-0/+2
| | | Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move web/api context related testing function into a separate package (#26859)wxiaoguang2023-09-014-58/+58
| | | | | | | | | Just like `models/unittest`, the testing helper functions should be in a separate package: `contexttest` And complete the TODO: > // TODO: move this function to other packages, because it depends on "models" package
* Fix some slice append usages (#26778)Chongyi Zheng2023-08-291-3/+3
| | | Co-authored-by: delvh <dev.lh@web.de>
* Sync repo's IsEmpty status correctly (#26517)wxiaoguang2023-08-171-1/+3
| | | Close #26509
* Refactor "Content" for file uploading (#25851)wxiaoguang2023-07-181-11/+12
| | | | | | | | | | | | | | | Before: the concept "Content string" is used everywhere. It has some problems: 1. Sometimes it means "base64 encoded content", sometimes it means "raw binary content" 2. It doesn't work with large files, eg: uploading a 1G LFS file would make Gitea process OOM This PR does the refactoring: use "ContentReader" / "ContentBase64" instead of "Content" This PR is not breaking because the key in API JSON is still "content": `` ContentBase64 string `json:"content"` ``
* Do not "guess" the file encoding/BOM when using API to upload files (#25828)wxiaoguang2023-07-121-102/+3
| | | | | | | | Related issue: #18368 It doesn't seem right to "guess" the file encoding/BOM when using API to upload files. The API should save the uploaded content as-is.
* Replace `interface{}` with `any` (#25686)silverwind2023-07-041-2/+2
| | | | | Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`. Basically the same [as golang did](https://github.com/golang/go/commit/2580d0e08d5e9f979b943758d3c49877fb2324cb).
* Sync branches into databases (#22743)Lunny Xiao2023-06-292-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Related #14180 Related #25233 Related #22639 Close #19786 Related #12763 This PR will change all the branches retrieve method from reading git data to read database to reduce git read operations. - [x] Sync git branches information into database when push git data - [x] Create a new table `Branch`, merge some columns of `DeletedBranch` into `Branch` table and drop the table `DeletedBranch`. - [x] Read `Branch` table when visit `code` -> `branch` page - [x] Read `Branch` table when list branch names in `code` page dropdown - [x] Read `Branch` table when list git ref compare page - [x] Provide a button in admin page to manually sync all branches. - [x] Sync branches if repository is not empty but database branches are empty when visiting pages with branches list - [x] Use `commit_time desc` as the default FindBranch order by to keep consistent as before and deleted branches will be always at the end. --------- Co-authored-by: Jason Song <i@wolfogre.com>
* Refactor web package and context package (#25298)wxiaoguang2023-06-184-11/+11
| | | | | | | | | | | | | 1. The "web" package shouldn't depends on "modules/context" package, instead, let each "web context" register themselves to the "web" package. 2. The old Init/Free doesn't make sense, so simplify it * The ctx in "Init(ctx)" is never used, and shouldn't be used that way * The "Free" is never called and shouldn't be called because the SSPI instance is shared --------- Co-authored-by: Giteabot <teabot@gitea.io>
* Fix profile render when the README.md size is larger than 1024 bytes (#25131)yp053272023-06-131-1/+1
| | | | | | | | | | | | | | | | | Fixes https://github.com/go-gitea/gitea/issues/25094 `GetBlobContent` will only get the first 1024 bytes, if the README.md size is larger than 1024 bytes, We can not render the rest of them. After this fix, we should provide the limited size to read when call `GetBlobContent`. After: ![image](https://github.com/go-gitea/gitea/assets/18380374/22a42936-4cf8-40b4-a5c7-e384082beb0d) --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Update repo's default branch when adding new files in an empty one (#25017)wxiaoguang2023-05-311-8/+8
| | | | | | Fix #25014 Only API needs this fix. On the Web UI, users could only add new file on the default branch.
* API endpoint for changing/creating/deleting multiple files (#24887)Denys Konovalov2023-05-293-368/+279
| | | | | | | | | | | This PR creates an API endpoint for creating/updating/deleting multiple files in one API call similar to the solution provided by [GitLab](https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions). To archive this, the CreateOrUpdateRepoFile and DeleteRepoFIle functions in files service are unified into one function supporting multiple files and actions. Resolves #14619
* Support uploading file to empty repo by API (#24357)wxiaoguang2023-04-261-0/+5
| | | | | | The uploading API already works (the only nit is the the IsEmpty flag is out-of-sync, this PR also fixes it) Close #14633
* Allow adding new files to an empty repo (#24164)wxiaoguang2023-04-191-4/+15
| | | ![image](https://user-images.githubusercontent.com/2114189/232561612-2bfcfd0a-fc04-47ba-965f-5d0bcea46c54.png)
* Introduce path Clean/Join helper functions (#23495)wxiaoguang2023-03-211-1/+1
| | | | | | | | | | | | | | | Since #23493 has conflicts with latest commits, this PR is my proposal for fixing #23371 Details are in the comments And refactor the `modules/options` module, to make it always use "filepath" to access local files. Benefits: * No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))), "/")` any more (not only one before) * The function behaviors are clearly defined
* fix submodule is nil panic (#23588)来自村里的小螃蟹2023-03-201-1/+3
| | | | | | | | | | #23587 submodule path is nil It is panic a nil error --------- Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* Use CleanPath instead of path.Clean (#23371)Lunny Xiao2023-03-081-2/+2
| | | As title.
* Return empty url for submodule tree entries (#23043)Zettat1232023-02-211-0/+5
| | | | | | | | | | | | Close #22614. Refer to [Github's API](https://docs.github.com/en/rest/git/trees?apiVersion=2022-11-28#get-a-tree), if a tree entry is a submodule, its url will be an empty string. --------- Co-authored-by: delvh <leon@kske.dev> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Add context cache as a request level cache (#22294)Lunny Xiao2023-02-154-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To avoid duplicated load of the same data in an HTTP request, we can set a context cache to do that. i.e. Some pages may load a user from a database with the same id in different areas on the same page. But the code is hidden in two different deep logic. How should we share the user? As a result of this PR, now if both entry functions accept `context.Context` as the first parameter and we just need to refactor `GetUserByID` to reuse the user from the context cache. Then it will not be loaded twice on an HTTP request. But of course, sometimes we would like to reload an object from the database, that's why `RemoveContextData` is also exposed. The core context cache is here. It defines a new context ```go type cacheContext struct { ctx context.Context data map[any]map[any]any lock sync.RWMutex } var cacheContextKey = struct{}{} func WithCacheContext(ctx context.Context) context.Context { return context.WithValue(ctx, cacheContextKey, &cacheContext{ ctx: ctx, data: make(map[any]map[any]any), }) } ``` Then you can use the below 4 methods to read/write/del the data within the same context. ```go func GetContextData(ctx context.Context, tp, key any) any func SetContextData(ctx context.Context, tp, key, value any) func RemoveContextData(ctx context.Context, tp, key any) func GetWithContextCache[T any](ctx context.Context, cacheGroupKey string, cacheTargetID any, f func() (T, error)) (T, error) ``` Then let's take a look at how `system.GetString` implement it. ```go func GetSetting(ctx context.Context, key string) (string, error) { return cache.GetWithContextCache(ctx, contextCacheKey, key, func() (string, error) { return cache.GetString(genSettingCacheKey(key), func() (string, error) { res, err := GetSettingNoCache(ctx, key) if err != nil { return "", err } return res.SettingValue, nil }) }) } ``` First, it will check if context data include the setting object with the key. If not, it will query from the global cache which may be memory or a Redis cache. If not, it will get the object from the database. In the end, if the object gets from the global cache or database, it will be set into the context cache. An object stored in the context cache will only be destroyed after the context disappeared.
* escape filename when assemble URL (#22850)sillyguodong2023-02-121-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes: #22843 ### Cause: https://github.com/go-gitea/gitea/blob/affdd40296960a08a4223330ccbd1fb88c96ea1a/services/repository/files/content.go#L161 Previously, we did not escape the **"%"** that might be in "treePath" when call "url.parse()". ![image](https://user-images.githubusercontent.com/33891828/218066318-5a909e50-2a17-46e6-b32f-684b2aa4b91f.png) This function will check whether "%" is the beginning of an escape character. Obviously, the "%" in the example (hello%mother.txt) is not that. So, the function will return a error. ### Solution: We can escape "treePath" by call "url.PathEscape()" function firstly. ### Screenshot: ![image](https://user-images.githubusercontent.com/33891828/218069781-1a030f8b-18d0-4804-b0f8-73997849ef43.png) --------- Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Andrew Thornton <art27@cantab.net>
* Refactor git command package to improve security and maintainability (#22678)wxiaoguang2023-02-044-15/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR follows #21535 (and replace #22592) ## Review without space diff https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1 ## Purpose of this PR 1. Make git module command completely safe (risky user inputs won't be passed as argument option anymore) 2. Avoid low-level mistakes like https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918 3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg` type 4. Simplify code when using git command ## The main idea of this PR * Move the `git.CmdArg` to the `internal` package, then no other package except `git` could use it. Then developers could never do `AddArguments(git.CmdArg(userInput))` any more. * Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already trusted arguments. It's only used in a few cases, for example: use git arguments from config file, help unit test with some arguments. * Introduce `AddOptionValues` and `AddOptionFormat`, they make code more clear and simple: * Before: `AddArguments("-m").AddDynamicArguments(message)` * After: `AddOptionValues("-m", message)` * - * Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'", sig.Name, sig.Email)))` * After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)` ## FAQ ### Why these changes were not done in #21535 ? #21535 is mainly a search&replace, it did its best to not change too much logic. Making the framework better needs a lot of changes, so this separate PR is needed as the second step. ### The naming of `AddOptionXxx` According to git's manual, the `--xxx` part is called `option`. ### How can it guarantee that `internal.CmdArg` won't be not misused? Go's specification guarantees that. Trying to access other package's internal package causes compilation error. And, `golangci-lint` also denies the git/internal package. Only the `git/command.go` can use it carefully. ### There is still a `ToTrustedCmdArgs`, will it still allow developers to make mistakes and pass untrusted arguments? Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code will be very complex (see the changes for examples). Then developers and reviewers can know that something might be unreasonable. ### Why there was a `CmdArgCheck` and why it's removed? At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck` was introduced as a hacky patch. Now, almost all code could be written as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for `CmdArgCheck` anymore. ### Why many codes for `signArg == ""` is deleted? Because in the old code, `signArg` could never be empty string, it's either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just dead code. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Supports wildcard protected branch (#20825)Lunny Xiao2023-01-162-6/+10
| | | | | | | | | | | | | | | | | This PR introduce glob match for protected branch name. The separator is `/` and you can use `*` matching non-separator chars and use `**` across separator. It also supports input an exist or non-exist branch name as matching condition and branch name condition has high priority than glob rule. Should fix #2529 and #15705 screenshots <img width="1160" alt="image" src="https://user-images.githubusercontent.com/81045/205651179-ebb5492a-4ade-4bb4-a13c-965e8c927063.png"> Co-authored-by: zeripath <art27@cantab.net>
* Use context parameter in models/git (#22367)Jason Song2023-01-094-9/+11
| | | | | | | | | After #22362, we can feel free to use transactions without `db.DefaultContext`. And there are still lots of models using `db.DefaultContext`, I think we should refactor them carefully and one by one. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Use complete SHA to create and query commit status (#22244)Jason Song2022-12-272-2/+5
| | | | | | | Fix #13485. Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* refactor some functions to support ctx as first parameter (#21878)Lunny Xiao2022-12-031-1/+1
| | | | Co-authored-by: KN4CK3R <admin@oldschoolhack.me> Co-authored-by: Lauris BH <lauris@nix.lv>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-2715-30/+15
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)delvh2022-10-247-25/+25
| | | | | | | | | Found using `find . -type f -name '*.go' -print -exec vim {} -c ':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;` Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Andrew Thornton <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor git command arguments and make all arguments to be safe to be used ↵wxiaoguang2022-10-234-18/+11
| | | | | | | (#21535) Follow #21464 Make all git command arguments strictly safe. Most changes are one-to-one replacing, keep all existing logic.
* Refactor parseTreeEntries, speed up tree list (#21368)wxiaoguang2022-10-081-1/+1
| | | | | Close #20315 (fix the panic when parsing invalid input), Speed up #20231 (use ls-tree without size field) Introduce ListEntriesRecursiveFast (ls-tree without size) and ListEntriesRecursiveWithSize (ls-tree with size)
generated by cgit v1.2.3 (git 2.39.1) at 2025-04-15 02:28:56 +0000