Handle more pathological branch and tag names (#11843) (#11863)
Backport #11843
It's possible to push quite pathological appearing branch names to gitea
using git push gitea reasonable-branch:refs/heads/-- at which point
large parts of the UI will break. Similarly you can git push origin
reasonable-tag:refs/tags/-- which wil return an error.
This PR fixes the problems these cause. It also changes the code from
creating branches to pushing to ensure that branch restoration has to
pass hooks.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Change language statistics to save size instead of percentage (#11681) (#11690)
* Change language statistics to save size instead of percentage (#11681)
* Change language statistics to save size instead of percentage in database
Co-Authored-By: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Do not exclude if only language
* Fix edge cases with special langauges
Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Fix language stat calculation (#11692)
* Fix language stat calculation
* Group languages and ignore 0 size files
* remove unneeded code
Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Refactor Cron and merge dashboard tasks
* Merge Cron and Dashboard tasks
* Make every cron task report a system notice on completion
* Refactor the creation of these tasks
* Ensure that execution counts of tasks is correct
* Allow cron tasks to be started from the cron page
* golangci-lint fixes
* Enforce that only one task with the same name can be registered
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix name check
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @guillep2k
* as per @lafriks
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add git.CommandContext variants
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Properly generate ref URLs
Tags used to not generate correct URLs (src/branch/tags/1.0.0 instead of
src/tags/1.0.0).
Also cleans up some code around it with the created helper functions.
* Fix formatting and create migration
* Add copyright head to utils_test
* Use a raw query for the ref migration
* Remove semicolon
* Quote column and table names in migration SQL
* Change || to CONCAT, since MSSQL does not support ||
* Make migration engine aware
* Add missing import
* Move ref EndName and URL to the issue service
* Fix tests
* Add test for commit refs
* Update issue.go
* Use the right command for building JavaScript bundles
* Prepare for merge
* Check for refs/* before prepending in migration
* Update services/issue/issue_test.go
* Update modules/git/utils_test.go
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: techknowlogick <matti@mdranta.net>
Allow compare page to look up base, head, own-fork, forkbase-of-head (#11327)
* Allow compare page to look up base, head, own-fork, forkbase-of-head
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @guillep2k
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update routers/repo/compare.go
* as per @guillep2k
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Rationalise the names a little
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Rationalise the names a little (2)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix 500 with fork of fork
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent 500 on compare different trees
Signed-off-by: Andrew Thornton <art27@cantab.net>
* dotdotdot is perfectly valid in both usernames and repo names
Signed-off-by: Andrew Thornton <art27@cantab.net>
* ensure we can set the head and base repos too
Signed-off-by: Andrew Thornton <art27@cantab.net>
* ensure we can set the head and base repos too (2)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* only set headRepo == baseRepo if isSameRepo
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Manage port in submodule refurl
Fix #11304
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* URLJoin is causes a cyclic dependency and possibly isn't what what we want anyway
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Protect against leading .. in scp syntax
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Support unicode emojis and remove emojify.js (#11032)
* Support unicode emojis and remove emojify.js
This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.
This works in a few ways:
First it adds emoji parsing support into gitea itself. This allows us to
* Render emojis from valid alias (:smile:)
* Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
* Easily allow for custom "emoji"
* Support all emoji rendering and features without javascript
* Uses plain unicode and lets the system render in appropriate emoji font
* Doesn't leave us relying on external sources for updates/fixes/features
That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)
For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.
The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.
I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.
I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.
Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.
Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130
* add new shared function emojiHTML
* don't increase emoji size in issue title
* Update templates/repo/issue/view_content/add_reaction.tmpl
Co-Authored-By: 6543 <6543@obermui.de>
* Support for emoji rendering in various templates
* Render code and review comments as they should be
* Better way to handle mail subjects
* insert unicode from tribute selection
* Add template helper for plain text when needed
* Use existing replace function I forgot about
* Don't include emoji greater than Unicode Version 12
Only include emoji and aliases in JSON
* Update build/generate-emoji.go
* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have
* final updates
* code review
* code review
* hard code gitea custom emoji to match previous behavior
* Update .eslintrc
Co-Authored-By: silverwind <me@silverwind.io>
* disable preempt
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Disable new signal-based asynchronous goroutine preemption from GO 1.14 in git env (#11237)
As seen in trouble shooting #11032 the new feature of Go 1.14 is causing several second delays in startup in certain situations. Debugging shows it spending several seconds handling SIGURG commands during init:
```
6922:04:51.984234 trace init() ./modules/queue/unique_queue_wrapped.go
remote: ) = 69 <0.000012>
remote: [pid 15984] 22:04:51 write(1, "\ttime taken: 236.761\302\265s\n\n", 25 time taken: 236.761µs
remote:
remote: ) = 25 <0.000011>
remote: [pid 15984] 22:04:51 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
remote: [pid 15984] 22:04:52 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
remote: [pid 15984] 22:04:52 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
```
This causes up to 20 seconds added to a push in some cases as it happens for each call of the gitea hook command. This is likely the cause of #10661 as well and would start to effect users once we release 1.12 which would be the first release compiled with Go 1.14. I suspect this is just a slight issue with the upstream implementatation as there have been a few very similar bugs fixed and reported:
https://github.com/golang/go/issues/37741https://github.com/golang/go/issues/37942
We should revisit this in the future and see if a newer version of Go has solved it, but for now disable this option in the environment that gitea hook runs in to avoid it.
Handle push rejection in branch and upload (#10854)
* Handle push rejections and push out-of-date in branch creation and
file upload.
* Remove the duplicated sanitize from services/pull/merge
* Move the errors Err(Merge)PushOutOfDate and ErrPushRejected to
modules/git
* Handle errors better in the upload file dialogs
Fix #10460
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Fix panic in API pulls when headbranch does not exist (#10676)
* Fix panic in API pulls when headbranch does not exist
* refix other reference to plumbing.ErrReferenceNotFound
Signed-off-by: Andrew Thornton <art27@cantab.net>
Code Refactor of IssueWatch related things (#10401)
* refactor
* optimize
* remove Iretating function
LoadWatchUsers do not load Users into IW object and it is used only in api ... so move this logic
* remove unessesary
* Apply suggestions from code review
Thx
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* make Tests more robust
* fix rebase
* restart CI
* CI no dont hit sqlites deadlock
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Add detected file language to code search (#10256)
Move langauge detection to separate module to be more reusable
Add option to disable vendored file exclusion from file search
Allways show all language stats for search
* Implementation for calculating language statistics
Impement saving code language statistics to database
Implement rendering langauge stats
Add primary laguage to show in repository list
Implement repository stats indexer queue
Add indexer test
Refactor to use queue module
* Do not timeout for queues
Cache last commit to accelerate the repository directory page visit (#10069)
* Cache last commit to accelerate the repository directory page visit
* Default use default cache configuration
* add tests for last commit cache
* Simplify last commit cache
* Revert Enabled back
* Change the last commit cache default ttl to 8760h
* Fix test
Some refactor on git diff and ignore getting commit information failed on migrating pull request review comments (#9996)
* Some refactor on git diff and ignore getting commit information failed on migrating pull request review comments
* fix test
* fix lint
* Change error log to warn
Add require signed commit for protected branch (#9708)
* Add require signed commit for protected branch
* Fix fmt
* Make editor show if they will be signed
* bugfix
* Add basic merge check and better information for CRUD
* linting comment
* Add descriptors to merge signing
* Slight refactor
* Slight improvement to appearances
* Handle Merge API
* manage CRUD API
* Move error to error.go
* Remove fix to delete.go
* prep for merge
* need to tolerate \r\n in message
* check protected branch before trying to load it
* Apply suggestions from code review
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* fix commit-reader
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Mark PR reviews as stale at push and allow to dismiss stale approvals (#9532)
Fix #5997.
If a push causes the patch/diff of a PR towards target branch to change, all existing reviews for the PR will be set and shown as stale.
New branch protection option to dismiss stale approvals are added.
To show that a review is not based on the latest PR changes, an hourglass is shown
* Adds functionality to change target branch of created pull requests
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Use const instead of var in JavaScript additions
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Check if branches are equal and if PR already exists before changing target branch
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Make sure to check all commits
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Print error messages for user as error flash message
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Disallow changing target branch of closed or merged pull requests
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Resolve conflicts after merge of upstream/master
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Change order of branch select fields
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Removes duplicate check
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Use ctx.Tr for translations
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Recompile JS
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Use correct translation namespace
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Remove redundant if condition
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Moves most change branch logic into pull service
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Completes comment
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Add Ref to ChangesPayload for logging changed target branches
instead of creating a new struct
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Revert changes to go.mod
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Directly use createComment method
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Return 404 if pull request is not found. Move written check up
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Remove variable declaration
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Return client errors on change pull request target errors
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Return error in commit.HasPreviousCommit
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Adds blank line
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Test patch before persisting new target branch
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Update patch before testing (not working)
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Removes patch calls when changeing pull request target
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Removes unneeded check for base name
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Moves ChangeTargetBranch completely to pull service. Update patch status.
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Set webhook mode after errors were validated
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Update PR in one transaction
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Move logic for check if head is equal with branch to pull model
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Adds missing comment and simplify return
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
* Adjust CreateComment method call
Signed-off-by: Mario Lubenka <mario.lubenka@googlemail.com>
Graceful: Xorm, RepoIndexer, Cron and Others (#9282)
* Change graceful to use a singleton obtained through GetManager instead of a global.
* Graceful: Make TestPullRequests shutdownable
* Graceful: Make the cron tasks graceful
* Graceful: AddTestPullRequest run in graceful ctx
* Graceful: SyncMirrors shutdown
* Graceful: SetDefaultContext for Xorm to be HammerContext
* Avoid starting graceful for migrate commands and checkout
* Graceful: DeliverHooks now can be shutdown
* Fix multiple syncing errors in modules/sync/UniqueQueue & Make UniqueQueue closable
* Begin the process of making the repo indexer shutdown gracefully
Remove SavePatch and generate patches on the fly (#9302)
* Save patches to temporary files
* Remove SavePatch and generate patches on the fly
* Use ioutil.TempDir
* fixup! Use ioutil.TempDir
* fixup! fixup! Use ioutil.TempDir
* RemoveAll LocalCopyPath() in initIntergrationTest
* Default to status checking on PR creation
* Remove unnecessary set to StatusChecking
* Protect against unable to load repo
* Handle conflicts
* Restore original conflict setting
* In TestPullRequests update status to StatusChecking before running TestPatch
The underlying implementation of os.exec uses channels and goroutines.
It is possible to have time-variant error values returned from Cmd.Wait
depending on which comes first.
Also, the git subcommand and options should be separated tokens.
Fixes a flaky test in modules/git/command_test.go
Graceful: Cancel Process on monitor pages & HammerTime (#9213)
* Graceful: Create callbacks to with contexts
* Graceful: Say when Gitea is completely finished
* Graceful: Git and Process within HammerTime
Force all git commands to terminate at HammerTime
Force all process commands to terminate at HammerTime
Move almost all git processes to run as git Commands
* Graceful: Always Hammer after Shutdown
* ProcessManager: Add cancel functionality
* Fix tests
* Make sure that process.Manager.Kill() cancels
* Make threadsafe access to Processes and remove own unused Kill
* Remove cmd from the process manager as it is no longer used
* the default context is the correct context
* get rid of double till
In investigating #7947 it has become clear that the storage component of go-git repositories needs closing.
This PR adds this Close function and adds the Close functions as necessary.
In TransferOwnership the ctx.Repo.GitRepo is closed if it is open to help prevent the risk of multiple open files.
Fixes #7947
Migrate temp_repo.go to use git.NewCommand (#8918)
This PR migrates temp_repo.go to use git.NewCommand instead creating processes by itself - this fixes the problem underlying PR #8905.
There are other places that run git outside of the controlled locale defined in #8548 but temp_repo.go is the only cause of failure of local testing in cases where English is not the default - implying that error messages from those other commands are not interpreted.
Replaces #8905
Adjust error reporting from merge failures and use LC_ALL=C for git (#8548)
There are two major components to this PR:
* This PR handles merge and rebase failures from merging a little more nicely with Flash errors rather a 500.
* All git commands are run in the LC_ALL="C" environment to ensure that error messages are in English. This DefaultLocale is defined in a way that if necessary (due to platform weirdness) it can be overridden at build time using LDFLAGS="-X "code.gitea.io/gitea/modules/git.DefaultLocale=C"" with C changed for the locale as necessary.
Lauris BH
zeripath
Sijmen Schoon
mrsdizzie
6543
John Olheiser
Lunny Xiao
SpaWn2KiLl
David Svantesson
Mario Lubenka
Mura Li
John Olheiser
Benno
jaqra