Auto merge pull requests when all checks succeeded via API (#9307)

* Fix indention

Signed-off-by: kolaente <k@knt.li>

* Add option to merge a pr right now without waiting for the checks to succeed

Signed-off-by: kolaente <k@knt.li>

* Fix lint

Signed-off-by: kolaente <k@knt.li>

* Add scheduled pr merge to tables used for testing

Signed-off-by: kolaente <k@knt.li>

* Add status param to make GetPullRequestByHeadBranch reusable

Signed-off-by: kolaente <k@knt.li>

* Move "Merge now" to a seperate button to make the ui clearer

Signed-off-by: kolaente <k@knt.li>

* Update models/scheduled_pull_request_merge.go

Co-authored-by: 赵智超 <1012112796@qq.com>

* Update web_src/js/index.js

Co-authored-by: 赵智超 <1012112796@qq.com>

* Update web_src/js/index.js

Co-authored-by: 赵智超 <1012112796@qq.com>

* Re-add migration after merge

* Fix frontend lint

* Fix version compare

* Add vendored dependencies

* Add basic tets

* Make sure the api route is capable of scheduling PRs for merging

* Fix comparing version

* make vendor

* adopt refactor

* apply suggestion: User -> Doer

* init var once

* Fix Test

* Update templates/repo/issue/view_content/comments.tmpl

* adopt

* nits

* next

* code format

* lint

* use same name schema; rm CreateUnScheduledPRToAutoMergeComment

* API: can not create schedule twice

* Add TestGetBranchNamesForSha

* nits

* new go routine for each pull to merge

* Update models/pull.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* Update models/scheduled_pull_request_merge.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* fix & add renaming sugestions

* Update services/automerge/pull_auto_merge.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* fix conflict relicts

* apply latest refactors

* fix: migration after merge

* Update models/error.go

Co-authored-by: delvh <dev.lh@web.de>

* Update options/locale/locale_en-US.ini

Co-authored-by: delvh <dev.lh@web.de>

* Update options/locale/locale_en-US.ini

Co-authored-by: delvh <dev.lh@web.de>

* adapt latest refactors

* fix test

* use more context

* skip potential edgecases

* document func usage

* GetBranchNamesForSha() -> GetRefsBySha()

* start refactoring

* ajust to new changes

* nit

* docu nit

* the great check move

* move checks for branchprotection into own package

* resolve todo now ...

* move & rename

* unexport if posible

* fix

* check if merge is allowed before merge on scheduled pull

* debugg

* wording

* improve SetDefaults & nits

* NotAllowedToMerge -> DisallowedToMerge

* fix test

* merge files

* use package "errors"

* merge files

* add string names

* other implementation for gogit

* adapt refactor

* more context for models/pull.go

* GetUserRepoPermission use context

* more ctx

* use context for loading pull head/base-repo

* more ctx

* more ctx

* models.LoadIssueCtx()

* models.LoadIssueCtx()

* Handle pull_service.Merge in one DB transaction

* add TODOs

* next

* next

* next

* more ctx

* more ctx

* Start refactoring structure of old pull code ...

* move code into new packages

* shorter names ... and finish **restructure**

* Update models/branches.go

Co-authored-by: zeripath <art27@cantab.net>

* finish UpdateProtectBranch

* more and fix

* update datum

* template: use "svg" helper

* rename prQueue 2 prPatchCheckerQueue

* handle automerge in queue

* lock pull on git&db actions ...

* lock pull on git&db actions ...

* add TODO notes

* the regex

* transaction in tests

* GetRepositoryByIDCtx

* shorter table name and lint fix

* close transaction bevore notify

* Update models/pull.go

* next

* CheckPullMergable check all branch protections!

* Update routers/web/repo/pull.go

* CheckPullMergable check all branch protections!

* Revert "PullService lock via pullID (#19520)" (for now...)

This reverts commit 6cde7c9159a5ea75a10356feb7b8c7ad4c434a9a.

* Update services/pull/check.go

* Use for a repo action one database transaction

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* Update services/issue/status.go

Co-authored-by: delvh <dev.lh@web.de>

* Update services/issue/status.go

Co-authored-by: delvh <dev.lh@web.de>

* use db.WithTx()

* gofmt

* make pr.GetDefaultMergeMessage() context aware

* make MergePullRequestForm.SetDefaults context aware

* use db.WithTx()

* pull.SetMerged only with context

* fix deadlock in `test-sqlite\#TestAPIBranchProtection`

* dont forget templates

* db.WithTx allow to set the parentCtx

* handle db transaction in service packages but not router

* issue_service.ChangeStatus just had caused another deadlock :/
it has to do something with how notification package is handled

* if we merge a pull in one database transaktion, we get a lock, because merge infoce internal api that cant handle open db sessions to the same repo

* ajust to current master

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* dont open db transaction in router

* make generate-swagger

* one _success less

* wording nit

* rm

* adapt

* remove not needed test files

* rm less diff & use attr in JS

* ...

* Update services/repository/files/commit.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

* ajust db schema for PullAutoMerge

* skip broken pull refs

* more context in error messages

* remove webUI part for another pull

* remove more WebUI only parts

* API: add CancleAutoMergePR

* Apply suggestions from code review

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

* fix lint

* Apply suggestions from code review

* cancle -> cancel

Co-authored-by: delvh <dev.lh@web.de>

* change queue identifyer

* fix swagger

* prevent nil issue

* fix and dont drop error

* as per @zeripath

* Update integrations/git_test.go

Co-authored-by: delvh <dev.lh@web.de>

* Update integrations/git_test.go

Co-authored-by: delvh <dev.lh@web.de>

* more declarative integration tests (dedup code)

* use assert.False/True helper

Co-authored-by: 赵智超 <1012112796@qq.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

1 files changed, 241 insertions, 0 deletions

diff --git a/services/automerge/automerge.go b/services/automerge/automerge.go
new file mode 100644
index 0000000000..389546ed57
--- /dev/null
+++ b/services/automerge/automerge.go
@@ -0,0 +1,241 @@
+// Copyright 2021 Gitea. All rights reserved.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package automerge
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strconv"
+	"strings"
+
+	"code.gitea.io/gitea/models"
+	pull_model "code.gitea.io/gitea/models/pull"
+	repo_model "code.gitea.io/gitea/models/repo"
+	user_model "code.gitea.io/gitea/models/user"
+	"code.gitea.io/gitea/modules/git"
+	"code.gitea.io/gitea/modules/graceful"
+	"code.gitea.io/gitea/modules/log"
+	"code.gitea.io/gitea/modules/process"
+	"code.gitea.io/gitea/modules/queue"
+	pull_service "code.gitea.io/gitea/services/pull"
+)
+
+// prAutoMergeQueue represents a queue to handle update pull request tests
+var prAutoMergeQueue queue.UniqueQueue
+
+// Init runs the task queue to that handles auto merges
+func Init() error {
+	prAutoMergeQueue = queue.CreateUniqueQueue("pr_auto_merge", handle, "")
+	if prAutoMergeQueue == nil {
+		return fmt.Errorf("Unable to create pr_auto_merge Queue")
+	}
+	go graceful.GetManager().RunWithShutdownFns(prAutoMergeQueue.Run)
+	return nil
+}
+
+// handle passed PR IDs and test the PRs
+func handle(data ...queue.Data) []queue.Data {
+	for _, d := range data {
+		var id int64
+		var sha string
+		if _, err := fmt.Sscanf(d.(string), "%d_%s", &id, &sha); err != nil {
+			log.Error("could not parse data from pr_auto_merge queue (%v): %v", d, err)
+			continue
+		}
+		handlePull(id, sha)
+	}
+	return nil
+}
+
+func addToQueue(pr *models.PullRequest, sha string) {
+	if err := prAutoMergeQueue.PushFunc(fmt.Sprintf("%d_%s", pr.ID, sha), func() error {
+		log.Trace("Adding pullID: %d to the pull requests patch checking queue with sha %s", pr.ID, sha)
+		return nil
+	}); err != nil {
+		log.Error("Error adding pullID: %d to the pull requests patch checking queue %v", pr.ID, err)
+	}
+}
+
+// ScheduleAutoMerge if schedule is false and no error, pull can be merged directly
+func ScheduleAutoMerge(ctx context.Context, doer *user_model.User, pull *models.PullRequest, style repo_model.MergeStyle, message string) (scheduled bool, err error) {
+	lastCommitStatus, err := pull_service.GetPullRequestCommitStatusState(ctx, pull)
+	if err != nil {
+		return false, err
+	}
+
+	// we don't need to schedule
+	if lastCommitStatus.IsSuccess() {
+		return false, nil
+	}
+
+	return true, pull_model.ScheduleAutoMerge(ctx, doer, pull.ID, style, message)
+}
+
+// MergeScheduledPullRequest merges a previously scheduled pull request when all checks succeeded
+func MergeScheduledPullRequest(ctx context.Context, sha string, repo *repo_model.Repository) error {
+	pulls, err := getPullRequestsByHeadSHA(ctx, sha, repo, func(pr *models.PullRequest) bool {
+		return !pr.HasMerged && pr.CanAutoMerge()
+	})
+	if err != nil {
+		return err
+	}
+
+	for _, pr := range pulls {
+		addToQueue(pr, sha)
+	}
+
+	return nil
+}
+
+func getPullRequestsByHeadSHA(ctx context.Context, sha string, repo *repo_model.Repository, filter func(*models.PullRequest) bool) (map[int64]*models.PullRequest, error) {
+	gitRepo, err := git.OpenRepository(ctx, repo.RepoPath())
+	if err != nil {
+		return nil, err
+	}
+	defer gitRepo.Close()
+
+	refs, err := gitRepo.GetRefsBySha(sha, "")
+	if err != nil {
+		return nil, err
+	}
+
+	pulls := make(map[int64]*models.PullRequest)
+
+	for _, ref := range refs {
+		// Each pull branch starts with refs/pull/ we then go from there to find the index of the pr and then
+		// use that to get the pr.
+		if strings.HasPrefix(ref, git.PullPrefix) {
+			parts := strings.Split(ref[len(git.PullPrefix):], "/")
+
+			// e.g. 'refs/pull/1/head' would be []string{"1", "head"}
+			if len(parts) != 2 {
+				log.Error("getPullRequestsByHeadSHA found broken pull ref [%s] on repo [%-v]", ref, repo)
+				continue
+			}
+
+			prIndex, err := strconv.ParseInt(parts[0], 10, 64)
+			if err != nil {
+				log.Error("getPullRequestsByHeadSHA found broken pull ref [%s] on repo [%-v]", ref, repo)
+				continue
+			}
+
+			p, err := models.GetPullRequestByIndexCtx(ctx, repo.ID, prIndex)
+			if err != nil {
+				// If there is no pull request for this branch, we don't try to merge it.
+				if models.IsErrPullRequestNotExist(err) {
+					continue
+				}
+				return nil, err
+			}
+
+			if filter(p) {
+				pulls[p.ID] = p
+			}
+		}
+	}
+
+	return pulls, nil
+}
+
+func handlePull(pullID int64, sha string) {
+	ctx, _, finished := process.GetManager().AddContext(graceful.GetManager().HammerContext(),
+		fmt.Sprintf("Handle AutoMerge of pull[%d] with sha[%s]", pullID, sha))
+	defer finished()
+
+	pr, err := models.GetPullRequestByID(ctx, pullID)
+	if err != nil {
+		log.Error("GetPullRequestByID[%d]: %v", pullID, err)
+		return
+	}
+
+	// Check if there is a scheduled pr in the db
+	exists, scheduledPRM, err := pull_model.GetScheduledMergeByPullID(ctx, pr.ID)
+	if err != nil {
+		log.Error("pull[%d] GetScheduledMergeByPullID: %v", pr.ID, err)
+		return
+	}
+	if !exists {
+		return
+	}
+
+	// Get all checks for this pr
+	// We get the latest sha commit hash again to handle the case where the check of a previous push
+	// did not succeed or was not finished yet.
+
+	if err = pr.LoadHeadRepoCtx(ctx); err != nil {
+		log.Error("pull[%d] LoadHeadRepoCtx: %v", pr.ID, err)
+		return
+	}
+
+	headGitRepo, err := git.OpenRepository(ctx, pr.HeadRepo.RepoPath())
+	if err != nil {
+		log.Error("OpenRepository: %v", err)
+		return
+	}
+	defer headGitRepo.Close()
+
+	headBranchExist := headGitRepo.IsBranchExist(pr.HeadBranch)
+
+	if pr.HeadRepo == nil || !headBranchExist {
+		log.Warn("Head branch of auto merge pr does not exist [HeadRepoID: %d, Branch: %s, PR ID: %d]", pr.HeadRepoID, pr.HeadBranch, pr.ID)
+		return
+	}
+
+	// Check if all checks succeeded
+	pass, err := pull_service.IsPullCommitStatusPass(ctx, pr)
+	if err != nil {
+		log.Error("IsPullCommitStatusPass: %v", err)
+		return
+	}
+	if !pass {
+		log.Info("Scheduled auto merge pr has unsuccessful status checks [PullID: %d]", pr.ID)
+		return
+	}
+
+	// Merge if all checks succeeded
+	doer, err := user_model.GetUserByIDCtx(ctx, scheduledPRM.DoerID)
+	if err != nil {
+		log.Error("GetUserByIDCtx: %v", err)
+		return
+	}
+
+	perm, err := models.GetUserRepoPermission(ctx, pr.HeadRepo, doer)
+	if err != nil {
+		log.Error("GetUserRepoPermission: %v", err)
+		return
+	}
+
+	if err := pull_service.CheckPullMergable(ctx, doer, &perm, pr, false, false); err != nil {
+		if errors.Is(pull_service.ErrUserNotAllowedToMerge, err) {
+			log.Info("PR %d was scheduled to automerge by an unauthorized user", pr.ID)
+			return
+		}
+		log.Error("pull[%d] CheckPullMergable: %v", pr.ID, err)
+		return
+	}
+
+	var baseGitRepo *git.Repository
+	if pr.BaseRepoID == pr.HeadRepoID {
+		baseGitRepo = headGitRepo
+	} else {
+		if err = pr.LoadBaseRepoCtx(ctx); err != nil {
+			log.Error("LoadBaseRepoCtx: %v", err)
+			return
+		}
+
+		baseGitRepo, err = git.OpenRepository(ctx, pr.BaseRepo.RepoPath())
+		if err != nil {
+			log.Error("OpenRepository: %v", err)
+			return
+		}
+		defer baseGitRepo.Close()
+	}
+
+	if err := pull_service.Merge(pr, doer, baseGitRepo, scheduledPRM.MergeStyle, "", scheduledPRM.Message); err != nil {
+		log.Error("pull_service.Merge: %v", err)
+		return
+	}
+}