12 Commits (eab20cb6bd7bcbc1c3fffd5a41f1d1029ed30e27)

Author	SHA1	Message	Date
wxiaoguang	6bc3079c00	Refactor git command package to improve security and maintainability (#22678) ... This PR follows #21535 (and replace #22592) ## Review without space diff https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1 ## Purpose of this PR 1. Make git module command completely safe (risky user inputs won't be passed as argument option anymore) 2. Avoid low-level mistakes like https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918 3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg` type 4. Simplify code when using git command ## The main idea of this PR * Move the `git.CmdArg` to the `internal` package, then no other package except `git` could use it. Then developers could never do `AddArguments(git.CmdArg(userInput))` any more. * Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already trusted arguments. It's only used in a few cases, for example: use git arguments from config file, help unit test with some arguments. * Introduce `AddOptionValues` and `AddOptionFormat`, they make code more clear and simple: * Before: `AddArguments("-m").AddDynamicArguments(message)` * After: `AddOptionValues("-m", message)` * - * Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'", sig.Name, sig.Email)))` * After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)` ## FAQ ### Why these changes were not done in #21535 ? #21535 is mainly a search&replace, it did its best to not change too much logic. Making the framework better needs a lot of changes, so this separate PR is needed as the second step. ### The naming of `AddOptionXxx` According to git's manual, the `--xxx` part is called `option`. ### How can it guarantee that `internal.CmdArg` won't be not misused? Go's specification guarantees that. Trying to access other package's internal package causes compilation error. And, `golangci-lint` also denies the git/internal package. Only the `git/command.go` can use it carefully. ### There is still a `ToTrustedCmdArgs`, will it still allow developers to make mistakes and pass untrusted arguments? Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code will be very complex (see the changes for examples). Then developers and reviewers can know that something might be unreasonable. ### Why there was a `CmdArgCheck` and why it's removed? At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck` was introduced as a hacky patch. Now, almost all code could be written as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for `CmdArgCheck` anymore. ### Why many codes for `signArg == ""` is deleted? Because in the old code, `signArg` could never be empty string, it's either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just dead code. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>	2 years ago
flynnnnnnnnnn	e81ccc406b	Implement FSFE REUSE for golang files (#21840) ... Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>	2 years ago
kolaente	59b30f060a	Auto merge pull requests when all checks succeeded via API (#9307) ... * Fix indention Signed-off-by: kolaente <k@knt.li> * Add option to merge a pr right now without waiting for the checks to succeed Signed-off-by: kolaente <k@knt.li> * Fix lint Signed-off-by: kolaente <k@knt.li> * Add scheduled pr merge to tables used for testing Signed-off-by: kolaente <k@knt.li> * Add status param to make GetPullRequestByHeadBranch reusable Signed-off-by: kolaente <k@knt.li> * Move "Merge now" to a seperate button to make the ui clearer Signed-off-by: kolaente <k@knt.li> * Update models/scheduled_pull_request_merge.go Co-authored-by: 赵智超 <1012112796@qq.com> * Update web_src/js/index.js Co-authored-by: 赵智超 <1012112796@qq.com> * Update web_src/js/index.js Co-authored-by: 赵智超 <1012112796@qq.com> * Re-add migration after merge * Fix frontend lint * Fix version compare * Add vendored dependencies * Add basic tets * Make sure the api route is capable of scheduling PRs for merging * Fix comparing version * make vendor * adopt refactor * apply suggestion: User -> Doer * init var once * Fix Test * Update templates/repo/issue/view_content/comments.tmpl * adopt * nits * next * code format * lint * use same name schema; rm CreateUnScheduledPRToAutoMergeComment * API: can not create schedule twice * Add TestGetBranchNamesForSha * nits * new go routine for each pull to merge * Update models/pull.go Co-authored-by: a1012112796 <1012112796@qq.com> * Update models/scheduled_pull_request_merge.go Co-authored-by: a1012112796 <1012112796@qq.com> * fix & add renaming sugestions * Update services/automerge/pull_auto_merge.go Co-authored-by: a1012112796 <1012112796@qq.com> * fix conflict relicts * apply latest refactors * fix: migration after merge * Update models/error.go Co-authored-by: delvh <dev.lh@web.de> * Update options/locale/locale_en-US.ini Co-authored-by: delvh <dev.lh@web.de> * Update options/locale/locale_en-US.ini Co-authored-by: delvh <dev.lh@web.de> * adapt latest refactors * fix test * use more context * skip potential edgecases * document func usage * GetBranchNamesForSha() -> GetRefsBySha() * start refactoring * ajust to new changes * nit * docu nit * the great check move * move checks for branchprotection into own package * resolve todo now ... * move & rename * unexport if posible * fix * check if merge is allowed before merge on scheduled pull * debugg * wording * improve SetDefaults & nits * NotAllowedToMerge -> DisallowedToMerge * fix test * merge files * use package "errors" * merge files * add string names * other implementation for gogit * adapt refactor * more context for models/pull.go * GetUserRepoPermission use context * more ctx * use context for loading pull head/base-repo * more ctx * more ctx * models.LoadIssueCtx() * models.LoadIssueCtx() * Handle pull_service.Merge in one DB transaction * add TODOs * next * next * next * more ctx * more ctx * Start refactoring structure of old pull code ... * move code into new packages * shorter names ... and finish **restructure** * Update models/branches.go Co-authored-by: zeripath <art27@cantab.net> * finish UpdateProtectBranch * more and fix * update datum * template: use "svg" helper * rename prQueue 2 prPatchCheckerQueue * handle automerge in queue * lock pull on git&db actions ... * lock pull on git&db actions ... * add TODO notes * the regex * transaction in tests * GetRepositoryByIDCtx * shorter table name and lint fix * close transaction bevore notify * Update models/pull.go * next * CheckPullMergable check all branch protections! * Update routers/web/repo/pull.go * CheckPullMergable check all branch protections! * Revert "PullService lock via pullID (#19520)" (for now...) This reverts commit 6cde7c9159a5ea75a10356feb7b8c7ad4c434a9a. * Update services/pull/check.go * Use for a repo action one database transaction * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: delvh <dev.lh@web.de> * Update services/issue/status.go Co-authored-by: delvh <dev.lh@web.de> * Update services/issue/status.go Co-authored-by: delvh <dev.lh@web.de> * use db.WithTx() * gofmt * make pr.GetDefaultMergeMessage() context aware * make MergePullRequestForm.SetDefaults context aware * use db.WithTx() * pull.SetMerged only with context * fix deadlock in `test-sqlite\#TestAPIBranchProtection` * dont forget templates * db.WithTx allow to set the parentCtx * handle db transaction in service packages but not router * issue_service.ChangeStatus just had caused another deadlock :/ it has to do something with how notification package is handled * if we merge a pull in one database transaktion, we get a lock, because merge infoce internal api that cant handle open db sessions to the same repo * ajust to current master * Apply suggestions from code review Co-authored-by: delvh <dev.lh@web.de> * dont open db transaction in router * make generate-swagger * one _success less * wording nit * rm * adapt * remove not needed test files * rm less diff & use attr in JS * ... * Update services/repository/files/commit.go Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> * ajust db schema for PullAutoMerge * skip broken pull refs * more context in error messages * remove webUI part for another pull * remove more WebUI only parts * API: add CancleAutoMergePR * Apply suggestions from code review Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> * fix lint * Apply suggestions from code review * cancle -> cancel Co-authored-by: delvh <dev.lh@web.de> * change queue identifyer * fix swagger * prevent nil issue * fix and dont drop error * as per @zeripath * Update integrations/git_test.go Co-authored-by: delvh <dev.lh@web.de> * Update integrations/git_test.go Co-authored-by: delvh <dev.lh@web.de> * more declarative integration tests (dedup code) * use assert.False/True helper Co-authored-by: 赵智超 <1012112796@qq.com> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>	3 years ago
wxiaoguang	c18d8d6968	Remove legacy `+build:` constraint (#19582) ... Go 1.17 and later use modern `//go:build` constraints, the old `// +build:` constraints should be removed.	3 years ago
6543	3e88af898a	Make git.OpenRepository accept Context (#19260) ... * OpenRepositoryCtx -> OpenRepository * OpenRepository -> openRepositoryWithDefaultContext, only for internal usage	3 years ago
zeripath	889a8c268c	Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) ... Strangely #19038 appears to relate to an issue whereby a tag appears to be listed in `git show-ref --tags` but then does not appear when `git show-ref --tags -- short_name` is called. As a solution though I propose to stop the second call as it is unnecessary and only likely to cause problems. I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added. I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. Fix #19038 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>	3 years ago
zeripath	5cb0c9aa0d	Propagate context and ensure git commands run in request context (#17868) ... This PR continues the work in #17125 by progressively ensuring that git commands run within the request context. This now means that the if there is a git repo already open in the context it will be used instead of reopening it. Signed-off-by: Andrew Thornton <art27@cantab.net>	3 years ago
zeripath	9e6e1dc950	Improve checkBranchName (#17901) ... The current implementation of checkBranchName is highly inefficient involving opening the repository, the listing all of the branch names checking them individually before then using using opened repo to get the tags. This PR avoids this by simply walking the references from show-ref instead of opening the repository (in the nogogit case). Signed-off-by: Andrew Thornton <art27@cantab.net>	3 years ago
John Olheiser	648464b504	Add bundle download for repository (#14538) ... * Add bundle download Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix fmt Signed-off-by: jolheiser <john.olheiser@gmail.com> * Fix build tags Signed-off-by: jolheiser <john.olheiser@gmail.com> * Download specific commit Signed-off-by: jolheiser <john.olheiser@gmail.com>	3 years ago
zeripath	196593e2e9	More efficiently parse shas for shaPostProcessor (#16101) ... * More efficiently parse shas for shaPostProcessor The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends which is fine if there is only one thing that matches a sha - however if there are multiple things then this becomes wildly inefficient. This PR provides functions for both backends which are much faster to use. Fix #16092 * Add ShaExistCache to RenderContext Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de>	3 years ago
6543	0d1444751f	[API] Add pagination to ListBranches (#14524) ... * make PaginateUserSlice generic -> PaginateSlice * Add pagination to ListBranches * add skip, limit to Repository.GetBranches() * Move routers/api/v1/utils/utils PaginateSlice -> modules/util/paginate.go * repo_module.GetBranches paginate * fix & rename & more logging * better description Co-authored-by: zeripath <art27@cantab.net> Co-authored-by: a1012112796 <1012112796@qq.com>	4 years ago
zeripath	511f6138d4	Use native git variants by default with go-git variants as build tag (#13673) ... * Move last commit cache back into modules/git Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove go-git from the interface for last commit cache Signed-off-by: Andrew Thornton <art27@cantab.net> * move cacheref to last_commit_cache Signed-off-by: Andrew Thornton <art27@cantab.net> * Remove go-git from routers/private/hook Signed-off-by: Andrew Thornton <art27@cantab.net> * Move FindLFSFiles to pipeline Signed-off-by: Andrew Thornton <art27@cantab.net> * Make no-go-git variants Signed-off-by: Andrew Thornton <art27@cantab.net> * Submodule RefID Signed-off-by: Andrew Thornton <art27@cantab.net> * fix issue with GetCommitsInfo Signed-off-by: Andrew Thornton <art27@cantab.net> * fix GetLastCommitForPaths Signed-off-by: Andrew Thornton <art27@cantab.net> * Improve efficiency Signed-off-by: Andrew Thornton <art27@cantab.net> * More efficiency Signed-off-by: Andrew Thornton <art27@cantab.net> * even faster Signed-off-by: Andrew Thornton <art27@cantab.net> * Reduce duplication * As per @lunny Signed-off-by: Andrew Thornton <art27@cantab.net> * attempt to fix drone Signed-off-by: Andrew Thornton <art27@cantab.net> * fix test-tags Signed-off-by: Andrew Thornton <art27@cantab.net> * default to use no-go-git variants and add gogit build tag Signed-off-by: Andrew Thornton <art27@cantab.net> * placate lint Signed-off-by: Andrew Thornton <art27@cantab.net> * as per @6543 Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: techknowlogick <techknowlogick@gitea.io>	4 years ago