|
|
|
// Copyright 2017 The Gitea Authors. All rights reserved.
|
|
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
|
|
|
|
package test
|
|
|
|
|
|
|
|
import (
|
|
|
|
scontext "context"
|
|
|
|
"io"
|
|
|
|
"net/http"
|
|
|
|
"net/http/httptest"
|
|
|
|
"net/url"
|
|
|
|
"testing"
|
|
|
|
|
|
|
|
access_model "code.gitea.io/gitea/models/perm/access"
|
|
|
|
repo_model "code.gitea.io/gitea/models/repo"
|
|
|
|
"code.gitea.io/gitea/models/unittest"
|
|
|
|
user_model "code.gitea.io/gitea/models/user"
|
|
|
|
"code.gitea.io/gitea/modules/context"
|
|
|
|
"code.gitea.io/gitea/modules/git"
|
Make HTML template functions support context (#24056)
# Background
Golang template is not friendly for large projects, and Golang template
team is quite slow, related:
* `https://github.com/golang/go/issues/54450`
Without upstream support, we can also have our solution to make HTML
template functions support context.
It helps a lot, the above Golang template issue `#54450` explains a lot:
1. It makes `{{Locale.Tr}}` could be used in any template, without
passing unclear `(dict "root" . )` anymore.
2. More and more functions need `context`, like `avatar`, etc, we do not
need to do `(dict "Context" $.Context)` anymore.
3. Many request-related functions could be shared by parent&children
templates, like "user setting" / "system setting"
See the test `TestScopedTemplateSetFuncMap`, one template set, two
`Execute` calls with different `CtxFunc`.
# The Solution
Instead of waiting for upstream, this PR re-uses the escaped HTML
template trees, use `AddParseTree` to add related templates/trees to a
new template instance, then the new template instance can have its own
FuncMap , the function calls in the template trees will always use the
new template's FuncMap.
`template.New` / `template.AddParseTree` / `adding-FuncMap` are all
quite fast, so the performance is not affected.
The details:
1. Make a new `html/template/Template` for `all` templates
2. Add template code to the `all` template
3. Freeze the `all` template, reset its exec func map, it shouldn't
execute any template.
4. When a router wants to render a template by its `name`
1. Find the `name` in `all`
2. Find all its related sub templates
3. Escape all related templates (just like what the html template
package does)
4. Add the escaped parse-trees of related templates into a new (scoped)
`text/template/Template`
5. Add context-related func map into the new (scoped) text template
6. Execute the new (scoped) text template
7. To improve performance, the escaped templates are cached to `template
sets`
# FAQ
## There is a `unsafe` call, is this PR unsafe?
This PR is safe. Golang has strict language definition, it's safe to do
so: https://pkg.go.dev/unsafe#Pointer (1) Conversion of a *T1 to Pointer
to *T2
## What if Golang template supports such feature in the future?
The public structs/interfaces/functions introduced by this PR is quite
simple, the code of `HTMLRender` is not changed too much. It's very easy
to switch to the official mechanism if there would be one.
## Does this PR change the template execution behavior?
No, see the tests (welcome to design more tests if it's necessary)
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2 years ago
|
|
|
"code.gitea.io/gitea/modules/templates"
|
|
|
|
"code.gitea.io/gitea/modules/translation"
|
|
|
|
"code.gitea.io/gitea/modules/web/middleware"
|
|
|
|
|
|
|
|
chi "github.com/go-chi/chi/v5"
|
|
|
|
"github.com/stretchr/testify/assert"
|
|
|
|
)
|
|
|
|
|
|
|
|
// MockContext mock context for unit tests
|
Rewrite queue (#24505)
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2 years ago
|
|
|
// TODO: move this function to other packages, because it depends on "models" package
|
|
|
|
func MockContext(t *testing.T, path string) *context.Context {
|
|
|
|
resp := &mockResponseWriter{}
|
|
|
|
ctx := context.Context{
|
|
|
|
Render: &mockRender{},
|
|
|
|
Data: make(middleware.ContextData),
|
|
|
|
Flash: &middleware.Flash{
|
|
|
|
Values: make(url.Values),
|
|
|
|
},
|
|
|
|
Resp: context.NewResponse(resp),
|
|
|
|
Locale: &translation.MockLocale{},
|
|
|
|
}
|
|
|
|
defer ctx.Close()
|
|
|
|
|
|
|
|
requestURL, err := url.Parse(path)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
req := &http.Request{
|
|
|
|
URL: requestURL,
|
|
|
|
Form: url.Values{},
|
|
|
|
}
|
|
|
|
|
|
|
|
chiCtx := chi.NewRouteContext()
|
|
|
|
req = req.WithContext(scontext.WithValue(req.Context(), chi.RouteCtxKey, chiCtx))
|
|
|
|
ctx.Req = context.WithContext(req, &ctx)
|
|
|
|
return &ctx
|
|
|
|
}
|
|
|
|
|
|
|
|
// LoadRepo load a repo into a test context.
|
|
|
|
func LoadRepo(t *testing.T, ctx *context.Context, repoID int64) {
|
|
|
|
ctx.Repo = &context.Repository{}
|
|
|
|
ctx.Repo.Repository = unittest.AssertExistsAndLoadBean(t, &repo_model.Repository{ID: repoID})
|
|
|
|
var err error
|
|
|
|
ctx.Repo.Owner, err = user_model.GetUserByID(ctx, ctx.Repo.Repository.OwnerID)
|
Add Organization Wide Labels (#10814)
* Add organization wide labels
Implement organization wide labels similar to organization wide
webhooks. This lets you create individual labels for organizations that can be used
for all repos under that organization (so being able to reuse the same
label across multiple repos).
This makes it possible for small organizations with many repos to use
labels effectively.
Fixes #7406
* Add migration
* remove comments
* fix tests
* Update options/locale/locale_en-US.ini
Removed unused translation string
* show org labels in issue search label filter
* Use more clear var name
* rename migration after merge from master
* comment typo
* update migration again after rebase with master
* check for orgID <=0 per guillep2k review
* fmt
* Apply suggestions from code review
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* remove unused code
* Make sure RepoID is 0 when searching orgID per code review
* more changes/code review requests
* More descriptive translation var per code review
* func description/delete comment when issue label deleted instead of hiding it
* remove comment
* only use issues in that repo when calculating number of open issues for org label on repo label page
* Add integration test for IssuesSearch API with labels
* remove unused function
* Update models/issue_label.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Use subquery in GetLabelIDsInReposByNames
* Fix tests to use correct orgID
* fix more tests
* IssuesSearch api now uses new BuildLabelNamesIssueIDsCondition. Add a few more tests as well
* update comment for clarity
* Revert previous code change now that we can use the new BuildLabelNamesIssueIDsCondition
* Don't sort repos by date in IssuesSearch API
After much debugging I've found a strange issue where in some cases MySQL will return a different result than other enigines if a query is sorted by a null collumn. For example with our integration test data where we don't set updated_unix in repository fixtures:
SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `avatar`, `created_unix`, `updated_unix` FROM `repository` ORDER BY updated_unix DESC LIMIT 15 OFFSET 45
Returns different results for MySQL than other engines. However, the similar query:
SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `avatar`, `created_unix`, `updated_unix` FROM `repository` ORDER BY updated_unix DESC LIMIT 15 OFFSET 30
Returns the same results.
This causes integration tests to fail on MySQL in certain cases but would never show up in a real installation. Since this API call always returns issues based on the optionally provided repo_priority_id or the issueID itself, there is no change to results by changing the repo sorting method used to get ids earlier in the function.
* linter is back!
* code review
* remove now unused option
* Fix newline at end of files
* more unused code
* update to master
* check for matching ids before query
* Update models/issue_label.go
Co-Authored-By: 6543 <6543@obermui.de>
* Update models/issue_label.go
* update comments
* Update routers/org/setting.go
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
5 years ago
|
|
|
assert.NoError(t, err)
|
|
|
|
ctx.Repo.RepoLink = ctx.Repo.Repository.Link()
|
|
|
|
ctx.Repo.Permission, err = access_model.GetUserRepoPermission(ctx, ctx.Repo.Repository, ctx.Doer)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
}
|
|
|
|
|
|
|
|
// LoadRepoCommit loads a repo's commit into a test context.
|
|
|
|
func LoadRepoCommit(t *testing.T, ctx *context.Context) {
|
|
|
|
gitRepo, err := git.OpenRepository(ctx, ctx.Repo.Repository.RepoPath())
|
|
|
|
assert.NoError(t, err)
|
|
|
|
defer gitRepo.Close()
|
|
|
|
branch, err := gitRepo.GetHEADBranch()
|
|
|
|
assert.NoError(t, err)
|
|
|
|
assert.NotNil(t, branch)
|
|
|
|
if branch != nil {
|
|
|
|
ctx.Repo.Commit, err = gitRepo.GetBranchCommit(branch.Name)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// LoadUser load a user into a test context.
|
|
|
|
func LoadUser(t *testing.T, ctx *context.Context, userID int64) {
|
|
|
|
ctx.Doer = unittest.AssertExistsAndLoadBean(t, &user_model.User{ID: userID})
|
|
|
|
}
|
|
|
|
|
|
|
|
// LoadGitRepo load a git repo into a test context. Requires that ctx.Repo has
|
|
|
|
// already been populated.
|
|
|
|
func LoadGitRepo(t *testing.T, ctx *context.Context) {
|
|
|
|
assert.NoError(t, ctx.Repo.Repository.LoadOwner(ctx))
|
|
|
|
var err error
|
|
|
|
ctx.Repo.GitRepo, err = git.OpenRepository(ctx, ctx.Repo.Repository.RepoPath())
|
|
|
|
assert.NoError(t, err)
|
|
|
|
}
|
|
|
|
|
|
|
|
type mockResponseWriter struct {
|
|
|
|
httptest.ResponseRecorder
|
|
|
|
size int
|
|
|
|
}
|
|
|
|
|
|
|
|
func (rw *mockResponseWriter) Write(b []byte) (int, error) {
|
|
|
|
rw.size += len(b)
|
|
|
|
return rw.ResponseRecorder.Write(b)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (rw *mockResponseWriter) Status() int {
|
|
|
|
return rw.ResponseRecorder.Code
|
|
|
|
}
|
|
|
|
|
|
|
|
func (rw *mockResponseWriter) Written() bool {
|
|
|
|
return rw.ResponseRecorder.Code > 0
|
|
|
|
}
|
|
|
|
|
|
|
|
func (rw *mockResponseWriter) Size() int {
|
|
|
|
return rw.size
|
|
|
|
}
|
|
|
|
|
|
|
|
func (rw *mockResponseWriter) Push(target string, opts *http.PushOptions) error {
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
|
|
|
type mockRender struct{}
|
|
|
|
|
Make HTML template functions support context (#24056)
# Background
Golang template is not friendly for large projects, and Golang template
team is quite slow, related:
* `https://github.com/golang/go/issues/54450`
Without upstream support, we can also have our solution to make HTML
template functions support context.
It helps a lot, the above Golang template issue `#54450` explains a lot:
1. It makes `{{Locale.Tr}}` could be used in any template, without
passing unclear `(dict "root" . )` anymore.
2. More and more functions need `context`, like `avatar`, etc, we do not
need to do `(dict "Context" $.Context)` anymore.
3. Many request-related functions could be shared by parent&children
templates, like "user setting" / "system setting"
See the test `TestScopedTemplateSetFuncMap`, one template set, two
`Execute` calls with different `CtxFunc`.
# The Solution
Instead of waiting for upstream, this PR re-uses the escaped HTML
template trees, use `AddParseTree` to add related templates/trees to a
new template instance, then the new template instance can have its own
FuncMap , the function calls in the template trees will always use the
new template's FuncMap.
`template.New` / `template.AddParseTree` / `adding-FuncMap` are all
quite fast, so the performance is not affected.
The details:
1. Make a new `html/template/Template` for `all` templates
2. Add template code to the `all` template
3. Freeze the `all` template, reset its exec func map, it shouldn't
execute any template.
4. When a router wants to render a template by its `name`
1. Find the `name` in `all`
2. Find all its related sub templates
3. Escape all related templates (just like what the html template
package does)
4. Add the escaped parse-trees of related templates into a new (scoped)
`text/template/Template`
5. Add context-related func map into the new (scoped) text template
6. Execute the new (scoped) text template
7. To improve performance, the escaped templates are cached to `template
sets`
# FAQ
## There is a `unsafe` call, is this PR unsafe?
This PR is safe. Golang has strict language definition, it's safe to do
so: https://pkg.go.dev/unsafe#Pointer (1) Conversion of a *T1 to Pointer
to *T2
## What if Golang template supports such feature in the future?
The public structs/interfaces/functions introduced by this PR is quite
simple, the code of `HTMLRender` is not changed too much. It's very easy
to switch to the official mechanism if there would be one.
## Does this PR change the template execution behavior?
No, see the tests (welcome to design more tests if it's necessary)
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
2 years ago
|
|
|
func (tr *mockRender) TemplateLookup(tmpl string) (templates.TemplateExecutor, error) {
|
|
|
|
return nil, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
func (tr *mockRender) HTML(w io.Writer, status int, _ string, _ interface{}) error {
|
|
|
|
if resp, ok := w.(http.ResponseWriter); ok {
|
|
|
|
resp.WriteHeader(status)
|
|
|
|
}
|
|
|
|
return nil
|
|
|
|
}
|