This PR implemented object storages(LFS/Packages/Attachments and etc.)
for Azure Blob Storage. It depends on azure official golang SDK and can
support both the azure blob storage cloud service and azurite mock
server.
Replace #25458Fix#22527
- [x] CI Tests
- [x] integration test, MSSQL integration tests will now based on
azureblob
- [x] unit test
- [x] CLI Migrate Storage
- [x] Documentation for configuration added
------
TODO (other PRs):
- [ ] Improve performance of `blob download`.
---------
Co-authored-by: yp05327 <576951401@qq.com>
This PR do some performance optimzations.
- [x] Add `index` for the column `comment_id` of `Attachment` table to
accelerate query from the database.
- [x] Remove unnecessary database queries when viewing issues. Before
some conditions which id = 0 will be sent to the database
- [x] Remove duplicated load posters
- [x] Batch loading attachements, isread of comments on viewing issue
---------
Co-authored-by: Zettat123 <zettat123@gmail.com>
As promised in #23817, I have this made a PR to make Release Download
URLs predictable. It currently follows the schema
`<repo>/releases/download/<tag>/<filename>`. this already works, but it
is nowhere shown in the UI or the API. The Problem is, that it is
currently possible to have multiple files with the same name (why do we
even allow this) for a release. I had written some Code to check, if a
Release has 2 or more files with the same Name. If yes, it uses the old
`attachments/<uuid>` URlL if no it uses the new fancy URL.
I had also changed `<repo>/releases/download/<tag>/<filename>` to
directly serve the File instead of redirecting, so people who who use
automatic update checker don't end up with the `attachments/<uuid>` URL.
Fixes#10919
---------
Co-authored-by: a1012112796 <1012112796@qq.com>
Original Issue: https://github.com/go-gitea/gitea/issues/22102
This addition would be a big benefit for design and art teams using the
issue tracking.
The preview will be the latest "image type" attachments on an issue-
simple, and allows for automatic updates of the cover image as issue
progress is made!
This would make Gitea competitive with Trello... wouldn't it be amazing
to say goodbye to Atlassian products? Ha.
First image is the most recent, the SQL will fetch up to 5 latest images
(URL string).
All images supported by browsers plus upcoming formats: *.avif *.bmp
*.gif *.jpg *.jpeg *.jxl *.png *.svg *.webp
The CSS will try to center-align images until it cannot, then it will
left align with overflow hidden. Single images get to be slightly
larger!
Tested so far on: Chrome, Firefox, Android Chrome, Android Firefox.
Current revision with light and dark themes:
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The doctor check `storages` currently only checks the attachment
storage. This PR adds some basic garbage collection functionality for
the other types of storage.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
A lot of our code is repeatedly testing if individual errors are
specific types of Not Exist errors. This is repetitative and unnecesary.
`Unwrap() error` provides a common way of labelling an error as a
NotExist error and we can/should use this.
This PR has chosen to use the common `io/fs` errors e.g.
`fs.ErrNotExist` for our errors. This is in some ways not completely
correct as these are not filesystem errors but it seems like a
reasonable thing to do and would allow us to simplify a lot of our code
to `errors.Is(err, fs.ErrNotExist)` instead of
`package.IsErr...NotExist(err)`
I am open to suggestions to use a different base error - perhaps
`models/db.ErrNotExist` if that would be felt to be better.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
This is an extensive PR attempting to fix these issues.
1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future.
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
* DBContext is just a Context
This PR removes some of the specialness from the DBContext and makes it context
This allows us to simplify the GetEngine code to wrap around any context in future
and means that we can change our loadRepo(e Engine) functions to simply take contexts.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix unit tests
Signed-off-by: Andrew Thornton <art27@cantab.net>
* another place that needs to set the initial context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* avoid race
Signed-off-by: Andrew Thornton <art27@cantab.net>
* change attachment error
Signed-off-by: Andrew Thornton <art27@cantab.net>
When create a new issue or comment and paste/upload an attachment/image, it will not assign an issue id before submit. So if user give up the creating, the attachments will lost key feature and become dirty content. We don't know if we need to delete the attachment even if the repository deleted.
This PR add a repo_id in attachment table so that even if a new upload attachment with no issue_id or release_id but should have repo_id. When deleting a repository, they could also be deleted.
Co-authored-by: 6543 <6543@obermui.de>
* chore: rewrite format.
* chore: update format
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
* chore: update format
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
* chore: Adjacent parameters with the same type should be grouped together
* chore: update format.
* Add a storage layer for attachments
* Fix some bug
* fix test
* Fix copyright head and lint
* Fix bug
* Add setting for minio and flags for migrate-storage
* Add documents
* fix lint
* Add test for minio store type on attachments
* fix test
* fix test
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Add warning when storage migrated successfully
* Fix drone
* fix test
* rebase
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* refactor the codes
* add trace
* Fix test
* remove log on xorm
* Fi download bug
* Add a storage layer for attachments
* Add setting for minio and flags for migrate-storage
* fix lint
* Add test for minio store type on attachments
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix drone
* fix test
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* refactor the codes
* add trace
* Fix test
* Add URL function to serve attachments directly from S3/Minio
* Add ability to enable/disable redirection in attachment configuration
* Fix typo
* Add a storage layer for attachments
* Add setting for minio and flags for migrate-storage
* fix lint
* Add test for minio store type on attachments
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix drone
* fix test
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* don't change unrelated files
* Fix lint
* Fix build
* update go.mod and go.sum
* Use github.com/minio/minio-go/v6
* Remove unused function
* Upgrade minio to v7 and some other improvements
* fix lint
* Fix go mod
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Tyler <tystuyfzand@gmail.com>
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.
Fix#12339
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
* Use ID or Where to instead directly use Get when load object from database
* Apply suggestions from code review
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
* test: add current attachement responses
* refactor: check if attachement is linked and accessible by user
* chore: clean TODO
* fix: typo attachement -> attachment
* revert un-needed go.sum change
* refactor: move models logic to models
* fix TestCreateIssueAttachment which was wrongly successful
* fix unit tests with unittype added
* fix unit tests with changes
* use a valid uuid format for pgsql int. test
* test: add unit test TestLinkedRepository
* refactor: allow uploader to access unlinked attachement
* add missing blank line
* refactor: move to a separate function repo.GetAttachment
* typo
* test: remove err test return
* refactor: use repo perm for access checking generally + 404 for all reject
* display ui time with customize time location
* fix lint
* rename UILocation to DefaultUILocation
* move time related functions to modules/timeutil
* fix tests
* fix tests
* fix build
* fix swagger
* Add size column to attachment
Migrate attachments by calculating file sizes
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Calculate attachment size on creation
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Log error instead of returning error
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Improve release page ui by adding size
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Add documentation to a.MustSize()
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Add Attachment API
* repos/:owner/:repo/releases (add attachments)
* repos/:owner/:repo/releases/:id (add attachments)
* repos/:owner/:repo/releases/:id/attachments
* repos/:owner/:repo/releases/:id/attachments/:attachment_id
Signed-off-by: Jonas Franz <info@jonasfranz.de>
* Add unit tests for new attachment functions
Fix comments
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* fix lint
* Update vendor.json
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* remove version of sdk
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Fix unit tests
Add missing license header
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Add CreateReleaseAttachment
Add EditReleaseAttachment
Add DeleteReleaseAttachment
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Add filename query parameter for choosing another name for an attachment
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Fix order of imports
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Restricting updatable attachment columns
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* gofmt
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Update go-sdk
Replace Attachments with Assets
Signed-off-by: Jonas Franz <info@jonasfranz.de>
* Update go-sdk
Signed-off-by: Jonas Franz <info@jonasfranz.de>
* Updating go-sdk and regenerating swagger
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Add missing file of go-sdk
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Change origin of code.gitea.io/sdk to code.gitea.io/sdk
Update code.gitea.io/sdk
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Update swagger
Signed-off-by: Jonas Franz <info@jonasfranz.software>
* Update updateAttachment
* refactor struct's time to remove unnecessary memory usage
* use AsTimePtr simple code
* fix tests
* fix time compare
* fix template on gpg
* use AddDuration instead of Add
* use created & updated instead BeforeInsert & BeforeUpdate
* fix vendor checksum
* only show generated SQL when development mode
* remove extra update column updated_unix
* remove trace config
Lunny Xiao
JakobDev
Nathaniel Sabanski
flynnnnnnnnnn
KN4CK3R
zeripath
delvh
6543
Bo-Yi Wu
Antoine GIRARD
gdeverlant
Jonas Franz
Lauris BH
