Backport #26122 by @Zettat123
This PR
- Fix#26093. Replace `time.Time` with `timeutil.TimeStamp`
- Fix#26135. Add missing `xorm:"extends"` to `CountLFSMetaObject` for
LFS meta object query
- Add a unit test for LFS meta object garbage collection
Co-authored-by: Zettat123 <zettat123@gmail.com>
This PR adds a task to the cron service to allow garbage collection of
LFS meta objects. As repositories may have a large number of
LFSMetaObjects, an updated column is added to this table and it is used
to perform a generational GC to attempt to reduce the amount of work.
(There may need to be a bit more work here but this is probably enough
for the moment.)
Fix#7045
Signed-off-by: Andrew Thornton <art27@cantab.net>
After #22362, we can feel free to use transactions without
`db.DefaultContext`.
And there are still lots of models using `db.DefaultContext`, I think we
should refactor them carefully and one by one.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The recent PR adding orphaned checks to the LFS storage is not
sufficient to completely GC LFS, as it is possible for LFSMetaObjects to
remain associated with repos but still need to be garbage collected.
Imagine a situation where a branch is uploaded containing LFS files but
that branch is later completely deleted. The LFSMetaObjects will remain
associated with the Repository but the Repository will no longer contain
any pointers to the object.
This PR adds a second doctor command to perform a full GC.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
The doctor check `storages` currently only checks the attachment
storage. This PR adds some basic garbage collection functionality for
the other types of storage.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Fix#19513
This PR introduce a new db method `InTransaction(context.Context)`,
and also builtin check on `db.TxContext` and `db.WithTx`.
There is also a new method `db.AutoTx` has been introduced but could be used by other PRs.
`WithTx` will always open a new transaction, if a transaction exist in context, return an error.
`AutoTx` will try to open a new transaction if no transaction exist in context.
That means it will always enter a transaction if there is no error.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
A lot of our code is repeatedly testing if individual errors are
specific types of Not Exist errors. This is repetitative and unnecesary.
`Unwrap() error` provides a common way of labelling an error as a
NotExist error and we can/should use this.
This PR has chosen to use the common `io/fs` errors e.g.
`fs.ErrNotExist` for our errors. This is in some ways not completely
correct as these are not filesystem errors but it seems like a
reasonable thing to do and would allow us to simplify a lot of our code
to `errors.Is(err, fs.ErrNotExist)` instead of
`package.IsErr...NotExist(err)`
I am open to suggestions to use a different base error - perhaps
`models/db.ErrNotExist` if that would be felt to be better.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
* When non-admin users use code search, get code unit accessible repos in one main query
* Modified some comments to match the changes
* Removed unnecessary check for Access Mode in Collaboration table
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Move access and repo permission to models/perm/access
* fix test
* Move some git related files into sub package models/git
* Fix build
* fix git test
* move lfs to sub package
* move more git related functions to models/git
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Move some repository related code into sub package
* Move more repository functions out of models
* Fix lint
* Some performance optimization for webhooks and others
* some refactors
* Fix lint
* Fix
* Update modules/repository/delete.go
Co-authored-by: delvh <dev.lh@web.de>
* Fix test
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
Co-authored-by: delvh <dev.lh@web.de>
Yet another issue has come up where the logging from SyncMirrors does not provide
enough context. This PR adds more context to these logging events.
Related #19038
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR reworked the Find pointer files feature in Settings -> LFS page.
When a LFS object is missing from database but exists in LFS content store, admin can associate it to the repository by clicking the Associate button.
This PR is not perfect (because the LFS module itself should be improved too), it's just a nice-to-have feature to help users recover their LFS repositories (eg: database was lost / table was truncated)
* Some refactors related repository model
* Move more methods out of repository
* Move repository into models/repo
* Fix test
* Fix test
* some improvements
* Remove unnecessary function
* DBContext is just a Context
This PR removes some of the specialness from the DBContext and makes it context
This allows us to simplify the GetEngine code to wrap around any context in future
and means that we can change our loadRepo(e Engine) functions to simply take contexts.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix unit tests
Signed-off-by: Andrew Thornton <art27@cantab.net>
* another place that needs to set the initial context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* avoid race
Signed-off-by: Andrew Thornton <art27@cantab.net>
* change attachment error
Signed-off-by: Andrew Thornton <art27@cantab.net>
* _ to unused func options
* rm useless brakets
* rm trifial non used models functions
* rm dead code
* rm dead global vars
* fix routers/api/v1/repo/issue.go
* dont overload import module
* Implemented LFS client.
* Implemented scanning for pointer files.
* Implemented downloading of lfs files.
* Moved model-dependent code into services.
* Removed models dependency. Added TryReadPointerFromBuffer.
* Migrated code from service to module.
* Centralised storage creation.
* Removed dependency from models.
* Moved ContentStore into modules.
* Share structs between server and client.
* Moved method to services.
* Implemented lfs download on clone.
* Implemented LFS sync on clone and mirror update.
* Added form fields.
* Updated templates.
* Fixed condition.
* Use alternate endpoint.
* Added missing methods.
* Fixed typo and make linter happy.
* Detached pointer parser from gogit dependency.
* Fixed TestGetLFSRange test.
* Added context to support cancellation.
* Use ReadFull to probably read more data.
* Removed duplicated code from models.
* Moved scan implementation into pointer_scanner_nogogit.
* Changed method name.
* Added comments.
* Added more/specific log/error messages.
* Embedded lfs.Pointer into models.LFSMetaObject.
* Moved code from models to module.
* Moved code from models to module.
* Moved code from models to module.
* Reduced pointer usage.
* Embedded type.
* Use promoted fields.
* Fixed unexpected eof.
* Added unit tests.
* Implemented migration of local file paths.
* Show an error on invalid LFS endpoints.
* Hide settings if not used.
* Added LFS info to mirror struct.
* Fixed comment.
* Check LFS endpoint.
* Manage LFS settings from mirror page.
* Fixed selector.
* Adjusted selector.
* Added more tests.
* Added local filesystem migration test.
* Fixed typo.
* Reset settings.
* Added special windows path handling.
* Added unit test for HTTPClient.
* Added unit test for BasicTransferAdapter.
* Moved into util package.
* Test if LFS endpoint is allowed.
* Added support for git://
* Just use a static placeholder as the displayed url may be invalid.
* Reverted to original code.
* Added "Advanced Settings".
* Updated wording.
* Added discovery info link.
* Implemented suggestion.
* Fixed missing format parameter.
* Added Pointer.IsValid().
* Always remove model on error.
* Added suggestions.
* Use channel instead of array.
* Update routers/repo/migrate.go
* fmt
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
* chore: rewrite format.
* chore: update format
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
* chore: update format
Signed-off-by: Bo-Yi Wu <appleboy.tw@gmail.com>
* chore: Adjacent parameters with the same type should be grouped together
* chore: update format.
* LFS support to be stored on minio
* Fix test
* Fix lint
* Fix lint
* Fix check
* Fix test
* Update documents and add migration for LFS
* Fix some bugs
* Restricted users (#4334): initial implementation
* Add User.IsRestricted & UI to edit it
* Pass user object instead of user id to places where IsRestricted flag matters
* Restricted users: maintain access rows for all referenced repos (incl public)
* Take logged in user & IsRestricted flag into account in org/repo listings, searches and accesses
* Add basic repo access tests for restricted users
Signed-off-by: Manush Dodunekov <manush@stendahls.se>
* Mention restricted users in the faq
Signed-off-by: Manush Dodunekov <manush@stendahls.se>
* Revert unnecessary change `.isUserPartOfOrg` -> `.IsUserPartOfOrg`
Signed-off-by: Manush Dodunekov <manush@stendahls.se>
* Remove unnecessary `org.IsOrganization()` call
Signed-off-by: Manush Dodunekov <manush@stendahls.se>
* Revert to an `int64` keyed `accessMap`
* Add type `userAccess`
* Add convenience func updateUserAccess()
* Turn accessMap into a `map[int64]userAccess`
Signed-off-by: Manush Dodunekov <manush@stendahls.se>
* or even better: `map[int64]*userAccess`
* updateUserAccess(): use tighter syntax as suggested by lafriks
* even tighter
* Avoid extra loop
* Don't disclose limited orgs to unauthenticated users
* Don't assume block only applies to orgs
* Use an array of `VisibleType` for filtering
* fix yet another thinko
* Ok - no need for u
* Revert "Ok - no need for u"
This reverts commit 5c3e886aab.
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
This PR adds basic repository LFS management UI including the ability to find all possible pointers within the repository. Locks are not managed at present but would be addable through some simple additions.
* Add basic repository lfs management
* add auto-associate function
* Add functionality to find commits with this lfs file
* Add link to find commits on the lfs file view
* Adjust commit view to state the likely branch causing the commit
* Only read Oid from database
* display ui time with customize time location
* fix lint
* rename UILocation to DefaultUILocation
* move time related functions to modules/timeutil
* fix tests
* fix tests
* fix build
* fix swagger
* Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile
* Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview
* Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile
* Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles
* Move branch checkout functions out of repo_editor.go as they are no longer used there
* BUGFIX: The default permissions should be 100644
This is a change from the previous code but is more in keeping
with the default behaviour of git.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Standardise cleanUploadFilename to more closely match git
See verify_path in: 7f4e641693/read-cache.c (L951)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Redirect on bad paths
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Refactor to move the uploading functions out to a module
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add LFS support
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update upload.go attribution header
Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution.
* Delete upload files after session committed
* Ensure that GIT_AUTHOR_NAME etc. are valid for git
see #5774
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add in test cases per @lafriks comment
* Add space between gitea and github imports
Signed-off-by: Andrew Thornton <art27@cantab.net>
* more examples in TestCleanUploadName
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix formatting
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set the SSH_ORIGINAL_COMMAND to ensure hooks are run
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Switch off SSH_ORIGINAL_COMMAND
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Pooled and buffered gzip implementation
* Add test for gzip
* Add integration test
* Ensure lfs check within transaction
The previous code made it possible for a race condition to occur whereby a LFSMetaObject could be checked into the database twice. We should check if the LFSMetaObject is within the database and insert it if not in one transaction.
* Try to avoid primary key problem in postgres
The integration tests are being affected by
https://github.com/go-testfixtures/testfixtures/issues/39 if we set the
primary key high enough, keep a count of this and remove at the end of
each test we shouldn't be affected by this.
* refactor struct's time to remove unnecessary memory usage
* use AsTimePtr simple code
* fix tests
* fix time compare
* fix template on gpg
* use AddDuration instead of Add
* use created & updated instead BeforeInsert & BeforeUpdate
* fix vendor checksum
* only show generated SQL when development mode
* remove extra update column updated_unix
* remove trace config
* Import github.com/git-lfs/lfs-test-server as lfs module base
Imported commit is 3968aac269a77b73924649b9412ae03f7ccd3198
Removed:
Dockerfile CONTRIBUTING.md mgmt* script/ vendor/ kvlogger.go
.dockerignore .gitignore README.md
* Remove config, add JWT support from github.com/mgit-at/lfs-test-server
Imported commit f0cdcc5a01599c5a955dc1bbf683bb4acecdba83
* Add LFS settings
* Add LFS meta object model
* Add LFS routes and initialization
* Import github.com/dgrijalva/jwt-go into vendor/
* Adapt LFS module: handlers, routing, meta store
* Move LFS routes to /user/repo/info/lfs/*
* Add request header checks to LFS BatchHandler / PostHandler
* Implement LFS basic authentication
* Rework JWT secret generation / load
* Implement LFS SSH token authentication with JWT
Specification: https://github.com/github/git-lfs/tree/master/docs/api
* Integrate LFS settings into install process
* Remove LFS objects when repository is deleted
Only removes objects from content store when deleted repo is the only
referencing repository
* Make LFS module stateless
Fixes bug where LFS would not work after installation without
restarting Gitea
* Change 500 'Internal Server Error' to 400 'Bad Request'
* Change sql query to xorm call
* Remove unneeded type from LFS module
* Change internal imports to code.gitea.io/gitea/
* Add Gitea authors copyright
* Change basic auth realm to "gitea-lfs"
* Add unique indexes to LFS model
* Use xorm count function in LFS check on repository delete
* Return io.ReadCloser from content store and close after usage
* Add LFS info to runWeb()
* Export LFS content store base path
* LFS file download from UI
* Work around git-lfs client issue with unauthenticated requests
Returning a dummy Authorization header for unauthenticated requests
lets git-lfs client skip asking for auth credentials
See: https://github.com/github/git-lfs/issues/1088
* Fix unauthenticated UI downloads from public repositories
* Authentication check order, Finish LFS file view logic
* Ignore LFS hooks if installed for current OS user
Fixes Gitea UI actions for repositories tracking LFS files.
Checks for minimum needed git version by parsing the semantic version
string.
* Hide LFS metafile diff from commit view, marking as binary
* Show LFS notice if file in commit view is tracked
* Add notbefore/nbf JWT claim
* Correct lint suggestions - comments for structs and functions
- Add comments to LFS model
- Function comment for GetRandomBytesAsBase64
- LFS server function comments and lint variable suggestion
* Move secret generation code out of conditional
Ensures no LFS code may run with an empty secret
* Do not hand out JWT tokens if LFS server support is disabled
Giteabot
silverwind
Lunny Xiao
Yarden Shoham
zeripath
Jason Song
flynnnnnnnnnn
delvh
Hugo Hoitink
6543
wxiaoguang
KN4CK3R
Bo-Yi Wu
John Olheiser
Manush Dodunekov
Lauris BH
Fabian Zaremba