Some specific events on Gitlab issues and merge requests are stored
separately from comments as "resource state events". With this change,
all relevant resource state events are downloaded during issue and merge
request migration, and converted to comments.
This PR also updates the template used to render comments to add support
for migrated comments of these types.
ref: https://docs.gitlab.com/ee/api/resource_state_events.html
(cherry picked from commit 17f170ee3724d8bdf2ddaad4211b12433f78ff0e)
GitLab generates "system notes" whenever an event happens within the
platform. Unlike Gitea, those events are stored and retrieved as text
comments with no semantic details. The only way to tell whether a
comment was generated in this manner is the `system` flag on the note
type.
This PR adds detection for a new specific kind of event: Changing the
target branch of a PR. When detected, it is downloaded using Gitea's
type for this event, and eventually uploaded into Gitea in the expected
format, i.e. with no text content in the comment.
This PR also updates the template used to render comments to add support
for migrated comments of this type.
ref:
11bd6dc826/app/services/system_notes/merge_requests_service.rb (L102)
(cherry picked from commit 6e5966597c2d498d1a8540dad965461d44ff8e57)
Now we can get object format name from git command line or from the
database repository table. Assume the column is right, we don't need to
read from git command line every time.
This also fixed a possible bug that the object format is wrong when
migrating a sha256 repository from external.
<img width="658" alt="image"
src="https://github.com/go-gitea/gitea/assets/81045/6e9a9dcf-13bf-4267-928b-6bf2c2560423">
(cherry picked from commit b79c30435f439af8243ee281310258cdf141e27b)
Conflicts:
routers/web/repo/blame.go
services/agit/agit.go
context
GitLab generates "system notes" whenever an event happens within the
platform. Unlike Gitea, those events are stored and retrieved as text
comments with no semantic details. The only way to tell whether a
comment was generated in this manner is the `system` flag on the note
type.
This PR adds detection for two specific kinds of events: Scheduling and
un-scheduling of automatic merges on a PR. When detected, they are
downloaded using Gitea's type for these events, and eventually uploaded
into Gitea in the expected format, i.e. with no text content in the
comment.
This PR also updates the template used to render comments to add support
for migrated comments of these two types.
ref:
11bd6dc826/app/services/system_notes/merge_requests_service.rb (L6-L17)
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
(cherry picked from commit a70c00b80bcb5de8479e407f1b8f08dcf756019d)
So the caller can check log events at the desired level instead of
being limited to the default level log.INFO
(cherry picked from commit 2fbf5f9555641a1244576df92cb518f8ad76c162)
(cherry picked from commit e2137a3147389114475db787522d5c22ff249d2c)
Closes#1789.
The bug was due to the fact that GitLab does not guarantee that issue numbers are created sequentially: some identifiers can be skipped. Therefore, the new pull requests numbers should not be offset by the number of issues, but by the maximum issue number.
See for instance https://gitlab.com/troyengel/archbuild/-/issues/?sort=created_date&state=all&first_page_size=20, where there is only a singe issue with number "2".
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/1790
Co-authored-by: Antonin Delpeuch <antonin@delpeuch.eu>
Co-committed-by: Antonin Delpeuch <antonin@delpeuch.eu>
(cherry picked from commit 2c185c39fe600041701d5f59cb1076a788815cb4)
(cherry picked from commit 8f68dc4c9c2f0acab55d59a496b0f141befad969)
(cherry picked from commit 7e932b7fca1b119e7cc646183c383ba51a5f1d14)
(cherry picked from commit 6bbe75ecf8ac502bd42ff5765e6e7733f290a54e)
(cherry picked from commit b18c2e8d658c3311e0a299696bd1b6612c52ef13)
Conflicts:
services/migrations/gitlab.go
https://codeberg.org/forgejo/forgejo/pulls/2075
(cherry picked from commit abc129c762b3c1a992ad5c67adf62d8336eadbbe)
(cherry picked from commit 28884fac10c455a9f40bebd961fca40afd4a749e)
(cherry picked from commit 5f528dd85fa6705c60d15bce616a46f00df1b85b)
(cherry picked from commit cb9b8a31b25b27fa5e386b20f02e532bbf3462a2)
(cherry picked from commit 97f02df163e2b4e23b82e23e5ef57a586b17f021)
(cherry picked from commit 4611e10e6acef9185e1f8c84e63d1fe95fcf6436)
Fix gitlab migration unit test
Closes#1837.
The differences in dates can be explained by commit e19b9653ea, which
changed the order in which "created_date" and "updated_date" are
considered.
(cherry picked from commit b0bba20aa44e30ef0296b89f336d426224d73a16)
Mock HTTP requests in GitLab migration test
This introduces a new utility which can be added to other tests
making HTTP calls to a live service, to cache the responses of this
service in the repository.
(cherry picked from commit 52053b138948bd74c7eb88c0796c2e18f4247f3c)
Enable mocked HTTP responses for GitLab migration test
(cherry picked from commit 19cefc4de24b935a6a5c92be8360301f196f3aa5)
Simplify HTTP mocking utility in unit tests
Follow-up to https://codeberg.org/forgejo/forgejo/pulls/1841
(cherry picked from commit ca517c8bb4bf97f061b8b19fd3303d734f46660c)
(cherry picked from commit b227e0dd6bdf2dc3e8679443fc538fbce4b3bcf5)
(cherry picked from commit 6cc9d06556cda6c952a0542284fbe504114971ce)
(cherry picked from commit f0746e648dc30510d655b8a3b821199b2638800f)
(cherry picked from commit 414193341b8493723c16694789cbc08dc80b9ce5)
(cherry picked from commit 6e93df3bbb6c589502afc9dc74a7ae1a7c0f7da8)
(cherry picked from commit db0dbab5527c9f1783fd0eddb057c2d91cbb67e4)
(cherry picked from commit 8f9c9c63fbd3f266bb29d38791e83dc369cc1350)
(cherry picked from commit e74e26203095b675ccedbc2e166faed59369d467)
(cherry picked from commit 2e0933edcfa102b578fb3c2500f9e6af9e5ba1c7)
(cherry picked from commit 65060c69616631221d3dd9ef8b48fbcb007ad0c6)
- If there's a code comment that's received during the migration that
contains no diffhunk, skip it. This either means it was commenting on
old diffhunk or it's just a general codecomment. Forgejo supports
neither of such type of code comment.
- Resolves https://codeberg.org/forgejo/forgejo/issues/1407
(cherry picked from commit ae463c7c559e02975ce5e758d8780def978eebee)
(cherry picked from commit bf48f02a86d6a193417f13a77031b8207a173dca)
(cherry picked from commit 10c3f102fa9135de37e9f73137ae5a9cf7072635)
(cherry picked from commit 828b4cc10cd0fc7e2540fe75e88b6ebf978c5c84)
(cherry picked from commit 6427fa65b641a32ead53779e3e7bda97704567df)
(cherry picked from commit 5b7a43c43fed0eb39e84edd652a699461f14fbbb)
(cherry picked from commit 4eef0fce72894fba2a8a138836421588f96f1087)
(cherry picked from commit a46192a4a6cffa1122ffc0c4781ba93c3067b05d)
(cherry picked from commit 107a9b8233731b3ac2d3a5474795a227c5bb8c0d)
(cherry picked from commit 308251fc48b674e58d27acf1ccf1bc00b5fe2d54)
(cherry picked from commit 017c4a53c5c8e3e2f1a1d8a06b1f975697584973)
(cherry picked from commit 4534a3393b5a6beb500eb36d92ac87dda485b984)
(cherry picked from commit 74e0c1663d27afc98b77c59db9f9a1593f7ea766)
(cherry picked from commit 9b17353f85c4f6273aec64996e70594fc2b8f37e)
(cherry picked from commit 09b6f58304f526c2fc8c9aeecf238b8bfa9ab1c5)
(cherry picked from commit bc649733a121503bd2c8855a7bcac5bfce883363)
(cherry picked from commit f1d4c783e272b10d3193e78e0bbbca1b2f7ef75c)
(cherry picked from commit d6850bc3087ba40e61099700be97818501472cbe)
(cherry picked from commit 21230d2d24dc9c96811891e55e2bb4974f8940ca)
(cherry picked from commit 569b27838254cb4a7bf54a4a3af7c829845442b3)
[FEAT] add Forgejo Git Service (squash) register a Forgejo factory
If the Forgejo factory for the Forgejo service is not registered,
newDownloader will fallback to a git service and not migrate issues
etc.
Refs: https://codeberg.org/forgejo/forgejo/issues/1678
(cherry picked from commit 51938cd1613c789c7176ca59592689c3bf055f45)
[FEAT] add Forgero Git Service
Signed-off-by: cassiozareck <cassiomilczareck@gmail.com>
(cherry picked from commit a878adfe628cf6dc367a17c3715fcd3499aa02b6)
Adding description and Forgejo SVG
(cherry picked from commit 13738c03804d019f28550e46a4ebc37dbe3a5cfc)
Undo reordering and tmpl redirection
(cherry picked from commit 9ae51c46f42acecac834371857e638098ebf6d27)
(cherry picked from commit 70fffdc61d06dd1d70b6a31496676a23d3d0c2fc)
(cherry picked from commit c0ebfa9da3db3e60d7b403a1bf8b8a19c32c5dc7)
(cherry picked from commit 9922c92787eccaba0021486ba0a3eb28583969e1)
(cherry picked from commit 00c0effbc74aedc7a4167a69c8a410ef324d576b)
(cherry picked from commit e4c9525b137205fa9ffdb4e0d7492bbbda9be6b5)
(cherry picked from commit 09d7b83211652d045975b0e3bb790856267d52a5)
(cherry picked from commit bbcd5975c91f6932f7f2ee07fbd63e84560ba96a)
(cherry picked from commit 55c70a0e18d33d8ac0da9ffb97f6d994ed88a319)
(cherry picked from commit 76596410c0dd0137cd497c9728c3e1d1c98f2430)
(cherry picked from commit 1308043931388bf6de691ad0f766861b77fb08a5)
(cherry picked from commit 919d6aedfed6abc8ec9def19f8deec2ee413252b)
[FEAT] add Forgero Git Service (squash) more tests
Previously only Gitea service was being tested under self-hosted migrations. Since Forgejo is also self-hosted and in fact use the same downloader/migrator we can add to this suite another test that will do the same, migrating the same repository under the same local instance but for the Forgejo service (represented by 9)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/1709
Co-authored-by: zareck <cassiomilczareck@gmail.com>
Co-committed-by: zareck <cassiomilczareck@gmail.com>
(cherry picked from commit 40a4b8f1a8637f78cf2f48104f0b336377652df9)
(cherry picked from commit 3198b4a64240b7d4e8b33d8b858a12d046db38c9)
(cherry picked from commit 4edda1f3890eb1b5bb9b1eeec1214dbc11f8e343)
(cherry picked from commit 4d91b77d29fd4b20be12bf21c31447722ff6da40)
(cherry picked from commit afe85c52e3c1c165c171443e3ba79caef1560e0d)
(cherry picked from commit 5ea7df79adfba4a85c7ebbccfb7da15b48eef19c)
(cherry picked from commit a667182542abab8ebb29905fb38afe509682c220)
(cherry picked from commit a9bebb1e71b8a20bb19352357a5b71b9b84c0d21)
(cherry picked from commit 4831a89e460bb982a497b6f22613149840b13a9c)
(cherry picked from commit e02a74651f9813cc72c64e391a2fa6e3c282ce3f)
(cherry picked from commit 05dcef59aa4d05b040fe4ae24d73f9d9660e6ed2)
(cherry picked from commit c8bac187f983150150a2652724bab8f923be44e0)
(cherry picked from commit c87903a0cc75daeee8783d9774158711011d4382)
## Purpose
This is a refactor toward building an abstraction over managing git
repositories.
Afterwards, it does not matter anymore if they are stored on the local
disk or somewhere remote.
## What this PR changes
We used `git.OpenRepository` everywhere previously.
Now, we should split them into two distinct functions:
Firstly, there are temporary repositories which do not change:
```go
git.OpenRepository(ctx, diskPath)
```
Gitea managed repositories having a record in the database in the
`repository` table are moved into the new package `gitrepo`:
```go
gitrepo.OpenRepository(ctx, repo_model.Repo)
```
Why is `repo_model.Repository` the second parameter instead of file
path?
Because then we can easily adapt our repository storage strategy.
The repositories can be stored locally, however, they could just as well
be stored on a remote server.
## Further changes in other PRs
- A Git Command wrapper on package `gitrepo` could be created. i.e.
`NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir:
repo.RepoPath()}`, the directory should be empty before invoking this
method and it can be filled in the function only. #28940
- Remove the `RepoPath()`/`WikiPath()` functions to reduce the
possibility of mistakes.
---------
Co-authored-by: delvh <dev.lh@web.de>
- Remove `ObjectFormatID`
- Remove function `ObjectFormatFromID`.
- Use `Sha1ObjectFormat` directly but not a pointer because it's an
empty struct.
- Store `ObjectFormatName` in `repository` struct
Refactor Hash interfaces and centralize hash function. This will allow
easier introduction of different hash function later on.
This forms the "no-op" part of the SHA256 enablement patch.
Due to a bug in the GitLab API, the diff_refs field is populated in the
response when fetching an individual merge request, but not when
fetching a list of them. That field is used to populate the merge base
commit SHA.
While there is detection for the merge base even when not populated by
the downloader, that detection is not flawless. Specifically, when a
GitLab merge request has a single commit, and gets merged with the
squash strategy, the base branch will be fast-forwarded instead of a
separate squash or merge commit being created. The merge base detection
attempts to find the last commit on the base branch that is also on the
PR branch, but in the fast-forward case that is the PR's only commit.
Assuming the head commit is also the merge base results in the import of
a PR with 0 commits and no diff.
This PR uses the individual merge request endpoint to fetch merge
request data with the diff_refs field. With its data, the base merge
commit can be properly set, which—by not relying on the detection
mentioned above—correctly imports PRs that were "merged" by
fast-forwarding the base branch.
ref: https://gitlab.com/gitlab-org/gitlab/-/issues/29620
Before this PR, the PR migration code populates Gitea's MergedCommitID
field by using GitLab's merge_commit_sha field. However, that field is
only populated when the PR was merged using a merge strategy. When a
squash strategy is used, squash_commit_sha is populated instead.
Given that Gitea does not keep track of merge and squash commits
separately, this PR simply populates Gitea's MergedCommitID by using
whichever field is present in the GitLab API response.
Part of #27065
This PR touches functions used in templates. As templates are not static
typed, errors are harder to find, but I hope I catch it all. I think
some tests from other persons do not hurt.
This PR removed `unittest.MainTest` the second parameter
`TestOptions.GiteaRoot`. Now it detects the root directory by current
working directory.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
To avoid deadlock problem, almost database related functions should be
have ctx as the first parameter.
This PR do a refactor for some of these functions.
Related #14180
Related #25233
Related #22639Close#19786
Related #12763
This PR will change all the branches retrieve method from reading git
data to read database to reduce git read operations.
- [x] Sync git branches information into database when push git data
- [x] Create a new table `Branch`, merge some columns of `DeletedBranch`
into `Branch` table and drop the table `DeletedBranch`.
- [x] Read `Branch` table when visit `code` -> `branch` page
- [x] Read `Branch` table when list branch names in `code` page dropdown
- [x] Read `Branch` table when list git ref compare page
- [x] Provide a button in admin page to manually sync all branches.
- [x] Sync branches if repository is not empty but database branches are
empty when visiting pages with branches list
- [x] Use `commit_time desc` as the default FindBranch order by to keep
consistent as before and deleted branches will be always at the end.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close#12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
This PR
- [x] Move some functions from `issues.go` to `issue_stats.go` and
`issue_label.go`
- [x] Remove duplicated issue options `UserIssueStatsOption` to keep
only one `IssuesOptions`
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
Before, there was a `log/buffer.go`, but that design is not general, and
it introduces a lot of irrelevant `Content() (string, error) ` and
`return "", fmt.Errorf("not supported")` .
And the old `log/buffer.go` is difficult to use, developers have to
write a lot of `Contains` and `Sleep` code.
The new `LogChecker` is designed to be a general approach to help to
assert some messages appearing or not appearing in logs.
Since #23493 has conflicts with latest commits, this PR is my proposal
for fixing #23371
Details are in the comments
And refactor the `modules/options` module, to make it always use
"filepath" to access local files.
Benefits:
* No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))),
"/")` any more (not only one before)
* The function behaviors are clearly defined
Earl Warren
