Commit graph

3306 commits

Author SHA1 Message Date
zeripath e0853d4a21
Add API Token Cache (#16547)
One of the issues holding back performance of the API is the problem of hashing.
Whilst banning BASIC authentication with passwords will help, the API Token scheme
still requires a PBKDF2 hash - which means that heavy API use (using Tokens) can
still cause enormous numbers of hash computations.

A slight solution to this whilst we consider moving to using JWT based tokens and/or
a session orientated solution is to simply cache the successful tokens. This has some
security issues but this should be balanced by the security issues of load from
hashing.

Related #14668

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-08-17 14:30:42 -04:00
Lunny Xiao 74d75eb69c
Fix spelling of HookProcReceiveResult (#16690) 2021-08-14 13:17:10 +02:00
zeripath 23a87a003e
Ensure empty lines are copiable and final new line too (#16678)
* Ensure empty lines are copiable and final new line too

When files are highlighted the newline character needs to be added in a whitespace
compliant mode. Also ensure the final empty newline is rendered.

Fix #16434

* Add test and ensure spans closed

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-14 01:16:56 +02:00
6543 3a6edd3685
Update issue_index to finish migration (#16685)
* update issue_index to finish migration

* One Func to RecalculateIssueIndexForRepo
2021-08-13 21:06:18 +08:00
zeripath 5fbccad906
Fix NPE in fuzzer (#16680)
The fuzzer found an issue with the issue pattern processor where there is a spurious
path.Clean which does not need to be there. This PR also sets the default AppURL for
the fuzzer too.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-13 00:22:05 +02:00
6543 2289580bb7
[API] generalize list header (#16551)
* Add info about list endpoints to CONTRIBUTING.md

* Let all list endpoints return X-Total-Count header 

* Add TODOs for GetCombinedCommitStatusByRef

* Fix models/issue_stopwatch.go

* Rrefactor models.ListDeployKeys

* Introduce helper func and use them for SetLinkHeader related func
2021-08-12 14:43:08 +02:00
nitul1991 2d25b7d44b
Add an api endpoint to fetch git notes (#15373) (#16649)
close #15373
2021-08-11 03:01:40 +02:00
6543 c4d70a0325
Rename ctx.Form() to ctx.FormString() and move code into own file (#16571)
Followup from #16562 prepare for #16567

* Rename ctx.Form() to ctx.FormString()
* Reimplement FormX func to need less code and cpu cycles
* Move code into own file
2021-08-11 02:31:13 +02:00
6543 89245ee309
Upgrade github.com/google/go-github v32.1.0 -> v37.0.0 (#16661)
* Upgrade github.com/google/go-github vv32.1.0 -> v37.0.0

* refactor: use GetX() func to reduce code
2021-08-10 11:49:43 +02:00
KN4CK3R d9ef43a712
Replace list.List with slices (#16311)
* Replaced list with slice.

* Fixed usage of pointer to temporary variable.

* Replaced LIFO list with slice.

* Lint

* Removed type check.

* Removed duplicated code.

* Lint

* Fixed merge.

Co-authored-by: 6543 <6543@obermui.de>
2021-08-09 14:08:51 -04:00
nitul1991 23d438f565
Change the implementation of the go-git version of GetNote to mirror the non go-git version when passed a non-existent commit (#16658)
Fixes #16657
2021-08-09 16:24:34 +01:00
Romain 14762abf0b
Separate open and closed issue in metrics (#16637)
* Get the issue counts in one query

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-08-07 12:43:50 +03:00
zeripath afd88a2418
Allow setting X-FRAME-OPTIONS (#16643)
* Allow setting X-FRAME-OPTIONS

This PR provides a mechanism to set the X-FRAME-OPTIONS header.

Fix #7951

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

Co-authored-by: John Olheiser <john.olheiser@gmail.com>

Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2021-08-06 16:47:10 -04:00
Richard Mahn fd39e6dc6a
Fixes #16557 - duplicate csv import (#16631)
Remove the unnecessary duplicate package import for `encoding/csv` in `modules/csv/csv.go`

Fix #16557
2021-08-05 17:56:11 +01:00
zeripath 7c4172ef71
Pass down SignedUserName down to AccessLogger context (#16605)
* Pass down SignedUserName down to AccessLogger context

Unfortunately when the AccessLogger was moved back before the contexters the
SignedUserName reporting was lost. This is due to Request.WithContext leading to a
shallow copy of the Request and the modules/context/Context being within that request.

This PR adds a new context variable of a string pointer which is set and handled
in the contexters.

Fix #16600

Signed-off-by: Andrew Thornton <art27@cantab.net>

* handle nil ptr issue

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-08-04 13:26:30 -04:00
zeripath 1fc7d6d6ad
Fix table alignment in markdown (#16596)
Set the TableOptions in markdown to allow alignment of the tables to work correctly

Fix #15959

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-02 17:11:18 +01:00
Lunny Xiao 33e0b38287
Rename context.Query to context.Form (#16562) 2021-07-29 03:42:15 +02:00
a1012112796 3705168837
Add agit flow support in gitea (#14295)
* feature: add agit flow support

ref: https://git-repo.info/en/2020/03/agit-flow-and-git-repo/

example:

```Bash
git checkout -b test
echo "test" >> README.md
git commit -m "test"
git push origin HEAD:refs/for/master -o topic=test
```

Signed-off-by: a1012112796 <1012112796@qq.com>

* fix lint

* simplify code add fix some nits

* update merge help message

* Apply suggestions from code review. Thanks @jiangxin

* add forced-update message

* fix lint

* splite writePktLine

* add refs/for/<target-branch>/<topic-branch> support also

* Add test code add fix api

* fix lint

* fix test

* skip test if git version < 2.29

* try test with git 2.30.1

* fix permission check bug

* fix some nit

* logic implify and test code update

* fix bug

* apply suggestions from code review

* prepare for merge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix permission check bug

- test code update
- apply suggestions from code review @zeripath

Signed-off-by: a1012112796 <1012112796@qq.com>

* fix bug when target branch isn't exist

* prevent some special push and fix some nits

* fix lint

* try splite

* Apply suggestions from code review

- fix permission check
- handle user rename

* fix version negotiation

* remane

* fix template

* handle empty repo

* ui: fix  branch link under the title

* fix nits

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-07-28 17:42:56 +08:00
zeripath fd15fd4c67
Handle too long PR titles correctly (#16517)
The CompareAndPullRequestPost handler for POST to /compare
incorrectly handles returning errors to the user. For a start
it does not set the necessary markers to switch SimpleMDE
but it also does not immediately return to the form.

This PR fixes this by setting the appropriate values, fixing
the templates and preventing the suggestion of a too long
title.

Fix #16507

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-25 03:59:27 +01:00
Lunny Xiao 9f31f3aa8a
Add an abstract json layout to make it's easier to change json library (#16528)
* Add an abstract json layout to make it's easier to change json library

* Fix import

* Fix import sequence

* Fix blank lines

* Fix blank lines
2021-07-24 18:03:58 +02:00
techknowlogick 29a22ade82
switch to maintained lib (#16532)
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-07-24 13:00:41 +02:00
zeripath 5d2e11eedb
Refactor: Move login out of models (#16199)
`models` does far too much. In particular it handles all `UserSignin`.

It shouldn't be responsible for calling LDAP, SMTP or PAM for signing in.

Therefore we should move this code out of `models`.

This code has to depend on `models` - therefore it belongs in `services`.

There is a package in `services` called `auth` and clearly this functionality belongs in there.

Plan:

- [x] Change `auth.Auth` to `auth.Method` - as they represent methods of authentication.
- [x] Move `models.UserSignIn` into `auth`
- [x] Move `models.ExternalUserLogin`
- [x] Move most of the `LoginVia*` methods to `auth` or subpackages
- [x] Move Resynchronize functionality to `auth`
  - Involved some restructuring of `models/ssh_key.go` to reduce the size of this massive file and simplify its files.
- [x] Move the rest of the LDAP functionality in to the ldap subpackage
- [x] Re-factor the login sources to express an interfaces `auth.Source`?
  - I've done this through some smaller interfaces Authenticator and Synchronizable - which would allow us to extend things in future
- [x] Now LDAP is out of models - need to think about modules/auth/ldap and I think all of that functionality might just be moveable
- [x] Similarly a lot Oauth2 functionality need not be in models too and should be moved to services/auth/source/oauth2
  - [x] modules/auth/oauth2/oauth2.go uses xorm... This is naughty - probably need to move this into models.
  - [x] models/oauth2.go - mostly should be in modules/auth/oauth2 or services/auth/source/oauth2 
- [x] More simplifications of login_source.go may need to be done
- Allow wiring in of notify registration -  *this can now easily be done - but I think we should do it in another PR*  - see #16178 
- More refactors...?
  - OpenID should probably become an auth Method but I think that can be left for another PR
  - Methods should also probably be cleaned up  - again another PR I think.
  - SSPI still needs more refactors.* Rename auth.Auth auth.Method
* Restructure ssh_key.go

- move functions from models/user.go that relate to ssh_key to ssh_key
- split ssh_key.go to try create clearer function domains for allow for
future refactors here.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-24 11:16:34 +01:00
zeripath f135a818f5
Make Mermaid.js limit configurable (#16519)
* Make Mermaid.js limit configurable

Add `MERMAID_MAX_SOURCE_CHARACTERS` to `[markup]` settings
to make the maximum size of a mermaid render configurable.

Fix #16513

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fixup! Make Mermaid.js limit configurable

* Update custom/conf/app.example.ini

Co-authored-by: silverwind <me@silverwind.io>

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

Co-authored-by: silverwind <me@silverwind.io>

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-07-24 00:21:51 -04:00
lengyuqu 078e2b2c39
Add support for corporate WeChat webhooks (#15910)
* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* Update templates/admin/hook_new.tmpl

Co-authored-by: a1012112796 <1012112796@qq.com>

* Update services/webhook/wechatwork.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* 修善wechatwork

* 修善wechatwork

* fix

* Update locale_cs-CZ.ini

fix

* fix build

* fix

* fix build

* make webhooks.zh-cn.md

* delet unnecessary blank line

* delet unnecessary blank line

* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* Update templates/admin/hook_new.tmpl

Co-authored-by: a1012112796 <1012112796@qq.com>

* Update services/webhook/wechatwork.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* 修善wechatwork

* 修善wechatwork

* fix

* fix build

* fix

* fix build

* make webhooks.zh-cn.md

* delet unnecessary blank line

* delet unnecessary blank line

* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* fix

* fix

* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* fix wechat

* fix wechat

* fix wechat

* fix wechat

* Fix invalid params and typo of email templates (#16394)

Signed-off-by: Meano <meanocat@gmail.com>

* Add LRU mem cache implementation (#16226)

The current default memory cache implementation is unbounded in size and number of
objects cached. This is hardly ideal.

This PR proposes creating a TwoQueue LRU cache as the underlying cache for Gitea.
The cache is limited by the number of objects stored in the cache (rather than size)
for simplicity. The default number of objects is 50000 - which is perhaps too small
as most of our objects cached are going to be much less than 1kB.

It may be worth considering using a different LRU implementation that actively limits
sizes or avoids GC - however, this is just a beginning implementation.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* [skip ci] Updated translations via Crowdin

* Replace `plugins/docker` with `techknowlogick/drone-docker`in ci (#16407)

* plugins/docker -> techknowlogick/drone-docker

* It is multi-arch

* docs: rewrite email setup (#16404)

* Add intro for both the docs page and mailer methods
  * Fix numbering level in SMTP section
  * Recommends implicit TLS

Signed-off-by: Bagas Sanjaya <bagasdotme@gmail.com>

* Validate Issue Index before querying DB (#16406)

* Fix external renderer (#16401)

* fix external renderer

* use GBackground context as fallback

* no fallback, return error

Co-authored-by: Lauris BH <lauris@nix.lv>

* Add checkbox to delete pull branch after successful merge (#16049)

* Add checkbox to delete pull branch after successful merge

* Omit DeleteBranchAfterMerge field in json

* Log a warning instead of error when PR head branch deleted

* Add DefaultDeleteBranchAfterMerge to PullRequestConfig

* Add support for delete_branch_after_merge via API

* Fix for API: the branch should be deleted from the HEAD repo

If head and base repo are the same, reuse the already opened ctx.Repo.GitRepo

* Don't delegate to CleanupBranch, only reuse branch deletion code

CleanupBranch contains too much logic that has already been performed by the Merge

* Reuse gitrepo in MergePullRequest

Co-authored-by: Andrew Thornton <art27@cantab.net>

* [skip ci] Updated translations via Crowdin

* Detect encoding changes while parsing diff (#16330)

* Detect encoding changes while parsing diff

* Let branch/tag name be a valid ref to get CI status (#16400)

* fix #16384#

* refactor: move shared helper func to utils package

* extend Tests

* use ctx.Repo.GitRepo if not nil

* fix

* fix

* 企业微信webhook

* 企业微信webhook

* 企业微信webhook

* fix build

* fix build

* Apply suggestions from code review

Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: myheavily <myheavily>
Co-authored-by: zhaoxin <gitea@fake.local>
Co-authored-by: Meano <Meano@foxmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: GiteaBot <teabot@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Bagas Sanjaya <bagasdotme@gmail.com>
Co-authored-by: Norwin <noerw@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Jimmy Praet <jimmy.praet@telenet.be>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-07-23 12:41:27 +08:00
zeripath 1ce4fb256f
Restore creation of git-daemon-export-ok files (#16508)
Somewhere along the line the creation of git-daemon-export-ok
files disappeared but the updating of these files when
repo visibility changes remained. The problem is that the
current state will create files even when the org or user
is private.

This PR restores creation correctly.

Fix #15521

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-22 12:53:54 +01:00
zeripath 28f6f7bb03
Restore CORS on git smart http protocol (#16496)
Unfortunately the chi changes have resulted in the CORS headers for the
git smart http protocol going missing.

This is mostly because the OPTIONS method is not being handled by
httpBase anymore.

This PR adds a GetOptions, PostOptions and Options methods to web
handler to allow OPTIONS method requests to still reach the httpBase
function.

Fix #16350
Close #16491

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-07-21 11:32:35 +08:00
zeripath 49bd9a1111
Fix race in log (#16490)
A race has been detected in #1441 relating to getting log levels.

This PR protects the GetLevel and GetStacktraceLevel calls with a RW mutex.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-20 20:09:29 +01:00
zeripath 97381aad5d
Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479)
Fix #16427 (again!)

* handle sharing violation error code

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-20 20:23:01 +03:00
6543 b26c3b482f
Add TestPrepareWikiFileName (#16487)
* Add TestPrepareWikiFileName

* use LsTree as LsFiles is index only

* ajust other tests

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-07-20 15:16:20 +02:00
Lunny Xiao 43262226db
Fix data race in bleve indexer (#16474)
* Fix data race in bleve indexer
2021-07-18 21:59:10 +02:00
Lars Lehtonen 2344007fa3
modules/markup/markdown: fix dropped test error (#16438) 2021-07-18 14:09:34 +02:00
zeripath e83abfc289
Prevent race in TestPersistableChannelQueue (#16468)
* Prevent race in TestPersistableChannelQueue

A slight race has become apparent in the TestPersistableChannelQueue.

This PR simply adds locking to prevent the race.

* make print value of "$(GOTESTFLAGS)" on test-backend and unit-test-coverage


Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-07-17 19:09:56 +02:00
zeripath b08e14bbcf
Retry rename on lock induced failures (re-fix) (#16461)
Unfortunately #16435 asserts the wrong error and should use
os.LinkError not os.PathError.

Fix #16439

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-16 19:16:04 +02:00
Josef Fröhle 8d962daed6
cleanup code issueFullPattern in modules/markup (#16419)
fix #16415
2021-07-15 22:33:56 +02:00
Richard Nienaber 908136c557
add configuration option to restrict users by default (#16256)
* add configuration option to restrict users by default

* default IsRestricted permission only set on sign up

setting this in the model messes with other workflows (e.g. syncing LDAP users) where the IsRestricted permission needs to be explicitly set and not overridden by a config value

* fix formatting

* Apply suggestions from code review

* ensure newly created user is set to restricted

* ensure imports are in the correct order

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-07-15 15:19:48 -04:00
Jimmy Praet 8df3d6575a
Change @every 24h default schedules to @midnight (#16431) 2021-07-15 11:55:48 -04:00
zeripath 33a8eec33e
Retry rename on lock induced failures (#16435)
* Retry rename on lock induced failures

Due to external locking on Windows it is possible for an
os.Rename to fail if the files or directories are being
used elsewhere.

This PR simply suggests retrying the rename again similar
to how we handle the os.Remove problems.

Fix #16427

Signed-off-by: Andrew Thornton <art27@cantab.net>

* resolve CI fail

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-07-15 11:46:07 -04:00
zeripath 3dcb3e9073
Second attempt at preventing zombies (#16326)
* Second attempt at preventing zombies

* Ensure that the pipes are closed in ssh.go
* Ensure that a cancellable context is passed up in cmd/* http requests
* Make cmd.fail return properly so defers are obeyed
* Ensure that something is sent to stdout in case of blocks here

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint 2

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint 3

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fixup

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Apply suggestions from code review

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-14 10:43:13 -04:00
Stanley Hu 3dba75fb97
Support HTTP/2 in Let's Encrypt (#16371)
Modify the tlsConfig.NextProtos for Let's Encrypt and built-in HTTPS server in order to support HTTP/2.

Co-authored-by: 6543 <6543@obermui.de>
2021-07-13 18:17:46 +01:00
zeripath b82293270c
Add option to provide signature for a token to verify key ownership (#14054)
* Add option to provide signed token to verify key ownership

Currently we will only allow a key to be matched to a user if it matches
an activated email address. This PR provides a different mechanism - if
the user provides a signature for automatically generated token (based
on the timestamp, user creation time, user ID, username and primary
email.

* Ensure verified keys can act for all active emails for the user

* Add code to mark keys as verified

* Slight UI adjustments

* Slight UI adjustments 2

* Simplify signature verification slightly

* fix postgres test

* add api routes

* handle swapped primary-keys

* Verify the no-reply address for verified keys

* Only add email addresses that are activated to keys

* Fix committer shortcut properly

* Restructure gpg_keys.go

* Use common Verification Token code

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-13 15:28:07 +02:00
Lunny Xiao 67f135ca5d
Fix archive error when rename repo or user (#16399)
Use repo id instead of full name to generate archive path
2021-07-13 14:16:31 +02:00
Jimmy Praet 78118a3b02
Add checkbox to delete pull branch after successful merge (#16049)
* Add checkbox to delete pull branch after successful merge

* Omit DeleteBranchAfterMerge field in json

* Log a warning instead of error when PR head branch deleted

* Add DefaultDeleteBranchAfterMerge to PullRequestConfig

* Add support for delete_branch_after_merge via API

* Fix for API: the branch should be deleted from the HEAD repo

If head and base repo are the same, reuse the already opened ctx.Repo.GitRepo

* Don't delegate to CleanupBranch, only reuse branch deletion code

CleanupBranch contains too much logic that has already been performed by the Merge

* Reuse gitrepo in MergePullRequest

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-07-13 01:26:25 +02:00
6543 46a4c6835d
Fix external renderer (#16401)
* fix external renderer

* use GBackground context as fallback

* no fallback, return error

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-12 17:13:59 -04:00
zeripath 2f725cbc9e
Add LRU mem cache implementation (#16226)
The current default memory cache implementation is unbounded in size and number of
objects cached. This is hardly ideal.

This PR proposes creating a TwoQueue LRU cache as the underlying cache for Gitea.
The cache is limited by the number of objects stored in the cache (rather than size)
for simplicity. The default number of objects is 50000 - which is perhaps too small
as most of our objects cached are going to be much less than 1kB.

It may be worth considering using a different LRU implementation that actively limits
sizes or avoids GC - however, this is just a beginning implementation.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-10 23:54:15 +02:00
6543 91162bbaea
Update bluemonday to v1.0.15 (#16379)
* update github.com/microcosm-cc/bluemonday

* add exec flag to contrib/update_dependencies.sh

* Fix TESTS
2021-07-09 03:30:31 +02:00
zeripath d06f9ce274
Redirect on bad CSRF instead of presenting bad page (#14937)
The current CSRF handler is a bit harsh with bad CSRF tokens on webpages
I think we can be a little kinder and redirect to base page with a flash error

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-08 15:57:24 +02:00
luzpaz e0296b6a6d
Fix various documentation, user-facing, and source comment typos (#16367)
* Fix various doc, user-facing, and source comment typos

Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
2021-07-08 13:38:13 +02:00
a1012112796 5bb97a12d7
Creating a repo from a template repo via API (#15958)
* Creating a repo from a template repo via API

fix #15934
ref:
https://docs.github.com/en/rest/reference/repos#create-a-repository-using-a-template

Signed-off-by: a1012112796 <1012112796@qq.com>
2021-07-05 17:29:08 +02:00
zeripath 32fd11395b
Fix relative links in postprocessed images (#16334)
If a pre-post-processed file contains relative img tags these need to be updated
and joined correctly with the prefix. Finally, the node attributes need to be updated.

Fix #16308

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-07-04 10:26:04 +01:00
zeripath 62c278e4ab
Fix modified files list in webhooks when there is a space (#16288)
* Fix modified files list in webhooks when there is a space

There is an unfortunate bug with GetCommitFileStatus where files with
spaces are misparsed and split at the space.

There is a second bug because modern gits detect renames meaning that
this function no longer works correctly.

There is a third bug in that merge commits don't have their modified
files detected correctly.

Fix #15865


Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-02 21:23:37 +02:00
6543 836884429a
Add forge emojies (#16296)
* codeberg :codeberg:
* gitlab :gitlab:
* git :git:
* github :github:
* gogs :gogs:
2021-07-02 16:04:57 +02:00
6543 dfa18a8b1c
Introduce NotifySubjectType (#16320)
* Introduce NotifySubjectType

* update swagger docs
2021-07-01 12:51:24 +02:00
zeripath 9979983283
Update Go-Git to take advantage of LargeObjectThreshold (#16316)
Following the merging of https://github.com/go-git/go-git/pull/330 we
can now add a setting to avoid go-git reading and caching large objects.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 22:58:45 +02:00
zeripath 302e8b6d02
Prevent zombie processes (#16314)
Unfortunately go doesn't always ensure that execd processes are completely
waited for. On linux this means that zombie processes can occur.

This PR ensures that these are waited for by using signal notifier in serv and
passing a context elsewhere.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 22:07:23 +02:00
KN4CK3R 7d70a6eff8
Fix webhook commits wrong hash on HEAD reset (#16283)
Use `..` instead of `...` with `rev-list`. In combination with #16282 the receiver can get the correct commit. The behaviour is now like Github.

fixes #11802
2021-06-30 19:49:06 +02:00
Lunny Xiao 0966349354
Make the github migration less rate limit waiting to get comment per page from repository but not per issue (#16070)
* Make the github migration less rate limit waiting to get comment per page from repository but not per issue

* Fix lint

* adjust Downloader interface

* Fix missed reviews

* Fix test

* Remove unused struct
2021-06-30 15:23:49 +08:00
zeripath add74fb368
Fix panic in recursive cache (#16298)
There is a bug with last commit cache recursive cache where the last
commit information that refers to the current tree itself will cause a
panic due to its path ("") not being included in the expected tree entry
paths.

This PR fixes this by skipping the missing entry.

Fix #16290

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-29 16:12:43 -04:00
6543 65548359cc
Add custom emoji support (#16004) 2021-06-29 16:28:38 +02:00
KN4CK3R aac663e0da
Implemented head_commit for webhooks (#16282)
* Removed Len field.

* Added head_commit webhook field.

* Added comment for returns.
2021-06-29 15:34:03 +02:00
zeripath f825f20d49
Upgrade Gliderlabs SSH to 0.3.3 and add FailedConnectionCallback (#16278)
* Upgrade Gliderlabs SSH to 0.3.3 and add FailedConnectionCallback

Following the merging of https://github.com/gliderlabs/ssh/pull/143 we
can now report connections to the ssh server that have failed before
public key exchange has completed using the standard fail2ban message.

This PR updates Gliderlabs SSH and adds a callback that will provide this
logging.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* move the callback to its own function to make the logging appear little nicer

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-28 13:05:27 -04:00
6543 59c58553ba
Fix API to return author for author on commits(#16276) 2021-06-27 23:15:42 +01:00
KN4CK3R 9b1b4b5433
Refactor Webhook + Add X-Hub-Signature (#16176)
This PR removes multiple unneeded fields from the `HookTask` struct and adds the two headers `X-Hub-Signature` and `X-Hub-Signature-256`.

## ⚠️ BREAKING ⚠️ 

* The `Secret` field is no longer passed as part of the payload.
* "Breaking" change (or fix?): The webhook history shows the real called url and not the url registered in the webhook (`deliver.go`@129).

Close #16115
Fixes #7788
Fixes #11755

Co-authored-by: zeripath <art27@cantab.net>
2021-06-27 20:21:09 +01:00
6543 0b27b93728
Make allowed Visiblity modes configurable for Users (#16271)
Now that #16069 is merged, some sites may wish to enforce that users are all public, limited or private, and/or disallow users from becoming private.

This PR adds functionality and settings to constrain a user's ability to change their visibility.

Co-authored-by: zeripath <art27@cantab.net>
2021-06-27 19:47:35 +01:00
Steven f533b5d5cf
Make app.ini more restrictive on new installations (#16266)
Signed-off-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>
2021-06-27 11:07:36 +01:00
zeripath 35f37a3625
Add --quiet and --verbose to gitea web to control initial logging (#16260)
One of the repeatedly reported issues has been that gitea produces too much console
logging during set up even if the console logger is turned off.

Fundamentally this is due to some otherwise very helpful logging that has to occur
before logging is set up. This has come to a head with the merging of #16243 where
otherwise potentially helpful Trace logging in the git module now appears on the
console.

This PR proposes three things:

1. Change the initial default logger to Info not Trace.
2. Change the logging for the AppPath things to Info in recompense.
3. Add two new command line options to gitea web: --quiet and --verbose

`gitea web -q` or `gitea web --quiet` will only log Fatal level initially.
`gitea web -verbose` will log at Trace.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-26 20:56:58 -04:00
Sergey Dryabzhinsky 22a0636544
Add Visible modes function from Organisation to Users too (#16069)
You can limit or hide organisations. This pull make it also posible for users

- new strings to translte
- add checkbox to user profile form
- add checkbox to admin user.edit form
- filter explore page user search
- filter api admin and public user searches
- allow admins view "hidden" users
- add app option DEFAULT_USER_VISIBILITY
- rewrite many files to use Visibility field
- check for teams intersection
- fix context output
- right fake 404 if not visible

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-06-26 20:53:14 +01:00
zeripath 19ac575d57
Limit stdout tracelog to actual stdout (#16258)
Related #16243

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-26 13:47:56 +01:00
Lunny Xiao e3c626834b
Let package git depend on setting but not opposite (#15241)
* Let package git depend on setting but not opposite

* private some package variables
2021-06-26 13:28:55 +02:00
zeripath 62a4879e84
Improve efficiency in FindRenderizableReferenceNumeric and getReferences (#16251)
* Fuzzer finds an NPE due to incorrect URLPrefix

The Fuzzer is running on a non-repo urlprefix which is incorrect for RenderRaw


* Make FindRenderizableReferenceNumeric and getReferences more efficient

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-26 01:11:33 +02:00
ayb 9b33d18899
Added support for gopher URLs. (#14749)
* Added support for gopher URLs.

* Add setting and make this user settable instead

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-06-25 18:38:27 -04:00
6543 3ef23d5411
Use gitea logging module for git module (#16243)
remove log() func from gogs times and switch to proper logging

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-06-25 17:54:08 +01:00
KN4CK3R 44b8b07631
Add tag protection (#15629)
* Added tag protection in hook.

* Prevent UI tag creation if protected.

* Added settings page.

* Added tests.

* Added suggestions.

* Moved tests.

* Use individual errors.

* Removed unneeded methods.

* Switched delete selector.

* Changed method names.

* No reason to be unique.

* Allow editing of protected tags.

* Removed unique key from migration.

* Added docs page.

* Changed date.

* Respond with 404 to not found tags.

* Replaced glob with regex pattern.

* Added support for glob and regex pattern.

* Updated documentation.

* Changed white* to allow*.

* Fixed edit button link.

* Added cancel button.

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-06-25 16:28:55 +02:00
Mura Li fd6b1be1b6
Replace ARCCache with TwoQueueCache to avoid patent issue (#16240)
Co-authored-by: Mura Li <typeless@users.noreply.github.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-24 14:37:07 -04:00
KN4CK3R 4cc63e9919
Fix diff expansion is missing final line in a file (#16222)
* Fixed down offset.

* Fixed wrong line count result.
2021-06-24 17:47:46 +02:00
Lunny Xiao b223d36195
Rework repository archive (#14723)
* Use storage to store archive files

* Fix backend lint

* Add archiver table on database

* Finish archive download

* Fix test

* Add database migrations

* Add status for archiver

* Fix lint

* Add queue

* Add doctor to check and delete old archives

* Improve archive queue

* Fix tests

* improve archive storage

* Delete repo archives

* Add missing fixture

* fix fixture

* Fix fixture

* Fix test

* Fix archiver cleaning

* Fix bug

* Add docs for repository archive storage

* remove repo-archive configuration

* Fix test

* Fix test

* Fix lint

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 17:12:38 -04:00
KN4CK3R c9c7afda1a
Add sanitizer rules per renderer (#16110)
* Added sanitizer rules per renderer.

* Updated documentation.

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 17:09:51 -04:00
Viktor Yakovchuk 08f4b3f312
Fix 500 Error with branch and tag sharing the same name #15592 (#16040)
* Fix 500 Error with branch and tag sharing the same name #15592

Fixed 500 error while create Pull request when there are more
than one sources (branch, tag) with the same name

Fix #15592

Signed-off-by: Viktor Yakovchuk <viktor@yakovchuk.net>

* fix logging

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-23 17:08:26 -04:00
Lunny Xiao f2babf3346
Add code block highlight to orgmode back (#14222)
Fix missed orgmode code block hightlight

Co-authored-by: zeripath <art27@cantab.net>
2021-06-23 22:45:47 +02:00
6543 58501a2682
[API] GET / SET User Settings (#16169)
* API: GET/SET User Settings

* linter

* Apply suggestions from code review

* Update modules/structs/user.go

* lint

* fix swagger

* move User2UserSettings to convert

* as per @zeripath "preferences" -> "settings"

Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 15:58:44 -04:00
KN4CK3R 383ffcfa34
Small refactoring of modules/private (#15947)
* Use correct variable name.

* doer is never nil here.

* Use status code constants.

* Replaced generic map with concrete struct.

* Fixed windows lint.

* Removed unused method.

* Changed error codes.

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 15:38:19 -04:00
KN4CK3R 31f6b95dfa
Use data path instead of config path (#16227) 2021-06-23 16:56:25 +02:00
Jimmy Praet 17030ced75
Improve notifications for WIP draft PR's (#14663)
* #14559 Reduce amount of email notifications for WIP draft PR's

don't notify repo watchers of WIP draft PR's

* #13190 Notification when WIP Pull Request is ready for review

* Send email notification to repo watchers when WIP PR is created

* Send ui notification to repo watchers when WIP PR is created

* send specific email notification when PR is marked ready for review

instead of reusing the CreatePullRequest action

* Fix lint error

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 00:14:22 -04:00
zeripath d55b5eb0d3
Use html.Parse rather than html.ParseFragment (#16223)
* Use html.Parse rather than html.ParseFragment
  There have been a few issues with html.ParseFragment - just use html.Parse instead.

* Skip document node

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-22 00:12:22 +02:00
zeripath 196593e2e9
More efficiently parse shas for shaPostProcessor (#16101)
* More efficiently parse shas for shaPostProcessor

The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.

This PR provides functions for both backends which are much faster to use.

Fix #16092

* Add ShaExistCache to RenderContext

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-21 00:39:12 +02:00
zeripath 23358bc55d
Use git log name-status in get last commit (#16059)
* Improve get last commit using git log --name-status

git log --name-status -c provides information about the diff between a
commit and its parents. Using this and adjusting the algorithm to use
the first change to a path allows for a much faster generation of commit
info.

There is a subtle change in the results generated but this will cause
the results to more closely match those from elsewhere.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-21 01:00:46 +03:00
parnic 35742d4af7
Reintroduce squash merge default comment as a config setting (#16134)
* Reinstate most of commit 09304db9a5

* Move the behaviour behind a config setting

* Also fix the initial #12365
2021-06-19 00:08:22 +02:00
6543 75205b5669
Fix some API bugs (#16184)
* Repository object only count releases as releases (fix #16144)

* EditOrg respect RepoAdminChangeTeamAccess option (fix #16013)
2021-06-18 01:24:55 +02:00
KN4CK3R 29695cd6d5
Add asymmetric JWT signing (#16010)
* Added asymmetric token signing.

* Load signing key from settings.

* Added optional kid parameter.

* Updated documentation.

* Add "kid" to token header.
2021-06-17 23:56:46 +02:00
6543 f7cd394680
[API] Add repoCreateTag (#16165)
* Add API CreateTag

* Add Test

* API: expose Tag Message
2021-06-17 18:04:10 +02:00
Mura Li 19dedc3fa5
Speed up git diff highlight generation (#16180)
Co-authored-by: Mura Li <typeless@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-17 10:55:16 -04:00
6543 b3fbd37e99
[API] expose repo.GetReviewers() & repo.GetAssignees() (#16168)
* API: expose repo.GetReviewers() & repo.GetAssignees()

* Add tests

* fix unrelated swagger query type
2021-06-17 16:02:34 +02:00
zeripath 0db1048c3a
Run processors on whole of text (#16155)
There is an inefficiency in the design of our processors which means that Emoji
and other processors run in order n^2 time.

This PR forces the processors to process the entirety of text node before passing
back up. The fundamental inefficiency remains but it should be significantly
ameliorated.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-17 11:35:05 +01:00
6543 6ad5d0a306
[API] ListReleases add filter for draft and pre-releases (#16175)
* invent ctx.QueryOptionalBool

* [API] ListReleases add draft and pre-release filter

* Add X-Total-Count header

* Add a release to fixtures

* Add TEST for API ListReleases
2021-06-17 10:58:10 +02:00
6543 c9d053f0ca
[API] User expose counters (#16167) 2021-06-17 09:17:35 +02:00
zeripath 1ec9e906dc
Ensure settings for Service and Mailer are read on the install page (#15943)
* Ensure settings for Service and Mailer are read on the install page

NewContext does not set the mailer or service settings so add
a new function that will run this.

Fix #15894

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-16 19:32:57 -04:00
zeripath ffbf35b7e9
Clean-up the settings hierarchy for issue_indexer queue (#16001)
There are a couple of settings in `[indexer]` relating to the `issue_indexer` queue
which override settings in unpredictable ways. This PR adjusts this hierarchy and makes
explicit that these settings are deprecated.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-16 18:19:20 -04:00
zeripath 5f4522cd1f
Fix missing discard in repo_language_stats (#16030)
Set the missing discard(1) in repo_language_stats.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-16 18:16:47 -04:00
zeripath 6d69df2804
Add Status Updates whilst Gitea migrations are occurring (#15076)
* Add migrating message

Signed-off-by: Andrew Thornton <art27@cantab.net>

* simplify messenger

Signed-off-by: Andrew Thornton <art27@cantab.net>

* make messenger an interface

Signed-off-by: Andrew Thornton <art27@cantab.net>

* rename

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prepare for merge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per tech

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-16 18:02:24 -04:00
zeripath f4d3bf7867
issue-keyword class is being incorrectly stripped off spans (#16163)
Bluemonday sanitizer regexp rules are not additive, so the addition of the icons,
emojis and chroma syntax policy has led to this being stripped.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-16 03:02:03 +02:00
zeripath 95352e6bd5
Make command in authorized keys a template (#16003)
Fix #15595
Replaces #15978

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-14 21:15:12 -04:00
KN4CK3R 440039c0cc
Add push to remote mirror repository (#15157)
* Added push mirror model.

* Integrated push mirror into queue.

* Moved methods into own file.

* Added basic implementation.

* Mirror wiki too.

* Removed duplicated method.

* Get url for different remotes.

* Added migration.

* Unified remote url access.

* Add/Remove push mirror remotes.

* Prevent hangs with missing credentials.

* Moved code between files.

* Changed sanitizer interface.

* Added push mirror backend methods.

* Only update the mirror remote.

* Limit refs on push.

* Added UI part.

* Added missing table.

* Delete mirror if repository gets removed.

* Changed signature. Handle object errors.

* Added upload method.

* Added "upload" unit tests.

* Added transfer adapter unit tests.

* Send correct headers.

* Added pushing of LFS objects.

* Added more logging.

* Simpler body handling.

* Process files in batches to reduce HTTP calls.

* Added created timestamp.

* Fixed invalid column name.

* Changed name to prevent xorm auto setting.

* Remove table header im empty.

* Strip exit code from error message.

* Added docs page about mirroring.

* Fixed date.

* Fixed merge errors.

* Moved test to integrations.

* Added push mirror test.

* Added test.
2021-06-14 19:20:43 +02:00
a1012112796 a005265718
small refactor for retry downloader (#16137)
Signed-off-by: a1012112796 <1012112796@qq.com>
2021-06-13 16:59:19 +02:00
zeripath daa5a23548
Set self-adjusting deadline for connection writing (#16068)
* Set self-adjusting deadline for connection writing

In #16055 it appears that the simple 5s deadline doesn't work for large
file writes. Now we can't - or at least shouldn't just set no deadline
as go will happily let these connections block indefinitely. However,
what seems reasonable is to set some minimum rate we expect for writing.

This PR suggests the following algorithm:

* Every write has a minimum timeout of 5s (adjustable at compile time.)
* If there has been a previous write - then consider its previous
deadline, add half of the minimum timeout + 2s per kb about to written.
* If that new deadline is after the minimum timeout use that.

Fix #16055

* Linearly increase timeout

* Make PerWriteTimeout, PerWritePerKbTimeouts configurable

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-11 00:25:25 +03:00
Lunny Xiao fb3ffeb18d
Add sso.Group, context.Auth, context.APIAuth to allow auth special routes (#16086)
* Add sso.Group, context.Auth, context.APIAuth to allow auth special routes

* Remove unnecessary check

* Rename sso -> auth

* remove unused method of Auth interface
2021-06-09 19:53:16 +02:00
KN4CK3R 21cde5c439
Fix data URI scramble (#16098)
* Removed unused method.

* No prefix for data uris.

* Added test to prevent regressions.
2021-06-07 18:55:26 +02:00
KN4CK3R 3607f79d78
Fixed assert statements. (#16089) 2021-06-07 07:27:09 +02:00
zeripath 51775f65bc
Make commit info cancelable (#16032)
* Make modules/context.Context a context.Context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Simplify context calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the base context for requests to the HammerContext

Signed-off-by: Andrew Thornton <art27@cantab.net>

* pass context into get-last-commit

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Make commit_info cancellable

Signed-off-by: Andrew Thornton <art27@cantab.net>

* use context as context

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-06 19:44:58 -04:00
Lunny Xiao b6762e2306
Fix regression of renderer (#16091)
* Fix regression of renderer

* Fix render setting load twice bug
2021-06-06 18:50:07 -04:00
KN4CK3R ee5e1c4a88
Rewrite of the LFS server (#15523)
* Restructured code. Moved static checks out of loop.

* Restructured batch api. Add support for individual errors.

* Let router decide if LFS is enabled.

* Renamed methods.

* Return correct status from verify handler.

* Unified media type check in router.

* Changed error code according to spec.

* Moved checks into router.

* Removed invalid v1 api methods.

* Unified methods.

* Display better error messages.

* Added size parameter. Create meta object on upload.

* Use object error on invalid size.

* Skip upload if object exists.

* Moved methods.

* Suppress fields in response.

* Changed error on accept.

* Added tests.

* Use ErrorResponse object.

* Test against message property.

* Add support for the old invalid lfs client.

* Fixed the check because MinIO wraps the error.

* Use individual repositories.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-06 02:59:27 +03:00
zeripath fe18a85f54
Fix panic (#16072)
There is an incorrect casting in the wrapped queue.

Fix #16071

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-05 15:23:22 +01:00
KN4CK3R 8e262104c2
Add Image Diff for SVG files (#14867)
* Added type sniffer.

* Switched content detection from base to typesniffer.

* Added GuessContentType to Blob.

* Moved image info logic to client.
Added support for SVG images in diff.

* Restore old blocked svg behaviour.

* Added missing image formats.

* Execute image diff only when container is visible.

* add margin to spinner

* improve BIN tag on image diffs

* Default to render view.

* Show image diff on incomplete diff.

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-05 15:32:19 +03:00
Lunny Xiao 7979c3654e
Copy git data from disk when restore repository (#16066) 2021-06-04 16:14:20 +03:00
Norwin cb940c4312
Encrypt migration credentials at rest (#15895)
* encrypt migration credentials in task persistence

Not sure this is the best approach, we could encrypt the entire
`PayloadContent` instead. Also instead of clearing individual fields in
payload content, we could just delete the task once it has
(successfully) finished..?

* remove credentials of past migrations

* only run DB migration for completed tasks

* fix binding

* add omitempty

* never serialize unencrypted credentials

* fix import order

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-31 04:25:47 -04:00
zeripath 256b1a3561
Fix bug in reverse proxy (#16026)
Unfortunately go panics you try to cast a nil interface{} as another primitive
therefore you need to check interfaces are not nil before casting.

Fix #16025

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-31 02:54:16 -04:00
zeripath 3183a465d7
Make modules/context.Context a context.Context (#16031)
* Make modules/context.Context a context.Context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Simplify context calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the base context for requests to the HammerContext

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-31 02:18:11 -04:00
Lunny Xiao effad26c0e
Improve assets handler middleware (#15961)
* Use route to serve assets but not middleware

* Fix build error with bindata tag

* convert path to absolute

* fix build

* reduce function stack

* Add tests for assets

* Remove test for assets because they are not generated

* Use a http function to serve assets

* Still use middleware to serve assets then less middleware stack for assets

* Move serveContent to original position

* remove unnecessary blank line change

* Fix bug for /assets* requests

* clean code

Co-authored-by: zeripath <art27@cantab.net>
2021-05-30 18:25:11 +08:00
Jimmy Praet d79c8bc302
Don't manipulate input params in email notification (#16011) 2021-05-30 10:38:38 +01:00
Lunny Xiao a9daebca22
Fix get tag when migration (#16014)
Co-authored-by: zeripath <art27@cantab.net>
2021-05-29 16:04:58 -04:00
KN4CK3R efe77eec85
Unified custom config creation (#16012)
* Unified custom config creation.

* Fixed log message.

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-29 19:44:14 +01:00
zeripath 6d6a65cf5c
Allow Token/Basic auth on raw paths (#15987)
It appears that people have been using token authentication to navigate to raw paths
and recent changes have broken this. Whilst ideally these paths would not be being used
like this - it was not the intention to be a breaking change.

This PR restores access to these paths.

Fix #13772

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-27 19:46:11 +01:00
zeripath 6d39053711
Fix setting of SameSite on cookies (#15989)
Fix #15972

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-26 21:01:07 -04:00
zeripath c1a80b7d6a
Use filepath.ToSlash and Join in indexer defaults and queues (#15971)
As revealed by #15964 there is inconsistent use of filepath Join and path Join
for these directories. The best thing to do is to use filepath.Join but then ToSlash
them for consistency.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2021-05-25 22:50:35 -04:00
zeripath 3aaf64885f
Change default queue settings to be low go-routines (#15964)
This PR suggests a change to the default configuration for queues:

* Use a common DATADIR for the queues
* Set starting workers to 0 and make boost a single worker

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-24 02:23:55 +03:00
KN4CK3R 6021fbfe7a
Make tasklist checkboxes clickable (#15791)
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-05-23 17:14:03 +03:00
silverwind be745be0a4
Double the avatar size factor (#15941)
* Double the avatar size factor

This results on finer Avatar rendering on Hi-DPI display.

* fix test

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-21 21:18:43 -04:00
a1012112796 fec8324026
add a new internal hook to save ssh log (#15787)
* add a new internal hook to save ssh log

as title, when a ssh error ocure like #15785.
only when switch ``RUN_MODE`` to dev can we
found which error is ocure. But this way is
not a good idea for production envirment.

this changes try save ssh error mesage to the
log file like other log by a new internal hook.
I think it's usefull for find error message
in production envirment. Thanks.

Signed-off-by: a1012112796 <1012112796@qq.com>

* rename and fix nit

* Update modules/private/hook.go

Co-authored-by: silverwind <me@silverwind.io>

Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-21 17:37:16 -04:00
zeripath 17be645498
Encrypt LDAP bind password in db with SECRET_KEY (#15547)
* Encrypt LDAP bind password in db with SECRET_KEY

The LDAP source bind password are currently stored in plaintext in the db
This PR simply encrypts them with the setting.SECRET_KEY.

Fix #15460

Signed-off-by: Andrew Thornton <art27@cantab.net>

* remove ui warning regarding unencrypted password

Co-authored-by: silverwind <me@silverwind.io>
2021-05-20 09:29:57 +01:00
Norwin a137ee7a68
git migration: don't prompt interactively for clone credentials (#15902)
* don't prompt interactively for clone credentials

* apply GIT_TERMINAL_PROMPT=0 to all git cmds

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-17 18:59:31 +08:00
6543 e0c93fed05
GitHub: migrate draft releases too (#15884)
* GitHub: migrate draft releases too

* refactor
2021-05-15 18:37:17 -04:00
zeripath 8e32eeb5de
Hold the event source when there are no listeners (#15725)
* Hold the event source when there are no listeners

The event source does not need to run when there are no listeners. Therefore
pause it when there are none.

* add some more logging

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 23:46:13 +02:00
zeripath f582ec4e53
Create a session on ReverseProxy and ensure that ReverseProxy users cannot change username (#15304)
* Create a session on ReverseProxy and ensure that ReverseProxy users cannot change username

ReverseProxy users should generate a session on reverse proxy username change.

Also prevent ReverseProxy users from changing their username.

Fix #2407

* add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 20:33:13 +02:00
zeripath 17c5c654a5
Prevent double-login for Git HTTP and LFS and simplify login (#15303)
* Prevent double-login for Git HTTP and LFS and simplify login

There are a number of inconsistencies with our current methods for
logging in for git and lfs. The first is that there is a double login
process. This is particularly evident in 1.13 where there are no less
than 4 hash checks for basic authentication due to the previous
IsPasswordSet behaviour.

This duplicated code had individual inconsistencies that were not
helpful and caused confusion.

This PR does the following:

* Remove the specific login code from the git and lfs handlers except
for the lfs special bearer token
* Simplify the meaning of DisableBasicAuthentication to allow Token and
Oauth2 sign-in.
* The removal of the specific code from git and lfs means that these
both now have the same login semantics and can - if not
DisableBasicAuthentication - login from external services. Further it
allows Oauth2 token authentication as per our standard mechanisms.
* The change in the recovery handler prevents the service from
re-attempting to login - primarily because this could easily cause a
further panic and it is wasteful.

* add test

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-05-15 17:32:09 +02:00
zeripath ba526ceffe
Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow level queue, reduce goroutines etc (#15693)
* move shutdownfns, terminatefns and hammerfns out of separate goroutines

Coalesce the shutdownfns etc into a list of functions that get run at shutdown
rather then have them run at goroutines blocked on selects.

This may help reduce the background select/poll load in certain
configurations.

* The LevelDB queues can actually wait on empty instead of polling

Slight refactor to cause leveldb queues to wait on empty instead of polling.

* Shutdown the shadow level queue once it is empty

* Remove bytefifo additional goroutine for readToChan as it can just be run in run

* Remove additional removeWorkers goroutine for workers

* Simplify the AtShutdown and AtTerminate functions and add Channel Flusher

* Add shutdown flusher to CUQ

* move persistable channel shutdown stuff to Shutdown Fn

* Ensure that UPCQ has the correct config

* handle shutdown during the flushing

* reduce risk of race between zeroBoost and addWorkers

* prevent double shutdown

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 16:22:26 +02:00
zeripath d0ea10b21f
Close the gitrepo when deleting the repository (#15876)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-14 21:19:38 +01:00
KN4CK3R df72cf6211
Fix LFS commit finder not working (#15856)
* Create a copy of the sha bytes.


Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-05-14 14:12:11 +01:00
zeripath 1a5659943e
Add timeout to writing to responses (#15831)
In #15826 it has become apparent that there are a few occasions when a response can
hang during writing, and because there is no timeout go will happily just block
interminably. This PR adds a fixed 5 second timeout to all writes to a connection.

Fix #15826

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-14 13:26:03 +01:00
zeripath 2d87a84709
Stop calling WriteHeader in Write (#15862)
Fixes http: superfluous response.WriteHeader call from code.gitea.io/gitea/modules/context.(*Response).WriteHeader (response.go:67)

* Looking again we don't need this writeHeader as all of our downstream
implementations will always do it for us

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-05-14 11:05:50 +03:00
mlpo 52f8dcda43
Allow custom highlight mapping beyond file extensions (#15808)
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-05-13 12:31:23 +03:00
yan 27b29ffb22
fix truncate utf8 string (#15828)
* fix truncate utf8 string.

* revoke truncated user info.
2021-05-13 08:50:57 +01:00
zeripath 2a565478d1
Tagger can be empty, as can Commit and Author - tolerate this (#15835)
Unfortunately some old repositories can have tags with empty Tagger, Commit
or Author. Go-Git variants will always have empty values for these whereas
the native git variant leaves them at nil. The simplest solution is just to
always have these set to empty Signatures.

v156 migration also makes the incorrect assumption that these cannot be empty.
Therefore add some handling to this and add logging and adjust broken
logging elsewhere in this migration.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-12 01:11:42 -04:00
KN4CK3R 3d7d750a99
Fix individual tests (addition to #15802) (#15818)
* Decouple TestAction_GetRepoLink and TestSizedAvatarLink.

* Load database for TestCheckGPGUserEmail.

* Load database for TestMakeIDsFromAPIAssigneesToAdd.

* Load database for TestGetUserIDsByNames and TestGetMaileableUsersByIDs.

* Load database for TestUser_ToUser.

* Load database for TestRepository_EditWikiPage.

* Include AppSubURL in test.

* Prevent panic with empty slice.

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-12 00:13:42 -04:00
zeripath aa65a607e4
Queue manager FlushAll can loop rapidly - add delay (#15733)
* Queue manager FlushAll can loop rapidly - add delay

Add delay within FlushAll to prevent rapid loop when workers are busy

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per lunny

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-05-12 00:22:08 +01:00
KN4CK3R f8335444de
Add Active and ProhibitLogin to API (#15689)
* Added active and prohibit_login.

* Do not omit fields for normal users.
2021-05-11 02:22:29 +02:00
Adam Szatyin d86d123322
Add mimetype mapping settings (#15133)
* Fix APK's Content-Type header

* Fix case sensitive comparison

* Add custom mime type mapping for downloadable files

* Add documentation for MIME type mapping

* Rename download.mimetype.mapping configuration to repository.mimetype_mapping

Co-authored-by: zeripath <art27@cantab.net>
2021-05-10 16:38:08 -04:00
John Olheiser 2f65c6b2f0
Add err to log (#15813)
Signed-off-by: jolheiser <john.olheiser@gmail.com>
2021-05-10 20:48:23 +01:00
Lunny Xiao e5723d6556
Move restore repo to internal router and invoke from command to avoid open the same db file or queues files (#15790)
* Move restore repo to internal router and invoke from command to avoid open the same db file or queues files

* Follow @zeripath's review

* set no timeout for resotre repo private request

* make restore repo cancelable
2021-05-10 15:57:45 +08:00
silverwind 1e6fa57acb
Use single shared random string generation function (#15741)
* Use single shared random string generation function

- Replace 3 functions that do the same with 1 shared one
- Use crypto/rand over math/rand for a stronger RNG
- Output only alphanumerical for URL compatibilty

Fixes: #15536

* use const string method

* Update modules/avatar/avatar.go

Co-authored-by: a1012112796 <1012112796@qq.com>

Co-authored-by: a1012112796 <1012112796@qq.com>
2021-05-10 07:45:17 +01:00
zeripath 270aab429e
On open repository open common cat file batch and batch-check (#15667)
Use common git cat-file --batch and git cat-file --batch-check to
significantly reduce calls to git.
    
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-10 03:27:03 +02:00
silverwind 17a7797cdd
Fix incorrect asset URL (#15805)
Fixes another regression from https://github.com/go-gitea/gitea/pull/15219.
2021-05-09 21:03:09 +01:00
Steven 2e0f315617
Respect default merge message syntax when parsing item references (#15772)
* Respect merge message structure for parsing item references

Signed-off-by: JustusBunsi <61625851+justusbunsi@users.noreply.github.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-05-09 19:25:23 +01:00
zeripath a2df265476
Add trace logging to SSO methods (#15803)
It is currenly impossible to detect which "SSO" method is responsible for login. This
PR adds some basic trace logging to these methods.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-09 18:04:53 +02:00
6543 4ea0b46d9b
Tests should use test files (#15801) 2021-05-09 17:20:33 +02:00
6543 a229e34387
Allow only internal registration (#15795)
* Add ALLOW_ONLY_INTERNAL_REGISTRATION into settings

* OpenID respect setting too
2021-05-09 16:13:35 +02:00
KN4CK3R e818e9150f
Fixed individual markdown tests. (#15802) 2021-05-09 15:29:49 +02:00
zeripath e22ee468cf
Exponential Backoff for ByteFIFO (#15724)
This PR is another in the vein of queue improvements. It suggests an
exponential backoff for bytefifo queues to reduce the load from queue
polling. This will mostly be useful for redis queues.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-05-08 17:29:47 +01:00
silverwind 24ad131221
Rename StaticUrlPrefix to AssetUrlPrefix (#15779)
Use a new name for this template/frontend variable to make it distinct
from the server variable StaticURLPrefix.
2021-05-08 16:27:25 +02:00
silverwind d4f28fd4ad
Fix URL of gitea emoji (#15770)
Fixes regression from #15219
2021-05-07 17:34:33 +02:00
6543 16034bb613
OAuth2 login: Set account link to "login" as default behavior (#15768) 2021-05-07 16:15:16 +02:00
zeripath 8f0539c235
Set GIT_DIR correctly if it is not set (#15751)
* Set GIT_DIR correctly if it is not set

* Expand out templates

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-07 15:19:09 +02:00
zeripath bdc1697729
Drop back to use IsAnInteractiveSession for SVC (#15749)
* Drop back to use IsAnInteractiveSession for SVC

There is an apparent permission change problem when using
IsWindowsService to determine if the SVC manager should be
used.

This PR simply drops back to using IsAnInteractiveSession as
this does not change behaviour.

Fix #15454

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Yes staticcheck I know this is deprecated

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Just leave me alone lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-05-07 11:27:31 +02:00
zeripath 114c85e6ef
Fix bug where repositories appear unadopted (#15757)
Fix bug where repositories with capital letters in their names appear unadopted.

Fix #15755

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-07 09:21:34 +08:00
a1012112796 3088866531
fix some ui bug about draft release (#15137)
* fix some ui bug about draft release

- should not show draft release in tag list because
  it will't create real tag
- still show draft release without tag and commit message
  for draft release instead of 404 error
- remove tag load for attachement links because it's useless

Signed-off-by: a1012112796 <1012112796@qq.com>

* add test code

* fix test

That's because has added a new release in relaese test database.

* fix dropdown link for draft release
2021-05-06 05:12:50 +02:00
zeripath eedc0c8324
Defer closing the gitrepo until the end of the wrapped context functions (#15653)
There was a mistake in #15372 where deferral of gitrepo close occurs before it should.

This PR fixes this.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-06 01:30:25 +02:00
zeripath 47fd156936
Use route rather than use thus reducing the number of stack frames (#15301)
Since the move to Chi the number of stack frames has proliferated somewhat catastrophically and we're up to 96 frames with multiple tests of the url outside of a trie which is inefficient.

This PR reduces the number of stack frames by 6 through careful use of Route, moves Captcha into its own router so that it only fires on Captcha routes, similarly for avatars and repo-avatars.

The robots.txt, / and apple-touch-icon.png are moved out of requiring Contexter.

It moves access logger higher in the stack frame because there is no reason why it can't be higher.

Extract from #15186
Contains #15292
2021-05-04 22:48:31 +01:00
zeripath 1b017fe7ca
Fix setting redis db path (#15698)
There is a bug setting the redis db in the common nosql manager whereby the db path
always fails.

This PR fixes this.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-03 13:24:24 -04:00
zeripath d11b9fbcce
Prevent race in TestChannelQueue_Batch (#15703)
There is a potential race in TestChannelQueue_Batch due to boost workers starting up

This PR simply removes the boosts from this test.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-03 17:16:59 +01:00
KN4CK3R 8c8471e754
Addition to (Add Location, Website and Description to API #15675) (#15690)
* Use same name as other structs.

* Sync with normal forms.

* Edit description with API.

* Workaround for nil value.
2021-05-02 21:03:15 +02:00
zeripath 0590176a23
Only use boost workers for leveldb shadow queues (#15696)
* The leveldb shadow queue of a persistable channel queue should always start with 0
workers and just use boost to add additional workers if necessary.

* create a zero boost so that if there are no workers in a pool - boost to start the workers

* actually set timeout appropriately on boosted workers

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-02 08:22:30 +01:00
zeripath a926ff919d
Performance improvement for last commit cache and show-ref (#15455)
* Improve performance when there are multiple commits in the last commit cache

* read refs directly if we can

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-02 03:16:08 +02:00
a1012112796 ae6d7860be
add cron job to delete old actions from database (#15688)
that's a way to save database storage space.

Signed-off-by: a1012112796 <1012112796@qq.com>
2021-05-01 14:17:02 +02:00
KN4CK3R 67e10397af
Add Location, Website and Biography to API (#15675) 2021-05-01 10:05:55 +01:00
KN4CK3R ee3fb92419
Delete references if repository gets deleted (#15681)
* Remove DeletedBranch and LFSLocks.

* Sort beans.

Co-authored-by: zeripath <art27@cantab.net>
2021-04-30 21:10:39 +02:00
KN4CK3R c80d7f33b6
Unified link creation. (#15619) 2021-04-30 19:25:13 +02:00
KN4CK3R 2e85165349
Delete protected branch if repository gets removed (#15658)
* Added missing error parameters.

* Delete protected branch if repository gets removed.

* Added doctor fix.
2021-04-30 00:48:52 +02:00
a1012112796 1e877613bf
add /assets as root dir of public files (#15219)
* add `/assets` as root dir of public files

Signed-off-by: a1012112796 <1012112796@qq.com>

* move serviceworker.js

* make fmt

* fix some link

* fix test

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-04-28 12:35:06 +00:00
Lunny Xiao 3d5bb3e6a3
fix webhook timeout bug (#15613)
* Also fix the potential problem in httplib
2021-04-25 21:48:12 +03:00
zeripath d6a33cef23
If the default branch is not present do not report error on stats indexing (follow-up of #15546) (#15583)
#15546 doesn't completely fix this problem because the error returned is an ObjectNotExist
error not a BranchNotExist error.

Add test for ErrObjectNotExist too

Fix #15257

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-22 17:35:29 +02:00
zeripath f719ffc783
If the default branch is not present do not report error on stats indexing (#15546)
* If the default branch is not present do not report error on stats indexing

Fix #15257

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per lunny

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-22 09:19:21 +08:00
zeripath 1cd8d0ca0e
Fix NPE on view commit with notes (#15561)
Fix #15558

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-21 19:00:27 +01:00
Lunny Xiao 2255afffad
Fix lfs management find (#15537)
Fix #15236

* Do not do 40byte conversion within ParseTreeLine
* Missed a to40ByteSHA

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-04-21 18:31:59 +02:00
Lunny Xiao 9d99f6ab19
Refactor renders (#15175)
* Refactor renders

* Some performance optimization

* Fix comment

* Transform reader

* Fix csv test

* Fix test

* Fix tests

* Improve optimaziation

* Fix test

* Fix test

* Detect file encoding with reader

* Improve optimaziation

* reduce memory usage

* improve code

* fix build

* Fix test

* Fix for go1.15

* Fix render

* Fix comment

* Fix lint

* Fix test

* Don't use NormalEOF when unnecessary

* revert change on util.go

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* rename function

* Take NormalEOF back

Co-authored-by: zeripath <art27@cantab.net>
2021-04-19 18:25:08 -04:00
Lunny Xiao a3c4c57b7c
Fix bug on commit graph (#15517) 2021-04-17 10:27:25 +01:00
Kyle D f44543a1bb
Disable Stars config option (#14653)
* Add config option to disable stars

* Replace "stars" with watched in user profile

* Add documentation
2021-04-15 18:53:57 +02:00
6543 2a42d80d14
migration: github: if rate limit is not enabled, ignore it (#15490) 2021-04-15 15:34:22 +02:00
Lunny Xiao dc5a1d617d
Fix potential copy lfs records failure when fork a repository (#15441) 2021-04-14 22:15:28 +01:00
Naohisa Murakami 1426601cf7
Use index of the supported tags to choose user lang (#15452)
Fix #14793.

The previous implementation used the first return value of matcher.Match, which is the chosen language tag but may contain extensions such as de-DE-u-rg-chzzzz.

As mentioned in the documentation of language package, matcher.Match also returns the index of the supported tags, so I think it is better to use it rather than manipulate the returned language tag.
2021-04-14 19:52:01 +01:00
zeripath 1ee776970a
Fix ambiguous argument error on tags (#15432)
There is a weird gotcha with GetTagCommitID that because it uses git rev-list
can cause an ambiguous argument error.

This PR simply makes tags use the same code as branches.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-14 16:22:37 +02:00
zeripath 8e2a8efd84
Prevent superfluous response.WriteHeader (#15456)
This PR simply checks the status before writing the header.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-14 14:57:18 +02:00
Martin Michaelis 55eb1745bd
OAuth2 auto-register (#5123)
* Refactored handleOAuth2SignIn in routers/user/auth.go

The function handleOAuth2SignIn was called twice but some code path could only
be reached by one of the invocations. Moved the unnecessary code path out of
handleOAuth2SignIn.


* Refactored user creation

There was common code to create a user and display the correct error message.
And after the creation the only user should be an admin and if enabled a
confirmation email should be sent. This common code is now abstracted into
two functions and a helper function to call both.

* Added auto-register for OAuth2 users

If enabled new OAuth2 users will be registered with their OAuth2 details.
The UserID, Name and Email fields from the gothUser are used.
Therefore the OpenID Connect provider needs additional scopes to return
the coresponding claims.

* Added error for missing fields in OAuth2 response

* Linking and auto linking on oauth2 registration

* Set default username source to nickname

* Add automatic oauth2 scopes for github and google

* Add hint to change the openid connect scopes if fields are missing

* Extend info about auto linking security risk

Co-authored-by: Viktor Kuzmin <kvaster@gmail.com>
Signed-off-by: Martin Michaelis <code@mgjm.de>
2021-04-14 14:02:12 +02:00
Lunny Xiao 66f0fd0959
Add NeedPostProcess for Parser interface to improve performance of csv render (#15153) 2021-04-13 15:06:31 +08:00
KN4CK3R a35a5b225c
Add ETag header (#15370)
* Add ETag header.

* Comply with RFC 7232.

* Moved logic into httpcache.go

* Changed name.

* Lint

* Implemented If-None-Match list.

* Fixed missing header on *

* Removed weak etag support.

* Removed * support.

* Added unit test.

* Lint

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-12 10:49:26 -04:00
6543 83d771c1c7
use repo1_bare to test against (#15402) 2021-04-11 11:45:29 -04:00
zeripath c680eb2cc7
Links in markdown should be absolute to the repository not the server (#15088)
* Links in markdown should be absolute to the repository not the server

Fix #15075

Signed-off-by: Andrew Thornton <art27@cantab.net>

* match github

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-04-11 00:26:28 +08:00
zeripath 84f5a0bc62
Always set the merge base used to merge the commit (#15352)
The issue is that the TestPatch will reset the PR MergeBase - and it is possible for TestPatch to update the MergeBase whilst a merge is ongoing. The ensuing merge will then complete but it doesn't re-set the MergeBase it used to merge the PR.

Fixes the intermittent error in git test.

Signed-off-by: Andrew Thornton art27@cantab.net
2021-04-10 09:27:29 +01:00
zeripath 136a20926c
Turn RepoRef and RepoAssignment back into func(*Context) (#15372)
Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-09 20:26:08 -04:00
zeripath b9ed3cbc26
Upgrade to bluemonday 1.0.7 (#15379)
* Upgrade to bluemonday 1.0.7

Fix #15349

Signed-off-by: Andrew Thornton <art27@cantab.net>

* resolve unit test

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-09 19:13:06 -04:00
6543 9c4601bdf8
Code Formats, Nits & Unused Func/Var deletions (#15286)
* _ to unused func options

* rm useless brakets

* rm trifial non used models functions

* rm dead code

* rm dead global vars

* fix routers/api/v1/repo/issue.go

* dont overload import module
2021-04-09 09:40:34 +02:00
6543 147826a577
[API] pull notification subject status: add "merged" (#15344)
Current subject status can be "", "open" and "closed". This add "merged" to it.
2021-04-09 03:36:23 +02:00
KN4CK3R c03e488e14
Add LFS Migration and Mirror (#14726)
* Implemented LFS client.

* Implemented scanning for pointer files.

* Implemented downloading of lfs files.

* Moved model-dependent code into services.

* Removed models dependency. Added TryReadPointerFromBuffer.

* Migrated code from service to module.

* Centralised storage creation.

* Removed dependency from models.

* Moved ContentStore into modules.

* Share structs between server and client.

* Moved method to services.

* Implemented lfs download on clone.

* Implemented LFS sync on clone and mirror update.

* Added form fields.

* Updated templates.

* Fixed condition.

* Use alternate endpoint.

* Added missing methods.

* Fixed typo and make linter happy.

* Detached pointer parser from gogit dependency.

* Fixed TestGetLFSRange test.

* Added context to support cancellation.

* Use ReadFull to probably read more data.

* Removed duplicated code from models.

* Moved scan implementation into pointer_scanner_nogogit.

* Changed method name.

* Added comments.

* Added more/specific log/error messages.

* Embedded lfs.Pointer into models.LFSMetaObject.

* Moved code from models to module.

* Moved code from models to module.

* Moved code from models to module.

* Reduced pointer usage.

* Embedded type.

* Use promoted fields.

* Fixed unexpected eof.

* Added unit tests.

* Implemented migration of local file paths.

* Show an error on invalid LFS endpoints.

* Hide settings if not used.

* Added LFS info to mirror struct.

* Fixed comment.

* Check LFS endpoint.

* Manage LFS settings from mirror page.

* Fixed selector.

* Adjusted selector.

* Added more tests.

* Added local filesystem migration test.

* Fixed typo.

* Reset settings.

* Added special windows path handling.

* Added unit test for HTTPClient.

* Added unit test for BasicTransferAdapter.

* Moved into util package.

* Test if LFS endpoint is allowed.

* Added support for git://

* Just use a static placeholder as the displayed url may be invalid.

* Reverted to original code.

* Added "Advanced Settings".

* Updated wording.

* Added discovery info link.

* Implemented suggestion.

* Fixed missing format parameter.

* Added Pointer.IsValid().

* Always remove model on error.

* Added suggestions.

* Use channel instead of array.

* Update routers/repo/migrate.go

* fmt

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-04-08 18:25:57 -04:00
zeripath fa3895ce81
Move modules/forms to services/forms (#15305)
Forms are dependent on models and therefore should be in services.

This PR also removes the old auth. aliasing

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-06 20:44:05 +01:00
zeripath b101fa83a6
Fix bug in Wrap (#15302)
Whilst doing other work I have noticed that there is an issue with Wrap when passing an
http.Handler - the next should be the next handler in line not empty.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-06 17:40:56 +02:00
KN4CK3R 5f18404045
Close file on invalid range (Addition to #15166) (#15268)
* Close file on invalid range.

* Close on seek error

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Moved 'Seek' into server.

* io.ReadSeekCloser is only available in Go 1.16

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-04-06 21:22:34 +08:00
zeripath 04196b7658
Update to bluemonday-1.0.6 (#15294)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-05 17:38:31 -04:00
6543 16dea6cebd
[refactor] replace int with httpStatusCodes (#15282)
* replace "200" (int) with "http.StatusOK" (const)

* ctx.Error & ctx.HTML

* ctx.JSON Part1

* ctx.JSON Part2

* ctx.JSON Part3
2021-04-05 11:30:52 -04:00
zeripath 0bb8bd8190
Add size to Save function (#15264)
Fix #15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-03 18:19:59 +02:00
KN4CK3R 3cc7d27b6f
Close file on invalid range (#15166)
* Close file on invalid range.

* Close on seek error

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-04-03 14:44:12 +02:00
6543 80d6c6d7de
[refactor] mailer service (#15072)
* Unexport SendUserMail

* Instead of "[]*models.User" or "[]string" lists infent "[]*MailRecipient" for mailer

* adopt

* code format

* TODOs for "i18n"

* clean

* no fallback for lang -> just use english

* lint

* exec testComposeIssueCommentMessage per lang and use only emails

* rm MailRecipient

* Dont reload from users from db if you alredy have in ram

* nits

* minimize diff

Signed-off-by: 6543 <6543@obermui.de>

* localize subjects

* linter ...

* Tr extend

* start tmpl edit ...

* Apply suggestions from code review

* use translation.Locale

* improve mailIssueCommentBatch

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add i18n to datas

Signed-off-by: Andrew Thornton <art27@cantab.net>

* a comment

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-04-02 11:25:13 +01:00
zeripath ff460ca74d
Speed up enry.IsVendor (#15213)
`enry.IsVendor` is kinda slow as it simply iterates across all regexps.
This PR ajdusts the regexps to combine them to make this process a
little quicker.

Related #15143

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-01 19:41:09 +02:00
a1012112796 43fb4921e3
response simple text message for not html request when 404 (#15229)
* response simple text message for not html request when response 404

Signed-off-by: a1012112796 <1012112796@qq.com>
2021-04-01 17:11:42 +02:00
a1012112796 007fb00c0e
response 404 for diff/patch of a commit that not exist (#15221)
* response 404 for diff/patch of a commit that not exist

fix #15217

Signed-off-by: a1012112796 <1012112796@qq.com>

* Update routers/repo/commit.go

Co-authored-by: silverwind <me@silverwind.io>

* use ctx.NotFound()

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
2021-04-01 08:17:14 +03:00
zeripath 113c1557ff
Fix regression from #14623 - use debug SVC handler only on interactive sessions (#15210)
Unfortunately #14623 changed from the deprecated IsInteractiveSession to
IsWindowsService without recognising that they are the complement of
each other.

This means that Windows SVC control is not working correctly. This PR
adds some Tracing statements but also fixes the bug.

Fix #15159

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-31 22:48:48 +03:00
a1012112796 f5b6dc9759
add 'fonts' into 'KnownPublicEntries' (#15188)
fix #15184

Signed-off-by: a1012112796 <1012112796@qq.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-03-30 15:47:34 +03:00
KN4CK3R 0c6137617f
Add Tabular Diff for CSV files (#14661)
Implements request #14320 The rendering of CSV files does match the diff style.

* Moved CSV logic into base package.

* Added method to create a tabular diff.

* Added CSV compare context.

* Added CSV diff template.

* Use new table style in CSV markup.

* Added file size limit for CSV rendering.

* Display CSV parser errors in diff.

* Lazy read single file.

* Lazy read rows for full diff.

* Added unit tests for various CSV changes.
2021-03-29 22:44:28 +02:00
a1012112796 3273fb9af1
use level config in main section when subsection not set level (#15176)
in previouse if a log subsetcion not set level
it will use ``info`` as default value.

this pr will make default value (``[log] -> LEVEL``) useable.

example config:
```INI
[log]
MODE = console
LEVEL = Trace

[log.console]
LEVEL =
STDERR = false
```

previous result:
```JSON
// console:
{
  "level": "info",
  ...................
}
```

after change:

```JSON
// console:
{
  "level": "track",
  ...................
}
```

Signed-off-by: a1012112796 <1012112796@qq.com>

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-28 20:08:19 +01:00
zeripath 82d1a7fb17
Update repository size on cron gc task (#15177)
git gc cron could change the size of the repository therefore we should update the
size of the repo stored in our database.

Also significantly improve the efficiency of counting lfs associated with the
repository
2021-03-28 05:56:28 +02:00
sotho e7609929c1
Expose resolver via API (#15167)
* Expose resolver via API
2021-03-28 00:37:51 +01:00
6543 290cf75f93
[refactor] Unify the export of user data via API (#15144)
* [refactor] unify how user data is exported via API

* test time via unix timestamp
2021-03-27 17:45:26 +01:00
parnic f4d27498bd
Add DefaultMergeStyle option to repository (#14789)
Fixes #12293
2021-03-27 15:55:40 +01:00
6543 dc56fb7c84
fix regression of 15139 (#15164) 2021-03-26 03:46:41 +01:00
zeripath b68eb54f95
Clusterfuzz found another way (#15160)
Clusterfuzz found another way so I found another way to stop it

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-26 01:21:00 +02:00
sotho f2844b7583
Fix wrong user returned in API (#15139)
The API call: GET /repos/{owner}/{repo}/pulls/{index}/reviews/{id}/comments
returns always the reviewer, but should return the poster.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-03-24 21:26:15 +01:00
6543 5f038cd7fe
[Vendor] update gitea-sdk v0.14.0 (#15103)
* upgraded code.gitea.io/sdk/gitea v0.13.2 => v0.14.0

* rm workaround
2021-03-22 18:03:18 +01:00
zeripath a587a28434
Fix another clusterfuzz identified issue (#15096)
* Fix another clusterfuzz identified issue

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-22 02:45:01 +01:00
6543 24f7bd5899
Use i18n.Reset to reload locales (#15073) 2021-03-21 16:11:36 +01:00
a1012112796 17731e05ff
fix double 'push tag' action feed (#15078)
Signed-off-by: a1012112796 <1012112796@qq.com>
2021-03-21 11:11:22 +01:00
6543 78e8f62706
[Refactor] remove possible resource leak (#15067)
* move "copy uploaded lfs files 2 repo" to own function for "defer file.Close()"

* rm type overload

* Update modules/repofiles/upload.go

Co-authored-by: zeripath <art27@cantab.net>
2021-03-21 00:30:29 +08:00
Lunny Xiao cf549500e0
Fix bug when upload on web (#15042)
* Fix bug when upload on web

* move into own function

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-03-19 23:49:29 +01:00
6543 a3a65137ba
Delete Labels & IssueLabels on Repo Delete too (#15039)
* Doctor: find IssueLabels without existing label

* on Repo Delete: delete labels & issue_labels too

* performance nits

* Add Migration: Delete orphaned IssueLabels

* Migration v174: use Sync2

* USE sess !!!

* better func name

* code format & comment

* RAW SQL

* Update models/migrations/v176.go

* next try?
2021-03-19 21:01:24 +02:00
zeripath dace0ce1b1
Prevent addition of labels from outside the repository or organisation in issues (#14912)
* Never add labels not from this repository or organisation and remove org labels on transfer

Prevent the addition of labels from outside of the repository or
organisation and remove organisation labels on transfer.

Related #14908

Signed-off-by: Andrew Thornton <art27@cantab.net>

* switch to use sql

Signed-off-by: Andrew Thornton <art27@cantab.net>

* remove AS

Signed-off-by: Andrew Thornton <art27@cantab.net>

* subquery alias

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Give me some AS?

Signed-off-by: Andrew Thornton <art27@cantab.net>

* double AS

Signed-off-by: Andrew Thornton <art27@cantab.net>

* try try again

Signed-off-by: Andrew Thornton <art27@cantab.net>

* once more around the merry go round

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix api problem

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add outside label consistency check into doctor

This PR adds another consistency check into doctor in order to detect
labels that have been added from outside of repositories and organisations

Fix #14908

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix migration

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prep for merge

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-19 21:25:14 +08:00
zeripath c0c052bdbb
another clusterfuzz spotted issue (#15032)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 22:37:11 +01:00
KN4CK3R e8ad6c1ff3
Do not convert file path to lowercase (#15023)
* Do not convert file path to lowercase.

* lint

* Check against lowercase hostname.
2021-03-18 13:58:47 +00:00
zeripath 070c57867d
Fix postgres ID sequences broken by recreate-table (#15015)
* Fix postgres ID sequences broken by recreate-table

Unfortunately there is a subtle problem with recreatetable on postgres which
leads to the sequences not being renamed and not being left at 0.

Fix #14725

Signed-off-by: Andrew Thornton <art27@cantab.net>

* let us try information_schema instead

Signed-off-by: Andrew Thornton <art27@cantab.net>

* try again

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-18 02:06:40 -04:00
zeripath 71aca93dec
Remove extraneous logging (#15020)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 04:25:52 +01:00
a1012112796 27b8eff96b
support gitmailmap in GetCodeActivityStats() (#15009)
ref:
- https://git-scm.com/docs/gitmailmap
- https://git-scm.com/docs/git-log#Documentation/git-log.txt-emaNem

Signed-off-by: a1012112796 <1012112796@qq.com>

Co-authored-by: zeripath <art27@cantab.net>
2021-03-17 17:00:26 -04:00
Norwin f4629fc472
Move repo.CloseIssuesViaCommitInAnyBranch to issue settings (#14965) 2021-03-16 02:00:52 +01:00
zeripath ed31ddc29a
Fix several render issues (#14986)
* Fix an issue with panics related to attributes
* Wrap goldmark render in a recovery function
* Reduce memory use in render emoji
* Use a pipe for rendering goldmark - still needs more work and a limiter

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-16 00:20:05 +01:00
Lauris BH 044cd4d016
Add reverse proxy configuration support for remote IP address (#14959)
* Add reverse proxy configuration support for remote IP address validation

* Trust all IP addresses in containerized environments by default

* Use single option to specify networks and proxy IP addresses. By default trust all loopback IPs

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-16 00:27:28 +02:00
zeripath 6e423d5573
Ensure validation occurs on clone addresses too (#14994)
* Ensure validation occurs on clone addresses too

Fix #14984

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Fix api tests

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-15 17:52:11 -04:00
zeripath 164e35ead3
Make sure sibling images get a link too (#14979)
* Make sure sibling images get a link too

Due a problem with the ast.Walker in the our transformer in goldmark
an image with a sibling image will not be transformed to gain a parent
link. This PR fixes this.

Fix #12925

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-14 16:36:51 +00:00
zeripath 91ee3be588
Prevent incorrect HTML escaping in swagger.json (#14957)
* Prevent incorrect HTML escaping in swagger.json

Fix #14706

Signed-off-by: Andrew Thornton <art27@cantab.net>

* oops add it to the helper

Signed-off-by: Andrew Thornton <art27@cantab.net>

* try again

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-11 23:43:04 -05:00
zeripath c8e5c79cfd
Add ui.explore settings to control view of explore pages (2) (#14094)
This is an alternative PR to #13687.

Add `[ui.explore]` settings to allow restricting the
explore pages to logged in users only and to disable the users explore page.

The two proposed settings are:

- `REQUIRE_SIGNIN_VIEW`: Only allows access to the explore pages if the
user is signed in. Also restricts
  - `/api/v1/user/search`
  - `/api/v1/users/{username}`
  - `/api/v1/users/{username}/repos`
  - but does not restrict `/api/v1/users/{username}/heatmap`
- `DISABLE_USERS_PAGE`: Disables the /explore/users page

Fix #2908

Close #13687 

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-11 13:40:54 +00:00
6543 177da717a7
[API] get pull, return head branch sha, even if deleted (#14931)
* API: return head branch sha, even if deleted

* relax if ref not resolvable
2021-03-08 21:48:31 +01:00
6543 14d8cb7819
Move Workaround for #12675 into it's own function (#14922)
* Move Workatround for #12675 into it's own function

* use more reliable solution (as tea do)
2021-03-08 20:00:19 +01:00
zeripath eb576269d4
Re-enable import local paths after reversion from #13610 (#14925)
PR #13610 unfortunately disabled importing repositories from local paths.
This PR restores this functionality.

Fix #14700

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-08 13:10:17 +00:00
zeripath c03f530212
Make internal SSH server host key path configurable (#14918)
* Make SSH server host key path configurable

* make it possible to have multiple keys

* Make gitea.rsa the default key

* Add some more logging

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-08 03:43:59 +01:00
zeripath 9b261f52f0
Add SameSite setting for cookies (#14900)
Add SameSite setting for cookies and rationalise the cookie setting code. Switches SameSite to Lax by default. 

There is a possible future extension of differentiating which cookies could be set at Strict by default but that is for a future PR.

Fix #5583

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-07 08:12:43 +00:00
zeripath a68db9076a
Fix race in LFS ContentStore.Put(...) (#14895)
Continuing on from #14888

The previous implementation has race whereby an incomplete upload or
hash mismatch upload can end up in the ContentStore. This PR moves the
validation into the reader so that if there is a hash error or size
mismatch the reader will return with an error instead of an io.EOF
causing the storage to abort the storage.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-06 20:21:56 +00:00
zeripath 20f13bfdfc
Fix a couple of issues with a feeds (#14897)
@CirnoT spotted a couple of issues with feeds on discord.

This PR fixes both of these.
2021-03-06 05:09:49 +01:00
zeripath 144cfe5720
Fix race in local storage (#14888)
LocalStorage should only put completed files in position

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-05 14:19:17 +01:00
a1012112796 a5279b74b6
Make manual merge autodetection optional and add manual merge as merge method (#12543)
* Make auto check manual merge as a chooseable mod and add manual merge way on ui

as title, Before this pr, we use same way with GH to check manually merge.
It good, but in some special cases, misjudgments can occur. and it's hard
to fix this bug. So I add option to allow repo manager block "auto check manual merge"
function, Then it will have same style like gitlab(allow empty pr). and to compensate for
not being able to detect THE PR merge automatically, I added a manual approach.

Signed-off-by: a1012112796 <1012112796@qq.com>

* make swager

* api support

* ping ci

* fix TestPullCreate_EmptyChangesWithCommits

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* Apply review suggestions and add test

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* fix build

* test error message

* make fmt

* Fix indentation issues identified by @silverwind

Co-authored-by: silverwind <me@silverwind.io>

* Fix tests and make manually merged disabled error on API the same

Signed-off-by: Andrew Thornton <art27@cantab.net>

* a small nit

* fix wrong commit id error

* fix bug

* simple test

* fix test

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-03 22:41:23 -05:00
zeripath 523efa433b
Move Bleve and Elastic code indexers to use a common cat-file --batch (#14781)
* Extract out the common cat-file batch calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Move bleve and elastic indexers to use a common cat-file --batch when indexing

Signed-off-by: Andrew Thornton <art27@cantab.net>

* move catfilebatch to batch_reader and rename to batch_reader.go

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-03 21:57:01 -05:00
zeripath 59d1cc49f1
Fix paging of file commit logs (#14831)
Unfortunately `git log revision ... --skip=x -- path` skips the number of commits
not the number of commits relating to the path.

This PR changes the function to have a reader that reads and skips the
necessary number of commits by hand instead.

Fix #8716

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-04 02:48:19 +02:00
zeripath 0bdeb2b302
Add missing repo.projects unit into swagger (#14876)
Fix #14875

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-03 23:44:30 +01:00
zeripath f0e15250b9
Migrate to use jsoniter instead of encoding/json (#14841)
* Migrate to use jsoniter

* fix tests

* update gitea.com/go-chi/binding

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-01 22:08:10 +01:00
a1012112796 6cf97df9fd
remove duplicate define of CheckAttribute() (#14837)
Signed-off-by: a1012112796 <1012112796@qq.com>
2021-03-01 14:14:17 +02:00
6543 a4148c0f12
Repository transfer has to be confirmed, if user can not create repo for new owner (#14792)
* make repo as "pending transfer" if on transfer start doer has no right to create repo in new destination

* if new pending transfer ocured, create UI & Mail notifications
2021-03-01 01:47:30 +01:00
a1012112796 83cf1a894e
Create tag on ui (#13467)
Support create single tag directly

support create tag with message from create release ui

Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-02-28 19:57:45 +00:00
zeripath cf29cb30d3
Prevent use of double sub-path and incorrect asset path in manifest (#14827)
MakeAbsoluteAssetURL should just url join the static url prefix on to appurl
if it is not an absolute path - this is because StaticURLPrefix is an absolute
prefix not a relative prefix to the app sub url.

Fix #14422

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-28 12:29:22 +00:00
zeripath 3d8b5ad5f3
Fix a couple of CommentAsPatch issues. (#14804)
* CutDiffAroundLine makes the incorrect assumption that `---` and `+++` always represent part of the header of a diff.

This PR adds a flag to its parsing to prevent this problem and adds a streaming parsing technique to CutDiffAroundLine using an io.pipe instead of just sending data to an unbounded buffer.

Fix #14711

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Handle unquoted comment patch files

When making comment patches unfortunately the patch does not always quote the filename
This makes the diff --git header ambiguous again.

This PR finally adds handling for ambiguity in to parse patch

Fix #14812

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add in testing for no error

There is no way currently for CutDiffAroundLine in this test to cause an
error however, it should still be tested.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-27 19:46:14 +01:00
Lunny Xiao e79dae29cc
Fix repo-restore bug with poster not replaced (#14759)
* Fix restore bug

* Fix restore bug

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
2021-02-22 11:57:41 +08:00
Lunny Xiao bd9361384a
Fix go get (#14758)
* Fix go get

* Fix default branch

Co-authored-by: 6543 <6543@obermui.de>
2021-02-22 00:51:15 +08:00
6543 190bd0accd
Export LFS & TimeTracking function status (#14753) 2021-02-20 11:17:51 +01:00
zeripath 65c940f66c
Prevent endless loop if templates missing (#14752)
Since the chi upgrade if the templates are missing an endless loop will occur if
status/500.tmpl is missing.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-02-20 12:26:57 +08:00
Norwin 91424fff3d
stopwatch notif: check if timetracking is enabled (#14750)
fixes #14435
2021-02-20 00:06:56 +01:00
zeripath aa4f9180e4
Clarify the suffices and prefixes of setting.AppSubURL and setting.AppURL (#12999)
Also removes some unnecessary uses of fmt.Sprintf and adds documentation
strings

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-19 22:36:43 +01:00
Lunny Xiao 6362b24a59
Fix when a commit not found returned 500 (#14732)
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-02-18 19:40:17 -05:00
6543 8d5c795cc4
[API] Add Restricted Field to User (#14630)
* Expose Restricted field for User

* Add Option to Change Restricted on User via adminEditUser API

* Add test who change restricted & test if it changed it ...

* make generate-swagger

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-02-18 16:25:35 +08:00
zeripath 7ab6c77b41
Remove NULs byte arrays passed to PostProcess (#14587)
PostProcess is supposed to be parsing and handling HTML
fragments, but on fuzzing it appears that there is a weird
issue with NUL elements that could cause a memory address
error in downstream libraries.

The simplest solution is to strip out the weird NULs - they
should not be there in any case and would be stripped out
anyway.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-18 02:32:14 +01:00
Lunny Xiao ec06eb112c
Fix github download on migration (#14703)
* Fix github download on migration

* Use Context for Client

Co-authored-by: zeripath <art27@cantab.net>
2021-02-18 01:36:49 +01:00
zeripath ae7e6cd474
Reduce calls to git cat-file -s (#14682)
* Reduce calls to git cat-file -s

There are multiple places where there are repeated calls to git cat-file
-s due to the blobs not being created with their size.

Through judicious use of git ls-tree -l and slight adjustments to the
indexer code we can avoid a lot of these calls.

* simplify by always expecting the long format

* Also always set the sized field and tell the indexer the update is sized
2021-02-17 22:32:25 +01:00
zeripath 7ba158183a
Use cat-file --batch in GetLanguageStats (#14685)
* Use cat-file --batch in GetLanguageStats

This PR moves to using a single cat-file --batch in GetLanguageStats
significantly reducing the number of processes spawned during language stat
processing.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/git/repo_language_stats_nogogit.go

Co-authored-by: a1012112796 <1012112796@qq.com>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-02-17 14:32:47 -05:00
zeripath ad43b119a8
Add Password Algorithm option to install page (#14701)
Add Password Algorithm option to install page

Fix #14674 

Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2021-02-16 23:37:20 +01:00
zeripath 66a148e398
Restore detection of branches are equal on compare page (#14586)
Somehow the test for detecting if branches are equal broke
this PR restores this functionality.

Fix #14502

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-16 16:39:45 +01:00
zeripath 0a9a484e1e
Create DB session provider(based on xorm) (#13031)
* Create Xorm session provider

This PR creates a Xorm session provider which creates
the appropriate Session table for macaron/session.

Fix #7137

Signed-off-by: Andrew Thornton <art27@cantab.net>

* extraneous l

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* use key instead of ID to be compatible with go-macaron/session

Signed-off-by: Andrew Thornton <art27@cantab.net>

* And change the migration too.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update spacing of imports

Co-authored-by: 6543 <6543@obermui.de>

* Update modules/session/xorm.go

Co-authored-by: techknowlogick <matti@mdranta.net>

* add xorm provider to the virtual provider

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prep for master merge

* prep for merge master

* As per @lunny

* move migration out of the way

* Move to call this db session as per @lunny

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-02-15 00:33:31 -05:00
Damien Goutte-Gattat fc4a8c2980
Allow blocking some email domains from registering an account (#14667)
Gitea allows to whitelist email domains so that only email addresses from certain domains are allowed to register an account, but does not currently allows to do the opposite: blacklisting email domains so that addresses from certain domains are *forbidden* to register an account.

The idea has been briefly mentioned in the discussion about issue #6350, but never implemented. This PR does that.

The rationale is that, in my experience of running a Gitea instance, *a single email domain* is responsible for *most* of the spam accounts, and for *all* of the spam accounts that manage to get past the email confirmation step. So on top of the other spam mitigation measures already available (email confirmation, CAPTCHA, etc.), having the option to block a particularly annoying domain would be helpful.

close #13628
2021-02-15 00:31:29 +01:00
zeripath c9f1baf620
Prevent template renderer from rendering error (#14646)
When there is a panic during template rendering unrolled/render
will automatically render the error. This leads to the
panic being displayed in the page and not a 500 page

Fix #14467
Fix #14525

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-13 19:41:53 +00:00
zeripath b3c2e23cbb
Prevent race in PersistableChannelUniqueQueue.Has (#14651)
There is potentially a race with a slow starting internal
queue causing a NPE if Has is checked before the internal
queue has been setup.

This PR adds a lock on the Has() fn.

Fix #14311

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-13 20:02:09 +01:00
Lunny Xiao 5e4fa7c703
Turn default hash password algorightm back to pbkdf2 from argon2 until we found a better one (#14673)
* Turn default hash password algorightm back to pbkdf2 from argon2 until we found a better one

* Add a warning on document
2021-02-13 18:03:29 +01:00
6543 0a4c2d75cd
rm gogs relict (#14665) 2021-02-14 00:07:17 +08:00
Paweł Bogusławski 7d7007dca7
Added option to disable webhooks (#13176)
* Added option to disable web hooks

This mod introduces DISABLE_WEB_HOOKS parameter in [security] section
of app.ini (by default set to false). If set to true it disables web
hooks feature. Any existing undelivered web hook tasks will be cancelled.
Any existing web hook definitions will be left untouched in db but
its delivery tasks will be ignored.

Author-Change-Id: IB#1105130

* Webhook spelling fixed

Webhook spelling fixed.

Fixes: 07df6614dc84cdd2e9f39c57577fa1062bd70012
Related: https://github.com/go-gitea/gitea/pull/13176#pullrequestreview-510868421
Author-Change-Id: IB#1105174

* Parameter description fixed

Parameter description fixed.

Fixes: 07df6614dc84cdd2e9f39c57577fa1062bd70012
Related: https://github.com/go-gitea/gitea/pull/13176#pullrequestreview-514086107
Author-Change-Id: IB#1105174
2021-02-11 12:34:34 -05:00
a1012112796 ac701637b4
Add dismiss review feature (#12674)
* Add dismiss review feature

refs:
    https://github.blog/2016-10-12-dismissing-reviews-on-pull-requests/
    https://developer.github.com/v3/pulls/reviews/#dismiss-a-review-for-a-pull-request

* change modal ui and error message

* Add unDismissReview api

Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-02-11 18:32:25 +01:00
Romain c69c01d2b6
Sort / Move project boards (#14634)
Sort Project board (#14533)
2021-02-11 17:32:27 +01:00
6543 ac97ea573c
[Vendor] Update go-redis to v8.5.0 (#13749)
* Update go-redis to v8.4.0

* github.com/go-redis/redis/v8  v8.4.0 -> v8.5.0

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* TODO

* Use the Queue termination channel as the default context for pushes

Signed-off-by: Andrew Thornton <art27@cantab.net>

* missed one

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-02-10 21:28:32 +00:00
zeripath 0a23079485
Do not assume all 40 char strings are SHA1s (#14624)
GetCommit() assumes that all 40 char strings are SHA1s. This leads to an
error if you try to do a PR on a branch which is 40 characters long.

This PR attempts the SHA first - and if it fails will switch to using rev-parse.

Fix #14470

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-10 07:37:25 +00:00
zeripath f9abf94bd9
HasPreviousCommit causes recursive load of commits unnecessarily (#14598)
This PR improves HasPreviousCommit to prevent the automatic and recursive loading
of previous commits using git merge-base --is-ancestor and git rev-list

Fix #13684

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-10 07:00:57 +00:00
zeripath f82b1dd7c3
Prevent adding nil label to .AddedLabels or .RemovedLabels (#14623)
* Prevent adding nil label to .AddedLabels or .RemovedLabels

There are possibly a few old databases out there with malmigrated data that can
cause panics with empty labels being migrated.

This PR adds a few tests to prevent nil labels being added.

Fix #14466

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add doctor command to remove the broken label comments

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-02-10 10:50:44 +08:00
zeripath 30f7ddb833
Ensure memcache TTL cannot be over 30 days (#14592)
Memcached TTL cannot be > 30 days and if it is attempted the TTL is interpreted as
a unix timestamp.

This PR ensures that the TTL is switched to a unix timestamp in those cases.

Fix #14571

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-09 22:29:03 +00:00
zeripath e65cfabda7
Remove spurious DataAsync Error logging (#14599)
Breaking the pipe is a valid way of killing a piped command and any error from
a broken cat-file batch command should be passed back up to the writer any way
therefore specifically logging it is unnecessary.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-07 21:04:58 +00:00
Lunny Xiao 240fea8c14
Fix rate limit bug when downloading assets on migrating from github (#14564) 2021-02-07 15:56:11 +00:00
6543 cbe7f5296e
[API] Add affected files of commits to commit struct (#14579)
* Add files affected by a commit to gitea API -- similar to github

* Add files affected by a commit to gitea API

* Fix stupid error

* Fix other stupid typo

* Generate swagger tmpl

* Comply with convert to git commit refacto

* update swagger docs

* extend test

* format code

* Update integrations/api_repo_git_commits_test.go

* Update modules/convert/git_commit.go

Co-authored-by: Laurent Cahour <laurent.cahour@dont-nod.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-02-07 15:43:40 +01:00
John Olheiser 19fccdc45d
Fix locale init (#14582)
just log if lang is already loaded since we can not reload it

Co-authored-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-02-05 22:41:45 +01:00
6543 87009ab40a
Reduce data races (#14549)
* Add race conditions into test

* Fix Race in GetManager()

* DataAsync() use error chan

* just log no chan

* finish
2021-02-03 22:36:38 +01:00
6543 0d1444751f
[API] Add pagination to ListBranches (#14524)
* make PaginateUserSlice generic -> PaginateSlice

* Add pagination to ListBranches

* add skip, limit to Repository.GetBranches()

* Move routers/api/v1/utils/utils PaginateSlice -> modules/util/paginate.go

* repo_module.GetBranches paginate

* fix & rename & more logging

* better description

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: a1012112796 <1012112796@qq.com>
2021-02-03 20:06:13 +01:00
Stefan eea4197cd9
Internal ssh server respect Ciphers, MACs and KeyExchanges settings (#14523) 2021-01-30 14:20:32 +01:00
Lunny Xiao 5e20fd6dbf
Move middlewares to web/middleware (#14480)
Co-authored-by: 6543 <6543@obermui.de>
2021-01-30 10:55:53 +02:00
6543 0e0424c8ec
Add Doctor FixWrongUserType (#14522)
* Add Doctor FixWrongUserType

* use NoAutoTime
2021-01-29 19:30:43 +01:00
Lunny Xiao 0cd87d64ff
Update docs and comments to remove macaron (#14491) 2021-01-29 16:35:30 +01:00
Lunny Xiao 87b098f3b6
Fix json charset bug (#14514) 2021-01-29 08:42:47 -05:00
a1012112796 f19da14c34
enhancement: add signoff option in commit form (#14516)
Signed-off-by: a1012112796 <1012112796@qq.com>
2021-01-29 16:57:45 +08:00
Lunny Xiao f761c82c94
Fix load time bug (#14508) 2021-01-29 05:33:47 +01:00
6543 99b7af6fc8
Add some Unit-Tests (#14500)
* fix url

* modules/auth/pa: coverage: 40#.0%

* modules/base coverage: 67.6% -> 89.9%

* modules/cache coverage: 0% -> 12.0%

* modules/convert coverage: 27.1% -> 29.7%
2021-01-28 19:08:11 +01:00
Lunny Xiao 154b23da0a
Fix display since time round (#14226)
* Fix display since time round

* Fix since time

* Fix tests
2021-01-28 13:29:22 +01:00
6543 a7cfb9f2c3
Fix JSON Header response (#14499) 2021-01-27 15:46:22 -06:00
Lunny Xiao a51cc6dea4
Fix access log (#14475)
Fix #14121, #14478.

The `AccessLog` middleware has to be after `Contexter` or `APIContexter` so that we can get `LoginUserName` if possible.
And also there is a **BREAK** change that it removed internal API access log.
2021-01-27 18:46:35 +01:00
Lunny Xiao 41c0776568
Fix captcha (#14488)
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-01-27 22:56:54 +08:00
Lunny Xiao 669ff8e9b1
Fix switch language (#14484)
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-01-27 16:20:42 +02:00
PhilAtWysdom af7f71207c
Fix: url.Values map was not initialized (#14485)
Values map was not initialized, leading to error 500 on submission of initial configuration

Co-authored-by: 6543 <6543@obermui.de>
2021-01-27 15:33:32 +02:00
Jui-Nan Lin c10503afec
[Feature] add precise search type for Elastic Search (#12869)
* feat: add type query parameters for specifying precise search

* feat: add select dropdown in search box

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-01-27 12:00:35 +02:00
Brad Albright a598877fdf
Cron job to cleanup hook_task table (#13080)
Close **Prune hook_task Table (#10741)**

Added a cron job to delete webhook deliveries in the hook_task table. It can be turned on/off and the schedule controlled globally via app.ini. The data can be deleted by either the age of the delivery which is the default or by deleting the all but the most recent deliveries _per webhook_.

Note: I had previously submitted pr #11416  but I closed it when I realized that I had deleted per repository instead of per webhook. Also, I decided allowing the settings to be overridden via the ui was overkill. Also this version allows the deletion by age which is probably what most people would want.
2021-01-26 22:02:42 +01:00
Lunny Xiao 6433ba0ec3
Move macaron to chi (#14293)
Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR.

- [x] Define `context.ResponseWriter` interface with an implementation `context.Response`.
- [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before.
- [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic .
- [x] Use https://github.com/unrolled/render instead of macaron's internal render
- [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip
- [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK**
- [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha
- [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache
- [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding
- [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors
- [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation`
- [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle.
- [x] Removed macaron log service because it's not need any more. **BREAK**
- [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition.
- [x] Move Git HTTP protocol implementation to use routers directly.
- [x] Fix the problem that chi routes don't support trailing slash but macaron did.
- [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render. 

Notices:
- Chi router don't support request with trailing slash
- Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI.

Co-authored-by: 6543 <6543@obermui.de>
2021-01-26 16:36:53 +01:00
Dofri Jónsson 012bf388ab
Fix incorrect key name so registerManualConfirm setting works as expected. (#14455) 2021-01-26 09:20:59 +08:00
Andrew Bezold bc05ddc0eb
Redirect on changed user and org name (#11649)
* Add redirect for user

* Add redirect for orgs

* Add user redirect test

* Appease linter

* Add comment to DeleteUserRedirect function

* Fix locale changes

* Fix GetUserByParams

* Fix orgAssignment

* Remove debug logging

* Add redirect prompt

* Dont Export DeleteUserRedirect & only use it within a session

* Unexport newUserRedirect

* cleanup

* Fix & Dedub API code

* Format Template

* Add Migration & rm dublicat

* Refactor: unexport newRepoRedirect() & rm dedub del exec

* if this fails we'll need to re-rename the user directory

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-01-24 16:23:05 +01:00
6543 4acb499f30
just overload to not get it by mistake again ... (#14440) 2021-01-23 21:33:43 +02:00
6543 a0e424da85
Enhance Ghost comment mitigation Settings (#14392)
* refactor models.DeleteComment and delete related reactions too

* use deleteComment for UserDeleteWithCommentsMaxDays in DeleteUser

* nits

* Use time.Duration as other time settings have

* docs

* Resolve Fixme & fix potential deadlock

* Disabled by Default

* Update Config Value Description

* switch args

* Update models/issue_comment.go

Co-authored-by: zeripath <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-01-21 21:56:19 -05:00
6543 81c833d92d
Add support to migrate from gogs (#14342)
Add support to migrate gogs:

  *  issues
  *  comments
  *  labels
  *  milestones
  *  wiki


Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-01-21 20:33:58 +01:00
Norwin b5570d3e68
Display current stopwatch in navbar (#14122)
* add notification about running stopwatch to header

* serialize seconds, duration in stopwatches api

* ajax update stopwatch

i should get my testenv working locally...

* new variant: hover dialog

* noscript compatibility

* js: live-update stopwatch time

* js live update robustness
2021-01-21 15:51:52 +01:00
Artemis Tosini cb08248c33
Add support for ed25519_sk and ecdsa_sk SSH keys (#13462)
* Add support for ed25519_sk and ecdsa_sk SSH keys

These start with sk-ssh-ed25519@openssh.com and sk-ecdsa-sha2-nistp256@openssh.com.
They are supported in recent versions of go x/crypto/ssh and OpenSSH 8.2
or higher.

* skip ssh-keygen

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-01-20 20:36:55 +00:00
zeripath 172229966c
Prevent panic on fuzzer provided string (#14405)
* Prevent panic on fuzzer provided string

The fuzzer has found that providing a <body> tag with an attribute to
PostProcess causes a panic. This PR removes any rendered html or body
tags from the output.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Placate lint

* placate lint again

Signed-off-by: Andrew Thornton <art27@cantab.net>

* minor cleanup

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-01-20 23:10:50 +08:00
Lunny Xiao 135b0e502d
Fix log http status is always zero (#14400)
* Fix log http status is always zero

* Fix lint

Co-authored-by: 6543 <6543@obermui.de>
2021-01-19 20:47:43 -05:00
Chester Liu 0c0445c97a
Add pager to the branches page (#14202)
* Add pager to the branches page

* override pageSize if bigger than max

* Make branches commit range configurable

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: silverwind <me@silverwind.io>
2021-01-19 05:07:38 +01:00
6543 2686e6bbbe
Check if label template exist first (#14384)
* add check

* refactor

* rollback repo on error after session closed
2021-01-18 21:00:50 +01:00
Lauris BH f5abe2f563
Upgrade blevesearch dependency to v2.0.1 (#14346)
* Upgrade blevesearch dependency to v2.0.1

* Update rupture to v1.0.0

* Fix test
2021-01-17 20:21:14 -05:00
6543 21da519c0c
Implement ghost comment mitigation (#14349)
* Implement ghost comment mitigation

Adds a config option USER_DELETE_WITH_COMMENTS_MAX_DAYS to the [service] section. See https://codeberg.org/Codeberg/Discussion/issues/24 for the underlying issue.

* cleanup

* use setting module correctly

* add to docs

Co-authored-by: Moritz Marquardt <git@momar.de>
2021-01-17 21:48:38 +01:00
Lauris BH 0a3c3357f3
Sort issue search results by revelance (#14353) 2021-01-16 12:55:17 +08:00
Kyle D bfd0c47ef6
Kd/fix allow svg doctype (#14344)
* make svg regex case-insensitive & use strict word boundary

* allow doctype svg

* add doctype tests

* allow <!DOCTYPE svg> and <svg/>
2021-01-15 17:38:41 +08:00
Lunny Xiao 84b147c7f0
Use IsProd instead of testing if it's equal. (#14336)
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-14 23:17:03 +02:00
Lunny Xiao f76c30094f
Fix typo (#14332) 2021-01-14 23:35:10 +08:00
Jonathan Tran 81467e6f35
Display SVG files as images instead of text (#14101)
* Change to display SVG files as images

* Remove unsafe styles from SVG CSP

* Add integration test to test SVG headers

* Add config setting to disable SVG rendering

* Add test for img tag when loading SVG image

* Remove the Raw view button for svg files since we don't fully support this

* Fix copyright year

* Rename and move config setting

* Add setting to cheat sheet in docs

* Fix so that comment matches cheat sheet

* Add allowing styles in CSP based on pull request feedback

* Re-enable raw button since we show SVG styles now

* Change so that SVG files are editable

* Add UI to toggle between source and rendered image for SVGs

* Change to show blame button for SVG images

* Fix to update ctx data

* Add test for DetectContentType when file is longer than sniffLen

Co-authored-by: Jonathan Tran <jon@allspice.io>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
2021-01-12 22:45:19 -05:00
6543 6b3b6f1833
Add option to change username to the admin panel (#14229)
Co-authored-by: Bwko <bouwko@gmail.com>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-10 14:14:02 +02:00
Jimmy Praet bcb7f35221
Do not reload page after adding comments in Pull Request reviews (#13877)
Fixed #8861 
* use ajax on PR review page

* handle review comments

* extract duplicate code

FetchCodeCommentsByLine was initially more or less copied from fetchCodeCommentsByReview. Now they both use a common findCodeComments function instead

* use the Engine that was passed into the method

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-01-08 23:49:55 +02:00
Lunny Xiao 3c96a37162
Some code improvements (#14266) 2021-01-06 19:23:57 +00:00
Lunny Xiao a1c9e8f266
Fix windows build error (#14263)
* fix build

* take flash error message back and fix more windows lint error

* performance optimization

* own step to check lint for windows

Co-authored-by: 6543 <6543@obermui.de>
2021-01-06 09:38:00 +08:00
6543 325add71cf
Add option for administrator to reset user 2FA (#14243)
* Frontend

* Backend

* only show 2FA-Reset option if posible
2021-01-05 14:54:48 +01:00
Lunny Xiao 15a475b7db
Fix recovery middleware to render gitea style page. (#13857)
* Some changes to fix recovery

* Move Recovery to middlewares

* Remove trace code

* Fix lint

* add session middleware and remove dependent on macaron for sso

* Fix panic 500 page rendering

* Fix bugs

* Fix fmt

* Fix vendor

* recover unnecessary change

* Fix lint and addd some comments about the copied codes.

* Use util.StatDir instead of com.StatDir

Co-authored-by: 6543 <6543@obermui.de>
2021-01-05 21:05:40 +08:00
Nuno Silva 44ff1d6a1e
Render links for commit hashes followed by comma (#14224)
Regex test cases: https://regex101.com/r/mVbPxM/2/

fixes #14223
2021-01-03 23:11:10 +08:00
Paul Barton 7576e37a65
Add MirrorInterval to the API (#14163)
* Added MirrorInterval to the API

* Remove MirrorInterval from CreateRepository

* Removed Duplicate UpdateMirror Function

* Updated Error Logging

* Update Log Message for is not Mirror

Co-authored-by: 6543 <6543@obermui.de>

* Delete Debug Statement that snuck in

Co-authored-by: zeripath <art27@cantab.net>

* Add Check for If Interval is too small

* Output to API Call

* Add Error Object when time is Less than Min Interval

* Frequency Error Message

Co-authored-by: zeripath <art27@cantab.net>

* Allow Zero Mirror Interval

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-02 18:47:47 -05:00
Jimmy Praet e6acce649b
Send notifications for mentions in pulls, issues, (code-)comments (#14218)
Fixes #14187: mention handling extracted from email notification code
Fixes #14013: add notification for mentions in pull request code comments
Fixes #13450: Not receiving any emails with setting "Only Email on Mention"
2021-01-02 18:04:02 +01:00
Lunny Xiao d44f192d3e
When visit /favicon.ico but the static file is not exist return 404 but not continue to handle the route (#14211)
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-01-01 17:05:45 -05:00
silverwind 1bf7d71a0a
Consolidate Logos and update README header (#14136)
* Consolidate Logos and update README header

- Remove unused `logo-lg.png`, `logo-sm.png` and `logo-192.png`.
- Consolidate `favicon.svg` and `logo.svg` to just `logo.svg`.
- Remove Safari Mask icon, it seems to work fine with just `favicon.png` (no SVG support).
- Remove Fluid Icon. It only served Firefox and SVG works just fine there.
- Update customization instructions.
- Update README.md to use SVG icon, increase logo size and center it and badges.

* Update README_ZH.md

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>

* Update README_ZH.md

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-01-01 14:04:35 -05:00
Johnny Oskarsson a07e67d9cc
Minimal OpenID Connect implementation (#14139)
This is "minimal" in the sense that only the Authorization Code Flow
from OpenID Connect Core is implemented.  No discovery, no configuration
endpoint, and no user scope management.

OpenID Connect is an extension to the (already implemented) OAuth 2.0
protocol, and essentially an `id_token` JWT is added to the access token
endpoint response when using the Authorization Code Flow.  I also added
support for the "nonce" field since it is required to be used in the
id_token if the client decides to include it in its initial request.

In order to enable this extension an OAuth 2.0 scope containing
"openid" is needed. Other OAuth 2.0 requests should not be impacted by
this change.

This minimal implementation is enough to enable single sign-on (SSO)
for other sites, e.g. by using something like `mod_auth_openidc` to
only allow access to a CI server if a user has logged into Gitea.

Fixes: #1310

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-02 00:33:27 +08:00
Chester Liu 632800eda7
Check for 'main' as potential default branch name (#14193) 2020-12-30 15:46:26 +00:00
silverwind 8e5aea88c7
Improve label and text wrapping (#14113)
* Improve label wrapping

- Adjust issue list styles so labels can wrap on the same line as the
  text. This relies on `display: inline` with the HTML whitespace being
  used as the separator.
- Add global word-break: break-word. This should generally avoid text
  overflows in various places.

* add whitespace to history labels

* use overflow-wrap

* restore word-break rules

* use correct pre

* use better selector for middle align

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-12-29 18:48:28 -05:00
Lunny Xiao 11555d850b
Fix bug of link query order on markdown render (#14156)
* Fix bug of link query order on markdown render

* Fix bluemonday bug and fix one wrong test

Co-authored-by: 6543 <6543@obermui.de>
2020-12-29 00:28:27 +08:00
Timo Ley 2b2a4a53bf
Custom icons for OAuth sources (#14161)
* Add Icon URL to Backend

* Template for Icon URL

* Localization & Edit Icon URL

* Improve Custom Icon URL

* Removed not working else

Co-authored-by: 6543 <6543@obermui.de>
2020-12-28 03:35:55 +01:00
Lunny Xiao dd08853b10
Dump github/gitlab/gitea repository data to a local directory and restore to gitea (#12244)
* Dump github/gitlab repository data to a local directory

* Fix lint

* Adjust directory structure

* Allow migration special units

* Allow migration ignore release assets

* Fix lint

* Add restore repository

* stage the changes

* Merge

* Fix lint

* Update the interface

* Add some restore methods

* Finish restore

* Add comments

* Fix restore

* Add a token flag

* Fix bug

* Fix test

* Fix test

* Fix bug

* Fix bug

* Fix lint

* Fix restore

* refactor downloader

* fmt

* Fix bug isEnd detection on getIssues

* Refactor maxPerPage

* Remove unused codes

* Remove unused codes

* Fix bug

* Fix restore

* Fix dump

* Uploader should not depend downloader

* use release attachment name but not id

* Fix restore bug

* Fix lint

* Fix restore bug

* Add a method of DownloadFunc for base.Release to make uploader not depend on downloader

* fix Release yml marshal

* Fix trace information

* Fix bug when dump & restore

* Save relative path on yml file

* Fix bug

* Use relative path

* Update docs

* Use git service string but not int

* Recognize clone addr to service type
2020-12-27 11:34:19 +08:00
6543 212fa340cf
Migration: drop to long repo topics (#14152) 2020-12-27 09:23:57 +08:00
6543 a19447aed1
migrate from com.* to alternatives (#14103)
* remove github.com/unknwon/com from models

* dont use "com.ToStr()"

* replace "com.ToStr" with "fmt.Sprint" where its easy to do

* more refactor

* fix test

* just "proxy" Copy func for now

* as per @lunny
2020-12-25 11:59:32 +02:00
Daniil Pankratov 5a94db37ed
Fix creating OAuth2 auth source from CLI (#14116)
Fix creation OAuth2 auth source from CLI.

Fix #8356

Co-authored-by: Daniil Pankratov <daniil.pankratov@t-systems.com>
2020-12-24 19:47:17 +00:00
Lunny Xiao 19ae6439b0
Improve vfsgen to not unzip bindata files but send to browser directly (#7109)
* Don't unzip files from bindata but send to browser directly

* remove dependent for httpgzip

* Add tests for parseAcceptEncoding

* Update docs for ENABLE_GZIP

* Fix bug

* Fix bug

Co-authored-by: zeripath <art27@cantab.net>
2020-12-24 12:25:17 +08:00
6543 5b958315c1
API: Make BasicAuth Prefix case insensitive (#14106) 2020-12-23 20:52:46 +01:00