Commit graph

28 commits

Author SHA1 Message Date
Jason Song e253888a0e
Fix isAllowed of escapeStreamer ()
The use of `sort.Search` is wrong: The slice should be sorted, and
`return >= 0` doen't mean it exists, see the
[manual](https://pkg.go.dev/sort#Search).

Could be fixed like this if we really need it:

```diff
diff --git a/modules/charset/escape_stream.go b/modules/charset/escape_stream.go
index 823b63513..fcf1ffbc1 100644
--- a/modules/charset/escape_stream.go
+++ b/modules/charset/escape_stream.go
@@ -20,6 +20,9 @@ import (
 var defaultWordRegexp = regexp.MustCompile(`(-?\d*\.\d\w*)|([^\` + "`" + `\~\!\@\#\$\%\^\&\*\(\)\-\=\+\[\{\]\}\\\|\;\:\'\"\,\.\<\>\/\?\s\x00-\x1f]+)`)

 func NewEscapeStreamer(locale translation.Locale, next HTMLStreamer, allowed ...rune) HTMLStreamer {
+       sort.Slice(allowed, func(i, j int) bool {
+               return allowed[i] < allowed[j]
+       })
        return &escapeStreamer{
                escaped:                 &EscapeStatus{},
                PassthroughHTMLStreamer: *NewPassthroughStreamer(next),
@@ -284,14 +287,8 @@ func (e *escapeStreamer) runeTypes(runes ...rune) (types []runeType, confusables
 }

 func (e *escapeStreamer) isAllowed(r rune) bool {
-       if len(e.allowed) == 0 {
-               return false
-       }
-       if len(e.allowed) == 1 {
-               return e.allowed[0] == r
-       }
-
-       return sort.Search(len(e.allowed), func(i int) bool {
+       i := sort.Search(len(e.allowed), func(i int) bool {
                return e.allowed[i] >= r
-       }) >= 0
+       })
+       return i < len(e.allowed) && e.allowed[i] == r
 }
```

But I don't think so, a map is better to do it.
2023-02-09 20:51:36 +08:00
crystal 5d9c64b3fe
Fix line spacing for plaintext previews ()
Adding `<br>` between each line is not necessary since the entire file
is rendered inside a `<pre>`

fixes https://codeberg.org/Codeberg/Community/issues/915
2023-02-01 22:51:02 -06:00
zeripath 6e22605793
Ensure that plain files are rendered correctly even when containing ambiguous characters ()
As recognised in  the rendering of plain text files is somewhat
incorrect when there are ambiguous characters as the html code is double
escaped. In fact there are several more problems here.

We have a residual isRenderedHTML which is actually simply escaping the
file - not rendering it. This is badly named and gives the wrong
impression.

There is also unusual behaviour whether the file is called a Readme or
not and there is no way to get to the source code if the file is called
README.

In reality what should happen is different depending on whether the file
is being rendered a README at the bottom of the directory view or not.

1. If it is rendered as a README on a directory - it should simply be
escaped and rendered as `<pre>` text.
2. If it is rendered as a file then it should be rendered as source
code.

This PR therefore does:
1. Rename IsRenderedHTML to IsPlainText
2. Readme files rendered at the bottom of the directory are rendered
without line numbers
3. Otherwise plain text files are rendered as source code.

Replace 

Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-12-17 22:22:25 +02:00
silverwind 0585ac3ac6
Update go dev dependencies ()
`golangci-lint`
[deprecated](https://github.com/golangci/golangci-lint/issues/1841) a
bunch of linters, removed them.
2022-12-08 16:21:37 +08:00
zeripath a08584ee36
Ensure that Chinese punctuation is not ambiguous when locale is Chinese ()
Although there are per-locale fallbacks for ambiguity the locale names
for Chinese do not quite match our locales. This PR simply maps zh-CN on
to zh-hans and other zh variants on to zh-hant.

Ref 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-12-04 17:57:30 +00:00
flynnnnnnnnnn e81ccc406b
Implement FSFE REUSE for golang files ()
Change all license headers to comply with REUSE specification.

Fix 

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
zeripath 8080e23c9b
Move go-licenses to generate and separate generate into a frontend and backend component ()
The `go-licenses` make task introduced in  is being run on make vendor
and occasionally causes an empty go-licenses file if the vendors need to
change. This should be moved to the generate task as it is a generated file.

Now because of this change we also need to split generation into two separate 
steps:

1. `generate-backend`
2. `generate-frontend`

In the future it would probably be useful to make `generate-swagger` part of `generate-frontend` but it's not tolerated with our .drone.yml

Ref 

Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
2022-09-05 14:04:18 +08:00
zeripath bb0ff77e46
Share HTML template renderers and create a watcher framework ()
The recovery, API, Web and package frameworks all create their own HTML
Renderers. This increases the memory requirements of Gitea
unnecessarily with duplicate templates being kept in memory.

Further the reloading framework in dev mode for these involves locking
and recompiling all of the templates on each load. This will potentially
hide concurrency issues and it is inefficient.

This PR stores the templates renderer in the context and stores this
context in the NormalRoutes, it then creates a fsnotify.Watcher
framework to watch files.

The watching framework is then extended to the mailer templates which
were previously not being reloaded in dev.

Then the locales are simplified to a similar structure.

Fix  
Fix 
Fix 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-08-28 10:43:25 +01:00
Jason Song 15b189b570
Avoid frequent string2bytes conversions ()
Fix 
2022-08-24 12:50:13 +01:00
zeripath 99efa02edf
Switch Unicode Escaping to a VSCode-like system ()
This PR rewrites the invisible unicode detection algorithm to more
closely match that of the Monaco editor on the system. It provides a
technique for detecting ambiguous characters and relaxes the detection
of combining marks.

Control characters are in addition detected as invisible in this
implementation whereas they are not on monaco but this is related to
font issues.

Close 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-08-13 19:32:34 +01:00
luzpaz d29d6d1991
Fix various typos ()
* Fix various typos

Found via `codespell -q 3 -S ./options/locale,./options/license,./public/vendor -L actived,allways,attachements,ba,befores,commiter,pullrequest,pullrequests,readby,splitted,te,unknwon`

Co-authored-by: zeripath <art27@cantab.net>
2022-07-12 23:32:37 +02:00
Wim cb50375e2b
Add more linters to improve code readability ()
Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability

- nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length.
- unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions
- wastedassign - https://github.com/sanposhiho/wastedassign -  wastedassign finds wasted assignment statements.
- notlintlint -  Reports ill-formed or insufficient nolint directives
- stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent
  - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
2022-06-20 12:02:49 +02:00
zeripath bc4764ffc6
Detect truncated utf-8 characters at the end of content as still representing utf-8 ()
Our character detection algorithm can potentially incorrectly detect utf-8 as iso-8859-x
if there is a truncated character at the end of the partially read file.

This PR changes the detection algorithm to truncated utf8 characters at the end of the
buffer.

Fix 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-05-21 14:06:24 +01:00
Gusted bf2867dec2
Don't treat BOM escape sequence as hidden character. ()
* Don't treat BOM escape sequence as hidden character.

- BOM sequence is a common non-harmfull escape sequence, it shouldn't be
shown as hidden character.
- Follows GitHub's behavior.
- Resolves 

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-02-26 16:48:23 +00:00
zeripath 4b3ebda0e7
Fix panic in EscapeReader ()
There is a potential panic due to a mistaken resetting of the length parameter when
multibyte characters go over a read boundary.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-02-19 15:25:31 +00:00
6543 54e9ee37a7
format with gofumpt ()
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
zeripath 21ed4fd8da
Add warning for BIDI characters in page renders and in diffs ()
Fix 

Given the comments I've adjusted this somewhat. The numbers of characters detected are increased and include things like the use of U+300 to make à instead of à and non-breaking spaces.

There is a button which can be used to escape the content to show it.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Gwyneth Morgan <gwymor@tilde.club>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-01-07 02:18:52 +01:00
Gusted ff2fd08228
Simplify parameter types ()
Remove repeated type declarations in function definitions.
2021-12-20 04:41:31 +00:00
KN4CK3R f99d50fc9f
Read expected buffer size ()
* Read expected buffer size.

* Changed name.
2021-10-24 22:12:43 +01:00
Eng Zer Jun f2e7d5477f
refactor: move from io/ioutil to io and os package ()
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-09-22 13:38:34 +08:00
Lunny Xiao 9d99f6ab19
Refactor renders ()
* Refactor renders

* Some performance optimization

* Fix comment

* Transform reader

* Fix csv test

* Fix test

* Fix tests

* Improve optimaziation

* Fix test

* Fix test

* Detect file encoding with reader

* Improve optimaziation

* reduce memory usage

* improve code

* fix build

* Fix test

* Fix for go1.15

* Fix render

* Fix comment

* Fix lint

* Fix test

* Don't use NormalEOF when unnecessary

* revert change on util.go

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* rename function

* Take NormalEOF back

Co-authored-by: zeripath <art27@cantab.net>
2021-04-19 18:25:08 -04:00
zeripath e429c1164e
Ensure that the detected charset order is set in chardet test ()
TestToUTF8WithFallback is the cause of recurrent spurious test failures
even despite code to set the detected charset order.

The reason why this happens is because the preferred detected charset order
is not being initialised for these tests.

This PR simply ensures that this is set at the start of each test and would
allow different tests to be written to allow differing orders.

Replaces 
Close 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-08-23 14:15:29 +01:00
zeripath a1ad188326
Fix chardet test and add ordering option ()
* Fix chardet test and add ordering option

Signed-off-by: Andrew Thornton <art27@cantab.net>

* minor fixes

Signed-off-by: Andrew Thornton <art27@cantab.net>

* remove log

Signed-off-by: Andrew Thornton <art27@cantab.net>

* remove log2

Signed-off-by: Andrew Thornton <art27@cantab.net>

* only iterate through top results

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

* slight restructure of for loop

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-06-02 19:20:19 -03:00
Antoine GIRARD 81a52442a1 deps: update and fix chardet import () 2019-12-14 02:15:48 +02:00
guillep2k 356e1a70ea Reduce test sensibility () 2019-10-07 01:49:14 -04:00
guillep2k 2628b15ee3 Fix utf8 tests ()
* Prevent compiler environment from making the tests fail

* Remove unused function

* Pass lint
2019-09-21 13:01:34 -04:00
guillep2k 6097ff68e7 Make encoding tests independent of LOCALE settings ()
* Make encoding tests independent of LOCALE settings

* Fix fmt

* Force CI to restart
2019-09-02 19:08:07 -04:00
guillep2k 5a44be627c Convert files to utf-8 for indexing ()
* Convert files to utf-8 for indexing

* Move utf8 functions to modules/base

* Bump repoIndexerLatestVersion to 3

* Add tests for base/encoding.go

* Changes to pass gosimple

* Move UTF8 funcs into new modules/charset package
2019-08-15 20:07:28 +08:00