summaryrefslogtreecommitdiffstats
path: root/modules/charset
Commit message (Collapse)AuthorAgeFilesLines
* Update go tool dependencies, restructure lint targets (#24239)silverwind2023-04-221-1/+3
| | | | | | | | - Update all tool dependencies to latest tag - Remove unused errcheck, it is part of golangci-lint - Include main.go in air - Enable wastedassign again now that it's [generics-compatible](https://github.com/golangci/golangci-lint/pull/3689) - Restructured lint targets to new `lint-*` namespace
* Refactor locale number (#24134)wxiaoguang2023-04-171-11/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Before, the `GiteaLocaleNumber.js` was just written as a a drop-in replacement for old `js-pretty-number`. Actually, we can use Golang's `text` package to format. This PR partially completes the TODOs in `GiteaLocaleNumber.js`: > if we have complete backend locale support (eg: Golang "x/text" package), we can drop this component. > tooltip: only 2 usages of this, we can replace it with Golang's "x/text/number" package in the future. This PR also helps #24131 Screenshots: <details> ![image](https://user-images.githubusercontent.com/2114189/232179420-b1b9974b-9d96-4408-b209-b80182c8b359.png) ![image](https://user-images.githubusercontent.com/2114189/232179416-14f36aa0-3f3e-4ac9-b366-7bd3a4464a11.png) </details>
* Use data-tooltip-content for tippy tooltip (#23649)wxiaoguang2023-03-242-5/+5
| | | | | | | | | | | | | | | | | Follow: * #23574 * Remove all ".tooltip[data-content=...]" Major changes: * Remove "tooltip" class, use "[data-tooltip-content=...]" instead of ".tooltip[data-content=...]" * Remove legacy `data-position`, it's dead code since last Fomantic Tooltip -> Tippy Tooltip refactoring * Rename reaction attribute from `data-content` to `data-reaction-content` * Add comments for some `data-content`: `{{/* used by the form */}}` * Remove empty "ui" class * Use "text color" for SVG icons (a few)
* Fix isAllowed of escapeStreamer (#22814)Jason Song2023-02-091-17/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The use of `sort.Search` is wrong: The slice should be sorted, and `return >= 0` doen't mean it exists, see the [manual](https://pkg.go.dev/sort#Search). Could be fixed like this if we really need it: ```diff diff --git a/modules/charset/escape_stream.go b/modules/charset/escape_stream.go index 823b63513..fcf1ffbc1 100644 --- a/modules/charset/escape_stream.go +++ b/modules/charset/escape_stream.go @@ -20,6 +20,9 @@ import ( var defaultWordRegexp = regexp.MustCompile(`(-?\d*\.\d\w*)|([^\` + "`" + `\~\!\@\#\$\%\^\&\*\(\)\-\=\+\[\{\]\}\\\|\;\:\'\"\,\.\<\>\/\?\s\x00-\x1f]+)`) func NewEscapeStreamer(locale translation.Locale, next HTMLStreamer, allowed ...rune) HTMLStreamer { + sort.Slice(allowed, func(i, j int) bool { + return allowed[i] < allowed[j] + }) return &escapeStreamer{ escaped: &EscapeStatus{}, PassthroughHTMLStreamer: *NewPassthroughStreamer(next), @@ -284,14 +287,8 @@ func (e *escapeStreamer) runeTypes(runes ...rune) (types []runeType, confusables } func (e *escapeStreamer) isAllowed(r rune) bool { - if len(e.allowed) == 0 { - return false - } - if len(e.allowed) == 1 { - return e.allowed[0] == r - } - - return sort.Search(len(e.allowed), func(i int) bool { + i := sort.Search(len(e.allowed), func(i int) bool { return e.allowed[i] >= r - }) >= 0 + }) + return i < len(e.allowed) && e.allowed[i] == r } ``` But I don't think so, a map is better to do it.
* Fix line spacing for plaintext previews (#22699)crystal2023-02-011-5/+1
| | | | | | Adding `<br>` between each line is not necessary since the entire file is rendered inside a `<pre>` fixes https://codeberg.org/Codeberg/Community/issues/915
* Ensure that plain files are rendered correctly even when containing ↵zeripath2022-12-171-1/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ambiguous characters (#22017) As recognised in #21841 the rendering of plain text files is somewhat incorrect when there are ambiguous characters as the html code is double escaped. In fact there are several more problems here. We have a residual isRenderedHTML which is actually simply escaping the file - not rendering it. This is badly named and gives the wrong impression. There is also unusual behaviour whether the file is called a Readme or not and there is no way to get to the source code if the file is called README. In reality what should happen is different depending on whether the file is being rendered a README at the bottom of the directory view or not. 1. If it is rendered as a README on a directory - it should simply be escaped and rendered as `<pre>` text. 2. If it is rendered as a file then it should be rendered as source code. This PR therefore does: 1. Rename IsRenderedHTML to IsPlainText 2. Readme files rendered at the bottom of the directory are rendered without line numbers 3. Otherwise plain text files are rendered as source code. Replace #21841 Signed-off-by: Andrew Thornton <art27@cantab.net> Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Update go dev dependencies (#22064)silverwind2022-12-081-1/+1
| | | | | `golangci-lint` [deprecated](https://github.com/golangci/golangci-lint/issues/1841) a bunch of linters, removed them.
* Ensure that Chinese punctuation is not ambiguous when locale is Chinese (#22019)zeripath2022-12-041-0/+6
| | | | | | | | | | Although there are per-locale fallbacks for ambiguity the locale names for Chinese do not quite match our locales. This PR simply maps zh-CN on to zh-hans and other zh variants on to zh-hant. Ref #20999 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Implement FSFE REUSE for golang files (#21840)flynnnnnnnnnn2022-11-2715-34/+19
| | | | | | | | | Change all license headers to comply with REUSE specification. Fix #16132 Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github> Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Move go-licenses to generate and separate generate into a frontend and ↵zeripath2022-09-052-0/+22
| | | | | | | | | | | | | | | | | | | | | | backend component (#21061) The `go-licenses` make task introduced in #21034 is being run on make vendor and occasionally causes an empty go-licenses file if the vendors need to change. This should be moved to the generate task as it is a generated file. Now because of this change we also need to split generation into two separate steps: 1. `generate-backend` 2. `generate-frontend` In the future it would probably be useful to make `generate-swagger` part of `generate-frontend` but it's not tolerated with our .drone.yml Ref #21034 Signed-off-by: Andrew Thornton <art27@cantab.net> Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: delvh <dev.lh@web.de>
* Share HTML template renderers and create a watcher framework (#20218)zeripath2022-08-281-4/+11
| | | | | | | | | | | | | | | | | | | | | | | | | The recovery, API, Web and package frameworks all create their own HTML Renderers. This increases the memory requirements of Gitea unnecessarily with duplicate templates being kept in memory. Further the reloading framework in dev mode for these involves locking and recompiling all of the templates on each load. This will potentially hide concurrency issues and it is inefficient. This PR stores the templates renderer in the context and stores this context in the NormalRoutes, it then creates a fsnotify.Watcher framework to watch files. The watching framework is then extended to the mailer templates which were previously not being reloaded in dev. Then the locales are simplified to a similar structure. Fix #20210 Fix #20211 Fix #20217 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Avoid frequent string2bytes conversions (#20940)Jason Song2022-08-241-7/+8
| | | Fix #20939
* Switch Unicode Escaping to a VSCode-like system (#19990)zeripath2022-08-1314-273/+1956
| | | | | | | | | | | | | | | This PR rewrites the invisible unicode detection algorithm to more closely match that of the Monaco editor on the system. It provides a technique for detecting ambiguous characters and relaxes the detection of combining marks. Control characters are in addition detected as invisible in this implementation whereas they are not on monaco but this is related to font issues. Close #19913 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix various typos (#20338)luzpaz2022-07-121-3/+3
| | | | | | | * Fix various typos Found via `codespell -q 3 -S ./options/locale,./options/license,./public/vendor -L actived,allways,attachements,ba,befores,commiter,pullrequest,pullrequests,readby,splitted,te,unknwon` Co-authored-by: zeripath <art27@cantab.net>
* Add more linters to improve code readability (#19989)Wim2022-06-202-5/+5
| | | | | | | | | | Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability - nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length. - unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions - wastedassign - https://github.com/sanposhiho/wastedassign - wastedassign finds wasted assignment statements. - notlintlint - Reports ill-formed or insufficient nolint directives - stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
* Detect truncated utf-8 characters at the end of content as still ↵zeripath2022-05-212-1/+164
| | | | | | | | | | | | | | representing utf-8 (#19773) Our character detection algorithm can potentially incorrectly detect utf-8 as iso-8859-x if there is a truncated character at the end of the partially read file. This PR changes the detection algorithm to truncated utf8 characters at the end of the buffer. Fix #19743 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Don't treat BOM escape sequence as hidden character. (#18909)Gusted2022-02-262-4/+25
| | | | | | | | | | | * Don't treat BOM escape sequence as hidden character. - BOM sequence is a common non-harmfull escape sequence, it shouldn't be shown as hidden character. - Follows GitHub's behavior. - Resolves #18837 Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Fix panic in EscapeReader (#18820)zeripath2022-02-192-0/+10
| | | | | | There is a potential panic due to a mistaken resetting of the length parameter when multibyte characters go over a read boundary. Signed-off-by: Andrew Thornton <art27@cantab.net>
* format with gofumpt (#18184)65432022-01-202-33/+65
| | | | | | | | | | | * gofumpt -w -l . * gofumpt -w -l -extra . * Add linter * manual fix * change make fmt
* Add warning for BIDI characters in page renders and in diffs (#17562)zeripath2022-01-072-0/+432
| | | | | | | | | | | | Fix #17514 Given the comments I've adjusted this somewhat. The numbers of characters detected are increased and include things like the use of U+300 to make à instead of à and non-breaking spaces. There is a button which can be used to escape the content to show it. Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: Gwyneth Morgan <gwymor@tilde.club> Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Simplify parameter types (#18006)Gusted2021-12-201-3/+3
| | | Remove repeated type declarations in function definitions.
* Read expected buffer size (#17409)KN4CK3R2021-10-241-2/+3
| | | | | | * Read expected buffer size. * Changed name.
* refactor: move from io/ioutil to io and os package (#17109)Eng Zer Jun2021-09-221-2/+1
| | | | | | | | | The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactor renders (#15175)Lunny Xiao2021-04-191-18/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Refactor renders * Some performance optimization * Fix comment * Transform reader * Fix csv test * Fix test * Fix tests * Improve optimaziation * Fix test * Fix test * Detect file encoding with reader * Improve optimaziation * reduce memory usage * improve code * fix build * Fix test * Fix for go1.15 * Fix render * Fix comment * Fix lint * Fix test * Don't use NormalEOF when unnecessary * revert change on util.go * Apply suggestions from code review Co-authored-by: zeripath <art27@cantab.net> * rename function * Take NormalEOF back Co-authored-by: zeripath <art27@cantab.net>
* Ensure that the detected charset order is set in chardet test (#12574)zeripath2020-08-231-5/+23
| | | | | | | | | | | | | | | TestToUTF8WithFallback is the cause of recurrent spurious test failures even despite code to set the detected charset order. The reason why this happens is because the preferred detected charset order is not being initialised for these tests. This PR simply ensures that this is set at the start of each test and would allow different tests to be written to allow differing orders. Replaces #12571 Close #12571 Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix chardet test and add ordering option (#11621)zeripath2020-06-022-4/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix chardet test and add ordering option Signed-off-by: Andrew Thornton <art27@cantab.net> * minor fixes Signed-off-by: Andrew Thornton <art27@cantab.net> * remove log Signed-off-by: Andrew Thornton <art27@cantab.net> * remove log2 Signed-off-by: Andrew Thornton <art27@cantab.net> * only iterate through top results Signed-off-by: Andrew Thornton <art27@cantab.net> * Update docs/content/doc/advanced/config-cheat-sheet.en-us.md * slight restructure of for loop Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* deps: update and fix chardet import (#9351)Antoine GIRARD2019-12-141-1/+1
|
* Reduce test sensibility (#8393)guillep2k2019-10-071-1/+2
|
* Fix utf8 tests (#8192)guillep2k2019-09-212-35/+81
| | | | | | | | * Prevent compiler environment from making the tests fail * Remove unused function * Pass lint
* Make encoding tests independent of LOCALE settings (#8018)guillep2k2019-09-021-12/+26
| | | | | | | | * Make encoding tests independent of LOCALE settings * Fix fmt * Force CI to restart
* Convert files to utf-8 for indexing (#7814)guillep2k2019-08-152-0/+343
* Convert files to utf-8 for indexing * Move utf8 functions to modules/base * Bump repoIndexerLatestVersion to 3 * Add tests for base/encoding.go * Changes to pass gosimple * Move UTF8 funcs into new modules/charset package
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-29 03:59:36 +0000