Don't manually replace whitespace during render (#10291) (#10315)
* Don't manually replace whitespace during render
For historical reasons Gitea manually alters the urlPrefix and replaces
a whitespace with a +. This Works for URLs, but we're also passing
urlPrefix to git calls and adding the + is breaking the tree path.
Goldmark will automatically convert a white space to the proper %20, so
we should leave the string as is which lets us pass it to git unmodified
and then let Goldmark fix it.
Also fixed separate bug in URLJoin I noticed while testing where it will
silently discard sections of a path that have # in them (possibly
others). We should just escape it first.
Fixes 10156
* Escape elems as well
* Revert "Escape elems as well"
This reverts commit 8bf49596fe.
* restart ci
* remove changes to URLJoin
* restart ci
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: techknowlogick <matti@mdranta.net>
Change markdown rendering from blackfriday to goldmark (#9533)
* Move to goldmark
Markdown rendering moved from blackfriday to the goldmark.
Multiple subtle changes required to the goldmark extensions to keep
current rendering and defaults.
Can go further with goldmark linkify and have this work within markdown
rendering making the link processor unnecessary.
Need to think about how to go about allowing extensions - at present it
seems that these would be hard to do without recompilation.
* linter fixes
Co-authored-by: Lauris BH <lauris@nix.lv>
* Prefix all user-generated IDs in markup
* Add user-content- to IDs in unit-tests
* fixup markdown_test.go
* update the hrefs for the wiki test
* Add blackfriday extension regex
Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Support custom sanitization policy
Allowing the gitea administrator to configure sanitization policy allows
them to couple external renders and custom templates to support more
markup. In particular, the `pandoc` renderer allows generating KaTeX
annotations, wrapping them in `<span>` elements with class `math` and
either `inline` or `display` (depending on whether or not inline or
block mode was requested).
This iteration gives the administrator whitelisting powers; carefully
crafted regexes will thus let through only the desired attributes
necessary to support their custom markup.
Resolves: #9054
Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* Document new sanitization configuration
- Adds basic documentation to app.ini.sample,
- Adds an example to the Configuration Cheat Sheet, and
- Adds extended information to External Renderers section.
Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* Drop extraneous length check in newMarkupSanitizer(...)
Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* Fix plural ELEMENT and ALLOW_ATTR in docs
These were left over from their initial names. Make them singular to
conform with the current expectations.
Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* Add support for local vs. remote xrefs
* Add doc for references
* Docs: fix cases not currently supported
* One more doc fix
* Doc: mentions for teams and orgs
* Change !num ref concept, no change in functionality
* Fix test
* Improve table of issue reference types
* Fix paragraph mark
Convert EOL to UNIX-style to render MD properly (#8925)
* Convert EOL to UNIX-style to render MD properly
* Update modules/markup/markdown/markdown.go
Co-Authored-By: zeripath <art27@cantab.net>
* Fix lint optimization
* Check for empty content before conversion
* Update modules/util/util.go
Co-Authored-By: zeripath <art27@cantab.net>
* Improved checks and tests
* Add paragraph render test
* Improve speed even more, improve tests
* Small improvement by @gary-kim
* Fix test for DOS
* More improvements
* Restart CI
Make link last commit massages in repository home page and commit tables (#8006)
* Make link last commit massages in repository home page and commit tables
* Use RenderCommitMessageLink instead surround with a
* deleted __debug_bin file
* Exclude email to link from latest commit title
* Exclude email processor from commit table
Co-Authored-By: mrsdizzie <info@mrsdizzie.com>
* Add class parameter to a html element creator functions.
Make links underline dashed that are not commit
* fix tests
* Show dashed underline when also not hovered
* feat: highlight issue references with :
e.g. #1287: my commit msg
e.g. ABC-1234: my commit msg
* ref: update model regex to consistent with issueNumericPattern
* test: check highlight issue with : in commits messages
* detect csv delimiter in csv rendering
fixes #7868
* make linter happy
* fix failing testcase & use ints where possible
* expose markup type to template
previously all markup had the .markdown class, which is incorrect,
as it applies markdown CSS & JS logic to CSV rendering
* fix build (missing `make css`)
* ignore quoted csv content for delimiter scoring
also fix html generation
Check commit message hashes before making links (#7713)
* Check commit message hashes before making links
Previously, when formatting commit messages, anything
that looked like SHA1 hashes was turned into a link
using regex. This meant that certain phrases or numbers
such as `777777` or `deadbeef` could be recognized as a commit
even if the repository has no commit with those hashes.
This change will make it so that anything that looks
like a SHA1 hash using regex will then also be checked
to ensure that there is a commit in the repository
with that hash before making a link.
Signed-off-by: Gary Kim <gary@garykim.dev>
* Use gogit to check if commit exists
This commit modifies the commit hash check
in the render for commit messages to use
gogit for better performance.
Signed-off-by: Gary Kim <gary@garykim.dev>
* Make code cleaner
Signed-off-by: Gary Kim <gary@garykim.dev>
* Use rev-parse to check if commit exists
Signed-off-by: Gary Kim <gary@garykim.dev>
* Add and modify tests for checking hashes in html link rendering
Signed-off-by: Gary Kim <gary@garykim.dev>
* Return error in sha1CurrentPatternProcessor
Co-Authored-By: mrsdizzie <info@mrsdizzie.com>
* Import Gitea log module
Signed-off-by: Gary Kim <gary@garykim.dev>
* Revert "Return error in sha1CurrentPatternProcessor"
This reverts commit 28f561cac4.
Signed-off-by: Gary Kim <gary@garykim.dev>
* Add debug logging to sha1CurrentPatternProcessor
This will log errors by the git command run in
sha1CurrentPatternProcessor if the error is one
that was unexpected.
Signed-off-by: Gary Kim <gary@garykim.dev>
Since #6273 was merged, we now have access to proper context metas
always. Update SHA generated links to use these instead of urlPrefix.
Update tests as well.
Fixes #4536.
* Improve issue autolinks
Update autolinks to match what github does here:
Issue in same repo: #1
Issue in different repo: org/repo#1
Fixes #6264
* Use setting.AppURL when parsing URL
Using setting.AppURL here is a more reliable way of parsing the current
URL and what other functions in this file seem to use.
* Make ComposeMetas always return a valid context
* Add per repository markdown renderers for better context
* Update for use of context metas
Now that we include the user and repo name inside context metas, update
various code and tests for this new logic
Use stricter boundaries for auto-link detection (#6522)
* Use stricter boundaries for auto-link detection
Currently autolinks use \W for boundary detection which creates many
situations of inserting links into places they don't belong (paths,
URLs, UUIDs, etc...)
This fixes that by replacing \W and only allowing these matches to touch
an open paren or bracket (matching what seems to be Github behavior) in
addition to whitespace and start of line. Similar for ending boundary as
well.
Fixes #6149
(and probably others)
* Update test
Replace incorrect test with a value that is a valid username, based on:
"Username should contain only alphanumeric, dash ('-'), underscore ('_')
and dot ('.') characters."
* Also allow for period at the end
Matching Github behavior
* Fix email regex to work properly with specificed boundaries
Create a specific capture group for email address and then use
FindStringSubmatchIndex to allow for non-matching patterns as
boundaries.
* Add Tests
Add tests for new behavior -- including tests for email addresses which
were absent before.
This improves the SHA1 link detection to not pick up extraneous
non-whitespace characters at the end of the URL. The '.' is a special
case handled in code itself because of missing regexp lookahead
support.
Regex test cases: https://regex101.com/r/xUMlqh/3
* Panic don't fatal on create new logger
Fixes #5854
Signed-off-by: Andrew Thornton <art27@cantab.net>
* partial broken
* Update the logging infrastrcture
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Reset the skip levels for Fatal and Error
Signed-off-by: Andrew Thornton <art27@cantab.net>
* broken ncsa
* More log.Error fixes
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove nal
* set log-levels to lowercase
* Make console_test test all levels
* switch to lowercased levels
* OK now working
* Fix vetting issues
* Fix lint
* Fix tests
* change default logging to match current gitea
* Improve log testing
Signed-off-by: Andrew Thornton <art27@cantab.net>
* reset error skip levels to 0
* Update documentation and access logger configuration
* Redirect the router log back to gitea if redirect macaron log but also allow setting the log level - i.e. TRACE
* Fix broken level caching
* Refactor the router log
* Add Router logger
* Add colorizing options
* Adjust router colors
* Only create logger if they will be used
* update app.ini.sample
* rename Attribute ColorAttribute
* Change from white to green for function
* Set fatal/error levels
* Restore initial trace logger
* Fix Trace arguments in modules/auth/auth.go
* Properly handle XORMLogger
* Improve admin/config page
* fix fmt
* Add auto-compression of old logs
* Update error log levels
* Remove the unnecessary skip argument from Error, Fatal and Critical
* Add stacktrace support
* Fix tests
* Remove x/sync from vendors?
* Add stderr option to console logger
* Use filepath.ToSlash to protect against Windows in tests
* Remove prefixed underscores from names in colors.go
* Remove not implemented database logger
This was removed from Gogs on 4 Mar 2016 but left in the configuration
since then.
* Ensure that log paths are relative to ROOT_PATH
* use path.Join
* rename jsonConfig to logConfig
* Rename "config" to "jsonConfig" to make it clearer
* Requested changes
* Requested changes: XormLogger
* Try to color the windows terminal
If successful default to colorizing the console logs
* fixup
* Colorize initially too
* update vendor
* Colorize logs on default and remove if this is not a colorizing logger
* Fix documentation
* fix test
* Use go-isatty to detect if on windows we are on msys or cygwin
* Fix spelling mistake
* Add missing vendors
* More changes
* Rationalise the ANSI writer protection
* Adjust colors on advice from @0x5c
* Make Flags a comma separated list
* Move to use the windows constant for ENABLE_VIRTUAL_TERMINAL_PROCESSING
* Ensure matching is done on the non-colored message - to simpify EXPRESSION
* Migrate to go modules
* make vendor
* Update mvdan.cc/xurls
* make vendor
* Update code.gitea.io/git
* make fmt-check
* Update github.com/go-sql-driver/mysql
* make vendor
* Markdown: enable some more extensions
Improve Markdown-rendering by enabling some extensions:
- enable definitions lists
- enable footnotes
- enable header-ids and automatically generate header-ids (for linking to README-sections or creating table-of-contents for larger READMEs)
* Markdown: update and exted tests
Update and add tests for additionally enabled Markdown-extensions.
* Replace linkRegex with xurls library
Rather than maintaining a complicated regex to match URLs for
autolinking, gitea can use this existing go library that takes care of
the matching with very little code change to gitea itself. After
spending a while trying to find the perfect regex for all cases this library
still works better as it is more flexible than a single regex ever will be.
This will also fix the following issues: #5844#3095#3381
This passes all our current tests and I've added new ones mentioned in
those issues as well.
* Use xurls.StrictMatchingScheme instead of xurls.Strict
This is much faster and we only care about https? links to preserve
existing behavior.
The visitLinksForShortLinks feature would look inside of an <a> tag and
run shortLinkProcessorFull on any text, which attempts to create links
out of potential 'short links' like [[test]] [[link|example]] etc...
This makes no sense because you can't have nested links within an <a>
tag. Specifically, the html5 standard says <a> tags can't include
interactive content if they contain the href attribute:
http://w3c.github.io/html/single-page.html#the-a-element
And also defines an <a> element with a href attribute as interactive:
http://w3c.github.io/html/single-page.html#interactive-content
Therefore you can't really put a link inside of another link. In
practice none of this works anyways since browsers won't render it, it
would probably be broken if they tried, and it is causing a bug
(#4946). No current tests rely on this behavior either.
This removes the feature and also explicitly excludes the
current visitNodeForShortLinks from looking in <a> tags.
Modify the current linkRegex to require http|https which appears to be
the intended behavior based on the comments. Right now, it also matches
anything starting with www as well. Also add testing for linkRegex
This PR makes it possible for the markdown renderer to render images and media straight from the LFS.
Fix #5746
Signed-off-by: Andrew Thornton [art27@cantab.net](mailto:art27@cantab.net)
Recover panic in orgmode.Render if bad orgfile (#4982) (#5903)
This PR protects against the panic referred to in chaseadmsio/goorgeous#82
by recovering from the panic and just returning the raw bytes if
there is an error.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Pass link prefixes to external markup parsers (#5201)
* Pass environment variables for URL prefixes to external markup parser
Signed-off-by: Nicolas Lenz <nicolas@eisfunke.com>
* Document external markup link prefix environment variables
Signed-off-by: Nicolas Lenz <nicolas@eisfunke.com>
* Run format on link prefix changes
Signed-off-by: Nicolas Lenz <nicolas@eisfunke.com>
zeripath
6543
mrsdizzie
John Olheiser
Moritz
Alexander Scheel
guillep2k
Lauris BH
jaqra
Rinat
Tamal Saha
Norwin
Gary Kim
Christian Muehlhaeuser
Lunny Xiao
silverwind
kolaente
Mura Li
Roland Koebler
mrsdizzie
Khaled Hamed
L.E.R
Nicolas Lenz
techknowlogick