Increase default LFS auth timeout from 20m to 24h (#24628)
According to the discussion with DanielGibson, the default "20m" seems
too short. It would make LFS fail if the file is large / network is
slow.
I think relaxing this timeout doesn't have side affect. So change the
default value to 24h, IMO that should be long enough.
## :warning: BREAKING
If admins want the previous timeout, they should set the setting
`[server].LFS_HTTP_AUTH_EXPIRY`.
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
Fix the permission of team's `Actions` unit issue (#24536)
close #24449
The unit of `Actions` should be contorlled not only by
`repository.DISABLED_REPO_UNITS` but also by `actions.ENABLED`
in the `app.ini`.
Previously, the permission of the team's `Actions` unit was not
controlled by `actions.Enabled`. So, even if the user sets
`actions.Enabled` to false, he can still select the permission of the
`Actions` unit for the team.
This PR makes the permissions of the team's `Actions` unit also
controlled by `actions.Enabled`. Just append`TypeActions` into
`DisabledRepoUnits` slice when initializing if `actions.Enabled` is
false.
### Changes:
If `Actions` is set disbaled in `app.ini`, like below:
```yaml
[actions]
ENABLED = false
```
1. If user try to create/edit a team, will prompt user that `Actions` is disabled.
2. `actions` is not displayed in the sidebar on the team details page
Co-authored-by: @awkwardbunny
This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.
---------
Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
## Needs and benefits
[Livebook](https://livebook.dev/) notebooks are used for code
documentation and for deep dives and note-taking in the elixir
ecosystem. Rendering these in these as Markdown on frogejo has many
benefits, since livemd is a subset of markdown. Some of the benefits
are:
- New users of elixir and livebook are scared by unformated .livemd
files, but are shown what they expect
- Sharing a notebook is as easy as sharing a link, no need to install
the software in order to see the results.
[goldmark-meraid ](https://github.com/abhinav/goldmark-mermaid) is a
mermaid-js parser already included in gitea. This makes the .livemd
rendering integration feature complete. With this PR class diagrams, ER
Diagrams, flow charts and much more will be rendered perfectly.
With the additional functionality gitea will be an ideal tool for
sharing resources with fellow software engineers working in the elixir
ecosystem. Allowing the git forge to be used without needing to install
any software.
## Feature Description
This issue requests the .livemd extension to be added as a Markdown
language extension.
- `.livemd` is the extension of Livebook which is an Elixir version of
Jupyter Notebook.
- `.livemd` is` a subset of Markdown.
This would require the .livemd to be recognized as a markdown file. The
Goldmark the markdown parser should handle the parsing and rendering
automatically.
Here is the corresponding commit for GitHub linguist:
https://github.com/github/linguist/pull/5672
Here is a sample page of a livemd file:
https://github.com/github/linguist/blob/master/samples/Markdown/livebook.livemd
## Screenshots
The first screenshot shows how github shows the sample .livemd in the
browser.
The second screenshot shows how mermaid js, renders my development
notebook and its corresponding ER Diagram. The source code can be found
here:
79615f7428/termiNotes.livemd
## Testing
I just changed the file extension from `.livemd`to `.md`and the document
already renders perfectly on codeberg. Check you can it out
[here](https://codeberg.org/lgh/Termi/src/branch/livemd2md/termiNotes.md)
---------
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Refactor `setting.Other` and remove unused `SHOW_FOOTER_BRANDING` (#24270)
The `SHOW_FOOTER_BRANDING` came from year 2015, and it seems nobody ever
uses it. It only shows an GitHub icon which seems unrelated to Gitea, it
doesn't do what document says. So, remove it.
## :warning: Breaking
Users can now remove the key `[other].SHOW_FOOTER_BRANDING` from their
app.ini.
Close #24062
At the beginning, I just wanted to fix the warning mentioned by #24062
But, the cookie code really doesn't look good to me, so clean up them.
Complete the TODO on `SetCookie`:
> TODO: Copied from gitea.com/macaron/macaron and should be improved
after macaron removed.
The default access log format has been unnecessarily escaped, leading to
spurious backslashes appearing in log lines.
Additionally, the `RemoteAddr` field includes the port, which breaks
most log parsers attempting to process it. I've added a call to
`net.SplitHostPort()` attempting to isolate the address alone, with a
fallback to the original address if it errs.
Signed-off-by: Gary Moon <gary@garymoon.net>
Use a general approach to access custom/static/builtin assets (#24022)
The idea is to use a Layered Asset File-system (modules/assetfs/layered.go)
For example: when there are 2 layers: "custom", "builtin", when access
to asset "my/page.tmpl", the Layered Asset File-system will first try to
use "custom" assets, if not found, then use "builtin" assets.
This approach will hugely simplify a lot of code, make them testable.
Other changes:
* Simplify the AssetsHandlerFunc code
* Simplify the `gitea embedded` sub-command code
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Prefer native parser for SSH public key parsing (#23798)
Without this patch, the setting SSH.StartBuiltinServer decides whether
the native (Go) implementation is used rather than calling 'ssh-keygen'.
It's possible for 'using ssh-keygen' and 'using the built-in server' to
be independent.
In fact, the gitea rootless container doesn't ship ssh-keygen and can be
configured to use the host's SSH server - which will cause the public
key parsing mechanism to break.
This commit changes the decision to be based on SSH.KeygenPath instead.
Any existing configurations with a custom KeygenPath set will continue
to function. The new default value of '' selects the native version. The
downside of this approach is that anyone who has relying on plain
'ssh-keygen' to have special properties will now be using the native
version instead.
I assume the exec-variant is only there because /x/crypto/ssh didn't
support ssh-ed25519 until 2016. I don't see any other reason for using
it so it might be an acceptable risk.
Fixes #23363
EDIT: this message was garbled when I tried to get the commit
description back in.. Trying to reconstruct it:
## :warning: BREAKING :warning: Users who don't have SSH.KeygenPath
explicitly set and rely on the ssh-keygen binary need to set
SSH.KeygenPath to 'ssh-keygen' in order to be able to continue using it
for public key parsing.
There was something else but I can't remember at the moment.
EDIT2: It was about `make test` and `make lint`. Can't get them to run.
To reproduce the issue, I installed `golang` in `docker.io/node:16` and
got:
```
...
go: mvdan.cc/xurls/v2@v2.4.0: unknown revision mvdan.cc/xurls/v2.4.0
go: gotest.tools/v3@v3.4.0: unknown revision gotest.tools/v3.4.0
...
go: gotest.tools/v3@v3.0.3: unknown revision gotest.tools/v3.0.3
...
go: error loading module requirements
```
Signed-off-by: Leon M. Busch-George <leon@georgemail.eu>
Add ONLY_SHOW_RELEVANT_REPOS back, fix explore page bug, make code more strict (#23766)
Follow #21962
After I eat my own dogfood, I would say that
ONLY_SHOW_RELEVANT_REPOS=false is necessary for many private/enterprise
instances, because many private repositories do not have
"description/topic", users just want to search by their names.
This PR also adds `PageIsExploreRepositories` check, to make code more
strict, because the `search` template is shared for different purpose.
And during the test, I found a bug that the "Search" button didn't
respect the "relevant" parameter, so this PR fixes the bug by the way
together.
I think this PR needs to be backported.
Editor preview support for external renderers (#23333)
Remove `[repository.editor] PREVIEWABLE_FILE_MODES` setting that seemed
like it was intended to support this but did not work. Instead, whenever
viewing a file shows a preview, also have a Preview tab in the file
editor.
Add new `/markup` web and API endpoints with `comment`, `gfm`,
`markdown` and new `file` mode that uses a file path to determine the
renderer.
Remove `/markdown` web endpoint but keep the API for backwards and
GitHub compatibility.
## ⚠️ BREAKING ⚠️
The `[repository.editor] PREVIEWABLE_FILE_MODES` setting was removed.
This setting served no practical purpose and was not working correctly.
Instead a preview tab is always shown in the file editor when supported.
---------
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Move pidfile creation from setting to web cmd package (#23285)
Creating pid file should not belong to setting package and only web
command needs that. So this PR moves pidfile creation from setting
package to web command package to keep setting package more readable.
I marked this as `break` because the PIDFile path moved. For those who
have used the pid build argument, it has to be changed.
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
This PR adds support for reflogs on all repositories. It does this by
adding a global configuration entry.
Implements #14865
---------
Signed-off-by: Philip Peterson <philip.c.peterson@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Refactor `setting.Database.UseXXX` to methods (#23354)
Replace #23350.
Refactor `setting.Database.UseMySQL` to
`setting.Database.Type.IsMySQL()`.
To avoid mismatching between `Type` and `UseXXX`.
This refactor can fix the bug mentioned in #23350, so it should be
backported.
Make CI use a dummy password hasher for all tests (#22983)
During the recent hash algorithm change it became clear that the choice
of password hash algorithm plays a role in the time taken for CI to run.
Therefore as attempt to improve CI we should consider using a dummy
hashing algorithm instead of a real hashing algorithm.
This PR creates a dummy algorithm which is then set as the default
hashing algorithm during tests that use the fixtures. This hopefully
will cause a reduction in the time it takes for CI to run.
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Refactor the setting to make unit test easier (#22405)
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.
- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.
Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
Provide the ability to set password hash algorithm parameters (#22942)
This PR refactors and improves the password hashing code within gitea
and makes it possible for server administrators to set the password
hashing parameters
In addition it takes the opportunity to adjust the settings for `pbkdf2`
in order to make the hashing a little stronger.
The majority of this work was inspired by PR #14751 and I would like to
thank @boppy for their work on this.
Thanks to @gusted for the suggestion to adjust the `pbkdf2` hashing
parameters.
Close #14751
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Added a new captcha(cloudflare turnstile) and its corresponding
document. Cloudflare turnstile official instructions are here:
https://developers.cloudflare.com/turnstile
Signed-off-by: ByLCY <bylcy@bylcy.dev>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Jason Song <i@wolfogre.com>
Every user can already disable the filter manually, so the explicit
setting is absolutely useless and only complicates the logic.
Previously, there was also unexpected behavior when multiple query
parameters were present.
---------
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Repositories: by default disable all units except code and pulls on forks (#22541)
Most of the time forks are used for contributing code only, so not
having
issues, projects, release and packages is a better default for such
cases.
They can still be enabled in the settings.
A new option `DEFAULT_FORK_REPO_UNITS` is added to configure the default
units on forks.
Also add missing `repo.packages` unit to documentation.
code by: @brechtvl
## :warning: BREAKING :warning:
When forking a repository, the fork will now have issues, projects,
releases, packages and wiki disabled. These can be enabled in the
repository settings afterwards. To change back to the previous default
behavior, configure `DEFAULT_FORK_REPO_UNITS` to be the same value as
`DEFAULT_REPO_UNITS`.
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
closes #13585
fixes #9067
fixes #2386
ref #6226
ref #6219
fixes #745
This PR adds support to process incoming emails to perform actions.
Currently I added handling of replies and unsubscribing from
issues/pulls. In contrast to #13585 the IMAP IDLE command is used
instead of polling which results (in my opinion 😉) in cleaner code.
Procedure:
- When sending an issue/pull reply email, a token is generated which is
present in the Reply-To and References header.
- IMAP IDLE waits until a new email arrives
- The token tells which action should be performed
A possible signature and/or reply gets stripped from the content.
I added a new service to the drone pipeline to test the receiving of
incoming mails. If we keep this in, we may test our outgoing emails too
in future.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
KN4CK3R
rune
wxiaoguang
Lunny Xiao
sillyguodong
Yarden Shoham
contra-bit
silverwind
Gary Moon
Leon Busch-George
techknowlogick
Brecht Van Lommel
Philip Peterson
Jason Song
Yarden Shoham
zeripath
ByLCY
delvh
