Refactor and enhance issue indexer to support both searching, filtering and paging (#26012)
Fix #24662.
Replace #24822 and #25708 (although it has been merged)
## Background
In the past, Gitea supported issue searching with a keyword and
conditions in a less efficient way. It worked by searching for issues
with the keyword and obtaining limited IDs (as it is heavy to get all)
on the indexer (bleve/elasticsearch/meilisearch), and then querying with
conditions on the database to find a subset of the found IDs. This is
why the results could be incomplete.
To solve this issue, we need to store all fields that could be used as
conditions in the indexer and support both keyword and additional
conditions when searching with the indexer.
## Major changes
- Redefine `IndexerData` to include all fields that could be used as
filter conditions.
- Refactor `Search(ctx context.Context, kw string, repoIDs []int64,
limit, start int, state string)` to `Search(ctx context.Context, options
*SearchOptions)`, so it supports more conditions now.
- Change the data type stored in `issueIndexerQueue`. Use
`IndexerMetadata` instead of `IndexerData` in case the data has been
updated while it is in the queue. This also reduces the storage size of
the queue.
- Enhance searching with Bleve/Elasticsearch/Meilisearch, make them
fully support `SearchOptions`. Also, update the data versions.
- Keep most logic of database indexer, but remove
`issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is
the entry point to search issues.
- Start a Meilisearch instance to test it in unit tests.
- Add unit tests with almost full coverage to test
Bleve/Elasticsearch/Meilisearch indexer.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This PR cleans up the docs in a way to make them simpler to ingest by
our [docs repo](https://gitea.com/gitea/gitea-docusaurus).
1. It includes all of the sed invocations our ingestion did, removing
the need to do it at build time.
2. It replaces the shortcode variable replacement method with
`@variable@` style, simply for easier sed invocations when required.
3. It removes unused files and moves the docs up a level as cleanup.
---------
Signed-off-by: jolheiser <john.olheiser@gmail.com>
Use Actions git context instead of dynamically created buildkit one (#25381)
The [docker/build-push-action@v2
action](https://github.com/docker/build-push-action) by default ignores
the checkout created using the actions/checkout@v2 action. When you pass
a git build context to docker build, it wouldn't include the .git
directory.
By passing `context: .` to the build step then it'll use the Actions git
context which includes the git fetch from the earlier step.
`docker-dryrun` runs on almost any PR, which is a huge waste of CI
resources. Run it only when the Dockerfiles change and also add a step
that verifies the rootless file.
---------
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Clean up pyproject.toml and package.json, fix poetry options (#25327)
- Reduce `pyproject.toml` and `package.json` to the minimal required
format, removing unneeded properties. `build-system` is not needed as
per
[this](https://github.com/python-poetry/poetry/issues/8110#issuecomment-1595846841).
- Fix `poetry.toml` options they were wrong previously.
- Add dependencies of poetry files to templates `files-changed`.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
So I found this [linter](https://github.com/Riverside-Healthcare/djlint)
which features a mode for go templates, so I gave it a try and it did
find a number of valid issue, like unbalanced tags etc. It also has a
number of bugs, I had to disable/workaround many issues.
Given that this linter is written in python, this does add a dependency
on `python` >= 3.8 and `poetry` to the development environment to be
able to run this linter locally.
- `e.g.` prefixes on placeholders are removed because the linter had a
false-positive on `placeholder="e.g. cn=Search"` for the `attr=value`
syntax and it's not ideal anyways to write `e.g.` into a placeholder
because a placeholder is meant to hold a sample value.
- In `templates/repo/settings/options.tmpl` I simplified the logic to
not conditionally create opening tags without closing tags because this
stuff confuses the linter (and possibly the reader as well).
The `setup-go` actions did not all have `check-latest` which means they
use some cached version of go that currently still resolves to go1.20.4,
as seen in a number of recent runs that currently fail at govulncheck
because of it:
````
Run actions/setup-go@v4
Setup go version spec >=1.20
Attempting to resolve the latest version from the manifest...
matching >=1.20...
Resolved as '1.20.4'
````
Add the
[check-latest](https://github.com/actions/setup-go#check-latest-version)
option which should guarantee that this cache is skipped.
- Merge the file filters into `files-changed.yml`
- Remove unused yaml anchors like `&backend`
- Merge the `compliance-docs` workflow into `compliance`
- Add actions linting
- Misc cleanups for whitespace and step names
Use file filters action instead of Github's files filter (#24877)
Inspired by
https://github.com/go-gitea/gitea/pull/24530#issuecomment-1558815301
This PR use a file filter action to do different CI jobs according
changed files types. All types are defined in
`.github/file-filters.yml`. Now there are 4 types, `docs`, `backend`,
`frontend` and `build`. Then if a PR only changed docs files, those CI
jobs which passed the conditions will run, and other types are also like
this.
---------
Co-authored-by: silverwind <me@silverwind.io>
minio storage iterator shows different behavior with local fs iterator.
in local fs storage:
``` go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "prefix/xxx.file"
})
```
in minio storage:
```go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "xxx.file"
})
```
I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
- Remove actions name where command is descriptive enough
- Use kebab-case instead of snake-case for step names
- Use shorter job names because to make PR checks more readable
- Remove duplicate `checks-backend`
---------
Co-authored-by: Yarden Shoham <git@yardenshoham.com>
Replaces #24641
Currently, unit tests fail when run locally (unless users have minio
instance running). This PR only requires redis unit tests if in CI.
- Only run redis unit tests when `CI` env variable is set
- Add minio as a service in unit tests actions
With comments happening on closed issues/prs this locks issues that have
been closed for >45days. This allows for comments on recently closed
issues/prs to still happen.
---------
Co-authored-by: silverwind <me@silverwind.io>
similar to #24314, this moves drone cron pipelines to (GitHub) Actions.
As these are mostly compatible with Gitea Actions, when we start to
dogfood, these will already be migrated.
I also removed the discord notify pipeline, as it hasn't been working
for several months.
switch to use Actions from drone for cron (#24314)
Move drone cron pipelines to (GitHub) Actions. As these are mostly
compatible with Gitea Actions, when we start to dogfood, these will
already be migrated.
Update `pull_request_template.md` because:
- It's a kind idea to hide the tips. However, it's easier to include
them in the commit message by mistake when you cannot see them. Check
`git log | grep 'Please check the following:'`. So don't hide it, expose
it and help fix it.
- "for backports" is much clearer than "for bug fixes". I saw someone
post a PR to a release branch because they believed it was the right way
for a bugfix.
- "Allow edits by maintainers", or we have to ask the contributor to
update the branch and they could be confused.
- Remind the contributor that the words could be included in the commit
message, to avoid some words like "Hello", "Sorry". If they really need
them, they can separate them with a line, like:
```markdown
Close #xxxx
Because ... Then ... Finally ...
---
Hello, this is my first time opening a pull request. Sorry for any mistakes.
```
And the merger should be careful, check and delete the extra content
before merging.
---------
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Make the proformas clearer that we need DEBUG logs (#18700)
It is ridiculous how few of our bug reporter are giving us DEBUG level logs.
This has to change and I think the proforma is not making it clear enough that
they have to give us these logs.
This PR changes the issue proformas to tell people to give us these logs.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Create issue template using new format
This issue template has been based off of the current issue creation workflow defined in the ISSUE_TEMPLATE.md file
This commit allows for that issue template (which is a legacy method of creating an issue) to be deleted
* Delete old issue template file
This commit deletes the old issue template file, as it is both legacy and replaced by ISSUE_TEMPLATE/issue.yaml
* Adjust file upload and remove code of conduct checkbox
* Block blank issue creation
* Update feedback
Removed most requirements
Updated prompt to include instruction to state whether using try.gitea.io
Added default value to Gitea version
* Create Feature Request issue form
* Update and rename issue.yaml to bug-report.yaml
* Enable blank issue creation
Co-authored-by: a101211279 <1012112796@qq.com>
* Update config.yaml
* Remove default & placeholder from Gitea version
* Create issue template in .gitea directory
* Add relevant bug report message to feature request
* Adjust name to remove banner
There is a banner that tells you to create a config.yml when you already have a config.yaml
Renaming the file stops this from occurring.
* Add UI bug report form
* Remove unnecessary question
Co-authored-by: a101211279 <1012112796@qq.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Jason Song
John Olheiser
Lunny Xiao
isla w
silverwind
techknowlogick
a1012112796
FuXiaoHei
Kyle D
Eugene
delvh
zeripath
wxiaoguang
qwerty287
Gabriel
Ikko Ashimine