You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

pull-db-tests.yml 6.6KB

Refactor and enhance issue indexer to support both searching, filtering and paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
10 months ago
Refactor and enhance issue indexer to support both searching, filtering and paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
10 months ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225
  1. name: db-tests
  2. on:
  3. pull_request:
  4. concurrency:
  5. group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
  6. cancel-in-progress: true
  7. jobs:
  8. files-changed:
  9. uses: ./.github/workflows/files-changed.yml
  10. test-pgsql:
  11. if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
  12. needs: files-changed
  13. runs-on: ubuntu-latest
  14. services:
  15. pgsql:
  16. image: postgres:12
  17. env:
  18. POSTGRES_DB: test
  19. POSTGRES_PASSWORD: postgres
  20. ports:
  21. - "5432:5432"
  22. ldap:
  23. image: gitea/test-openldap:latest
  24. ports:
  25. - "389:389"
  26. - "636:636"
  27. minio:
  28. # as github actions doesn't support "entrypoint", we need to use a non-official image
  29. # that has a custom entrypoint set to "minio server /data"
  30. image: bitnami/minio:2023.8.31
  31. env:
  32. MINIO_ROOT_USER: 123456
  33. MINIO_ROOT_PASSWORD: 12345678
  34. ports:
  35. - "9000:9000"
  36. steps:
  37. - uses: actions/checkout@v4
  38. - uses: actions/setup-go@v5
  39. with:
  40. go-version-file: go.mod
  41. check-latest: true
  42. - name: Add hosts to /etc/hosts
  43. run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 pgsql ldap minio" | sudo tee -a /etc/hosts'
  44. - run: make deps-backend
  45. - run: make backend
  46. env:
  47. TAGS: bindata
  48. - name: run migration tests
  49. run: make test-pgsql-migration
  50. - name: run tests
  51. run: make test-pgsql
  52. timeout-minutes: 50
  53. env:
  54. TAGS: bindata gogit
  55. RACE_ENABLED: true
  56. TEST_TAGS: gogit
  57. TEST_LDAP: 1
  58. USE_REPO_TEST_DIR: 1
  59. test-sqlite:
  60. if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
  61. needs: files-changed
  62. runs-on: ubuntu-latest
  63. steps:
  64. - uses: actions/checkout@v4
  65. - uses: actions/setup-go@v5
  66. with:
  67. go-version-file: go.mod
  68. check-latest: true
  69. - run: make deps-backend
  70. - run: make backend
  71. env:
  72. TAGS: bindata gogit sqlite sqlite_unlock_notify
  73. - name: run migration tests
  74. run: make test-sqlite-migration
  75. - name: run tests
  76. run: make test-sqlite
  77. timeout-minutes: 50
  78. env:
  79. TAGS: bindata gogit sqlite sqlite_unlock_notify
  80. RACE_ENABLED: true
  81. TEST_TAGS: gogit sqlite sqlite_unlock_notify
  82. USE_REPO_TEST_DIR: 1
  83. test-unit:
  84. if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
  85. needs: files-changed
  86. runs-on: ubuntu-latest
  87. services:
  88. elasticsearch:
  89. image: elasticsearch:7.5.0
  90. env:
  91. discovery.type: single-node
  92. ports:
  93. - "9200:9200"
  94. meilisearch:
  95. image: getmeili/meilisearch:v1.2.0
  96. env:
  97. MEILI_ENV: development # disable auth
  98. ports:
  99. - "7700:7700"
  100. redis:
  101. image: redis
  102. options: >- # wait until redis has started
  103. --health-cmd "redis-cli ping"
  104. --health-interval 5s
  105. --health-timeout 3s
  106. --health-retries 10
  107. ports:
  108. - 6379:6379
  109. minio:
  110. image: bitnami/minio:2021.3.17
  111. env:
  112. MINIO_ACCESS_KEY: 123456
  113. MINIO_SECRET_KEY: 12345678
  114. ports:
  115. - "9000:9000"
  116. steps:
  117. - uses: actions/checkout@v4
  118. - uses: actions/setup-go@v5
  119. with:
  120. go-version-file: go.mod
  121. check-latest: true
  122. - name: Add hosts to /etc/hosts
  123. run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 mysql elasticsearch meilisearch smtpimap" | sudo tee -a /etc/hosts'
  124. - run: make deps-backend
  125. - run: make backend
  126. env:
  127. TAGS: bindata
  128. - name: unit-tests
  129. run: make unit-test-coverage test-check
  130. env:
  131. TAGS: bindata
  132. RACE_ENABLED: true
  133. GITHUB_READ_TOKEN: ${{ secrets.GITHUB_READ_TOKEN }}
  134. - name: unit-tests-gogit
  135. run: make unit-test-coverage test-check
  136. env:
  137. TAGS: bindata gogit
  138. RACE_ENABLED: true
  139. GITHUB_READ_TOKEN: ${{ secrets.GITHUB_READ_TOKEN }}
  140. test-mysql:
  141. if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
  142. needs: files-changed
  143. runs-on: ubuntu-latest
  144. services:
  145. mysql:
  146. image: mysql:8.0
  147. env:
  148. MYSQL_ALLOW_EMPTY_PASSWORD: true
  149. MYSQL_DATABASE: testgitea
  150. ports:
  151. - "3306:3306"
  152. elasticsearch:
  153. image: elasticsearch:7.5.0
  154. env:
  155. discovery.type: single-node
  156. ports:
  157. - "9200:9200"
  158. smtpimap:
  159. image: tabascoterrier/docker-imap-devel:latest
  160. ports:
  161. - "25:25"
  162. - "143:143"
  163. - "587:587"
  164. - "993:993"
  165. steps:
  166. - uses: actions/checkout@v4
  167. - uses: actions/setup-go@v5
  168. with:
  169. go-version-file: go.mod
  170. check-latest: true
  171. - name: Add hosts to /etc/hosts
  172. run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 mysql elasticsearch smtpimap" | sudo tee -a /etc/hosts'
  173. - run: make deps-backend
  174. - run: make backend
  175. env:
  176. TAGS: bindata
  177. - name: run migration tests
  178. run: make test-mysql-migration
  179. - name: run tests
  180. run: make integration-test-coverage
  181. env:
  182. TAGS: bindata
  183. RACE_ENABLED: true
  184. USE_REPO_TEST_DIR: 1
  185. TEST_INDEXER_CODE_ES_URL: "http://elastic:changeme@elasticsearch:9200"
  186. test-mssql:
  187. if: needs.files-changed.outputs.backend == 'true' || needs.files-changed.outputs.actions == 'true'
  188. needs: files-changed
  189. runs-on: ubuntu-latest
  190. services:
  191. mssql:
  192. image: mcr.microsoft.com/mssql/server:2017-latest
  193. env:
  194. ACCEPT_EULA: Y
  195. MSSQL_PID: Standard
  196. SA_PASSWORD: MwantsaSecurePassword1
  197. ports:
  198. - "1433:1433"
  199. steps:
  200. - uses: actions/checkout@v4
  201. - uses: actions/setup-go@v5
  202. with:
  203. go-version-file: go.mod
  204. check-latest: true
  205. - name: Add hosts to /etc/hosts
  206. run: '[ -e "/.dockerenv" ] || [ -e "/run/.containerenv" ] || echo "127.0.0.1 mssql" | sudo tee -a /etc/hosts'
  207. - run: make deps-backend
  208. - run: make backend
  209. env:
  210. TAGS: bindata
  211. - run: make test-mssql-migration
  212. - name: run tests
  213. run: make test-mssql
  214. timeout-minutes: 50
  215. env:
  216. TAGS: bindata
  217. USE_REPO_TEST_DIR: 1