From 148b40840657260d103eda1897d61f2d949656de Mon Sep 17 00:00:00 2001 From: James Moger Date: Thu, 6 Mar 2014 13:39:11 -0500 Subject: [PATCH] Document ticket replication, mirroring, and advanced administration --- build.xml | 4 +- src/site/tickets_replication.mkd | 135 +++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+), 1 deletion(-) create mode 100644 src/site/tickets_replication.mkd diff --git a/build.xml b/build.xml index 71e7430b..beefc54b 100644 --- a/build.xml +++ b/build.xml @@ -570,7 +570,8 @@ - + + @@ -909,6 +910,7 @@ + diff --git a/src/site/tickets_replication.mkd b/src/site/tickets_replication.mkd new file mode 100644 index 00000000..542fd5fc --- /dev/null +++ b/src/site/tickets_replication.mkd @@ -0,0 +1,135 @@ +## Ticket Replication & Advanced Administration + +*SINCE 1.4.0* + +**Ticket Replication** +Gitblit does *not* provide a generic/universal replication mechanism that works across all persistence backends. + +**Advanced Administration** +Gitblit does *not* provide a generic/universal for advanced administration (i.e. manually tweaking ticket data) however each service does have a strategy for that case. + +### FileTicketService + +#### Ticket Replication +Replication is not supported. + +#### Advanced Administration +Use your favorite text editor to **carefully** manipulate a ticket's journal file. I recommend using a JSON validation service to ensure your changes are valid JSON. + +After you've done this, you will need to reset Gitblit's internal ticket cache and you may need to reindex the tickets, depending on your changes. + +### BranchTicketService + +#### Ticket Replication +Gitblit supports ticket replication for a couple of scenarios with the *BranchTicketService*. This requires that the Gitblit instance receiving the ticket data be configured for the *BranchTicketService*. Likewise, the source of the ticket data must be a repository that has ticket data persisted using the *BranchTicketService*. + +##### Manually Pushing refs/gitblit/tickets + +Let's say you wanted to create a perfect clone of the Gitblit repository hosted at https://dev.gitblit.com in your own Gitblit instance. We'll use this repository as an example because it is configured for the *BranchTicketService*. + +**Assumptions** + +1. We are pushing to our local Gitblit with the admin account, or some other privileged account +2. Our local Gitblit is configured for create-on-push +3. Our local Gitblit is configured for the *BranchTicketService* + +**Procedure** + +1. First we'll clone a mirror of the source repository:
git clone --mirror https://dev.gitblit.com/r/gitblit.git
+2. Then we'll add a remote for our local Gitblit instance:
cd gitblit.git
git remote add local https://localhost:8443/gitblit.git
+3. Then we'll push *everything* to our local Gitblit:
git push --mirror local
+ +If your push was successful you should have a new repository with the entire official Gitblit tickets data. + +##### Mirroring refs/gitblit/tickets + +Gitblit 1.4.0 introduces a mirroring service. This is not the same as the federation feature - although there are similarities. + +If you setup a mirror of another Gitblit repository which uses the *BranchTicketService* **AND** your Gitblit instance is configured for *BranchTicketService*, then your Gitblit will automatically fetch and reindex all tickets without intervention or further configuration. + +**Things to note about mirrors...** + +1. You must set *git.enableMirroring=true* and optionally change *git.mirrorPeriod* +2. Mirrors are read-only. You can not push to a mirror. You can not manipulate a mirror's ticket data. +3. Mirrors are a Git feature - not a Gitblit invention. To create one you must currently use Git within your *git.repositoriesFolder*, you must reset your cache, and you must trigger a ticket reindex.
git clone --mirror <url>
curl --insecure --user admin:admin "https://localhost:8443/rpc?req=clear_repository_cache"
curl --insecure --user admin:admin "https://localhost:8443/rpc?req=reindex_tickets&name=<repo>"
+4. After you have indexed the repository, Gitblit will take over and incrementally update your tickets data on each fetch. + +#### Advanced Administration +Repository owners or Gitblit administrators have the option of manually editing ticket data. To do this you must fetch and checkout the `refs/gitblit/tickets` ref. This orphan branch is where ticket data is stored. You may then use a text editor to **carefully** manipulate journals and push your changes back upstream. I recommend using a JSON validation tool to ensure your changes are valid JSON. + + git fetch origin refs/gitblit/tickets + git checkout -B tix FETCH_HEAD + ...fix data... + git add . + git commit + git push origin HEAD:refs/gitblit/tickets + +Gitblit will identify the incoming `refs/gitblit/tickets` ref update and will incrementally index the changed tickets OR, if the update is non-fast-forward, all tickets on that branch will be reindexed. + +### RedisTicketService + +#### Ticket Replication +Redis is capable of sophisticated replication and clustering. I have not configured Redis replication myself. If this topic interests you please document your procedure and open a pull request to improve this section for others who may also be interested in Redis replication. + +#### Advanced Administration +You can directly manipulate the journals in Redis. The most convenient way do manipulate data is using the simple, but very competent, [RedisDesktopManager](http://redisdesktop.com). It even provides JSON pretty printing which faciliates editing. + +After you've done this, you will need to reset Gitblit's internal ticket cache and you may need to reindex the tickets, depending on your changes. + +The schema of the Redis backend looks like this *repository:object:id*. + + redis 127.0.0.1:6379> keys * + 1) "~james/mytickets.git:ticket:8" + 2) "~james/mytickets.git:journal:8" + 3) "~james/mytickets.git:ticket:4" + 4) "~james/mytickets.git:counter" + 5) "~james/mytickets.git:journal:2" + 6) "~james/mytickets.git:journal:4" + 7) "~james/mytickets.git:journal:7" + 8) "~james/mytickets.git:ticket:3" + 9) "~james/mytickets.git:ticket:6" + 10) "~james/mytickets.git:journal:1" + 11) "~james/mytickets.git:ticket:2" + 12) "~james/mytickets.git:journal:6" + 13) "~james/mytickets.git:ticket:7" + 14) "~james/mytickets.git:ticket:1" + 15) "~james/mytickets.git:journal:3" + +**Some notes about the Redis backend** +The *ticket* object keys are provided as a convenience for integration with other systems. Gitblit does not read those keys, but it does update them. + +The *journal* object keys are the important ones. Gitblit maintains ticket change journals. The *journal* object keys are Redis LISTs where each list entry is a JSON change document. + +The other important object key is the *counter* which is used to assign ticket ids. + +### Resetting the Tickets Cache and Reindexing Tickets + +Reindexing can be memory exhaustive. It obviously depends on the number of tickets you have. Normally, you won't need to manually reindex but if you do, offline reindexing is recommended. + +#### Offline Reindexing + +##### Gitblit GO + +Gitblit GO ships with a script that executes the *com.gitblit.ReindexTickets* tool included in the Gitblit jar file. This tool will reindex *all* tickets in *all* repositories **AND** must be run when Gitblit is offline. + + reindex-tickets + +##### Gitblit WAR/Express + +Gitblit WAR/Express does not ship with anything other than the WAR, but you can still reindex tickets offline with a little extra effort. + +*Windows* + + java -cp "C:/path/to/WEB-INF/lib/*" com.gitblit.ReindexTickets --baseFolder + +*Linux/Unix/Mac OSX* + + java -cp /path/to/WEB-INF/lib/* com.gitblit.ReindexTickets --baseFolder + +#### Live Reindexing + +You can trigger a live reindex of tickets for any backend using Gitblit's RPC interface and curl or your browser. This will also reset Gitblit's internal ticket cache. Use of this RPC requires *web.enableRpcServlet=true* and *web.enableRpcManagement=true* along with administrator credentials. + + curl --insecure --user admin:admin "https://localhost:8443/rpc?req=reindex_tickets" + curl --insecure --user admin:admin "https://localhost:8443/rpc?req=reindex_tickets&name=gitblit.git" + -- 2.39.5