Ticket tracker with patchset contributions
A basic issue tracker styled as a hybrid of GitHub and BitBucket issues.
You may attach commits to an existing ticket or you can push a single
commit to create a *proposal* ticket.
Tickets keep track of patchsets (one or more commits) and allow patchset
rewriting (rebase, amend, squash) by detecing the non-fast-forward
update and assigning a new patchset number to the new commits.
Ticket tracker
--------------
The ticket tracker stores tickets as an append-only journal of changes.
The journals are deserialized and a ticket is built by applying the
journal entries. Tickets are indexed using Apache Lucene and all
queries and searches are executed against this Lucene index.
There is one trade-off to this persistence design: user attributions are
non-relational.
What does that mean? Each journal entry stores the username of the
author. If the username changes in the user service, the journal entry
will not reflect that change because the values are hard-coded.
Here are a few reasons/justifications for this design choice:
1. commit identifications (author, committer, tagger) are non-relational
2. maintains the KISS principle
3. your favorite text editor can still be your administration tool
Persistence Choices
-------------------
**FileTicketService**: stores journals on the filesystem
**BranchTicketService**: stores journals on an orphan branch
**RedisTicketService**: stores journals in a Redis key-value datastore
It should be relatively straight-forward to develop other backends
(MongoDB, etc) as long as the journal design is preserved.
Pushing Commits
---------------
Each push to a ticket is identified as a patchset revision. A patchset
revision may add commits to the patchset (fast-forward) OR a patchset
revision may rewrite history (rebase, squash, rebase+squash, or amend).
Patchset authors should not be afraid to polish, revise, and rewrite
their code before merging into the proposed branch.
Gitblit will create one ref for each patchset. These refs are updated
for fast-forward pushes or created for rewrites. They are formatted as
`refs/tickets/{shard}/{id}/{patchset}`. The *shard* is the last two
digits of the id. If the id < 10, prefix a 0. The *shard* is always
two digits long. The shard's purpose is to ensure Gitblit doesn't
exceed any filesystem directory limits for file creation.
**Creating a Proposal Ticket**
You may create a new change proposal ticket just by pushing a **single
commit** to `refs/for/{branch}` where branch is the proposed integration
branch OR `refs/for/new` or `refs/for/default` which both will use the
default repository branch.
git push origin HEAD:refs/for/new
**Updating a Patchset**
The safe way to update an existing patchset is to push to the patchset
ref.
git push origin HEAD:refs/heads/ticket/{id}
This ensures you do not accidentally create a new patchset in the event
that the patchset was updated after you last pulled.
The not-so-safe way to update an existing patchset is to push using the
magic ref.
git push origin HEAD:refs/for/{id}
This push ref will update an exisitng patchset OR create a new patchset
if the update is non-fast-forward.
**Rebasing, Squashing, Amending**
Gitblit makes rebasing, squashing, and amending patchsets easy.
Normally, pushing a non-fast-forward update would require rewind (RW+)
repository permissions. Gitblit provides a magic ref which will allow
ticket participants to rewrite a ticket patchset as long as the ticket
is open.
git push origin HEAD:refs/for/{id}
Pushing changes to this ref allows the patchset authors to rebase,
squash, or amend the patchset commits without requiring client-side use
of the *--force* flag on push AND without requiring RW+ permission to
the repository. Since each patchset is tracked with a ref it is easy to
recover from accidental non-fast-forward updates.
Features
--------
- Ticket tracker with status changes and responsible assignments
- Patchset revision scoring mechanism
- Update/Rewrite patchset handling
- Close-on-push detection
- Server-side Merge button for simple merges
- Comments with Markdown syntax support
- Rich mail notifications
- Voting
- Mentions
- Watch lists
- Querying
- Searches
- Partial miletones support
- Multiple backend options
10 years ago Ticket tracker with patchset contributions
A basic issue tracker styled as a hybrid of GitHub and BitBucket issues.
You may attach commits to an existing ticket or you can push a single
commit to create a *proposal* ticket.
Tickets keep track of patchsets (one or more commits) and allow patchset
rewriting (rebase, amend, squash) by detecing the non-fast-forward
update and assigning a new patchset number to the new commits.
Ticket tracker
--------------
The ticket tracker stores tickets as an append-only journal of changes.
The journals are deserialized and a ticket is built by applying the
journal entries. Tickets are indexed using Apache Lucene and all
queries and searches are executed against this Lucene index.
There is one trade-off to this persistence design: user attributions are
non-relational.
What does that mean? Each journal entry stores the username of the
author. If the username changes in the user service, the journal entry
will not reflect that change because the values are hard-coded.
Here are a few reasons/justifications for this design choice:
1. commit identifications (author, committer, tagger) are non-relational
2. maintains the KISS principle
3. your favorite text editor can still be your administration tool
Persistence Choices
-------------------
**FileTicketService**: stores journals on the filesystem
**BranchTicketService**: stores journals on an orphan branch
**RedisTicketService**: stores journals in a Redis key-value datastore
It should be relatively straight-forward to develop other backends
(MongoDB, etc) as long as the journal design is preserved.
Pushing Commits
---------------
Each push to a ticket is identified as a patchset revision. A patchset
revision may add commits to the patchset (fast-forward) OR a patchset
revision may rewrite history (rebase, squash, rebase+squash, or amend).
Patchset authors should not be afraid to polish, revise, and rewrite
their code before merging into the proposed branch.
Gitblit will create one ref for each patchset. These refs are updated
for fast-forward pushes or created for rewrites. They are formatted as
`refs/tickets/{shard}/{id}/{patchset}`. The *shard* is the last two
digits of the id. If the id < 10, prefix a 0. The *shard* is always
two digits long. The shard's purpose is to ensure Gitblit doesn't
exceed any filesystem directory limits for file creation.
**Creating a Proposal Ticket**
You may create a new change proposal ticket just by pushing a **single
commit** to `refs/for/{branch}` where branch is the proposed integration
branch OR `refs/for/new` or `refs/for/default` which both will use the
default repository branch.
git push origin HEAD:refs/for/new
**Updating a Patchset**
The safe way to update an existing patchset is to push to the patchset
ref.
git push origin HEAD:refs/heads/ticket/{id}
This ensures you do not accidentally create a new patchset in the event
that the patchset was updated after you last pulled.
The not-so-safe way to update an existing patchset is to push using the
magic ref.
git push origin HEAD:refs/for/{id}
This push ref will update an exisitng patchset OR create a new patchset
if the update is non-fast-forward.
**Rebasing, Squashing, Amending**
Gitblit makes rebasing, squashing, and amending patchsets easy.
Normally, pushing a non-fast-forward update would require rewind (RW+)
repository permissions. Gitblit provides a magic ref which will allow
ticket participants to rewrite a ticket patchset as long as the ticket
is open.
git push origin HEAD:refs/for/{id}
Pushing changes to this ref allows the patchset authors to rebase,
squash, or amend the patchset commits without requiring client-side use
of the *--force* flag on push AND without requiring RW+ permission to
the repository. Since each patchset is tracked with a ref it is easy to
recover from accidental non-fast-forward updates.
Features
--------
- Ticket tracker with status changes and responsible assignments
- Patchset revision scoring mechanism
- Update/Rewrite patchset handling
- Close-on-push detection
- Server-side Merge button for simple merges
- Comments with Markdown syntax support
- Rich mail notifications
- Voting
- Mentions
- Watch lists
- Querying
- Searches
- Partial miletones support
- Multiple backend options
10 years ago |
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531 |
- /*
- * Copyright 2013 gitblit.com.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
- package com.gitblit.tickets;
-
- import java.net.URI;
- import java.util.ArrayList;
- import java.util.Collections;
- import java.util.List;
- import java.util.Set;
-
- import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
-
- import redis.clients.jedis.Client;
- import redis.clients.jedis.Jedis;
- import redis.clients.jedis.JedisPool;
- import redis.clients.jedis.Protocol;
- import redis.clients.jedis.Transaction;
- import redis.clients.jedis.exceptions.JedisException;
-
- import com.gitblit.Keys;
- import com.gitblit.manager.INotificationManager;
- import com.gitblit.manager.IRepositoryManager;
- import com.gitblit.manager.IRuntimeManager;
- import com.gitblit.manager.IUserManager;
- import com.gitblit.models.RepositoryModel;
- import com.gitblit.models.TicketModel;
- import com.gitblit.models.TicketModel.Attachment;
- import com.gitblit.models.TicketModel.Change;
- import com.gitblit.utils.ArrayUtils;
- import com.gitblit.utils.StringUtils;
-
- /**
- * Implementation of a ticket service based on a Redis key-value store. All
- * tickets are persisted in the Redis store so it must be configured for
- * durability otherwise tickets are lost on a flush or restart. Tickets are
- * indexed with Lucene and all queries are executed against the Lucene index.
- *
- * @author James Moger
- *
- */
- public class RedisTicketService extends ITicketService {
-
- private final JedisPool pool;
-
- private enum KeyType {
- journal, ticket, counter
- }
-
- public RedisTicketService(
- IRuntimeManager runtimeManager,
- INotificationManager notificationManager,
- IUserManager userManager,
- IRepositoryManager repositoryManager) {
-
- super(runtimeManager,
- notificationManager,
- userManager,
- repositoryManager);
-
- String redisUrl = settings.getString(Keys.tickets.redis.url, "");
- this.pool = createPool(redisUrl);
- }
-
- @Override
- public RedisTicketService start() {
- return this;
- }
-
- @Override
- protected void resetCachesImpl() {
- }
-
- @Override
- protected void resetCachesImpl(RepositoryModel repository) {
- }
-
- @Override
- protected void close() {
- pool.destroy();
- }
-
- @Override
- public boolean isReady() {
- return pool != null;
- }
-
- /**
- * Constructs a key for use with a key-value data store.
- *
- * @param key
- * @param repository
- * @param id
- * @return a key
- */
- private String key(RepositoryModel repository, KeyType key, String id) {
- StringBuilder sb = new StringBuilder();
- sb.append(repository.name).append(':');
- sb.append(key.name());
- if (!StringUtils.isEmpty(id)) {
- sb.append(':');
- sb.append(id);
- }
- return sb.toString();
- }
-
- /**
- * Constructs a key for use with a key-value data store.
- *
- * @param key
- * @param repository
- * @param id
- * @return a key
- */
- private String key(RepositoryModel repository, KeyType key, long id) {
- return key(repository, key, "" + id);
- }
-
- private boolean isNull(String value) {
- return value == null || "nil".equals(value);
- }
-
- private String getUrl() {
- Jedis jedis = pool.getResource();
- try {
- if (jedis != null) {
- Client client = jedis.getClient();
- return client.getHost() + ":" + client.getPort() + "/" + client.getDB();
- }
- } catch (JedisException e) {
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return null;
- }
-
- /**
- * Ensures that we have a ticket for this ticket id.
- *
- * @param repository
- * @param ticketId
- * @return true if the ticket exists
- */
- @Override
- public boolean hasTicket(RepositoryModel repository, long ticketId) {
- if (ticketId <= 0L) {
- return false;
- }
- Jedis jedis = pool.getResource();
- if (jedis == null) {
- return false;
- }
- try {
- Boolean exists = jedis.exists(key(repository, KeyType.journal, ticketId));
- return exists != null && exists;
- } catch (JedisException e) {
- log.error("failed to check hasTicket from Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return false;
- }
-
- /**
- * Assigns a new ticket id.
- *
- * @param repository
- * @return a new long ticket id
- */
- @Override
- public synchronized long assignNewId(RepositoryModel repository) {
- Jedis jedis = pool.getResource();
- try {
- String key = key(repository, KeyType.counter, null);
- String val = jedis.get(key);
- if (isNull(val)) {
- jedis.set(key, "0");
- }
- long ticketNumber = jedis.incr(key);
- return ticketNumber;
- } catch (JedisException e) {
- log.error("failed to assign new ticket id in Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return 0L;
- }
-
- /**
- * Returns all the tickets in the repository. Querying tickets from the
- * repository requires deserializing all tickets. This is an expensive
- * process and not recommended. Tickets should be indexed by Lucene and
- * queries should be executed against that index.
- *
- * @param repository
- * @param filter
- * optional filter to only return matching results
- * @return a list of tickets
- */
- @Override
- public List<TicketModel> getTickets(RepositoryModel repository, TicketFilter filter) {
- Jedis jedis = pool.getResource();
- List<TicketModel> list = new ArrayList<TicketModel>();
- if (jedis == null) {
- return list;
- }
- try {
- // Deserialize each journal, build the ticket, and optionally filter
- Set<String> keys = jedis.keys(key(repository, KeyType.journal, "*"));
- for (String key : keys) {
- // {repo}:journal:{id}
- String id = key.split(":")[2];
- long ticketId = Long.parseLong(id);
- List<Change> changes = getJournal(jedis, repository, ticketId);
- if (ArrayUtils.isEmpty(changes)) {
- log.warn("Empty journal for {}:{}", repository, ticketId);
- continue;
- }
- TicketModel ticket = TicketModel.buildTicket(changes);
- ticket.project = repository.projectPath;
- ticket.repository = repository.name;
- ticket.number = ticketId;
-
- // add the ticket, conditionally, to the list
- if (filter == null) {
- list.add(ticket);
- } else {
- if (filter.accept(ticket)) {
- list.add(ticket);
- }
- }
- }
-
- // sort the tickets by creation
- Collections.sort(list);
- } catch (JedisException e) {
- log.error("failed to retrieve tickets from Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return list;
- }
-
- /**
- * Retrieves the ticket from the repository by first looking-up the changeId
- * associated with the ticketId.
- *
- * @param repository
- * @param ticketId
- * @return a ticket, if it exists, otherwise null
- */
- @Override
- protected TicketModel getTicketImpl(RepositoryModel repository, long ticketId) {
- Jedis jedis = pool.getResource();
- if (jedis == null) {
- return null;
- }
-
- try {
- List<Change> changes = getJournal(jedis, repository, ticketId);
- if (ArrayUtils.isEmpty(changes)) {
- log.warn("Empty journal for {}:{}", repository, ticketId);
- return null;
- }
- TicketModel ticket = TicketModel.buildTicket(changes);
- ticket.project = repository.projectPath;
- ticket.repository = repository.name;
- ticket.number = ticketId;
- log.debug("rebuilt ticket {} from Redis @ {}", ticketId, getUrl());
- return ticket;
- } catch (JedisException e) {
- log.error("failed to retrieve ticket from Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return null;
- }
-
- /**
- * Returns the journal for the specified ticket.
- *
- * @param repository
- * @param ticketId
- * @return a list of changes
- */
- private List<Change> getJournal(Jedis jedis, RepositoryModel repository, long ticketId) throws JedisException {
- if (ticketId <= 0L) {
- return new ArrayList<Change>();
- }
- List<String> entries = jedis.lrange(key(repository, KeyType.journal, ticketId), 0, -1);
- if (entries.size() > 0) {
- // build a json array from the individual entries
- StringBuilder sb = new StringBuilder();
- sb.append("[");
- for (String entry : entries) {
- sb.append(entry).append(',');
- }
- sb.setLength(sb.length() - 1);
- sb.append(']');
- String journal = sb.toString();
-
- return TicketSerializer.deserializeJournal(journal);
- }
- return new ArrayList<Change>();
- }
-
- @Override
- public boolean supportsAttachments() {
- return false;
- }
-
- /**
- * Retrieves the specified attachment from a ticket.
- *
- * @param repository
- * @param ticketId
- * @param filename
- * @return an attachment, if found, null otherwise
- */
- @Override
- public Attachment getAttachment(RepositoryModel repository, long ticketId, String filename) {
- return null;
- }
-
- /**
- * Deletes a ticket.
- *
- * @param ticket
- * @return true if successful
- */
- @Override
- protected boolean deleteTicketImpl(RepositoryModel repository, TicketModel ticket, String deletedBy) {
- boolean success = false;
- if (ticket == null) {
- throw new RuntimeException("must specify a ticket!");
- }
-
- Jedis jedis = pool.getResource();
- if (jedis == null) {
- return false;
- }
-
- try {
- // atomically remove ticket
- Transaction t = jedis.multi();
- t.del(key(repository, KeyType.ticket, ticket.number));
- t.del(key(repository, KeyType.journal, ticket.number));
- t.exec();
-
- success = true;
- log.debug("deleted ticket {} from Redis @ {}", "" + ticket.number, getUrl());
- } catch (JedisException e) {
- log.error("failed to delete ticket from Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
-
- return success;
- }
-
- /**
- * Commit a ticket change to the repository.
- *
- * @param repository
- * @param ticketId
- * @param change
- * @return true, if the change was committed
- */
- @Override
- protected boolean commitChangeImpl(RepositoryModel repository, long ticketId, Change change) {
- Jedis jedis = pool.getResource();
- if (jedis == null) {
- return false;
- }
- try {
- List<Change> changes = getJournal(jedis, repository, ticketId);
- changes.add(change);
- // build a new effective ticket from the changes
- TicketModel ticket = TicketModel.buildTicket(changes);
-
- String object = TicketSerializer.serialize(ticket);
- String journal = TicketSerializer.serialize(change);
-
- // atomically store ticket
- Transaction t = jedis.multi();
- t.set(key(repository, KeyType.ticket, ticketId), object);
- t.rpush(key(repository, KeyType.journal, ticketId), journal);
- t.exec();
-
- log.debug("updated ticket {} in Redis @ {}", "" + ticketId, getUrl());
- return true;
- } catch (JedisException e) {
- log.error("failed to update ticket cache in Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return false;
- }
-
- /**
- * Deletes all Tickets for the rpeository from the Redis key-value store.
- *
- */
- @Override
- protected boolean deleteAllImpl(RepositoryModel repository) {
- Jedis jedis = pool.getResource();
- if (jedis == null) {
- return false;
- }
-
- boolean success = false;
- try {
- Set<String> keys = jedis.keys(repository.name + ":*");
- if (keys.size() > 0) {
- Transaction t = jedis.multi();
- t.del(keys.toArray(new String[keys.size()]));
- t.exec();
- }
- success = true;
- } catch (JedisException e) {
- log.error("failed to delete all tickets in Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return success;
- }
-
- @Override
- protected boolean renameImpl(RepositoryModel oldRepository, RepositoryModel newRepository) {
- Jedis jedis = pool.getResource();
- if (jedis == null) {
- return false;
- }
-
- boolean success = false;
- try {
- Set<String> oldKeys = jedis.keys(oldRepository.name + ":*");
- Transaction t = jedis.multi();
- for (String oldKey : oldKeys) {
- String newKey = newRepository.name + oldKey.substring(oldKey.indexOf(':'));
- t.rename(oldKey, newKey);
- }
- t.exec();
- success = true;
- } catch (JedisException e) {
- log.error("failed to rename tickets in Redis @ " + getUrl(), e);
- pool.returnBrokenResource(jedis);
- jedis = null;
- } finally {
- if (jedis != null) {
- pool.returnResource(jedis);
- }
- }
- return success;
- }
-
- private JedisPool createPool(String url) {
- JedisPool pool = null;
- if (!StringUtils.isEmpty(url)) {
- try {
- URI uri = URI.create(url);
- if (uri.getScheme() != null && uri.getScheme().equalsIgnoreCase("redis")) {
- int database = Protocol.DEFAULT_DATABASE;
- String password = null;
- if (uri.getUserInfo() != null) {
- password = uri.getUserInfo().split(":", 2)[1];
- }
- if (uri.getPath().indexOf('/') > -1) {
- database = Integer.parseInt(uri.getPath().split("/", 2)[1]);
- }
- pool = new JedisPool(new GenericObjectPoolConfig(), uri.getHost(), uri.getPort(), Protocol.DEFAULT_TIMEOUT, password, database);
- } else {
- pool = new JedisPool(url);
- }
- } catch (JedisException e) {
- log.error("failed to create a Redis pool!", e);
- }
- }
- return pool;
- }
-
- @Override
- public String toString() {
- String url = getUrl();
- return getClass().getSimpleName() + " (" + (url == null ? "DISABLED" : url) + ")";
- }
- }
|