diff options
author | James Moger <james.moger@gitblit.com> | 2013-12-09 17:19:03 -0500 |
---|---|---|
committer | James Moger <james.moger@gitblit.com> | 2014-03-03 21:34:32 -0500 |
commit | 5e3521f8496511db4df45f011ea72f25623ad90f (patch) | |
tree | 98b4f516d59833b5a8c1ccbcd45672e5b9f3add2 /src/main/java/com/gitblit/tickets | |
parent | 94e12c168f5eec300fd23d0de25c7dc93a96c429 (diff) | |
download | gitblit-5e3521f8496511db4df45f011ea72f25623ad90f.tar.gz gitblit-5e3521f8496511db4df45f011ea72f25623ad90f.zip |
Ticket tracker with patchset contributions
A basic issue tracker styled as a hybrid of GitHub and BitBucket issues.
You may attach commits to an existing ticket or you can push a single
commit to create a *proposal* ticket.
Tickets keep track of patchsets (one or more commits) and allow patchset
rewriting (rebase, amend, squash) by detecing the non-fast-forward
update and assigning a new patchset number to the new commits.
Ticket tracker
--------------
The ticket tracker stores tickets as an append-only journal of changes.
The journals are deserialized and a ticket is built by applying the
journal entries. Tickets are indexed using Apache Lucene and all
queries and searches are executed against this Lucene index.
There is one trade-off to this persistence design: user attributions are
non-relational.
What does that mean? Each journal entry stores the username of the
author. If the username changes in the user service, the journal entry
will not reflect that change because the values are hard-coded.
Here are a few reasons/justifications for this design choice:
1. commit identifications (author, committer, tagger) are non-relational
2. maintains the KISS principle
3. your favorite text editor can still be your administration tool
Persistence Choices
-------------------
**FileTicketService**: stores journals on the filesystem
**BranchTicketService**: stores journals on an orphan branch
**RedisTicketService**: stores journals in a Redis key-value datastore
It should be relatively straight-forward to develop other backends
(MongoDB, etc) as long as the journal design is preserved.
Pushing Commits
---------------
Each push to a ticket is identified as a patchset revision. A patchset
revision may add commits to the patchset (fast-forward) OR a patchset
revision may rewrite history (rebase, squash, rebase+squash, or amend).
Patchset authors should not be afraid to polish, revise, and rewrite
their code before merging into the proposed branch.
Gitblit will create one ref for each patchset. These refs are updated
for fast-forward pushes or created for rewrites. They are formatted as
`refs/tickets/{shard}/{id}/{patchset}`. The *shard* is the last two
digits of the id. If the id < 10, prefix a 0. The *shard* is always
two digits long. The shard's purpose is to ensure Gitblit doesn't
exceed any filesystem directory limits for file creation.
**Creating a Proposal Ticket**
You may create a new change proposal ticket just by pushing a **single
commit** to `refs/for/{branch}` where branch is the proposed integration
branch OR `refs/for/new` or `refs/for/default` which both will use the
default repository branch.
git push origin HEAD:refs/for/new
**Updating a Patchset**
The safe way to update an existing patchset is to push to the patchset
ref.
git push origin HEAD:refs/heads/ticket/{id}
This ensures you do not accidentally create a new patchset in the event
that the patchset was updated after you last pulled.
The not-so-safe way to update an existing patchset is to push using the
magic ref.
git push origin HEAD:refs/for/{id}
This push ref will update an exisitng patchset OR create a new patchset
if the update is non-fast-forward.
**Rebasing, Squashing, Amending**
Gitblit makes rebasing, squashing, and amending patchsets easy.
Normally, pushing a non-fast-forward update would require rewind (RW+)
repository permissions. Gitblit provides a magic ref which will allow
ticket participants to rewrite a ticket patchset as long as the ticket
is open.
git push origin HEAD:refs/for/{id}
Pushing changes to this ref allows the patchset authors to rebase,
squash, or amend the patchset commits without requiring client-side use
of the *--force* flag on push AND without requiring RW+ permission to
the repository. Since each patchset is tracked with a ref it is easy to
recover from accidental non-fast-forward updates.
Features
--------
- Ticket tracker with status changes and responsible assignments
- Patchset revision scoring mechanism
- Update/Rewrite patchset handling
- Close-on-push detection
- Server-side Merge button for simple merges
- Comments with Markdown syntax support
- Rich mail notifications
- Voting
- Mentions
- Watch lists
- Querying
- Searches
- Partial miletones support
- Multiple backend options
Diffstat (limited to 'src/main/java/com/gitblit/tickets')
-rw-r--r-- | src/main/java/com/gitblit/tickets/BranchTicketService.java | 799 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/FileTicketService.java | 467 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/ITicketService.java | 1088 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/NullTicketService.java | 129 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/QueryBuilder.java | 222 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/QueryResult.java | 114 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/RedisTicketService.java | 534 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/TicketIndexer.java | 657 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/TicketLabel.java | 77 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/TicketMilestone.java | 53 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/TicketNotifier.java | 617 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/TicketResponsible.java | 59 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/TicketSerializer.java | 175 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/commands.md | 11 | ||||
-rw-r--r-- | src/main/java/com/gitblit/tickets/email.css | 38 |
15 files changed, 5040 insertions, 0 deletions
diff --git a/src/main/java/com/gitblit/tickets/BranchTicketService.java b/src/main/java/com/gitblit/tickets/BranchTicketService.java new file mode 100644 index 00000000..14ed8094 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/BranchTicketService.java @@ -0,0 +1,799 @@ +/* + * Copyright 2014 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.IOException; +import java.text.MessageFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TreeSet; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicLong; + +import javax.inject.Inject; + +import org.eclipse.jgit.api.errors.ConcurrentRefUpdateException; +import org.eclipse.jgit.api.errors.JGitInternalException; +import org.eclipse.jgit.dircache.DirCache; +import org.eclipse.jgit.dircache.DirCacheBuilder; +import org.eclipse.jgit.dircache.DirCacheEntry; +import org.eclipse.jgit.internal.JGitText; +import org.eclipse.jgit.lib.CommitBuilder; +import org.eclipse.jgit.lib.FileMode; +import org.eclipse.jgit.lib.ObjectId; +import org.eclipse.jgit.lib.ObjectInserter; +import org.eclipse.jgit.lib.PersonIdent; +import org.eclipse.jgit.lib.RefUpdate; +import org.eclipse.jgit.lib.RefUpdate.Result; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevCommit; +import org.eclipse.jgit.revwalk.RevTree; +import org.eclipse.jgit.revwalk.RevWalk; +import org.eclipse.jgit.treewalk.CanonicalTreeParser; +import org.eclipse.jgit.treewalk.TreeWalk; + +import com.gitblit.Constants; +import com.gitblit.manager.INotificationManager; +import com.gitblit.manager.IRepositoryManager; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.manager.IUserManager; +import com.gitblit.models.PathModel; +import com.gitblit.models.RefModel; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Attachment; +import com.gitblit.models.TicketModel.Change; +import com.gitblit.utils.ArrayUtils; +import com.gitblit.utils.JGitUtils; +import com.gitblit.utils.StringUtils; + +/** + * Implementation of a ticket service based on an orphan branch. All tickets + * are serialized as a list of JSON changes and persisted in a hashed directory + * structure, similar to the standard git loose object structure. + * + * @author James Moger + * + */ +public class BranchTicketService extends ITicketService { + + public static final String BRANCH = "refs/gitblit/tickets"; + + private static final String JOURNAL = "journal.json"; + + private static final String ID_PATH = "id/"; + + private final Map<String, AtomicLong> lastAssignedId; + + @Inject + public BranchTicketService( + IRuntimeManager runtimeManager, + INotificationManager notificationManager, + IUserManager userManager, + IRepositoryManager repositoryManager) { + + super(runtimeManager, + notificationManager, + userManager, + repositoryManager); + + lastAssignedId = new ConcurrentHashMap<String, AtomicLong>(); + } + + @Override + public BranchTicketService start() { + return this; + } + + @Override + protected void resetCachesImpl() { + lastAssignedId.clear(); + } + + @Override + protected void resetCachesImpl(RepositoryModel repository) { + if (lastAssignedId.containsKey(repository.name)) { + lastAssignedId.get(repository.name).set(0); + } + } + + @Override + protected void close() { + } + + /** + * Returns a RefModel for the refs/gitblit/tickets branch in the repository. + * If the branch can not be found, null is returned. + * + * @return a refmodel for the gitblit tickets branch or null + */ + private RefModel getTicketsBranch(Repository db) { + List<RefModel> refs = JGitUtils.getRefs(db, Constants.R_GITBLIT); + for (RefModel ref : refs) { + if (ref.reference.getName().equals(BRANCH)) { + return ref; + } + } + return null; + } + + /** + * Creates the refs/gitblit/tickets branch. + * @param db + */ + private void createTicketsBranch(Repository db) { + JGitUtils.createOrphanBranch(db, BRANCH, null); + } + + /** + * Returns the ticket path. This follows the same scheme as Git's object + * store path where the first two characters of the hash id are the root + * folder with the remaining characters as a subfolder within that folder. + * + * @param ticketId + * @return the root path of the ticket content on the refs/gitblit/tickets branch + */ + private String toTicketPath(long ticketId) { + StringBuilder sb = new StringBuilder(); + sb.append(ID_PATH); + long m = ticketId % 100L; + if (m < 10) { + sb.append('0'); + } + sb.append(m); + sb.append('/'); + sb.append(ticketId); + return sb.toString(); + } + + /** + * Returns the path to the attachment for the specified ticket. + * + * @param ticketId + * @param filename + * @return the path to the specified attachment + */ + private String toAttachmentPath(long ticketId, String filename) { + return toTicketPath(ticketId) + "/attachments/" + filename; + } + + /** + * Reads a file from the tickets branch. + * + * @param db + * @param file + * @return the file content or null + */ + private String readTicketsFile(Repository db, String file) { + RevWalk rw = null; + try { + ObjectId treeId = db.resolve(BRANCH + "^{tree}"); + if (treeId == null) { + return null; + } + rw = new RevWalk(db); + RevTree tree = rw.lookupTree(treeId); + if (tree != null) { + return JGitUtils.getStringContent(db, tree, file, Constants.ENCODING); + } + } catch (IOException e) { + log.error("failed to read " + file, e); + } finally { + if (rw != null) { + rw.release(); + } + } + return null; + } + + /** + * Writes a file to the tickets branch. + * + * @param db + * @param file + * @param content + * @param createdBy + * @param msg + */ + private void writeTicketsFile(Repository db, String file, String content, String createdBy, String msg) { + if (getTicketsBranch(db) == null) { + createTicketsBranch(db); + } + + DirCache newIndex = DirCache.newInCore(); + DirCacheBuilder builder = newIndex.builder(); + ObjectInserter inserter = db.newObjectInserter(); + + try { + // create an index entry for the revised index + final DirCacheEntry idIndexEntry = new DirCacheEntry(file); + idIndexEntry.setLength(content.length()); + idIndexEntry.setLastModified(System.currentTimeMillis()); + idIndexEntry.setFileMode(FileMode.REGULAR_FILE); + + // insert new ticket index + idIndexEntry.setObjectId(inserter.insert(org.eclipse.jgit.lib.Constants.OBJ_BLOB, + content.getBytes(Constants.ENCODING))); + + // add to temporary in-core index + builder.add(idIndexEntry); + + Set<String> ignorePaths = new HashSet<String>(); + ignorePaths.add(file); + + for (DirCacheEntry entry : getTreeEntries(db, ignorePaths)) { + builder.add(entry); + } + + // finish temporary in-core index used for this commit + builder.finish(); + + // commit the change + commitIndex(db, newIndex, createdBy, msg); + + } catch (ConcurrentRefUpdateException e) { + log.error("", e); + } catch (IOException e) { + log.error("", e); + } finally { + inserter.release(); + } + } + + /** + * Ensures that we have a ticket for this ticket id. + * + * @param repository + * @param ticketId + * @return true if the ticket exists + */ + @Override + public boolean hasTicket(RepositoryModel repository, long ticketId) { + boolean hasTicket = false; + Repository db = repositoryManager.getRepository(repository.name); + try { + RefModel ticketsBranch = getTicketsBranch(db); + if (ticketsBranch == null) { + return false; + } + String ticketPath = toTicketPath(ticketId); + RevCommit tip = JGitUtils.getCommit(db, BRANCH); + hasTicket = !JGitUtils.getFilesInPath(db, ticketPath, tip).isEmpty(); + } finally { + db.close(); + } + return hasTicket; + } + + /** + * Assigns a new ticket id. + * + * @param repository + * @return a new long id + */ + @Override + public synchronized long assignNewId(RepositoryModel repository) { + long newId = 0L; + Repository db = repositoryManager.getRepository(repository.name); + try { + if (getTicketsBranch(db) == null) { + createTicketsBranch(db); + } + + // identify current highest ticket id by scanning the paths in the tip tree + if (!lastAssignedId.containsKey(repository.name)) { + lastAssignedId.put(repository.name, new AtomicLong(0)); + } + AtomicLong lastId = lastAssignedId.get(repository.name); + if (lastId.get() <= 0) { + List<PathModel> paths = JGitUtils.getDocuments(db, Arrays.asList("json"), BRANCH); + for (PathModel path : paths) { + String name = path.name.substring(path.name.lastIndexOf('/') + 1); + if (!JOURNAL.equals(name)) { + continue; + } + String tid = path.path.split("/")[2]; + long ticketId = Long.parseLong(tid); + if (ticketId > lastId.get()) { + lastId.set(ticketId); + } + } + } + + // assign the id and touch an empty journal to hold it's place + newId = lastId.incrementAndGet(); + String journalPath = toTicketPath(newId) + "/" + JOURNAL; + writeTicketsFile(db, journalPath, "", "gitblit", "assigned id #" + newId); + } finally { + db.close(); + } + return newId; + } + + /** + * Returns all the tickets in the repository. Querying tickets from the + * repository requires deserializing all tickets. This is an expensive + * process and not recommended. Tickets are indexed by Lucene and queries + * should be executed against that index. + * + * @param repository + * @param filter + * optional filter to only return matching results + * @return a list of tickets + */ + @Override + public List<TicketModel> getTickets(RepositoryModel repository, TicketFilter filter) { + List<TicketModel> list = new ArrayList<TicketModel>(); + + Repository db = repositoryManager.getRepository(repository.name); + try { + RefModel ticketsBranch = getTicketsBranch(db); + if (ticketsBranch == null) { + return list; + } + + // Collect the set of all json files + List<PathModel> paths = JGitUtils.getDocuments(db, Arrays.asList("json"), BRANCH); + + // Deserialize each ticket and optionally filter out unwanted tickets + for (PathModel path : paths) { + String name = path.name.substring(path.name.lastIndexOf('/') + 1); + if (!JOURNAL.equals(name)) { + continue; + } + String json = readTicketsFile(db, path.path); + if (StringUtils.isEmpty(json)) { + // journal was touched but no changes were written + continue; + } + try { + // Reconstruct ticketId from the path + // id/26/326/journal.json + String tid = path.path.split("/")[2]; + long ticketId = Long.parseLong(tid); + List<Change> changes = TicketSerializer.deserializeJournal(json); + if (ArrayUtils.isEmpty(changes)) { + log.warn("Empty journal for {}:{}", repository, path.path); + continue; + } + TicketModel ticket = TicketModel.buildTicket(changes); + ticket.project = repository.projectPath; + ticket.repository = repository.name; + ticket.number = ticketId; + + // add the ticket, conditionally, to the list + if (filter == null) { + list.add(ticket); + } else { + if (filter.accept(ticket)) { + list.add(ticket); + } + } + } catch (Exception e) { + log.error("failed to deserialize {}/{}\n{}", + new Object [] { repository, path.path, e.getMessage()}); + log.error(null, e); + } + } + + // sort the tickets by creation + Collections.sort(list); + return list; + } finally { + db.close(); + } + } + + /** + * Retrieves the ticket from the repository by first looking-up the changeId + * associated with the ticketId. + * + * @param repository + * @param ticketId + * @return a ticket, if it exists, otherwise null + */ + @Override + protected TicketModel getTicketImpl(RepositoryModel repository, long ticketId) { + Repository db = repositoryManager.getRepository(repository.name); + try { + List<Change> changes = getJournal(db, ticketId); + if (ArrayUtils.isEmpty(changes)) { + log.warn("Empty journal for {}:{}", repository, ticketId); + return null; + } + TicketModel ticket = TicketModel.buildTicket(changes); + if (ticket != null) { + ticket.project = repository.projectPath; + ticket.repository = repository.name; + ticket.number = ticketId; + } + return ticket; + } finally { + db.close(); + } + } + + /** + * Returns the journal for the specified ticket. + * + * @param db + * @param ticketId + * @return a list of changes + */ + private List<Change> getJournal(Repository db, long ticketId) { + RefModel ticketsBranch = getTicketsBranch(db); + if (ticketsBranch == null) { + return new ArrayList<Change>(); + } + + if (ticketId <= 0L) { + return new ArrayList<Change>(); + } + + String journalPath = toTicketPath(ticketId) + "/" + JOURNAL; + String json = readTicketsFile(db, journalPath); + if (StringUtils.isEmpty(json)) { + return new ArrayList<Change>(); + } + List<Change> list = TicketSerializer.deserializeJournal(json); + return list; + } + + @Override + public boolean supportsAttachments() { + return true; + } + + /** + * Retrieves the specified attachment from a ticket. + * + * @param repository + * @param ticketId + * @param filename + * @return an attachment, if found, null otherwise + */ + @Override + public Attachment getAttachment(RepositoryModel repository, long ticketId, String filename) { + if (ticketId <= 0L) { + return null; + } + + // deserialize the ticket model so that we have the attachment metadata + TicketModel ticket = getTicket(repository, ticketId); + Attachment attachment = ticket.getAttachment(filename); + + // attachment not found + if (attachment == null) { + return null; + } + + // retrieve the attachment content + Repository db = repositoryManager.getRepository(repository.name); + try { + String attachmentPath = toAttachmentPath(ticketId, attachment.name); + RevTree tree = JGitUtils.getCommit(db, BRANCH).getTree(); + byte[] content = JGitUtils.getByteContent(db, tree, attachmentPath, false); + attachment.content = content; + attachment.size = content.length; + return attachment; + } finally { + db.close(); + } + } + + /** + * Deletes a ticket from the repository. + * + * @param ticket + * @return true if successful + */ + @Override + protected synchronized boolean deleteTicketImpl(RepositoryModel repository, TicketModel ticket, String deletedBy) { + if (ticket == null) { + throw new RuntimeException("must specify a ticket!"); + } + + boolean success = false; + Repository db = repositoryManager.getRepository(ticket.repository); + try { + RefModel ticketsBranch = getTicketsBranch(db); + + if (ticketsBranch == null) { + throw new RuntimeException(BRANCH + " does not exist!"); + } + String ticketPath = toTicketPath(ticket.number); + + TreeWalk treeWalk = null; + try { + ObjectId treeId = db.resolve(BRANCH + "^{tree}"); + + // Create the in-memory index of the new/updated ticket + DirCache index = DirCache.newInCore(); + DirCacheBuilder builder = index.builder(); + + // Traverse HEAD to add all other paths + treeWalk = new TreeWalk(db); + int hIdx = -1; + if (treeId != null) { + hIdx = treeWalk.addTree(treeId); + } + treeWalk.setRecursive(true); + while (treeWalk.next()) { + String path = treeWalk.getPathString(); + CanonicalTreeParser hTree = null; + if (hIdx != -1) { + hTree = treeWalk.getTree(hIdx, CanonicalTreeParser.class); + } + if (!path.startsWith(ticketPath)) { + // add entries from HEAD for all other paths + if (hTree != null) { + final DirCacheEntry entry = new DirCacheEntry(path); + entry.setObjectId(hTree.getEntryObjectId()); + entry.setFileMode(hTree.getEntryFileMode()); + + // add to temporary in-core index + builder.add(entry); + } + } + } + + // finish temporary in-core index used for this commit + builder.finish(); + + success = commitIndex(db, index, deletedBy, "- " + ticket.number); + + } catch (Throwable t) { + log.error(MessageFormat.format("Failed to delete ticket {0,number,0} from {1}", + ticket.number, db.getDirectory()), t); + } finally { + // release the treewalk + treeWalk.release(); + } + } finally { + db.close(); + } + return success; + } + + /** + * Commit a ticket change to the repository. + * + * @param repository + * @param ticketId + * @param change + * @return true, if the change was committed + */ + @Override + protected synchronized boolean commitChangeImpl(RepositoryModel repository, long ticketId, Change change) { + boolean success = false; + + Repository db = repositoryManager.getRepository(repository.name); + try { + DirCache index = createIndex(db, ticketId, change); + success = commitIndex(db, index, change.author, "#" + ticketId); + + } catch (Throwable t) { + log.error(MessageFormat.format("Failed to commit ticket {0,number,0} to {1}", + ticketId, db.getDirectory()), t); + } finally { + db.close(); + } + return success; + } + + /** + * Creates an in-memory index of the ticket change. + * + * @param changeId + * @param change + * @return an in-memory index + * @throws IOException + */ + private DirCache createIndex(Repository db, long ticketId, Change change) + throws IOException, ClassNotFoundException, NoSuchFieldException { + + String ticketPath = toTicketPath(ticketId); + DirCache newIndex = DirCache.newInCore(); + DirCacheBuilder builder = newIndex.builder(); + ObjectInserter inserter = db.newObjectInserter(); + + Set<String> ignorePaths = new TreeSet<String>(); + try { + // create/update the journal + // exclude the attachment content + List<Change> changes = getJournal(db, ticketId); + changes.add(change); + String journal = TicketSerializer.serializeJournal(changes).trim(); + + byte [] journalBytes = journal.getBytes(Constants.ENCODING); + String journalPath = ticketPath + "/" + JOURNAL; + final DirCacheEntry journalEntry = new DirCacheEntry(journalPath); + journalEntry.setLength(journalBytes.length); + journalEntry.setLastModified(change.date.getTime()); + journalEntry.setFileMode(FileMode.REGULAR_FILE); + journalEntry.setObjectId(inserter.insert(org.eclipse.jgit.lib.Constants.OBJ_BLOB, journalBytes)); + + // add journal to index + builder.add(journalEntry); + ignorePaths.add(journalEntry.getPathString()); + + // Add any attachments to the index + if (change.hasAttachments()) { + for (Attachment attachment : change.attachments) { + // build a path name for the attachment and mark as ignored + String path = toAttachmentPath(ticketId, attachment.name); + ignorePaths.add(path); + + // create an index entry for this attachment + final DirCacheEntry entry = new DirCacheEntry(path); + entry.setLength(attachment.content.length); + entry.setLastModified(change.date.getTime()); + entry.setFileMode(FileMode.REGULAR_FILE); + + // insert object + entry.setObjectId(inserter.insert(org.eclipse.jgit.lib.Constants.OBJ_BLOB, attachment.content)); + + // add to temporary in-core index + builder.add(entry); + } + } + + for (DirCacheEntry entry : getTreeEntries(db, ignorePaths)) { + builder.add(entry); + } + + // finish the index + builder.finish(); + } finally { + inserter.release(); + } + return newIndex; + } + + /** + * Returns all tree entries that do not match the ignore paths. + * + * @param db + * @param ignorePaths + * @param dcBuilder + * @throws IOException + */ + private List<DirCacheEntry> getTreeEntries(Repository db, Collection<String> ignorePaths) throws IOException { + List<DirCacheEntry> list = new ArrayList<DirCacheEntry>(); + TreeWalk tw = null; + try { + tw = new TreeWalk(db); + ObjectId treeId = db.resolve(BRANCH + "^{tree}"); + int hIdx = tw.addTree(treeId); + tw.setRecursive(true); + + while (tw.next()) { + String path = tw.getPathString(); + CanonicalTreeParser hTree = null; + if (hIdx != -1) { + hTree = tw.getTree(hIdx, CanonicalTreeParser.class); + } + if (!ignorePaths.contains(path)) { + // add all other tree entries + if (hTree != null) { + final DirCacheEntry entry = new DirCacheEntry(path); + entry.setObjectId(hTree.getEntryObjectId()); + entry.setFileMode(hTree.getEntryFileMode()); + list.add(entry); + } + } + } + } finally { + if (tw != null) { + tw.release(); + } + } + return list; + } + + private boolean commitIndex(Repository db, DirCache index, String author, String message) throws IOException, ConcurrentRefUpdateException { + boolean success = false; + + ObjectId headId = db.resolve(BRANCH + "^{commit}"); + if (headId == null) { + // create the branch + createTicketsBranch(db); + headId = db.resolve(BRANCH + "^{commit}"); + } + ObjectInserter odi = db.newObjectInserter(); + try { + // Create the in-memory index of the new/updated ticket + ObjectId indexTreeId = index.writeTree(odi); + + // Create a commit object + PersonIdent ident = new PersonIdent(author, "gitblit@localhost"); + CommitBuilder commit = new CommitBuilder(); + commit.setAuthor(ident); + commit.setCommitter(ident); + commit.setEncoding(Constants.ENCODING); + commit.setMessage(message); + commit.setParentId(headId); + commit.setTreeId(indexTreeId); + + // Insert the commit into the repository + ObjectId commitId = odi.insert(commit); + odi.flush(); + + RevWalk revWalk = new RevWalk(db); + try { + RevCommit revCommit = revWalk.parseCommit(commitId); + RefUpdate ru = db.updateRef(BRANCH); + ru.setNewObjectId(commitId); + ru.setExpectedOldObjectId(headId); + ru.setRefLogMessage("commit: " + revCommit.getShortMessage(), false); + Result rc = ru.forceUpdate(); + switch (rc) { + case NEW: + case FORCED: + case FAST_FORWARD: + success = true; + break; + case REJECTED: + case LOCK_FAILURE: + throw new ConcurrentRefUpdateException(JGitText.get().couldNotLockHEAD, + ru.getRef(), rc); + default: + throw new JGitInternalException(MessageFormat.format( + JGitText.get().updatingRefFailed, BRANCH, commitId.toString(), + rc)); + } + } finally { + revWalk.release(); + } + } finally { + odi.release(); + } + return success; + } + + @Override + protected boolean deleteAllImpl(RepositoryModel repository) { + Repository db = repositoryManager.getRepository(repository.name); + try { + RefModel branch = getTicketsBranch(db); + if (branch != null) { + return JGitUtils.deleteBranchRef(db, BRANCH); + } + return true; + } catch (Exception e) { + log.error(null, e); + } finally { + db.close(); + } + return false; + } + + @Override + protected boolean renameImpl(RepositoryModel oldRepository, RepositoryModel newRepository) { + return true; + } + + @Override + public String toString() { + return getClass().getSimpleName(); + } +} diff --git a/src/main/java/com/gitblit/tickets/FileTicketService.java b/src/main/java/com/gitblit/tickets/FileTicketService.java new file mode 100644 index 00000000..8375a2ba --- /dev/null +++ b/src/main/java/com/gitblit/tickets/FileTicketService.java @@ -0,0 +1,467 @@ +/* + * Copyright 2014 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.File; +import java.io.IOException; +import java.text.MessageFormat; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicLong; + +import javax.inject.Inject; + +import org.eclipse.jgit.lib.Repository; + +import com.gitblit.Constants; +import com.gitblit.manager.INotificationManager; +import com.gitblit.manager.IRepositoryManager; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.manager.IUserManager; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Attachment; +import com.gitblit.models.TicketModel.Change; +import com.gitblit.utils.ArrayUtils; +import com.gitblit.utils.FileUtils; +import com.gitblit.utils.StringUtils; + +/** + * Implementation of a ticket service based on a directory within the repository. + * All tickets are serialized as a list of JSON changes and persisted in a hashed + * directory structure, similar to the standard git loose object structure. + * + * @author James Moger + * + */ +public class FileTicketService extends ITicketService { + + private static final String JOURNAL = "journal.json"; + + private static final String TICKETS_PATH = "tickets/"; + + private final Map<String, AtomicLong> lastAssignedId; + + @Inject + public FileTicketService( + IRuntimeManager runtimeManager, + INotificationManager notificationManager, + IUserManager userManager, + IRepositoryManager repositoryManager) { + + super(runtimeManager, + notificationManager, + userManager, + repositoryManager); + + lastAssignedId = new ConcurrentHashMap<String, AtomicLong>(); + } + + @Override + public FileTicketService start() { + return this; + } + + @Override + protected void resetCachesImpl() { + lastAssignedId.clear(); + } + + @Override + protected void resetCachesImpl(RepositoryModel repository) { + if (lastAssignedId.containsKey(repository.name)) { + lastAssignedId.get(repository.name).set(0); + } + } + + @Override + protected void close() { + } + + /** + * Returns the ticket path. This follows the same scheme as Git's object + * store path where the first two characters of the hash id are the root + * folder with the remaining characters as a subfolder within that folder. + * + * @param ticketId + * @return the root path of the ticket content in the ticket directory + */ + private String toTicketPath(long ticketId) { + StringBuilder sb = new StringBuilder(); + sb.append(TICKETS_PATH); + long m = ticketId % 100L; + if (m < 10) { + sb.append('0'); + } + sb.append(m); + sb.append('/'); + sb.append(ticketId); + return sb.toString(); + } + + /** + * Returns the path to the attachment for the specified ticket. + * + * @param ticketId + * @param filename + * @return the path to the specified attachment + */ + private String toAttachmentPath(long ticketId, String filename) { + return toTicketPath(ticketId) + "/attachments/" + filename; + } + + /** + * Ensures that we have a ticket for this ticket id. + * + * @param repository + * @param ticketId + * @return true if the ticket exists + */ + @Override + public boolean hasTicket(RepositoryModel repository, long ticketId) { + boolean hasTicket = false; + Repository db = repositoryManager.getRepository(repository.name); + try { + String journalPath = toTicketPath(ticketId) + "/" + JOURNAL; + hasTicket = new File(db.getDirectory(), journalPath).exists(); + } finally { + db.close(); + } + return hasTicket; + } + + /** + * Assigns a new ticket id. + * + * @param repository + * @return a new long id + */ + @Override + public synchronized long assignNewId(RepositoryModel repository) { + long newId = 0L; + Repository db = repositoryManager.getRepository(repository.name); + try { + if (!lastAssignedId.containsKey(repository.name)) { + lastAssignedId.put(repository.name, new AtomicLong(0)); + } + AtomicLong lastId = lastAssignedId.get(repository.name); + if (lastId.get() <= 0) { + // identify current highest ticket id by scanning the paths in the tip tree + File dir = new File(db.getDirectory(), TICKETS_PATH); + dir.mkdirs(); + List<File> journals = findAll(dir, JOURNAL); + for (File journal : journals) { + // Reconstruct ticketId from the path + // id/26/326/journal.json + String path = FileUtils.getRelativePath(dir, journal); + String tid = path.split("/")[1]; + long ticketId = Long.parseLong(tid); + if (ticketId > lastId.get()) { + lastId.set(ticketId); + } + } + } + + // assign the id and touch an empty journal to hold it's place + newId = lastId.incrementAndGet(); + String journalPath = toTicketPath(newId) + "/" + JOURNAL; + File journal = new File(db.getDirectory(), journalPath); + journal.getParentFile().mkdirs(); + journal.createNewFile(); + } catch (IOException e) { + log.error("failed to assign ticket id", e); + return 0L; + } finally { + db.close(); + } + return newId; + } + + /** + * Returns all the tickets in the repository. Querying tickets from the + * repository requires deserializing all tickets. This is an expensive + * process and not recommended. Tickets are indexed by Lucene and queries + * should be executed against that index. + * + * @param repository + * @param filter + * optional filter to only return matching results + * @return a list of tickets + */ + @Override + public List<TicketModel> getTickets(RepositoryModel repository, TicketFilter filter) { + List<TicketModel> list = new ArrayList<TicketModel>(); + + Repository db = repositoryManager.getRepository(repository.name); + try { + // Collect the set of all json files + File dir = new File(db.getDirectory(), TICKETS_PATH); + List<File> journals = findAll(dir, JOURNAL); + + // Deserialize each ticket and optionally filter out unwanted tickets + for (File journal : journals) { + String json = null; + try { + json = new String(FileUtils.readContent(journal), Constants.ENCODING); + } catch (Exception e) { + log.error(null, e); + } + if (StringUtils.isEmpty(json)) { + // journal was touched but no changes were written + continue; + } + try { + // Reconstruct ticketId from the path + // id/26/326/journal.json + String path = FileUtils.getRelativePath(dir, journal); + String tid = path.split("/")[1]; + long ticketId = Long.parseLong(tid); + List<Change> changes = TicketSerializer.deserializeJournal(json); + if (ArrayUtils.isEmpty(changes)) { + log.warn("Empty journal for {}:{}", repository, journal); + continue; + } + TicketModel ticket = TicketModel.buildTicket(changes); + ticket.project = repository.projectPath; + ticket.repository = repository.name; + ticket.number = ticketId; + + // add the ticket, conditionally, to the list + if (filter == null) { + list.add(ticket); + } else { + if (filter.accept(ticket)) { + list.add(ticket); + } + } + } catch (Exception e) { + log.error("failed to deserialize {}/{}\n{}", + new Object [] { repository, journal, e.getMessage()}); + log.error(null, e); + } + } + + // sort the tickets by creation + Collections.sort(list); + return list; + } finally { + db.close(); + } + } + + private List<File> findAll(File dir, String filename) { + List<File> list = new ArrayList<File>(); + for (File file : dir.listFiles()) { + if (file.isDirectory()) { + list.addAll(findAll(file, filename)); + } else if (file.isFile()) { + if (file.getName().equals(filename)) { + list.add(file); + } + } + } + return list; + } + + /** + * Retrieves the ticket from the repository by first looking-up the changeId + * associated with the ticketId. + * + * @param repository + * @param ticketId + * @return a ticket, if it exists, otherwise null + */ + @Override + protected TicketModel getTicketImpl(RepositoryModel repository, long ticketId) { + Repository db = repositoryManager.getRepository(repository.name); + try { + List<Change> changes = getJournal(db, ticketId); + if (ArrayUtils.isEmpty(changes)) { + log.warn("Empty journal for {}:{}", repository, ticketId); + return null; + } + TicketModel ticket = TicketModel.buildTicket(changes); + if (ticket != null) { + ticket.project = repository.projectPath; + ticket.repository = repository.name; + ticket.number = ticketId; + } + return ticket; + } finally { + db.close(); + } + } + + /** + * Returns the journal for the specified ticket. + * + * @param db + * @param ticketId + * @return a list of changes + */ + private List<Change> getJournal(Repository db, long ticketId) { + if (ticketId <= 0L) { + return new ArrayList<Change>(); + } + + String journalPath = toTicketPath(ticketId) + "/" + JOURNAL; + File journal = new File(db.getDirectory(), journalPath); + if (!journal.exists()) { + return new ArrayList<Change>(); + } + + String json = null; + try { + json = new String(FileUtils.readContent(journal), Constants.ENCODING); + } catch (Exception e) { + log.error(null, e); + } + if (StringUtils.isEmpty(json)) { + return new ArrayList<Change>(); + } + List<Change> list = TicketSerializer.deserializeJournal(json); + return list; + } + + @Override + public boolean supportsAttachments() { + return true; + } + + /** + * Retrieves the specified attachment from a ticket. + * + * @param repository + * @param ticketId + * @param filename + * @return an attachment, if found, null otherwise + */ + @Override + public Attachment getAttachment(RepositoryModel repository, long ticketId, String filename) { + if (ticketId <= 0L) { + return null; + } + + // deserialize the ticket model so that we have the attachment metadata + TicketModel ticket = getTicket(repository, ticketId); + Attachment attachment = ticket.getAttachment(filename); + + // attachment not found + if (attachment == null) { + return null; + } + + // retrieve the attachment content + Repository db = repositoryManager.getRepository(repository.name); + try { + String attachmentPath = toAttachmentPath(ticketId, attachment.name); + File file = new File(db.getDirectory(), attachmentPath); + if (file.exists()) { + attachment.content = FileUtils.readContent(file); + attachment.size = attachment.content.length; + } + return attachment; + } finally { + db.close(); + } + } + + /** + * Deletes a ticket from the repository. + * + * @param ticket + * @return true if successful + */ + @Override + protected synchronized boolean deleteTicketImpl(RepositoryModel repository, TicketModel ticket, String deletedBy) { + if (ticket == null) { + throw new RuntimeException("must specify a ticket!"); + } + + boolean success = false; + Repository db = repositoryManager.getRepository(ticket.repository); + try { + String ticketPath = toTicketPath(ticket.number); + File dir = new File(db.getDirectory(), ticketPath); + if (dir.exists()) { + success = FileUtils.delete(dir); + } + success = true; + } finally { + db.close(); + } + return success; + } + + /** + * Commit a ticket change to the repository. + * + * @param repository + * @param ticketId + * @param change + * @return true, if the change was committed + */ + @Override + protected synchronized boolean commitChangeImpl(RepositoryModel repository, long ticketId, Change change) { + boolean success = false; + + Repository db = repositoryManager.getRepository(repository.name); + try { + List<Change> changes = getJournal(db, ticketId); + changes.add(change); + String journal = TicketSerializer.serializeJournal(changes).trim(); + + String journalPath = toTicketPath(ticketId) + "/" + JOURNAL; + File file = new File(db.getDirectory(), journalPath); + file.getParentFile().mkdirs(); + FileUtils.writeContent(file, journal); + success = true; + } catch (Throwable t) { + log.error(MessageFormat.format("Failed to commit ticket {0,number,0} to {1}", + ticketId, db.getDirectory()), t); + } finally { + db.close(); + } + return success; + } + + @Override + protected boolean deleteAllImpl(RepositoryModel repository) { + Repository db = repositoryManager.getRepository(repository.name); + try { + File dir = new File(db.getDirectory(), TICKETS_PATH); + return FileUtils.delete(dir); + } catch (Exception e) { + log.error(null, e); + } finally { + db.close(); + } + return false; + } + + @Override + protected boolean renameImpl(RepositoryModel oldRepository, RepositoryModel newRepository) { + return true; + } + + @Override + public String toString() { + return getClass().getSimpleName(); + } +} diff --git a/src/main/java/com/gitblit/tickets/ITicketService.java b/src/main/java/com/gitblit/tickets/ITicketService.java new file mode 100644 index 00000000..d04cd5e1 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/ITicketService.java @@ -0,0 +1,1088 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.IOException; +import java.text.MessageFormat; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.BitSet; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.lib.StoredConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.gitblit.IStoredSettings; +import com.gitblit.Keys; +import com.gitblit.manager.INotificationManager; +import com.gitblit.manager.IRepositoryManager; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.manager.IUserManager; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Attachment; +import com.gitblit.models.TicketModel.Change; +import com.gitblit.models.TicketModel.Field; +import com.gitblit.models.TicketModel.Patchset; +import com.gitblit.models.TicketModel.Status; +import com.gitblit.tickets.TicketIndexer.Lucene; +import com.gitblit.utils.DiffUtils; +import com.gitblit.utils.DiffUtils.DiffStat; +import com.gitblit.utils.StringUtils; +import com.google.common.cache.Cache; +import com.google.common.cache.CacheBuilder; + +/** + * Abstract parent class of a ticket service that stubs out required methods + * and transparently handles Lucene indexing. + * + * @author James Moger + * + */ +public abstract class ITicketService { + + private static final String LABEL = "label"; + + private static final String MILESTONE = "milestone"; + + private static final String STATUS = "status"; + + private static final String COLOR = "color"; + + private static final String DUE = "due"; + + private static final String DUE_DATE_PATTERN = "yyyy-MM-dd"; + + /** + * Object filter interface to querying against all available ticket models. + */ + public interface TicketFilter { + + boolean accept(TicketModel ticket); + } + + protected final Logger log; + + protected final IStoredSettings settings; + + protected final IRuntimeManager runtimeManager; + + protected final INotificationManager notificationManager; + + protected final IUserManager userManager; + + protected final IRepositoryManager repositoryManager; + + protected final TicketIndexer indexer; + + private final Cache<TicketKey, TicketModel> ticketsCache; + + private final Map<String, List<TicketLabel>> labelsCache; + + private final Map<String, List<TicketMilestone>> milestonesCache; + + private static class TicketKey { + final String repository; + final long ticketId; + + TicketKey(RepositoryModel repository, long ticketId) { + this.repository = repository.name; + this.ticketId = ticketId; + } + + @Override + public int hashCode() { + return (repository + ticketId).hashCode(); + } + + @Override + public boolean equals(Object o) { + if (o instanceof TicketKey) { + return o.hashCode() == hashCode(); + } + return false; + } + + @Override + public String toString() { + return repository + ":" + ticketId; + } + } + + + /** + * Creates a ticket service. + */ + public ITicketService( + IRuntimeManager runtimeManager, + INotificationManager notificationManager, + IUserManager userManager, + IRepositoryManager repositoryManager) { + + this.log = LoggerFactory.getLogger(getClass()); + this.settings = runtimeManager.getSettings(); + this.runtimeManager = runtimeManager; + this.notificationManager = notificationManager; + this.userManager = userManager; + this.repositoryManager = repositoryManager; + + this.indexer = new TicketIndexer(runtimeManager); + + CacheBuilder<Object, Object> cb = CacheBuilder.newBuilder(); + this.ticketsCache = cb + .maximumSize(1000) + .expireAfterAccess(30, TimeUnit.MINUTES) + .build(); + + this.labelsCache = new ConcurrentHashMap<String, List<TicketLabel>>(); + this.milestonesCache = new ConcurrentHashMap<String, List<TicketMilestone>>(); + } + + /** + * Start the service. + * + */ + public abstract ITicketService start(); + + /** + * Stop the service. + * + */ + public final ITicketService stop() { + indexer.close(); + ticketsCache.invalidateAll(); + repositoryManager.closeAll(); + close(); + return this; + } + + /** + * Creates a ticket notifier. The ticket notifier is not thread-safe! + * + */ + public TicketNotifier createNotifier() { + return new TicketNotifier( + runtimeManager, + notificationManager, + userManager, + repositoryManager, + this); + } + + /** + * Returns the ready status of the ticket service. + * + * @return true if the ticket service is ready + */ + public boolean isReady() { + return true; + } + + /** + * Returns true if the new patchsets can be accepted for this repository. + * + * @param repository + * @return true if patchsets are being accepted + */ + public boolean isAcceptingNewPatchsets(RepositoryModel repository) { + return isReady() + && settings.getBoolean(Keys.tickets.acceptNewPatchsets, true) + && repository.acceptNewPatchsets + && isAcceptingTicketUpdates(repository); + } + + /** + * Returns true if new tickets can be manually created for this repository. + * This is separate from accepting patchsets. + * + * @param repository + * @return true if tickets are being accepted + */ + public boolean isAcceptingNewTickets(RepositoryModel repository) { + return isReady() + && settings.getBoolean(Keys.tickets.acceptNewTickets, true) + && repository.acceptNewTickets + && isAcceptingTicketUpdates(repository); + } + + /** + * Returns true if ticket updates are allowed for this repository. + * + * @param repository + * @return true if tickets are allowed to be updated + */ + public boolean isAcceptingTicketUpdates(RepositoryModel repository) { + return isReady() + && repository.isBare + && !repository.isFrozen + && !repository.isMirror; + } + + /** + * Returns true if the repository has any tickets + * @param repository + * @return true if the repository has tickets + */ + public boolean hasTickets(RepositoryModel repository) { + return indexer.hasTickets(repository); + } + + /** + * Closes any open resources used by this service. + */ + protected abstract void close(); + + /** + * Reset all caches in the service. + */ + public final synchronized void resetCaches() { + ticketsCache.invalidateAll(); + labelsCache.clear(); + milestonesCache.clear(); + resetCachesImpl(); + } + + protected abstract void resetCachesImpl(); + + /** + * Reset any caches for the repository in the service. + */ + public final synchronized void resetCaches(RepositoryModel repository) { + List<TicketKey> repoKeys = new ArrayList<TicketKey>(); + for (TicketKey key : ticketsCache.asMap().keySet()) { + if (key.repository.equals(repository.name)) { + repoKeys.add(key); + } + } + ticketsCache.invalidateAll(repoKeys); + labelsCache.remove(repository.name); + milestonesCache.remove(repository.name); + resetCachesImpl(repository); + } + + protected abstract void resetCachesImpl(RepositoryModel repository); + + + /** + * Returns the list of labels for the repository. + * + * @param repository + * @return the list of labels + */ + public List<TicketLabel> getLabels(RepositoryModel repository) { + String key = repository.name; + if (labelsCache.containsKey(key)) { + return labelsCache.get(key); + } + List<TicketLabel> list = new ArrayList<TicketLabel>(); + Repository db = repositoryManager.getRepository(repository.name); + try { + StoredConfig config = db.getConfig(); + Set<String> names = config.getSubsections(LABEL); + for (String name : names) { + TicketLabel label = new TicketLabel(name); + label.color = config.getString(LABEL, name, COLOR); + list.add(label); + } + labelsCache.put(key, Collections.unmodifiableList(list)); + } catch (Exception e) { + log.error("invalid tickets settings for " + repository, e); + } finally { + db.close(); + } + return list; + } + + /** + * Returns a TicketLabel object for a given label. If the label is not + * found, a ticket label object is created. + * + * @param repository + * @param label + * @return a TicketLabel + */ + public TicketLabel getLabel(RepositoryModel repository, String label) { + for (TicketLabel tl : getLabels(repository)) { + if (tl.name.equalsIgnoreCase(label)) { + String q = QueryBuilder.q(Lucene.rid.matches(repository.getRID())).and(Lucene.labels.matches(label)).build(); + tl.tickets = indexer.queryFor(q, 1, 0, Lucene.number.name(), true); + return tl; + } + } + return new TicketLabel(label); + } + + /** + * Creates a label. + * + * @param repository + * @param milestone + * @param createdBy + * @return the label + */ + public synchronized TicketLabel createLabel(RepositoryModel repository, String label, String createdBy) { + TicketLabel lb = new TicketMilestone(label); + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + StoredConfig config = db.getConfig(); + config.setString(LABEL, label, COLOR, lb.color); + config.save(); + } catch (IOException e) { + log.error("failed to create label " + label + " in " + repository, e); + } finally { + db.close(); + } + return lb; + } + + /** + * Updates a label. + * + * @param repository + * @param label + * @param createdBy + * @return true if the update was successful + */ + public synchronized boolean updateLabel(RepositoryModel repository, TicketLabel label, String createdBy) { + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + StoredConfig config = db.getConfig(); + config.setString(LABEL, label.name, COLOR, label.color); + config.save(); + + return true; + } catch (IOException e) { + log.error("failed to update label " + label + " in " + repository, e); + } finally { + db.close(); + } + return false; + } + + /** + * Renames a label. + * + * @param repository + * @param oldName + * @param newName + * @param createdBy + * @return true if the rename was successful + */ + public synchronized boolean renameLabel(RepositoryModel repository, String oldName, String newName, String createdBy) { + if (StringUtils.isEmpty(newName)) { + throw new IllegalArgumentException("new label can not be empty!"); + } + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + TicketLabel label = getLabel(repository, oldName); + StoredConfig config = db.getConfig(); + config.unsetSection(LABEL, oldName); + config.setString(LABEL, newName, COLOR, label.color); + config.save(); + + for (QueryResult qr : label.tickets) { + Change change = new Change(createdBy); + change.unlabel(oldName); + change.label(newName); + updateTicket(repository, qr.number, change); + } + + return true; + } catch (IOException e) { + log.error("failed to rename label " + oldName + " in " + repository, e); + } finally { + db.close(); + } + return false; + } + + /** + * Deletes a label. + * + * @param repository + * @param label + * @param createdBy + * @return true if the delete was successful + */ + public synchronized boolean deleteLabel(RepositoryModel repository, String label, String createdBy) { + if (StringUtils.isEmpty(label)) { + throw new IllegalArgumentException("label can not be empty!"); + } + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + StoredConfig config = db.getConfig(); + config.unsetSection(LABEL, label); + config.save(); + + return true; + } catch (IOException e) { + log.error("failed to delete label " + label + " in " + repository, e); + } finally { + db.close(); + } + return false; + } + + /** + * Returns the list of milestones for the repository. + * + * @param repository + * @return the list of milestones + */ + public List<TicketMilestone> getMilestones(RepositoryModel repository) { + String key = repository.name; + if (milestonesCache.containsKey(key)) { + return milestonesCache.get(key); + } + List<TicketMilestone> list = new ArrayList<TicketMilestone>(); + Repository db = repositoryManager.getRepository(repository.name); + try { + StoredConfig config = db.getConfig(); + Set<String> names = config.getSubsections(MILESTONE); + for (String name : names) { + TicketMilestone milestone = new TicketMilestone(name); + milestone.status = Status.fromObject(config.getString(MILESTONE, name, STATUS), milestone.status); + milestone.color = config.getString(MILESTONE, name, COLOR); + String due = config.getString(MILESTONE, name, DUE); + if (!StringUtils.isEmpty(due)) { + try { + milestone.due = new SimpleDateFormat(DUE_DATE_PATTERN).parse(due); + } catch (ParseException e) { + log.error("failed to parse {} milestone {} due date \"{}\"", + new Object [] { repository, name, due }); + } + } + list.add(milestone); + } + milestonesCache.put(key, Collections.unmodifiableList(list)); + } catch (Exception e) { + log.error("invalid tickets settings for " + repository, e); + } finally { + db.close(); + } + return list; + } + + /** + * Returns the list of milestones for the repository that match the status. + * + * @param repository + * @param status + * @return the list of milestones + */ + public List<TicketMilestone> getMilestones(RepositoryModel repository, Status status) { + List<TicketMilestone> matches = new ArrayList<TicketMilestone>(); + for (TicketMilestone milestone : getMilestones(repository)) { + if (status == milestone.status) { + matches.add(milestone); + } + } + return matches; + } + + /** + * Returns the specified milestone or null if the milestone does not exist. + * + * @param repository + * @param milestone + * @return the milestone or null if it does not exist + */ + public TicketMilestone getMilestone(RepositoryModel repository, String milestone) { + for (TicketMilestone ms : getMilestones(repository)) { + if (ms.name.equalsIgnoreCase(milestone)) { + String q = QueryBuilder.q(Lucene.rid.matches(repository.getRID())).and(Lucene.milestone.matches(milestone)).build(); + ms.tickets = indexer.queryFor(q, 1, 0, Lucene.number.name(), true); + return ms; + } + } + return null; + } + + /** + * Creates a milestone. + * + * @param repository + * @param milestone + * @param createdBy + * @return the milestone + */ + public synchronized TicketMilestone createMilestone(RepositoryModel repository, String milestone, String createdBy) { + TicketMilestone ms = new TicketMilestone(milestone); + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + StoredConfig config = db.getConfig(); + config.setString(MILESTONE, milestone, STATUS, ms.status.name()); + config.setString(MILESTONE, milestone, COLOR, ms.color); + config.save(); + + milestonesCache.remove(repository.name); + } catch (IOException e) { + log.error("failed to create milestone " + milestone + " in " + repository, e); + } finally { + db.close(); + } + return ms; + } + + /** + * Updates a milestone. + * + * @param repository + * @param milestone + * @param createdBy + * @return true if successful + */ + public synchronized boolean updateMilestone(RepositoryModel repository, TicketMilestone milestone, String createdBy) { + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + StoredConfig config = db.getConfig(); + config.setString(MILESTONE, milestone.name, STATUS, milestone.status.name()); + config.setString(MILESTONE, milestone.name, COLOR, milestone.color); + if (milestone.due != null) { + config.setString(MILESTONE, milestone.name, DUE, + new SimpleDateFormat(DUE_DATE_PATTERN).format(milestone.due)); + } + config.save(); + + milestonesCache.remove(repository.name); + return true; + } catch (IOException e) { + log.error("failed to update milestone " + milestone + " in " + repository, e); + } finally { + db.close(); + } + return false; + } + + /** + * Renames a milestone. + * + * @param repository + * @param oldName + * @param newName + * @param createdBy + * @return true if successful + */ + public synchronized boolean renameMilestone(RepositoryModel repository, String oldName, String newName, String createdBy) { + if (StringUtils.isEmpty(newName)) { + throw new IllegalArgumentException("new milestone can not be empty!"); + } + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + TicketMilestone milestone = getMilestone(repository, oldName); + StoredConfig config = db.getConfig(); + config.unsetSection(MILESTONE, oldName); + config.setString(MILESTONE, newName, STATUS, milestone.status.name()); + config.setString(MILESTONE, newName, COLOR, milestone.color); + if (milestone.due != null) { + config.setString(MILESTONE, milestone.name, DUE, + new SimpleDateFormat(DUE_DATE_PATTERN).format(milestone.due)); + } + config.save(); + + milestonesCache.remove(repository.name); + + TicketNotifier notifier = createNotifier(); + for (QueryResult qr : milestone.tickets) { + Change change = new Change(createdBy); + change.setField(Field.milestone, newName); + TicketModel ticket = updateTicket(repository, qr.number, change); + notifier.queueMailing(ticket); + } + notifier.sendAll(); + + return true; + } catch (IOException e) { + log.error("failed to rename milestone " + oldName + " in " + repository, e); + } finally { + db.close(); + } + return false; + } + /** + * Deletes a milestone. + * + * @param repository + * @param milestone + * @param createdBy + * @return true if successful + */ + public synchronized boolean deleteMilestone(RepositoryModel repository, String milestone, String createdBy) { + if (StringUtils.isEmpty(milestone)) { + throw new IllegalArgumentException("milestone can not be empty!"); + } + Repository db = null; + try { + db = repositoryManager.getRepository(repository.name); + StoredConfig config = db.getConfig(); + config.unsetSection(MILESTONE, milestone); + config.save(); + + milestonesCache.remove(repository.name); + + return true; + } catch (IOException e) { + log.error("failed to delete milestone " + milestone + " in " + repository, e); + } finally { + db.close(); + } + return false; + } + + /** + * Assigns a new ticket id. + * + * @param repository + * @return a new ticket id + */ + public abstract long assignNewId(RepositoryModel repository); + + /** + * Ensures that we have a ticket for this ticket id. + * + * @param repository + * @param ticketId + * @return true if the ticket exists + */ + public abstract boolean hasTicket(RepositoryModel repository, long ticketId); + + /** + * Returns all tickets. This is not a Lucene search! + * + * @param repository + * @return all tickets + */ + public List<TicketModel> getTickets(RepositoryModel repository) { + return getTickets(repository, null); + } + + /** + * Returns all tickets that satisfy the filter. Retrieving tickets from the + * service requires deserializing all journals and building ticket models. + * This is an expensive process and not recommended. Instead, the queryFor + * method should be used which executes against the Lucene index. + * + * @param repository + * @param filter + * optional issue filter to only return matching results + * @return a list of tickets + */ + public abstract List<TicketModel> getTickets(RepositoryModel repository, TicketFilter filter); + + /** + * Retrieves the ticket. + * + * @param repository + * @param ticketId + * @return a ticket, if it exists, otherwise null + */ + public final TicketModel getTicket(RepositoryModel repository, long ticketId) { + TicketKey key = new TicketKey(repository, ticketId); + TicketModel ticket = ticketsCache.getIfPresent(key); + + if (ticket == null) { + // load & cache ticket + ticket = getTicketImpl(repository, ticketId); + if (ticket.hasPatchsets()) { + Repository r = repositoryManager.getRepository(repository.name); + try { + Patchset patchset = ticket.getCurrentPatchset(); + DiffStat diffStat = DiffUtils.getDiffStat(r, patchset.base, patchset.tip); + // diffstat could be null if we have ticket data without the + // commit objects. e.g. ticket replication without repo + // mirroring + if (diffStat != null) { + ticket.insertions = diffStat.getInsertions(); + ticket.deletions = diffStat.getDeletions(); + } + } finally { + r.close(); + } + } + if (ticket != null) { + ticketsCache.put(key, ticket); + } + } + return ticket; + } + + /** + * Retrieves the ticket. + * + * @param repository + * @param ticketId + * @return a ticket, if it exists, otherwise null + */ + protected abstract TicketModel getTicketImpl(RepositoryModel repository, long ticketId); + + /** + * Get the ticket url + * + * @param ticket + * @return the ticket url + */ + public String getTicketUrl(TicketModel ticket) { + final String canonicalUrl = settings.getString(Keys.web.canonicalUrl, "https://localhost:8443"); + final String hrefPattern = "{0}/tickets?r={1}&h={2,number,0}"; + return MessageFormat.format(hrefPattern, canonicalUrl, ticket.repository, ticket.number); + } + + /** + * Get the compare url + * + * @param base + * @param tip + * @return the compare url + */ + public String getCompareUrl(TicketModel ticket, String base, String tip) { + final String canonicalUrl = settings.getString(Keys.web.canonicalUrl, "https://localhost:8443"); + final String hrefPattern = "{0}/compare?r={1}&h={2}..{3}"; + return MessageFormat.format(hrefPattern, canonicalUrl, ticket.repository, base, tip); + } + + /** + * Returns true if attachments are supported. + * + * @return true if attachments are supported + */ + public abstract boolean supportsAttachments(); + + /** + * Retrieves the specified attachment from a ticket. + * + * @param repository + * @param ticketId + * @param filename + * @return an attachment, if found, null otherwise + */ + public abstract Attachment getAttachment(RepositoryModel repository, long ticketId, String filename); + + /** + * Creates a ticket. Your change must include a repository, author & title, + * at a minimum. If your change does not have those minimum requirements a + * RuntimeException will be thrown. + * + * @param repository + * @param change + * @return true if successful + */ + public TicketModel createTicket(RepositoryModel repository, Change change) { + return createTicket(repository, 0L, change); + } + + /** + * Creates a ticket. Your change must include a repository, author & title, + * at a minimum. If your change does not have those minimum requirements a + * RuntimeException will be thrown. + * + * @param repository + * @param ticketId (if <=0 the ticket id will be assigned) + * @param change + * @return true if successful + */ + public TicketModel createTicket(RepositoryModel repository, long ticketId, Change change) { + + if (repository == null) { + throw new RuntimeException("Must specify a repository!"); + } + if (StringUtils.isEmpty(change.author)) { + throw new RuntimeException("Must specify a change author!"); + } + if (!change.hasField(Field.title)) { + throw new RuntimeException("Must specify a title!"); + } + + change.watch(change.author); + + if (ticketId <= 0L) { + ticketId = assignNewId(repository); + } + + change.setField(Field.status, Status.New); + + boolean success = commitChangeImpl(repository, ticketId, change); + if (success) { + TicketModel ticket = getTicket(repository, ticketId); + indexer.index(ticket); + return ticket; + } + return null; + } + + /** + * Updates a ticket. + * + * @param repository + * @param ticketId + * @param change + * @return the ticket model if successful + */ + public final TicketModel updateTicket(RepositoryModel repository, long ticketId, Change change) { + if (change == null) { + throw new RuntimeException("change can not be null!"); + } + + if (StringUtils.isEmpty(change.author)) { + throw new RuntimeException("must specify a change author!"); + } + + TicketKey key = new TicketKey(repository, ticketId); + ticketsCache.invalidate(key); + + boolean success = commitChangeImpl(repository, ticketId, change); + if (success) { + TicketModel ticket = getTicket(repository, ticketId); + ticketsCache.put(key, ticket); + indexer.index(ticket); + return ticket; + } + return null; + } + + /** + * Deletes all tickets in every repository. + * + * @return true if successful + */ + public boolean deleteAll() { + List<String> repositories = repositoryManager.getRepositoryList(); + BitSet bitset = new BitSet(repositories.size()); + for (int i = 0; i < repositories.size(); i++) { + String name = repositories.get(i); + RepositoryModel repository = repositoryManager.getRepositoryModel(name); + boolean success = deleteAll(repository); + bitset.set(i, success); + } + boolean success = bitset.cardinality() == repositories.size(); + if (success) { + indexer.deleteAll(); + resetCaches(); + } + return success; + } + + /** + * Deletes all tickets in the specified repository. + * @param repository + * @return true if succesful + */ + public boolean deleteAll(RepositoryModel repository) { + boolean success = deleteAllImpl(repository); + if (success) { + resetCaches(repository); + indexer.deleteAll(repository); + } + return success; + } + + protected abstract boolean deleteAllImpl(RepositoryModel repository); + + /** + * Handles repository renames. + * + * @param oldRepositoryName + * @param newRepositoryName + * @return true if successful + */ + public boolean rename(RepositoryModel oldRepository, RepositoryModel newRepository) { + if (renameImpl(oldRepository, newRepository)) { + resetCaches(oldRepository); + indexer.deleteAll(oldRepository); + reindex(newRepository); + return true; + } + return false; + } + + protected abstract boolean renameImpl(RepositoryModel oldRepository, RepositoryModel newRepository); + + /** + * Deletes a ticket. + * + * @param repository + * @param ticketId + * @param deletedBy + * @return true if successful + */ + public boolean deleteTicket(RepositoryModel repository, long ticketId, String deletedBy) { + TicketModel ticket = getTicket(repository, ticketId); + boolean success = deleteTicketImpl(repository, ticket, deletedBy); + if (success) { + ticketsCache.invalidate(new TicketKey(repository, ticketId)); + indexer.delete(ticket); + return true; + } + return false; + } + + /** + * Deletes a ticket. + * + * @param repository + * @param ticket + * @param deletedBy + * @return true if successful + */ + protected abstract boolean deleteTicketImpl(RepositoryModel repository, TicketModel ticket, String deletedBy); + + + /** + * Updates the text of an ticket comment. + * + * @param ticket + * @param commentId + * the id of the comment to revise + * @param updatedBy + * the author of the updated comment + * @param comment + * the revised comment + * @return the revised ticket if the change was successful + */ + public final TicketModel updateComment(TicketModel ticket, String commentId, + String updatedBy, String comment) { + Change revision = new Change(updatedBy); + revision.comment(comment); + revision.comment.id = commentId; + RepositoryModel repository = repositoryManager.getRepositoryModel(ticket.repository); + TicketModel revisedTicket = updateTicket(repository, ticket.number, revision); + return revisedTicket; + } + + /** + * Deletes a comment from a ticket. + * + * @param ticket + * @param commentId + * the id of the comment to delete + * @param deletedBy + * the user deleting the comment + * @return the revised ticket if the deletion was successful + */ + public final TicketModel deleteComment(TicketModel ticket, String commentId, String deletedBy) { + Change deletion = new Change(deletedBy); + deletion.comment(""); + deletion.comment.id = commentId; + deletion.comment.deleted = true; + RepositoryModel repository = repositoryManager.getRepositoryModel(ticket.repository); + TicketModel revisedTicket = updateTicket(repository, ticket.number, deletion); + return revisedTicket; + } + + /** + * Commit a ticket change to the repository. + * + * @param repository + * @param ticketId + * @param change + * @return true, if the change was committed + */ + protected abstract boolean commitChangeImpl(RepositoryModel repository, long ticketId, Change change); + + + /** + * Searches for the specified text. This will use the indexer, if available, + * or will fall back to brute-force retrieval of all tickets and string + * matching. + * + * @param repository + * @param text + * @param page + * @param pageSize + * @return a list of matching tickets + */ + public List<QueryResult> searchFor(RepositoryModel repository, String text, int page, int pageSize) { + return indexer.searchFor(repository, text, page, pageSize); + } + + /** + * Queries the index for the matching tickets. + * + * @param query + * @param page + * @param pageSize + * @param sortBy + * @param descending + * @return a list of matching tickets or an empty list + */ + public List<QueryResult> queryFor(String query, int page, int pageSize, String sortBy, boolean descending) { + return indexer.queryFor(query, page, pageSize, sortBy, descending); + } + + /** + * Destroys an existing index and reindexes all tickets. + * This operation may be expensive and time-consuming. + */ + public void reindex() { + long start = System.nanoTime(); + indexer.deleteAll(); + for (String name : repositoryManager.getRepositoryList()) { + RepositoryModel repository = repositoryManager.getRepositoryModel(name); + try { + List<TicketModel> tickets = getTickets(repository); + if (!tickets.isEmpty()) { + log.info("reindexing {} tickets from {} ...", tickets.size(), repository); + indexer.index(tickets); + System.gc(); + } + } catch (Exception e) { + log.error("failed to reindex {}", repository.name); + log.error(null, e); + } + } + long end = System.nanoTime(); + long secs = TimeUnit.NANOSECONDS.toMillis(end - start); + log.info("reindexing completed in {} msecs.", secs); + } + + /** + * Destroys any existing index and reindexes all tickets. + * This operation may be expensive and time-consuming. + */ + public void reindex(RepositoryModel repository) { + long start = System.nanoTime(); + List<TicketModel> tickets = getTickets(repository); + indexer.index(tickets); + log.info("reindexing {} tickets from {} ...", tickets.size(), repository); + long end = System.nanoTime(); + long secs = TimeUnit.NANOSECONDS.toMillis(end - start); + log.info("reindexing completed in {} msecs.", secs); + } + + /** + * Synchronously executes the runnable. This is used for special processing + * of ticket updates, namely merging from the web ui. + * + * @param runnable + */ + public synchronized void exec(Runnable runnable) { + runnable.run(); + } +} diff --git a/src/main/java/com/gitblit/tickets/NullTicketService.java b/src/main/java/com/gitblit/tickets/NullTicketService.java new file mode 100644 index 00000000..cc893025 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/NullTicketService.java @@ -0,0 +1,129 @@ +/* + * Copyright 2014 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.util.Collections; +import java.util.List; + +import javax.inject.Inject; + +import com.gitblit.manager.INotificationManager; +import com.gitblit.manager.IRepositoryManager; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.manager.IUserManager; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Attachment; +import com.gitblit.models.TicketModel.Change; + +/** + * Implementation of a ticket service that rejects everything. + * + * @author James Moger + * + */ +public class NullTicketService extends ITicketService { + + @Inject + public NullTicketService( + IRuntimeManager runtimeManager, + INotificationManager notificationManager, + IUserManager userManager, + IRepositoryManager repositoryManager) { + + super(runtimeManager, + notificationManager, + userManager, + repositoryManager); + } + + @Override + public boolean isReady() { + return false; + } + + @Override + public NullTicketService start() { + return this; + } + + @Override + protected void resetCachesImpl() { + } + + @Override + protected void resetCachesImpl(RepositoryModel repository) { + } + + @Override + protected void close() { + } + + @Override + public boolean hasTicket(RepositoryModel repository, long ticketId) { + return false; + } + + @Override + public synchronized long assignNewId(RepositoryModel repository) { + return 0L; + } + + @Override + public List<TicketModel> getTickets(RepositoryModel repository, TicketFilter filter) { + return Collections.emptyList(); + } + + @Override + protected TicketModel getTicketImpl(RepositoryModel repository, long ticketId) { + return null; + } + + @Override + public boolean supportsAttachments() { + return false; + } + + @Override + public Attachment getAttachment(RepositoryModel repository, long ticketId, String filename) { + return null; + } + + @Override + protected synchronized boolean deleteTicketImpl(RepositoryModel repository, TicketModel ticket, String deletedBy) { + return false; + } + + @Override + protected synchronized boolean commitChangeImpl(RepositoryModel repository, long ticketId, Change change) { + return false; + } + + @Override + protected boolean deleteAllImpl(RepositoryModel repository) { + return false; + } + + @Override + protected boolean renameImpl(RepositoryModel oldRepository, RepositoryModel newRepository) { + return false; + } + + @Override + public String toString() { + return getClass().getSimpleName(); + } +} diff --git a/src/main/java/com/gitblit/tickets/QueryBuilder.java b/src/main/java/com/gitblit/tickets/QueryBuilder.java new file mode 100644 index 00000000..17aeb988 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/QueryBuilder.java @@ -0,0 +1,222 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import com.gitblit.utils.StringUtils; + +/** + * A Lucene query builder. + * + * @author James Moger + * + */ +public class QueryBuilder { + + private final QueryBuilder parent; + private String q; + private transient StringBuilder sb; + private int opCount; + + public static QueryBuilder q(String kernel) { + return new QueryBuilder(kernel); + } + + private QueryBuilder(QueryBuilder parent) { + this.sb = new StringBuilder(); + this.parent = parent; + } + + public QueryBuilder() { + this(""); + } + + public QueryBuilder(String query) { + this.sb = new StringBuilder(query == null ? "" : query); + this.parent = null; + } + + public boolean containsField(String field) { + return sb.toString().contains(field + ":"); + } + + /** + * Creates a new AND subquery. Make sure to call endSubquery to + * get return *this* query. + * + * e.g. field:something AND (subquery) + * + * @return a subquery + */ + public QueryBuilder andSubquery() { + sb.append(" AND ("); + return new QueryBuilder(this); + } + + /** + * Creates a new OR subquery. Make sure to call endSubquery to + * get return *this* query. + * + * e.g. field:something OR (subquery) + * + * @return a subquery + */ + public QueryBuilder orSubquery() { + sb.append(" OR ("); + return new QueryBuilder(this); + } + + /** + * Ends a subquery and returns the parent query. + * + * @return the parent query + */ + public QueryBuilder endSubquery() { + this.q = sb.toString().trim(); + if (q.length() > 0) { + parent.sb.append(q).append(')'); + } + return parent; + } + + /** + * Append an OR condition. + * + * @param condition + * @return + */ + public QueryBuilder or(String condition) { + return op(condition, " OR "); + } + + /** + * Append an AND condition. + * + * @param condition + * @return + */ + public QueryBuilder and(String condition) { + return op(condition, " AND "); + } + + /** + * Append an AND NOT condition. + * + * @param condition + * @return + */ + public QueryBuilder andNot(String condition) { + return op(condition, " AND NOT "); + } + + /** + * Nest this query as a subquery. + * + * e.g. field:something AND field2:something else + * ==> (field:something AND field2:something else) + * + * @return this query nested as a subquery + */ + public QueryBuilder toSubquery() { + if (opCount > 1) { + sb.insert(0, '(').append(')'); + } + return this; + } + + /** + * Nest this query as an AND subquery of the condition + * + * @param condition + * @return the query nested as an AND subquery of the specified condition + */ + public QueryBuilder subqueryOf(String condition) { + if (!StringUtils.isEmpty(condition)) { + toSubquery().and(condition); + } + return this; + } + + /** + * Removes a condition from the query. + * + * @param condition + * @return the query + */ + public QueryBuilder remove(String condition) { + int start = sb.indexOf(condition); + if (start == 0) { + // strip first condition + sb.replace(0, condition.length(), ""); + } else if (start > 1) { + // locate condition in query + int space1 = sb.lastIndexOf(" ", start - 1); + int space0 = sb.lastIndexOf(" ", space1 - 1); + if (space0 > -1 && space1 > -1) { + String conjunction = sb.substring(space0, space1).trim(); + if ("OR".equals(conjunction) || "AND".equals(conjunction)) { + // remove the conjunction + sb.replace(space0, start + condition.length(), ""); + } else { + // unknown conjunction + sb.replace(start, start + condition.length(), ""); + } + } else { + sb.replace(start, start + condition.length(), ""); + } + } + return this; + } + + /** + * Generate the return the Lucene query. + * + * @return the generated query + */ + public String build() { + if (parent != null) { + throw new IllegalAccessError("You can not build a subquery! endSubquery() instead!"); + } + this.q = sb.toString().trim(); + + // cleanup paranthesis + while (q.contains("()")) { + q = q.replace("()", ""); + } + if (q.length() > 0) { + if (q.charAt(0) == '(' && q.charAt(q.length() - 1) == ')') { + // query is wrapped by unnecessary paranthesis + q = q.substring(1, q.length() - 1); + } + } + return q; + } + + private QueryBuilder op(String condition, String op) { + opCount++; + if (!StringUtils.isEmpty(condition)) { + if (sb.length() != 0) { + sb.append(op); + } + sb.append(condition); + } + return this; + } + + @Override + public String toString() { + return sb.toString().trim(); + } +}
\ No newline at end of file diff --git a/src/main/java/com/gitblit/tickets/QueryResult.java b/src/main/java/com/gitblit/tickets/QueryResult.java new file mode 100644 index 00000000..9f5d3a55 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/QueryResult.java @@ -0,0 +1,114 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Date; +import java.util.List; + +import com.gitblit.models.TicketModel.Patchset; +import com.gitblit.models.TicketModel.Status; +import com.gitblit.models.TicketModel.Type; +import com.gitblit.utils.StringUtils; + +/** + * Represents the results of a query to the ticket index. + * + * @author James Moger + * + */ +public class QueryResult implements Serializable { + + private static final long serialVersionUID = 1L; + + public String project; + public String repository; + public long number; + public String createdBy; + public Date createdAt; + public String updatedBy; + public Date updatedAt; + public String dependsOn; + public String title; + public String body; + public Status status; + public String responsible; + public String milestone; + public String topic; + public Type type; + public String mergeSha; + public String mergeTo; + public List<String> labels; + public List<String> attachments; + public List<String> participants; + public List<String> watchedby; + public List<String> mentions; + public Patchset patchset; + public int commentsCount; + public int votesCount; + public int approvalsCount; + + public int docId; + public int totalResults; + + public Date getDate() { + return updatedAt == null ? createdAt : updatedAt; + } + + public boolean isProposal() { + return type != null && Type.Proposal == type; + } + + public boolean isMerged() { + return Status.Merged == status && !StringUtils.isEmpty(mergeSha); + } + + public boolean isWatching(String username) { + return watchedby != null && watchedby.contains(username); + } + + public List<String> getLabels() { + List<String> list = new ArrayList<String>(); + if (labels != null) { + list.addAll(labels); + } + if (topic != null) { + list.add(topic); + } + Collections.sort(list); + return list; + } + + @Override + public boolean equals(Object o) { + if (o instanceof QueryResult) { + return hashCode() == o.hashCode(); + } + return false; + } + + @Override + public int hashCode() { + return (repository + number).hashCode(); + } + + @Override + public String toString() { + return repository + "-" + number; + } +} diff --git a/src/main/java/com/gitblit/tickets/RedisTicketService.java b/src/main/java/com/gitblit/tickets/RedisTicketService.java new file mode 100644 index 00000000..5653f698 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/RedisTicketService.java @@ -0,0 +1,534 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Set; + +import javax.inject.Inject; + +import org.apache.commons.pool2.impl.GenericObjectPoolConfig; + +import redis.clients.jedis.Client; +import redis.clients.jedis.Jedis; +import redis.clients.jedis.JedisPool; +import redis.clients.jedis.Protocol; +import redis.clients.jedis.Transaction; +import redis.clients.jedis.exceptions.JedisException; + +import com.gitblit.Keys; +import com.gitblit.manager.INotificationManager; +import com.gitblit.manager.IRepositoryManager; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.manager.IUserManager; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Attachment; +import com.gitblit.models.TicketModel.Change; +import com.gitblit.utils.ArrayUtils; +import com.gitblit.utils.StringUtils; + +/** + * Implementation of a ticket service based on a Redis key-value store. All + * tickets are persisted in the Redis store so it must be configured for + * durability otherwise tickets are lost on a flush or restart. Tickets are + * indexed with Lucene and all queries are executed against the Lucene index. + * + * @author James Moger + * + */ +public class RedisTicketService extends ITicketService { + + private final JedisPool pool; + + private enum KeyType { + journal, ticket, counter + } + + @Inject + public RedisTicketService( + IRuntimeManager runtimeManager, + INotificationManager notificationManager, + IUserManager userManager, + IRepositoryManager repositoryManager) { + + super(runtimeManager, + notificationManager, + userManager, + repositoryManager); + + String redisUrl = settings.getString(Keys.tickets.redis.url, ""); + this.pool = createPool(redisUrl); + } + + @Override + public RedisTicketService start() { + return this; + } + + @Override + protected void resetCachesImpl() { + } + + @Override + protected void resetCachesImpl(RepositoryModel repository) { + } + + @Override + protected void close() { + pool.destroy(); + } + + @Override + public boolean isReady() { + return pool != null; + } + + /** + * Constructs a key for use with a key-value data store. + * + * @param key + * @param repository + * @param id + * @return a key + */ + private String key(RepositoryModel repository, KeyType key, String id) { + StringBuilder sb = new StringBuilder(); + sb.append(repository.name).append(':'); + sb.append(key.name()); + if (!StringUtils.isEmpty(id)) { + sb.append(':'); + sb.append(id); + } + return sb.toString(); + } + + /** + * Constructs a key for use with a key-value data store. + * + * @param key + * @param repository + * @param id + * @return a key + */ + private String key(RepositoryModel repository, KeyType key, long id) { + return key(repository, key, "" + id); + } + + private boolean isNull(String value) { + return value == null || "nil".equals(value); + } + + private String getUrl() { + Jedis jedis = pool.getResource(); + try { + if (jedis != null) { + Client client = jedis.getClient(); + return client.getHost() + ":" + client.getPort() + "/" + client.getDB(); + } + } catch (JedisException e) { + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return null; + } + + /** + * Ensures that we have a ticket for this ticket id. + * + * @param repository + * @param ticketId + * @return true if the ticket exists + */ + @Override + public boolean hasTicket(RepositoryModel repository, long ticketId) { + if (ticketId <= 0L) { + return false; + } + Jedis jedis = pool.getResource(); + if (jedis == null) { + return false; + } + try { + Boolean exists = jedis.exists(key(repository, KeyType.journal, ticketId)); + return exists != null && !exists; + } catch (JedisException e) { + log.error("failed to check hasTicket from Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return false; + } + + /** + * Assigns a new ticket id. + * + * @param repository + * @return a new long ticket id + */ + @Override + public synchronized long assignNewId(RepositoryModel repository) { + Jedis jedis = pool.getResource(); + try { + String key = key(repository, KeyType.counter, null); + String val = jedis.get(key); + if (isNull(val)) { + jedis.set(key, "0"); + } + long ticketNumber = jedis.incr(key); + return ticketNumber; + } catch (JedisException e) { + log.error("failed to assign new ticket id in Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return 0L; + } + + /** + * Returns all the tickets in the repository. Querying tickets from the + * repository requires deserializing all tickets. This is an expensive + * process and not recommended. Tickets should be indexed by Lucene and + * queries should be executed against that index. + * + * @param repository + * @param filter + * optional filter to only return matching results + * @return a list of tickets + */ + @Override + public List<TicketModel> getTickets(RepositoryModel repository, TicketFilter filter) { + Jedis jedis = pool.getResource(); + List<TicketModel> list = new ArrayList<TicketModel>(); + if (jedis == null) { + return list; + } + try { + // Deserialize each journal, build the ticket, and optionally filter + Set<String> keys = jedis.keys(key(repository, KeyType.journal, "*")); + for (String key : keys) { + // {repo}:journal:{id} + String id = key.split(":")[2]; + long ticketId = Long.parseLong(id); + List<Change> changes = getJournal(jedis, repository, ticketId); + if (ArrayUtils.isEmpty(changes)) { + log.warn("Empty journal for {}:{}", repository, ticketId); + continue; + } + TicketModel ticket = TicketModel.buildTicket(changes); + ticket.project = repository.projectPath; + ticket.repository = repository.name; + ticket.number = ticketId; + + // add the ticket, conditionally, to the list + if (filter == null) { + list.add(ticket); + } else { + if (filter.accept(ticket)) { + list.add(ticket); + } + } + } + + // sort the tickets by creation + Collections.sort(list); + } catch (JedisException e) { + log.error("failed to retrieve tickets from Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return list; + } + + /** + * Retrieves the ticket from the repository by first looking-up the changeId + * associated with the ticketId. + * + * @param repository + * @param ticketId + * @return a ticket, if it exists, otherwise null + */ + @Override + protected TicketModel getTicketImpl(RepositoryModel repository, long ticketId) { + Jedis jedis = pool.getResource(); + if (jedis == null) { + return null; + } + + try { + List<Change> changes = getJournal(jedis, repository, ticketId); + if (ArrayUtils.isEmpty(changes)) { + log.warn("Empty journal for {}:{}", repository, ticketId); + return null; + } + TicketModel ticket = TicketModel.buildTicket(changes); + ticket.project = repository.projectPath; + ticket.repository = repository.name; + ticket.number = ticketId; + log.debug("rebuilt ticket {} from Redis @ {}", ticketId, getUrl()); + return ticket; + } catch (JedisException e) { + log.error("failed to retrieve ticket from Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return null; + } + + /** + * Returns the journal for the specified ticket. + * + * @param repository + * @param ticketId + * @return a list of changes + */ + private List<Change> getJournal(Jedis jedis, RepositoryModel repository, long ticketId) throws JedisException { + if (ticketId <= 0L) { + return new ArrayList<Change>(); + } + List<String> entries = jedis.lrange(key(repository, KeyType.journal, ticketId), 0, -1); + if (entries.size() > 0) { + // build a json array from the individual entries + StringBuilder sb = new StringBuilder(); + sb.append("["); + for (String entry : entries) { + sb.append(entry).append(','); + } + sb.setLength(sb.length() - 1); + sb.append(']'); + String journal = sb.toString(); + + return TicketSerializer.deserializeJournal(journal); + } + return new ArrayList<Change>(); + } + + @Override + public boolean supportsAttachments() { + return false; + } + + /** + * Retrieves the specified attachment from a ticket. + * + * @param repository + * @param ticketId + * @param filename + * @return an attachment, if found, null otherwise + */ + @Override + public Attachment getAttachment(RepositoryModel repository, long ticketId, String filename) { + return null; + } + + /** + * Deletes a ticket. + * + * @param ticket + * @return true if successful + */ + @Override + protected boolean deleteTicketImpl(RepositoryModel repository, TicketModel ticket, String deletedBy) { + boolean success = false; + if (ticket == null) { + throw new RuntimeException("must specify a ticket!"); + } + + Jedis jedis = pool.getResource(); + if (jedis == null) { + return false; + } + + try { + // atomically remove ticket + Transaction t = jedis.multi(); + t.del(key(repository, KeyType.ticket, ticket.number)); + t.del(key(repository, KeyType.journal, ticket.number)); + t.exec(); + + success = true; + log.debug("deleted ticket {} from Redis @ {}", "" + ticket.number, getUrl()); + } catch (JedisException e) { + log.error("failed to delete ticket from Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + + return success; + } + + /** + * Commit a ticket change to the repository. + * + * @param repository + * @param ticketId + * @param change + * @return true, if the change was committed + */ + @Override + protected boolean commitChangeImpl(RepositoryModel repository, long ticketId, Change change) { + Jedis jedis = pool.getResource(); + if (jedis == null) { + return false; + } + try { + List<Change> changes = getJournal(jedis, repository, ticketId); + changes.add(change); + // build a new effective ticket from the changes + TicketModel ticket = TicketModel.buildTicket(changes); + + String object = TicketSerializer.serialize(ticket); + String journal = TicketSerializer.serialize(change); + + // atomically store ticket + Transaction t = jedis.multi(); + t.set(key(repository, KeyType.ticket, ticketId), object); + t.rpush(key(repository, KeyType.journal, ticketId), journal); + t.exec(); + + log.debug("updated ticket {} in Redis @ {}", "" + ticketId, getUrl()); + return true; + } catch (JedisException e) { + log.error("failed to update ticket cache in Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return false; + } + + /** + * Deletes all Tickets for the rpeository from the Redis key-value store. + * + */ + @Override + protected boolean deleteAllImpl(RepositoryModel repository) { + Jedis jedis = pool.getResource(); + if (jedis == null) { + return false; + } + + boolean success = false; + try { + Set<String> keys = jedis.keys(repository.name + ":*"); + if (keys.size() > 0) { + Transaction t = jedis.multi(); + t.del(keys.toArray(new String[keys.size()])); + t.exec(); + } + success = true; + } catch (JedisException e) { + log.error("failed to delete all tickets in Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return success; + } + + @Override + protected boolean renameImpl(RepositoryModel oldRepository, RepositoryModel newRepository) { + Jedis jedis = pool.getResource(); + if (jedis == null) { + return false; + } + + boolean success = false; + try { + Set<String> oldKeys = jedis.keys(oldRepository.name + ":*"); + Transaction t = jedis.multi(); + for (String oldKey : oldKeys) { + String newKey = newRepository.name + oldKey.substring(oldKey.indexOf(':')); + t.rename(oldKey, newKey); + } + t.exec(); + success = true; + } catch (JedisException e) { + log.error("failed to rename tickets in Redis @ " + getUrl(), e); + pool.returnBrokenResource(jedis); + jedis = null; + } finally { + if (jedis != null) { + pool.returnResource(jedis); + } + } + return success; + } + + private JedisPool createPool(String url) { + JedisPool pool = null; + if (!StringUtils.isEmpty(url)) { + try { + URI uri = URI.create(url); + if (uri.getScheme() != null && uri.getScheme().equalsIgnoreCase("redis")) { + int database = Protocol.DEFAULT_DATABASE; + String password = null; + if (uri.getUserInfo() != null) { + password = uri.getUserInfo().split(":", 2)[1]; + } + if (uri.getPath().indexOf('/') > -1) { + database = Integer.parseInt(uri.getPath().split("/", 2)[1]); + } + pool = new JedisPool(new GenericObjectPoolConfig(), uri.getHost(), uri.getPort(), Protocol.DEFAULT_TIMEOUT, password, database); + } else { + pool = new JedisPool(url); + } + } catch (JedisException e) { + log.error("failed to create a Redis pool!", e); + } + } + return pool; + } + + @Override + public String toString() { + String url = getUrl(); + return getClass().getSimpleName() + " (" + (url == null ? "DISABLED" : url) + ")"; + } +} diff --git a/src/main/java/com/gitblit/tickets/TicketIndexer.java b/src/main/java/com/gitblit/tickets/TicketIndexer.java new file mode 100644 index 00000000..3929a000 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/TicketIndexer.java @@ -0,0 +1,657 @@ +/* + * Copyright 2014 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.File; +import java.io.IOException; +import java.text.MessageFormat; +import java.text.ParseException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Date; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; + +import org.apache.lucene.analysis.standard.StandardAnalyzer; +import org.apache.lucene.document.Document; +import org.apache.lucene.document.Field.Store; +import org.apache.lucene.document.IntField; +import org.apache.lucene.document.LongField; +import org.apache.lucene.document.TextField; +import org.apache.lucene.index.DirectoryReader; +import org.apache.lucene.index.IndexWriter; +import org.apache.lucene.index.IndexWriterConfig; +import org.apache.lucene.index.IndexWriterConfig.OpenMode; +import org.apache.lucene.queryparser.classic.QueryParser; +import org.apache.lucene.search.BooleanClause.Occur; +import org.apache.lucene.search.BooleanQuery; +import org.apache.lucene.search.IndexSearcher; +import org.apache.lucene.search.Query; +import org.apache.lucene.search.ScoreDoc; +import org.apache.lucene.search.Sort; +import org.apache.lucene.search.SortField; +import org.apache.lucene.search.SortField.Type; +import org.apache.lucene.search.TopFieldDocs; +import org.apache.lucene.search.TopScoreDocCollector; +import org.apache.lucene.store.Directory; +import org.apache.lucene.store.FSDirectory; +import org.apache.lucene.util.Version; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.gitblit.Keys; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Attachment; +import com.gitblit.models.TicketModel.Patchset; +import com.gitblit.models.TicketModel.Status; +import com.gitblit.utils.FileUtils; +import com.gitblit.utils.StringUtils; + +/** + * Indexes tickets in a Lucene database. + * + * @author James Moger + * + */ +public class TicketIndexer { + + /** + * Fields in the Lucene index + */ + public static enum Lucene { + + rid(Type.STRING), + did(Type.STRING), + project(Type.STRING), + repository(Type.STRING), + number(Type.LONG), + title(Type.STRING), + body(Type.STRING), + topic(Type.STRING), + created(Type.LONG), + createdby(Type.STRING), + updated(Type.LONG), + updatedby(Type.STRING), + responsible(Type.STRING), + milestone(Type.STRING), + status(Type.STRING), + type(Type.STRING), + labels(Type.STRING), + participants(Type.STRING), + watchedby(Type.STRING), + mentions(Type.STRING), + attachments(Type.INT), + content(Type.STRING), + patchset(Type.STRING), + comments(Type.INT), + mergesha(Type.STRING), + mergeto(Type.STRING), + patchsets(Type.INT), + votes(Type.INT); + + final Type fieldType; + + Lucene(Type fieldType) { + this.fieldType = fieldType; + } + + public String colon() { + return name() + ":"; + } + + public String matches(String value) { + if (StringUtils.isEmpty(value)) { + return ""; + } + boolean not = value.charAt(0) == '!'; + if (not) { + return "!" + name() + ":" + escape(value.substring(1)); + } + return name() + ":" + escape(value); + } + + public String doesNotMatch(String value) { + if (StringUtils.isEmpty(value)) { + return ""; + } + return "NOT " + name() + ":" + escape(value); + } + + public String isNotNull() { + return matches("[* TO *]"); + } + + public SortField asSortField(boolean descending) { + return new SortField(name(), fieldType, descending); + } + + private String escape(String value) { + if (value.charAt(0) != '"') { + if (value.indexOf('/') > -1) { + return "\"" + value + "\""; + } + } + return value; + } + + public static Lucene fromString(String value) { + for (Lucene field : values()) { + if (field.name().equalsIgnoreCase(value)) { + return field; + } + } + return created; + } + } + + private final Logger log = LoggerFactory.getLogger(getClass()); + + private final Version luceneVersion = Version.LUCENE_46; + + private final File luceneDir; + + private IndexWriter writer; + + private IndexSearcher searcher; + + public TicketIndexer(IRuntimeManager runtimeManager) { + this.luceneDir = runtimeManager.getFileOrFolder(Keys.tickets.indexFolder, "${baseFolder}/tickets/lucene"); + } + + /** + * Close all writers and searchers used by the ticket indexer. + */ + public void close() { + closeSearcher(); + closeWriter(); + } + + /** + * Deletes the entire ticket index for all repositories. + */ + public void deleteAll() { + close(); + FileUtils.delete(luceneDir); + } + + /** + * Deletes all tickets for the the repository from the index. + */ + public boolean deleteAll(RepositoryModel repository) { + try { + IndexWriter writer = getWriter(); + StandardAnalyzer analyzer = new StandardAnalyzer(luceneVersion); + QueryParser qp = new QueryParser(luceneVersion, Lucene.rid.name(), analyzer); + BooleanQuery query = new BooleanQuery(); + query.add(qp.parse(repository.getRID()), Occur.MUST); + + int numDocsBefore = writer.numDocs(); + writer.deleteDocuments(query); + writer.commit(); + closeSearcher(); + int numDocsAfter = writer.numDocs(); + if (numDocsBefore == numDocsAfter) { + log.debug(MessageFormat.format("no records found to delete in {0}", repository)); + return false; + } else { + log.debug(MessageFormat.format("deleted {0} records in {1}", numDocsBefore - numDocsAfter, repository)); + return true; + } + } catch (Exception e) { + log.error("error", e); + } + return false; + } + + /** + * Bulk Add/Update tickets in the Lucene index + * + * @param tickets + */ + public void index(List<TicketModel> tickets) { + try { + IndexWriter writer = getWriter(); + for (TicketModel ticket : tickets) { + Document doc = ticketToDoc(ticket); + writer.addDocument(doc); + } + writer.commit(); + closeSearcher(); + } catch (Exception e) { + log.error("error", e); + } + } + + /** + * Add/Update a ticket in the Lucene index + * + * @param ticket + */ + public void index(TicketModel ticket) { + try { + IndexWriter writer = getWriter(); + delete(ticket.repository, ticket.number, writer); + Document doc = ticketToDoc(ticket); + writer.addDocument(doc); + writer.commit(); + closeSearcher(); + } catch (Exception e) { + log.error("error", e); + } + } + + /** + * Delete a ticket from the Lucene index. + * + * @param ticket + * @throws Exception + * @return true, if deleted, false if no record was deleted + */ + public boolean delete(TicketModel ticket) { + try { + IndexWriter writer = getWriter(); + return delete(ticket.repository, ticket.number, writer); + } catch (Exception e) { + log.error("Failed to delete ticket " + ticket.number, e); + } + return false; + } + + /** + * Delete a ticket from the Lucene index. + * + * @param repository + * @param ticketId + * @throws Exception + * @return true, if deleted, false if no record was deleted + */ + private boolean delete(String repository, long ticketId, IndexWriter writer) throws Exception { + StandardAnalyzer analyzer = new StandardAnalyzer(luceneVersion); + QueryParser qp = new QueryParser(luceneVersion, Lucene.did.name(), analyzer); + BooleanQuery query = new BooleanQuery(); + query.add(qp.parse(StringUtils.getSHA1(repository + ticketId)), Occur.MUST); + + int numDocsBefore = writer.numDocs(); + writer.deleteDocuments(query); + writer.commit(); + closeSearcher(); + int numDocsAfter = writer.numDocs(); + if (numDocsBefore == numDocsAfter) { + log.debug(MessageFormat.format("no records found to delete in {0}", repository)); + return false; + } else { + log.debug(MessageFormat.format("deleted {0} records in {1}", numDocsBefore - numDocsAfter, repository)); + return true; + } + } + + /** + * Returns true if the repository has tickets in the index. + * + * @param repository + * @return true if there are indexed tickets + */ + public boolean hasTickets(RepositoryModel repository) { + return !queryFor(Lucene.rid.matches(repository.getRID()), 1, 0, null, true).isEmpty(); + } + + /** + * Search for tickets matching the query. The returned tickets are + * shadows of the real ticket, but suitable for a results list. + * + * @param repository + * @param text + * @param page + * @param pageSize + * @return search results + */ + public List<QueryResult> searchFor(RepositoryModel repository, String text, int page, int pageSize) { + if (StringUtils.isEmpty(text)) { + return Collections.emptyList(); + } + Set<QueryResult> results = new LinkedHashSet<QueryResult>(); + StandardAnalyzer analyzer = new StandardAnalyzer(luceneVersion); + try { + // search the title, description and content + BooleanQuery query = new BooleanQuery(); + QueryParser qp; + + qp = new QueryParser(luceneVersion, Lucene.title.name(), analyzer); + qp.setAllowLeadingWildcard(true); + query.add(qp.parse(text), Occur.SHOULD); + + qp = new QueryParser(luceneVersion, Lucene.body.name(), analyzer); + qp.setAllowLeadingWildcard(true); + query.add(qp.parse(text), Occur.SHOULD); + + qp = new QueryParser(luceneVersion, Lucene.content.name(), analyzer); + qp.setAllowLeadingWildcard(true); + query.add(qp.parse(text), Occur.SHOULD); + + IndexSearcher searcher = getSearcher(); + Query rewrittenQuery = searcher.rewrite(query); + + log.debug(rewrittenQuery.toString()); + + TopScoreDocCollector collector = TopScoreDocCollector.create(5000, true); + searcher.search(rewrittenQuery, collector); + int offset = Math.max(0, (page - 1) * pageSize); + ScoreDoc[] hits = collector.topDocs(offset, pageSize).scoreDocs; + for (int i = 0; i < hits.length; i++) { + int docId = hits[i].doc; + Document doc = searcher.doc(docId); + QueryResult result = docToQueryResult(doc); + if (repository != null) { + if (!result.repository.equalsIgnoreCase(repository.name)) { + continue; + } + } + results.add(result); + } + } catch (Exception e) { + log.error(MessageFormat.format("Exception while searching for {0}", text), e); + } + return new ArrayList<QueryResult>(results); + } + + /** + * Search for tickets matching the query. The returned tickets are + * shadows of the real ticket, but suitable for a results list. + * + * @param text + * @param page + * @param pageSize + * @param sortBy + * @param desc + * @return + */ + public List<QueryResult> queryFor(String queryText, int page, int pageSize, String sortBy, boolean desc) { + if (StringUtils.isEmpty(queryText)) { + return Collections.emptyList(); + } + + Set<QueryResult> results = new LinkedHashSet<QueryResult>(); + StandardAnalyzer analyzer = new StandardAnalyzer(luceneVersion); + try { + QueryParser qp = new QueryParser(luceneVersion, Lucene.content.name(), analyzer); + Query query = qp.parse(queryText); + + IndexSearcher searcher = getSearcher(); + Query rewrittenQuery = searcher.rewrite(query); + + log.debug(rewrittenQuery.toString()); + + Sort sort; + if (sortBy == null) { + sort = new Sort(Lucene.created.asSortField(desc)); + } else { + sort = new Sort(Lucene.fromString(sortBy).asSortField(desc)); + } + int maxSize = 5000; + TopFieldDocs docs = searcher.search(rewrittenQuery, null, maxSize, sort, false, false); + int size = (pageSize <= 0) ? maxSize : pageSize; + int offset = Math.max(0, (page - 1) * size); + ScoreDoc[] hits = subset(docs.scoreDocs, offset, size); + for (int i = 0; i < hits.length; i++) { + int docId = hits[i].doc; + Document doc = searcher.doc(docId); + QueryResult result = docToQueryResult(doc); + result.docId = docId; + result.totalResults = docs.totalHits; + results.add(result); + } + } catch (Exception e) { + log.error(MessageFormat.format("Exception while searching for {0}", queryText), e); + } + return new ArrayList<QueryResult>(results); + } + + private ScoreDoc [] subset(ScoreDoc [] docs, int offset, int size) { + if (docs.length >= (offset + size)) { + ScoreDoc [] set = new ScoreDoc[size]; + System.arraycopy(docs, offset, set, 0, set.length); + return set; + } else if (docs.length >= offset) { + ScoreDoc [] set = new ScoreDoc[docs.length - offset]; + System.arraycopy(docs, offset, set, 0, set.length); + return set; + } else { + return new ScoreDoc[0]; + } + } + + private IndexWriter getWriter() throws IOException { + if (writer == null) { + Directory directory = FSDirectory.open(luceneDir); + + if (!luceneDir.exists()) { + luceneDir.mkdirs(); + } + + StandardAnalyzer analyzer = new StandardAnalyzer(luceneVersion); + IndexWriterConfig config = new IndexWriterConfig(luceneVersion, analyzer); + config.setOpenMode(OpenMode.CREATE_OR_APPEND); + writer = new IndexWriter(directory, config); + } + return writer; + } + + private synchronized void closeWriter() { + try { + if (writer != null) { + writer.close(); + } + } catch (Exception e) { + log.error("failed to close writer!", e); + } finally { + writer = null; + } + } + + private IndexSearcher getSearcher() throws IOException { + if (searcher == null) { + searcher = new IndexSearcher(DirectoryReader.open(getWriter(), true)); + } + return searcher; + } + + private synchronized void closeSearcher() { + try { + if (searcher != null) { + searcher.getIndexReader().close(); + } + } catch (Exception e) { + log.error("failed to close searcher!", e); + } finally { + searcher = null; + } + } + + /** + * Creates a Lucene document from a ticket. + * + * @param ticket + * @return a Lucene document + */ + private Document ticketToDoc(TicketModel ticket) { + Document doc = new Document(); + // repository and document ids for Lucene querying + toDocField(doc, Lucene.rid, StringUtils.getSHA1(ticket.repository)); + toDocField(doc, Lucene.did, StringUtils.getSHA1(ticket.repository + ticket.number)); + + toDocField(doc, Lucene.project, ticket.project); + toDocField(doc, Lucene.repository, ticket.repository); + toDocField(doc, Lucene.number, ticket.number); + toDocField(doc, Lucene.title, ticket.title); + toDocField(doc, Lucene.body, ticket.body); + toDocField(doc, Lucene.created, ticket.created); + toDocField(doc, Lucene.createdby, ticket.createdBy); + toDocField(doc, Lucene.updated, ticket.updated); + toDocField(doc, Lucene.updatedby, ticket.updatedBy); + toDocField(doc, Lucene.responsible, ticket.responsible); + toDocField(doc, Lucene.milestone, ticket.milestone); + toDocField(doc, Lucene.topic, ticket.topic); + toDocField(doc, Lucene.status, ticket.status.name()); + toDocField(doc, Lucene.comments, ticket.getComments().size()); + toDocField(doc, Lucene.type, ticket.type == null ? null : ticket.type.name()); + toDocField(doc, Lucene.mergesha, ticket.mergeSha); + toDocField(doc, Lucene.mergeto, ticket.mergeTo); + toDocField(doc, Lucene.labels, StringUtils.flattenStrings(ticket.getLabels(), ";").toLowerCase()); + toDocField(doc, Lucene.participants, StringUtils.flattenStrings(ticket.getParticipants(), ";").toLowerCase()); + toDocField(doc, Lucene.watchedby, StringUtils.flattenStrings(ticket.getWatchers(), ";").toLowerCase()); + toDocField(doc, Lucene.mentions, StringUtils.flattenStrings(ticket.getMentions(), ";").toLowerCase()); + toDocField(doc, Lucene.votes, ticket.getVoters().size()); + + List<String> attachments = new ArrayList<String>(); + for (Attachment attachment : ticket.getAttachments()) { + attachments.add(attachment.name.toLowerCase()); + } + toDocField(doc, Lucene.attachments, StringUtils.flattenStrings(attachments, ";")); + + List<Patchset> patches = ticket.getPatchsets(); + if (!patches.isEmpty()) { + toDocField(doc, Lucene.patchsets, patches.size()); + Patchset patchset = patches.get(patches.size() - 1); + String flat = + patchset.number + ":" + + patchset.rev + ":" + + patchset.tip + ":" + + patchset.base + ":" + + patchset.commits; + doc.add(new org.apache.lucene.document.Field(Lucene.patchset.name(), flat, TextField.TYPE_STORED)); + } + + doc.add(new TextField(Lucene.content.name(), ticket.toIndexableString(), Store.NO)); + + return doc; + } + + private void toDocField(Document doc, Lucene lucene, Date value) { + if (value == null) { + return; + } + doc.add(new LongField(lucene.name(), value.getTime(), Store.YES)); + } + + private void toDocField(Document doc, Lucene lucene, long value) { + doc.add(new LongField(lucene.name(), value, Store.YES)); + } + + private void toDocField(Document doc, Lucene lucene, int value) { + doc.add(new IntField(lucene.name(), value, Store.YES)); + } + + private void toDocField(Document doc, Lucene lucene, String value) { + if (StringUtils.isEmpty(value)) { + return; + } + doc.add(new org.apache.lucene.document.Field(lucene.name(), value, TextField.TYPE_STORED)); + } + + /** + * Creates a query result from the Lucene document. This result is + * not a high-fidelity representation of the real ticket, but it is + * suitable for display in a table of search results. + * + * @param doc + * @return a query result + * @throws ParseException + */ + private QueryResult docToQueryResult(Document doc) throws ParseException { + QueryResult result = new QueryResult(); + result.project = unpackString(doc, Lucene.project); + result.repository = unpackString(doc, Lucene.repository); + result.number = unpackLong(doc, Lucene.number); + result.createdBy = unpackString(doc, Lucene.createdby); + result.createdAt = unpackDate(doc, Lucene.created); + result.updatedBy = unpackString(doc, Lucene.updatedby); + result.updatedAt = unpackDate(doc, Lucene.updated); + result.title = unpackString(doc, Lucene.title); + result.body = unpackString(doc, Lucene.body); + result.status = Status.fromObject(unpackString(doc, Lucene.status), Status.New); + result.responsible = unpackString(doc, Lucene.responsible); + result.milestone = unpackString(doc, Lucene.milestone); + result.topic = unpackString(doc, Lucene.topic); + result.type = TicketModel.Type.fromObject(unpackString(doc, Lucene.type), TicketModel.Type.defaultType); + result.mergeSha = unpackString(doc, Lucene.mergesha); + result.mergeTo = unpackString(doc, Lucene.mergeto); + result.commentsCount = unpackInt(doc, Lucene.comments); + result.votesCount = unpackInt(doc, Lucene.votes); + result.attachments = unpackStrings(doc, Lucene.attachments); + result.labels = unpackStrings(doc, Lucene.labels); + result.participants = unpackStrings(doc, Lucene.participants); + result.watchedby = unpackStrings(doc, Lucene.watchedby); + result.mentions = unpackStrings(doc, Lucene.mentions); + + if (!StringUtils.isEmpty(doc.get(Lucene.patchset.name()))) { + // unpack most recent patchset + String [] values = doc.get(Lucene.patchset.name()).split(":", 5); + + Patchset patchset = new Patchset(); + patchset.number = Integer.parseInt(values[0]); + patchset.rev = Integer.parseInt(values[1]); + patchset.tip = values[2]; + patchset.base = values[3]; + patchset.commits = Integer.parseInt(values[4]); + + result.patchset = patchset; + } + + return result; + } + + private String unpackString(Document doc, Lucene lucene) { + return doc.get(lucene.name()); + } + + private List<String> unpackStrings(Document doc, Lucene lucene) { + if (!StringUtils.isEmpty(doc.get(lucene.name()))) { + return StringUtils.getStringsFromValue(doc.get(lucene.name()), ";"); + } + return null; + } + + private Date unpackDate(Document doc, Lucene lucene) { + String val = doc.get(lucene.name()); + if (!StringUtils.isEmpty(val)) { + long time = Long.parseLong(val); + Date date = new Date(time); + return date; + } + return null; + } + + private long unpackLong(Document doc, Lucene lucene) { + String val = doc.get(lucene.name()); + if (StringUtils.isEmpty(val)) { + return 0; + } + long l = Long.parseLong(val); + return l; + } + + private int unpackInt(Document doc, Lucene lucene) { + String val = doc.get(lucene.name()); + if (StringUtils.isEmpty(val)) { + return 0; + } + int i = Integer.parseInt(val); + return i; + } +}
\ No newline at end of file diff --git a/src/main/java/com/gitblit/tickets/TicketLabel.java b/src/main/java/com/gitblit/tickets/TicketLabel.java new file mode 100644 index 00000000..686ce88b --- /dev/null +++ b/src/main/java/com/gitblit/tickets/TicketLabel.java @@ -0,0 +1,77 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.Serializable; +import java.util.List; + +import com.gitblit.utils.StringUtils; + +/** + * A ticket label. + * + * @author James Moger + * + */ +public class TicketLabel implements Serializable { + + private static final long serialVersionUID = 1L; + + public final String name; + + public String color; + + public List<QueryResult> tickets; + + + public TicketLabel(String name) { + this.name = name; + this.color = StringUtils.getColor(name); + } + + public int getTotalTickets() { + return tickets == null ? 0 : tickets.size(); + } + + public int getOpenTickets() { + int cnt = 0; + if (tickets != null) { + for (QueryResult ticket : tickets) { + if (!ticket.status.isClosed()) { + cnt++; + } + } + } + return cnt; + } + + public int getClosedTickets() { + int cnt = 0; + if (tickets != null) { + for (QueryResult ticket : tickets) { + if (ticket.status.isClosed()) { + cnt++; + } + } + } + return cnt; + } + + @Override + public String toString() { + return name; + } +} diff --git a/src/main/java/com/gitblit/tickets/TicketMilestone.java b/src/main/java/com/gitblit/tickets/TicketMilestone.java new file mode 100644 index 00000000..c6b4fcca --- /dev/null +++ b/src/main/java/com/gitblit/tickets/TicketMilestone.java @@ -0,0 +1,53 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.util.Date; + +import com.gitblit.models.TicketModel.Status; + +/** + * A ticket milestone. + * + * @author James Moger + * + */ +public class TicketMilestone extends TicketLabel { + + private static final long serialVersionUID = 1L; + + public Status status; + + public Date due; + + public TicketMilestone(String name) { + super(name); + status = Status.Open; + } + + public int getProgress() { + int total = getTotalTickets(); + if (total == 0) { + return 0; + } + return (int) (((getClosedTickets() * 1f) / (total * 1f)) * 100); + } + + @Override + public String toString() { + return name; + } +} diff --git a/src/main/java/com/gitblit/tickets/TicketNotifier.java b/src/main/java/com/gitblit/tickets/TicketNotifier.java new file mode 100644 index 00000000..b4c3baeb --- /dev/null +++ b/src/main/java/com/gitblit/tickets/TicketNotifier.java @@ -0,0 +1,617 @@ +/* + * Copyright 2014 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.IOException; +import java.io.InputStream; +import java.text.DateFormat; +import java.text.MessageFormat; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TreeMap; +import java.util.TreeSet; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apache.commons.io.IOUtils; +import org.apache.log4j.Logger; +import org.eclipse.jgit.diff.DiffEntry.ChangeType; +import org.eclipse.jgit.lib.Repository; +import org.eclipse.jgit.revwalk.RevCommit; +import org.slf4j.LoggerFactory; + +import com.gitblit.Constants; +import com.gitblit.IStoredSettings; +import com.gitblit.Keys; +import com.gitblit.git.PatchsetCommand; +import com.gitblit.manager.INotificationManager; +import com.gitblit.manager.IRepositoryManager; +import com.gitblit.manager.IRuntimeManager; +import com.gitblit.manager.IUserManager; +import com.gitblit.models.Mailing; +import com.gitblit.models.PathModel.PathChangeModel; +import com.gitblit.models.RepositoryModel; +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Change; +import com.gitblit.models.TicketModel.Field; +import com.gitblit.models.TicketModel.Patchset; +import com.gitblit.models.TicketModel.Review; +import com.gitblit.models.TicketModel.Status; +import com.gitblit.models.UserModel; +import com.gitblit.utils.ArrayUtils; +import com.gitblit.utils.DiffUtils; +import com.gitblit.utils.DiffUtils.DiffStat; +import com.gitblit.utils.JGitUtils; +import com.gitblit.utils.MarkdownUtils; +import com.gitblit.utils.StringUtils; + +/** + * Formats and queues ticket/patch notifications for dispatch to the + * mail executor upon completion of a push or a ticket update. Messages are + * created as Markdown and then transformed to html. + * + * @author James Moger + * + */ +public class TicketNotifier { + + protected final Map<Long, Mailing> queue = new TreeMap<Long, Mailing>(); + + private final String SOFT_BRK = "\n"; + + private final String HARD_BRK = "\n\n"; + + private final String HR = "----\n\n"; + + private final IStoredSettings settings; + + private final INotificationManager notificationManager; + + private final IUserManager userManager; + + private final IRepositoryManager repositoryManager; + + private final ITicketService ticketService; + + private final String addPattern = "<span style=\"color:darkgreen;\">+{0}</span>"; + private final String delPattern = "<span style=\"color:darkred;\">-{0}</span>"; + + public TicketNotifier( + IRuntimeManager runtimeManager, + INotificationManager notificationManager, + IUserManager userManager, + IRepositoryManager repositoryManager, + ITicketService ticketService) { + + this.settings = runtimeManager.getSettings(); + this.notificationManager = notificationManager; + this.userManager = userManager; + this.repositoryManager = repositoryManager; + this.ticketService = ticketService; + } + + public void sendAll() { + for (Mailing mail : queue.values()) { + notificationManager.send(mail); + } + } + + public void sendMailing(TicketModel ticket) { + queueMailing(ticket); + sendAll(); + } + + /** + * Queues an update notification. + * + * @param ticket + * @return a notification object used for testing + */ + public Mailing queueMailing(TicketModel ticket) { + try { + // format notification message + String markdown = formatLastChange(ticket); + + StringBuilder html = new StringBuilder(); + html.append("<head>"); + html.append(readStyle()); + html.append("</head>"); + html.append("<body>"); + html.append(MarkdownUtils.transformGFM(settings, markdown, ticket.repository)); + html.append("</body>"); + + Mailing mailing = Mailing.newHtml(); + mailing.from = getUserModel(ticket.updatedBy == null ? ticket.createdBy : ticket.updatedBy).getDisplayName(); + mailing.subject = getSubject(ticket); + mailing.content = html.toString(); + mailing.id = "ticket." + ticket.number + "." + StringUtils.getSHA1(ticket.repository + ticket.number); + + setRecipients(ticket, mailing); + queue.put(ticket.number, mailing); + + return mailing; + } catch (Exception e) { + Logger.getLogger(getClass()).error("failed to queue mailing for #" + ticket.number, e); + } + return null; + } + + protected String getSubject(TicketModel ticket) { + Change lastChange = ticket.changes.get(ticket.changes.size() - 1); + boolean newTicket = lastChange.isStatusChange() && ticket.changes.size() == 1; + String re = newTicket ? "" : "Re: "; + String subject = MessageFormat.format("{0}[{1}] {2} (#{3,number,0})", + re, StringUtils.stripDotGit(ticket.repository), ticket.title, ticket.number); + return subject; + } + + protected String formatLastChange(TicketModel ticket) { + Change lastChange = ticket.changes.get(ticket.changes.size() - 1); + UserModel user = getUserModel(lastChange.author); + + // define the fields we do NOT want to see in an email notification + Set<TicketModel.Field> fieldExclusions = new HashSet<TicketModel.Field>(); + fieldExclusions.addAll(Arrays.asList(Field.watchers, Field.voters)); + + StringBuilder sb = new StringBuilder(); + boolean newTicket = false; + boolean isFastForward = true; + List<RevCommit> commits = null; + DiffStat diffstat = null; + + String pattern; + if (lastChange.isStatusChange()) { + Status state = lastChange.getStatus(); + switch (state) { + case New: + // new ticket + newTicket = true; + fieldExclusions.add(Field.status); + fieldExclusions.add(Field.title); + fieldExclusions.add(Field.body); + if (lastChange.hasPatchset()) { + pattern = "**{0}** is proposing a change."; + } else { + pattern = "**{0}** created this ticket."; + } + sb.append(MessageFormat.format(pattern, user.getDisplayName())); + break; + default: + // some form of resolved + if (lastChange.hasField(Field.mergeSha)) { + // closed by push (merged patchset) + pattern = "**{0}** closed this ticket by pushing {1} to {2}."; + + // identify patch that closed the ticket + String merged = ticket.mergeSha; + for (Patchset patchset : ticket.getPatchsets()) { + if (patchset.tip.equals(ticket.mergeSha)) { + merged = patchset.toString(); + break; + } + } + sb.append(MessageFormat.format(pattern, user.getDisplayName(), merged, ticket.mergeTo)); + } else { + // workflow status change by user + pattern = "**{0}** changed the status of this ticket to **{1}**."; + sb.append(MessageFormat.format(pattern, user.getDisplayName(), lastChange.getStatus().toString().toUpperCase())); + } + break; + } + sb.append(HARD_BRK); + } else if (lastChange.hasPatchset()) { + // patchset uploaded + Patchset patchset = lastChange.patchset; + String base = ""; + // determine the changed paths + Repository repo = null; + try { + repo = repositoryManager.getRepository(ticket.repository); + if (patchset.isFF() && (patchset.rev > 1)) { + // fast-forward update, just show the new data + isFastForward = true; + Patchset prev = ticket.getPatchset(patchset.number, patchset.rev - 1); + base = prev.tip; + } else { + // proposal OR non-fast-forward update + isFastForward = false; + base = patchset.base; + } + + diffstat = DiffUtils.getDiffStat(repo, base, patchset.tip); + commits = JGitUtils.getRevLog(repo, base, patchset.tip); + } catch (Exception e) { + Logger.getLogger(getClass()).error("failed to get changed paths", e); + } finally { + repo.close(); + } + + // describe the patchset + String compareUrl = ticketService.getCompareUrl(ticket, base, patchset.tip); + if (patchset.isFF()) { + pattern = "**{0}** added {1} {2} to patchset {3}."; + sb.append(MessageFormat.format(pattern, user.getDisplayName(), patchset.added, patchset.added == 1 ? "commit" : "commits", patchset.number)); + } else { + pattern = "**{0}** uploaded patchset {1}. *({2})*"; + sb.append(MessageFormat.format(pattern, user.getDisplayName(), patchset.number, patchset.type.toString().toUpperCase())); + } + sb.append(HARD_BRK); + sb.append(MessageFormat.format("{0} {1}, {2} {3}, <span style=\"color:darkgreen;\">+{4} insertions</span>, <span style=\"color:darkred;\">-{5} deletions</span> from {6}. [compare]({7})", + commits.size(), commits.size() == 1 ? "commit" : "commits", + diffstat.paths.size(), + diffstat.paths.size() == 1 ? "file" : "files", + diffstat.getInsertions(), + diffstat.getDeletions(), + isFastForward ? "previous revision" : "merge base", + compareUrl)); + + // note commit additions on a rebase,if any + switch (lastChange.patchset.type) { + case Rebase: + if (lastChange.patchset.added > 0) { + sb.append(SOFT_BRK); + sb.append(MessageFormat.format("{0} {1} added.", lastChange.patchset.added, lastChange.patchset.added == 1 ? "commit" : "commits")); + } + break; + default: + break; + } + sb.append(HARD_BRK); + } else if (lastChange.hasReview()) { + // review + Review review = lastChange.review; + pattern = "**{0}** has reviewed patchset {1,number,0} revision {2,number,0}."; + sb.append(MessageFormat.format(pattern, user.getDisplayName(), review.patchset, review.rev)); + sb.append(HARD_BRK); + + String d = settings.getString(Keys.web.datestampShortFormat, "yyyy-MM-dd"); + String t = settings.getString(Keys.web.timeFormat, "HH:mm"); + DateFormat df = new SimpleDateFormat(d + " " + t); + List<Change> reviews = ticket.getReviews(ticket.getPatchset(review.patchset, review.rev)); + sb.append("| Date | Reviewer | Score | Description |\n"); + sb.append("| :--- | :------------ | :---: | :----------- |\n"); + for (Change change : reviews) { + String name = change.author; + UserModel u = userManager.getUserModel(change.author); + if (u != null) { + name = u.getDisplayName(); + } + String score; + switch (change.review.score) { + case approved: + score = MessageFormat.format(addPattern, change.review.score.getValue()); + break; + case vetoed: + score = MessageFormat.format(delPattern, Math.abs(change.review.score.getValue())); + break; + default: + score = "" + change.review.score.getValue(); + } + String date = df.format(change.date); + sb.append(String.format("| %1$s | %2$s | %3$s | %4$s |\n", + date, name, score, change.review.score.toString())); + } + sb.append(HARD_BRK); + } else if (lastChange.hasComment()) { + // comment update + sb.append(MessageFormat.format("**{0}** commented on this ticket.", user.getDisplayName())); + sb.append(HARD_BRK); + } else { + // general update + pattern = "**{0}** has updated this ticket."; + sb.append(MessageFormat.format(pattern, user.getDisplayName())); + sb.append(HARD_BRK); + } + + // ticket link + sb.append(MessageFormat.format("[view ticket {0,number,0}]({1})", + ticket.number, ticketService.getTicketUrl(ticket))); + sb.append(HARD_BRK); + + if (newTicket) { + // ticket title + sb.append(MessageFormat.format("### {0}", ticket.title)); + sb.append(HARD_BRK); + + // ticket description, on state change + if (StringUtils.isEmpty(ticket.body)) { + sb.append("<span style=\"color: #888;\">no description entered</span>"); + } else { + sb.append(ticket.body); + } + sb.append(HARD_BRK); + sb.append(HR); + } + + // field changes + if (lastChange.hasFieldChanges()) { + Map<Field, String> filtered = new HashMap<Field, String>(); + for (Map.Entry<Field, String> fc : lastChange.fields.entrySet()) { + if (!fieldExclusions.contains(fc.getKey())) { + // field is included + filtered.put(fc.getKey(), fc.getValue()); + } + } + + // sort by field ordinal + List<Field> fields = new ArrayList<Field>(filtered.keySet()); + Collections.sort(fields); + + if (filtered.size() > 0) { + sb.append(HARD_BRK); + sb.append("| Field Changes ||\n"); + sb.append("| ------------: | :----------- |\n"); + for (Field field : fields) { + String value; + if (filtered.get(field) == null) { + value = ""; + } else { + value = filtered.get(field).replace("\r\n", "<br/>").replace("\n", "<br/>").replace("|", "|"); + } + sb.append(String.format("| **%1$s:** | %2$s |\n", field.name(), value)); + } + sb.append(HARD_BRK); + } + } + + // new comment + if (lastChange.hasComment()) { + sb.append(HR); + sb.append(lastChange.comment.text); + sb.append(HARD_BRK); + } + + // insert the patchset details and review instructions + if (lastChange.hasPatchset() && ticket.isOpen()) { + if (commits != null && commits.size() > 0) { + // append the commit list + String title = isFastForward ? "Commits added to previous patchset revision" : "All commits in patchset"; + sb.append(MessageFormat.format("| {0} |||\n", title)); + sb.append("| SHA | Author | Title |\n"); + sb.append("| :-- | :----- | :---- |\n"); + for (RevCommit commit : commits) { + sb.append(MessageFormat.format("| {0} | {1} | {2} |\n", + commit.getName(), commit.getAuthorIdent().getName(), + StringUtils.trimString(commit.getShortMessage(), Constants.LEN_SHORTLOG).replace("|", "|"))); + } + sb.append(HARD_BRK); + } + + if (diffstat != null) { + // append the changed path list + String title = isFastForward ? "Files changed since previous patchset revision" : "All files changed in patchset"; + sb.append(MessageFormat.format("| {0} |||\n", title)); + sb.append("| :-- | :----------- | :-: |\n"); + for (PathChangeModel path : diffstat.paths) { + String add = MessageFormat.format(addPattern, path.insertions); + String del = MessageFormat.format(delPattern, path.deletions); + String diff = null; + switch (path.changeType) { + case ADD: + diff = add; + break; + case DELETE: + diff = del; + break; + case MODIFY: + if (path.insertions > 0 && path.deletions > 0) { + // insertions & deletions + diff = add + "/" + del; + } else if (path.insertions > 0) { + // just insertions + diff = add; + } else { + // just deletions + diff = del; + } + break; + default: + diff = path.changeType.name(); + break; + } + sb.append(MessageFormat.format("| {0} | {1} | {2} |\n", + getChangeType(path.changeType), path.name, diff)); + } + sb.append(HARD_BRK); + } + + sb.append(formatPatchsetInstructions(ticket, lastChange.patchset)); + } + + return sb.toString(); + } + + protected String getChangeType(ChangeType type) { + String style = null; + switch (type) { + case ADD: + style = "color:darkgreen;"; + break; + case COPY: + style = ""; + break; + case DELETE: + style = "color:darkred;"; + break; + case MODIFY: + style = ""; + break; + case RENAME: + style = ""; + break; + default: + break; + } + String code = type.name().toUpperCase().substring(0, 1); + if (style == null) { + return code; + } else { + return MessageFormat.format("<strong><span style=\"{0}padding:2px;margin:2px;border:1px solid #ddd;\">{1}</span></strong>", style, code); + } + } + + /** + * Generates patchset review instructions for command-line git + * + * @param patchset + * @return instructions + */ + protected String formatPatchsetInstructions(TicketModel ticket, Patchset patchset) { + String canonicalUrl = settings.getString(Keys.web.canonicalUrl, "https://localhost:8443"); + String repositoryUrl = canonicalUrl + Constants.R_PATH + ticket.repository; + + String ticketBranch = Repository.shortenRefName(PatchsetCommand.getTicketBranch(ticket.number)); + String patchsetBranch = PatchsetCommand.getPatchsetBranch(ticket.number, patchset.number); + String reviewBranch = PatchsetCommand.getReviewBranch(ticket.number); + + String instructions = readResource("commands.md"); + instructions = instructions.replace("${ticketId}", "" + ticket.number); + instructions = instructions.replace("${patchset}", "" + patchset.number); + instructions = instructions.replace("${repositoryUrl}", repositoryUrl); + instructions = instructions.replace("${ticketRef}", ticketBranch); + instructions = instructions.replace("${patchsetRef}", patchsetBranch); + instructions = instructions.replace("${reviewBranch}", reviewBranch); + + return instructions; + } + + /** + * Gets the usermodel for the username. Creates a temp model, if required. + * + * @param username + * @return a usermodel + */ + protected UserModel getUserModel(String username) { + UserModel user = userManager.getUserModel(username); + if (user == null) { + // create a temporary user model (for unit tests) + user = new UserModel(username); + } + return user; + } + + /** + * Set the proper recipients for a ticket. + * + * @param ticket + * @param mailing + */ + protected void setRecipients(TicketModel ticket, Mailing mailing) { + RepositoryModel repository = repositoryManager.getRepositoryModel(ticket.repository); + + // + // Direct TO recipients + // + Set<String> toAddresses = new TreeSet<String>(); + for (String name : ticket.getParticipants()) { + UserModel user = userManager.getUserModel(name); + if (user != null) { + if (!StringUtils.isEmpty(user.emailAddress)) { + if (user.canView(repository)) { + toAddresses.add(user.emailAddress); + } else { + LoggerFactory.getLogger(getClass()).warn( + MessageFormat.format("ticket {0}-{1,number,0}: {2} can not receive notification", + repository.name, ticket.number, user.username)); + } + } + } + } + mailing.setRecipients(toAddresses); + + // + // CC recipients + // + Set<String> ccs = new TreeSet<String>(); + + // cc users mentioned in last comment + Change lastChange = ticket.changes.get(ticket.changes.size() - 1); + if (lastChange.hasComment()) { + Pattern p = Pattern.compile("\\s@([A-Za-z0-9-_]+)"); + Matcher m = p.matcher(lastChange.comment.text); + while (m.find()) { + String username = m.group(); + ccs.add(username); + } + } + + // cc users who are watching the ticket + ccs.addAll(ticket.getWatchers()); + + // TODO cc users who are watching the repository + + Set<String> ccAddresses = new TreeSet<String>(); + for (String name : ccs) { + UserModel user = userManager.getUserModel(name); + if (user != null) { + if (!StringUtils.isEmpty(user.emailAddress)) { + if (user.canView(repository)) { + ccAddresses.add(user.emailAddress); + } else { + LoggerFactory.getLogger(getClass()).warn( + MessageFormat.format("ticket {0}-{1,number,0}: {2} can not receive notification", + repository.name, ticket.number, user.username)); + } + } + } + } + + // cc repository mailing list addresses + if (!ArrayUtils.isEmpty(repository.mailingLists)) { + ccAddresses.addAll(repository.mailingLists); + } + ccAddresses.addAll(settings.getStrings(Keys.mail.mailingLists)); + + mailing.setCCs(ccAddresses); + } + + protected String readStyle() { + StringBuilder sb = new StringBuilder(); + sb.append("<style>\n"); + sb.append(readResource("email.css")); + sb.append("</style>\n"); + return sb.toString(); + } + + protected String readResource(String resource) { + StringBuilder sb = new StringBuilder(); + InputStream is = null; + try { + is = getClass().getResourceAsStream(resource); + List<String> lines = IOUtils.readLines(is); + for (String line : lines) { + sb.append(line).append('\n'); + } + } catch (IOException e) { + + } finally { + if (is != null) { + try { + is.close(); + } catch (IOException e) { + } + } + } + return sb.toString(); + } +} diff --git a/src/main/java/com/gitblit/tickets/TicketResponsible.java b/src/main/java/com/gitblit/tickets/TicketResponsible.java new file mode 100644 index 00000000..12621c6c --- /dev/null +++ b/src/main/java/com/gitblit/tickets/TicketResponsible.java @@ -0,0 +1,59 @@ +/* + * Copyright 2014 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.io.Serializable; + +import org.parboiled.common.StringUtils; + +import com.gitblit.models.UserModel; + +/** + * A ticket responsible. + * + * @author James Moger + * + */ +public class TicketResponsible implements Serializable, Comparable<TicketResponsible> { + + private static final long serialVersionUID = 1L; + + public final String displayname; + + public final String username; + + public final String email; + + public TicketResponsible(UserModel user) { + this(user.getDisplayName(), user.username, user.emailAddress); + } + + public TicketResponsible(String displayname, String username, String email) { + this.displayname = displayname; + this.username = username; + this.email = email; + } + + @Override + public String toString() { + return displayname + (StringUtils.isEmpty(username) ? "" : (" (" + username + ")")); + } + + @Override + public int compareTo(TicketResponsible o) { + return toString().compareTo(o.toString()); + } +} diff --git a/src/main/java/com/gitblit/tickets/TicketSerializer.java b/src/main/java/com/gitblit/tickets/TicketSerializer.java new file mode 100644 index 00000000..2a71af33 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/TicketSerializer.java @@ -0,0 +1,175 @@ +/* + * Copyright 2013 gitblit.com. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.gitblit.tickets; + +import java.lang.reflect.Type; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Date; +import java.util.List; + +import com.gitblit.models.TicketModel; +import com.gitblit.models.TicketModel.Change; +import com.gitblit.models.TicketModel.Score; +import com.gitblit.utils.ArrayUtils; +import com.gitblit.utils.JsonUtils.ExcludeField; +import com.gitblit.utils.JsonUtils.GmtDateTypeAdapter; +import com.google.gson.ExclusionStrategy; +import com.google.gson.Gson; +import com.google.gson.GsonBuilder; +import com.google.gson.JsonDeserializationContext; +import com.google.gson.JsonDeserializer; +import com.google.gson.JsonElement; +import com.google.gson.JsonPrimitive; +import com.google.gson.JsonSerializationContext; +import com.google.gson.JsonSerializer; +import com.google.gson.JsonSyntaxException; +import com.google.gson.reflect.TypeToken; + +/** + * Serializes and deserializes tickets, change, and journals. + * + * @author James Moger + * + */ +public class TicketSerializer { + + protected static final Type JOURNAL_TYPE = new TypeToken<Collection<Change>>() {}.getType(); + + public static List<Change> deserializeJournal(String json) { + Collection<Change> list = gson().fromJson(json, JOURNAL_TYPE); + return new ArrayList<Change>(list); + } + + public static TicketModel deserializeTicket(String json) { + return gson().fromJson(json, TicketModel.class); + } + + public static TicketLabel deserializeLabel(String json) { + return gson().fromJson(json, TicketLabel.class); + } + + public static TicketMilestone deserializeMilestone(String json) { + return gson().fromJson(json, TicketMilestone.class); + } + + + public static String serializeJournal(List<Change> changes) { + try { + Gson gson = gson(); + return gson.toJson(changes); + } catch (Exception e) { + // won't happen + } + return null; + } + + public static String serialize(TicketModel ticket) { + if (ticket == null) { + return null; + } + try { + Gson gson = gson( + new ExcludeField("com.gitblit.models.TicketModel$Attachment.content"), + new ExcludeField("com.gitblit.models.TicketModel$Attachment.deleted"), + new ExcludeField("com.gitblit.models.TicketModel$Comment.deleted")); + return gson.toJson(ticket); + } catch (Exception e) { + // won't happen + } + return null; + } + + public static String serialize(Change change) { + if (change == null) { + return null; + } + try { + Gson gson = gson( + new ExcludeField("com.gitblit.models.TicketModel$Attachment.content")); + return gson.toJson(change); + } catch (Exception e) { + // won't happen + } + return null; + } + + public static String serialize(TicketLabel label) { + if (label == null) { + return null; + } + try { + Gson gson = gson(); + return gson.toJson(label); + } catch (Exception e) { + // won't happen + } + return null; + } + + public static String serialize(TicketMilestone milestone) { + if (milestone == null) { + return null; + } + try { + Gson gson = gson(); + return gson.toJson(milestone); + } catch (Exception e) { + // won't happen + } + return null; + } + + // build custom gson instance with GMT date serializer/deserializer + // http://code.google.com/p/google-gson/issues/detail?id=281 + public static Gson gson(ExclusionStrategy... strategies) { + GsonBuilder builder = new GsonBuilder(); + builder.registerTypeAdapter(Date.class, new GmtDateTypeAdapter()); + builder.registerTypeAdapter(Score.class, new ScoreTypeAdapter()); + if (!ArrayUtils.isEmpty(strategies)) { + builder.setExclusionStrategies(strategies); + } + return builder.create(); + } + + private static class ScoreTypeAdapter implements JsonSerializer<Score>, JsonDeserializer<Score> { + + private ScoreTypeAdapter() { + } + + @Override + public synchronized JsonElement serialize(Score score, Type type, + JsonSerializationContext jsonSerializationContext) { + return new JsonPrimitive(score.getValue()); + } + + @Override + public synchronized Score deserialize(JsonElement jsonElement, Type type, + JsonDeserializationContext jsonDeserializationContext) { + try { + int value = jsonElement.getAsInt(); + for (Score score : Score.values()) { + if (score.getValue() == value) { + return score; + } + } + return Score.not_reviewed; + } catch (Exception e) { + throw new JsonSyntaxException(jsonElement.getAsString(), e); + } + } + } +} diff --git a/src/main/java/com/gitblit/tickets/commands.md b/src/main/java/com/gitblit/tickets/commands.md new file mode 100644 index 00000000..25c24f4f --- /dev/null +++ b/src/main/java/com/gitblit/tickets/commands.md @@ -0,0 +1,11 @@ +#### To review with Git + +on a detached HEAD... + + git fetch ${repositoryUrl} ${ticketRef} && git checkout FETCH_HEAD + +on a new branch... + + git fetch ${repositoryUrl} ${ticketRef} && git checkout -B ${reviewBranch} FETCH_HEAD + + diff --git a/src/main/java/com/gitblit/tickets/email.css b/src/main/java/com/gitblit/tickets/email.css new file mode 100644 index 00000000..3b815420 --- /dev/null +++ b/src/main/java/com/gitblit/tickets/email.css @@ -0,0 +1,38 @@ +table { + border:1px solid #ddd; + margin: 15px 0px; +} + +th { + font-weight: bold; + border-bottom: 1px solid #ddd; +} + +td, th { + padding: 4px 8px; + vertical-align: top; +} + +a { + color: #2F58A0; +} + +a:hover { + color: #002060; +} + +body { + color: black; +} + +pre { + background-color: rgb(250, 250, 250); + border: 1px solid rgb(221, 221, 221); + border-radius: 4px 4px 4px 4px; + display: block; + font-size: 12px; + line-height: 18px; + margin: 9px 0; + padding: 8.5px; + white-space: pre-wrap; +} |