Federation pull_scripts request. Documentation.

docs/00_index.mkd

@@ -73,7 +73,7 @@ Administrators can create and manage all repositories, user accounts, and teams
 
 ### Backup Strategy
 
-Gitblit includes a backup mechanism (*federation*) which can be used to backup repositories and, optionally, user accounts & server settings from your Gitblit instance to another Gitblit instance or to a [Gitblit Federation Client](http://code.google.com/p/gitblit/downloads/detail?name=%FEDCLIENT%).  Similarly, you can use the federation mechanism to aggregate individual workspace Gitblit instances to a common, centralized server.
+Gitblit includes a backup mechanism (*federation*) which can be used to backup repositories and, optionally, user accounts, team definitions, server settings, & Groovy push hook scripts from your Gitblit instance to another Gitblit instance or to a [Gitblit Federation Client](http://code.google.com/p/gitblit/downloads/detail?name=%FEDCLIENT%).  Similarly, you can use the federation mechanism to aggregate individual workspace Gitblit instances to a common, centralized server.
 
 ### Java Runtime Requirement
 

docs/01_features.mkd

@@ -17,6 +17,7 @@
 - Administrators may create, edit, rename, or delete users through the web UI or RPC interface
 - Administrators may create, edit, rename, or delete teams through the web UI or RPC interface
 - Repository Owners may edit repositories through the web UI
+- Gravatar integration
 - Git-notes display support
 - Branch metrics (uses Google Charts)
 - HEAD and Branch RSS feeds

docs/02_federation.mkd

@@ -13,7 +13,7 @@ Please review all the documentation to understand how it works and its limitatio
 
 ### Important Changes to Note
 
-The *Gitblit 0.8.0* federation protocol adds retrieval of team definitions.  Older clients will not know to request team information. 
+The *Gitblit 0.8.0* federation protocol adds retrieval of teams and referenced push scripts.  Older clients will not know to request team or push script information. 
 
 The *Gitblit 0.7.0* federation protocol is incompatible with the 0.6.0 federation protocol because of a change in the way timestamps are formatted.
 
@@ -54,7 +54,7 @@ String usersAndRepositoriesToken = SHA1(passphrase + "-USERS_AND_REPOSITORIES");
 String repositoriesToken = SHA1(passphrase + "-REPOSITORIES");
 %ENDCODE%
     
-The *ALL* token allows another Gitblit instance to pull all your repositories, user accounts, and server settings.  
+The *ALL* token allows another Gitblit instance to pull all your repositories, user accounts, server settings, and referenced push scripts.  
 The *USERS_AND_REPOSITORIES* token allows another Gitblit instance to pull all your repositories and  user accounts.  
 The *REPOSITORIES* token only allows pulling of the repositories.
 
@@ -173,9 +173,17 @@ The pulling Gitblit instance will store a registration-specific `gitblit.propert
 
 These settings are unused by the pulling Gitblit instance.
 
+#### Push Scripts 
+
+Your Groovy push scripts are only pulled when using the *ALL* token.
+
+The pulling Gitblit instance will retrieve any referenced (i.e. used) push script and store it locally as *registration_scriptName.groovy* in the *federation.N.folder* folder.
+
+These scripts are unused by the pulling Gitblit instance.
+
 ### Collisions and Conflict Resolution
 
-Gitblit does **not** detect conflict and it does **not** offer conflict resolution of repositories, users, or settings.
+Gitblit does **not** detect conflict and it does **not** offer conflict resolution of repositories, users, teams, or settings.
 
 If an object exists locally that has the same name as the remote object, it is assumed they are the same and the contents of the remote object are merged into the local object.  If you can not guarantee that this is the case, then you should not store any federated repositories directly in *git.repositoriesFolder* and you should not enable *mergeAccounts*.
 
@@ -250,9 +258,9 @@ These examples would be entered into the `gitblit.properties` file of the pullin
 
 This assumes that the *token* is the *ALL* token from the origin gitblit instance.
 
-The repositories, example1_users.conf, and example1_gitblit.properties will be put in *git.repositoriesFolder* and the origin user accounts will be merged into the local user accounts, including passwords and all roles.  The Gitblit instance will also send a status acknowledgment to the origin Gitblit instance at the end of the pull operation.  The status report will include the state of each repository pull (EXCLUDED, SKIPPED, NOCHANGE, PULLED, MIRRORED).  This way the origin Gitblit instance can monitor the health of its mirrors.
+The repositories, example1_users.conf, example1_gitblit.propertiesn and all example1_scripts.groovy will be put in *git.repositoriesFolder* and the origin user accounts will be merged into the local user accounts, including passwords and all roles.  The Gitblit instance will also send a status acknowledgment to the origin Gitblit instance at the end of the pull operation.  The status report will include the state of each repository pull (EXCLUDED, SKIPPED, NOCHANGE, PULLED, MIRRORED).  This way the origin Gitblit instance can monitor the health of its mirrors.
 
-This example is considered *nearly* perfect because while the origin Gitblit's server settings are pulled and saved locally, they are not merged with your server settings so its not a true mirror, but its likely the mirror you'd want to configure.
+This example is considered *nearly* perfect because while the origin Gitblit's server settings & push scripts are pulled and saved locally, they are not merged with your server settings so its not a true mirror.
 
     federation.example1.url = https://go.gitblit.com
     federation.example1.token = 6f3b8a24bf970f17289b234284c94f43eb42f0e4

docs/04_releases.mkd

@@ -5,7 +5,7 @@
 
 - updated: Gitblit GO is now monolithic like the WAR build. (issue 30)  
 This change helps adoption of GO in environments without an internet connection or with a restricted connection.
-- added: Groovy 1.8.4 and a push hook script mechanism.  Hook scripts can be set per-repository, pre-team, or globally for all repositories.  
+- added: Groovy 1.8.4 and a push hook script mechanism.  Hook scripts can be set per-repository, per-team, or globally for all repositories.  
 Unfortunately this adds another 6 MB to the 8MB Gitblit package, but it allows for a *very* powerful, flexible, platform-independent hook script mechanism.  
     **New:** *groovy.scriptsFolder = groovy*  
     **New:** *groovy.preReceiveScripts =*  
@@ -18,11 +18,11 @@ Unfortunately this adds another 6 MB to the 8MB Gitblit package, but it allows f
 This user service implementation allows for serialization and deserialization of more sophisticated Gitblit User objects and will open the door for more advanced Gitblit features. For upgrading installations, a `users.conf` file will automatically be created for you from your existing `users.properties` file on your first launch of Gitblit.  You will have to manually set *realm.userService=users.conf* to switch to the new user service.  
 The original `users.properties` file and it's corresponding implementation are **deprecated**.  
     **New:** *realm.userService = users.conf*
-- added: Teams for specifying user-repository access in bulk
+- added: Teams for specifying user-repository access in bulk.  Teams may also specify mailing lists addresses and pre- & post- receive hook scripts.
 - added: Gitblit Express bundle to get started running Gitblit on RedHat's OpenShift cloud
 - added: optional Gravatar integration  
     **New:** *web.allowGravatar = true*   
-- added: multi-repository activity page.  this is a timeline of commit activity over the last N days for one or more repositories.  
+- added: aggregated repository activity page.  this is a timeline of commit activity over the last N days for one or more repositories.  
    **New:** *web.activityDuration = 14*  
    **New:** *web.timeFormat = HH:mm*  
    **New:** *web.datestampLongFormat = EEEE, MMMM d, yyyy*  

src/com/gitblit/Constants.java

@@ -117,7 +117,7 @@ public class Constants {
 	 * Enumeration representing the types of federation requests.
 	 */
 	public static enum FederationRequest {
-		POKE, PROPOSAL, PULL_REPOSITORIES, PULL_USERS, PULL_TEAMS, PULL_SETTINGS, STATUS;
+		POKE, PROPOSAL, PULL_REPOSITORIES, PULL_USERS, PULL_TEAMS, PULL_SETTINGS, PULL_SCRIPTS, STATUS;
 
 		public static FederationRequest fromName(String name) {
 			for (FederationRequest type : values()) {

src/com/gitblit/FederationPullExecutor.java

@@ -50,6 +50,7 @@ import com.gitblit.models.RepositoryModel;
 import com.gitblit.models.TeamModel;
 import com.gitblit.models.UserModel;
 import com.gitblit.utils.FederationUtils;
+import com.gitblit.utils.FileUtils;
 import com.gitblit.utils.JGitUtils;
 import com.gitblit.utils.JGitUtils.CloneResult;
 import com.gitblit.utils.StringUtils;
@@ -281,6 +282,8 @@ public class FederationPullExecutor implements Runnable {
 			r.close();
 		}
 
+		IUserService userService = null;
+
 		try {
 			// Pull USERS
 			// TeamModels are automatically pulled because they are contained
@@ -290,7 +293,7 @@ public class FederationPullExecutor implements Runnable {
 			if (users != null && users.size() > 0) {
 				File realmFile = new File(registrationFolderFile, registration.name + "_users.conf");
 				realmFile.delete();
-				ConfigUserService userService = new ConfigUserService(realmFile);
+				userService = new ConfigUserService(realmFile);
 				for (UserModel user : users) {
 					userService.updateUserModel(user.username, user);
 
@@ -357,6 +360,27 @@ public class FederationPullExecutor implements Runnable {
 					registration.name, registration.url), e);
 		}
 
+		try {
+			// Pull TEAMS
+			// We explicitly pull these even though they are embedded in
+			// UserModels because it is possible to use teams to specify
+			// mailing lists or push scripts without specifying users.
+			if (userService != null) {
+				Collection<TeamModel> teams = FederationUtils.getTeams(registration);
+				if (teams != null && teams.size() > 0) {
+					for (TeamModel team : teams) {
+						userService.updateTeamModel(team);
+					}
+				}
+			}
+		} catch (ForbiddenException e) {
+			// ignore forbidden exceptions
+		} catch (IOException e) {
+			logger.warn(MessageFormat.format(
+					"Failed to retrieve TEAMS from federated gitblit ({0} @ {1})",
+					registration.name, registration.url), e);
+		}
+
 		try {
 			// Pull SETTINGS
 			Map<String, String> settings = FederationUtils.getSettings(registration);
@@ -375,6 +399,27 @@ public class FederationPullExecutor implements Runnable {
 					"Failed to retrieve SETTINGS from federated gitblit ({0} @ {1})",
 					registration.name, registration.url), e);
 		}
+		
+		try {
+			// Pull SCRIPTS
+			Map<String, String> scripts = FederationUtils.getScripts(registration);
+			if (scripts != null && scripts.size() > 0) {
+				for (Map.Entry<String, String> script : scripts.entrySet()) {
+					String scriptName = script.getKey();
+					if (scriptName.endsWith(".groovy")) {
+						scriptName = scriptName.substring(0,  scriptName.indexOf(".groovy"));
+					}
+					File file = new File(registrationFolderFile, registration.name + "_" + scriptName + ".groovy");
+					FileUtils.writeContent(file, script.getValue());
+				}
+			}
+		} catch (ForbiddenException e) {
+			// ignore forbidden exceptions
+		} catch (IOException e) {
+			logger.warn(MessageFormat.format(
+					"Failed to retrieve SCRIPTS from federated gitblit ({0} @ {1})",
+					registration.name, registration.url), e);
+		}
 	}
 
 	/**

src/com/gitblit/FederationServlet.java

@@ -15,12 +15,15 @@
  */
 package com.gitblit;
 
+import java.io.File;
 import java.text.MessageFormat;
 import java.util.ArrayList;
 import java.util.Date;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 
 import javax.servlet.http.HttpServletResponse;
 
@@ -30,6 +33,7 @@ import com.gitblit.models.FederationProposal;
 import com.gitblit.models.TeamModel;
 import com.gitblit.models.UserModel;
 import com.gitblit.utils.FederationUtils;
+import com.gitblit.utils.FileUtils;
 import com.gitblit.utils.HttpUtils;
 import com.gitblit.utils.StringUtils;
 import com.gitblit.utils.TimeUtils;
@@ -216,6 +220,41 @@ public class FederationServlet extends JsonServlet {
 					teams.add(user);
 				}
 				result = teams;
+			} else if (FederationRequest.PULL_SCRIPTS.equals(reqType)) {
+				// pull scripts
+				if (!GitBlit.self().validateFederationRequest(reqType, token)) {
+					// invalid token to pull script
+					logger.warn(MessageFormat.format(
+							"Federation token from {0} not authorized to pull SCRIPTS",
+							request.getRemoteAddr()));
+					response.sendError(HttpServletResponse.SC_FORBIDDEN);
+					return;
+				}
+				Map<String, String> scripts = new HashMap<String, String>();
+				
+				Set<String> names = new HashSet<String>();
+				names.addAll(GitBlit.getStrings(Keys.groovy.preReceiveScripts));
+				names.addAll(GitBlit.getStrings(Keys.groovy.postReceiveScripts));
+				for (TeamModel team :  GitBlit.self().getAllTeams()) {
+					names.addAll(team.preReceiveScripts);
+					names.addAll(team.postReceiveScripts);
+				}
+				File scriptsFolder = GitBlit.getFileOrFolder(Keys.groovy.scriptsFolder, "groovy");
+				for (String name : names) {
+					File file = new File(scriptsFolder, name);
+					if (!file.exists() && !file.getName().endsWith(".groovy")) {
+						file = new File(scriptsFolder, name + ".groovy");
+					}
+					if (file.exists()) {
+						// read the script
+						String content = FileUtils.readContent(file, "\n");
+						scripts.put(name, content);
+					} else {
+						// missing script?!
+						logger.warn(MessageFormat.format("Failed to find push script \"{0}\"", name));
+					}
+				}
+				result = scripts;
 			}
 		}
 

src/com/gitblit/GitBlit.java

@@ -1253,6 +1253,7 @@ public class GitBlit implements ServletContextListener {
 		case PULL_TEAMS:
 			return token.equals(all) || token.equals(unr);
 		case PULL_SETTINGS:
+		case PULL_SCRIPTS:
 			return token.equals(all);
 		}
 		return false;

src/com/gitblit/utils/FederationUtils.java

@@ -285,7 +285,8 @@ public class FederationUtils {
 	}
 
 	/**
-	 * Tries to pull the gitblit team definitions from the remote gitblit instance.
+	 * Tries to pull the gitblit team definitions from the remote gitblit
+	 * instance.
 	 * 
 	 * @param registration
 	 * @return a collection of TeamModel objects
@@ -312,6 +313,19 @@ public class FederationUtils {
 		return settings;
 	}
 
+	/**
+	 * Tries to pull the referenced scripts from the remote gitblit instance.
+	 * 
+	 * @param registration
+	 * @return a map of the remote gitblit scripts by script name
+	 * @throws Exception
+	 */
+	public static Map<String, String> getScripts(FederationModel registration) throws Exception {
+		String url = asLink(registration.url, registration.token, FederationRequest.PULL_SCRIPTS);
+		Map<String, String> scripts = JsonUtils.retrieveJson(url, SETTINGS_TYPE);
+		return scripts;
+	}
+
 	/**
 	 * Send an status acknowledgment to the remote Gitblit server.
 	 * 

src/com/gitblit/utils/JsonUtils.java

@@ -108,6 +108,19 @@ public class JsonUtils {
 			UnauthorizedException {
 		return retrieveJson(url, type, null, null);
 	}
+	
+	/**
+	 * Reads a gson object from the specified url.
+	 * 
+	 * @param url
+	 * @param type
+	 * @return the deserialized object
+	 * @throws {@link IOException}
+	 */
+	public static <X> X retrieveJson(String url, Class<? extends X> clazz) throws IOException,
+			UnauthorizedException {
+		return retrieveJson(url, clazz, null, null);
+	}
 
 	/**
 	 * Reads a gson object from the specified url.

tests/com/gitblit/tests/FederationTests.java

@@ -156,4 +156,11 @@ public class FederationTests {
 		assertNotNull(teams);
 		assertTrue(teams.size() > 0);
 	}
+	
+	@Test
+	public void testPullScripts() throws Exception {
+		Map<String, String> scripts = FederationUtils.getScripts(getRegistration());
+		assertNotNull(scripts);
+		assertTrue(scripts.keySet().contains("sendmail"));
+	}
 }