You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

AppServer.java 9.0KB

Support LFS protocol and a file system based LFS storage Implement LfsProtocolServlet handling the "Git LFS v1 Batch API" protocol [1]. Add a simple file system based LFS content store and the debug-lfs-store command to simplify testing. Introduce a LargeFileRepository interface to enable additional storage implementation while reusing the same protocol implementation. At the client side we have to configure the lfs.url, specify that we use the batch API and we don't use authentication: [lfs] url = http://host:port/lfs batch = true [lfs "http://host:port/lfs"] access = none the git-lfs client appends the "objects/batch" to the lfs.url. Hard code an Authorization header in the FileLfsRepository.getAction because then git-lfs client will skip asking for credentials. It will just forward the Authorization header from the response to the download/upload request. The FileLfsServlet supports file content storage for "Large File Storage" (LFS) server as defined by the Github LFS API [2]. - upload and download of large files is probably network bound hence use an asynchronous servlet for good scalability - simple object storage in file system with 2 level fan-out - use LockFile to protect writing large objects against multiple concurrent uploads of the same object - to prevent corrupt uploads the uploaded file is rejected if its hash doesn't match id given in URL The debug-lfs-store command is used to run the LfsProtocolServlet and, optionally, the FileLfsServlet which makes it easier to setup a local test server. [1] https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md [2] https://github.com/github/git-lfs/tree/master/docs/api Bug: 472961 Change-Id: I7378da5575159d2195138d799704880c5c82d5f3 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
8 years ago
Support LFS protocol and a file system based LFS storage Implement LfsProtocolServlet handling the "Git LFS v1 Batch API" protocol [1]. Add a simple file system based LFS content store and the debug-lfs-store command to simplify testing. Introduce a LargeFileRepository interface to enable additional storage implementation while reusing the same protocol implementation. At the client side we have to configure the lfs.url, specify that we use the batch API and we don't use authentication: [lfs] url = http://host:port/lfs batch = true [lfs "http://host:port/lfs"] access = none the git-lfs client appends the "objects/batch" to the lfs.url. Hard code an Authorization header in the FileLfsRepository.getAction because then git-lfs client will skip asking for credentials. It will just forward the Authorization header from the response to the download/upload request. The FileLfsServlet supports file content storage for "Large File Storage" (LFS) server as defined by the Github LFS API [2]. - upload and download of large files is probably network bound hence use an asynchronous servlet for good scalability - simple object storage in file system with 2 level fan-out - use LockFile to protect writing large objects against multiple concurrent uploads of the same object - to prevent corrupt uploads the uploaded file is rejected if its hash doesn't match id given in URL The debug-lfs-store command is used to run the LfsProtocolServlet and, optionally, the FileLfsServlet which makes it easier to setup a local test server. [1] https://github.com/github/git-lfs/blob/master/docs/api/http-v1-batch.md [2] https://github.com/github/git-lfs/tree/master/docs/api Bug: 472961 Change-Id: I7378da5575159d2195138d799704880c5c82d5f3 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Signed-off-by: Sasa Zivkov <sasa.zivkov@sap.com>
8 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302
  1. /*
  2. * Copyright (C) 2010, 2012 Google Inc.
  3. * and other copyright owners as documented in the project's IP log.
  4. *
  5. * This program and the accompanying materials are made available
  6. * under the terms of the Eclipse Distribution License v1.0 which
  7. * accompanies this distribution, is reproduced below, and is
  8. * available at http://www.eclipse.org/org/documents/edl-v10.php
  9. *
  10. * All rights reserved.
  11. *
  12. * Redistribution and use in source and binary forms, with or
  13. * without modification, are permitted provided that the following
  14. * conditions are met:
  15. *
  16. * - Redistributions of source code must retain the above copyright
  17. * notice, this list of conditions and the following disclaimer.
  18. *
  19. * - Redistributions in binary form must reproduce the above
  20. * copyright notice, this list of conditions and the following
  21. * disclaimer in the documentation and/or other materials provided
  22. * with the distribution.
  23. *
  24. * - Neither the name of the Eclipse Foundation, Inc. nor the
  25. * names of its contributors may be used to endorse or promote
  26. * products derived from this software without specific prior
  27. * written permission.
  28. *
  29. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
  30. * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
  31. * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
  32. * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  33. * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
  34. * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  35. * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
  36. * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
  37. * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
  38. * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
  39. * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
  40. * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
  41. * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  42. */
  43. package org.eclipse.jgit.junit.http;
  44. import static org.junit.Assert.assertFalse;
  45. import static org.junit.Assert.assertTrue;
  46. import java.net.InetAddress;
  47. import java.net.URI;
  48. import java.net.URISyntaxException;
  49. import java.net.UnknownHostException;
  50. import java.util.ArrayList;
  51. import java.util.List;
  52. import org.eclipse.jetty.security.AbstractLoginService;
  53. import org.eclipse.jetty.security.Authenticator;
  54. import org.eclipse.jetty.security.ConstraintMapping;
  55. import org.eclipse.jetty.security.ConstraintSecurityHandler;
  56. import org.eclipse.jetty.security.authentication.BasicAuthenticator;
  57. import org.eclipse.jetty.server.Connector;
  58. import org.eclipse.jetty.server.HttpConfiguration;
  59. import org.eclipse.jetty.server.HttpConnectionFactory;
  60. import org.eclipse.jetty.server.Server;
  61. import org.eclipse.jetty.server.ServerConnector;
  62. import org.eclipse.jetty.server.handler.ContextHandlerCollection;
  63. import org.eclipse.jetty.servlet.ServletContextHandler;
  64. import org.eclipse.jetty.util.security.Constraint;
  65. import org.eclipse.jetty.util.security.Credential;
  66. import org.eclipse.jetty.util.security.Credential.Crypt;
  67. import org.eclipse.jgit.transport.URIish;
  68. /**
  69. * Tiny web application server for unit testing.
  70. * <p>
  71. * Tests should start the server in their {@code setUp()} method and stop the
  72. * server in their {@code tearDown()} method. Only while started the server's
  73. * URL and/or port number can be obtained.
  74. */
  75. public class AppServer {
  76. /** Realm name for the secure access areas. */
  77. public static final String realm = "Secure Area";
  78. /** Username for secured access areas. */
  79. public static final String username = "agitter";
  80. /** Password for {@link #username} in secured access areas. */
  81. public static final String password = "letmein";
  82. static {
  83. // Install a logger that throws warning messages.
  84. //
  85. final String prop = "org.eclipse.jetty.util.log.class";
  86. System.setProperty(prop, RecordingLogger.class.getName());
  87. }
  88. private final Server server;
  89. private final ServerConnector connector;
  90. private final ContextHandlerCollection contexts;
  91. private final TestRequestLog log;
  92. public AppServer() {
  93. this(0);
  94. }
  95. /**
  96. * @param port
  97. * the http port number
  98. * @since 4.2
  99. */
  100. public AppServer(int port) {
  101. server = new Server();
  102. HttpConfiguration http_config = new HttpConfiguration();
  103. http_config.setSecureScheme("https");
  104. http_config.setSecurePort(8443);
  105. http_config.setOutputBufferSize(32768);
  106. connector = new ServerConnector(server,
  107. new HttpConnectionFactory(http_config));
  108. connector.setPort(port);
  109. try {
  110. final InetAddress me = InetAddress.getByName("localhost");
  111. connector.setHost(me.getHostAddress());
  112. } catch (UnknownHostException e) {
  113. throw new RuntimeException("Cannot find localhost", e);
  114. }
  115. contexts = new ContextHandlerCollection();
  116. log = new TestRequestLog();
  117. log.setHandler(contexts);
  118. server.setConnectors(new Connector[] { connector });
  119. server.setHandler(log);
  120. }
  121. /**
  122. * Create a new servlet context within the server.
  123. * <p>
  124. * This method should be invoked before the server is started, once for each
  125. * context the caller wants to register.
  126. *
  127. * @param path
  128. * path of the context; use "/" for the root context if binding
  129. * to the root is desired.
  130. * @return the context to add servlets into.
  131. */
  132. public ServletContextHandler addContext(String path) {
  133. assertNotYetSetUp();
  134. if ("".equals(path))
  135. path = "/";
  136. ServletContextHandler ctx = new ServletContextHandler();
  137. ctx.setContextPath(path);
  138. contexts.addHandler(ctx);
  139. return ctx;
  140. }
  141. public ServletContextHandler authBasic(ServletContextHandler ctx) {
  142. assertNotYetSetUp();
  143. auth(ctx, new BasicAuthenticator());
  144. return ctx;
  145. }
  146. static class TestLoginService extends AbstractLoginService {
  147. private String role;
  148. TestLoginService(String role) {
  149. this.role = role;
  150. }
  151. @Override
  152. protected String[] loadRoleInfo(UserPrincipal user) {
  153. return new String[] { role };
  154. }
  155. @Override
  156. protected UserPrincipal loadUserInfo(String name) {
  157. return new UserPrincipal(username,
  158. Credential.getCredential(Crypt.crypt(username, password)));
  159. }
  160. }
  161. private void auth(ServletContextHandler ctx, Authenticator authType) {
  162. final String role = "can-access";
  163. TestLoginService users = new TestLoginService(role);
  164. ConstraintMapping cm = new ConstraintMapping();
  165. cm.setConstraint(new Constraint());
  166. cm.getConstraint().setAuthenticate(true);
  167. cm.getConstraint().setDataConstraint(Constraint.DC_NONE);
  168. cm.getConstraint().setRoles(new String[] { role });
  169. cm.setPathSpec("/*");
  170. ConstraintSecurityHandler sec = new ConstraintSecurityHandler();
  171. sec.setRealmName(realm);
  172. sec.setAuthenticator(authType);
  173. sec.setLoginService(users);
  174. sec.setConstraintMappings(new ConstraintMapping[] { cm });
  175. sec.setHandler(ctx);
  176. contexts.removeHandler(ctx);
  177. contexts.addHandler(sec);
  178. }
  179. /**
  180. * Start the server on a random local port.
  181. *
  182. * @throws Exception
  183. * the server cannot be started, testing is not possible.
  184. */
  185. public void setUp() throws Exception {
  186. RecordingLogger.clear();
  187. log.clear();
  188. server.start();
  189. }
  190. /**
  191. * Shutdown the server.
  192. *
  193. * @throws Exception
  194. * the server refuses to halt, or wasn't running.
  195. */
  196. public void tearDown() throws Exception {
  197. RecordingLogger.clear();
  198. log.clear();
  199. server.setStopTimeout(0);
  200. server.stop();
  201. }
  202. /**
  203. * Get the URI to reference this server.
  204. * <p>
  205. * The returned URI includes the proper host name and port number, but does
  206. * not contain a path.
  207. *
  208. * @return URI to reference this server's root context.
  209. */
  210. public URI getURI() {
  211. assertAlreadySetUp();
  212. String host = connector.getHost();
  213. if (host.contains(":") && !host.startsWith("["))
  214. host = "[" + host + "]";
  215. final String uri = "http://" + host + ":" + getPort();
  216. try {
  217. return new URI(uri);
  218. } catch (URISyntaxException e) {
  219. throw new RuntimeException("Unexpected URI error on " + uri, e);
  220. }
  221. }
  222. /** @return the local port number the server is listening on. */
  223. public int getPort() {
  224. assertAlreadySetUp();
  225. return connector.getLocalPort();
  226. }
  227. /** @return all requests since the server was started. */
  228. public List<AccessEvent> getRequests() {
  229. return new ArrayList<AccessEvent>(log.getEvents());
  230. }
  231. /**
  232. * @param base
  233. * base URI used to access the server.
  234. * @param path
  235. * the path to locate requests for, relative to {@code base}.
  236. * @return all requests which match the given path.
  237. */
  238. public List<AccessEvent> getRequests(URIish base, String path) {
  239. return getRequests(HttpTestCase.join(base, path));
  240. }
  241. /**
  242. * @param path
  243. * the path to locate requests for.
  244. * @return all requests which match the given path.
  245. */
  246. public List<AccessEvent> getRequests(String path) {
  247. ArrayList<AccessEvent> r = new ArrayList<AccessEvent>();
  248. for (AccessEvent event : log.getEvents()) {
  249. if (event.getPath().equals(path)) {
  250. r.add(event);
  251. }
  252. }
  253. return r;
  254. }
  255. private void assertNotYetSetUp() {
  256. assertFalse("server is not running", server.isRunning());
  257. }
  258. private void assertAlreadySetUp() {
  259. assertTrue("server is running", server.isRunning());
  260. }
  261. }