You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

PacketLineIn.java 10KB

Client-side protocol V2 support for fetching Make all transports request protocol V2 when fetching. Depending on the transport, set the GIT_PROTOCOL environment variable (file and ssh), pass the Git-Protocol header (http), or set the hidden "\0version=2\0" (git anon). We'll fall back to V0 if the server doesn't reply with a version 2 answer. A user can control which protocol the client requests via the git config protocol.version; if not set, JGit requests protocol V2 for fetching. Pushing always uses protocol V0 still. In the API, there is only a new Transport.openFetch() version that takes a collection of RefSpecs plus additional patterns to construct the Ref prefixes for the "ls-refs" command in protocol V2. If none are given, the server will still advertise all refs, even in protocol V2. BasePackConnection.readAdvertisedRefs() handles falling back to protocol V0. It newly returns true if V0 was used and the advertised refs were read, and false if V2 is used and an explicit "ls-refs" is needed. (This can't be done transparently inside readAdvertisedRefs() because a "stateless RPC" transport like TransportHttp may need to open a new connection for writing.) BasePackFetchConnection implements the changes needed for the protocol V2 "fetch" command (stateless protocol, simplified ACK handling, delimiters, section headers). In TransportHttp, change readSmartHeaders() to also recognize the "version 2" packet line as a valid smart server indication. Adapt tests, and run all the HTTP tests not only with both HTTP connection factories (JDK and Apache HttpClient) but also with both protocol V0 and V2. The SSH tests are much slower and much more focused on the SSH protocol and SSH key handling. Factor out two very simple cloning and pulling tests and make those run with protocol V2. Bug: 553083 Change-Id: I357c7f5daa7efb2872f1c64ee6f6d54229031ae1 Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
3 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411
  1. /*
  2. * Copyright (C) 2008, 2010 Google Inc.
  3. * Copyright (C) 2008, 2009 Robin Rosenberg <robin.rosenberg@dewire.com>
  4. * Copyright (C) 2008, 2020 Shawn O. Pearce <spearce@spearce.org> and others
  5. *
  6. * This program and the accompanying materials are made available under the
  7. * terms of the Eclipse Distribution License v. 1.0 which is available at
  8. * https://www.eclipse.org/org/documents/edl-v10.php.
  9. *
  10. * SPDX-License-Identifier: BSD-3-Clause
  11. */
  12. package org.eclipse.jgit.transport;
  13. import static java.nio.charset.StandardCharsets.UTF_8;
  14. import java.io.IOException;
  15. import java.io.InputStream;
  16. import java.io.UncheckedIOException;
  17. import java.text.MessageFormat;
  18. import java.util.Iterator;
  19. import org.eclipse.jgit.errors.PackProtocolException;
  20. import org.eclipse.jgit.internal.JGitText;
  21. import org.eclipse.jgit.lib.MutableObjectId;
  22. import org.eclipse.jgit.util.IO;
  23. import org.eclipse.jgit.util.RawParseUtils;
  24. import org.slf4j.Logger;
  25. import org.slf4j.LoggerFactory;
  26. /**
  27. * Read Git style pkt-line formatting from an input stream.
  28. * <p>
  29. * This class is not thread safe and may issue multiple reads to the underlying
  30. * stream for each method call made.
  31. * <p>
  32. * This class performs no buffering on its own. This makes it suitable to
  33. * interleave reads performed by this class with reads performed directly
  34. * against the underlying InputStream.
  35. */
  36. public class PacketLineIn {
  37. private static final Logger log = LoggerFactory.getLogger(PacketLineIn.class);
  38. /**
  39. * Magic return from {@link #readString()} when a flush packet is found.
  40. *
  41. * @deprecated Callers should use {@link #isEnd(String)} to check if a
  42. * string is the end marker, or
  43. * {@link PacketLineIn#readStrings()} to iterate over all
  44. * strings in the input stream until the marker is reached.
  45. */
  46. @Deprecated
  47. public static final String END = new String(); /* must not string pool */
  48. /**
  49. * Magic return from {@link #readString()} when a delim packet is found.
  50. *
  51. * @since 5.0
  52. * @deprecated Callers should use {@link #isDelimiter(String)} to check if a
  53. * string is the delimiter.
  54. */
  55. @Deprecated
  56. public static final String DELIM = new String(); /* must not string pool */
  57. enum AckNackResult {
  58. /** NAK */
  59. NAK,
  60. /** ACK */
  61. ACK,
  62. /** ACK + continue */
  63. ACK_CONTINUE,
  64. /** ACK + common */
  65. ACK_COMMON,
  66. /** ACK + ready */
  67. ACK_READY;
  68. }
  69. private final byte[] lineBuffer = new byte[SideBandOutputStream.SMALL_BUF];
  70. private final InputStream in;
  71. private long limit;
  72. /**
  73. * Create a new packet line reader.
  74. *
  75. * @param in
  76. * the input stream to consume.
  77. */
  78. public PacketLineIn(InputStream in) {
  79. this(in, 0);
  80. }
  81. /**
  82. * Create a new packet line reader.
  83. *
  84. * @param in
  85. * the input stream to consume.
  86. * @param limit
  87. * bytes to read from the input; unlimited if set to 0.
  88. * @since 4.7
  89. */
  90. public PacketLineIn(InputStream in, long limit) {
  91. this.in = in;
  92. this.limit = limit;
  93. }
  94. /**
  95. * Parses a ACK/NAK line in protocol V2.
  96. *
  97. * @param line
  98. * to parse
  99. * @param returnedId
  100. * in case of {@link AckNackResult#ACK_COMMON ACK_COMMON}
  101. * @return one of {@link AckNackResult#NAK NAK},
  102. * {@link AckNackResult#ACK_COMMON ACK_COMMON}, or
  103. * {@link AckNackResult#ACK_READY ACK_READY}
  104. * @throws IOException
  105. * on protocol or transport errors
  106. */
  107. static AckNackResult parseACKv2(String line, MutableObjectId returnedId)
  108. throws IOException {
  109. if ("NAK".equals(line)) { //$NON-NLS-1$
  110. return AckNackResult.NAK;
  111. }
  112. if (line.startsWith("ACK ") && line.length() == 44) { //$NON-NLS-1$
  113. returnedId.fromString(line.substring(4, 44));
  114. return AckNackResult.ACK_COMMON;
  115. }
  116. if ("ready".equals(line)) { //$NON-NLS-1$
  117. return AckNackResult.ACK_READY;
  118. }
  119. if (line.startsWith("ERR ")) { //$NON-NLS-1$
  120. throw new PackProtocolException(line.substring(4));
  121. }
  122. throw new PackProtocolException(
  123. MessageFormat.format(JGitText.get().expectedACKNAKGot, line));
  124. }
  125. AckNackResult readACK(MutableObjectId returnedId) throws IOException {
  126. final String line = readString();
  127. if (line.length() == 0)
  128. throw new PackProtocolException(JGitText.get().expectedACKNAKFoundEOF);
  129. if ("NAK".equals(line)) //$NON-NLS-1$
  130. return AckNackResult.NAK;
  131. if (line.startsWith("ACK ")) { //$NON-NLS-1$
  132. returnedId.fromString(line.substring(4, 44));
  133. if (line.length() == 44)
  134. return AckNackResult.ACK;
  135. final String arg = line.substring(44);
  136. switch (arg) {
  137. case " continue": //$NON-NLS-1$
  138. return AckNackResult.ACK_CONTINUE;
  139. case " common": //$NON-NLS-1$
  140. return AckNackResult.ACK_COMMON;
  141. case " ready": //$NON-NLS-1$
  142. return AckNackResult.ACK_READY;
  143. default:
  144. break;
  145. }
  146. }
  147. if (line.startsWith("ERR ")) //$NON-NLS-1$
  148. throw new PackProtocolException(line.substring(4));
  149. throw new PackProtocolException(MessageFormat.format(JGitText.get().expectedACKNAKGot, line));
  150. }
  151. /**
  152. * Read a single UTF-8 encoded string packet from the input stream.
  153. * <p>
  154. * If the string ends with an LF, it will be removed before returning the
  155. * value to the caller. If this automatic trimming behavior is not desired,
  156. * use {@link #readStringRaw()} instead.
  157. *
  158. * @return the string. {@link #END} if the string was the magic flush
  159. * packet, {@link #DELIM} if the string was the magic DELIM
  160. * packet.
  161. * @throws java.io.IOException
  162. * the stream cannot be read.
  163. */
  164. public String readString() throws IOException {
  165. int len = readLength();
  166. if (len == 0) {
  167. log.debug("git< 0000"); //$NON-NLS-1$
  168. return END;
  169. }
  170. if (len == 1) {
  171. log.debug("git< 0001"); //$NON-NLS-1$
  172. return DELIM;
  173. }
  174. len -= 4; // length header (4 bytes)
  175. if (len == 0) {
  176. log.debug("git< "); //$NON-NLS-1$
  177. return ""; //$NON-NLS-1$
  178. }
  179. byte[] raw;
  180. if (len <= lineBuffer.length)
  181. raw = lineBuffer;
  182. else
  183. raw = new byte[len];
  184. IO.readFully(in, raw, 0, len);
  185. if (raw[len - 1] == '\n')
  186. len--;
  187. String s = RawParseUtils.decode(UTF_8, raw, 0, len);
  188. log.debug("git< " + s); //$NON-NLS-1$
  189. return s;
  190. }
  191. /**
  192. * Get an iterator to read strings from the input stream.
  193. *
  194. * @return an iterator that calls {@link #readString()} until {@link #END}
  195. * is encountered.
  196. *
  197. * @throws IOException
  198. * on failure to read the initial packet line.
  199. * @since 5.4
  200. */
  201. public PacketLineInIterator readStrings() throws IOException {
  202. return new PacketLineInIterator(this);
  203. }
  204. /**
  205. * Read a single UTF-8 encoded string packet from the input stream.
  206. * <p>
  207. * Unlike {@link #readString()} a trailing LF will be retained.
  208. *
  209. * @return the string. {@link #END} if the string was the magic flush
  210. * packet.
  211. * @throws java.io.IOException
  212. * the stream cannot be read.
  213. */
  214. public String readStringRaw() throws IOException {
  215. int len = readLength();
  216. if (len == 0) {
  217. log.debug("git< 0000"); //$NON-NLS-1$
  218. return END;
  219. }
  220. len -= 4; // length header (4 bytes)
  221. byte[] raw;
  222. if (len <= lineBuffer.length)
  223. raw = lineBuffer;
  224. else
  225. raw = new byte[len];
  226. IO.readFully(in, raw, 0, len);
  227. String s = RawParseUtils.decode(UTF_8, raw, 0, len);
  228. log.debug("git< " + s); //$NON-NLS-1$
  229. return s;
  230. }
  231. /**
  232. * Check if a string is the delimiter marker.
  233. *
  234. * @param s
  235. * the string to check
  236. * @return true if the given string is {@link #DELIM}, otherwise false.
  237. * @since 5.4
  238. */
  239. @SuppressWarnings({ "ReferenceEquality", "StringEquality" })
  240. public static boolean isDelimiter(String s) {
  241. return s == DELIM;
  242. }
  243. /**
  244. * Get the delimiter marker.
  245. * <p>
  246. * Intended for use only in tests.
  247. *
  248. * @return The delimiter marker.
  249. */
  250. static String delimiter() {
  251. return DELIM;
  252. }
  253. /**
  254. * Get the end marker.
  255. * <p>
  256. * Intended for use only in tests.
  257. *
  258. * @return The end marker.
  259. */
  260. static String end() {
  261. return END;
  262. }
  263. /**
  264. * Check if a string is the packet end marker.
  265. *
  266. * @param s
  267. * the string to check
  268. * @return true if the given string is {@link #END}, otherwise false.
  269. * @since 5.4
  270. */
  271. @SuppressWarnings({ "ReferenceEquality", "StringEquality" })
  272. public static boolean isEnd(String s) {
  273. return s == END;
  274. }
  275. void discardUntilEnd() throws IOException {
  276. for (;;) {
  277. int n = readLength();
  278. if (n == 0) {
  279. break;
  280. }
  281. IO.skipFully(in, n - 4);
  282. }
  283. }
  284. int readLength() throws IOException {
  285. IO.readFully(in, lineBuffer, 0, 4);
  286. int len;
  287. try {
  288. len = RawParseUtils.parseHexInt16(lineBuffer, 0);
  289. } catch (ArrayIndexOutOfBoundsException err) {
  290. throw invalidHeader(err);
  291. }
  292. if (len == 0) {
  293. return 0;
  294. } else if (len == 1) {
  295. return 1;
  296. } else if (len < 4) {
  297. throw invalidHeader();
  298. }
  299. if (limit != 0) {
  300. int n = len - 4;
  301. if (limit < n) {
  302. limit = -1;
  303. try {
  304. IO.skipFully(in, n);
  305. } catch (IOException e) {
  306. // Ignore failure discarding packet over limit.
  307. }
  308. throw new InputOverLimitIOException();
  309. }
  310. // if set limit must not be 0 (means unlimited).
  311. limit = n < limit ? limit - n : -1;
  312. }
  313. return len;
  314. }
  315. private IOException invalidHeader() {
  316. return new IOException(MessageFormat.format(JGitText.get().invalidPacketLineHeader,
  317. "" + (char) lineBuffer[0] + (char) lineBuffer[1] //$NON-NLS-1$
  318. + (char) lineBuffer[2] + (char) lineBuffer[3]));
  319. }
  320. private IOException invalidHeader(Throwable cause) {
  321. IOException ioe = invalidHeader();
  322. ioe.initCause(cause);
  323. return ioe;
  324. }
  325. /**
  326. * IOException thrown by read when the configured input limit is exceeded.
  327. *
  328. * @since 4.7
  329. */
  330. public static class InputOverLimitIOException extends IOException {
  331. private static final long serialVersionUID = 1L;
  332. }
  333. /**
  334. * Iterator over packet lines.
  335. * <p>
  336. * Calls {@link #readString()} on the {@link PacketLineIn} until
  337. * {@link #END} is encountered.
  338. *
  339. * @since 5.4
  340. *
  341. */
  342. public static class PacketLineInIterator implements Iterable<String> {
  343. private PacketLineIn in;
  344. private String current;
  345. PacketLineInIterator(PacketLineIn in) throws IOException {
  346. this.in = in;
  347. current = in.readString();
  348. }
  349. @Override
  350. public Iterator<String> iterator() {
  351. return new Iterator<String>() {
  352. @Override
  353. public boolean hasNext() {
  354. return !PacketLineIn.isEnd(current);
  355. }
  356. @Override
  357. public String next() {
  358. String next = current;
  359. try {
  360. current = in.readString();
  361. } catch (IOException e) {
  362. throw new UncheckedIOException(e);
  363. }
  364. return next;
  365. }
  366. };
  367. }
  368. }
  369. }