You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

RepositoryTestCase.java 16KB

Persist minimal racy threshold and allow manual configuration To enable persisting the minimal racy threshold per FileStore add a new config option to the user global git configuration: - Config section is "filesystem" - Config subsection is concatenation of - Java vendor (system property "java.vendor") - Java version (system property "java.version") - FileStore's name, on Windows we use the attribute volume:vsn instead since the name is not necessarily unique. - separated by '|' e.g. "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1" The same prefix is used as for filesystem timestamp resolution, so both values are stored in the same config section - The config key for minmal racy threshold is "minRacyThreshold" as a time value, supported time units are those supported by DefaultTypedConfigGetter#getTimeUnit - measure for 3 seconds to limit runtime which depends on hardware, OS and Java version being used If the minimal racy threshold is configured for a given FileStore the configured value is used instead of measuring it. When the minimal racy threshold was measured it is persisted in the user global git configuration. Rename FileStoreAttributeCache to FileStoreAttributes since this class is now declared public in order to enable exposing all attributes in one object. Example: [filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"] timestampResolution = 7000 nanoseconds minRacyThreshold = 3440 microseconds Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
4 years ago
Measure minimum racy interval to auto-configure FileSnapshot By running FileSnapshotTest#detectFileModified we found that the sum of measured filesystem timestamp resolution and measured clock resolution may yield a too small interval after a file has been modified which we need to consider racily clean. In our tests we didn't find this behavior on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11 this effect was not observed. On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the following test results using Java 8 and 11: In 23-98% of 10000 test runs (depending on filesystem type and Java version) the test failed, which means the effective interval which needs to be considered racily clean after a file was modified is larger than the measured file timestamp resolution. "delta" is the observed interval after a file has been modified but FileSnapshot did not yet detect the modification: "resolution" is the measured sum of file timestamp resolution and clock resolution seen in Java. Java version filesystem failures resolution min delta max delta 1.8.0_212-b04 btrfs 98.6% 1 ms 3.6 ms 6.6 ms 1.8.0_212-b04 ext4 82.6% 3 ms 1.1 ms 4.1 ms 1.8.0_212-b04 xfs 23.8% 4 ms 3.7 ms 3.9 ms 1.8.0_212-b04 zfs 23.1% 3 ms 4.8 ms 5.0 ms 11.0.3+7 btrfs 98.1% 3 us 0.7 ms 4.7 ms 11.0.3+7 ext4 98.1% 6 us 0.7 ms 4.7 ms 11.0.3+7 xfs 98.5% 7 us 0.1 ms 8.0 ms 11.0.3+7 zfs 98.4% 7 us 0.7 ms 5.2 ms Mac OS 1.8.0_212 APFS 0% 1 s 11.0.3+7 APFS 0% 6 us The observed delta is not distributed according to a normal gaussian distribution but rather random in the observed range between "min delta" and "max delta". Run this test after measuring file timestamp resolution in FS.FileAttributeCache to auto-configure JGit since it's unclear what mechanism is causing this effect. In FileSnapshot#isRacyClean use the maximum of the measured timestamp resolution and the measured "delta" as explained above to decide if a given FileSnapshot is to be considered racily clean. Add a 30% safety margin to ensure we are on the safe side. Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
4 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550
  1. /*
  2. * Copyright (C) 2009, Google Inc.
  3. * Copyright (C) 2007-2008, Robin Rosenberg <robin.rosenberg@dewire.com>
  4. * Copyright (C) 2006-2007, Shawn O. Pearce <spearce@spearce.org>
  5. * Copyright (C) 2009, Yann Simon <yann.simon.fr@gmail.com> and others
  6. *
  7. * This program and the accompanying materials are made available under the
  8. * terms of the Eclipse Distribution License v. 1.0 which is available at
  9. * https://www.eclipse.org/org/documents/edl-v10.php.
  10. *
  11. * SPDX-License-Identifier: BSD-3-Clause
  12. */
  13. package org.eclipse.jgit.junit;
  14. import static java.nio.charset.StandardCharsets.UTF_8;
  15. import static org.junit.Assert.assertEquals;
  16. import java.io.File;
  17. import java.io.FileInputStream;
  18. import java.io.FileNotFoundException;
  19. import java.io.FileOutputStream;
  20. import java.io.IOException;
  21. import java.io.InputStreamReader;
  22. import java.io.Reader;
  23. import java.nio.file.Path;
  24. import java.time.Instant;
  25. import java.util.Map;
  26. import java.util.concurrent.TimeUnit;
  27. import org.eclipse.jgit.api.Git;
  28. import org.eclipse.jgit.api.errors.GitAPIException;
  29. import org.eclipse.jgit.dircache.DirCacheBuilder;
  30. import org.eclipse.jgit.dircache.DirCacheCheckout;
  31. import org.eclipse.jgit.dircache.DirCacheEntry;
  32. import org.eclipse.jgit.internal.storage.file.FileRepository;
  33. import org.eclipse.jgit.lib.AnyObjectId;
  34. import org.eclipse.jgit.lib.Constants;
  35. import org.eclipse.jgit.lib.FileMode;
  36. import org.eclipse.jgit.lib.ObjectId;
  37. import org.eclipse.jgit.lib.ObjectInserter;
  38. import org.eclipse.jgit.lib.RefUpdate;
  39. import org.eclipse.jgit.lib.Repository;
  40. import org.eclipse.jgit.revwalk.RevCommit;
  41. import org.eclipse.jgit.revwalk.RevWalk;
  42. import org.eclipse.jgit.treewalk.FileTreeIterator;
  43. import org.eclipse.jgit.util.FS;
  44. import org.eclipse.jgit.util.FileUtils;
  45. import org.junit.After;
  46. import org.junit.Before;
  47. /**
  48. * Base class for most JGit unit tests.
  49. *
  50. * Sets up a predefined test repository and has support for creating additional
  51. * repositories and destroying them when the tests are finished.
  52. */
  53. public abstract class RepositoryTestCase extends LocalDiskRepositoryTestCase {
  54. /**
  55. * Copy a file
  56. *
  57. * @param src
  58. * @param dst
  59. * @throws IOException
  60. */
  61. protected static void copyFile(File src, File dst)
  62. throws IOException {
  63. try (FileInputStream fis = new FileInputStream(src);
  64. FileOutputStream fos = new FileOutputStream(dst)) {
  65. final byte[] buf = new byte[4096];
  66. int r;
  67. while ((r = fis.read(buf)) > 0) {
  68. fos.write(buf, 0, r);
  69. }
  70. }
  71. }
  72. /**
  73. * Write a trash file
  74. *
  75. * @param name
  76. * @param data
  77. * @return the trash file
  78. * @throws IOException
  79. */
  80. protected File writeTrashFile(String name, String data)
  81. throws IOException {
  82. return JGitTestUtil.writeTrashFile(db, name, data);
  83. }
  84. /**
  85. * Create a symbolic link
  86. *
  87. * @param link
  88. * the path of the symbolic link to create
  89. * @param target
  90. * the target of the symbolic link
  91. * @return the path to the symbolic link
  92. * @throws Exception
  93. * @since 4.2
  94. */
  95. protected Path writeLink(String link, String target)
  96. throws Exception {
  97. return JGitTestUtil.writeLink(db, link, target);
  98. }
  99. /**
  100. * Write a trash file
  101. *
  102. * @param subdir
  103. * @param name
  104. * @param data
  105. * @return the trash file
  106. * @throws IOException
  107. */
  108. protected File writeTrashFile(final String subdir, final String name,
  109. final String data)
  110. throws IOException {
  111. return JGitTestUtil.writeTrashFile(db, subdir, name, data);
  112. }
  113. /**
  114. * Read content of a file
  115. *
  116. * @param name
  117. * @return the file's content
  118. * @throws IOException
  119. */
  120. protected String read(String name) throws IOException {
  121. return JGitTestUtil.read(db, name);
  122. }
  123. /**
  124. * Check if file exists
  125. *
  126. * @param name
  127. * file name
  128. * @return if the file exists
  129. */
  130. protected boolean check(String name) {
  131. return JGitTestUtil.check(db, name);
  132. }
  133. /**
  134. * Delete a trash file
  135. *
  136. * @param name
  137. * file name
  138. * @throws IOException
  139. */
  140. protected void deleteTrashFile(String name) throws IOException {
  141. JGitTestUtil.deleteTrashFile(db, name);
  142. }
  143. /**
  144. * Check content of a file.
  145. *
  146. * @param f
  147. * @param checkData
  148. * expected content
  149. * @throws IOException
  150. */
  151. protected static void checkFile(File f, String checkData)
  152. throws IOException {
  153. try (Reader r = new InputStreamReader(new FileInputStream(f),
  154. UTF_8)) {
  155. if (checkData.length() > 0) {
  156. char[] data = new char[checkData.length()];
  157. assertEquals(data.length, r.read(data));
  158. assertEquals(checkData, new String(data));
  159. }
  160. assertEquals(-1, r.read());
  161. }
  162. }
  163. /** Test repository, initialized for this test case. */
  164. protected FileRepository db;
  165. /** Working directory of {@link #db}. */
  166. protected File trash;
  167. /** {@inheritDoc} */
  168. @Override
  169. @Before
  170. public void setUp() throws Exception {
  171. super.setUp();
  172. db = createWorkRepository();
  173. trash = db.getWorkTree();
  174. }
  175. @Override
  176. @After
  177. public void tearDown() throws Exception {
  178. db.close();
  179. super.tearDown();
  180. }
  181. /**
  182. * Represent the state of the index in one String. This representation is
  183. * useful when writing tests which do assertions on the state of the index.
  184. * By default information about path, mode, stage (if different from 0) is
  185. * included. A bitmask controls which additional info about
  186. * modificationTimes, smudge state and length is included.
  187. * <p>
  188. * The format of the returned string is described with this BNF:
  189. *
  190. * <pre>
  191. * result = ( "[" path mode stage? time? smudge? length? sha1? content? "]" )* .
  192. * mode = ", mode:" number .
  193. * stage = ", stage:" number .
  194. * time = ", time:t" timestamp-index .
  195. * smudge = "" | ", smudged" .
  196. * length = ", length:" number .
  197. * sha1 = ", sha1:" hex-sha1 .
  198. * content = ", content:" blob-data .
  199. * </pre>
  200. *
  201. * 'stage' is only presented when the stage is different from 0. All
  202. * reported time stamps are mapped to strings like "t0", "t1", ... "tn". The
  203. * smallest reported time-stamp will be called "t0". This allows to write
  204. * assertions against the string although the concrete value of the time
  205. * stamps is unknown.
  206. *
  207. * @param includedOptions
  208. * a bitmask constructed out of the constants {@link #MOD_TIME},
  209. * {@link #SMUDGE}, {@link #LENGTH}, {@link #CONTENT_ID} and
  210. * {@link #CONTENT} controlling which info is present in the
  211. * resulting string.
  212. * @return a string encoding the index state
  213. * @throws IllegalStateException
  214. * @throws IOException
  215. */
  216. public String indexState(int includedOptions)
  217. throws IllegalStateException, IOException {
  218. return indexState(db, includedOptions);
  219. }
  220. /**
  221. * Resets the index to represent exactly some filesystem content. E.g. the
  222. * following call will replace the index with the working tree content:
  223. * <p>
  224. * <code>resetIndex(new FileSystemIterator(db))</code>
  225. * <p>
  226. * This method can be used by testcases which first prepare a new commit
  227. * somewhere in the filesystem (e.g. in the working-tree) and then want to
  228. * have an index which matches their prepared content.
  229. *
  230. * @param treeItr
  231. * a {@link org.eclipse.jgit.treewalk.FileTreeIterator} which
  232. * determines which files should go into the new index
  233. * @throws FileNotFoundException
  234. * @throws IOException
  235. */
  236. protected void resetIndex(FileTreeIterator treeItr)
  237. throws FileNotFoundException, IOException {
  238. try (ObjectInserter inserter = db.newObjectInserter()) {
  239. DirCacheBuilder builder = db.lockDirCache().builder();
  240. DirCacheEntry dce;
  241. while (!treeItr.eof()) {
  242. long len = treeItr.getEntryLength();
  243. dce = new DirCacheEntry(treeItr.getEntryPathString());
  244. dce.setFileMode(treeItr.getEntryFileMode());
  245. dce.setLastModified(treeItr.getEntryLastModifiedInstant());
  246. dce.setLength((int) len);
  247. try (FileInputStream in = new FileInputStream(
  248. treeItr.getEntryFile())) {
  249. dce.setObjectId(
  250. inserter.insert(Constants.OBJ_BLOB, len, in));
  251. }
  252. builder.add(dce);
  253. treeItr.next(1);
  254. }
  255. builder.commit();
  256. inserter.flush();
  257. }
  258. }
  259. /**
  260. * Helper method to map arbitrary objects to user-defined names. This can be
  261. * used create short names for objects to produce small and stable debug
  262. * output. It is guaranteed that when you lookup the same object multiple
  263. * times even with different nameTemplates this method will always return
  264. * the same name which was derived from the first nameTemplate.
  265. * nameTemplates can contain "%n" which will be replaced by a running number
  266. * before used as a name.
  267. *
  268. * @param l
  269. * the object to lookup
  270. * @param lookupTable
  271. * a table storing object-name mappings.
  272. * @param nameTemplate
  273. * the name for that object. Can contain "%n" which will be
  274. * replaced by a running number before used as a name. If the
  275. * lookup table already contains the object this parameter will
  276. * be ignored
  277. * @return a name of that object. Is not guaranteed to be unique. Use
  278. * nameTemplates containing "%n" to always have unique names
  279. */
  280. public static String lookup(Object l, String nameTemplate,
  281. Map<Object, String> lookupTable) {
  282. String name = lookupTable.get(l);
  283. if (name == null) {
  284. name = nameTemplate.replaceAll("%n",
  285. Integer.toString(lookupTable.size()));
  286. lookupTable.put(l, name);
  287. }
  288. return name;
  289. }
  290. /**
  291. * Replaces '\' by '/'
  292. *
  293. * @param str
  294. * the string in which backslashes should be replaced
  295. * @return the resulting string with slashes
  296. * @since 4.2
  297. */
  298. public static String slashify(String str) {
  299. str = str.replace('\\', '/');
  300. return str;
  301. }
  302. /**
  303. * Waits until it is guaranteed that a subsequent file modification has a
  304. * younger modification timestamp than the modification timestamp of the
  305. * given file. This is done by touching a temporary file, reading the
  306. * lastmodified attribute and, if needed, sleeping. After sleeping this loop
  307. * starts again until the filesystem timer has advanced enough. The
  308. * temporary file will be created as a sibling of lastFile.
  309. *
  310. * @param lastFile
  311. * the file on which we want to wait until the filesystem timer
  312. * has advanced more than the lastmodification timestamp of this
  313. * file
  314. * @return return the last measured value of the filesystem timer which is
  315. * greater than then the lastmodification time of lastfile.
  316. * @throws InterruptedException
  317. * @throws IOException
  318. */
  319. public static Instant fsTick(File lastFile)
  320. throws InterruptedException,
  321. IOException {
  322. File tmp;
  323. FS fs = FS.DETECTED;
  324. if (lastFile == null) {
  325. lastFile = tmp = File
  326. .createTempFile("fsTickTmpFile", null);
  327. } else {
  328. if (!fs.exists(lastFile)) {
  329. throw new FileNotFoundException(lastFile.getPath());
  330. }
  331. tmp = File.createTempFile("fsTickTmpFile", null,
  332. lastFile.getParentFile());
  333. }
  334. long res = FS.getFileStoreAttributes(tmp.toPath())
  335. .getFsTimestampResolution().toNanos();
  336. long sleepTime = res / 10;
  337. try {
  338. Instant startTime = fs.lastModifiedInstant(lastFile);
  339. Instant actTime = fs.lastModifiedInstant(tmp);
  340. while (actTime.compareTo(startTime) <= 0) {
  341. TimeUnit.NANOSECONDS.sleep(sleepTime);
  342. FileUtils.touch(tmp.toPath());
  343. actTime = fs.lastModifiedInstant(tmp);
  344. }
  345. return actTime;
  346. } finally {
  347. FileUtils.delete(tmp);
  348. }
  349. }
  350. /**
  351. * Create a branch
  352. *
  353. * @param objectId
  354. * @param branchName
  355. * @throws IOException
  356. */
  357. protected void createBranch(ObjectId objectId, String branchName)
  358. throws IOException {
  359. RefUpdate updateRef = db.updateRef(branchName);
  360. updateRef.setNewObjectId(objectId);
  361. updateRef.update();
  362. }
  363. /**
  364. * Checkout a branch
  365. *
  366. * @param branchName
  367. * @throws IllegalStateException
  368. * @throws IOException
  369. */
  370. protected void checkoutBranch(String branchName)
  371. throws IllegalStateException, IOException {
  372. try (RevWalk walk = new RevWalk(db)) {
  373. RevCommit head = walk.parseCommit(db.resolve(Constants.HEAD));
  374. RevCommit branch = walk.parseCommit(db.resolve(branchName));
  375. DirCacheCheckout dco = new DirCacheCheckout(db,
  376. head.getTree().getId(), db.lockDirCache(),
  377. branch.getTree().getId());
  378. dco.setFailOnConflict(true);
  379. dco.checkout();
  380. }
  381. // update the HEAD
  382. RefUpdate refUpdate = db.updateRef(Constants.HEAD);
  383. refUpdate.setRefLogMessage("checkout: moving to " + branchName, false);
  384. refUpdate.link(branchName);
  385. }
  386. /**
  387. * Writes a number of files in the working tree. The first content specified
  388. * will be written into a file named '0', the second into a file named "1"
  389. * and so on. If <code>null</code> is specified as content then this file is
  390. * skipped.
  391. *
  392. * @param ensureDistinctTimestamps
  393. * if set to <code>true</code> then between two write operations
  394. * this method will wait to ensure that the second file will get
  395. * a different lastmodification timestamp than the first file.
  396. * @param contents
  397. * the contents which should be written into the files
  398. * @return the File object associated to the last written file.
  399. * @throws IOException
  400. * @throws InterruptedException
  401. */
  402. protected File writeTrashFiles(boolean ensureDistinctTimestamps,
  403. String... contents)
  404. throws IOException, InterruptedException {
  405. File f = null;
  406. for (int i = 0; i < contents.length; i++)
  407. if (contents[i] != null) {
  408. if (ensureDistinctTimestamps && (f != null))
  409. fsTick(f);
  410. f = writeTrashFile(Integer.toString(i), contents[i]);
  411. }
  412. return f;
  413. }
  414. /**
  415. * Commit a file with the specified contents on the specified branch,
  416. * creating the branch if it didn't exist before.
  417. * <p>
  418. * It switches back to the original branch after the commit if there was
  419. * one.
  420. *
  421. * @param filename
  422. * @param contents
  423. * @param branch
  424. * @return the created commit
  425. */
  426. protected RevCommit commitFile(String filename, String contents, String branch) {
  427. try (Git git = new Git(db)) {
  428. Repository repo = git.getRepository();
  429. String originalBranch = repo.getFullBranch();
  430. boolean empty = repo.resolve(Constants.HEAD) == null;
  431. if (!empty) {
  432. if (repo.findRef(branch) == null)
  433. git.branchCreate().setName(branch).call();
  434. git.checkout().setName(branch).call();
  435. }
  436. writeTrashFile(filename, contents);
  437. git.add().addFilepattern(filename).call();
  438. RevCommit commit = git.commit()
  439. .setMessage(branch + ": " + filename).call();
  440. if (originalBranch != null)
  441. git.checkout().setName(originalBranch).call();
  442. else if (empty)
  443. git.branchCreate().setName(branch).setStartPoint(commit).call();
  444. return commit;
  445. } catch (IOException | GitAPIException e) {
  446. throw new RuntimeException(e);
  447. }
  448. }
  449. /**
  450. * Create <code>DirCacheEntry</code>
  451. *
  452. * @param path
  453. * @param mode
  454. * @return the DirCacheEntry
  455. */
  456. protected DirCacheEntry createEntry(String path, FileMode mode) {
  457. return createEntry(path, mode, DirCacheEntry.STAGE_0, path);
  458. }
  459. /**
  460. * Create <code>DirCacheEntry</code>
  461. *
  462. * @param path
  463. * @param mode
  464. * @param content
  465. * @return the DirCacheEntry
  466. */
  467. protected DirCacheEntry createEntry(final String path, final FileMode mode,
  468. final String content) {
  469. return createEntry(path, mode, DirCacheEntry.STAGE_0, content);
  470. }
  471. /**
  472. * Create <code>DirCacheEntry</code>
  473. *
  474. * @param path
  475. * @param mode
  476. * @param stage
  477. * @param content
  478. * @return the DirCacheEntry
  479. */
  480. protected DirCacheEntry createEntry(final String path, final FileMode mode,
  481. final int stage, final String content) {
  482. final DirCacheEntry entry = new DirCacheEntry(path, stage);
  483. entry.setFileMode(mode);
  484. try (ObjectInserter.Formatter formatter = new ObjectInserter.Formatter()) {
  485. entry.setObjectId(formatter.idFor(
  486. Constants.OBJ_BLOB, Constants.encode(content)));
  487. }
  488. return entry;
  489. }
  490. /**
  491. * Create <code>DirCacheEntry</code>
  492. *
  493. * @param path
  494. * @param objectId
  495. * @return the DirCacheEntry
  496. */
  497. protected DirCacheEntry createGitLink(String path, AnyObjectId objectId) {
  498. final DirCacheEntry entry = new DirCacheEntry(path,
  499. DirCacheEntry.STAGE_0);
  500. entry.setFileMode(FileMode.GITLINK);
  501. entry.setObjectId(objectId);
  502. return entry;
  503. }
  504. /**
  505. * Assert files are equal
  506. *
  507. * @param expected
  508. * @param actual
  509. * @throws IOException
  510. */
  511. public static void assertEqualsFile(File expected, File actual)
  512. throws IOException {
  513. assertEquals(expected.getCanonicalFile(), actual.getCanonicalFile());
  514. }
  515. }