You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

FSTest.java 8.4KB

Persist minimal racy threshold and allow manual configuration To enable persisting the minimal racy threshold per FileStore add a new config option to the user global git configuration: - Config section is "filesystem" - Config subsection is concatenation of - Java vendor (system property "java.vendor") - Java version (system property "java.version") - FileStore's name, on Windows we use the attribute volume:vsn instead since the name is not necessarily unique. - separated by '|' e.g. "AdoptOpenJDK|1.8.0_212-b03|/dev/disk1s1" The same prefix is used as for filesystem timestamp resolution, so both values are stored in the same config section - The config key for minmal racy threshold is "minRacyThreshold" as a time value, supported time units are those supported by DefaultTypedConfigGetter#getTimeUnit - measure for 3 seconds to limit runtime which depends on hardware, OS and Java version being used If the minimal racy threshold is configured for a given FileStore the configured value is used instead of measuring it. When the minimal racy threshold was measured it is persisted in the user global git configuration. Rename FileStoreAttributeCache to FileStoreAttributes since this class is now declared public in order to enable exposing all attributes in one object. Example: [filesystem "AdoptOpenJDK|11.0.3|/dev/disk1s1"] timestampResolution = 7000 nanoseconds minRacyThreshold = 3440 microseconds Change-Id: I22195e488453aae8d011b0a8e3276fe3d99deaea Signed-off-by: Matthias Sohn <matthias.sohn@sap.com> Also-By: Marc Strapetz <marc.strapetz@syntevo.com>
4 years ago
Measure minimum racy interval to auto-configure FileSnapshot By running FileSnapshotTest#detectFileModified we found that the sum of measured filesystem timestamp resolution and measured clock resolution may yield a too small interval after a file has been modified which we need to consider racily clean. In our tests we didn't find this behavior on all systems we tested on, e.g. on MacOS using APFS and Java 8 and 11 this effect was not observed. On Linux (SLES 15, kernel 4.12.14-150.22-default) we collected the following test results using Java 8 and 11: In 23-98% of 10000 test runs (depending on filesystem type and Java version) the test failed, which means the effective interval which needs to be considered racily clean after a file was modified is larger than the measured file timestamp resolution. "delta" is the observed interval after a file has been modified but FileSnapshot did not yet detect the modification: "resolution" is the measured sum of file timestamp resolution and clock resolution seen in Java. Java version filesystem failures resolution min delta max delta 1.8.0_212-b04 btrfs 98.6% 1 ms 3.6 ms 6.6 ms 1.8.0_212-b04 ext4 82.6% 3 ms 1.1 ms 4.1 ms 1.8.0_212-b04 xfs 23.8% 4 ms 3.7 ms 3.9 ms 1.8.0_212-b04 zfs 23.1% 3 ms 4.8 ms 5.0 ms 11.0.3+7 btrfs 98.1% 3 us 0.7 ms 4.7 ms 11.0.3+7 ext4 98.1% 6 us 0.7 ms 4.7 ms 11.0.3+7 xfs 98.5% 7 us 0.1 ms 8.0 ms 11.0.3+7 zfs 98.4% 7 us 0.7 ms 5.2 ms Mac OS 1.8.0_212 APFS 0% 1 s 11.0.3+7 APFS 0% 6 us The observed delta is not distributed according to a normal gaussian distribution but rather random in the observed range between "min delta" and "max delta". Run this test after measuring file timestamp resolution in FS.FileAttributeCache to auto-configure JGit since it's unclear what mechanism is causing this effect. In FileSnapshot#isRacyClean use the maximum of the measured timestamp resolution and the measured "delta" as explained above to decide if a given FileSnapshot is to be considered racily clean. Add a 30% safety margin to ensure we are on the safe side. Change-Id: I1c8bb59f6486f174b7bbdc63072777ddbe06694d Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
4 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239
  1. /*
  2. * Copyright (C) 2012-2013, Robin Rosenberg <robin.rosenberg@dewire.com>
  3. * and other copyright owners as documented in the project's IP log.
  4. *
  5. * This program and the accompanying materials are made available
  6. * under the terms of the Eclipse Distribution License v1.0 which
  7. * accompanies this distribution, is reproduced below, and is
  8. * available at http://www.eclipse.org/org/documents/edl-v10.php
  9. *
  10. * All rights reserved.
  11. *
  12. * Redistribution and use in source and binary forms, with or
  13. * without modification, are permitted provided that the following
  14. * conditions are met:
  15. *
  16. * - Redistributions of source code must retain the above copyright
  17. * notice, this list of conditions and the following disclaimer.
  18. *
  19. * - Redistributions in binary form must reproduce the above
  20. * copyright notice, this list of conditions and the following
  21. * disclaimer in the documentation and/or other materials provided
  22. * with the distribution.
  23. *
  24. * - Neither the name of the Eclipse Foundation, Inc. nor the
  25. * names of its contributors may be used to endorse or promote
  26. * products derived from this software without specific prior
  27. * written permission.
  28. *
  29. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
  30. * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
  31. * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
  32. * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  33. * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
  34. * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  35. * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
  36. * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
  37. * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
  38. * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
  39. * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
  40. * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
  41. * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  42. */
  43. package org.eclipse.jgit.util;
  44. import static java.time.Instant.EPOCH;
  45. import static org.junit.Assert.assertEquals;
  46. import static org.junit.Assert.assertFalse;
  47. import static org.junit.Assert.assertTrue;
  48. import static org.junit.Assume.assumeTrue;
  49. import java.io.File;
  50. import java.io.IOException;
  51. import java.nio.charset.Charset;
  52. import java.nio.file.Files;
  53. import java.nio.file.Path;
  54. import java.nio.file.attribute.FileTime;
  55. import java.nio.file.attribute.PosixFileAttributeView;
  56. import java.nio.file.attribute.PosixFilePermission;
  57. import java.time.Duration;
  58. import java.time.ZoneId;
  59. import java.time.format.DateTimeFormatter;
  60. import java.util.Locale;
  61. import java.util.Set;
  62. import java.util.concurrent.TimeUnit;
  63. import org.eclipse.jgit.errors.CommandFailedException;
  64. import org.eclipse.jgit.junit.MockSystemReader;
  65. import org.eclipse.jgit.junit.RepositoryTestCase;
  66. import org.eclipse.jgit.lib.RepositoryCache;
  67. import org.junit.After;
  68. import org.junit.Assume;
  69. import org.junit.Before;
  70. import org.junit.Test;
  71. public class FSTest {
  72. private File trash;
  73. @Before
  74. public void setUp() throws Exception {
  75. SystemReader.setInstance(new MockSystemReader());
  76. trash = File.createTempFile("tmp_", "");
  77. trash.delete();
  78. assertTrue("mkdir " + trash, trash.mkdir());
  79. }
  80. @After
  81. public void tearDown() throws Exception {
  82. FileUtils.delete(trash, FileUtils.RECURSIVE | FileUtils.RETRY);
  83. }
  84. /**
  85. * The old File methods traverse symbolic links and look at the targets.
  86. * With symbolic links we usually want to modify/look at the link. For some
  87. * reason the executable attribute seems to always look at the target, but
  88. * for the other attributes like lastModified, hidden and exists we must
  89. * differ between the link and the target.
  90. *
  91. * @throws IOException
  92. * @throws InterruptedException
  93. */
  94. @Test
  95. public void testSymlinkAttributes() throws IOException, InterruptedException {
  96. Assume.assumeTrue(FS.DETECTED.supportsSymlinks());
  97. FS fs = FS.DETECTED;
  98. File link = new File(trash, "ä");
  99. File target = new File(trash, "å");
  100. fs.createSymLink(link, "å");
  101. assertTrue(fs.exists(link));
  102. String targetName = fs.readSymLink(link);
  103. assertEquals("å", targetName);
  104. assertTrue(fs.lastModifiedInstant(link).compareTo(EPOCH) > 0);
  105. assertTrue(fs.exists(link));
  106. assertFalse(fs.canExecute(link));
  107. assertEquals(2, fs.length(link));
  108. assertFalse(fs.exists(target));
  109. assertFalse(fs.isFile(target));
  110. assertFalse(fs.isDirectory(target));
  111. assertFalse(fs.canExecute(target));
  112. RepositoryTestCase.fsTick(link);
  113. // Now create the link target
  114. FileUtils.createNewFile(target);
  115. assertTrue(fs.exists(link));
  116. assertTrue(fs.lastModifiedInstant(link).compareTo(EPOCH) > 0);
  117. assertTrue(fs.lastModifiedInstant(target)
  118. .compareTo(fs.lastModifiedInstant(link)) > 0);
  119. assertFalse(fs.canExecute(link));
  120. fs.setExecute(target, true);
  121. assertFalse(fs.canExecute(link));
  122. assumeTrue(fs.supportsExecute());
  123. assertTrue(fs.canExecute(target));
  124. }
  125. @Test
  126. public void testExecutableAttributes() throws Exception {
  127. FS fs = FS.DETECTED.newInstance();
  128. // If this assumption fails the test is halted and ignored.
  129. assumeTrue(fs instanceof FS_POSIX);
  130. ((FS_POSIX) fs).setUmask(0022);
  131. File f = new File(trash, "bla");
  132. assertTrue(f.createNewFile());
  133. assertFalse(fs.canExecute(f));
  134. Set<PosixFilePermission> permissions = readPermissions(f);
  135. assertTrue(!permissions.contains(PosixFilePermission.OTHERS_EXECUTE));
  136. assertTrue(!permissions.contains(PosixFilePermission.GROUP_EXECUTE));
  137. assertTrue(!permissions.contains(PosixFilePermission.OWNER_EXECUTE));
  138. fs.setExecute(f, true);
  139. permissions = readPermissions(f);
  140. assertTrue("'owner' execute permission not set",
  141. permissions.contains(PosixFilePermission.OWNER_EXECUTE));
  142. assertTrue("'group' execute permission not set",
  143. permissions.contains(PosixFilePermission.GROUP_EXECUTE));
  144. assertTrue("'others' execute permission not set",
  145. permissions.contains(PosixFilePermission.OTHERS_EXECUTE));
  146. ((FS_POSIX) fs).setUmask(0033);
  147. fs.setExecute(f, false);
  148. assertFalse(fs.canExecute(f));
  149. fs.setExecute(f, true);
  150. permissions = readPermissions(f);
  151. assertTrue("'owner' execute permission not set",
  152. permissions.contains(PosixFilePermission.OWNER_EXECUTE));
  153. assertFalse("'group' execute permission set",
  154. permissions.contains(PosixFilePermission.GROUP_EXECUTE));
  155. assertFalse("'others' execute permission set",
  156. permissions.contains(PosixFilePermission.OTHERS_EXECUTE));
  157. }
  158. private Set<PosixFilePermission> readPermissions(File f) throws IOException {
  159. return Files
  160. .getFileAttributeView(f.toPath(), PosixFileAttributeView.class)
  161. .readAttributes().permissions();
  162. }
  163. @Test(expected = CommandFailedException.class)
  164. public void testReadPipePosixCommandFailure()
  165. throws CommandFailedException {
  166. FS fs = FS.DETECTED.newInstance();
  167. assumeTrue(fs instanceof FS_POSIX);
  168. FS.readPipe(fs.userHome(),
  169. new String[] { "/bin/sh", "-c", "exit 1" },
  170. Charset.defaultCharset().name());
  171. }
  172. @Test(expected = CommandFailedException.class)
  173. public void testReadPipeCommandStartFailure()
  174. throws CommandFailedException {
  175. FS fs = FS.DETECTED.newInstance();
  176. FS.readPipe(fs.userHome(),
  177. new String[] { "this-command-does-not-exist" },
  178. Charset.defaultCharset().name());
  179. }
  180. @Test
  181. public void testFsTimestampResolution() throws Exception {
  182. DateTimeFormatter formatter = DateTimeFormatter
  183. .ofPattern("uuuu-MMM-dd HH:mm:ss.nnnnnnnnn", Locale.ENGLISH)
  184. .withZone(ZoneId.systemDefault());
  185. Path dir = Files.createTempDirectory("probe-filesystem");
  186. Duration resolution = FS.getFileStoreAttributes(dir)
  187. .getFsTimestampResolution();
  188. long resolutionNs = resolution.toNanos();
  189. assertTrue(resolutionNs > 0);
  190. for (int i = 0; i < 10; i++) {
  191. Path f = null;
  192. try {
  193. f = dir.resolve("testTimestampResolution" + i);
  194. Files.createFile(f);
  195. FileUtils.touch(f);
  196. FileTime t1 = Files.getLastModifiedTime(f);
  197. TimeUnit.NANOSECONDS.sleep(resolutionNs);
  198. FileUtils.touch(f);
  199. FileTime t2 = Files.getLastModifiedTime(f);
  200. assertTrue(String.format(
  201. "expected t2=%s to be larger than t1=%s\nsince file timestamp resolution was measured to be %,d ns",
  202. formatter.format(t2.toInstant()),
  203. formatter.format(t1.toInstant()),
  204. Long.valueOf(resolutionNs)), t2.compareTo(t1) > 0);
  205. } finally {
  206. Files.delete(f);
  207. }
  208. }
  209. }
  210. // bug 548682
  211. @Test
  212. public void testRepoCacheRelativePathUnbornRepo() {
  213. assertFalse(RepositoryCache.FileKey
  214. .isGitRepository(new File("repo.git"), FS.DETECTED));
  215. }
  216. }