You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

FS_Win32.java 5.3KB

Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
6 years ago
Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
6 years ago
Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
6 years ago
Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
6 years ago
Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
6 years ago
Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>
6 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210
  1. /*
  2. * Copyright (C) 2009, Google Inc.
  3. * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org> and others
  4. *
  5. * This program and the accompanying materials are made available under the
  6. * terms of the Eclipse Distribution License v. 1.0 which is available at
  7. * https://www.eclipse.org/org/documents/edl-v10.php.
  8. *
  9. * SPDX-License-Identifier: BSD-3-Clause
  10. */
  11. package org.eclipse.jgit.util;
  12. import java.io.File;
  13. import java.io.IOException;
  14. import java.nio.charset.Charset;
  15. import java.nio.file.FileVisitOption;
  16. import java.nio.file.FileVisitResult;
  17. import java.nio.file.Files;
  18. import java.nio.file.Path;
  19. import java.nio.file.SimpleFileVisitor;
  20. import java.nio.file.attribute.BasicFileAttributes;
  21. import java.util.ArrayList;
  22. import java.util.Arrays;
  23. import java.util.EnumSet;
  24. import java.util.List;
  25. import org.eclipse.jgit.errors.CommandFailedException;
  26. import org.eclipse.jgit.treewalk.FileTreeIterator.FileEntry;
  27. import org.eclipse.jgit.treewalk.FileTreeIterator.FileModeStrategy;
  28. import org.eclipse.jgit.treewalk.WorkingTreeIterator.Entry;
  29. import org.slf4j.Logger;
  30. import org.slf4j.LoggerFactory;
  31. /**
  32. * FS implementation for Windows
  33. *
  34. * @since 3.0
  35. */
  36. public class FS_Win32 extends FS {
  37. private static final Logger LOG = LoggerFactory.getLogger(FS_Win32.class);
  38. /**
  39. * Constructor
  40. */
  41. public FS_Win32() {
  42. super();
  43. }
  44. /**
  45. * Constructor
  46. *
  47. * @param src
  48. * instance whose attributes to copy
  49. */
  50. protected FS_Win32(FS src) {
  51. super(src);
  52. }
  53. /** {@inheritDoc} */
  54. @Override
  55. public FS newInstance() {
  56. return new FS_Win32(this);
  57. }
  58. /** {@inheritDoc} */
  59. @Override
  60. public boolean supportsExecute() {
  61. return false;
  62. }
  63. /** {@inheritDoc} */
  64. @Override
  65. public boolean canExecute(File f) {
  66. return false;
  67. }
  68. /** {@inheritDoc} */
  69. @Override
  70. public boolean setExecute(File f, boolean canExec) {
  71. return false;
  72. }
  73. /** {@inheritDoc} */
  74. @Override
  75. public boolean isCaseSensitive() {
  76. return false;
  77. }
  78. /** {@inheritDoc} */
  79. @Override
  80. public boolean retryFailedLockFileCommit() {
  81. return true;
  82. }
  83. /** {@inheritDoc} */
  84. @Override
  85. public Entry[] list(File directory, FileModeStrategy fileModeStrategy) {
  86. List<Entry> result = new ArrayList<>();
  87. FS fs = this;
  88. boolean checkExecutable = fs.supportsExecute();
  89. try {
  90. Files.walkFileTree(directory.toPath(),
  91. EnumSet.noneOf(FileVisitOption.class), 1,
  92. new SimpleFileVisitor<Path>() {
  93. @Override
  94. public FileVisitResult visitFile(Path file,
  95. BasicFileAttributes attrs) throws IOException {
  96. File f = file.toFile();
  97. FS.Attributes attributes = new FS.Attributes(fs, f,
  98. true, attrs.isDirectory(),
  99. checkExecutable && f.canExecute(),
  100. attrs.isSymbolicLink(),
  101. attrs.isRegularFile(),
  102. attrs.creationTime().toMillis(),
  103. attrs.lastModifiedTime().toInstant(),
  104. attrs.size());
  105. result.add(new FileEntry(f, fs, attributes,
  106. fileModeStrategy));
  107. return FileVisitResult.CONTINUE;
  108. }
  109. @Override
  110. public FileVisitResult visitFileFailed(Path file,
  111. IOException exc) throws IOException {
  112. // Just ignore it
  113. return FileVisitResult.CONTINUE;
  114. }
  115. });
  116. } catch (IOException e) {
  117. // Ignore
  118. }
  119. if (result.isEmpty()) {
  120. return NO_ENTRIES;
  121. }
  122. return result.toArray(new Entry[0]);
  123. }
  124. /** {@inheritDoc} */
  125. @Override
  126. protected File discoverGitExe() {
  127. String path = SystemReader.getInstance().getenv("PATH"); //$NON-NLS-1$
  128. File gitExe = searchPath(path, "git.exe", "git.cmd"); //$NON-NLS-1$ //$NON-NLS-2$
  129. if (gitExe == null) {
  130. if (searchPath(path, "bash.exe") != null) { //$NON-NLS-1$
  131. // This isn't likely to work, but its worth trying:
  132. // If bash is in $PATH, git should also be in $PATH.
  133. String w;
  134. try {
  135. w = readPipe(userHome(),
  136. new String[]{"bash", "--login", "-c", "which git"}, // //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
  137. Charset.defaultCharset().name());
  138. } catch (CommandFailedException e) {
  139. LOG.warn(e.getMessage());
  140. return null;
  141. }
  142. if (!StringUtils.isEmptyOrNull(w)) {
  143. // The path may be in cygwin/msys notation so resolve it right away
  144. gitExe = resolve(null, w);
  145. }
  146. }
  147. }
  148. return gitExe;
  149. }
  150. /** {@inheritDoc} */
  151. @Override
  152. protected File userHomeImpl() {
  153. String home = SystemReader.getInstance().getenv("HOME"); //$NON-NLS-1$
  154. if (home != null) {
  155. return resolve(null, home);
  156. }
  157. String homeDrive = SystemReader.getInstance().getenv("HOMEDRIVE"); //$NON-NLS-1$
  158. if (homeDrive != null) {
  159. String homePath = SystemReader.getInstance().getenv("HOMEPATH"); //$NON-NLS-1$
  160. if (homePath != null) {
  161. return new File(homeDrive, homePath);
  162. }
  163. }
  164. String homeShare = SystemReader.getInstance().getenv("HOMESHARE"); //$NON-NLS-1$
  165. if (homeShare != null) {
  166. return new File(homeShare);
  167. }
  168. return super.userHomeImpl();
  169. }
  170. /** {@inheritDoc} */
  171. @Override
  172. public ProcessBuilder runInShell(String cmd, String[] args) {
  173. List<String> argv = new ArrayList<>(3 + args.length);
  174. argv.add("cmd.exe"); //$NON-NLS-1$
  175. argv.add("/c"); //$NON-NLS-1$
  176. argv.add(cmd);
  177. argv.addAll(Arrays.asList(args));
  178. ProcessBuilder proc = new ProcessBuilder();
  179. proc.command(argv);
  180. return proc;
  181. }
  182. /** {@inheritDoc} */
  183. @Override
  184. public Attributes getAttributes(File path) {
  185. return FileUtils.getFileAttributesBasic(this, path);
  186. }
  187. }