mirrors/jgit - jgit - source @ dussan.org

Thomas Wolf 4bfc6c2ae9 Significantly speed up FileTreeIterator on Windows Getting attributes of files on Windows is an expensive operation. Windows stores file attributes in the directory, so they are basically available "for free" when a directory is listed. The implementation of Java's Files.walkFileTree() takes advantage of that (at least in the OpenJDK implementation for Windows) and provides the attributes from the directory to a FileVisitor. Using Files.walkFileTree() with a maximum depth of 1 is thus a good approach on Windows to get both the file names and the attributes in one go. In my tests, this gives a significant speed-up of FileTreeIterator over the "normal" way: using File.listFiles() and then reading the attributes of each file individually. The speed-up is hard to quantify exactly, but in my tests I've observed consistently 30-40% for staging 500 files one after another, each individually, and up to 50% for individual TreeWalks with a FileTreeIterator. On Unix, this technique is detrimental. Unix stores file attributes differently, and getting attributes of individual files is not costly. On Unix, the old way of doing a listFiles() and getting individual attributes (both native operations) is about three times faster than using walkFileTree, which is implemented in Java. Therefore, move the operation to FS/FS_Win32 and call it from FileTreeIterator, so that we can have different implementations depending on the file system. A little performance test program is included as a JUnit test (to be run manually). While this does speed up things on Windows, it doesn't solve the basic problem of bug 532300: the iterator always gets the full directory listing and the attributes of all files, and the more files there are the longer that takes. Bug: 532300 Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>	6 years ago
..
FileTreeIteratorPerformanceTest.java	Significantly speed up FileTreeIterator on Windows	6 years ago

Thomas Wolf 4bfc6c2ae9 Significantly speed up FileTreeIterator on Windows

Getting attributes of files on Windows is an expensive operation.
Windows stores file attributes in the directory, so they are
basically available "for free" when a directory is listed. The
implementation of Java's Files.walkFileTree() takes advantage of
that (at least in the OpenJDK implementation for Windows) and
provides the attributes from the directory to a FileVisitor.

Using Files.walkFileTree() with a maximum depth of 1 is thus a
good approach on Windows to get both the file names and the
attributes in one go.

In my tests, this gives a significant speed-up of FileTreeIterator
over the "normal" way: using File.listFiles() and then reading the
attributes of each file individually. The speed-up is hard to
quantify exactly, but in my tests I've observed consistently 30-40%
for staging 500 files one after another, each individually, and up
to 50% for individual TreeWalks with a FileTreeIterator.

On Unix, this technique is detrimental. Unix stores file attributes
differently, and getting attributes of individual files is not costly.
On Unix, the old way of doing a listFiles() and getting individual
attributes (both native operations) is about three times faster than
using walkFileTree, which is implemented in Java.

Therefore, move the operation to FS/FS_Win32 and call it from
FileTreeIterator, so that we can have different implementations
depending on the file system.

A little performance test program is included as a JUnit test (to be
run manually).

While this does speed up things on Windows, it doesn't solve the basic
problem of bug 532300: the iterator always gets the full directory
listing and the attributes of all files, and the more files there are
the longer that takes.

Bug: 532300
Change-Id: Ic5facb871c725256c2324b0d97b95e6efc33282a
Signed-off-by: Thomas Wolf <thomas.wolf@paranor.ch>

6 years ago

FileTreeIteratorPerformanceTest.java

Significantly speed up FileTreeIterator on Windows

6 years ago