updating harness documentation for new features

2 files changed, 92 insertions, 24 deletions

diff --git a/testing-drivers/src/org/aspectj/testing/drivers/package.html b/testing-drivers/src/org/aspectj/testing/drivers/package.html
index 1bdc35634..6bd1738db 100644
--- a/testing-drivers/src/org/aspectj/testing/drivers/package.html
+++ b/testing-drivers/src/org/aspectj/testing/drivers/package.html
@@ -69,7 +69,12 @@ each component in the chain may accept and interpret:
      <code>-traceTestsMin</code> 
      <code>-hideStreams</code> and 
      <code>!eclipse</code>, used to emit tests results in a form
-     comparable by <code>org.aspectj.testing.util.TestDiffs</code>.     
+     comparable by <code>org.aspectj.testing.util.TestDiffs</code>. 
+     Note that output in the form emitted by <code>-traceTestsMin</code>
+     is input for the options to select tests by title
+     (e.g., <code>-ajctestTitleList</code>)
+     and for the tool which compare results from different test runs
+     (<code>org.aspectj.testing.util.TestDiffs</code>).    
      </td></tr>
      
   <tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and
@@ -95,8 +100,8 @@ each component in the chain may accept and interpret:
      hide streams for passing tests but emit them for failing tests.
      </td></tr>
     
-<tr><td rowspan="3" valign="top">AjcTest
-    <p>selection options for keywords, bugID (PR)
+<tr><td rowspan="4" valign="top">AjcTest
+    <p>selection options for keywords, bugID (PR), or title (description)
     </td></tr>
   <tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code>
      will either require or skip tests that have the specified keywords.
@@ -104,6 +109,32 @@ each component in the chain may accept and interpret:
   <tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code>
      will run only tests that are associated with one of the bug id's listed.
      </td></tr>
+  <tr><td><u>title</u>: 
+     <code>"-ajctestTitleContains=one,two"</code>
+     will require that the title (description) of the test contain 
+     either "one" or "two".
+     Use this to select a few tests you know generally.
+     <br>
+     <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
+     will require that the title (description) of the test be
+     equal to one listed in <code>../tests/ajcTestResults.txt</code>
+     as a line of the form "[PASS|FAIL] {title}(.."
+     (emitted by the <code>-traceTestsMin</code> option).
+     Use this to re-run a set of tests.
+     <br>
+     <code>"-ajctestTitleList=first title\, in theory, second title"</code>
+     will require that the title (description) of the test be
+     exactly "first title, in theory" or "second title".  
+     The entire option must be one argument on the command line.
+     This option only differs from the prior in not specifying
+     a valid file to read.
+     Use this when working with just a few specific tests.
+     <br>
+     <code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code>
+     is the same as the <code>-ajctestTitleList=..</code> variant,
+     except that only results prefixed "FAIL" are included.
+     Use this to re-run only the tests that failed from a large set.
+     </td></tr>
     
 <tr><td rowspan="6" valign="top">CompilerRun
 <p>compiler options and side-effects
@@ -208,6 +239,16 @@ Following are some sample configurations:
     (tests skipped, incomplete, failed, passed).<p>
   </li>
   
+<li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code>
+ <p>This writes to result.txt one line [PASS|FAIL] per test, plus a
+    1-line summary of the test results.<p>
+  </li>
+  
+<li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code>
+ <p>This re-runs any test that failed from the "results.txt" run,
+    verbosely logging any fails.<p>
+  </li>
+  
 <li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code>
  <p>Use this when running tests mainly to see if they pass or
     if the failure messages are typically enough information
@@ -222,7 +263,8 @@ Following are some sample configurations:
   </li>
 
 <li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code>
- <p>This would do four complete runs with the old (Ajc 1.0) compiler: one with
+ <p>Because of the trailing '-' on two of the options,
+    this would do four complete runs with the old (Ajc 1.0) compiler: one with
     no options, one with -lenient, one with -Xlint, and one with both.<p>
   </li>
   
@@ -238,7 +280,7 @@ If you have a set of options you use often, you can define a single-word
 option alias for it; see <code>Harness.optionAliases</code>.
 
 <p><u>Configuration</u>: Most tests use the library jars in 
-<code>aspectj/modules/lib/test</code>, defined in 
+<code>modules/lib/test</code>, defined in 
 <code>org.aspectj.testing.harness.bridge.Globals</code>.  
 Normally the harness finds these by relative path 
 <code>../lib/tests/*.jar</code>, which works whenever the tests are
@@ -254,8 +296,10 @@ The harness must be run in a compiler-compatible VM, and the
 compiler steps run in-process.
 However, the java steps can be run in forked mode, which is useful
 when compiling for a VM which can't run the compiler.
-Here's how you would run the harness to compile against a 
-specific version of 1.1:
+To compile for a different target VM could require
+setting the options for bootclasspath, target, and source.
+To run the harness so that any &lt;run.. tasks run in a
+separate vm, do something like this:
 <pre>
    java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
         -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
@@ -279,5 +323,6 @@ to the classpath, with an effect like these commands
      {mainClass} {option..}
 </pre>
 
+
 </body>
 </html>
diff --git a/tests/readme-tests-module.html b/tests/readme-tests-module.html
index c67a20bd7..bb47df117 100644
--- a/tests/readme-tests-module.html
+++ b/tests/readme-tests-module.html
@@ -76,8 +76,9 @@ Tests differ in the kind of harness they use and their target.
      </tr>
 </table>
 
-<h3><a name="#harness"></a>Harness</h3>
+<h3><a name="harness"></a>Harness</h3>
    
+<h4><a name="harness-tests"></a>Harness tests</h4>
 These are compiler test cases, the preferred form for
 any compiler tests.  The XML files are test definitions,
 and the subdirectories contain the test sources (many of which
@@ -108,6 +109,25 @@ segregated for convenience as follows:
        These are tests to exercise the test harness itself.
        </li>
 </ul>
+The tests specify most anything one could want to do with
+compile/run variants.  The best summary of the test specification
+format is the <a href="ajcTestSuite.dtd">ajcTestSuite.dtd</a>.
+
+<p>
+To write a new test, evolve an existing example.  See the dtd and/or
+the javadoc for the testing harness implementation in the 
+<code>testing</code> module for more details.
+
+<p>To write a test case for a bug, see 
+  <a href="readme-writing-compiler-tests.html">
+           readme-writing-compiler-tests.html</a>.
+   Many bugs that first present in an IDE or the Ant taskdef
+   are compiler bugs that can be written in this form.
+   We <em>much</em> prefer this automated form over
+   manual steps specified in a bug report.
+
+<h4><a name="harness-runs"></a>Running the harness</h4>
+<p>
 To run the harness, use <code>org.aspectj.testing.drivers.Harness</code>,
 in the <code>testing-drivers</code> module. Since that is the main class
 for the <code>testing-drivers</code> module, you can build and 
@@ -119,27 +139,30 @@ run the harness as follows:
   java -jar ../aj-build/jars/testing-drivers-all.jar ajcTests.xml
 </pre>
  
-For help on harness options, use <code>-help</code>.   
-For more help on options for harness components, see 
+The harness enable you to
+<ul>
+  <li>control output, hiding streams and logging messages</li>
+  <li>select tests using keywords, bug ID's, or titles</li>
+  <li>pass compiler arguments on the harness command line</li>
+  <li>use different compiler wrappers - the command-line compiler,
+      the AJDE IDE support (mostly sans GUI), or 
+      the Ant taskdef support.</li>
+  <li>fork java run steps</li>
+</ul>
+For a listing of harness options, use <code>-help</code>.   
+For a fuller description of harness options and components, see 
   <a href="../testing-drivers/src/org/aspectj/testing/drivers/package.html">
   the package documentation for the harness</a>.
-Note in particular how to select tests using keywords 
-and how compiler arguments can be passed on the harness command line.
-<p>
-To write a new test, evolve an existing example.  See the dtd and/or
-the javadoc for the testing harness implementation in the 
-<code>testing</code> module for more details.
-
-<p>To write a test case for a bug, see 
-  <a href="readme-writing-compiler-tests.html">
-           readme-writing-compiler-tests.html</a>.
            
-<h3><a name="#product"></a>Product</h3>
+<h3><a name="product"></a>Product</h3>
 Product tests target a built distribution or do compiler (argument) 
 testing not supported by the harness.
 Many have been superceded by ant tests.
+To build aspectj with itself, see
+<a href="product/build-aspectj/build.xml">
+         product/build-aspectj/build.xml</a>.
 
-<h3><a name="#ajctest"></a>AjcTest</h3>
+<h3><a name="ajctest"></a>AjcTest</h3>
    <a href="ajctest.xml">ajctest.xml</a>
    is an Ant script that compiles and runs large systems.
    The scripts tell the AjcTest task from the testing module to drive
@@ -150,14 +173,14 @@ Many have been superceded by ant tests.
    <!-- XXX install and correct -->
    
    
-<h3><a name="#jacks"></a>Jacks</h3>
+<h3><a name="jacks"></a>Jacks</h3>
 Jacks is the Jikes compiler test suite available at the 
     <a href="http://www-124.ibm.com/developerworks/oss/cvs/jikes/~checkout~/jacks/jacks.html">
        Jacks project page</a>.
 
 We have a bash script to run it using ajc and javac and compare
        the results.
-<h3><a name="#junit"></a>JUnit</h3>
+<h3><a name="junit"></a>JUnit</h3>
 The JUnit tests should be self-documenting.
 For more information on running them, see
 <a href="../build/readme-build-and-test-aspectj.html">