<code>-traceTestsMin</code>
<code>-hideStreams</code> and
<code>!eclipse</code>, used to emit tests results in a form
- comparable by <code>org.aspectj.testing.util.TestDiffs</code>.
+ comparable by <code>org.aspectj.testing.util.TestDiffs</code>.
+ Note that output in the form emitted by <code>-traceTestsMin</code>
+ is input for the options to select tests by title
+ (e.g., <code>-ajctestTitleList</code>)
+ and for the tool which compare results from different test runs
+ (<code>org.aspectj.testing.util.TestDiffs</code>).
</td></tr>
<tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and
hide streams for passing tests but emit them for failing tests.
</td></tr>
-<tr><td rowspan="3" valign="top">AjcTest
- <p>selection options for keywords, bugID (PR)
+<tr><td rowspan="4" valign="top">AjcTest
+ <p>selection options for keywords, bugID (PR), or title (description)
</td></tr>
<tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code>
will either require or skip tests that have the specified keywords.
<tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code>
will run only tests that are associated with one of the bug id's listed.
</td></tr>
+ <tr><td><u>title</u>:
+ <code>"-ajctestTitleContains=one,two"</code>
+ will require that the title (description) of the test contain
+ either "one" or "two".
+ Use this to select a few tests you know generally.
+ <br>
+ <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
+ will require that the title (description) of the test be
+ equal to one listed in <code>../tests/ajcTestResults.txt</code>
+ as a line of the form "[PASS|FAIL] {title}(.."
+ (emitted by the <code>-traceTestsMin</code> option).
+ Use this to re-run a set of tests.
+ <br>
+ <code>"-ajctestTitleList=first title\, in theory, second title"</code>
+ will require that the title (description) of the test be
+ exactly "first title, in theory" or "second title".
+ The entire option must be one argument on the command line.
+ This option only differs from the prior in not specifying
+ a valid file to read.
+ Use this when working with just a few specific tests.
+ <br>
+ <code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code>
+ is the same as the <code>-ajctestTitleList=..</code> variant,
+ except that only results prefixed "FAIL" are included.
+ Use this to re-run only the tests that failed from a large set.
+ </td></tr>
<tr><td rowspan="6" valign="top">CompilerRun
<p>compiler options and side-effects
(tests skipped, incomplete, failed, passed).<p>
</li>
+<li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code>
+ <p>This writes to result.txt one line [PASS|FAIL] per test, plus a
+ 1-line summary of the test results.<p>
+ </li>
+
+<li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code>
+ <p>This re-runs any test that failed from the "results.txt" run,
+ verbosely logging any fails.<p>
+ </li>
+
<li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code>
<p>Use this when running tests mainly to see if they pass or
if the failure messages are typically enough information
</li>
<li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code>
- <p>This would do four complete runs with the old (Ajc 1.0) compiler: one with
+ <p>Because of the trailing '-' on two of the options,
+ this would do four complete runs with the old (Ajc 1.0) compiler: one with
no options, one with -lenient, one with -Xlint, and one with both.<p>
</li>
option alias for it; see <code>Harness.optionAliases</code>.
<p><u>Configuration</u>: Most tests use the library jars in
-<code>aspectj/modules/lib/test</code>, defined in
+<code>modules/lib/test</code>, defined in
<code>org.aspectj.testing.harness.bridge.Globals</code>.
Normally the harness finds these by relative path
<code>../lib/tests/*.jar</code>, which works whenever the tests are
compiler steps run in-process.
However, the java steps can be run in forked mode, which is useful
when compiling for a VM which can't run the compiler.
-Here's how you would run the harness to compile against a
-specific version of 1.1:
+To compile for a different target VM could require
+setting the options for bootclasspath, target, and source.
+To run the harness so that any <run.. tasks run in a
+separate vm, do something like this:
<pre>
java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
-Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
{mainClass} {option..}
</pre>
+
</body>
</html>
</tr>
</table>
-<h3><a name="#harness"></a>Harness</h3>
+<h3><a name="harness"></a>Harness</h3>
+<h4><a name="harness-tests"></a>Harness tests</h4>
These are compiler test cases, the preferred form for
any compiler tests. The XML files are test definitions,
and the subdirectories contain the test sources (many of which
These are tests to exercise the test harness itself.
</li>
</ul>
+The tests specify most anything one could want to do with
+compile/run variants. The best summary of the test specification
+format is the <a href="ajcTestSuite.dtd">ajcTestSuite.dtd</a>.
+
+<p>
+To write a new test, evolve an existing example. See the dtd and/or
+the javadoc for the testing harness implementation in the
+<code>testing</code> module for more details.
+
+<p>To write a test case for a bug, see
+ <a href="readme-writing-compiler-tests.html">
+ readme-writing-compiler-tests.html</a>.
+ Many bugs that first present in an IDE or the Ant taskdef
+ are compiler bugs that can be written in this form.
+ We <em>much</em> prefer this automated form over
+ manual steps specified in a bug report.
+
+<h4><a name="harness-runs"></a>Running the harness</h4>
+<p>
To run the harness, use <code>org.aspectj.testing.drivers.Harness</code>,
in the <code>testing-drivers</code> module. Since that is the main class
for the <code>testing-drivers</code> module, you can build and
java -jar ../aj-build/jars/testing-drivers-all.jar ajcTests.xml
</pre>
-For help on harness options, use <code>-help</code>.
-For more help on options for harness components, see
+The harness enable you to
+<ul>
+ <li>control output, hiding streams and logging messages</li>
+ <li>select tests using keywords, bug ID's, or titles</li>
+ <li>pass compiler arguments on the harness command line</li>
+ <li>use different compiler wrappers - the command-line compiler,
+ the AJDE IDE support (mostly sans GUI), or
+ the Ant taskdef support.</li>
+ <li>fork java run steps</li>
+</ul>
+For a listing of harness options, use <code>-help</code>.
+For a fuller description of harness options and components, see
<a href="../testing-drivers/src/org/aspectj/testing/drivers/package.html">
the package documentation for the harness</a>.
-Note in particular how to select tests using keywords
-and how compiler arguments can be passed on the harness command line.
-<p>
-To write a new test, evolve an existing example. See the dtd and/or
-the javadoc for the testing harness implementation in the
-<code>testing</code> module for more details.
-
-<p>To write a test case for a bug, see
- <a href="readme-writing-compiler-tests.html">
- readme-writing-compiler-tests.html</a>.
-<h3><a name="#product"></a>Product</h3>
+<h3><a name="product"></a>Product</h3>
Product tests target a built distribution or do compiler (argument)
testing not supported by the harness.
Many have been superceded by ant tests.
+To build aspectj with itself, see
+<a href="product/build-aspectj/build.xml">
+ product/build-aspectj/build.xml</a>.
-<h3><a name="#ajctest"></a>AjcTest</h3>
+<h3><a name="ajctest"></a>AjcTest</h3>
<a href="ajctest.xml">ajctest.xml</a>
is an Ant script that compiles and runs large systems.
The scripts tell the AjcTest task from the testing module to drive
<!-- XXX install and correct -->
-<h3><a name="#jacks"></a>Jacks</h3>
+<h3><a name="jacks"></a>Jacks</h3>
Jacks is the Jikes compiler test suite available at the
<a href="http://www-124.ibm.com/developerworks/oss/cvs/jikes/~checkout~/jacks/jacks.html">
Jacks project page</a>.
We have a bash script to run it using ajc and javac and compare
the results.
-<h3><a name="#junit"></a>JUnit</h3>
+<h3><a name="junit"></a>JUnit</h3>
The JUnit tests should be self-documenting.
For more information on running them, see
<a href="../build/readme-build-and-test-aspectj.html">
projects / aspectj.git / commitdiff