diff options
| -rw-r--r-- | testing-drivers/src/org/aspectj/testing/drivers/package.html | 59 | ||||
| -rw-r--r-- | tests/readme-tests-module.html | 57 |
2 files changed, 92 insertions, 24 deletions
diff --git a/testing-drivers/src/org/aspectj/testing/drivers/package.html b/testing-drivers/src/org/aspectj/testing/drivers/package.html index 1bdc35634..6bd1738db 100644 --- a/testing-drivers/src/org/aspectj/testing/drivers/package.html +++ b/testing-drivers/src/org/aspectj/testing/drivers/package.html @@ -69,7 +69,12 @@ each component in the chain may accept and interpret: <code>-traceTestsMin</code> <code>-hideStreams</code> and <code>!eclipse</code>, used to emit tests results in a form - comparable by <code>org.aspectj.testing.util.TestDiffs</code>. + comparable by <code>org.aspectj.testing.util.TestDiffs</code>. + Note that output in the form emitted by <code>-traceTestsMin</code> + is input for the options to select tests by title + (e.g., <code>-ajctestTitleList</code>) + and for the tool which compare results from different test runs + (<code>org.aspectj.testing.util.TestDiffs</code>). </td></tr> <tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and @@ -95,8 +100,8 @@ each component in the chain may accept and interpret: hide streams for passing tests but emit them for failing tests. </td></tr> -<tr><td rowspan="3" valign="top">AjcTest - <p>selection options for keywords, bugID (PR) +<tr><td rowspan="4" valign="top">AjcTest + <p>selection options for keywords, bugID (PR), or title (description) </td></tr> <tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code> will either require or skip tests that have the specified keywords. @@ -104,6 +109,32 @@ each component in the chain may accept and interpret: <tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code> will run only tests that are associated with one of the bug id's listed. </td></tr> + <tr><td><u>title</u>: + <code>"-ajctestTitleContains=one,two"</code> + will require that the title (description) of the test contain + either "one" or "two". + Use this to select a few tests you know generally. + <br> + <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code> + will require that the title (description) of the test be + equal to one listed in <code>../tests/ajcTestResults.txt</code> + as a line of the form "[PASS|FAIL] {title}(.." + (emitted by the <code>-traceTestsMin</code> option). + Use this to re-run a set of tests. + <br> + <code>"-ajctestTitleList=first title\, in theory, second title"</code> + will require that the title (description) of the test be + exactly "first title, in theory" or "second title". + The entire option must be one argument on the command line. + This option only differs from the prior in not specifying + a valid file to read. + Use this when working with just a few specific tests. + <br> + <code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code> + is the same as the <code>-ajctestTitleList=..</code> variant, + except that only results prefixed "FAIL" are included. + Use this to re-run only the tests that failed from a large set. + </td></tr> <tr><td rowspan="6" valign="top">CompilerRun <p>compiler options and side-effects @@ -208,6 +239,16 @@ Following are some sample configurations: (tests skipped, incomplete, failed, passed).<p> </li> +<li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code> + <p>This writes to result.txt one line [PASS|FAIL] per test, plus a + 1-line summary of the test results.<p> + </li> + +<li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code> + <p>This re-runs any test that failed from the "results.txt" run, + verbosely logging any fails.<p> + </li> + <li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code> <p>Use this when running tests mainly to see if they pass or if the failure messages are typically enough information @@ -222,7 +263,8 @@ Following are some sample configurations: </li> <li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code> - <p>This would do four complete runs with the old (Ajc 1.0) compiler: one with + <p>Because of the trailing '-' on two of the options, + this would do four complete runs with the old (Ajc 1.0) compiler: one with no options, one with -lenient, one with -Xlint, and one with both.<p> </li> @@ -238,7 +280,7 @@ If you have a set of options you use often, you can define a single-word option alias for it; see <code>Harness.optionAliases</code>. <p><u>Configuration</u>: Most tests use the library jars in -<code>aspectj/modules/lib/test</code>, defined in +<code>modules/lib/test</code>, defined in <code>org.aspectj.testing.harness.bridge.Globals</code>. Normally the harness finds these by relative path <code>../lib/tests/*.jar</code>, which works whenever the tests are @@ -254,8 +296,10 @@ The harness must be run in a compiler-compatible VM, and the compiler steps run in-process. However, the java steps can be run in forked mode, which is useful when compiling for a VM which can't run the compiler. -Here's how you would run the harness to compile against a -specific version of 1.1: +To compile for a different target VM could require +setting the options for bootclasspath, target, and source. +To run the harness so that any <run.. tasks run in a +separate vm, do something like this: <pre> java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \ -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \ @@ -279,5 +323,6 @@ to the classpath, with an effect like these commands {mainClass} {option..} </pre> + </body> </html> diff --git a/tests/readme-tests-module.html b/tests/readme-tests-module.html index c67a20bd7..bb47df117 100644 --- a/tests/readme-tests-module.html +++ b/tests/readme-tests-module.html @@ -76,8 +76,9 @@ Tests differ in the kind of harness they use and their target. </tr> </table> -<h3><a name="#harness"></a>Harness</h3> +<h3><a name="harness"></a>Harness</h3> +<h4><a name="harness-tests"></a>Harness tests</h4> These are compiler test cases, the preferred form for any compiler tests. The XML files are test definitions, and the subdirectories contain the test sources (many of which @@ -108,6 +109,25 @@ segregated for convenience as follows: These are tests to exercise the test harness itself. </li> </ul> +The tests specify most anything one could want to do with +compile/run variants. The best summary of the test specification +format is the <a href="ajcTestSuite.dtd">ajcTestSuite.dtd</a>. + +<p> +To write a new test, evolve an existing example. See the dtd and/or +the javadoc for the testing harness implementation in the +<code>testing</code> module for more details. + +<p>To write a test case for a bug, see + <a href="readme-writing-compiler-tests.html"> + readme-writing-compiler-tests.html</a>. + Many bugs that first present in an IDE or the Ant taskdef + are compiler bugs that can be written in this form. + We <em>much</em> prefer this automated form over + manual steps specified in a bug report. + +<h4><a name="harness-runs"></a>Running the harness</h4> +<p> To run the harness, use <code>org.aspectj.testing.drivers.Harness</code>, in the <code>testing-drivers</code> module. Since that is the main class for the <code>testing-drivers</code> module, you can build and @@ -119,27 +139,30 @@ run the harness as follows: java -jar ../aj-build/jars/testing-drivers-all.jar ajcTests.xml </pre> -For help on harness options, use <code>-help</code>. -For more help on options for harness components, see +The harness enable you to +<ul> + <li>control output, hiding streams and logging messages</li> + <li>select tests using keywords, bug ID's, or titles</li> + <li>pass compiler arguments on the harness command line</li> + <li>use different compiler wrappers - the command-line compiler, + the AJDE IDE support (mostly sans GUI), or + the Ant taskdef support.</li> + <li>fork java run steps</li> +</ul> +For a listing of harness options, use <code>-help</code>. +For a fuller description of harness options and components, see <a href="../testing-drivers/src/org/aspectj/testing/drivers/package.html"> the package documentation for the harness</a>. -Note in particular how to select tests using keywords -and how compiler arguments can be passed on the harness command line. -<p> -To write a new test, evolve an existing example. See the dtd and/or -the javadoc for the testing harness implementation in the -<code>testing</code> module for more details. - -<p>To write a test case for a bug, see - <a href="readme-writing-compiler-tests.html"> - readme-writing-compiler-tests.html</a>. -<h3><a name="#product"></a>Product</h3> +<h3><a name="product"></a>Product</h3> Product tests target a built distribution or do compiler (argument) testing not supported by the harness. Many have been superceded by ant tests. +To build aspectj with itself, see +<a href="product/build-aspectj/build.xml"> + product/build-aspectj/build.xml</a>. -<h3><a name="#ajctest"></a>AjcTest</h3> +<h3><a name="ajctest"></a>AjcTest</h3> <a href="ajctest.xml">ajctest.xml</a> is an Ant script that compiles and runs large systems. The scripts tell the AjcTest task from the testing module to drive @@ -150,14 +173,14 @@ Many have been superceded by ant tests. <!-- XXX install and correct --> -<h3><a name="#jacks"></a>Jacks</h3> +<h3><a name="jacks"></a>Jacks</h3> Jacks is the Jikes compiler test suite available at the <a href="http://www-124.ibm.com/developerworks/oss/cvs/jikes/~checkout~/jacks/jacks.html"> Jacks project page</a>. We have a bash script to run it using ajc and javac and compare the results. -<h3><a name="#junit"></a>JUnit</h3> +<h3><a name="junit"></a>JUnit</h3> The JUnit tests should be self-documenting. For more information on running them, see <a href="../build/readme-build-and-test-aspectj.html"> |