From 9bf93f948ce3413eddcb4fffad6daed5a416f628 Mon Sep 17 00:00:00 2001 From: wisberg Date: Sun, 25 May 2003 01:37:49 +0000 Subject: [PATCH] updating harness documentation for new features --- .../org/aspectj/testing/drivers/package.html | 59 ++++++++++++++++--- tests/readme-tests-module.html | 57 ++++++++++++------ 2 files changed, 92 insertions(+), 24 deletions(-) diff --git a/testing-drivers/src/org/aspectj/testing/drivers/package.html b/testing-drivers/src/org/aspectj/testing/drivers/package.html index 1bdc35634..6bd1738db 100644 --- a/testing-drivers/src/org/aspectj/testing/drivers/package.html +++ b/testing-drivers/src/org/aspectj/testing/drivers/package.html @@ -69,7 +69,12 @@ each component in the chain may accept and interpret: -traceTestsMin -hideStreams and !eclipse, used to emit tests results in a form - comparable by org.aspectj.testing.util.TestDiffs. + comparable by org.aspectj.testing.util.TestDiffs. + Note that output in the form emitted by -traceTestsMin + is input for the options to select tests by title + (e.g., -ajctestTitleList) + and for the tool which compare results from different test runs + (org.aspectj.testing.util.TestDiffs). output: -hide{Compiler|Run}Streams will prevent output and @@ -95,8 +100,8 @@ each component in the chain may accept and interpret: hide streams for passing tests but emit them for failing tests. -AjcTest -

selection options for keywords, bugID (PR) +AjcTest +

selection options for keywords, bugID (PR), or title (description) keywords: -ajctest[Require|Skip]Keywords=one{,two} will either require or skip tests that have the specified keywords. @@ -104,6 +109,32 @@ each component in the chain may accept and interpret: Bugs: -ajctestPR=101{,102} will run only tests that are associated with one of the bug id's listed. + title: + "-ajctestTitleContains=one,two" + will require that the title (description) of the test contain + either "one" or "two". + Use this to select a few tests you know generally. +
+ "-ajctestTitleList=../tests/ajcTestResults.txt" + will require that the title (description) of the test be + equal to one listed in ../tests/ajcTestResults.txt + as a line of the form "[PASS|FAIL] {title}(.." + (emitted by the -traceTestsMin option). + Use this to re-run a set of tests. +
+ "-ajctestTitleList=first title\, in theory, second title" + will require that the title (description) of the test be + exactly "first title, in theory" or "second title". + The entire option must be one argument on the command line. + This option only differs from the prior in not specifying + a valid file to read. + Use this when working with just a few specific tests. +
+ "-ajctestTitleFailList=../tests/ajcTestResults.txt" + is the same as the -ajctestTitleList=.. variant, + except that only results prefixed "FAIL" are included. + Use this to re-run only the tests that failed from a large set. + CompilerRun

compiler options and side-effects @@ -208,6 +239,16 @@ Following are some sample configurations: (tests skipped, incomplete, failed, passed).

+

  • java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt +

    This writes to result.txt one line [PASS|FAIL] per test, plus a + 1-line summary of the test results.

    +

    • + +
  • java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt +

    This re-runs any test that failed from the "results.txt" run, + verbosely logging any fails.

    +

    • +
  • java {harness} -hideStreams -logMinFail {suiteFile}

    Use this when running tests mainly to see if they pass or if the failure messages are typically enough information @@ -222,7 +263,8 @@ Following are some sample configurations:

  • java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile} -

    This would do four complete runs with the old (Ajc 1.0) compiler: one with +

    Because of the trailing '-' on two of the options, + this would do four complete runs with the old (Ajc 1.0) compiler: one with no options, one with -lenient, one with -Xlint, and one with both.

    • @@ -238,7 +280,7 @@ If you have a set of options you use often, you can define a single-word option alias for it; see Harness.optionAliases.

    Configuration: Most tests use the library jars in -aspectj/modules/lib/test, defined in +modules/lib/test, defined in org.aspectj.testing.harness.bridge.Globals. Normally the harness finds these by relative path ../lib/tests/*.jar, which works whenever the tests are @@ -254,8 +296,10 @@ The harness must be run in a compiler-compatible VM, and the compiler steps run in-process. However, the java steps can be run in forked mode, which is useful when compiling for a VM which can't run the compiler. -Here's how you would run the harness to compile against a -specific version of 1.1: +To compile for a different target VM could require +setting the options for bootclasspath, target, and source. +To run the harness so that any <run.. tasks run in a +separate vm, do something like this: 

        java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
         -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
@@ -279,5 +323,6 @@ to the classpath, with an effect like these commands
      {mainClass} {option..}
    + diff --git a/tests/readme-tests-module.html b/tests/readme-tests-module.html index c67a20bd7..bb47df117 100644 --- a/tests/readme-tests-module.html +++ b/tests/readme-tests-module.html @@ -76,8 +76,9 @@ Tests differ in the kind of harness they use and their target. -

    Harness

    +

    Harness

    +

    Harness tests

    These are compiler test cases, the preferred form for any compiler tests. The XML files are test definitions, and the subdirectories contain the test sources (many of which @@ -108,6 +109,25 @@ segregated for convenience as follows: These are tests to exercise the test harness itself. +The tests specify most anything one could want to do with +compile/run variants. The best summary of the test specification +format is the ajcTestSuite.dtd. + +

    +To write a new test, evolve an existing example. See the dtd and/or +the javadoc for the testing harness implementation in the +testing module for more details. + +

    To write a test case for a bug, see + + readme-writing-compiler-tests.html. + Many bugs that first present in an IDE or the Ant taskdef + are compiler bugs that can be written in this form. + We much prefer this automated form over + manual steps specified in a bug report. + +

    Running the harness

    +

    To run the harness, use org.aspectj.testing.drivers.Harness, in the testing-drivers module. Since that is the main class for the testing-drivers module, you can build and @@ -119,27 +139,30 @@ run the harness as follows: java -jar ../aj-build/jars/testing-drivers-all.jar ajcTests.xml -For help on harness options, use -help. -For more help on options for harness components, see +The harness enable you to +

    +For a listing of harness options, use -help. +For a fuller description of harness options and components, see the package documentation for the harness. -Note in particular how to select tests using keywords -and how compiler arguments can be passed on the harness command line. -

    -To write a new test, evolve an existing example. See the dtd and/or -the javadoc for the testing harness implementation in the -testing module for more details. - -

    To write a test case for a bug, see - - readme-writing-compiler-tests.html. -

    Product

    +

    Product

    Product tests target a built distribution or do compiler (argument) testing not supported by the harness. Many have been superceded by ant tests. +To build aspectj with itself, see + + product/build-aspectj/build.xml. -

    AjcTest

    +

    AjcTest

    ajctest.xml is an Ant script that compiles and runs large systems. The scripts tell the AjcTest task from the testing module to drive @@ -150,14 +173,14 @@ Many have been superceded by ant tests. -

    Jacks

    +

    Jacks

    Jacks is the Jikes compiler test suite available at the Jacks project page. We have a bash script to run it using ajc and javac and compare the results. -

    JUnit

    +

    JUnit

    The JUnit tests should be self-documenting. For more information on running them, see -- 2.39.5