Harness Package Documentation

The AspectJ compiler test harness can compile and run AspectJ programs as specified by the test definitions. This document tells you how to run the harness. It describes the options you can specify on the command-line to control various components that form the harness, either to specify options that augment the test definitions or to change how the harness works, e.g., selecting particular tests or logging more information. For information on how to write a test definition, see readme-writing-compiler-tests.html in the testing module.

The harness drives compiler tests, using a chain of responsibility to map elements in the schema of a test definition to implementing classes.

Test feature	Description	Implementing class
(loading suites...)	general harness	Harness
(logging...)	subclass feature harness	FeatureHarness
`<suite>`	Test suite	AjcTest.Suite
`<ajc-test>`	Test case	AjcTest
`<compile>`	Initial (batch) compile run	CompilerRun
`<inc-compile>`	Incremental re-compile	IncCompilerRun
`<run>`	Run class	JavaRun

The compiler used is the AspectJ compiler ajc (optionally as wrapped by the Ant task or AJDE API's), but in principle any compiler accepting similar options can be used.

To run from the command-line, use Harness.main(String[]). To run programmatically, use Harness.getHarness(). Harness.runMain(String[]) takes arguments that each component in the chain may accept and interpret, so you can modify how the tests run by specifying the following arguments on the harness command line:

Component Options

Harness
suite files, harness verbosity, temp files, option variants

suite files: ajcTest-compliant .txt or .xml files are accepted.

-verboseHarness, -quietHarness: Log accepted options and skipped tests, or do not print even info messages.

-keepTemp: Normally the harness saves temp files until the end of the run, and deletes them. If you abort the run or specify -keepTemp, then temporary (sandbox) directories will remain for analysis. In either case, the file system accumulates all temporary directories and files used for a give harness run; for the ajcTests.xml suite, this runs into thousands of files.

-killTemp: The opposite of -keepTemp, this causes the harness to delete temporary (sandbox) directories at the end of each test run. In this case, the file system only accumulates files for the current test.

*- variants: Options with a trailing "-" cause two sets of option lists to be produced, one with and one without the corresponding option. E.g., "-emacssym-" will run the suite twice, once with and once without the "-emacssym" flag. That means if you use this on each of three options, you will get 8 variant sets (1 with no options, 1 with all 3 options, 3 with 2 options, and 3 with 1 option).

FeatureHarness
output and logging options

tracing: -progressDots will print "." for every passed test completed and "!" for every test completed but not passed. -traceTests will print a one-line summary for each test of the time and space taken and whether the test passed. -traceTestsMin will print only the test and whether it passed. -baseline is an alias for -traceTestsMin -hideStreams and !eclipse, used to emit tests results in a form comparable by org.aspectj.testing.util.TestDiffs. or usable to select tests by title for options like -ajctestTitleList.

output: -hide{Compiler|Run}Streams will prevent output and error streams from being printed to System.err and System.out, optionally only for run or compile steps.

logging: Log variants take the form -log{Min|Xml}[Fail|Pass|All]. The suffix {All|Pass|Fail} selects all tests or only passing or failing tests. The infix {Min} means to log with minimal information, typically only any fail messages. The infix {Xml} means to log the XML form of the test definition, so that you can inspect the input or re-run arbitrary tests. (You can also re-run a set of tests using keywords (e.g., "-ajctestsRequireKeywords=..." or using titles (e.g., "-ajctestsTitleFailList=ajcTestResults.txt".) Finally, the experimental option -XlogPublicType will log the XML test definition for any test run that emits any ERROR messages containing the text "public type".

interaction of output streams and logging: Streams will be emitted in real-time, before the test is logged, unless streams are hidden. When logging in normal (non-Min or -XML) form, the log will emit the streams with the test report, so e.g., you can use -hideStreams -logFail to hide streams for passing tests but emit them for failing tests in the context of the log.

AjcTest
selection options for keywords, bugID (PR), or title (description)

keywords: -ajctest[Require|Skip]Keywords=one{,two} will either require or skip a test that has one of the specified keywords.

Bugs: -ajctestPR=101{,102} will require that a test have one of the listed bug id's.

title: "-ajctestTitleContains=one,two" will require that the title (description) of a test contain one of the specified substrings, here either "one" or "two". Use this to select a few tests you know generally.
"-ajctestTitleList=first title\, in theory, second title" will require that the title (description) of a test be exactly "first title, in theory" or "second title". The entire option must be one argument on the command line. Use this when working with just a few specific tests.
"-ajctestTitleList=../tests/ajcTestResults.txt" will require that the title (description) of a test be equal to one listed in ../tests/ajcTestResults.txt as a line of the form "[PASS|FAIL] {title}(.." (This form is emitted by the -traceTestsMin option). This option only differs from the prior in that the parameter is a valid file to read. Use this to re-run a large set of tests.
"-ajctestTitleFailList=../tests/ajcTestResults.txt" is the same as the -ajctestTitleList={file} variant, except that only results prefixed "FAIL" are included. Use this to re-run only the tests that failed from a large set.

Combinations: all selectors are applied to each test, so all tests selected will comply with all constraints. Specifying lists within a particular constraints will match a union of tests for that constraint (e.g., all tests with bug id's 101 or 102), but there is no way to get a union of constraints (e.g., all tests with bug id's 101 or 102 or with keywords pure-java or knownLimitation). However, -ajctestSkipKeywords=... can return all tests without the specified keywords, so it can form unions like "all tests without the knownLimitation keyword, but with bug id's 101 or 102". Title lists can work similarly. E.g., to run the failed incremental tests from ajcTestResults.txt, specify -ajctestTitleFailList=../tests/ajcTestResults.txt -ajctestRequireKeywords=incremental-test.

CompilerRun
compiler options and side-effects

supported options: The command line passed to the compiler by CompilerRun is composed of entries derived from the <compile> attributes and recognized from the harness command line. <compile> has specific attributes like files, argfiles, classpath and sourceroot which translate directly to their counterparts. The output option -d is defined by CompilerRun and may not be specified (and -outjar is not supported). Most other compiler options are defined in CompilerRun.Spec.CRSOptions and may be specified on the harness command-line or in the options attribute of <compile>. In the options attribute, each argument is comma-delimited, so an option with an argument would look like <compile options="-source,1.4" ...>. If options collide, duplicates can be resolved using option dominance (below).

compiler selectors: Use -ajc or -eclipse to select the old (ajc 1.0) or new (eajc 1.1) compilers. Note that the old compiler is not available in the CVS source tree at eclipse.org. Use -ajdeCompiler to run a wrapper around the AJDE interface and -ajctaskCompiler to run a wrapper around the AjcTask (Ant task) interface.

option dominance [-|!|^]: Some tests require or prohibit certain options; likewise, sometime you want to force all tests run with or without an option specified on the command-line, regardless of its setting in the <compile options=".." ...> attribute. For this reason an option may be specified in the options attribute or on the harness command-line as -option, !option, or ^option.

- set: If the leading character of an option is "-", then it is set unless forced-off.

^ force-off: If the leading character of an option is "^", then it is forced off. Any other matching option will be removed.

! force-on: If the leading character of an option is "!", then it is forced on. Any other non-force-on matching option will be removed.

force conflict: If there are two matching force-on options, the test is skipped.

related options: Two options match if they are the same or if they are in the same family. For example, -ajc and eclipse are both compiler, and -source 1.4 and -source 1.3 are both source.

auto-skip: After combining global and local options, there may be conflicting or impossible options, which cause the test to be skipped:

semantic conflicts: two options may conflict in meaning - e.g., -lenient and -strict

impossible option: It may not be possible in the current configuration to implement an option - e.g., -usejavac or -eclipse when javac or the eclipse implementation is not on the classpath

source searching: Given -seek:{literal}, as a side-effect, CompilerRun will search source files for {literal}, emitting for each instance an INFO message of the form: found: {file}:{line}:{column} (Note that the harness does not display INFO messages unless -verboseHarness or -loud is used.)

JavaRun
Options and forking
options: options specified in the test are passed to the main method as they would be on the command-line. No options passed to the harness are passed through to the main class.

forking: Forking is useful to run in a different version of Java than can be supported by the harness (i.e., some 1.1 flavor); it's very time-consuming otherwise. Currently forking is only controllable through system properties of the invoking vm (defined in JavaRun.java):

javarun.fork: anything to run in a new vm.

javarun.java: path to the java executable to run (suffix included). If not supplied, the harness tries to find the java that invoked the harness.

javarun.java.home: the value of the JAVA_HOME environment variable, if it needs to be set.

javarun.bootclasspath: this is prepended to the run classpath. Multiple entries must be separated by the system-dependent path separator.

javarun.vmargs: this is added to the fork command-line right after java. Multiple entries must be separated by a comma (and the whole thing should be one parameter), e.g., -Djavarun.vmargs=-Dname=value,-Dname2="value 2"

Following are some sample configurations:

Component	Options
Harness suite files, harness verbosity, temp files, option variants
suite files: ajcTest-compliant .txt or .xml files are accepted.
`-verboseHarness`, `-quietHarness`: Log accepted options and skipped tests, or do not print even info messages.
`-keepTemp`: Normally the harness saves temp files until the end of the run, and deletes them. If you abort the run or specify `-keepTemp`, then temporary (sandbox) directories will remain for analysis. In either case, the file system accumulates all temporary directories and files used for a give harness run; for the `ajcTests.xml` suite, this runs into thousands of files.
`-killTemp`: The opposite of `-keepTemp`, this causes the harness to delete temporary (sandbox) directories at the end of each test run. In this case, the file system only accumulates files for the current test.
*- variants: Options with a trailing "-" cause two sets of option lists to be produced, one with and one without the corresponding option. E.g., "-emacssym-" will run the suite twice, once with and once without the "-emacssym" flag. That means if you use this on each of three options, you will get 8 variant sets (1 with no options, 1 with all 3 options, 3 with 2 options, and 3 with 1 option).
FeatureHarness output and logging options
tracing: `-progressDots` will print "." for every passed test completed and "!" for every test completed but not passed. `-traceTests` will print a one-line summary for each test of the time and space taken and whether the test passed. `-traceTestsMin` will print only the test and whether it passed. `-baseline` is an alias for `-traceTestsMin` `-hideStreams` and `!eclipse`, used to emit tests results in a form comparable by `org.aspectj.testing.util.TestDiffs`. or usable to select tests by title for options like `-ajctestTitleList`.
output: `-hide{Compiler\|Run}Streams` will prevent output and error streams from being printed to System.err and System.out, optionally only for run or compile steps.
logging: Log variants take the form `-log{Min\|Xml}[Fail\|Pass\|All]`. The suffix {All\|Pass\|Fail} selects all tests or only passing or failing tests. The infix {Min} means to log with minimal information, typically only any fail messages. The infix {Xml} means to log the XML form of the test definition, so that you can inspect the input or re-run arbitrary tests. (You can also re-run a set of tests using keywords (e.g., "`-ajctestsRequireKeywords=...`" or using titles (e.g., "`-ajctestsTitleFailList=ajcTestResults.txt`".) Finally, the experimental option `-XlogPublicType` will log the XML test definition for any test run that emits any ERROR messages containing the text "public type".
interaction of output streams and logging: Streams will be emitted in real-time, before the test is logged, unless streams are hidden. When logging in normal (non-Min or -XML) form, the log will emit the streams with the test report, so e.g., you can use -hideStreams -logFail to hide streams for passing tests but emit them for failing tests in the context of the log.
AjcTest selection options for keywords, bugID (PR), or title (description)
keywords: `-ajctest[Require\|Skip]Keywords=one{,two}` will either require or skip a test that has one of the specified keywords.
Bugs: `-ajctestPR=101{,102}` will require that a test have one of the listed bug id's.
title: `"-ajctestTitleContains=one,two"` will require that the title (description) of a test contain one of the specified substrings, here either "one" or "two". Use this to select a few tests you know generally. `"-ajctestTitleList=first title\, in theory, second title"` will require that the title (description) of a test be exactly "first title, in theory" or "second title". The entire option must be one argument on the command line. Use this when working with just a few specific tests. `"-ajctestTitleList=../tests/ajcTestResults.txt"` will require that the title (description) of a test be equal to one listed in `../tests/ajcTestResults.txt` as a line of the form "[PASS\|FAIL] {title}(.." (This form is emitted by the `-traceTestsMin` option). This option only differs from the prior in that the parameter is a valid file to read. Use this to re-run a large set of tests. `"-ajctestTitleFailList=../tests/ajcTestResults.txt"` is the same as the `-ajctestTitleList={file}` variant, except that only results prefixed "FAIL" are included. Use this to re-run only the tests that failed from a large set.
Combinations: all selectors are applied to each test, so all tests selected will comply with all constraints. Specifying lists within a particular constraints will match a union of tests for that constraint (e.g., all tests with bug id's 101 or 102), but there is no way to get a union of constraints (e.g., all tests with bug id's 101 or 102 or with keywords pure-java or knownLimitation). However, `-ajctestSkipKeywords=...` can return all tests without the specified keywords, so it can form unions like "all tests without the knownLimitation keyword, but with bug id's 101 or 102". Title lists can work similarly. E.g., to run the failed incremental tests from ajcTestResults.txt, specify `-ajctestTitleFailList=../tests/ajcTestResults.txt` `-ajctestRequireKeywords=incremental-test`.
CompilerRun compiler options and side-effects
supported options: The command line passed to the compiler by `CompilerRun` is composed of entries derived from the `<compile>` attributes and recognized from the harness command line. `<compile>` has specific attributes like `files`, `argfiles`, `classpath` and `sourceroot` which translate directly to their counterparts. The output option `-d` is defined by `CompilerRun` and may not be specified (and `-outjar` is not supported). Most other compiler options are defined in `CompilerRun.Spec.CRSOptions` and may be specified on the harness command-line or in the `options` attribute of `<compile>`. In the `options` attribute, each argument is comma-delimited, so an option with an argument would look like `<compile options="-source,1.4" ...>`. If options collide, duplicates can be resolved using option dominance (below).
compiler selectors: Use `-ajc` or `-eclipse` to select the old (ajc 1.0) or new (eajc 1.1) compilers. Note that the old compiler is not available in the CVS source tree at eclipse.org. Use `-ajdeCompiler` to run a wrapper around the AJDE interface and `-ajctaskCompiler` to run a wrapper around the AjcTask (Ant task) interface.
option dominance `[-\|!\|^]`: Some tests require or prohibit certain options; likewise, sometime you want to force all tests run with or without an option specified on the command-line, regardless of its setting in the `<compile options=".." ...>` attribute. For this reason an option may be specified in the options attribute or on the harness command-line as `-option`, `!option`, or `^option`. - set: If the leading character of an option is "-", then it is set unless forced-off. ^ force-off: If the leading character of an option is "^", then it is forced off. Any other matching option will be removed. ! force-on: If the leading character of an option is "!", then it is forced on. Any other non-force-on matching option will be removed. force conflict: If there are two matching force-on options, the test is skipped. related options: Two options match if they are the same or if they are in the same family. For example, `-ajc` and `eclipse` are both compiler, and `-source 1.4` and `-source 1.3` are both source.
auto-skip: After combining global and local options, there may be conflicting or impossible options, which cause the test to be skipped: semantic conflicts: two options may conflict in meaning - e.g., `-lenient` and `-strict` impossible option: It may not be possible in the current configuration to implement an option - e.g., `-usejavac` or `-eclipse` when javac or the eclipse implementation is not on the classpath
source searching: Given `-seek:{literal}`, as a side-effect, `CompilerRun` will search source files for {literal}, emitting for each instance an INFO message of the form: `found: {file}:{line}:{column}` (Note that the harness does not display INFO messages unless `-verboseHarness` or `-loud` is used.)
JavaRun Options and forking	options: options specified in the test are passed to the main method as they would be on the command-line. No options passed to the harness are passed through to the main class.
forking: Forking is useful to run in a different version of Java than can be supported by the harness (i.e., some 1.1 flavor); it's very time-consuming otherwise. Currently forking is only controllable through system properties of the invoking vm (defined in JavaRun.java): javarun.fork: anything to run in a new vm. javarun.java: path to the java executable to run (suffix included). If not supplied, the harness tries to find the java that invoked the harness. javarun.java.home: the value of the JAVA_HOME environment variable, if it needs to be set. javarun.bootclasspath: this is prepended to the run classpath. Multiple entries must be separated by the system-dependent path separator. javarun.vmargs: this is added to the fork command-line right after java. Multiple entries must be separated by a comma (and the whole thing should be one parameter), e.g., `-Djavarun.vmargs=-Dname=value,-Dname2="value 2"`

java {harness} -hideStreams {suiteFile}
Use this to output only a 1-line summary of the test results (tests skipped, incomplete, failed, passed).
java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt
This writes to result.txt one line [PASS|FAIL] per test, plus a 1-line summary of the test results.
java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt
This re-runs any test that failed from the "results.txt" run, verbosely logging any fails.
java {harness} -hideStreams -logMinFail {suiteFile}
Use this when running tests mainly to see if they pass or if the failure messages are typically enough information to indicate why the test is failing. It produces only minimal output for failed tests.
java {harness} -hideStreams -verboseHarness -logFail {suiteFile}
When it's not clear at first glance why a test is failing, before looking at the test code you can run it and print any harness or test setup failures and all the associated messages from the test components.
java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}
Because of the trailing '-' on two of the options, this would do four complete runs with the old (Ajc 1.0) compiler: one with no options, one with -lenient, one with -Xlint, and one with both.
java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}
Run any tests associated with bugs 101 and 102, with and without -Xlint, forcing off -usejavac and forcing the use of the new eclipse-based compiler.

If you have a set of options you use often, you can define a single-word option alias for it; see Harness.optionAliases.
Configuration: Most tests use the library jars in modules/lib/test, defined in org.aspectj.testing.harness.bridge.Globals. Normally the harness finds these by relative path ../lib/tests/*.jar, which works whenever the tests are run from a peer module directory. When running tests elsewhere, define the environment variable harness.libdir - e.g.,

    $ cd aspectj/tests
    $ java -Dharness.libdir=../modules/lib/test ...

Forking:: The harness must be run in a compiler-compatible VM, and the compiler steps run in-process. However, the java steps can be run in forked mode, which is useful when compiling for a VM which can't run the compiler. To compile for a different target VM could require setting the options for bootclasspath, target, and source. To run the harness so that any <run.. tasks run in a separate vm, do something like this:

   java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
        -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
        -Djavarun.java.home=d:\jdk1.1.8 \
        -Djavarun.fork=true \
        -jar ../aj-build/jars/testing-drivers-all.jar \
        ajcTests.xml -logFail

Here CompilerRun would add the bootclasspath as such when compiling. JavaRun would fork using the 1.1 vm and prepend the bootclasspath to the classpath, with an effect like these commands (ignoring the line splitting in the classpath):

   set JAVA_HOME=d:\jdk1.1.8
   d:\jdk1.1.8\bin\java.exe \
     -classpath "d:\jdk1.1.8\lib\classes.zip;
                 d:\aspectj-src\lib\test\testing-client.jar;
                 d:\aspectj-src\lib\test\aspectjrt.jar;
                 c:\TEMP\sandbox7wers\classes"
     {mainClass} {option..}