123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428 |
- <html>
- <head><title>Harness Package Documentation</title></head>
- <body>
- <p>
- The AspectJ compiler test harness can compile and run AspectJ programs
- as specified by the test definitions.
- This document tells you how to run the harness.
- It describes the options you can specify on the command-line to
- control various components that form the harness, either to
- specify options that augment the test definitions or to
- change how the harness works, e.g., selecting particular tests
- or logging more information.
- For information on how to write a test definition, see
- <code>readme-writing-compiler-tests.html</code> in the testing module.
- </p>
- <p>
- The harness drives compiler tests, using
- a chain of responsibility to map elements
- in the schema of a test definition to implementing classes.
-
- </p>
- <table border="1" cellpadding="1">
- <tr><th align="left">Test feature</th>
- <th align="left">Description</th>
- <th align="left">Implementing class</th>
- </tr>
- <tr><td>(loading suites...)</td>
- <td>general harness</td>
- <td>Harness</td>
- </tr>
- <tr><td>(logging...)</td>
- <td>subclass feature harness</td>
- <td>FeatureHarness</td>
- </tr>
- <tr><td><code><suite></code></td>
- <td>Test suite</td>
- <td>AjcTest.Suite</td>
- </tr>
- <tr><td> <code><ajc-test></code></td>
- <td>Test case</td>
- <td>AjcTest</td>
- </tr>
- <tr><td> <code><compile></code></td>
- <td>Initial (batch) compile run</td>
- <td>CompilerRun</td>
- </tr>
- <tr><td> <code><inc-compile></code></td>
- <td>Incremental re-compile</td>
- <td>IncCompilerRun</td>
- </tr>
- <tr><td> <code><run></code></td>
- <td>Run class</td>
- <td>JavaRun</td>
- </tr>
- </table>
- <!--
- general harness (Harness)
- subclass feature harness (FeatureHarness)
- <ajc-test> run component (AjcTest)
- <compile> {sub-} run component (CompilerRun)
- <inc-compile> {sub-} run component (IncCompilerRun)
- <run> {sub-} run component (JavaRun)
- ...
- -->
- <p/>
- The compiler used is the AspectJ compiler <code>ajc</code>
- (optionally as wrapped by the Ant task or AJDE API's), but
- in principle any compiler accepting similar options can be
- used.
- <p/>
- To run from the command-line, use
- <code>Harness.main(String[])</code>.
- To run programmatically, use <code>Harness.getHarness()</code>.
- <code>Harness.runMain(String[])</code> takes arguments that
- each component in the chain may accept and interpret, so
- you can modify how the tests run by specifying the following
- arguments on the harness command line:
- <p/>
- <table cellpadding="1" border="1">
- <tr><th>Component</th><th>Options</th></tr>
-
- <tr><td rowspan="6" valign="top">Harness
- <p>suite files, harness verbosity, temp files, option variants
- </p></td></tr>
- <tr><td><u>suite files</u>: ajcTest-compliant .txt or .xml files are accepted.
- <!-- XXX link to ajcTestSuite.dtd and .txt definitions -->
- </td></tr>
- <tr><td><u><code>-verboseHarness</code></u>,
- <u><code>-quietHarness</code></u>:
- Log accepted options and skipped tests,
- or do not print even info messages.
- </td></tr>
- <tr><td><u><code>-keepTemp</code></u>: Normally the harness saves temp files until
- the end of the run, and deletes them. If you abort the run or specify
- <code>-keepTemp</code>, then temporary (sandbox) directories will remain for analysis.
- In either case, the file system accumulates all temporary directories
- and files used for a give harness run; for the <code>ajcTests.xml</code>
- suite, this runs into thousands of files.
- </td></tr>
- <tr><td><u><code>-killTemp</code></u>: The opposite of <code>-keepTemp</code>,
- this causes the harness to delete temporary (sandbox) directories at
- the end of each test run.
- In this case, the file system only accumulates files for
- the current test.
- </td></tr>
- <tr><td><u>*- variants</u>: Options with a trailing "-" cause two sets of
- option lists to be produced, one with and one without the corresponding
- option. E.g., "-emacssym-" will run the suite twice, once with and
- once without the "-emacssym" flag.
- That means if you use this on each of three options, you will
- get 8 variant sets (1 with no options, 1 with all 3 options,
- 3 with 2 options, and 3 with 1 option).
- </td></tr>
-
- <tr><td rowspan="5" valign="top">FeatureHarness
- <p>output and logging options
- </p></td></tr>
- <tr><td><u>tracing</u>:
- <code>-progressDots</code> will print "." for every passed
- test completed and "!" for every test completed but not passed.
- <code>-traceTests</code> will print a one-line summary for each test
- of the time and space taken and whether the test passed.
- <code>-traceTestsMin</code> will print only the test and whether it passed.
- <code>-baseline</code> is an alias for
- <code>-traceTestsMin</code>
- <code>-hideStreams</code> and
- <code>!eclipse</code>, used to emit tests results in a form
- comparable by <code>org.aspectj.testing.util.TestDiffs</code>.
- or usable to select tests by title for options like
- <code>-ajctestTitleList</code>.
- </td></tr>
-
- <tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and
- error streams from being printed to System.err and System.out,
- optionally only for run or compile steps.
- </td></tr>
- <tr><td><u>logging</u>:
- Log variants take the form <code>-log{Min|Xml}[Fail|Pass|All]</code>.
- The suffix {All|Pass|Fail} selects all tests or only passing or failing tests.
- The infix {Min} means to log with minimal information, typically only any
- fail messages.
- The infix {Xml} means to log the XML form of the test definition, so that
- you can inspect the input or re-run arbitrary tests.
- (You can also re-run a set of tests using keywords
- (e.g., "<code>-ajctestsRequireKeywords=...</code>" or using titles
- (e.g., "<code>-ajctestsTitleFailList=ajcTestResults.txt</code>".)
- Finally, the experimental option <code>-XlogPublicType</code> will
- log the XML test definition for
- any test run that emits any ERROR messages containing the text "public type".
- </td></tr>
- <tr><td><u>interaction of output streams and logging</u>:
- Streams will be emitted in real-time,
- <em>before</em> the test is logged, unless streams are hidden.
- When logging in normal (non-Min or -XML) form, the log will emit the streams
- with the test report, so e.g., you can use -hideStreams -logFail to
- hide streams for passing tests but emit them for failing tests
- in the context of the log.
- </td></tr>
-
- <tr><td rowspan="5" valign="top">AjcTest
- <p>selection options for keywords, bugID (PR), or title (description)
- </p></td></tr>
- <tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code>
- will either require or skip a test that has one of the
- specified keywords.
- </td></tr>
- <tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code>
- will require that a test have one of the listed bug id's.
- </td></tr>
- <tr><td><u>title</u>:
- <code>"-ajctestTitleContains=one,two"</code>
- will require that the title (description) of a test contain
- one of the specified substrings, here either "one" or "two".
- Use this to select a few tests you know generally.
- <br/>
- <code>"-ajctestTitleList=first title\, in theory, second title"</code>
- will require that the title (description) of a test be
- exactly "first title, in theory" or "second title".
- The entire option must be one argument on the command line.
- Use this when working with just a few specific tests.
- <br/>
- <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
- will require that the title (description) of a test be
- equal to one listed in <code>../tests/ajcTestResults.txt</code>
- as a line of the form "[PASS|FAIL] {title}(.."
- (This form is emitted by the <code>-traceTestsMin</code> option).
- This option only differs from the prior in that the parameter
- is a valid file to read.
- Use this to re-run a large set of tests.
- <br/>
- <code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code>
- is the same as the <code>-ajctestTitleList={file}</code> variant,
- except that only results prefixed "FAIL" are included.
- Use this to re-run only the tests that failed from a large set.
- </td></tr>
-
- <tr><td><u>Combinations</u>: all selectors are applied to each test,
- so all tests selected will comply with all constraints.
- Specifying lists within a particular constraints will match
- a union of tests for that constraint
- (e.g., all tests with bug id's 101 or 102),
- but there is no way to get a union of constraints
- (e.g., all tests with bug id's 101 or 102 <em>or</em>
- with keywords pure-java or knownLimitation).
- However, <code>-ajctestSkipKeywords=...</code> can return all
- tests without the specified keywords, so it can form unions like
- "all tests without the knownLimitation keyword, but with
- bug id's 101 or 102".
- Title lists can work similarly. E.g., to run the failed
- incremental tests from ajcTestResults.txt, specify
- <code>-ajctestTitleFailList=../tests/ajcTestResults.txt</code>
- <code>-ajctestRequireKeywords=incremental-test</code>.
- </td></tr>
-
- <tr><td rowspan="6" valign="top">CompilerRun
- <p>compiler options and side-effects
- </p></td></tr>
- <tr><td><u>supported options</u>:
- The command line passed to the compiler by <code>CompilerRun</code>
- is composed of entries derived from the <code><compile></code>
- attributes and recognized from the harness command line.
- <code><compile></code> has specific attributes like
- <code>files</code>,
- <code>argfiles</code>,
- <code>classpath</code> and
- <code>sourceroot</code>
- which translate directly to their counterparts.
- The output option <code>-d</code> is defined by <code>CompilerRun</code> and
- may not be specified (and <code>-outjar</code> is not supported).
- Most other compiler options are defined in
- <code>CompilerRun.Spec.CRSOptions</code> and may be specified
- on the harness command-line
- or in the <code>options</code> attribute of
- <code><compile></code>.
- In the <code>options</code> attribute, each argument is comma-delimited,
- so an option with an argument would look like
- <code><compile options="-source,1.4" ...></code>.
- If options collide, duplicates
- can be resolved using option dominance (below).
- </td></tr>
- <tr><td><u>compiler selectors</u>:
- Use <code>-ajc</code> or <code>-eclipse</code> to select the old
- (ajc 1.0) or new (eajc 1.1) compilers.
- Note that the old compiler is not
- available in the CVS source tree at eclipse.org.
- Use <code>-ajdeCompiler</code> to run a wrapper around the
- AJDE interface
- and <code>-ajctaskCompiler</code> to run a wrapper around the
- AjcTask (Ant task) interface.
- </td></tr>
- <tr><td><u>option dominance <code>[-|!|^]</code></u>:
- Some tests require or prohibit certain options;
- likewise, sometime you want to force all tests
- run with or without an option specified on the command-line,
- regardless of its setting in the <code><compile options=".." ...></code>
- attribute.
- For this reason an option may be specified in the options attribute
- or on the harness command-line as
- <code>-option</code>,
- <code>!option</code>, or
- <code>^option</code>.
- <ul>
- <li><u>- set</u>: If the leading character of an option is "-", then it is set unless forced-off.</li>
- <li><u>^ force-off</u>: If the leading character of an option is "^", then it is forced off.
- Any other matching option will be removed.</li>
- <li><u>! force-on</u>: If the leading character of an option is "!", then it is forced on.
- Any other non-force-on matching option will be removed.</li>
- <li><u>force conflict</u>: If there are two matching force-on options, the test is skipped.</li>
- <li><u>related options</u>: Two options match if they are the same or
- if they are in the same family. For example, <code>-ajc</code> and
- <code>eclipse</code> are both compiler, and <code>-source 1.4</code>
- and <code>-source 1.3</code> are both source.
- <br/>
- </li>
- </ul>
- </td></tr>
- <tr><td><u>auto-skip</u>: After combining global and local options, there may be
- conflicting or impossible options, which cause the test to be skipped:
- <ul>
- <li><u>semantic conflicts</u>: two options may conflict in meaning
- - e.g., <code>-lenient</code> and <code>-strict</code></li>
- <li><u>impossible option</u>: It may not be possible in the current configuration to
- implement an option - e.g., <code>-usejavac</code> or <code>-eclipse</code>
- when javac or the eclipse implementation is not on the classpath
- <br/></li>
- </ul>
- </td></tr>
-
- <tr><td><u>source searching</u>: Given <code>-seek:{literal}</code>,
- as a side-effect,
- <code>CompilerRun</code> will search source files for {literal},
- emitting for each instance an INFO message of the form:
- <tt>found: {file}:{line}:{column}</tt>
- (Note that the harness does not display INFO messages unless <tt>-verboseHarness</tt>
- or <tt>-loud</tt> is used.)
- </td></tr>
-
-
- <tr><td rowspan="2" valign="top">JavaRun
- <p>Options and forking</p></td>
- <td><u>options</u>: options specified in the test are passed
- to the main method as they would be on the command-line.
- No options passed to the harness are passed through to
- the main class.
- </td></tr>
- <tr><td><u>forking</u>:
- Forking is useful to run in a different version of Java
- than can be supported by the harness (i.e., some 1.1 flavor);
- it's very time-consuming otherwise.
- Currently forking is only controllable through system properties
- of the invoking vm (defined in JavaRun.java):
- <ul>
- <li><u>javarun.fork</u>: anything to run in a new vm.
- </li>
- <li><u>javarun.java</u>: path to the java executable to run
- (suffix included). If not supplied, the harness tries to
- find the java that invoked the harness.
- </li>
- <li><u>javarun.java.home</u>: the value of the JAVA_HOME
- environment variable, if it needs to be set.
- </li>
- <li><u>javarun.bootclasspath</u>: this is prepended to the
- run classpath. Multiple entries must be separated by
- the system-dependent path separator.
- </li>
- <li><u>javarun.vmargs</u>: this is added to the fork command-line
- right after java. Multiple entries must be separated by a comma
- (and the whole thing should be one parameter), e.g.,
- <code>-Djavarun.vmargs=-Dname=value,-Dname2="value 2"</code>
- </li>
- </ul>
- </td></tr>
- </table>
- <br/>
- Following are some sample configurations:
- <ul>
- <li><code>java {harness} -hideStreams {suiteFile}</code>
- <p>Use this to output only a 1-line summary of the test results
- (tests skipped, incomplete, failed, passed).<br/></p>
- </li>
-
- <li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code>
- <p>This writes to result.txt one line [PASS|FAIL] per test, plus a
- 1-line summary of the test results.<br/></p>
- </li>
-
- <li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code>
- <p>This re-runs any test that failed from the "results.txt" run,
- verbosely logging any fails.<br/></p>
- </li>
-
- <li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code>
- <p>Use this when running tests mainly to see if they pass or
- if the failure messages are typically enough information
- to indicate why the test is failing. It produces only minimal
- output for failed tests.<br/></p>
- </li>
-
- <li><code>java {harness} -hideStreams -verboseHarness -logFail {suiteFile}</code>
- <p>When it's not clear at first glance why a test is failing, before
- looking at the test code you can run it and print any harness or test
- setup failures and all the associated messages from the test components.<br/></p>
- </li>
-
- <li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code>
- <p>Because of the trailing '-' on two of the options,
- this would do four complete runs with the old (Ajc 1.0) compiler: one with
- no options, one with -lenient, one with -Xlint, and one with both.<br/></p>
- </li>
-
-
- <li><code>java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}</code>
- <p>Run any tests associated with bugs 101 and 102, with and without -Xlint,
- forcing off -usejavac and forcing the use of the new eclipse-based compiler.<br/></p>
- </li>
-
- </ul>
-
- If you have a set of options you use often, you can define a single-word
- option alias for it; see <code>Harness.optionAliases</code>.
-
- <br/><u>Configuration</u>: Most tests use the library jars in
- <code>modules/lib/test</code>, defined in
- <code>org.aspectj.testing.harness.bridge.Globals</code>.
- Normally the harness finds these by relative path
- <code>../lib/tests/*.jar</code>, which works whenever the tests are
- run from a peer module directory. When running tests elsewhere,
- define the environment variable <code>harness.libdir</code> - e.g.,
- <pre>
- $ cd aspectj/tests
- $ java -Dharness.libdir=../modules/lib/test ...
- </pre>
-
- <br/><u>Forking:</u>:
- The harness must be run in a compiler-compatible VM, and the
- compiler steps run in-process.
- However, the java steps can be run in forked mode, which is useful
- when compiling for a VM which can't run the compiler.
- To compile for a different target VM could require
- setting the options for bootclasspath, target, and source.
- To run the harness so that any <run.. tasks run in a
- separate vm, do something like this:
- <pre>
- java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
- -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
- -Djavarun.java.home=d:\jdk1.1.8 \
- -Djavarun.fork=true \
- -jar ../aj-build/jars/testing-drivers-all.jar \
- ajcTests.xml -logFail
- </pre>
-
- Here <code>CompilerRun</code> would add the bootclasspath as such when compiling.
- JavaRun would fork using the 1.1 vm and prepend the bootclasspath
- to the classpath, with an effect like these commands
- (ignoring the line splitting in the classpath):
- <pre>
- set JAVA_HOME=d:\jdk1.1.8
- d:\jdk1.1.8\bin\java.exe \
- -classpath "d:\jdk1.1.8\lib\classes.zip;
- d:\aspectj-src\lib\test\testing-client.jar;
- d:\aspectj-src\lib\test\aspectjrt.jar;
- c:\TEMP\sandbox7wers\classes"
- {mainClass} {option..}
- </pre>
-
-
- </body>
- </html>
|