|
|
@@ -1,26 +1,69 @@ |
|
|
|
<html> |
|
|
|
<body> |
|
|
|
This harness drives ajc compiler tests. |
|
|
|
You can run it from the command-line or programmatically. |
|
|
|
It works through a chain of responsibility roughly mapped to |
|
|
|
the schematic of a test definition: |
|
|
|
<pre> |
|
|
|
general harness |
|
|
|
subclass feature harness |
|
|
|
ajc-test run component |
|
|
|
compile {sub-} run component |
|
|
|
inc-compile {sub-} run component |
|
|
|
run {sub-} run component |
|
|
|
This document you how to run the harness, |
|
|
|
specifying options on the command-line for the |
|
|
|
various components that form the harness. |
|
|
|
<p> |
|
|
|
The harness drives compiler tests, using |
|
|
|
a chain of responsibility to map elements |
|
|
|
in the schema of a test definition to implementing classes: |
|
|
|
<p> |
|
|
|
<table border="1" cellpadding="1"> |
|
|
|
<tr><th align="left">Test feature</th> |
|
|
|
<th align="left">Description</th> |
|
|
|
<th align="left">Implementing class</th> |
|
|
|
</tr> |
|
|
|
<tr><td>(loading suites...)</td> |
|
|
|
<td>general harness</td> |
|
|
|
<td>Harness</td> |
|
|
|
</tr> |
|
|
|
<tr><td>(logging...)</td> |
|
|
|
<td>subclass feature harness</td> |
|
|
|
<td>FeatureHarness</td> |
|
|
|
</tr> |
|
|
|
<tr><td><code><suite><code></td> |
|
|
|
<td>Test suite</td> |
|
|
|
<td>AjcTest.Suite</td> |
|
|
|
</tr> |
|
|
|
<tr><td> <code><ajc-test><code></td> |
|
|
|
<td>Test case</td> |
|
|
|
<td>AjcTest</td> |
|
|
|
</tr> |
|
|
|
<tr><td> <code><compile><code></td> |
|
|
|
<td>Initial (batch) compile run</td> |
|
|
|
<td>CompilerRun</td> |
|
|
|
</tr> |
|
|
|
<tr><td> <code><inc-compile><code></td> |
|
|
|
<td>Incremental re-compile</td> |
|
|
|
<td>IncCompilerRun</td> |
|
|
|
</tr> |
|
|
|
<tr><td> <code><run><code></td> |
|
|
|
<td>Run class</td> |
|
|
|
<td>JavaRun</td> |
|
|
|
</tr> |
|
|
|
</table> |
|
|
|
<!-- |
|
|
|
general harness (Harness) |
|
|
|
subclass feature harness (FeatureHarness) |
|
|
|
<ajc-test> run component (AjcTest) |
|
|
|
<compile> {sub-} run component (CompilerRun) |
|
|
|
<inc-compile> {sub-} run component (IncCompilerRun) |
|
|
|
<run> {sub-} run component (JavaRun) |
|
|
|
... |
|
|
|
</pre> |
|
|
|
This tells you how to run the harness and which options are supported |
|
|
|
by each component. |
|
|
|
--> |
|
|
|
<p> |
|
|
|
|
|
|
|
Use <code>Harness.main(String[])</code> from the command-line, |
|
|
|
or <code>Harness.getHarness()</code> programmatically. |
|
|
|
The compiler used is the AspectJ compiler <code>ajc</code> |
|
|
|
(optionally as wrapped by the Ant task or AJDE API's), but |
|
|
|
in principle any compiler accepting similar options can be |
|
|
|
used. |
|
|
|
<p> |
|
|
|
To run from the command-line, use |
|
|
|
<code>Harness.main(String[])</code>. |
|
|
|
To run programmatically, use <code>Harness.getHarness()</code>. |
|
|
|
<code>Harness.runMain(String[])</code> takes arguments that |
|
|
|
each component in the chain may accept and interpret: |
|
|
|
each component in the chain may accept and interpret, so |
|
|
|
you can modify how the tests run by specifying the following |
|
|
|
arguments on the harness command line: |
|
|
|
<p> |
|
|
|
<table cellpadding="1" border="1"> |
|
|
|
<tr><th>Component</th><th>Options</th></tr> |
|
|
@@ -31,7 +74,8 @@ each component in the chain may accept and interpret: |
|
|
|
<tr><td><u>suite files</u>: ajcTest-compliant .txt or .xml files are accepted. |
|
|
|
<!-- XXX link to ajcTestSuite.dtd and .txt definitions --> |
|
|
|
</td></tr> |
|
|
|
<tr><td><u><code>-verboseHarness</code>, <code>-quietHarness</code></u>: |
|
|
|
<tr><td><u><code>-verboseHarness</code></u>, |
|
|
|
<u><code>-quietHarness</code></u>: |
|
|
|
Log accepted options and skipped tests, |
|
|
|
or do not print even info messages. |
|
|
|
</td></tr> |
|
|
@@ -39,7 +83,8 @@ each component in the chain may accept and interpret: |
|
|
|
the end of the run, and deletes them. If you abort the run or specify |
|
|
|
<code>-keepTemp</code>, then temporary (sandbox) directories will remain for analysis. |
|
|
|
In either case, the file system accumulates all temporary directories |
|
|
|
and files used for a give harness run. |
|
|
|
and files used for a give harness run; for the <code>ajcTests.xml</code> |
|
|
|
suite, this runs into thousands of files. |
|
|
|
</td></tr> |
|
|
|
<tr><td><u><code>-killTemp</code></u>: The opposite of <code>-keepTemp</code>, |
|
|
|
this causes the harness to delete temporary (sandbox) directories at |
|
|
@@ -70,11 +115,8 @@ each component in the chain may accept and interpret: |
|
|
|
<code>-hideStreams</code> and |
|
|
|
<code>!eclipse</code>, used to emit tests results in a form |
|
|
|
comparable by <code>org.aspectj.testing.util.TestDiffs</code>. |
|
|
|
Note that output in the form emitted by <code>-traceTestsMin</code> |
|
|
|
is input for the options to select tests by title |
|
|
|
(e.g., <code>-ajctestTitleList</code>) |
|
|
|
and for the tool which compare results from different test runs |
|
|
|
(<code>org.aspectj.testing.util.TestDiffs</code>). |
|
|
|
or usable to select tests by title for options like |
|
|
|
<code>-ajctestTitleList</code>. |
|
|
|
</td></tr> |
|
|
|
|
|
|
|
<tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and |
|
|
@@ -87,60 +129,86 @@ each component in the chain may accept and interpret: |
|
|
|
The infix {Min} means to log with minimal information, typically only any |
|
|
|
fail messages. |
|
|
|
The infix {Xml} means to log the XML form of the test definition, so that |
|
|
|
you can inspect the input or re-run arbitrary tests. (For the latter, consider |
|
|
|
also using keywords, under <code>-ajctestsRequireKeywords=...</code> below.) |
|
|
|
you can inspect the input or re-run arbitrary tests. |
|
|
|
(You can also re-run a set of tests using keywords |
|
|
|
(e.g., "<code>-ajctestsRequireKeywords=...</code>" or using titles |
|
|
|
(e.g., "<code>-ajctestsTitleFailList=ajcTestResults.txt</code>".) |
|
|
|
Finally, the experimental option <code>-XlogPublicType</code> will |
|
|
|
log the XML test definition for |
|
|
|
any test run that emits any ERROR messages containing the text "public type". |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>interaction of output streams and logging</u>: |
|
|
|
Streams will be emitted before the test is logged, unless streams are hidden. |
|
|
|
Streams will be emitted in real-time, |
|
|
|
<em>before</em> the test is logged, unless streams are hidden. |
|
|
|
When logging in normal (non-Min or -XML) form, the log will emit the streams |
|
|
|
with the test report, so e.g., you can use -hideStreams -logFail to |
|
|
|
hide streams for passing tests but emit them for failing tests. |
|
|
|
hide streams for passing tests but emit them for failing tests |
|
|
|
in the context of the log. |
|
|
|
</td></tr> |
|
|
|
|
|
|
|
<tr><td rowspan="4" valign="top">AjcTest |
|
|
|
<tr><td rowspan="5" valign="top">AjcTest |
|
|
|
<p>selection options for keywords, bugID (PR), or title (description) |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code> |
|
|
|
will either require or skip tests that have the specified keywords. |
|
|
|
will either require or skip a test that has one of the |
|
|
|
specified keywords. |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code> |
|
|
|
will run only tests that are associated with one of the bug id's listed. |
|
|
|
will require that a test have one of the listed bug id's. |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>title</u>: |
|
|
|
<code>"-ajctestTitleContains=one,two"</code> |
|
|
|
will require that the title (description) of the test contain |
|
|
|
either "one" or "two". |
|
|
|
will require that the title (description) of a test contain |
|
|
|
one of the specified substrings, here either "one" or "two". |
|
|
|
Use this to select a few tests you know generally. |
|
|
|
<br> |
|
|
|
<code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code> |
|
|
|
will require that the title (description) of the test be |
|
|
|
equal to one listed in <code>../tests/ajcTestResults.txt</code> |
|
|
|
as a line of the form "[PASS|FAIL] {title}(.." |
|
|
|
(emitted by the <code>-traceTestsMin</code> option). |
|
|
|
Use this to re-run a set of tests. |
|
|
|
<br> |
|
|
|
<code>"-ajctestTitleList=first title\, in theory, second title"</code> |
|
|
|
will require that the title (description) of the test be |
|
|
|
will require that the title (description) of a test be |
|
|
|
exactly "first title, in theory" or "second title". |
|
|
|
The entire option must be one argument on the command line. |
|
|
|
This option only differs from the prior in not specifying |
|
|
|
a valid file to read. |
|
|
|
Use this when working with just a few specific tests. |
|
|
|
<br> |
|
|
|
<code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code> |
|
|
|
will require that the title (description) of a test be |
|
|
|
equal to one listed in <code>../tests/ajcTestResults.txt</code> |
|
|
|
as a line of the form "[PASS|FAIL] {title}(.." |
|
|
|
(This form is emitted by the <code>-traceTestsMin</code> option). |
|
|
|
This option only differs from the prior in that the parameter |
|
|
|
is a valid file to read. |
|
|
|
Use this to re-run a large set of tests. |
|
|
|
<br> |
|
|
|
<code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code> |
|
|
|
is the same as the <code>-ajctestTitleList=..</code> variant, |
|
|
|
is the same as the <code>-ajctestTitleList={file}</code> variant, |
|
|
|
except that only results prefixed "FAIL" are included. |
|
|
|
Use this to re-run only the tests that failed from a large set. |
|
|
|
</td></tr> |
|
|
|
|
|
|
|
<tr><td><u>Combinations</u>: all selectors are applied to each test, |
|
|
|
so all tests selected will comply with all constraints. |
|
|
|
Specifying lists within a particular constraints will match |
|
|
|
a union of tests for that constraint |
|
|
|
(e.g., all tests with bug id's 101 or 102), |
|
|
|
but there is no way to get a union of constraints |
|
|
|
(e.g., all tests with bug id's 101 or 102 <em>or</em> |
|
|
|
with keywords pure-java or knownLimitation). |
|
|
|
However, <code>-ajctestSkipKeywords=...</code> can return all |
|
|
|
tests without the specified keywords, so it can form unions like |
|
|
|
"all tests without the knownLimitation keyword, but with |
|
|
|
bug id's 101 or 102". |
|
|
|
Title lists can work similarly. E.g., to run the failed |
|
|
|
incremental tests from ajcTestResults.txt, specify |
|
|
|
<code>-ajctestTitleFailList=../tests/ajcTestResults.txt</code> |
|
|
|
<code>-ajctestRequireKeywords=incremental-test</code>. |
|
|
|
</td></tr> |
|
|
|
|
|
|
|
<tr><td rowspan="6" valign="top">CompilerRun |
|
|
|
<p>compiler options and side-effects |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>supported options</u>: Options given on the command-line have |
|
|
|
the same meaning as the options in the test specification. |
|
|
|
the same meaning they would have to the compiler |
|
|
|
(i.e., the same meaning as they would have as a value in the |
|
|
|
<code>options</code> attribute in the <compile> entity |
|
|
|
in the <ajc-test> test specification). |
|
|
|
Only one-word options are supported; for this reason, <code>-source 1.4</code> is |
|
|
|
specified as <code>-source14</code> and converted by CompilerRun back |
|
|
|
to <code>-source 1.4</code>. Unsupported options include |
|
|
@@ -149,6 +217,13 @@ each component in the chain may accept and interpret: |
|
|
|
<code>-classpath</code>, |
|
|
|
<code>-outjar</code>, and |
|
|
|
<code>-sourceroot</code>. |
|
|
|
(But <code>argfile</code>, |
|
|
|
<code>classpath</code> and |
|
|
|
<code>sourceroot</code> |
|
|
|
are attributes accepted in the <compile> entity, |
|
|
|
and in any case the test calculates and uses |
|
|
|
<code>-d</code> and |
|
|
|
<code>-classpath</code>.) |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>compiler selectors</u>: |
|
|
|
Use <code>-ajc</code> or <code>-eclipse</code> to select the old |
|
|
@@ -160,10 +235,13 @@ each component in the chain may accept and interpret: |
|
|
|
and <code>-ajctaskCompiler</code> to run a wrapper around the |
|
|
|
AjcTask (Ant task) interface. |
|
|
|
</td></tr> |
|
|
|
<tr><td><u>option dominance <code>-!^</code></u>: |
|
|
|
<tr><td><u>option dominance <code>[-|!|^]</code></u>: |
|
|
|
Some tests require or prohibit certain options; |
|
|
|
likewise, sometime the person running the tests wants to require that all tests |
|
|
|
run with or without an option specified on the command-line. CompilerRun supports encodings and |
|
|
|
likewise, sometime you want to force all tests |
|
|
|
run with or without an option specified on the command-line, |
|
|
|
regardless of its setting in the <compile options=".." ...> |
|
|
|
attribute. |
|
|
|
CompilerRun supports encodings and |
|
|
|
conflict resolution for these, so an option may be specified as |
|
|
|
<code>-option</code>, |
|
|
|
<code>!option</code>, or |