<html>
<body>
-This harness drives ajc compiler tests.
-You can run it from the command-line or programmatically.
-It works through a chain of responsibility roughly mapped to
-the schematic of a test definition:
-<pre>
- general harness
- subclass feature harness
- ajc-test run component
- compile {sub-} run component
- inc-compile {sub-} run component
- run {sub-} run component
+This document you how to run the harness,
+specifying options on the command-line for the
+various components that form the harness.
+<p>
+The harness drives compiler tests, using
+a chain of responsibility to map elements
+in the schema of a test definition to implementing classes:
+<p>
+<table border="1" cellpadding="1">
+<tr><th align="left">Test feature</th>
+ <th align="left">Description</th>
+ <th align="left">Implementing class</th>
+</tr>
+<tr><td>(loading suites...)</td>
+ <td>general harness</td>
+ <td>Harness</td>
+</tr>
+<tr><td>(logging...)</td>
+ <td>subclass feature harness</td>
+ <td>FeatureHarness</td>
+</tr>
+<tr><td><code><suite><code></td>
+ <td>Test suite</td>
+ <td>AjcTest.Suite</td>
+</tr>
+<tr><td> <code><ajc-test><code></td>
+ <td>Test case</td>
+ <td>AjcTest</td>
+</tr>
+<tr><td> <code><compile><code></td>
+ <td>Initial (batch) compile run</td>
+ <td>CompilerRun</td>
+</tr>
+<tr><td> <code><inc-compile><code></td>
+ <td>Incremental re-compile</td>
+ <td>IncCompilerRun</td>
+</tr>
+<tr><td> <code><run><code></td>
+ <td>Run class</td>
+ <td>JavaRun</td>
+</tr>
+</table>
+<!--
+ general harness (Harness)
+ subclass feature harness (FeatureHarness)
+ <ajc-test> run component (AjcTest)
+ <compile> {sub-} run component (CompilerRun)
+ <inc-compile> {sub-} run component (IncCompilerRun)
+ <run> {sub-} run component (JavaRun)
...
-</pre>
-This tells you how to run the harness and which options are supported
-by each component.
+-->
<p>
-
-Use <code>Harness.main(String[])</code> from the command-line,
-or <code>Harness.getHarness()</code> programmatically.
+The compiler used is the AspectJ compiler <code>ajc</code>
+(optionally as wrapped by the Ant task or AJDE API's), but
+in principle any compiler accepting similar options can be
+used.
+<p>
+To run from the command-line, use
+<code>Harness.main(String[])</code>.
+To run programmatically, use <code>Harness.getHarness()</code>.
<code>Harness.runMain(String[])</code> takes arguments that
-each component in the chain may accept and interpret:
+each component in the chain may accept and interpret, so
+you can modify how the tests run by specifying the following
+arguments on the harness command line:
<p>
<table cellpadding="1" border="1">
<tr><th>Component</th><th>Options</th></tr>
<tr><td><u>suite files</u>: ajcTest-compliant .txt or .xml files are accepted.
<!-- XXX link to ajcTestSuite.dtd and .txt definitions -->
</td></tr>
- <tr><td><u><code>-verboseHarness</code>, <code>-quietHarness</code></u>:
+ <tr><td><u><code>-verboseHarness</code></u>,
+ <u><code>-quietHarness</code></u>:
Log accepted options and skipped tests,
or do not print even info messages.
</td></tr>
the end of the run, and deletes them. If you abort the run or specify
<code>-keepTemp</code>, then temporary (sandbox) directories will remain for analysis.
In either case, the file system accumulates all temporary directories
- and files used for a give harness run.
+ and files used for a give harness run; for the <code>ajcTests.xml</code>
+ suite, this runs into thousands of files.
</td></tr>
<tr><td><u><code>-killTemp</code></u>: The opposite of <code>-keepTemp</code>,
this causes the harness to delete temporary (sandbox) directories at
<code>-hideStreams</code> and
<code>!eclipse</code>, used to emit tests results in a form
comparable by <code>org.aspectj.testing.util.TestDiffs</code>.
- Note that output in the form emitted by <code>-traceTestsMin</code>
- is input for the options to select tests by title
- (e.g., <code>-ajctestTitleList</code>)
- and for the tool which compare results from different test runs
- (<code>org.aspectj.testing.util.TestDiffs</code>).
+ or usable to select tests by title for options like
+ <code>-ajctestTitleList</code>.
</td></tr>
<tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and
The infix {Min} means to log with minimal information, typically only any
fail messages.
The infix {Xml} means to log the XML form of the test definition, so that
- you can inspect the input or re-run arbitrary tests. (For the latter, consider
- also using keywords, under <code>-ajctestsRequireKeywords=...</code> below.)
+ you can inspect the input or re-run arbitrary tests.
+ (You can also re-run a set of tests using keywords
+ (e.g., "<code>-ajctestsRequireKeywords=...</code>" or using titles
+ (e.g., "<code>-ajctestsTitleFailList=ajcTestResults.txt</code>".)
Finally, the experimental option <code>-XlogPublicType</code> will
log the XML test definition for
any test run that emits any ERROR messages containing the text "public type".
</td></tr>
<tr><td><u>interaction of output streams and logging</u>:
- Streams will be emitted before the test is logged, unless streams are hidden.
+ Streams will be emitted in real-time,
+ <em>before</em> the test is logged, unless streams are hidden.
When logging in normal (non-Min or -XML) form, the log will emit the streams
with the test report, so e.g., you can use -hideStreams -logFail to
- hide streams for passing tests but emit them for failing tests.
+ hide streams for passing tests but emit them for failing tests
+ in the context of the log.
</td></tr>
-<tr><td rowspan="4" valign="top">AjcTest
+<tr><td rowspan="5" valign="top">AjcTest
<p>selection options for keywords, bugID (PR), or title (description)
</td></tr>
<tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code>
- will either require or skip tests that have the specified keywords.
+ will either require or skip a test that has one of the
+ specified keywords.
</td></tr>
<tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code>
- will run only tests that are associated with one of the bug id's listed.
+ will require that a test have one of the listed bug id's.
</td></tr>
<tr><td><u>title</u>:
<code>"-ajctestTitleContains=one,two"</code>
- will require that the title (description) of the test contain
- either "one" or "two".
+ will require that the title (description) of a test contain
+ one of the specified substrings, here either "one" or "two".
Use this to select a few tests you know generally.
<br>
- <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
- will require that the title (description) of the test be
- equal to one listed in <code>../tests/ajcTestResults.txt</code>
- as a line of the form "[PASS|FAIL] {title}(.."
- (emitted by the <code>-traceTestsMin</code> option).
- Use this to re-run a set of tests.
- <br>
<code>"-ajctestTitleList=first title\, in theory, second title"</code>
- will require that the title (description) of the test be
+ will require that the title (description) of a test be
exactly "first title, in theory" or "second title".
The entire option must be one argument on the command line.
- This option only differs from the prior in not specifying
- a valid file to read.
Use this when working with just a few specific tests.
<br>
+ <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
+ will require that the title (description) of a test be
+ equal to one listed in <code>../tests/ajcTestResults.txt</code>
+ as a line of the form "[PASS|FAIL] {title}(.."
+ (This form is emitted by the <code>-traceTestsMin</code> option).
+ This option only differs from the prior in that the parameter
+ is a valid file to read.
+ Use this to re-run a large set of tests.
+ <br>
<code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code>
- is the same as the <code>-ajctestTitleList=..</code> variant,
+ is the same as the <code>-ajctestTitleList={file}</code> variant,
except that only results prefixed "FAIL" are included.
Use this to re-run only the tests that failed from a large set.
</td></tr>
+
+ <tr><td><u>Combinations</u>: all selectors are applied to each test,
+ so all tests selected will comply with all constraints.
+ Specifying lists within a particular constraints will match
+ a union of tests for that constraint
+ (e.g., all tests with bug id's 101 or 102),
+ but there is no way to get a union of constraints
+ (e.g., all tests with bug id's 101 or 102 <em>or</em>
+ with keywords pure-java or knownLimitation).
+ However, <code>-ajctestSkipKeywords=...</code> can return all
+ tests without the specified keywords, so it can form unions like
+ "all tests without the knownLimitation keyword, but with
+ bug id's 101 or 102".
+ Title lists can work similarly. E.g., to run the failed
+ incremental tests from ajcTestResults.txt, specify
+ <code>-ajctestTitleFailList=../tests/ajcTestResults.txt</code>
+ <code>-ajctestRequireKeywords=incremental-test</code>.
+ </td></tr>
<tr><td rowspan="6" valign="top">CompilerRun
<p>compiler options and side-effects
</td></tr>
<tr><td><u>supported options</u>: Options given on the command-line have
- the same meaning as the options in the test specification.
+ the same meaning they would have to the compiler
+ (i.e., the same meaning as they would have as a value in the
+ <code>options</code> attribute in the <compile> entity
+ in the <ajc-test> test specification).
Only one-word options are supported; for this reason, <code>-source 1.4</code> is
specified as <code>-source14</code> and converted by CompilerRun back
to <code>-source 1.4</code>. Unsupported options include
<code>-classpath</code>,
<code>-outjar</code>, and
<code>-sourceroot</code>.
+ (But <code>argfile</code>,
+ <code>classpath</code> and
+ <code>sourceroot</code>
+ are attributes accepted in the <compile> entity,
+ and in any case the test calculates and uses
+ <code>-d</code> and
+ <code>-classpath</code>.)
</td></tr>
<tr><td><u>compiler selectors</u>:
Use <code>-ajc</code> or <code>-eclipse</code> to select the old
and <code>-ajctaskCompiler</code> to run a wrapper around the
AjcTask (Ant task) interface.
</td></tr>
- <tr><td><u>option dominance <code>-!^</code></u>:
+ <tr><td><u>option dominance <code>[-|!|^]</code></u>:
Some tests require or prohibit certain options;
- likewise, sometime the person running the tests wants to require that all tests
- run with or without an option specified on the command-line. CompilerRun supports encodings and
+ likewise, sometime you want to force all tests
+ run with or without an option specified on the command-line,
+ regardless of its setting in the <compile options=".." ...>
+ attribute.
+ CompilerRun supports encodings and
conflict resolution for these, so an option may be specified as
<code>-option</code>,
<code>!option</code>, or
projects / aspectj.git / commitdiff