diff options
| author | Andy Clement <aclement@pivotal.io> | 2019-01-30 16:50:34 -0800 |
|---|---|---|
| committer | Andy Clement <aclement@pivotal.io> | 2019-01-30 16:50:34 -0800 |
| commit | d60de8d0b3e62eb36b612a824bb9345d865c0155 (patch) | |
| tree | 859519e87a2614dfc219785e5fbac19db926d4eb /testing-drivers/src/test/java/org/aspectj/testing/drivers/package.html | |
| parent | 24676e3ad3d20ad7d9e7ce1d3e03347883c1ddfd (diff) | |
| download | aspectj-d60de8d0b3e62eb36b612a824bb9345d865c0155.tar.gz aspectj-d60de8d0b3e62eb36b612a824bb9345d865c0155.zip | |
mavenizing testing-drivers - done'
Diffstat (limited to 'testing-drivers/src/test/java/org/aspectj/testing/drivers/package.html')
| -rw-r--r-- | testing-drivers/src/test/java/org/aspectj/testing/drivers/package.html | 428 |
1 files changed, 428 insertions, 0 deletions
diff --git a/testing-drivers/src/test/java/org/aspectj/testing/drivers/package.html b/testing-drivers/src/test/java/org/aspectj/testing/drivers/package.html new file mode 100644 index 000000000..a24466579 --- /dev/null +++ b/testing-drivers/src/test/java/org/aspectj/testing/drivers/package.html @@ -0,0 +1,428 @@ +<html> + <head><title>Harness Package Documentation</title></head> +<body> +<p> +The AspectJ compiler test harness can compile and run AspectJ programs +as specified by the test definitions. +This document tells you how to run the harness. +It describes the options you can specify on the command-line to +control various components that form the harness, either to +specify options that augment the test definitions or to +change how the harness works, e.g., selecting particular tests +or logging more information. +For information on how to write a test definition, see +<code>readme-writing-compiler-tests.html</code> in the testing module. +</p> +<p> +The harness drives compiler tests, using +a chain of responsibility to map elements +in the schema of a test definition to implementing classes. + +</p> +<table border="1" cellpadding="1"> +<tr><th align="left">Test feature</th> + <th align="left">Description</th> + <th align="left">Implementing class</th> +</tr> +<tr><td>(loading suites...)</td> + <td>general harness</td> + <td>Harness</td> +</tr> +<tr><td>(logging...)</td> + <td>subclass feature harness</td> + <td>FeatureHarness</td> +</tr> +<tr><td><code><suite></code></td> + <td>Test suite</td> + <td>AjcTest.Suite</td> +</tr> +<tr><td> <code><ajc-test></code></td> + <td>Test case</td> + <td>AjcTest</td> +</tr> +<tr><td> <code><compile></code></td> + <td>Initial (batch) compile run</td> + <td>CompilerRun</td> +</tr> +<tr><td> <code><inc-compile></code></td> + <td>Incremental re-compile</td> + <td>IncCompilerRun</td> +</tr> +<tr><td> <code><run></code></td> + <td>Run class</td> + <td>JavaRun</td> +</tr> +</table> +<!-- + general harness (Harness) + subclass feature harness (FeatureHarness) + <ajc-test> run component (AjcTest) + <compile> {sub-} run component (CompilerRun) + <inc-compile> {sub-} run component (IncCompilerRun) + <run> {sub-} run component (JavaRun) + ... +--> +<p/> +The compiler used is the AspectJ compiler <code>ajc</code> +(optionally as wrapped by the Ant task or AJDE API's), but +in principle any compiler accepting similar options can be +used. +<p/> +To run from the command-line, use +<code>Harness.main(String[])</code>. +To run programmatically, use <code>Harness.getHarness()</code>. +<code>Harness.runMain(String[])</code> takes arguments that +each component in the chain may accept and interpret, so +you can modify how the tests run by specifying the following +arguments on the harness command line: +<p/> +<table cellpadding="1" border="1"> +<tr><th>Component</th><th>Options</th></tr> + +<tr><td rowspan="6" valign="top">Harness + <p>suite files, harness verbosity, temp files, option variants + </p></td></tr> + <tr><td><u>suite files</u>: ajcTest-compliant .txt or .xml files are accepted. + <!-- XXX link to ajcTestSuite.dtd and .txt definitions --> + </td></tr> + <tr><td><u><code>-verboseHarness</code></u>, + <u><code>-quietHarness</code></u>: + Log accepted options and skipped tests, + or do not print even info messages. + </td></tr> + <tr><td><u><code>-keepTemp</code></u>: Normally the harness saves temp files until + the end of the run, and deletes them. If you abort the run or specify + <code>-keepTemp</code>, then temporary (sandbox) directories will remain for analysis. + In either case, the file system accumulates all temporary directories + and files used for a give harness run; for the <code>ajcTests.xml</code> + suite, this runs into thousands of files. + </td></tr> + <tr><td><u><code>-killTemp</code></u>: The opposite of <code>-keepTemp</code>, + this causes the harness to delete temporary (sandbox) directories at + the end of each test run. + In this case, the file system only accumulates files for + the current test. + </td></tr> + <tr><td><u>*- variants</u>: Options with a trailing "-" cause two sets of + option lists to be produced, one with and one without the corresponding + option. E.g., "-emacssym-" will run the suite twice, once with and + once without the "-emacssym" flag. + That means if you use this on each of three options, you will + get 8 variant sets (1 with no options, 1 with all 3 options, + 3 with 2 options, and 3 with 1 option). + </td></tr> + +<tr><td rowspan="5" valign="top">FeatureHarness + <p>output and logging options + </p></td></tr> + <tr><td><u>tracing</u>: + <code>-progressDots</code> will print "." for every passed + test completed and "!" for every test completed but not passed. + <code>-traceTests</code> will print a one-line summary for each test + of the time and space taken and whether the test passed. + <code>-traceTestsMin</code> will print only the test and whether it passed. + <code>-baseline</code> is an alias for + <code>-traceTestsMin</code> + <code>-hideStreams</code> and + <code>!eclipse</code>, used to emit tests results in a form + comparable by <code>org.aspectj.testing.util.TestDiffs</code>. + or usable to select tests by title for options like + <code>-ajctestTitleList</code>. + </td></tr> + + <tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and + error streams from being printed to System.err and System.out, + optionally only for run or compile steps. + </td></tr> + <tr><td><u>logging</u>: + Log variants take the form <code>-log{Min|Xml}[Fail|Pass|All]</code>. + The suffix {All|Pass|Fail} selects all tests or only passing or failing tests. + The infix {Min} means to log with minimal information, typically only any + fail messages. + The infix {Xml} means to log the XML form of the test definition, so that + you can inspect the input or re-run arbitrary tests. + (You can also re-run a set of tests using keywords + (e.g., "<code>-ajctestsRequireKeywords=...</code>" or using titles + (e.g., "<code>-ajctestsTitleFailList=ajcTestResults.txt</code>".) + Finally, the experimental option <code>-XlogPublicType</code> will + log the XML test definition for + any test run that emits any ERROR messages containing the text "public type". + </td></tr> + <tr><td><u>interaction of output streams and logging</u>: + Streams will be emitted in real-time, + <em>before</em> the test is logged, unless streams are hidden. + When logging in normal (non-Min or -XML) form, the log will emit the streams + with the test report, so e.g., you can use -hideStreams -logFail to + hide streams for passing tests but emit them for failing tests + in the context of the log. + </td></tr> + +<tr><td rowspan="5" valign="top">AjcTest + <p>selection options for keywords, bugID (PR), or title (description) + </p></td></tr> + <tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code> + will either require or skip a test that has one of the + specified keywords. + </td></tr> + <tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code> + will require that a test have one of the listed bug id's. + </td></tr> + <tr><td><u>title</u>: + <code>"-ajctestTitleContains=one,two"</code> + will require that the title (description) of a test contain + one of the specified substrings, here either "one" or "two". + Use this to select a few tests you know generally. + <br/> + <code>"-ajctestTitleList=first title\, in theory, second title"</code> + will require that the title (description) of a test be + exactly "first title, in theory" or "second title". + The entire option must be one argument on the command line. + Use this when working with just a few specific tests. + <br/> + <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code> + will require that the title (description) of a test be + equal to one listed in <code>../tests/ajcTestResults.txt</code> + as a line of the form "[PASS|FAIL] {title}(.." + (This form is emitted by the <code>-traceTestsMin</code> option). + This option only differs from the prior in that the parameter + is a valid file to read. + Use this to re-run a large set of tests. + <br/> + <code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code> + is the same as the <code>-ajctestTitleList={file}</code> variant, + except that only results prefixed "FAIL" are included. + Use this to re-run only the tests that failed from a large set. + </td></tr> + + <tr><td><u>Combinations</u>: all selectors are applied to each test, + so all tests selected will comply with all constraints. + Specifying lists within a particular constraints will match + a union of tests for that constraint + (e.g., all tests with bug id's 101 or 102), + but there is no way to get a union of constraints + (e.g., all tests with bug id's 101 or 102 <em>or</em> + with keywords pure-java or knownLimitation). + However, <code>-ajctestSkipKeywords=...</code> can return all + tests without the specified keywords, so it can form unions like + "all tests without the knownLimitation keyword, but with + bug id's 101 or 102". + Title lists can work similarly. E.g., to run the failed + incremental tests from ajcTestResults.txt, specify + <code>-ajctestTitleFailList=../tests/ajcTestResults.txt</code> + <code>-ajctestRequireKeywords=incremental-test</code>. + </td></tr> + +<tr><td rowspan="6" valign="top">CompilerRun +<p>compiler options and side-effects + </p></td></tr> + <tr><td><u>supported options</u>: + The command line passed to the compiler by <code>CompilerRun</code> + is composed of entries derived from the <code><compile></code> + attributes and recognized from the harness command line. + <code><compile></code> has specific attributes like + <code>files</code>, + <code>argfiles</code>, + <code>classpath</code> and + <code>sourceroot</code> + which translate directly to their counterparts. + The output option <code>-d</code> is defined by <code>CompilerRun</code> and + may not be specified (and <code>-outjar</code> is not supported). + Most other compiler options are defined in + <code>CompilerRun.Spec.CRSOptions</code> and may be specified + on the harness command-line + or in the <code>options</code> attribute of + <code><compile></code>. + In the <code>options</code> attribute, each argument is comma-delimited, + so an option with an argument would look like + <code><compile options="-source,1.4" ...></code>. + If options collide, duplicates + can be resolved using option dominance (below). + </td></tr> + <tr><td><u>compiler selectors</u>: + Use <code>-ajc</code> or <code>-eclipse</code> to select the old + (ajc 1.0) or new (eajc 1.1) compilers. + Note that the old compiler is not + available in the CVS source tree at eclipse.org. + Use <code>-ajdeCompiler</code> to run a wrapper around the + AJDE interface + and <code>-ajctaskCompiler</code> to run a wrapper around the + AjcTask (Ant task) interface. + </td></tr> + <tr><td><u>option dominance <code>[-|!|^]</code></u>: + Some tests require or prohibit certain options; + likewise, sometime you want to force all tests + run with or without an option specified on the command-line, + regardless of its setting in the <code><compile options=".." ...></code> + attribute. + For this reason an option may be specified in the options attribute + or on the harness command-line as + <code>-option</code>, + <code>!option</code>, or + <code>^option</code>. + <ul> + <li><u>- set</u>: If the leading character of an option is "-", then it is set unless forced-off.</li> + <li><u>^ force-off</u>: If the leading character of an option is "^", then it is forced off. + Any other matching option will be removed.</li> + <li><u>! force-on</u>: If the leading character of an option is "!", then it is forced on. + Any other non-force-on matching option will be removed.</li> + <li><u>force conflict</u>: If there are two matching force-on options, the test is skipped.</li> + <li><u>related options</u>: Two options match if they are the same or + if they are in the same family. For example, <code>-ajc</code> and + <code>eclipse</code> are both compiler, and <code>-source 1.4</code> + and <code>-source 1.3</code> are both source. + <br/> + </li> + </ul> + </td></tr> + <tr><td><u>auto-skip</u>: After combining global and local options, there may be + conflicting or impossible options, which cause the test to be skipped: + <ul> + <li><u>semantic conflicts</u>: two options may conflict in meaning + - e.g., <code>-lenient</code> and <code>-strict</code></li> + <li><u>impossible option</u>: It may not be possible in the current configuration to + implement an option - e.g., <code>-usejavac</code> or <code>-eclipse</code> + when javac or the eclipse implementation is not on the classpath + <br/></li> + </ul> + </td></tr> + + <tr><td><u>source searching</u>: Given <code>-seek:{literal}</code>, + as a side-effect, + <code>CompilerRun</code> will search source files for {literal}, + emitting for each instance an INFO message of the form: + <tt>found: {file}:{line}:{column}</tt> + (Note that the harness does not display INFO messages unless <tt>-verboseHarness</tt> + or <tt>-loud</tt> is used.) + </td></tr> + + + <tr><td rowspan="2" valign="top">JavaRun + <p>Options and forking</p></td> + <td><u>options</u>: options specified in the test are passed + to the main method as they would be on the command-line. + No options passed to the harness are passed through to + the main class. + </td></tr> + <tr><td><u>forking</u>: + Forking is useful to run in a different version of Java + than can be supported by the harness (i.e., some 1.1 flavor); + it's very time-consuming otherwise. + Currently forking is only controllable through system properties + of the invoking vm (defined in JavaRun.java): + <ul> + <li><u>javarun.fork</u>: anything to run in a new vm. + </li> + <li><u>javarun.java</u>: path to the java executable to run + (suffix included). If not supplied, the harness tries to + find the java that invoked the harness. + </li> + <li><u>javarun.java.home</u>: the value of the JAVA_HOME + environment variable, if it needs to be set. + </li> + <li><u>javarun.bootclasspath</u>: this is prepended to the + run classpath. Multiple entries must be separated by + the system-dependent path separator. + </li> + <li><u>javarun.vmargs</u>: this is added to the fork command-line + right after java. Multiple entries must be separated by a comma + (and the whole thing should be one parameter), e.g., + <code>-Djavarun.vmargs=-Dname=value,-Dname2="value 2"</code> + </li> + </ul> + </td></tr> +</table> +<br/> +Following are some sample configurations: +<ul> +<li><code>java {harness} -hideStreams {suiteFile}</code> + <p>Use this to output only a 1-line summary of the test results + (tests skipped, incomplete, failed, passed).<br/></p> + </li> + +<li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code> + <p>This writes to result.txt one line [PASS|FAIL] per test, plus a + 1-line summary of the test results.<br/></p> + </li> + +<li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code> + <p>This re-runs any test that failed from the "results.txt" run, + verbosely logging any fails.<br/></p> + </li> + +<li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code> + <p>Use this when running tests mainly to see if they pass or + if the failure messages are typically enough information + to indicate why the test is failing. It produces only minimal + output for failed tests.<br/></p> + </li> + +<li><code>java {harness} -hideStreams -verboseHarness -logFail {suiteFile}</code> + <p>When it's not clear at first glance why a test is failing, before + looking at the test code you can run it and print any harness or test + setup failures and all the associated messages from the test components.<br/></p> + </li> + +<li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code> + <p>Because of the trailing '-' on two of the options, + this would do four complete runs with the old (Ajc 1.0) compiler: one with + no options, one with -lenient, one with -Xlint, and one with both.<br/></p> + </li> + + +<li><code>java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}</code> + <p>Run any tests associated with bugs 101 and 102, with and without -Xlint, + forcing off -usejavac and forcing the use of the new eclipse-based compiler.<br/></p> + </li> + +</ul> + +If you have a set of options you use often, you can define a single-word +option alias for it; see <code>Harness.optionAliases</code>. + +<br/><u>Configuration</u>: Most tests use the library jars in +<code>modules/lib/test</code>, defined in +<code>org.aspectj.testing.harness.bridge.Globals</code>. +Normally the harness finds these by relative path +<code>../lib/tests/*.jar</code>, which works whenever the tests are +run from a peer module directory. When running tests elsewhere, +define the environment variable <code>harness.libdir</code> - e.g., +<pre> + $ cd aspectj/tests + $ java -Dharness.libdir=../modules/lib/test ... +</pre> + +<br/><u>Forking:</u>: +The harness must be run in a compiler-compatible VM, and the +compiler steps run in-process. +However, the java steps can be run in forked mode, which is useful +when compiling for a VM which can't run the compiler. +To compile for a different target VM could require +setting the options for bootclasspath, target, and source. +To run the harness so that any <run.. tasks run in a +separate vm, do something like this: +<pre> + java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \ + -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \ + -Djavarun.java.home=d:\jdk1.1.8 \ + -Djavarun.fork=true \ + -jar ../aj-build/jars/testing-drivers-all.jar \ + ajcTests.xml -logFail +</pre> + +Here <code>CompilerRun</code> would add the bootclasspath as such when compiling. +JavaRun would fork using the 1.1 vm and prepend the bootclasspath +to the classpath, with an effect like these commands +(ignoring the line splitting in the classpath): +<pre> + set JAVA_HOME=d:\jdk1.1.8 + d:\jdk1.1.8\bin\java.exe \ + -classpath "d:\jdk1.1.8\lib\classes.zip; + d:\aspectj-src\lib\test\testing-client.jar; + d:\aspectj-src\lib\test\aspectjrt.jar; + c:\TEMP\sandbox7wers\classes" + {mainClass} {option..} +</pre> + + +</body> +</html> |