mirrors
/
org.aspectj
mirror da https://github.com/eclipse/org.aspectj.git


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428
							<html>
	<head><title>Harness Package Documentation</title></head>
<body>
<p>
The AspectJ compiler test harness can compile and run AspectJ programs
as specified by the test definitions.
This document tells you how to run the harness.
It describes the options you can specify on the command-line to
control various components that form the harness, either to 
specify options that augment the test definitions or to
change how the harness works, e.g., selecting particular tests
or logging more information.
For information on how to write a test definition, see
<code>readme-writing-compiler-tests.html</code> in the testing module.
</p>
<p>
The harness drives compiler tests, using  
a chain of responsibility to map elements
in the schema of a test definition to implementing classes.

</p>
<table border="1" cellpadding="1">
<tr><th align="left">Test feature</th>
    <th align="left">Description</th>
    <th align="left">Implementing class</th>
</tr>
<tr><td>(loading suites...)</td>
    <td>general harness</td>
    <td>Harness</td>
</tr>
<tr><td>(logging...)</td>
    <td>subclass feature harness</td>
    <td>FeatureHarness</td>
</tr>
<tr><td><code>&lt;suite&gt;</code></td>
    <td>Test suite</td>
    <td>AjcTest.Suite</td>
</tr>
<tr><td>&nbsp;&nbsp;<code>&lt;ajc-test&gt;</code></td>
    <td>Test case</td>
    <td>AjcTest</td>
</tr>
<tr><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>&lt;compile&gt;</code></td>
    <td>Initial (batch) compile run</td>
    <td>CompilerRun</td>
</tr>
<tr><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>&lt;inc-compile&gt;</code></td>
    <td>Incremental re-compile</td>
    <td>IncCompilerRun</td>
</tr>
<tr><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>&lt;run&gt;</code></td>
    <td>Run class</td>
    <td>JavaRun</td>
</tr>
</table>
<!--
   general harness                                 (Harness)
      subclass feature harness                     (FeatureHarness)
        &lt;ajc-test&gt; run component             (AjcTest)
          &lt;compile&gt; {sub-} run component     (CompilerRun)
          &lt;inc-compile&gt; {sub-} run component (IncCompilerRun)
          &lt;run&gt; {sub-} run component         (JavaRun)
          ...
-->
<p/>
The compiler used is the AspectJ compiler <code>ajc</code>
(optionally as wrapped by the Ant task or AJDE API's), but
in principle any compiler accepting similar options can be
used.
<p/>
To run from the command-line, use 
<code>Harness.main(String[])</code>.
To run programmatically, use <code>Harness.getHarness()</code>.
<code>Harness.runMain(String[])</code> takes arguments that
each component in the chain may accept and interpret, so
you can modify how the tests run by specifying the following 
arguments on the harness command line:
<p/>
<table cellpadding="1" border="1">
<tr><th>Component</th><th>Options</th></tr>

<tr><td rowspan="6" valign="top">Harness 
    <p>suite files, harness verbosity, temp files, option variants
    </p></td></tr>
  <tr><td><u>suite files</u>: ajcTest-compliant .txt or .xml files are accepted.
     <!-- XXX link to ajcTestSuite.dtd and .txt definitions -->
     </td></tr>
  <tr><td><u><code>-verboseHarness</code></u>, 
  	  <u><code>-quietHarness</code></u>: 
      Log accepted options and skipped tests,
      or do not print even info messages.
     </td></tr>
  <tr><td><u><code>-keepTemp</code></u>: Normally the harness saves temp files until
     the end of the run, and deletes them.  If you abort the run or specify
     <code>-keepTemp</code>, then temporary (sandbox) directories will remain for analysis.
     In either case, the file system accumulates all temporary directories
     and files used for a give harness run; for the <code>ajcTests.xml</code>
     suite, this runs into thousands of files.
     </td></tr>
  <tr><td><u><code>-killTemp</code></u>: The opposite of <code>-keepTemp</code>,
     this causes the harness to delete temporary (sandbox) directories at 
     the end of each test run. 
     In this case, the file system only accumulates files for 
     the current test.
     </td></tr>
  <tr><td><u>*- variants</u>: Options with a trailing "-" cause two sets of
      option lists to be produced, one with and one without the corresponding
      option.  E.g., "-emacssym-" will run the suite twice, once with and
      once without the "-emacssym" flag.
      That means if you use this on each of three options, you will
      get 8 variant sets (1 with no options, 1 with all 3 options, 
      3 with 2 options, and 3 with 1 option). 
     </td></tr>
    
<tr><td rowspan="5" valign="top">FeatureHarness 
    <p>output and logging options
    </p></td></tr>
  <tr><td><u>tracing</u>: 
     <code>-progressDots</code> will print "." for every passed
     test completed and "!" for every test completed but not passed.
     <code>-traceTests</code> will print a one-line summary for each test
     of the time and space taken and whether the test passed.
     <code>-traceTestsMin</code> will print only the test and whether it passed.
     <code>-baseline</code> is an alias for
     <code>-traceTestsMin</code> 
     <code>-hideStreams</code> and 
     <code>!eclipse</code>, used to emit tests results in a form
     comparable by <code>org.aspectj.testing.util.TestDiffs</code>. 
     or usable to select tests by title for options like 
     <code>-ajctestTitleList</code>.
     </td></tr>
     
  <tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and
     error streams from being printed to System.err and System.out, 
     optionally only for run or compile steps.
     </td></tr>
  <tr><td><u>logging</u>: 
     Log variants take the form <code>-log{Min|Xml}[Fail|Pass|All]</code>.
     The suffix {All|Pass|Fail} selects all tests or only passing or failing tests.
     The infix {Min} means to log with minimal information, typically only any
     fail messages.
     The infix {Xml} means to log the XML form of the test definition, so that
     you can inspect the input or re-run arbitrary tests.  
     (You can also re-run a set of tests using keywords
     (e.g., "<code>-ajctestsRequireKeywords=...</code>" or using titles
     (e.g., "<code>-ajctestsTitleFailList=ajcTestResults.txt</code>".)
     Finally, the experimental option <code>-XlogPublicType</code> will 
      log the XML test definition for 
  	  any test run that emits any ERROR messages containing the text "public type".
     </td></tr>
  <tr><td><u>interaction of output streams and logging</u>: 
     Streams will be emitted in real-time,
     <em>before</em> the test is logged, unless streams are hidden.
     When logging in normal (non-Min or -XML) form, the log will emit the streams
     with the test report, so e.g., you can use -hideStreams -logFail to 
     hide streams for passing tests but emit them for failing tests
     in the context of the log.
     </td></tr>
    
<tr><td rowspan="5" valign="top">AjcTest
    <p>selection options for keywords, bugID (PR), or title (description)
    </p></td></tr>
  <tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code>
     will either require or skip a test that has one of the 
     specified keywords.
     </td></tr>
  <tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code>
     will require that a test have one of the listed bug id's.
     </td></tr>
  <tr><td><u>title</u>: 
     <code>"-ajctestTitleContains=one,two"</code>
     will require that the title (description) of a test contain 
     one of the specified substrings, here either "one" or "two".
     Use this to select a few tests you know generally.
     <br/>
     <code>"-ajctestTitleList=first title\, in theory, second title"</code>
     will require that the title (description) of a test be
     exactly "first title, in theory" or "second title".  
     The entire option must be one argument on the command line.
     Use this when working with just a few specific tests.
     <br/>
     <code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
     will require that the title (description) of a test be
     equal to one listed in <code>../tests/ajcTestResults.txt</code>
     as a line of the form "[PASS|FAIL] {title}(.."
     (This form is emitted by the <code>-traceTestsMin</code> option).
     This option only differs from the prior in that the parameter
     is a valid file to read.
     Use this to re-run a large set of tests.
     <br/>
     <code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code>
     is the same as the <code>-ajctestTitleList={file}</code> variant,
     except that only results prefixed "FAIL" are included.
     Use this to re-run only the tests that failed from a large set.
     </td></tr>
     
  <tr><td><u>Combinations</u>: all selectors are applied to each test,
     so all tests selected will comply with all constraints.
     Specifying lists within a particular constraints will match
     a union of tests for that constraint 
     (e.g., all tests with bug id's 101 or 102),     
     but there is no way to get a union of constraints 
     (e.g., all tests with bug id's 101 or 102 <em>or</em> 
     with keywords pure-java or knownLimitation).
     However, <code>-ajctestSkipKeywords=...</code> can return all
     tests without the specified keywords, so it can form unions like
     "all tests without the knownLimitation keyword, but with 
     bug id's 101 or 102".
     Title lists can work similarly.  E.g., to run the failed
     incremental tests from ajcTestResults.txt, specify
     <code>-ajctestTitleFailList=../tests/ajcTestResults.txt</code>
     <code>-ajctestRequireKeywords=incremental-test</code>.
     </td></tr>
    
<tr><td rowspan="6" valign="top">CompilerRun
<p>compiler options and side-effects
    </p></td></tr>
  <tr><td><u>supported options</u>: 
     The command line passed to the compiler by <code>CompilerRun</code>
     is composed of entries derived from the <code>&lt;compile&gt;</code>
     attributes and recognized from the harness command line.
     <code>&lt;compile&gt;</code> has specific attributes like
     <code>files</code>, 
     <code>argfiles</code>, 
     <code>classpath</code> and       
     <code>sourceroot</code> 
     which translate directly to their counterparts.
     The output option <code>-d</code> is defined by <code>CompilerRun</code> and
     may not be specified (and <code>-outjar</code> is not supported).
     Most other compiler options are defined in 
     <code>CompilerRun.Spec.CRSOptions</code> and may be specified
     on the harness command-line
     or in the <code>options</code> attribute of 
     <code>&lt;compile&gt;</code>.  
     In the <code>options</code> attribute, each argument is comma-delimited,
     so an option with an argument would look like 
     <code>&lt;compile options="-source,1.4" ...&gt;</code>.  
     If options collide, duplicates
     can be resolved using option dominance (below).
     </td></tr>
  <tr><td><u>compiler selectors</u>: 
  	 Use <code>-ajc</code> or <code>-eclipse</code> to select the old
     (ajc 1.0) or new (eajc 1.1) compilers.  
      Note that the old compiler is not
      available in the CVS source tree at eclipse.org.
      Use <code>-ajdeCompiler</code> to run a wrapper around the
      AJDE interface
      and <code>-ajctaskCompiler</code> to run a wrapper around the
      AjcTask (Ant task) interface.
     </td></tr>
  <tr><td><u>option dominance <code>[-|!|^]</code></u>: 
     Some tests require or prohibit certain options; 
     likewise, sometime you want to force all tests
     run with or without an option specified on the command-line,
     regardless of its setting in the <code>&lt;compile options=".." ...&gt;</code> 
     attribute. 
     For this reason an option may be specified in the options attribute
     or on the harness command-line as
      <code>-option</code>,
      <code>!option</code>, or 
      <code>^option</code>.
     <ul>
     <li><u>- set</u>: If the leading character of an option is "-", then it is set unless forced-off.</li>
     <li><u>^ force-off</u>: If the leading character of an option is "^", then it is forced off.
            Any other matching option will be removed.</li>
     <li><u>! force-on</u>: If the leading character of an option is "!", then it is forced on.
         Any other non-force-on matching option will be removed.</li>
     <li><u>force conflict</u>: If there are two matching force-on options, the test is skipped.</li>
     <li><u>related options</u>: Two options match if they are the same or
        if they are in the same family.  For example, <code>-ajc</code> and
        <code>eclipse</code> are both compiler, and <code>-source 1.4</code>
        and <code>-source 1.3</code> are both source.
		<br/>
        </li>
     </ul>
     </td></tr>
  <tr><td><u>auto-skip</u>: After combining global and local options, there may be
     conflicting or impossible options, which cause the test to be skipped:
     <ul>
     <li><u>semantic conflicts</u>: two options may conflict in meaning 
           - e.g., <code>-lenient</code> and <code>-strict</code></li>
     <li><u>impossible option</u>: It may not be possible in the current configuration to 
         implement an option - e.g., <code>-usejavac</code> or <code>-eclipse</code> 
         when javac or the eclipse implementation is not on the classpath		
		<br/></li>
     </ul>
     </td></tr>
     
  <tr><td><u>source searching</u>: Given <code>-seek:{literal}</code>,  
  	  as a side-effect, 
  	  <code>CompilerRun</code> will search source files for {literal},
      emitting for each instance an INFO message of the form: 
     <tt>found: {file}:{line}:{column}</tt>
      (Note that the harness does not display INFO messages unless <tt>-verboseHarness</tt>
      or <tt>-loud</tt> is used.) 
     </td></tr>
    
     
  <tr><td rowspan="2" valign="top">JavaRun
  	  <p>Options and forking</p></td>
     <td><u>options</u>: options specified in the test are passed
      to the main method as they would be on the command-line.
      No options passed to the harness are passed through to
      the main class.
     </td></tr>
  <tr><td><u>forking</u>: 
        Forking is useful to run in a different version of Java
        than can be supported by the harness (i.e., some 1.1 flavor);
        it's very time-consuming otherwise.
  		Currently forking is only controllable through system properties
  		of the invoking vm (defined in JavaRun.java):
  		<ul>
  		   <li><u>javarun.fork</u>: anything to run in a new vm.
  		      </li>
  		   <li><u>javarun.java</u>: path to the java executable to run
  		      (suffix included).  If not supplied, the harness tries to
  		      find the java that invoked the harness.
  		      </li>
  		   <li><u>javarun.java.home</u>: the value of the JAVA_HOME 
  		      environment variable, if it needs to be set.
  		      </li>
  		   <li><u>javarun.bootclasspath</u>: this is prepended to the
  		      run classpath.  Multiple entries must be separated by
  		      the system-dependent path separator.
  		      </li>
  		   <li><u>javarun.vmargs</u>: this is added to the fork command-line
			right after java.  Multiple entries must be separated by a comma
			(and the whole thing should be one parameter), e.g.,
			<code>-Djavarun.vmargs=-Dname=value,-Dname2="value 2"</code>
  		      </li>
  		</ul>
     </td></tr>
</table>
<br/>
Following are some sample configurations:
<ul>
<li><code>java {harness} -hideStreams {suiteFile}</code>
 <p>Use this to output only a 1-line summary of the test results
    (tests skipped, incomplete, failed, passed).<br/></p>
  </li>
  
<li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code>
 <p>This writes to result.txt one line [PASS|FAIL] per test, plus a
    1-line summary of the test results.<br/></p>
  </li>
  
<li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code>
 <p>This re-runs any test that failed from the "results.txt" run,
    verbosely logging any fails.<br/></p>
  </li>
  
<li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code>
 <p>Use this when running tests mainly to see if they pass or
    if the failure messages are typically enough information
    to indicate why the test is failing.  It produces only minimal
    output for failed tests.<br/></p>
  </li>
  
<li><code>java {harness} -hideStreams -verboseHarness -logFail {suiteFile}</code>
 <p>When it's not clear at first glance why a test is failing, before
    looking at the test code you can run it and print any harness or test
    setup failures and all the associated messages from the test components.<br/></p>
  </li>

<li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code>
 <p>Because of the trailing '-' on two of the options,
    this would do four complete runs with the old (Ajc 1.0) compiler: one with
    no options, one with -lenient, one with -Xlint, and one with both.<br/></p>
  </li>
  

<li><code>java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}</code>
 <p>Run any tests associated with bugs 101 and 102, with and without -Xlint,
    forcing off -usejavac and forcing the use of the new eclipse-based compiler.<br/></p>
  </li>
  
</ul>

If you have a set of options you use often, you can define a single-word
option alias for it; see <code>Harness.optionAliases</code>.

<br/><u>Configuration</u>: Most tests use the library jars in 
<code>modules/lib/test</code>, defined in 
<code>org.aspectj.testing.harness.bridge.Globals</code>.  
Normally the harness finds these by relative path 
<code>../lib/tests/*.jar</code>, which works whenever the tests are
run from a peer module directory.  When running tests elsewhere,
define the environment variable <code>harness.libdir</code> - e.g., 
<pre>
    $ cd aspectj/tests
    $ java -Dharness.libdir=../modules/lib/test ...
</pre>

<br/><u>Forking:</u>:
The harness must be run in a compiler-compatible VM, and the
compiler steps run in-process.
However, the java steps can be run in forked mode, which is useful
when compiling for a VM which can't run the compiler.
To compile for a different target VM could require
setting the options for bootclasspath, target, and source.
To run the harness so that any &lt;run.. tasks run in a
separate vm, do something like this:
<pre>
   java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
        -Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
        -Djavarun.java.home=d:\jdk1.1.8 \
        -Djavarun.fork=true \
        -jar ../aj-build/jars/testing-drivers-all.jar \
        ajcTests.xml -logFail
</pre>

Here <code>CompilerRun</code> would add the bootclasspath as such when compiling.
JavaRun would fork using the 1.1 vm and prepend the bootclasspath
to the classpath, with an effect like these commands
(ignoring the line splitting in the classpath):
<pre>
   set JAVA_HOME=d:\jdk1.1.8
   d:\jdk1.1.8\bin\java.exe \
     -classpath "d:\jdk1.1.8\lib\classes.zip;
                 d:\aspectj-src\lib\test\testing-client.jar;
                 d:\aspectj-src\lib\test\aspectjrt.jar;
                 c:\TEMP\sandbox7wers\classes"
     {mainClass} {option..}
</pre>


</body>
</html>