--- /dev/null
+<html>
+<!-- <![CDATA[ putDataHere ]]> -->
+<title>Writing tests for the AspectJ compiler
+</title>
+<body>
+<h2>Writing tests for the AspectJ compiler
+</h2>
+The AspectJ project has a harness
+which reads test specification files and run tests.
+The tests are usually simple scenarios like
+"compile and run" or "compile expecting error",
+but may involve multiple files, incremental compilation,
+classpath or aspectpath entries, etc.
+This document shows how to write tests that can be run
+by the harness and suggests some patterns to use in
+test code, discussing
+
+<ul>
+ <li><a href="#simple">Simple test definitions</a></li>
+ <li><a href="#sourceFiles">Test source files</a></li>
+ <li><a href="#incremental">Incremental tests</a></li>
+ <li><a href="#verifying">Verifying test steps</a></li>
+ <ul>
+ <li><a href="#messages">Messages</a></li>
+ <li><a href="#dirchanges">Changes in an output directory</a></li>
+ <li><a href="#tester">Runtime behavior</a></li>
+ </ul>
+ <li><a href="#compilerOptions">Compiler Options</a></li>
+ <li><a href="#background">Harness background</a></li>
+</ul>
+Most people just writing a test case need to know only the
+information in
+ <a href="#simple">Simple test definitions</a> and
+ <a href="#sourceFiles">Test source files</a>.
+
+<p>Related documents:
+<ul>
+ <li>For information on running the harness, see
+ <a href="readme-tests-module.html">
+ readme-tests-module.html</a>
+ </li>
+ <li>For example test definition files, see
+ <a href="ajcTests.xml">ajcTests.xml</a> and
+ <a href="ajcTestsFailing.xml">ajcTestsFailing.xml</a>.
+ </li>
+</ul>
+
+<a name="simple"></a>
+<h4>Simple Test definitions</h4>
+
+Here is a simple example to compile <code>Main.java</code>
+and expect an error on line 10:
+<pre>
+ <ajc-test dir="new" title="simple error test">
+ <compile files="Main.java">
+ <message kind="error" line="10"/>
+ </compile>
+ </ajc-test>
+</pre>
+
+<p>Here is an example to compile
+<code>pack/Aspect.java</code> and
+<code>pack2/Main.java</code> and
+run the main class:
+<pre>
+ <ajc-test dir="new" title="simple run test">
+ <compile files="pack/Aspect.java,pack1/Main.java"/>
+ <run class="pack1.Main"/>
+ </ajc-test>
+</pre>
+The compile and run steps of a given ajc-test share a common
+sandbox, so (e.g.,) the run step knows to set its classpath using
+the classes directory generated by the compile step.
+
+<p>More complex compilations are discussed in
+ <a href="#compilerOptions">Compiler Options</a> below.
+
+<a name="sourceFiles"></a>
+<h4>Test source files</h4>
+
+The <code>dir</code> attribute in the <code>ajc-test</code>
+element specifies a base test directory
+relative to the directory of the test specification file.
+All paths are specified relative to this base test directory.
+
+E.g., the last example used <code>dir="new"</code> and
+presumed the following directory structure:
+<pre>
+ {some dir} # test specification directory
+ {testDefinition}.xml
+ new/ # test base directory
+ pack/Aspect.java
+ pack2/Main.java
+</pre>
+Test cases with only one file in the default package can often
+share directories (e.g., see the many files in <a href="new">new/</a>),
+but usually a test case has its own directory.
+
+<a name="incremental"></a>
+<h4>Incremental tests</h4>
+
+Incremental tests are more complex because they involve
+updating source files before recompiling.
+Here's an example that
+<ul>
+ <li>compiles Main.java,
+ </li><li>runs it,
+ </li><li>updates the source to introduce an error,
+ </li><li>incrementally compile it (and detect the error)
+ </li><li>updates the source to fix the error,
+ </li><li>incrementally compile it (successfully), and
+ </li><li>runs it again.
+ </li>
+</ul>
+<pre>
+ <ajc-test dir="new/incremental1" title="incremental test">
+ <compile staging="true"
+ files="Main.java,DeleteMe.java"/>
+ <run class="Main"/>
+ <inc-compile tag="20">
+ <message kind="error" line="15">
+ </inc-compile>
+ <inc-compile tag="30"/>
+ <run class="Main"/>
+ </ajc-test>
+</pre>
+Take particular note of two attributes,
+<code>compile</code>'s "<code>staging</code>"
+and <code>inc-compile</code>'s "<code>tag</code>".
+First, the compile task enables staging, which copies the
+test source files to a temporary directory so they can be
+updated during the test without changing the actual sources.
+Second, incremental compiles specify a tag which identifies
+files to be deleted or copied into the test source staging directory
+before recompiling. The tag is a suffix identifying
+files in the test source directory specifying how the sources should
+be changed before that incremental compilation step.
+If there is a prefixing suffix "delete", then the file is deleted;
+otherwise, the file is copied (with the effect either of updating
+an existing file or adding a new file).
+
+<p>
+Thus, to understand what's happening in an incremental test
+requires comparing the tags with the files specified in
+the test source directory.
+For example, here is a directory layout for the test above:
+<pre>
+ {some dir}
+ {testDefinition}.xml
+ new/
+ incremental1/
+ DeleteMe.delete.30.java
+ DeleteMe.java
+ Main.20.java
+ Main.30.java
+ Main.java
+ NewFile.30.java
+</pre>
+The result will be one compile and two re-compiles:
+<ol>
+ <li>Compile Main.java and DeleteMe.java</li>
+ <li>Update Main.java with the contents of Main.20.java
+ and recompile </li>
+ <li>Delete DeleteMe.java,
+ add NewFile.java,
+ update Main.java with the contents of Main.30.java
+ and recompile</li>
+</ol>
+
+<a name="verifying"></a>
+<h4>Verifying test steps</h4>
+As seen above, two ways to verify that a compile was successful
+are to run the corresponding class or check the compiler messages.
+More generally, the harness can verify compile/run test steps by detecting the
+following things and comparing against expected behavior:
+<p>
+
+<table border="1" cellpadding="1">
+
+<tr><th>Detect </th>
+ <th>Evaluate</th>
+ </tr>
+<tr><td>Exceptions </td>
+ <td>signal failure </td>
+ </tr>
+
+<tr><td>Result value </td>
+ <td>heuristically compare with expected:
+ compiles not expecting errors are expected
+ to return a normal result status, and vice-versa.</td>
+ </tr>
+<tr><td>Messages (e.g., compiler warnings and errors)</td>
+ <td>Compare with expected messages</td>
+ </tr>
+
+<tr><td>Directory changes (e.g., <code>.class</code> files created) </td>
+ <td>Compare with expected changes</td>
+ </tr>
+<tr><td>Runtime behavior</td>
+ <td>Use <code>Tester</code> in test source code
+ to signal events for comparison with expected events.</td>
+ </tr>
+</table>
+<p>
+
+<a name="messages"></a>
+<h5>Messages</h5>
+
+In a test definition, a nested <code>message</code> element
+specifies a condition on the successful completion of the nesting
+ajc-test sub-element. In the earlier example, if
+the harness does not detect an error message on line 10 or if
+there are unexpected messages, then the compile step will be
+reported as failing:
+<pre>
+ <ajc-test dir="new" title="simple error test">
+ <compile files="Main.java">
+ <message kind="error" line="10"/>
+ </compile>
+ </ajc-test>
+</pre>
+Expected messages can be specified as sub-elements for the three
+ <code>ajc-test</code> elements
+ <code>compile</code>,
+ <code>inc-compile</code>, and
+ <code>run</code>.
+Messages require a kind (error or warning) and a line.
+To make specification easier, if an error is specified for a line,
+the harness accepts as expected any number of errors on that line.
+<p>
+The current harness has only been tested to validate compilation
+based on line numbers.
+Filename-based comparison is disabled as untested/unused,
+and run messages are handled wrongly;
+line-number comparison will fail since
+run messages do not have line numbers.
+
+<a name="dirchanges"></a>
+<h5>Changes in an output directory</h5>
+
+As with messages, specifying directory changes as a nested element
+operates as a condition on the successful completion of the
+nesting element. The harness will check for files added, removed,
+updated, or unchanged from a given directory. The directory is
+specified explicitly or using a token for the shared classes or
+run directory. For even more brevity, the harness supports a
+default suffix for the files.
+<p>
+Directory changes have been used only to validate changes in
+the classes directory.
+The current harness defaults to using the classes directory,
+and when using the classes directory uses <code>.class</code>
+as a default suffix.
+<p>
+Here's an example specification:
+<pre>
+ <ajc-test dir="new/dirchanges-test" title="dir-changes test">
+ <compile staging="true"
+ files="Main.java,DeleteMe.java,Unchanged.java"/>
+ <inc-compile tag="20">
+ <dir-changes updated="Main"
+ removed="DeleteMe"
+ unchanged="Unchanged"/>
+ </inc-compile>
+ </ajc-test>
+</pre>
+It checks after a recompile that
+<ul>
+ <li><code>Main.class</code> was updated</li>
+ <li><code>DeleteMe.class<code> was deleted</li>
+ <li><code>Unchanged.class</code> was not touched</li>
+</ul>
+
+<a name="tester"></a>
+<h5>Runtime behavior</h5>
+Code that tests aspects often falls into the pattern of comparing
+expected and actual events/signals. For example, to prove that
+before advice in fact ran before a particular method execution,
+you might generated and expect signals corresponding to
+
+<ol>
+ <li>method-call</li>
+ <li>before advice execution</li>
+ <li>method-execution</li>
+</ol>
+
+The <code>Tester</code> utility class provides API's for signalling
+actual and expecting events and comparing the two.
+Typically, events are symbolized and compared as String.
+Here's a small sample test case that for the scenario above:
+<pre>
+import org.aspectj.testing.Tester;
+
+public class Main implements Runnable {
+ public static void main(String[] args) {
+ Tester.expectEvent("before advice");
+ Tester.expectEvent("execute run");
+ new Main().run();
+ Tester.checkAllEvents();
+ }
+ public void run() {
+ Tester.event("execute run");
+ }
+}
+
+aspect A {
+ before () : target(Runnable) && execution(void run()) {
+ Tester.event("before advice");
+ }
+}
+</pre>
+
+If either the advice or the method does not run,
+the harness will report a failure.
+
+<p>
+<code>Tester</code> also has methods that operate like
+JUnit assertions as idioms to detect differences in
+expected and actual values, signalling appropriately.
+<p>
+
+<code>Tester</code> is at
+ <a href="../testing-client/src/org/aspectj/testing/Tester.java">
+ ../testing-client/src/org/aspectj/testing/Tester.java</a>
+and is built into
+ <a href="../lib/tests/testing-client.jar">
+ ../lib/tests/testing-client.jar</a>
+which is included on the classpath by the compile and run steps.
+
+<p>You can write runtime test cases without using Tester;
+simply throw some exception from the main thread to signal failure.
+
+<a name="compilerOptions"></a>
+<h4>Compiler options</h4>
+The harness does not support all of the AspectJ 1.1 compiler options.
+Flags are mainly supported through the a comma-delimited list in
+the <code>options</code> attribute:
+<pre>
+ <ajc-test dir="new" title="lint test">
+ <compile files="LintTest.java"
+ options="-Xlint,-emacssym">
+ <message kind="warning" line="22">
+ </compile>
+</pre>
+This should work even for complex single-arg options like
+<code>-g:none<code>, but will fail for comma-delimited single-arg options like
+<code>-g:lines,vars</code> because the comma delimiters
+are ambiguous (yes, a design bug!).
+For -source 1.4 and -source 1.3 options,
+use -source14 and -source13 (yes, a hack!).
+<p>
+The <code>compile</code> element has the following attributes
+which handle most of the other compiler arguments:
+<ul>
+ <li><code>files</code>: .aj and .java files are treated as source files,
+ but .jar files are extracted and passed to the compiler
+ as <code>-injars</code>
+ </li><li><code>classpath</code>: directories and jar files for the classpath
+ </li><li><code>aspectpath</code>: binary aspects in jar files
+ </li><li><code>argfiles</code>: argument list files
+ </li>
+</ul>
+Paths for these are all relative to the test base directory, and
+multiple entries are separated with commas.
+<p>
+Here is a cooked example that uses all <code>compiler</code> attributes:
+<pre>
+ <ajc-test dir="new" title="attributes test">
+ <compile files="Main.java,injar.jar"
+ staging="true"
+ options="-Xlint,-g:none"
+ argfiles="debug.lst,aspects/test.lst"
+ aspectpath="jars/requiredAspects.jar"
+ classpath="providedClassesDir,jars/required.jar"/>
+ <inc-compile tag="20"/>
+ </ajc-test>
+</pre>
+
+
+<h5>Unsupported compiler options</h5>
+The harness does not support the following AspectJ compiler
+options: -target {version}, -outjar {file}, -sourceroots {dir},
+-log {file}.
+The following are used but specification is not supported:
+-d {dir}, -sourceroots {dir}...
+
+<a name="background"></a>
+<h4>Background information on the Harness</h4>
+To make the test specifications as terse as possible,
+harness components for
+ <code>inc-compile</code> and <code>run</code> elements
+use information set up earlier by <code>compile</code>,
+some of which is only implicit.
+When a test is run, the harness creates a staging directory
+for temporary files and a sandbox for sharing information
+between test components, particularly classpath entries
+shared between the compile and run components.
+The compile and run components share classpath information
+through the sandbox, adding default libraries:
+<ul>
+ <li>Aside from any explicit classpath entries,
+ <code>compile</code> always includes the jars
+ <a href="../lib/tests/aspecjrt.jar">
+ ../lib/tests/aspecjrt.jar</a> and
+ <a href="../lib/tests/testing-client.jar">
+ ../lib/tests/testing-client.jar</a>
+ on the compile classpath.
+ </li>
+ <li><code>run</code> sets up its classpath as the compile
+ classpath plus the compile output (classes) directory
+ plus any entries on the aspectpath.
+ </li>
+</ul>
+
+The harness provides some more advance behaviors,
+which you might see specified in the tests.
+For more information, see the API documentation for the harness.
+<ul>
+ <li><u>option dominance and overriding</u>:
+ Both in test specifications and on the command line
+ you can force compiler options on or off.
+ These appear in the test specifications as options
+ with prefixes
+ '!' or '~' rather than '-' (e.g., '~emacssym' to force
+ the emacssym option off, even in tests that specify it).
+ </li>
+</ul>
+<hr>
+<small>last updated January 7, 2002 </small> <!-- CVS variable -->
+
+</body>
+</html>
projects / aspectj.git / commitdiff