|
|
@@ -0,0 +1,432 @@ |
|
|
|
<html> |
|
|
|
<!-- <![CDATA[ putDataHere ]]> --> |
|
|
|
<title>Writing tests for the AspectJ compiler |
|
|
|
</title> |
|
|
|
<body> |
|
|
|
<h2>Writing tests for the AspectJ compiler |
|
|
|
</h2> |
|
|
|
The AspectJ project has a harness |
|
|
|
which reads test specification files and run tests. |
|
|
|
The tests are usually simple scenarios like |
|
|
|
"compile and run" or "compile expecting error", |
|
|
|
but may involve multiple files, incremental compilation, |
|
|
|
classpath or aspectpath entries, etc. |
|
|
|
This document shows how to write tests that can be run |
|
|
|
by the harness and suggests some patterns to use in |
|
|
|
test code, discussing |
|
|
|
|
|
|
|
<ul> |
|
|
|
<li><a href="#simple">Simple test definitions</a></li> |
|
|
|
<li><a href="#sourceFiles">Test source files</a></li> |
|
|
|
<li><a href="#incremental">Incremental tests</a></li> |
|
|
|
<li><a href="#verifying">Verifying test steps</a></li> |
|
|
|
<ul> |
|
|
|
<li><a href="#messages">Messages</a></li> |
|
|
|
<li><a href="#dirchanges">Changes in an output directory</a></li> |
|
|
|
<li><a href="#tester">Runtime behavior</a></li> |
|
|
|
</ul> |
|
|
|
<li><a href="#compilerOptions">Compiler Options</a></li> |
|
|
|
<li><a href="#background">Harness background</a></li> |
|
|
|
</ul> |
|
|
|
Most people just writing a test case need to know only the |
|
|
|
information in |
|
|
|
<a href="#simple">Simple test definitions</a> and |
|
|
|
<a href="#sourceFiles">Test source files</a>. |
|
|
|
|
|
|
|
<p>Related documents: |
|
|
|
<ul> |
|
|
|
<li>For information on running the harness, see |
|
|
|
<a href="readme-tests-module.html"> |
|
|
|
readme-tests-module.html</a> |
|
|
|
</li> |
|
|
|
<li>For example test definition files, see |
|
|
|
<a href="ajcTests.xml">ajcTests.xml</a> and |
|
|
|
<a href="ajcTestsFailing.xml">ajcTestsFailing.xml</a>. |
|
|
|
</li> |
|
|
|
</ul> |
|
|
|
|
|
|
|
<a name="simple"></a> |
|
|
|
<h4>Simple Test definitions</h4> |
|
|
|
|
|
|
|
Here is a simple example to compile <code>Main.java</code> |
|
|
|
and expect an error on line 10: |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new" title="simple error test"> |
|
|
|
<compile files="Main.java"> |
|
|
|
<message kind="error" line="10"/> |
|
|
|
</compile> |
|
|
|
</ajc-test> |
|
|
|
</pre> |
|
|
|
|
|
|
|
<p>Here is an example to compile |
|
|
|
<code>pack/Aspect.java</code> and |
|
|
|
<code>pack2/Main.java</code> and |
|
|
|
run the main class: |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new" title="simple run test"> |
|
|
|
<compile files="pack/Aspect.java,pack1/Main.java"/> |
|
|
|
<run class="pack1.Main"/> |
|
|
|
</ajc-test> |
|
|
|
</pre> |
|
|
|
The compile and run steps of a given ajc-test share a common |
|
|
|
sandbox, so (e.g.,) the run step knows to set its classpath using |
|
|
|
the classes directory generated by the compile step. |
|
|
|
|
|
|
|
<p>More complex compilations are discussed in |
|
|
|
<a href="#compilerOptions">Compiler Options</a> below. |
|
|
|
|
|
|
|
<a name="sourceFiles"></a> |
|
|
|
<h4>Test source files</h4> |
|
|
|
|
|
|
|
The <code>dir</code> attribute in the <code>ajc-test</code> |
|
|
|
element specifies a base test directory |
|
|
|
relative to the directory of the test specification file. |
|
|
|
All paths are specified relative to this base test directory. |
|
|
|
|
|
|
|
E.g., the last example used <code>dir="new"</code> and |
|
|
|
presumed the following directory structure: |
|
|
|
<pre> |
|
|
|
{some dir} # test specification directory |
|
|
|
{testDefinition}.xml |
|
|
|
new/ # test base directory |
|
|
|
pack/Aspect.java |
|
|
|
pack2/Main.java |
|
|
|
</pre> |
|
|
|
Test cases with only one file in the default package can often |
|
|
|
share directories (e.g., see the many files in <a href="new">new/</a>), |
|
|
|
but usually a test case has its own directory. |
|
|
|
|
|
|
|
<a name="incremental"></a> |
|
|
|
<h4>Incremental tests</h4> |
|
|
|
|
|
|
|
Incremental tests are more complex because they involve |
|
|
|
updating source files before recompiling. |
|
|
|
Here's an example that |
|
|
|
<ul> |
|
|
|
<li>compiles Main.java, |
|
|
|
</li><li>runs it, |
|
|
|
</li><li>updates the source to introduce an error, |
|
|
|
</li><li>incrementally compile it (and detect the error) |
|
|
|
</li><li>updates the source to fix the error, |
|
|
|
</li><li>incrementally compile it (successfully), and |
|
|
|
</li><li>runs it again. |
|
|
|
</li> |
|
|
|
</ul> |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new/incremental1" title="incremental test"> |
|
|
|
<compile staging="true" |
|
|
|
files="Main.java,DeleteMe.java"/> |
|
|
|
<run class="Main"/> |
|
|
|
<inc-compile tag="20"> |
|
|
|
<message kind="error" line="15"> |
|
|
|
</inc-compile> |
|
|
|
<inc-compile tag="30"/> |
|
|
|
<run class="Main"/> |
|
|
|
</ajc-test> |
|
|
|
</pre> |
|
|
|
Take particular note of two attributes, |
|
|
|
<code>compile</code>'s "<code>staging</code>" |
|
|
|
and <code>inc-compile</code>'s "<code>tag</code>". |
|
|
|
First, the compile task enables staging, which copies the |
|
|
|
test source files to a temporary directory so they can be |
|
|
|
updated during the test without changing the actual sources. |
|
|
|
Second, incremental compiles specify a tag which identifies |
|
|
|
files to be deleted or copied into the test source staging directory |
|
|
|
before recompiling. The tag is a suffix identifying |
|
|
|
files in the test source directory specifying how the sources should |
|
|
|
be changed before that incremental compilation step. |
|
|
|
If there is a prefixing suffix "delete", then the file is deleted; |
|
|
|
otherwise, the file is copied (with the effect either of updating |
|
|
|
an existing file or adding a new file). |
|
|
|
|
|
|
|
<p> |
|
|
|
Thus, to understand what's happening in an incremental test |
|
|
|
requires comparing the tags with the files specified in |
|
|
|
the test source directory. |
|
|
|
For example, here is a directory layout for the test above: |
|
|
|
<pre> |
|
|
|
{some dir} |
|
|
|
{testDefinition}.xml |
|
|
|
new/ |
|
|
|
incremental1/ |
|
|
|
DeleteMe.delete.30.java |
|
|
|
DeleteMe.java |
|
|
|
Main.20.java |
|
|
|
Main.30.java |
|
|
|
Main.java |
|
|
|
NewFile.30.java |
|
|
|
</pre> |
|
|
|
The result will be one compile and two re-compiles: |
|
|
|
<ol> |
|
|
|
<li>Compile Main.java and DeleteMe.java</li> |
|
|
|
<li>Update Main.java with the contents of Main.20.java |
|
|
|
and recompile </li> |
|
|
|
<li>Delete DeleteMe.java, |
|
|
|
add NewFile.java, |
|
|
|
update Main.java with the contents of Main.30.java |
|
|
|
and recompile</li> |
|
|
|
</ol> |
|
|
|
|
|
|
|
<a name="verifying"></a> |
|
|
|
<h4>Verifying test steps</h4> |
|
|
|
As seen above, two ways to verify that a compile was successful |
|
|
|
are to run the corresponding class or check the compiler messages. |
|
|
|
More generally, the harness can verify compile/run test steps by detecting the |
|
|
|
following things and comparing against expected behavior: |
|
|
|
<p> |
|
|
|
|
|
|
|
<table border="1" cellpadding="1"> |
|
|
|
|
|
|
|
<tr><th>Detect </th> |
|
|
|
<th>Evaluate</th> |
|
|
|
</tr> |
|
|
|
<tr><td>Exceptions </td> |
|
|
|
<td>signal failure </td> |
|
|
|
</tr> |
|
|
|
|
|
|
|
<tr><td>Result value </td> |
|
|
|
<td>heuristically compare with expected: |
|
|
|
compiles not expecting errors are expected |
|
|
|
to return a normal result status, and vice-versa.</td> |
|
|
|
</tr> |
|
|
|
<tr><td>Messages (e.g., compiler warnings and errors)</td> |
|
|
|
<td>Compare with expected messages</td> |
|
|
|
</tr> |
|
|
|
|
|
|
|
<tr><td>Directory changes (e.g., <code>.class</code> files created) </td> |
|
|
|
<td>Compare with expected changes</td> |
|
|
|
</tr> |
|
|
|
<tr><td>Runtime behavior</td> |
|
|
|
<td>Use <code>Tester</code> in test source code |
|
|
|
to signal events for comparison with expected events.</td> |
|
|
|
</tr> |
|
|
|
</table> |
|
|
|
<p> |
|
|
|
|
|
|
|
<a name="messages"></a> |
|
|
|
<h5>Messages</h5> |
|
|
|
|
|
|
|
In a test definition, a nested <code>message</code> element |
|
|
|
specifies a condition on the successful completion of the nesting |
|
|
|
ajc-test sub-element. In the earlier example, if |
|
|
|
the harness does not detect an error message on line 10 or if |
|
|
|
there are unexpected messages, then the compile step will be |
|
|
|
reported as failing: |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new" title="simple error test"> |
|
|
|
<compile files="Main.java"> |
|
|
|
<message kind="error" line="10"/> |
|
|
|
</compile> |
|
|
|
</ajc-test> |
|
|
|
</pre> |
|
|
|
Expected messages can be specified as sub-elements for the three |
|
|
|
<code>ajc-test</code> elements |
|
|
|
<code>compile</code>, |
|
|
|
<code>inc-compile</code>, and |
|
|
|
<code>run</code>. |
|
|
|
Messages require a kind (error or warning) and a line. |
|
|
|
To make specification easier, if an error is specified for a line, |
|
|
|
the harness accepts as expected any number of errors on that line. |
|
|
|
<p> |
|
|
|
The current harness has only been tested to validate compilation |
|
|
|
based on line numbers. |
|
|
|
Filename-based comparison is disabled as untested/unused, |
|
|
|
and run messages are handled wrongly; |
|
|
|
line-number comparison will fail since |
|
|
|
run messages do not have line numbers. |
|
|
|
|
|
|
|
<a name="dirchanges"></a> |
|
|
|
<h5>Changes in an output directory</h5> |
|
|
|
|
|
|
|
As with messages, specifying directory changes as a nested element |
|
|
|
operates as a condition on the successful completion of the |
|
|
|
nesting element. The harness will check for files added, removed, |
|
|
|
updated, or unchanged from a given directory. The directory is |
|
|
|
specified explicitly or using a token for the shared classes or |
|
|
|
run directory. For even more brevity, the harness supports a |
|
|
|
default suffix for the files. |
|
|
|
<p> |
|
|
|
Directory changes have been used only to validate changes in |
|
|
|
the classes directory. |
|
|
|
The current harness defaults to using the classes directory, |
|
|
|
and when using the classes directory uses <code>.class</code> |
|
|
|
as a default suffix. |
|
|
|
<p> |
|
|
|
Here's an example specification: |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new/dirchanges-test" title="dir-changes test"> |
|
|
|
<compile staging="true" |
|
|
|
files="Main.java,DeleteMe.java,Unchanged.java"/> |
|
|
|
<inc-compile tag="20"> |
|
|
|
<dir-changes updated="Main" |
|
|
|
removed="DeleteMe" |
|
|
|
unchanged="Unchanged"/> |
|
|
|
</inc-compile> |
|
|
|
</ajc-test> |
|
|
|
</pre> |
|
|
|
It checks after a recompile that |
|
|
|
<ul> |
|
|
|
<li><code>Main.class</code> was updated</li> |
|
|
|
<li><code>DeleteMe.class<code> was deleted</li> |
|
|
|
<li><code>Unchanged.class</code> was not touched</li> |
|
|
|
</ul> |
|
|
|
|
|
|
|
<a name="tester"></a> |
|
|
|
<h5>Runtime behavior</h5> |
|
|
|
Code that tests aspects often falls into the pattern of comparing |
|
|
|
expected and actual events/signals. For example, to prove that |
|
|
|
before advice in fact ran before a particular method execution, |
|
|
|
you might generated and expect signals corresponding to |
|
|
|
|
|
|
|
<ol> |
|
|
|
<li>method-call</li> |
|
|
|
<li>before advice execution</li> |
|
|
|
<li>method-execution</li> |
|
|
|
</ol> |
|
|
|
|
|
|
|
The <code>Tester</code> utility class provides API's for signalling |
|
|
|
actual and expecting events and comparing the two. |
|
|
|
Typically, events are symbolized and compared as String. |
|
|
|
Here's a small sample test case that for the scenario above: |
|
|
|
<pre> |
|
|
|
import org.aspectj.testing.Tester; |
|
|
|
|
|
|
|
public class Main implements Runnable { |
|
|
|
public static void main(String[] args) { |
|
|
|
Tester.expectEvent("before advice"); |
|
|
|
Tester.expectEvent("execute run"); |
|
|
|
new Main().run(); |
|
|
|
Tester.checkAllEvents(); |
|
|
|
} |
|
|
|
public void run() { |
|
|
|
Tester.event("execute run"); |
|
|
|
} |
|
|
|
} |
|
|
|
|
|
|
|
aspect A { |
|
|
|
before () : target(Runnable) && execution(void run()) { |
|
|
|
Tester.event("before advice"); |
|
|
|
} |
|
|
|
} |
|
|
|
</pre> |
|
|
|
|
|
|
|
If either the advice or the method does not run, |
|
|
|
the harness will report a failure. |
|
|
|
|
|
|
|
<p> |
|
|
|
<code>Tester</code> also has methods that operate like |
|
|
|
JUnit assertions as idioms to detect differences in |
|
|
|
expected and actual values, signalling appropriately. |
|
|
|
<p> |
|
|
|
|
|
|
|
<code>Tester</code> is at |
|
|
|
<a href="../testing-client/src/org/aspectj/testing/Tester.java"> |
|
|
|
../testing-client/src/org/aspectj/testing/Tester.java</a> |
|
|
|
and is built into |
|
|
|
<a href="../lib/tests/testing-client.jar"> |
|
|
|
../lib/tests/testing-client.jar</a> |
|
|
|
which is included on the classpath by the compile and run steps. |
|
|
|
|
|
|
|
<p>You can write runtime test cases without using Tester; |
|
|
|
simply throw some exception from the main thread to signal failure. |
|
|
|
|
|
|
|
<a name="compilerOptions"></a> |
|
|
|
<h4>Compiler options</h4> |
|
|
|
The harness does not support all of the AspectJ 1.1 compiler options. |
|
|
|
Flags are mainly supported through the a comma-delimited list in |
|
|
|
the <code>options</code> attribute: |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new" title="lint test"> |
|
|
|
<compile files="LintTest.java" |
|
|
|
options="-Xlint,-emacssym"> |
|
|
|
<message kind="warning" line="22"> |
|
|
|
</compile> |
|
|
|
</pre> |
|
|
|
This should work even for complex single-arg options like |
|
|
|
<code>-g:none<code>, but will fail for comma-delimited single-arg options like |
|
|
|
<code>-g:lines,vars</code> because the comma delimiters |
|
|
|
are ambiguous (yes, a design bug!). |
|
|
|
For -source 1.4 and -source 1.3 options, |
|
|
|
use -source14 and -source13 (yes, a hack!). |
|
|
|
<p> |
|
|
|
The <code>compile</code> element has the following attributes |
|
|
|
which handle most of the other compiler arguments: |
|
|
|
<ul> |
|
|
|
<li><code>files</code>: .aj and .java files are treated as source files, |
|
|
|
but .jar files are extracted and passed to the compiler |
|
|
|
as <code>-injars</code> |
|
|
|
</li><li><code>classpath</code>: directories and jar files for the classpath |
|
|
|
</li><li><code>aspectpath</code>: binary aspects in jar files |
|
|
|
</li><li><code>argfiles</code>: argument list files |
|
|
|
</li> |
|
|
|
</ul> |
|
|
|
Paths for these are all relative to the test base directory, and |
|
|
|
multiple entries are separated with commas. |
|
|
|
<p> |
|
|
|
Here is a cooked example that uses all <code>compiler</code> attributes: |
|
|
|
<pre> |
|
|
|
<ajc-test dir="new" title="attributes test"> |
|
|
|
<compile files="Main.java,injar.jar" |
|
|
|
staging="true" |
|
|
|
options="-Xlint,-g:none" |
|
|
|
argfiles="debug.lst,aspects/test.lst" |
|
|
|
aspectpath="jars/requiredAspects.jar" |
|
|
|
classpath="providedClassesDir,jars/required.jar"/> |
|
|
|
<inc-compile tag="20"/> |
|
|
|
</ajc-test> |
|
|
|
</pre> |
|
|
|
|
|
|
|
|
|
|
|
<h5>Unsupported compiler options</h5> |
|
|
|
The harness does not support the following AspectJ compiler |
|
|
|
options: -target {version}, -outjar {file}, -sourceroots {dir}, |
|
|
|
-log {file}. |
|
|
|
The following are used but specification is not supported: |
|
|
|
-d {dir}, -sourceroots {dir}... |
|
|
|
|
|
|
|
<a name="background"></a> |
|
|
|
<h4>Background information on the Harness</h4> |
|
|
|
To make the test specifications as terse as possible, |
|
|
|
harness components for |
|
|
|
<code>inc-compile</code> and <code>run</code> elements |
|
|
|
use information set up earlier by <code>compile</code>, |
|
|
|
some of which is only implicit. |
|
|
|
When a test is run, the harness creates a staging directory |
|
|
|
for temporary files and a sandbox for sharing information |
|
|
|
between test components, particularly classpath entries |
|
|
|
shared between the compile and run components. |
|
|
|
The compile and run components share classpath information |
|
|
|
through the sandbox, adding default libraries: |
|
|
|
<ul> |
|
|
|
<li>Aside from any explicit classpath entries, |
|
|
|
<code>compile</code> always includes the jars |
|
|
|
<a href="../lib/tests/aspecjrt.jar"> |
|
|
|
../lib/tests/aspecjrt.jar</a> and |
|
|
|
<a href="../lib/tests/testing-client.jar"> |
|
|
|
../lib/tests/testing-client.jar</a> |
|
|
|
on the compile classpath. |
|
|
|
</li> |
|
|
|
<li><code>run</code> sets up its classpath as the compile |
|
|
|
classpath plus the compile output (classes) directory |
|
|
|
plus any entries on the aspectpath. |
|
|
|
</li> |
|
|
|
</ul> |
|
|
|
|
|
|
|
The harness provides some more advance behaviors, |
|
|
|
which you might see specified in the tests. |
|
|
|
For more information, see the API documentation for the harness. |
|
|
|
<ul> |
|
|
|
<li><u>option dominance and overriding</u>: |
|
|
|
Both in test specifications and on the command line |
|
|
|
you can force compiler options on or off. |
|
|
|
These appear in the test specifications as options |
|
|
|
with prefixes |
|
|
|
'!' or '~' rather than '-' (e.g., '~emacssym' to force |
|
|
|
the emacssym option off, even in tests that specify it). |
|
|
|
</li> |
|
|
|
</ul> |
|
|
|
<hr> |
|
|
|
<small>last updated January 7, 2002 </small> <!-- CVS variable --> |
|
|
|
|
|
|
|
</body> |
|
|
|
</html> |