diff options
| author | wisberg <wisberg> | 2003-01-07 23:57:28 +0000 |
|---|---|---|
| committer | wisberg <wisberg> | 2003-01-07 23:57:28 +0000 |
| commit | 4678f4e40b2eb3090b4fa0e5e18b988c3817df6c (patch) | |
| tree | b8fd6d3a8a3bab502e59531629bf0cd24f634af1 /tests/readme-writing-compiler-tests.html | |
| parent | 2d10f482f22cf36649707f70e4cc5d8634d5325c (diff) | |
| download | aspectj-4678f4e40b2eb3090b4fa0e5e18b988c3817df6c.tar.gz aspectj-4678f4e40b2eb3090b4fa0e5e18b988c3817df6c.zip | |
initial draft of doc on writing compiler tests
Diffstat (limited to 'tests/readme-writing-compiler-tests.html')
| -rw-r--r-- | tests/readme-writing-compiler-tests.html | 432 |
1 files changed, 432 insertions, 0 deletions
diff --git a/tests/readme-writing-compiler-tests.html b/tests/readme-writing-compiler-tests.html new file mode 100644 index 000000000..8cc1a4815 --- /dev/null +++ b/tests/readme-writing-compiler-tests.html @@ -0,0 +1,432 @@ +<html> +<!-- <![CDATA[ putDataHere ]]> --> +<title>Writing tests for the AspectJ compiler +</title> +<body> +<h2>Writing tests for the AspectJ compiler +</h2> +The AspectJ project has a harness +which reads test specification files and run tests. +The tests are usually simple scenarios like +"compile and run" or "compile expecting error", +but may involve multiple files, incremental compilation, +classpath or aspectpath entries, etc. +This document shows how to write tests that can be run +by the harness and suggests some patterns to use in +test code, discussing + +<ul> + <li><a href="#simple">Simple test definitions</a></li> + <li><a href="#sourceFiles">Test source files</a></li> + <li><a href="#incremental">Incremental tests</a></li> + <li><a href="#verifying">Verifying test steps</a></li> + <ul> + <li><a href="#messages">Messages</a></li> + <li><a href="#dirchanges">Changes in an output directory</a></li> + <li><a href="#tester">Runtime behavior</a></li> + </ul> + <li><a href="#compilerOptions">Compiler Options</a></li> + <li><a href="#background">Harness background</a></li> +</ul> +Most people just writing a test case need to know only the +information in + <a href="#simple">Simple test definitions</a> and + <a href="#sourceFiles">Test source files</a>. + +<p>Related documents: +<ul> + <li>For information on running the harness, see + <a href="readme-tests-module.html"> + readme-tests-module.html</a> + </li> + <li>For example test definition files, see + <a href="ajcTests.xml">ajcTests.xml</a> and + <a href="ajcTestsFailing.xml">ajcTestsFailing.xml</a>. + </li> +</ul> + +<a name="simple"></a> +<h4>Simple Test definitions</h4> + +Here is a simple example to compile <code>Main.java</code> +and expect an error on line 10: +<pre> + <ajc-test dir="new" title="simple error test"> + <compile files="Main.java"> + <message kind="error" line="10"/> + </compile> + </ajc-test> +</pre> + +<p>Here is an example to compile +<code>pack/Aspect.java</code> and +<code>pack2/Main.java</code> and +run the main class: +<pre> + <ajc-test dir="new" title="simple run test"> + <compile files="pack/Aspect.java,pack1/Main.java"/> + <run class="pack1.Main"/> + </ajc-test> +</pre> +The compile and run steps of a given ajc-test share a common +sandbox, so (e.g.,) the run step knows to set its classpath using +the classes directory generated by the compile step. + +<p>More complex compilations are discussed in + <a href="#compilerOptions">Compiler Options</a> below. + +<a name="sourceFiles"></a> +<h4>Test source files</h4> + +The <code>dir</code> attribute in the <code>ajc-test</code> +element specifies a base test directory +relative to the directory of the test specification file. +All paths are specified relative to this base test directory. + +E.g., the last example used <code>dir="new"</code> and +presumed the following directory structure: +<pre> + {some dir} # test specification directory + {testDefinition}.xml + new/ # test base directory + pack/Aspect.java + pack2/Main.java +</pre> +Test cases with only one file in the default package can often +share directories (e.g., see the many files in <a href="new">new/</a>), +but usually a test case has its own directory. + +<a name="incremental"></a> +<h4>Incremental tests</h4> + +Incremental tests are more complex because they involve +updating source files before recompiling. +Here's an example that +<ul> + <li>compiles Main.java, + </li><li>runs it, + </li><li>updates the source to introduce an error, + </li><li>incrementally compile it (and detect the error) + </li><li>updates the source to fix the error, + </li><li>incrementally compile it (successfully), and + </li><li>runs it again. + </li> +</ul> +<pre> + <ajc-test dir="new/incremental1" title="incremental test"> + <compile staging="true" + files="Main.java,DeleteMe.java"/> + <run class="Main"/> + <inc-compile tag="20"> + <message kind="error" line="15"> + </inc-compile> + <inc-compile tag="30"/> + <run class="Main"/> + </ajc-test> +</pre> +Take particular note of two attributes, +<code>compile</code>'s "<code>staging</code>" +and <code>inc-compile</code>'s "<code>tag</code>". +First, the compile task enables staging, which copies the +test source files to a temporary directory so they can be +updated during the test without changing the actual sources. +Second, incremental compiles specify a tag which identifies +files to be deleted or copied into the test source staging directory +before recompiling. The tag is a suffix identifying +files in the test source directory specifying how the sources should +be changed before that incremental compilation step. +If there is a prefixing suffix "delete", then the file is deleted; +otherwise, the file is copied (with the effect either of updating +an existing file or adding a new file). + +<p> +Thus, to understand what's happening in an incremental test +requires comparing the tags with the files specified in +the test source directory. +For example, here is a directory layout for the test above: +<pre> + {some dir} + {testDefinition}.xml + new/ + incremental1/ + DeleteMe.delete.30.java + DeleteMe.java + Main.20.java + Main.30.java + Main.java + NewFile.30.java +</pre> +The result will be one compile and two re-compiles: +<ol> + <li>Compile Main.java and DeleteMe.java</li> + <li>Update Main.java with the contents of Main.20.java + and recompile </li> + <li>Delete DeleteMe.java, + add NewFile.java, + update Main.java with the contents of Main.30.java + and recompile</li> +</ol> + +<a name="verifying"></a> +<h4>Verifying test steps</h4> +As seen above, two ways to verify that a compile was successful +are to run the corresponding class or check the compiler messages. +More generally, the harness can verify compile/run test steps by detecting the +following things and comparing against expected behavior: +<p> + +<table border="1" cellpadding="1"> + +<tr><th>Detect </th> + <th>Evaluate</th> + </tr> +<tr><td>Exceptions </td> + <td>signal failure </td> + </tr> + +<tr><td>Result value </td> + <td>heuristically compare with expected: + compiles not expecting errors are expected + to return a normal result status, and vice-versa.</td> + </tr> +<tr><td>Messages (e.g., compiler warnings and errors)</td> + <td>Compare with expected messages</td> + </tr> + +<tr><td>Directory changes (e.g., <code>.class</code> files created) </td> + <td>Compare with expected changes</td> + </tr> +<tr><td>Runtime behavior</td> + <td>Use <code>Tester</code> in test source code + to signal events for comparison with expected events.</td> + </tr> +</table> +<p> + +<a name="messages"></a> +<h5>Messages</h5> + +In a test definition, a nested <code>message</code> element +specifies a condition on the successful completion of the nesting +ajc-test sub-element. In the earlier example, if +the harness does not detect an error message on line 10 or if +there are unexpected messages, then the compile step will be +reported as failing: +<pre> + <ajc-test dir="new" title="simple error test"> + <compile files="Main.java"> + <message kind="error" line="10"/> + </compile> + </ajc-test> +</pre> +Expected messages can be specified as sub-elements for the three + <code>ajc-test</code> elements + <code>compile</code>, + <code>inc-compile</code>, and + <code>run</code>. +Messages require a kind (error or warning) and a line. +To make specification easier, if an error is specified for a line, +the harness accepts as expected any number of errors on that line. +<p> +The current harness has only been tested to validate compilation +based on line numbers. +Filename-based comparison is disabled as untested/unused, +and run messages are handled wrongly; +line-number comparison will fail since +run messages do not have line numbers. + +<a name="dirchanges"></a> +<h5>Changes in an output directory</h5> + +As with messages, specifying directory changes as a nested element +operates as a condition on the successful completion of the +nesting element. The harness will check for files added, removed, +updated, or unchanged from a given directory. The directory is +specified explicitly or using a token for the shared classes or +run directory. For even more brevity, the harness supports a +default suffix for the files. +<p> +Directory changes have been used only to validate changes in +the classes directory. +The current harness defaults to using the classes directory, +and when using the classes directory uses <code>.class</code> +as a default suffix. +<p> +Here's an example specification: +<pre> + <ajc-test dir="new/dirchanges-test" title="dir-changes test"> + <compile staging="true" + files="Main.java,DeleteMe.java,Unchanged.java"/> + <inc-compile tag="20"> + <dir-changes updated="Main" + removed="DeleteMe" + unchanged="Unchanged"/> + </inc-compile> + </ajc-test> +</pre> +It checks after a recompile that +<ul> + <li><code>Main.class</code> was updated</li> + <li><code>DeleteMe.class<code> was deleted</li> + <li><code>Unchanged.class</code> was not touched</li> +</ul> + +<a name="tester"></a> +<h5>Runtime behavior</h5> +Code that tests aspects often falls into the pattern of comparing +expected and actual events/signals. For example, to prove that +before advice in fact ran before a particular method execution, +you might generated and expect signals corresponding to + +<ol> + <li>method-call</li> + <li>before advice execution</li> + <li>method-execution</li> +</ol> + +The <code>Tester</code> utility class provides API's for signalling +actual and expecting events and comparing the two. +Typically, events are symbolized and compared as String. +Here's a small sample test case that for the scenario above: +<pre> +import org.aspectj.testing.Tester; + +public class Main implements Runnable { + public static void main(String[] args) { + Tester.expectEvent("before advice"); + Tester.expectEvent("execute run"); + new Main().run(); + Tester.checkAllEvents(); + } + public void run() { + Tester.event("execute run"); + } +} + +aspect A { + before () : target(Runnable) && execution(void run()) { + Tester.event("before advice"); + } +} +</pre> + +If either the advice or the method does not run, +the harness will report a failure. + +<p> +<code>Tester</code> also has methods that operate like +JUnit assertions as idioms to detect differences in +expected and actual values, signalling appropriately. +<p> + +<code>Tester</code> is at + <a href="../testing-client/src/org/aspectj/testing/Tester.java"> + ../testing-client/src/org/aspectj/testing/Tester.java</a> +and is built into + <a href="../lib/tests/testing-client.jar"> + ../lib/tests/testing-client.jar</a> +which is included on the classpath by the compile and run steps. + +<p>You can write runtime test cases without using Tester; +simply throw some exception from the main thread to signal failure. + +<a name="compilerOptions"></a> +<h4>Compiler options</h4> +The harness does not support all of the AspectJ 1.1 compiler options. +Flags are mainly supported through the a comma-delimited list in +the <code>options</code> attribute: +<pre> + <ajc-test dir="new" title="lint test"> + <compile files="LintTest.java" + options="-Xlint,-emacssym"> + <message kind="warning" line="22"> + </compile> +</pre> +This should work even for complex single-arg options like +<code>-g:none<code>, but will fail for comma-delimited single-arg options like +<code>-g:lines,vars</code> because the comma delimiters +are ambiguous (yes, a design bug!). +For -source 1.4 and -source 1.3 options, +use -source14 and -source13 (yes, a hack!). +<p> +The <code>compile</code> element has the following attributes +which handle most of the other compiler arguments: +<ul> + <li><code>files</code>: .aj and .java files are treated as source files, + but .jar files are extracted and passed to the compiler + as <code>-injars</code> + </li><li><code>classpath</code>: directories and jar files for the classpath + </li><li><code>aspectpath</code>: binary aspects in jar files + </li><li><code>argfiles</code>: argument list files + </li> +</ul> +Paths for these are all relative to the test base directory, and +multiple entries are separated with commas. +<p> +Here is a cooked example that uses all <code>compiler</code> attributes: +<pre> + <ajc-test dir="new" title="attributes test"> + <compile files="Main.java,injar.jar" + staging="true" + options="-Xlint,-g:none" + argfiles="debug.lst,aspects/test.lst" + aspectpath="jars/requiredAspects.jar" + classpath="providedClassesDir,jars/required.jar"/> + <inc-compile tag="20"/> + </ajc-test> +</pre> + + +<h5>Unsupported compiler options</h5> +The harness does not support the following AspectJ compiler +options: -target {version}, -outjar {file}, -sourceroots {dir}, +-log {file}. +The following are used but specification is not supported: +-d {dir}, -sourceroots {dir}... + +<a name="background"></a> +<h4>Background information on the Harness</h4> +To make the test specifications as terse as possible, +harness components for + <code>inc-compile</code> and <code>run</code> elements +use information set up earlier by <code>compile</code>, +some of which is only implicit. +When a test is run, the harness creates a staging directory +for temporary files and a sandbox for sharing information +between test components, particularly classpath entries +shared between the compile and run components. +The compile and run components share classpath information +through the sandbox, adding default libraries: +<ul> + <li>Aside from any explicit classpath entries, + <code>compile</code> always includes the jars + <a href="../lib/tests/aspecjrt.jar"> + ../lib/tests/aspecjrt.jar</a> and + <a href="../lib/tests/testing-client.jar"> + ../lib/tests/testing-client.jar</a> + on the compile classpath. + </li> + <li><code>run</code> sets up its classpath as the compile + classpath plus the compile output (classes) directory + plus any entries on the aspectpath. + </li> +</ul> + +The harness provides some more advance behaviors, +which you might see specified in the tests. +For more information, see the API documentation for the harness. +<ul> + <li><u>option dominance and overriding</u>: + Both in test specifications and on the command line + you can force compiler options on or off. + These appear in the test specifications as options + with prefixes + '!' or '~' rather than '-' (e.g., '~emacssym' to force + the emacssym option off, even in tests that specify it). + </li> +</ul> +<hr> +<small>last updated January 7, 2002 </small> <!-- CVS variable --> + +</body> +</html> |