123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675 |
- <html>
- <!-- <![CDATA[ putDataHere ]]> -->
- <head>
- <title>Writing tests for the AspectJ compiler
- </title>
- </head>
- <body>
- <h2>Writing tests for the AspectJ compiler
- </h2>
- The AspectJ project has a harness
- which reads test specification files and run tests.
- The tests are usually simple scenarios like
- "compile and run" or "compile expecting error",
- but may involve multiple files, incremental compilation,
- classpath or aspectpath entries, etc.
- This document shows how to write tests that can be run
- by the harness and suggests some patterns to use in
- test code, discussing
-
- <ul>
- <li><a href="#simple">Simple test definitions</a></li>
- <li><a href="#sourceFiles">Test source files</a></li>
- <li><a href="#incremental">Incremental tests</a></li>
- <li><a href="#verifying">Verifying test steps</a>
- <ul>
- <li><a href="#messages">Messages</a></li>
- <li><a href="#dirchanges">Changes in an output directory</a></li>
- <li><a href="#tester">Testing runtime behavior</a></li>
- </ul></li>
- <li><a href="#forking">Forking at runtime</a></li>
- <li><a href="#compilerOptions">Compiler Options</a></li>
- <li><a href="#background">Harness background</a></li>
- </ul>
- Most people just writing a test case need to know only the
- information in
- <a href="#simple">Simple test definitions</a> and
- <a href="#sourceFiles">Test source files</a>.
-
- <p>Related documents:</p>
- <ul>
- <li>For information on running the harness, see
- <a href="readme-tests-module.html">
- readme-tests-module.html</a>
- </li>
- <li>For example test definition files, see
- <a href="ajcTests.xml">ajcTests.xml</a> and
- <a href="ajcTestsFailing.xml">ajcTestsFailing.xml</a>.
- </li>
- </ul>
-
- <a name="simple"></a>
- <h4>Simple Test definitions</h4>
-
- Test definitions are specified in XML files.
- Here is a simple example to compile <code>Main.java</code>
- and expect an error on line 10 in a file <code>ajcSample.xml</code>:
- <pre>
- <!DOCTYPE suite SYSTEM "../tests/ajcTestSuite.dtd">
- <suite>
- <ajc-test dir="new" title="simple error test">
- <compile files="Main.java">
- <message kind="error" line="10"/>
- </compile>
- </ajc-test>
- </suite>
- </pre>
-
- <p>Here is an example to compile
- <code>pack/Aspect.java</code> and
- <code>pack2/Main.java</code> and
- run the main class:
- </p>
- <pre>
- <ajc-test dir="new" title="simple run test">
- <compile files="pack/Aspect.java,pack1/Main.java"/>
- <run class="pack1.Main"/>
- </ajc-test>
- </pre>
- The compile and run steps of a given ajc-test share a common
- sandbox, so (e.g.,) the run step knows to set its classpath using
- the classes directory generated by the compile step.
-
- <p>More complex compilations are discussed in
- <a href="#compilerOptions">Compiler Options</a> below.
- </p>
- <a name="sourceFiles"></a>
- <h4>Test source files</h4>
-
- The <code>dir</code> attribute in the <code>ajc-test</code>
- element specifies a base test directory
- relative to the directory of the test specification file.
- All paths are specified relative to this base test directory.
-
- E.g., the last example used <code>dir="new"</code> and
- presumed the following directory structure:
- <pre>
- {some dir} # test specification directory
- {testDefinition}.xml
- new/ # test base directory
- pack/Aspect.java
- pack2/Main.java
- </pre>
- Test cases with only one file in the default package can often
- share directories (e.g., see the many files in <a href="new">new/</a>),
- but usually a test case has its own directory.
-
- <a name="incremental"></a>
- <h4>Incremental tests</h4>
-
- Incremental tests are more complex because they involve
- updating source files before recompiling.
- Here's an example that
- <ul>
- <li>compiles Main.java,
- </li><li>runs it,
- </li><li>updates the source to introduce an error,
- </li><li>incrementally compile it (and detect the error)
- </li><li>updates the source to fix the error,
- </li><li>incrementally compile it (successfully), and
- </li><li>runs it again.
- </li>
- </ul>
- <pre>
- <ajc-test dir="new/incremental1" title="incremental test">
- <compile staging="true"
- sourceroots="."
- options="-incremental" />
- <run class="Main"/>
- <inc-compile tag="20">
- <message kind="error" line="15">
- </inc-compile>
- <inc-compile tag="30"/>
- <run class="Main"/>
- </ajc-test>
- </pre>
-
- To understand what's happening in this test would require
- looking at the source directory to see which files are tagged
- "20" and "30". But before walking through that, there's a
- variation of incremental building for AJDE. (The AJDE wrapper
- around the <code>ajc</code> compiler can also be driven by the
- test harness.)
-
- <p>
- In AJDE, incremental tests also involve the notion of "fresh builds",
- i.e., when the test reuses the same compiler and build configuration
- but rebuilds from scratch. In that case, there is still the
- question of whether source files should be updated; if not,
- the tag can have the special value "same". For example, if
- the last example had two more lines:
- </p>
- <pre>
- ...
- <inc-compile tag="30"/>
- <run class="Main"/>
-
- <inc-compile fresh="true" tag="same"/>
- <run class="Main"/>
- </ajc-test>
- </pre>
- The test would complete by completely rebuilding the same files
- and then running the main class. This option has no effect on
- the normal (ajc) compiler, and requires specifying
- <code>-ajdeCompiler</code> to the harness or compile step
- as an argument.
- <p/>
-
- To recap the attributes of note for setting up incremental tests:
- <ul>
- <li><code>compile</code> <code>staging="true"</code>:
- Incremental tests require staging, which copies the
- test source files to a temporary directory so they can be
- updated during the test without changing the actual sources.
- </li>
- <li><code>compile</code> <code>sourceroots="{..}"</code>:
- incremental mode only takes source files in the form of
- <code>-sourceroots</code> entries.
- </li>
- <li><code>compile</code> <code>options="-incremental{,..}"</code>:
- Specify the <code>-incremental</code> option to signal
- incremental mode to the compiler. If you do not include this,
- the harness will still run an incremental test, but the compiler
- will not do its additional checking it on input options.
- </li>
- <li><code>inc-compile</code> <code>tag="{##}"</code>:
- The required tag attribute is a suffix identifying
- files in the test source directory specifying how the sources should
- be changed before that incremental compilation step.
- If there is a prefixing suffix "delete", then the file is deleted;
- otherwise, the file is copied (with the effect either of updating
- an existing file or adding a new file).
- If the tag is "same", then no files are changed.
- </li>
- <li><code>inc-compile</code> <code>fresh="true"</code>:
- With the AJDE compiler, you can rebuild the current build
- configuration in its entirety. (By contrast, doing another
- <compile> task would re-initialize the AJDE compiler.)
- This option is ignored unless <code>-ajdeCompiler</code>
- is passed to the harness on the command line or to the
- immediately preceding <compile> task in the options.
- </li>
- </ul>
-
- <p/>
- Now, to get back to the question of
- what exactly is happening in an incremental test. To do so,
- compare the tags with the files specified in
- the test source directory; the tagged files are the updates
- for that particular step. (By convention the tags are numeric
- and in order, but they need not be.)
- For example, here are some sources for the test above:
- <pre>
- {some dir}
- {testDefinition}.xml
- new/
- incremental1/
- DeleteMe.delete.30.java
- DeleteMe.java
- Main.20.java
- Main.30.java
- Main.java
- NewFile.30.java
- </pre>
- Comparing this with the test specification, you can see
- the harness will run one compile and two re-compiles:
- <ol>
- <li>Initially compile <code>Main.java</code> and <code>DeleteMe.java</code>
- <pre>
- <compile staging="true"
- files="Main.java,DeleteMe.java"/>
-
- {some dir}
- {testDefinition}.xml
- new/
- incremental1/
- ...
- DeleteMe.java
- ...
- Main.java
- ...
- </pre>
- </li>
- <li>For incremental tag 20,
- update <code>Main.java</code> with the contents of
- <code>Main.20.java</code>
- and recompile, expecting an error on line 15:
- <pre>
- <inc-compile tag="20">
- <message kind="error" line="15">
- </inc-compile>
-
- {some dir}
- {testDefinition}.xml
- new/
- incremental1/
- ...
- Main.20.java
- ...
- </pre></li>
-
- <li>For incremental tag 30,
- delete <code>DeleteMe.java</code>,
- add <code>NewFile.java</code>,
- update <code>Main.java</code> with the contents of
- <code>Main.30.java</code>
- and recompile with no error or warning messages:
- <pre>
- <inc-compile tag="30"/>
-
- {some dir}
- {testDefinition}.xml
- new/
- incremental1/
- DeleteMe.delete.30.java
- ...
- Main.30.java
- ...
- NewFile.30.java
- </pre>
- </li>
- </ol>
-
- <a name="verifying"></a>
- <h4>Verifying test steps</h4>
- As seen above, two ways to verify that a compile was successful
- are to run the corresponding class or check the compiler messages.
- More generally, the harness can verify compile/run test steps by detecting the
- following things and comparing against expected behavior:
- <p/>
-
- <table border="1" cellpadding="1">
-
- <tr><th>Detect </th>
- <th>Evaluate</th>
- </tr>
- <tr><td>Exceptions </td>
- <td>signal failure </td>
- </tr>
-
- <tr><td>Result value </td>
- <td>heuristically compare with expected:
- compiles not expecting errors are expected
- to return a normal result status, and vice-versa.</td>
- </tr>
- <tr><td>Messages (e.g., compiler warnings and errors)</td>
- <td>Compare with expected messages</td>
- </tr>
-
- <tr><td>Directory changes (e.g., <code>.class</code> files created) </td>
- <td>Compare with expected changes</td>
- </tr>
- <tr><td>Runtime behavior</td>
- <td>Use <code>Tester</code> in test source code
- to signal events for comparison with expected events.
- To signal failure when running a forked test without Tester,
- throw an exception.</td>
- </tr>
- </table>
- <p/>
-
- <a name="messages"></a>
- <h5>Messages</h5>
-
- In a test definition, a nested <code>message</code> element
- specifies a condition on the successful completion of the nesting
- ajc-test sub-element. In the earlier example, if
- the harness does not detect an error message on line 10 or if
- there are unexpected messages, then the compile step will be
- reported as failing:
- <pre>
- <ajc-test dir="new" title="simple error test">
- <compile files="Main.java">
- <message kind="error" line="10"/>
- </compile>
- </ajc-test>
- </pre>
- Expected messages can be specified as sub-elements for the three
- <code>ajc-test</code> elements
- <code>compile</code>,
- <code>inc-compile</code>, and
- <code>run</code>.
- Messages require a kind (error or warning) and a line.
- To make specification easier, if an error is specified for a line,
- the harness accepts as expected any number of errors on that line.
- <p>
- Most messages fall into those categories.
- However, an IMessage also has a Throwable thrown, a String detail,
- and a List of ISourceLocation (essentially, "see also", to point
- to other relevant places in the code). The thrown element is not
- supported, but you can specify the others:
- </p>
- <pre>
- <ajc-test dir="new" title="simple error test">
- <compile files="Main.java">
- <message
- kind="error"
- line="10"
- file="Main.java"
- text="This join point should never happen!"
- detail="(Find the declaring code below.)">
- <source line="12" file="Main.java"/>
- <source line="14" file="Main.java"/>
- <message>
- </compile>
- </ajc-test>
- </pre>
-
- This compiler-error test specifies a single error message triggered
- on line 10 of Main.java, with some text and details and two other
- source locations that are relevant, on lines 12 and 14 of the same
- file.
-
- <p>
- When specifying messages, be sure to provide enough detail that
- the harness can distinguish expected messages. For example, if you
- only specify the line number, then it will match the message in
- any file (if there is more than one). If there are two or more
- messages expected on a line, provide enough information
- to distinguish them. If you are using text or detail attributes,
- do not use one string that is a prefix of the other, since it
- will match either message, and the other message might not match.
- </p>
- <p>
- The "info" messages are special in that they are normally ignored.
- To specify expected "info" messages, you have to list all the
- messages the compiler will issue, which can vary depending on
- the compiler settings. Use the option <code>^verbose</code> to
- force the compiler's <code>-verbose</code> option off.
- </p>
- <p>
- By the same token, if you don't specify any extra source locations,
- then they will not be checked. If you think it is a bug if they
- are issued, then you have to specify one if them. (There is
- currently no way to specify that a message has no extra
- source locations.)
- </p>
-
- <a name="dirchanges"></a>
- <h5>Changes in an output directory</h5>
-
- As with messages, specifying directory changes as a nested element
- operates as a condition on the successful completion of the
- nesting element. The harness will check for files added, removed,
- updated, or unchanged from a given directory. The directory is
- specified explicitly or using a token for the shared classes or
- run directory. For even more brevity, the harness supports a
- default suffix for the files.
- <p>
- Directory changes have been used only to validate changes in
- the classes directory.
- The current harness defaults to using the classes directory,
- and when using the classes directory uses <code>.class</code>
- as a default suffix.
- </p>
- <p>
- Here's an example specification:
- </p>
- <pre>
- <ajc-test dir="new/dirchanges-test" title="dir-changes test">
- <compile staging="true"
- files="Main.java,DeleteMe.java,Unchanged.java"/>
- <inc-compile tag="20">
- <dir-changes updated="Main"
- removed="DeleteMe"
- unchanged="Unchanged"/>
- </inc-compile>
- </ajc-test>
- </pre>
- It checks after a recompile that
- <ul>
- <li><code>Main.class</code> was updated</li>
- <li><code>DeleteMe.class</code> was deleted</li>
- <li><code>Unchanged.class</code> was not touched</li>
- </ul>
-
- <a name="tester"></a>
- <h5>Runtime behavior</h5>
- Code that tests aspects often falls into the pattern of comparing
- expected and actual events/signals. For example, to prove that
- before advice in fact ran before a particular method execution,
- you might generated and expect signals corresponding to
-
- <ol>
- <li>method-call</li>
- <li>before advice execution</li>
- <li>method-execution</li>
- </ol>
-
- The <code>Tester</code> utility class provides API's for signalling
- actual and expecting events and comparing the two.
- Typically, events are symbolized and compared as String.
- Here's a small sample test case that for the scenario above:
- <pre>
- import org.aspectj.testing.Tester;
-
- public class Main implements Runnable {
- public static void main(String[] args) {
- Tester.expectEvent("before advice");
- Tester.expectEvent("execute run");
- new Main().run();
- Tester.checkAllEvents();
- }
- public void run() {
- Tester.event("execute run");
- }
- }
-
- aspect A {
- before () : target(Runnable) && execution(void run()) {
- Tester.event("before advice");
- }
- }
- </pre>
-
- If either the advice or the method does not run,
- the harness will report a failure.
-
- <p>
- <code>Tester</code> also has methods that operate like
- JUnit assertions as idioms to detect differences in
- expected and actual values, signalling appropriately.
- </p>
-
- <code>Tester</code> is at
- <a href="../testing-client/src/org/aspectj/testing/Tester.java">
- ../testing-client/src/org/aspectj/testing/Tester.java</a>
- and is built into
- <a href="../lib/tests/testing-client.jar">
- ../lib/tests/testing-client.jar</a>
- which is included on the classpath by the compile and run steps.
-
- <p>You can write runtime test cases without using Tester;
- simply throw some exception from the main thread to signal failure.
- </p>
-
- <a name="forking"></a>
- <h4>Forking and load-time weaving</h4>
- <p>You can fork the tests that are run by setting global properties
- or by setting attributes to run. One attribute,
- <code>aspectpath</code>, forces the run to fork and uses
- load-time weaving to weave aspects with the classpath.
- </p>
- <p>To fork for all tests, set the global system property
- <code>javarun.fork</code> to "true". You can also
- set other properties to control how forking happens.
- (By default, forking uses the java executable that was
- used to run the harness.)
- The following causes all <code>run</code> steps to
- fork into the specified Java 1.1 VM.
- </p>
- <pre>
- java -Djavarun.fork=true \
- -Djavarun.java=c:/home/jdk11/bin/java.exe \
- -Djavarun.java.home=c:/home/jdk11 \
- -Djavarun.bootclasspath=c:/home/jdk11/lib/classes.zip \
- -Djavarun.vmargs=-Dname=value,-Dname2="value 2" \
- org.aspectj.testing.drivers.Harness ...
- </pre>
-
- <p>You can fork a specific test by setting
- the <code>fork</code> attribute (and optionally the
- <code>vmargs</code> attribute):
- </p>
- <pre>
- <ajc-test dir="new" title="show forking">
- <compile files="Main.java"/>
- <run class="Main" fork="true"
- vmargs="-Dname=value,-Dname2="value 2"/>
- </ajc-test>
- </pre>
-
- <p>As a special case of forking, load-time weaving sets up
- a Java 1.4 or later VM using our weaving class loader,
- if you specify the <code>aspectpath</code> attribute to
- <code>run</code>.
- In the following example, the main class and the aspect
- are compiled separately. The main class is then woven
- and run using load-time weaving.</p>
- <pre>
- <ajc-test dir="new" title="show LTW">
- <compile files="Aspect.java" outjar="out.jar"/>
- <compile files="Main.java"/>
- <run class="Main" aspectpath="out.jar"/>
- </ajc-test>
- </pre>
-
- <a name="compilerOptions"></a>
- <h4>Compiler options</h4>
- The harness does not support all of the AspectJ 1.1 compiler options.
- Flags are mainly supported through the a comma-delimited list in
- the <code>options</code> attribute:
- <pre>
- <ajc-test dir="new" title="lint test">
- <compile files="LintTest.java"
- options="-Xlint,-emacssym,-source,1.4">
- <message kind="warning" line="22">
- </compile>
- </pre>
- This should work even for complex single-arg options like
- <code>-g:none</code>, but will fail for comma-delimited single-arg options like
- <code>-g:lines,vars</code> because the comma delimiters
- are ambiguous (yes, a design bug!).
- <p/>
- The <code>compile</code> element has the following attributes
- which handle most of the other compiler arguments:
- <ul>
- <li><code>files</code>: .aj and .java files are treated as source files,
- but .jar/zip files are extracted and passed to the compiler
- as <code>-injars</code>
- and readable directories are passed as <code>-inpath</code>.
- </li><li><code>classpath</code>: directories and jar files for the classpath
- </li><li><code>aspectpath</code>: binary aspects in jar files
- </li><li><code>argfiles</code>: argument list files
- </li><li><code>sourceroots</code>: root directories for source files
- </li><li><code>xlintfile</code>: override org.aspectj.weaver.XlintDefault.properties
- </li>
- </ul>
- Paths for these are all relative to the test base directory, and
- multiple entries are separated with commas.
- (Use only one entry for xlintfile.)
- <p/>
- Here is a cooked example that uses all <code>compiler</code> attributes:
- <pre>
- <ajc-test dir="new" title="attributes test">
- <compile files="Main.java,injar.jar,some-directory"
- staging="true"
- options="-Xlint,-g:none"
- argfiles="debug.lst,aspects/test.lst"
- aspectpath="jars/requiredAspects.jar"
- xlintfile="ignore-all-but-typenotfound.properties"
- classpath="providedClassesDir,jars/required.jar"/>
- <inc-compile tag="20"/>
- </ajc-test>
- </pre>
-
- <h5>Test-only compiler attributes</h5>
- The following attributes of the compiler entity dictate harness behavior:
-
- <ul>
- <li><u>badInput</u>:
- To test invalid input, set the compiler <code>badInput</code> attribute
- to "<code>true</code>". This prevents the harness from aborting a test
- because a specified input file was not found. (However, there is no way
- to specify bad input for a directory in the files attribute intended for
- -inpath, because the harness selects readable directories.)
- </li>
-
-
- <li><u>includeClassesDir</u>:
- Set this in order to include the output classes directory explicitly
- on the classpath.
- </li>
- <li><u>reuseCompiler</u>:
- Set this to re-use a compiler from a previous compiler run.
- (This tests statefulness of a compiler across non-incremental runs.)
- </li>
- </ul>
-
- <h5>Unsupported compiler options</h5>
- The harness does not support the following AspectJ compiler
- options: <code>-outjar {file}, -log {file}</code>.
- (<code>-d {dir}</code> is used but specification is not supported.)
-
-
-
- <a name="background"></a>
- <h4>Background information on the Harness</h4>
- To make the test specifications as terse as possible,
- harness components for
- <code>inc-compile</code> and <code>run</code> elements
- use information set up earlier by <code>compile</code>,
- some of which is only implicit.
- When a test is run, the harness creates a staging directory
- for temporary files and a sandbox component for sharing information
- between test components, particularly classpath entries
- shared between the compile and run components.
- The compile and run components share classpath information
- through the sandbox, adding default libraries:
- <ul>
- <li>Aside from any explicit classpath entries,
- <code>compile</code> always includes the jars
- <a href="../lib/tests/aspecjrt.jar">
- ../lib/tests/aspecjrt.jar</a> and
- <a href="../lib/tests/testing-client.jar">
- ../lib/tests/testing-client.jar</a>
- on the compile classpath.
- </li>
- <li><code>run</code> sets up its classpath as the compile
- classpath plus the compile output (classes) directory
- plus any entries on the aspectpath.
- </li>
- </ul>
-
- The harness provides some more advance behaviors,
- which you might see specified in the tests.
- For more information, see the API documentation for the harness
- (<a href="../testing-drivers/src/org/aspectj/testing/drivers/package.html">
- org/aspectj/testing/drivers/package.html</a>).
- <ul>
- <li><u>option dominance and overriding</u>:
- Both in test specifications and on the command line
- you can force compiler options on or off.
- (Forcing means that, e.g., even if an option is enabled in the
- test specification, it can be disabled from the command line.)
- These appear in the test specifications as options
- with prefixes
- '!' or '^' rather than '-' (e.g., '^emacssym' to force
- the emacssym option off, even in tests that specify it).
- </li>
- </ul>
- <hr/>
- <small>last updated March 8, 2004 </small> <!-- CVS variable -->
-
- </body>
- </html>
|