From 4678f4e40b2eb3090b4fa0e5e18b988c3817df6c Mon Sep 17 00:00:00 2001 From: wisberg Date: Tue, 7 Jan 2003 23:57:28 +0000 Subject: [PATCH] initial draft of doc on writing compiler tests --- tests/readme-writing-compiler-tests.html | 432 +++++++++++++++++++++++ 1 file changed, 432 insertions(+) create mode 100644 tests/readme-writing-compiler-tests.html diff --git a/tests/readme-writing-compiler-tests.html b/tests/readme-writing-compiler-tests.html new file mode 100644 index 000000000..8cc1a4815 --- /dev/null +++ b/tests/readme-writing-compiler-tests.html @@ -0,0 +1,432 @@ + + +Writing tests for the AspectJ compiler + + +

Writing tests for the AspectJ compiler +

+The AspectJ project has a harness +which reads test specification files and run tests. +The tests are usually simple scenarios like +"compile and run" or "compile expecting error", +but may involve multiple files, incremental compilation, +classpath or aspectpath entries, etc. +This document shows how to write tests that can be run +by the harness and suggests some patterns to use in +test code, discussing + + +Most people just writing a test case need to know only the +information in + Simple test definitions and + Test source files. + +

Related documents: +

+ + +

Simple Test definitions

+ +Here is a simple example to compile Main.java +and expect an error on line 10: +
+    <ajc-test dir="new" title="simple error test">
+        <compile files="Main.java">
+            <message kind="error" line="10"/>
+        </compile>
+    </ajc-test>
+
+ +

Here is an example to compile +pack/Aspect.java and +pack2/Main.java and +run the main class: +

+    <ajc-test dir="new" title="simple run test">
+        <compile files="pack/Aspect.java,pack1/Main.java"/>
+        <run class="pack1.Main"/>
+    </ajc-test>
+
+The compile and run steps of a given ajc-test share a common +sandbox, so (e.g.,) the run step knows to set its classpath using +the classes directory generated by the compile step. + +

More complex compilations are discussed in + Compiler Options below. + + +

Test source files

+ +The dir attribute in the ajc-test +element specifies a base test directory +relative to the directory of the test specification file. +All paths are specified relative to this base test directory. + +E.g., the last example used dir="new" and +presumed the following directory structure: +
+    {some dir}                  # test specification directory
+        {testDefinition}.xml
+        new/                    # test base directory
+          pack/Aspect.java
+          pack2/Main.java
+
+Test cases with only one file in the default package can often +share directories (e.g., see the many files in new/), +but usually a test case has its own directory. + + +

Incremental tests

+ +Incremental tests are more complex because they involve +updating source files before recompiling. +Here's an example that + +
+    <ajc-test dir="new/incremental1" title="incremental test">
+        <compile staging="true"
+                   files="Main.java,DeleteMe.java"/>
+        <run class="Main"/>
+        <inc-compile tag="20">
+            <message kind="error" line="15">
+        </inc-compile>
+        <inc-compile tag="30"/>
+        <run class="Main"/>
+    </ajc-test>
+
+Take particular note of two attributes, +compile's "staging" +and inc-compile's "tag". +First, the compile task enables staging, which copies the +test source files to a temporary directory so they can be +updated during the test without changing the actual sources. +Second, incremental compiles specify a tag which identifies +files to be deleted or copied into the test source staging directory +before recompiling. The tag is a suffix identifying +files in the test source directory specifying how the sources should +be changed before that incremental compilation step. +If there is a prefixing suffix "delete", then the file is deleted; +otherwise, the file is copied (with the effect either of updating +an existing file or adding a new file). + +

+Thus, to understand what's happening in an incremental test +requires comparing the tags with the files specified in +the test source directory. +For example, here is a directory layout for the test above: +

+    {some dir}
+        {testDefinition}.xml
+        new/
+          incremental1/
+            DeleteMe.delete.30.java
+            DeleteMe.java
+            Main.20.java
+            Main.30.java
+            Main.java
+            NewFile.30.java
+
+The result will be one compile and two re-compiles: +
    +
  1. Compile Main.java and DeleteMe.java
    2. +
  2. Update Main.java with the contents of Main.20.java + and recompile
    3. +
  3. Delete DeleteMe.java, + add NewFile.java, + update Main.java with the contents of Main.30.java + and recompile
    4. +
+ + +

Verifying test steps

+As seen above, two ways to verify that a compile was successful +are to run the corresponding class or check the compiler messages. +More generally, the harness can verify compile/run test steps by detecting the +following things and comparing against expected behavior: +

+ + + + + + + + + + + + + + + + + + + + + + + +
Detect Evaluate
Exceptions signal failure
Result value heuristically compare with expected: + compiles not expecting errors are expected + to return a normal result status, and vice-versa.
Messages (e.g., compiler warnings and errors)Compare with expected messages
Directory changes (e.g., .class files created) Compare with expected changes
Runtime behaviorUse Tester in test source code + to signal events for comparison with expected events.
+

+ + +

Messages
+ +In a test definition, a nested message element +specifies a condition on the successful completion of the nesting +ajc-test sub-element. In the earlier example, if +the harness does not detect an error message on line 10 or if +there are unexpected messages, then the compile step will be +reported as failing: +
+    <ajc-test dir="new" title="simple error test">
+        <compile files="Main.java">
+            <message kind="error" line="10"/>
+        </compile>
+    </ajc-test>
+
+Expected messages can be specified as sub-elements for the three + ajc-test elements + compile, + inc-compile, and + run. +Messages require a kind (error or warning) and a line. +To make specification easier, if an error is specified for a line, +the harness accepts as expected any number of errors on that line. +

+The current harness has only been tested to validate compilation +based on line numbers. +Filename-based comparison is disabled as untested/unused, +and run messages are handled wrongly; +line-number comparison will fail since +run messages do not have line numbers. + + +

Changes in an output directory
+ +As with messages, specifying directory changes as a nested element +operates as a condition on the successful completion of the +nesting element. The harness will check for files added, removed, +updated, or unchanged from a given directory. The directory is +specified explicitly or using a token for the shared classes or +run directory. For even more brevity, the harness supports a +default suffix for the files. +

+Directory changes have been used only to validate changes in +the classes directory. +The current harness defaults to using the classes directory, +and when using the classes directory uses .class +as a default suffix. +

+Here's an example specification: +

+    <ajc-test dir="new/dirchanges-test" title="dir-changes test">
+        <compile staging="true"
+                   files="Main.java,DeleteMe.java,Unchanged.java"/>
+        <inc-compile tag="20">
+            <dir-changes updated="Main" 
+                         removed="DeleteMe"
+                       unchanged="Unchanged"/>
+        </inc-compile>
+    </ajc-test>
+
+It checks after a recompile that + + + +
Runtime behavior
+Code that tests aspects often falls into the pattern of comparing +expected and actual events/signals. For example, to prove that +before advice in fact ran before a particular method execution, +you might generated and expect signals corresponding to + +
    +
  1. method-call
    2. +
  2. before advice execution
    3. +
  3. method-execution
    4. +
+ +The Tester utility class provides API's for signalling +actual and expecting events and comparing the two. +Typically, events are symbolized and compared as String. +Here's a small sample test case that for the scenario above: +
+import org.aspectj.testing.Tester;
+
+public class Main implements Runnable {
+    public static void main(String[] args) {
+        Tester.expectEvent("before advice");
+        Tester.expectEvent("execute run");
+        new Main().run();     
+        Tester.checkAllEvents();
+    }
+    public void run() {
+        Tester.event("execute run");
+    }
+}
+
+aspect A {
+    before () : target(Runnable) && execution(void run()) {
+         Tester.event("before advice");
+    }
+}
+
+ +If either the advice or the method does not run, +the harness will report a failure. + +

+Tester also has methods that operate like +JUnit assertions as idioms to detect differences in +expected and actual values, signalling appropriately. +

+ +Tester is at + + ../testing-client/src/org/aspectj/testing/Tester.java +and is built into + + ../lib/tests/testing-client.jar +which is included on the classpath by the compile and run steps. + +

You can write runtime test cases without using Tester; +simply throw some exception from the main thread to signal failure. + + +

Compiler options

+The harness does not support all of the AspectJ 1.1 compiler options. +Flags are mainly supported through the a comma-delimited list in +the options attribute: +
+    <ajc-test dir="new" title="lint test">
+        <compile files="LintTest.java" 
+                 options="-Xlint,-emacssym">
+            <message kind="warning" line="22">
+        </compile>
+
+This should work even for complex single-arg options like +-g:none, but will fail for comma-delimited single-arg options like +-g:lines,vars because the comma delimiters +are ambiguous (yes, a design bug!). +For -source 1.4 and -source 1.3 options, +use -source14 and -source13 (yes, a hack!). +

+The compile element has the following attributes +which handle most of the other compiler arguments: +

    +
  • files: .aj and .java files are treated as source files, + but .jar files are extracted and passed to the compiler + as -injars +
  • classpath: directories and jar files for the classpath +
  • aspectpath: binary aspects in jar files +
  • argfiles: argument list files +
    • +
+Paths for these are all relative to the test base directory, and +multiple entries are separated with commas. +

+Here is a cooked example that uses all compiler attributes: +

+    <ajc-test dir="new" title="attributes test">
+        <compile files="Main.java,injar.jar" 
+               staging="true"
+               options="-Xlint,-g:none"
+              argfiles="debug.lst,aspects/test.lst"
+            aspectpath="jars/requiredAspects.jar"
+             classpath="providedClassesDir,jars/required.jar"/>
+        <inc-compile tag="20"/>
+    </ajc-test>
+
+ + +
Unsupported compiler options
+The harness does not support the following AspectJ compiler +options: -target {version}, -outjar {file}, -sourceroots {dir}, +-log {file}. +The following are used but specification is not supported: +-d {dir}, -sourceroots {dir}... + + +

Background information on the Harness

+To make the test specifications as terse as possible, +harness components for + inc-compile and run elements +use information set up earlier by compile, +some of which is only implicit. +When a test is run, the harness creates a staging directory +for temporary files and a sandbox for sharing information +between test components, particularly classpath entries +shared between the compile and run components. +The compile and run components share classpath information +through the sandbox, adding default libraries: +
    +
  • Aside from any explicit classpath entries, + compile always includes the jars + + ../lib/tests/aspecjrt.jar and + + ../lib/tests/testing-client.jar + on the compile classpath. +
    • +
  • run sets up its classpath as the compile + classpath plus the compile output (classes) directory + plus any entries on the aspectpath. +
    • +
+ +The harness provides some more advance behaviors, +which you might see specified in the tests. +For more information, see the API documentation for the harness. +
    +
  • option dominance and overriding: + Both in test specifications and on the command line + you can force compiler options on or off. + These appear in the test specifications as options + with prefixes + '!' or '~' rather than '-' (e.g., '~emacssym' to force + the emacssym option off, even in tests that specify it). +
    • +
+
+last updated January 7, 2002 + + + -- 2.39.5