AspectJ release testing

This describes tests to validate AspectJ before a release. WARNING: this file is being superceded by ../build/release-checklist.txt. Some tests may be omitted for pre-release distributions, but all should be run for final releases. For information on how to build and run tests, see ../build/readme-build-and-test-aspectj.html and readme-tests-module.html.

(Most of these can and should be rolled into an Ant script for automation; until that script is written, this describes tests and could and/or should be run.)

Before a release, run the following tests using a Java 1.3 VM (the minimum required by the AspectJ tools):

  1. JUnit tests. Run these before the final build using tests/junitModules.xml (which runs all the modules/{module}/testsrc/{module}ModuleTests.java).

  2. All compiler tests in tests/ajcTests.xml and tests/ajcTestsFailing.xml. Run these before the final build using the test harness, as follows: 
      cd tests/
  java -jar {..}/testing-drivers-all.jar ajcTests.xml -release
    The -release option skips tests marked purejava or knownLimitation and only emits one-line results for each test: 
    PASS one test identifier()
FAIL another test identifier()
...
    (Use -help on the harness to see the full definition of the -release alias.) Save the output to the appropriate subdirectory of the org.aspectj/releases directory with the name ajcTests-result-{version}.txt. or ajcTests-result-{version}-emacssym.txt.

    Compare results with a prior release or test run using TestDiffs: 

      set CLASSPATH="{..}/testing-drivers-all.jar"
  java org.aspectj.testing.util.TestDiffs {first-run}.txt {second-run}.txt
    This will generate lists of tests fixed, broken, missing, or added. For a release, no tests from ajcTests.xml should be broken, and missing tests (those in the first run absent from the second run) should be justified (usually the test was removed as invalid). A test will show up as missing from the first run and added to the second run if it is renamed between runs. For a final release, no test should be FAIL unless it is deferred (including those from ajcTestsFailing.xml).

    For (at least final) releases, run all relevant variants:

    • With and without -emacssym
    • 1.3 and 1.4 VM's
    • without -release and with purejava tests. While these may fail, they should not crash the compiler.
  3. Examples. Run these on the built/installed release. The following should build and run the spacewar example: 
        cd {aspectj-install}/doc/examples
    {ant 1.5.1} -f build.xml
    Also run 2-3 other targets from Ant and some examples directly:
    • The target all runs all examples, and nonGui runs those that do not require manual operation;
    • The targettracing-bc does bytecode weaving
    • The targettracing-adapter-ajc compiles the tracing example using the task and the Ajc compiler adapter.
    • Run the compiled examples in a 1.1 VM (the minimum required for running AspectJ code).

    This quick test verifies that the aspectjtools.jar and aspectjrt.jar are installed and have a matching version, that the examples actually compile and run, etc.

  4. If there are any bugs which are closed with this release but which do not have an automated test case run using JUnit or the harness, those bugs should be manually tested against the release.

  5. Any other tests warranted by release objectives. E.g., make sure it works as expected for any demos or for any clients (e.g., Emacs, AJDT).
  6. Compile large open-source Java projects using AspectJ.
  7. Run the AspectJ compiler against the Jacks test suite.
  8. Manually step through the ajde/script.html on the AspectJ browser.

Some failures with past releases to consider checking: