123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142 |
- <html>
- <title>AspectJ release testing</title>
- <body>
- <h1>AspectJ release testing</h1>
- This describes tests to validate AspectJ before a release.
-
- <u>WARNING<u>: this file is being superceded by
- <a href="../build/release-checklist.txt">
- ../build/release-checklist.txt</a>.
-
- Some tests may be omitted for pre-release distributions, but all
- should be run for final releases.
- For information on how to build and run tests, see
- <a href="../build/readme-build-and-test-aspectj.html">
- ../build/readme-build-and-test-aspectj.html</a>
- and
- <a href="readme-tests-module.html">
- readme-tests-module.html</a>.
-
- <p>(Most of these can and should be rolled into an Ant script
- for automation; until that script is written, this describes
- tests and could and/or should be run.)
-
- <p>Before a release, run the following tests using a Java 1.3 VM
- (the minimum required by the AspectJ tools):
- <ol>
- <li>JUnit tests. Run these before the final build
- using <code>tests/junitModules.xml</code>
- (which runs all the
- <code>modules/{module}/testsrc/{module}ModuleTests.java</code>).
- <p>
- </li>
-
- <li>All compiler tests in <code>tests/ajcTests.xml</code>
- and <code>tests/ajcTestsFailing.xml</code>.
- Run these before the final build using the test harness,
- as follows:
- <pre>
- cd tests/
- java -jar {..}/testing-drivers-all.jar ajcTests.xml -release
- </pre>
- The <code>-release</code> option skips tests
- marked <code>purejava</code> or <code>knownLimitation</code>
- and only emits one-line results for each test:
- <pre>
- PASS one test identifier()
- FAIL another test identifier()
- ...
- </pre>
- (Use <code>-help</code> on the harness to see the full
- definition of the <code>-release</code> alias.)
- Save the output
- to the appropriate subdirectory of the
- <a href="../../releases">org.aspectj/releases</a> directory
- with the name
- <code>ajcTests-result-{version}.txt</code>.
- or <code>ajcTests-result-{version}-emacssym.txt</code>.
- <p>
- Compare results with a prior release or test run using TestDiffs:
- <pre>
- set CLASSPATH="{..}/testing-drivers-all.jar"
- java org.aspectj.testing.util.TestDiffs {first-run}.txt {second-run}.txt
- </pre>
- This will generate lists of tests fixed, broken, missing, or
- added. For a release, no tests from <code>ajcTests.xml</code>
- should be broken, and missing tests (those in the first run absent from the
- second run) should be justified (usually the test was
- removed as invalid). A test will show up as
- missing from the first run and added to the second run if
- it is renamed between runs.
- For a final release, no test should be <code>FAIL</code> unless it is
- deferred (including those from <code>ajcTestsFailing.xml</code>).
- <p>
- For (at least final) releases, run all relevant variants:
- <ul>
- <li>With and without <code>-emacssym</code></li>
- <li>1.3 and 1.4 VM's</li>
- <li>without -release and with purejava tests. While
- these may fail, they should not crash the compiler.</li>
- </ul>
-
- <li>Examples. Run these on the built/installed release.
- The following should build and run the spacewar example:
- <pre>
- cd {aspectj-install}/doc/examples
- {ant 1.5.1} -f build.xml
- </pre>
- Also run 2-3 other targets from Ant and some examples directly:
- <ul>
- <li>
- The target <code>all</code>
- runs all examples, and <code>nonGui</code> runs those
- that do not require manual operation;
- </li>
-
- <li>The target<code>tracing-bc</code> does bytecode weaving
- </li>
- <li>The target<code>tracing-adapter-ajc</code> compiles the
- tracing example using the <javac> task and the
- Ajc compiler adapter.
- </li>
- <li>
- Run the compiled examples in a 1.1 VM
- (the minimum required for running AspectJ code).
- </li>
- </ul>
- <p>
- This quick test verifies that the
- <code>aspectjtools.jar</code> and <code>aspectjrt.jar</code>
- are installed and have a matching version, that the examples actually
- compile and run, etc.
- <p>
- </li>
- <li>If there are any bugs which are closed with this release
- but which do not have an automated test case run using
- JUnit or the harness, those bugs should be manually tested
- against the release.
- <p>
- </li>
- <li>Any other tests warranted by release objectives.
- E.g., make sure it works as expected for any demos or
- for any clients (e.g., Emacs, AJDT).
- </li>
- <li>Compile large open-source Java projects using AspectJ.
- </li>
- <li>Run the AspectJ compiler against the Jacks test suite.
- </li>
- <li>Manually step through the
- <a href="ajde/script.html">ajde/script.html</a>
- on the AspectJ browser.
- </ol>
- <p>Some failures with past releases to consider checking:
- <ul>
- <li>debug info not included in .class files</li>
- <li>API not included in documentation</li>
- <li>Ant variables not filtered into doc files (e.g., ant-tasks.html)</li>
- <li>Manual dates not changed in docs or readme</li>
- <li>Inconsistent version information between runtime and tools classes</li>
- </ul>
-
- </body>
- </html>
|