TODO: write test-release.xml ant script

21 years ago · 0557b906e8
--- a/build/readme-build-and-test-aspectj.html
+++ b/build/readme-build-and-test-aspectj.html
@@ -305,111 +305,10 @@ For bug fixes, associated tests in
  <code>tests/ajcTests.xml</code>.
  

 <p>Before a release, run the following tests using a Java 1.3 VM 
 (the minimum required by the AspectJ tools):
 <ol>
  <li>JUnit tests.  Run these before the final build
  using <code>tests/junitModules.xml</code>
  (which runs all the 
  <code>modules/{module}/testsrc/{module}ModuleTests.java</code>).
  <p>
  </li>
  
  <li>All compiler tests in <code>tests/ajcTests.xml</code>
    and  <code>tests/ajcTestsFailing.xml</code>.
    Run these before the final build using the test harness,
    as follows:
 <pre>
  cd tests/
  java -jar {..}/testing-drivers-all.jar ajcTests.xml -release 
 </pre>        
    The <code>-release</code> option skips tests 
    marked <code>purejava</code> or <code>knownLimitation</code>
    and only emits one-line results for each test:
 <pre>
 PASS one test identifier()
 FAIL another test identifier()
 ...
 </pre>
    (Use <code>-help</code> on the harness to see the full 
    definition of the <code>-release</code> alias.)
    Run with and without the <code>-emacssym</code> option, 
    saving the output
    to the appropriate subdirectory of the
      <a href="../../releases">org.aspectj/releases</a> directory
    with the name 
       <code>ajcTests-result-{version}.txt</code> 
    or <code>ajcTests-result-{version}-emacssym.txt</code>.
    <p>
    Compare results with a prior release or test run using TestDiffs:
 <pre>
  set CLASSPATH="{..}/testing-drivers-all.jar"
  java org.aspectj.testing.util.TestDiffs {first-run}.txt {second-run}.txt
 </pre>        
   This will generate lists of tests fixed, broken, missing, or
   added.  For a release, no tests from <code>ajcTests.xml</code>
   should be broken, and missing tests (those in the first run absent from the
   second run) should be justified (usually the test was
   removed as invalid). A test will show up as
   missing from the first run and added to the second run if
   it is renamed between runs.
   For a final release, no test should be <code>FAIL</code> unless it is 
   deferred (including those from <code>ajcTestsFailing.xml</code>).
   <p>
    For more information, see the instructions for building and running
    the test harness and the tests
       <a href="../tests/readme-tests-module.html">
                ../tests/readme-tests-module.html</a>.
  <p>
  </li>  
  <li>Examples.  Run these on the built/installed release.
 The following should build and run the spacewar example:
 <pre>
    cd {aspectj-install}/doc/examples
    {ant 1.5.1} -f build.xml
 </pre>
 Also run 2-3 other targets from Ant and some examples directly:
 <ul>
 <li>
 The target <code>all</code>
 runs all examples, and <code>nonGui</code> runs those
 that do not require manual operation; 
 </li>

 <li>The target<code>tracing-bc</code> does bytecode weaving;
 </li>
 <li>
 Run some of the compiled examples in a 1.1 VM
    (the minimum required for running AspectJ code).
 </li>
 </ul>
 <p>
 This quick test verifies that the 
 <code>aspectjtools.jar</code> and <code>aspectjrt.jar</code>
 are installed and have a matching version, that the examples actually
 compile and run, etc.
   <p>
   </li> 
   <li>If there are any bugs which are closed with this release
   but which do not have an automated test case run using
   JUnit or the harness, those bugs should be manually tested 
   against the release.
   <p>
   </li> 
   <li>Any other tests warranted by release objectives.
   E.g., make sure it works as expected for any demos or 
   for any clients (e.g., Emacs, AJDT).
   </li> 
 </ol>
 <p>Some failures with past releases to consider checking:
 <ul>
  <li>debug info not included in .class files</li>
  <li>API not included in documentation</li>
  <li>Ant variables not filtered into doc files (e.g., ant-tasks.html)</li>
  <li>Manual dates not changed in docs or readme</li>
  <li>Inconsistent version information between runtime and tools classes</li>
 </ul>
   
 <p>Before a release, run the release tests as described in
  <a href="../tests/readme-release-tests.html">
           ../tests/readme-release-tests.html</a>.
                      
 <h4>Release completion</h4>
 When the release build is accepted,
 tag the tree with the release version
--- a/tests/readme-release-tests.html
+++ b/tests/readme-release-tests.html
@@ -0,0 +1,138 @@
 <html>
 <title>AspectJ release testing</title>
 <body>
 <h1>AspectJ release testing</h1>
 This describes tests to validate AspectJ before a release.

 Some may be omitted for pre-release distributions, but all
 should be run for final releases.
 For information on how to build and run tests, see
  <a href="../build/readme-build-and-test-aspectj.html">
           ../build/readme-build-and-test-aspectj.html</a>
 and
  <a href="readme-tests-module.html">
           readme-tests-module.html</a>.

 <p>(Most of these can and should be rolled into an Ant script
 for automation; until that script is written, this describes
 tests and could and/or should be run.)

 <p>Before a release, run the following tests using a Java 1.3 VM 
 (the minimum required by the AspectJ tools):
 <ol>
  <li>JUnit tests.  Run these before the final build
  using <code>tests/junitModules.xml</code>
  (which runs all the 
  <code>modules/{module}/testsrc/{module}ModuleTests.java</code>).
  <p>
  </li>
  
  <li>All compiler tests in <code>tests/ajcTests.xml</code>
    and  <code>tests/ajcTestsFailing.xml</code>.
    Run these before the final build using the test harness,
    as follows:
 <pre>
  cd tests/
  java -jar {..}/testing-drivers-all.jar ajcTests.xml -release 
 </pre>
    The <code>-release</code> option skips tests 
    marked <code>purejava</code> or <code>knownLimitation</code>
    and only emits one-line results for each test:
 <pre>
 PASS one test identifier()
 FAIL another test identifier()
 ...
 </pre>
    (Use <code>-help</code> on the harness to see the full 
    definition of the <code>-release</code> alias.)
    Save the output
    to the appropriate subdirectory of the
      <a href="../../releases">org.aspectj/releases</a> directory
    with the name 
       <code>ajcTests-result-{version}.txt</code>.
    or <code>ajcTests-result-{version}-emacssym.txt</code>.
    <p>
    Compare results with a prior release or test run using TestDiffs:
 <pre>
  set CLASSPATH="{..}/testing-drivers-all.jar"
  java org.aspectj.testing.util.TestDiffs {first-run}.txt {second-run}.txt
 </pre>        
   This will generate lists of tests fixed, broken, missing, or
   added.  For a release, no tests from <code>ajcTests.xml</code>
   should be broken, and missing tests (those in the first run absent from the
   second run) should be justified (usually the test was
   removed as invalid). A test will show up as
   missing from the first run and added to the second run if
   it is renamed between runs.
   For a final release, no test should be <code>FAIL</code> unless it is 
   deferred (including those from <code>ajcTestsFailing.xml</code>).
   <p>
   For (at least final) releases, run all relevant variants:
   <ul>
     <li>With and without <code>-emacssym</code></li>
     <li>1.3 and 1.4 VM's</li>
     <li>without -release and with purejava tests.  While 
         these may fail, they should not crash the compiler.</li>
   </ul>

  <li>Examples.  Run these on the built/installed release.
 The following should build and run the spacewar example:
 <pre>
    cd {aspectj-install}/doc/examples
    {ant 1.5.1} -f build.xml
 </pre>
 Also run 2-3 other targets from Ant and some examples directly:
 <ul>
 <li>
 The target <code>all</code>
 runs all examples, and <code>nonGui</code> runs those
 that do not require manual operation; 
 </li>

 <li>The target<code>tracing-bc</code> does bytecode weaving
 </li>
 <li>The target<code>tracing-adapter-ajc</code> compiles the
    tracing example using the <javac> task and the 
    Ajc compiler adapter.
 </li>
 <li>
 Run the compiled examples  in a 1.1 VM
    (the minimum required for running AspectJ code).
 </li>
 </ul>
 <p>
 This quick test verifies that the 
 <code>aspectjtools.jar</code> and <code>aspectjrt.jar</code>
 are installed and have a matching version, that the examples actually
 compile and run, etc.
   <p>
   </li> 
   <li>If there are any bugs which are closed with this release
   but which do not have an automated test case run using
   JUnit or the harness, those bugs should be manually tested 
   against the release.
   <p>
   </li> 
   <li>Any other tests warranted by release objectives.
   E.g., make sure it works as expected for any demos or 
   for any clients (e.g., Emacs, AJDT).
   </li> 
   <li>Compile large open-source Java projects using AspectJ.       
   </li>
   <li>Run the AspectJ compiler against the Jacks test suite.
   </li>
   <li>Manually step through the 
       <a href="ajde/script.html">ajde/script.html</a>
       on the AspectJ browser.
 </ol>
 <p>Some failures with past releases to consider checking:
 <ul>
  <li>debug info not included in .class files</li>
  <li>API not included in documentation</li>
  <li>Ant variables not filtered into doc files (e.g., ant-tasks.html)</li>
  <li>Manual dates not changed in docs or readme</li>
  <li>Inconsistent version information between runtime and tools classes</li>
 </ul>
 
 </body>
 </html>