1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
|
<html>
<title>AspectJ tests</title>
<body>
<h1>AspectJ tests</h1>
<h3>Testing and Tests</h3>
Testing on the AspectJ project is fairly straightforward.
For compiler issues/bugs, we write harness tests.
For API issues/bugs, we write JUnit tests.
We also have other testware for system testing.
<p>
This module contains test data and scripts, while the testing-drivers
and testing modules contain the harness.
Tests differ in the kind of harness they use and their target.
(Many tests are unused and date back to earlier times in the project.)
<p>
<table border="1" cellpadding="1">
<tr><th align="left"><u>Test</u> Description</th>
<th align="left">Driver</th>
<th align="left">Sources</th>
</tr>
<tr><td><a href="#harness">Harness</a> tests
compile and run test sources.
</td>
<td>Harness in testing-drivers module</td>
<td><a href="ajcTests.xml">ajcTests.xml</a>,
and <a href="ajcTests.xml">ajcTests10.xml</a>,
referring to tests in
<a href="new">new/</a>,
<a href="base">base/</a>,
<a href="errors">errors/</a>,
<a href="design">design/</a>,
<a href="pureJava">pureJava/</a>,
etc.
</td>
</tr>
<tr><td><a href="#product">Product</a> tests use a variety of
Ant and Bash scripts to drive the product distributions.
<td>bash,
<a href="http://jakarta.apache.org/Ant">Ant</a>
</td>
<td><a href="product">product/..</a></td>
</tr>
<tr><td><a href="#ajctest">AjcTest</a> is a large taskdef which
compiles and runs large Java systems.
</td>
<td>AjcTest
<a href="http://jakarta.apache.org/Ant">Ant</a>
Task in testing module </td>
<td><a href="other-systems/">other-systems/..</a>,
<a href="XXX">aspectj-external-lib.jar?/other-systems/</a></td>
</tr>
<tr><td><a href="#jacks">Jacks</a> is the Jikes compiler test suite,
run for us using a bash script.
<td>bash</td>
<td>See the
<a href="http://www-124.ibm.com/developerworks/oss/cvs/jikes/~checkout~/jacks/jacks.html">
Jacks project page</a>
</td>
</tr>
<tr><td><a href="#junit">JUnit</a> tests normally are run directly
before checking in,
but there is an Ant script to run and report on them all.
</td>
<td><a href="junit.org">JUnit</a></td>
<td>Every module has a testsrc/ directory with JUnit tests.</td>
</tr>
</table>
<h3><a name="harness"></a>Harness</h3>
<h4><a name="harness-tests"></a>Harness tests</h4>
These are compiler test cases, the preferred form for
any compiler tests. The XML files are test definitions,
and the subdirectories contain the test sources (many of which
are unused at present). The test definitions are
segregated for convenience as follows:
<ul>
<li><a href="ajcTests.xml">ajcTests.xml</a>: main test suite.
After a release, these tests should always run and pass.
</li>
<li><a href="ajcTests10.xml">ajcTests10.xml</a>:
Tests for the 1.0 compiler. In some cases, error tests
differ between the 1.0 and 1.1 compiler only in the
errors expected to be generated.
</li>
<li><a href="ajcTestsBroken.xml">ajcTestsBroken.xml</a>:
These tests fail, and the tests are broken.
The tests should be fixed and rolled into ajcTests.xml
</li>
<li><a href="ajcTestsFailing.xml">ajcTestsFailing.xml</a>:
These tests fail, and the compiler is broken.
When fixed, these tests should be rolled into
<a href="ajcTests.xml">ajcTests.xml</a>.
</li>
<li><a href="ajcTestsAttic.xml">ajcTestsAttic.xml</a>:
These are broken or failing tests not slated to be fixed.
</li>
<li><a href="ajcHarnessTests.xml">ajcHarnessTests.xml</a>:
These are tests to exercise the test harness itself.
</li>
</ul>
The tests specify most anything one could want to do with
compile/run variants. The best summary of the test specification
format is the <a href="ajcTestSuite.dtd">ajcTestSuite.dtd</a>.
<p>
To write a new test, evolve an existing example. See the dtd and/or
the javadoc for the testing harness implementation in the
<code>testing</code> module for more details.
<p>To write a test case for a bug, see
<a href="readme-writing-compiler-tests.html">
readme-writing-compiler-tests.html</a>.
Many bugs that first present in an IDE or the Ant taskdef
are compiler bugs that can be written in this form.
We <em>much</em> prefer this automated form over
manual steps specified in a bug report.
<h4><a name="harness-runs"></a>Running the harness</h4>
<p>
To run the harness, use <code>org.aspectj.testing.drivers.Harness</code>,
in the <code>testing-drivers</code> module. Since that is the main class
for the <code>testing-drivers</code> module, you can build and
run the harness as follows:
<pre>
cd build/
../lib/ant/bin/ant build-testing-drivers
cd ../tests
java -jar ../aj-build/jars/testing-drivers-all.jar ajcTests.xml
</pre>
The harness enable you to
<ul>
<li>control output, hiding streams and logging messages</li>
<li>select tests using keywords, bug ID's, or titles</li>
<li>pass compiler arguments on the harness command line</li>
<li>use different compiler wrappers - the command-line compiler,
the AJDE IDE support (mostly sans GUI), or
the Ant taskdef support.</li>
<li>fork java run steps</li>
</ul>
For a listing of harness options, use <code>-help</code>.
For a fuller description of harness options and components, see
<a href="../testing-drivers/src/org/aspectj/testing/drivers/package.html">
the package documentation for the harness</a>.
<h3><a name="product"></a>Product</h3>
Product tests target a built distribution or do compiler (argument)
testing not supported by the harness.
Many have been superceded by ant tests.
To build aspectj with itself, see
<a href="product/build-aspectj/build.xml">
product/build-aspectj/build.xml</a>.
<h3><a name="ajctest"></a>AjcTest</h3>
<a href="ajctest.xml">ajctest.xml</a>
is an Ant script that compiles and runs large systems.
The scripts tell the AjcTest task from the testing module to drive
the sources in other-systems through compile/doc/run cycles.
To set up the tests, the script combines the aspects checked in to
other-systems/ with the Java source files from various open-source
projects, contained in aspectj-external-lib.zip.
<!-- XXX install and correct -->
<h3><a name="jacks"></a>Jacks</h3>
Jacks is the Jikes compiler test suite available at the
<a href="http://www-124.ibm.com/developerworks/oss/cvs/jikes/~checkout~/jacks/jacks.html">
Jacks project page</a>.
We have a bash script to run it using ajc and javac and compare
the results.
<h3><a name="junit"></a>JUnit</h3>
The JUnit tests should be self-documenting.
For more information on running them, see
<a href="../build/readme-build-and-test-aspectj.html">
../build/readme-build-and-test-aspectj.html</a>.
</body>
</html>
|
