1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
|
<html>
<head><title>Harness Package Documentation</title></head>
<body>
<p>
The AspectJ compiler test harness can compile and run AspectJ programs
as specified by the test definitions.
This document tells you how to run the harness.
It describes the options you can specify on the command-line to
control various components that form the harness, either to
specify options that augment the test definitions or to
change how the harness works, e.g., selecting particular tests
or logging more information.
For information on how to write a test definition, see
<code>readme-writing-compiler-tests.html</code> in the testing module.
</p>
<p>
The harness drives compiler tests, using
a chain of responsibility to map elements
in the schema of a test definition to implementing classes.
</p>
<table border="1" cellpadding="1">
<tr><th align="left">Test feature</th>
<th align="left">Description</th>
<th align="left">Implementing class</th>
</tr>
<tr><td>(loading suites...)</td>
<td>general harness</td>
<td>Harness</td>
</tr>
<tr><td>(logging...)</td>
<td>subclass feature harness</td>
<td>FeatureHarness</td>
</tr>
<tr><td><code><suite></code></td>
<td>Test suite</td>
<td>AjcTest.Suite</td>
</tr>
<tr><td> <code><ajc-test></code></td>
<td>Test case</td>
<td>AjcTest</td>
</tr>
<tr><td> <code><compile></code></td>
<td>Initial (batch) compile run</td>
<td>CompilerRun</td>
</tr>
<tr><td> <code><inc-compile></code></td>
<td>Incremental re-compile</td>
<td>IncCompilerRun</td>
</tr>
<tr><td> <code><run></code></td>
<td>Run class</td>
<td>JavaRun</td>
</tr>
</table>
<!--
general harness (Harness)
subclass feature harness (FeatureHarness)
<ajc-test> run component (AjcTest)
<compile> {sub-} run component (CompilerRun)
<inc-compile> {sub-} run component (IncCompilerRun)
<run> {sub-} run component (JavaRun)
...
-->
<p/>
The compiler used is the AspectJ compiler <code>ajc</code>
(optionally as wrapped by the Ant task or AJDE API's), but
in principle any compiler accepting similar options can be
used.
<p/>
To run from the command-line, use
<code>Harness.main(String[])</code>.
To run programmatically, use <code>Harness.getHarness()</code>.
<code>Harness.runMain(String[])</code> takes arguments that
each component in the chain may accept and interpret, so
you can modify how the tests run by specifying the following
arguments on the harness command line:
<p/>
<table cellpadding="1" border="1">
<tr><th>Component</th><th>Options</th></tr>
<tr><td rowspan="6" valign="top">Harness
<p>suite files, harness verbosity, temp files, option variants
</p></td></tr>
<tr><td><u>suite files</u>: ajcTest-compliant .txt or .xml files are accepted.
<!-- XXX link to ajcTestSuite.dtd and .txt definitions -->
</td></tr>
<tr><td><u><code>-verboseHarness</code></u>,
<u><code>-quietHarness</code></u>:
Log accepted options and skipped tests,
or do not print even info messages.
</td></tr>
<tr><td><u><code>-keepTemp</code></u>: Normally the harness saves temp files until
the end of the run, and deletes them. If you abort the run or specify
<code>-keepTemp</code>, then temporary (sandbox) directories will remain for analysis.
In either case, the file system accumulates all temporary directories
and files used for a give harness run; for the <code>ajcTests.xml</code>
suite, this runs into thousands of files.
</td></tr>
<tr><td><u><code>-killTemp</code></u>: The opposite of <code>-keepTemp</code>,
this causes the harness to delete temporary (sandbox) directories at
the end of each test run.
In this case, the file system only accumulates files for
the current test.
</td></tr>
<tr><td><u>*- variants</u>: Options with a trailing "-" cause two sets of
option lists to be produced, one with and one without the corresponding
option. E.g., "-emacssym-" will run the suite twice, once with and
once without the "-emacssym" flag.
That means if you use this on each of three options, you will
get 8 variant sets (1 with no options, 1 with all 3 options,
3 with 2 options, and 3 with 1 option).
</td></tr>
<tr><td rowspan="5" valign="top">FeatureHarness
<p>output and logging options
</p></td></tr>
<tr><td><u>tracing</u>:
<code>-progressDots</code> will print "." for every passed
test completed and "!" for every test completed but not passed.
<code>-traceTests</code> will print a one-line summary for each test
of the time and space taken and whether the test passed.
<code>-traceTestsMin</code> will print only the test and whether it passed.
<code>-baseline</code> is an alias for
<code>-traceTestsMin</code>
<code>-hideStreams</code> and
<code>!eclipse</code>, used to emit tests results in a form
comparable by <code>org.aspectj.testing.util.TestDiffs</code>.
or usable to select tests by title for options like
<code>-ajctestTitleList</code>.
</td></tr>
<tr><td><u>output</u>: <code>-hide{Compiler|Run}Streams</code> will prevent output and
error streams from being printed to System.err and System.out,
optionally only for run or compile steps.
</td></tr>
<tr><td><u>logging</u>:
Log variants take the form <code>-log{Min|Xml}[Fail|Pass|All]</code>.
The suffix {All|Pass|Fail} selects all tests or only passing or failing tests.
The infix {Min} means to log with minimal information, typically only any
fail messages.
The infix {Xml} means to log the XML form of the test definition, so that
you can inspect the input or re-run arbitrary tests.
(You can also re-run a set of tests using keywords
(e.g., "<code>-ajctestsRequireKeywords=...</code>" or using titles
(e.g., "<code>-ajctestsTitleFailList=ajcTestResults.txt</code>".)
Finally, the experimental option <code>-XlogPublicType</code> will
log the XML test definition for
any test run that emits any ERROR messages containing the text "public type".
</td></tr>
<tr><td><u>interaction of output streams and logging</u>:
Streams will be emitted in real-time,
<em>before</em> the test is logged, unless streams are hidden.
When logging in normal (non-Min or -XML) form, the log will emit the streams
with the test report, so e.g., you can use -hideStreams -logFail to
hide streams for passing tests but emit them for failing tests
in the context of the log.
</td></tr>
<tr><td rowspan="5" valign="top">AjcTest
<p>selection options for keywords, bugID (PR), or title (description)
</p></td></tr>
<tr><td><u>keywords</u>: <code>-ajctest[Require|Skip]Keywords=one{,two}</code>
will either require or skip a test that has one of the
specified keywords.
</td></tr>
<tr><td><u>Bugs</u>: <code>-ajctestPR=101{,102}</code>
will require that a test have one of the listed bug id's.
</td></tr>
<tr><td><u>title</u>:
<code>"-ajctestTitleContains=one,two"</code>
will require that the title (description) of a test contain
one of the specified substrings, here either "one" or "two".
Use this to select a few tests you know generally.
<br/>
<code>"-ajctestTitleList=first title\, in theory, second title"</code>
will require that the title (description) of a test be
exactly "first title, in theory" or "second title".
The entire option must be one argument on the command line.
Use this when working with just a few specific tests.
<br/>
<code>"-ajctestTitleList=../tests/ajcTestResults.txt"</code>
will require that the title (description) of a test be
equal to one listed in <code>../tests/ajcTestResults.txt</code>
as a line of the form "[PASS|FAIL] {title}(.."
(This form is emitted by the <code>-traceTestsMin</code> option).
This option only differs from the prior in that the parameter
is a valid file to read.
Use this to re-run a large set of tests.
<br/>
<code>"-ajctestTitleFailList=../tests/ajcTestResults.txt"</code>
is the same as the <code>-ajctestTitleList={file}</code> variant,
except that only results prefixed "FAIL" are included.
Use this to re-run only the tests that failed from a large set.
</td></tr>
<tr><td><u>Combinations</u>: all selectors are applied to each test,
so all tests selected will comply with all constraints.
Specifying lists within a particular constraints will match
a union of tests for that constraint
(e.g., all tests with bug id's 101 or 102),
but there is no way to get a union of constraints
(e.g., all tests with bug id's 101 or 102 <em>or</em>
with keywords pure-java or knownLimitation).
However, <code>-ajctestSkipKeywords=...</code> can return all
tests without the specified keywords, so it can form unions like
"all tests without the knownLimitation keyword, but with
bug id's 101 or 102".
Title lists can work similarly. E.g., to run the failed
incremental tests from ajcTestResults.txt, specify
<code>-ajctestTitleFailList=../tests/ajcTestResults.txt</code>
<code>-ajctestRequireKeywords=incremental-test</code>.
</td></tr>
<tr><td rowspan="6" valign="top">CompilerRun
<p>compiler options and side-effects
</p></td></tr>
<tr><td><u>supported options</u>:
The command line passed to the compiler by <code>CompilerRun</code>
is composed of entries derived from the <code><compile></code>
attributes and recognized from the harness command line.
<code><compile></code> has specific attributes like
<code>files</code>,
<code>argfiles</code>,
<code>classpath</code> and
<code>sourceroot</code>
which translate directly to their counterparts.
The output option <code>-d</code> is defined by <code>CompilerRun</code> and
may not be specified (and <code>-outjar</code> is not supported).
Most other compiler options are defined in
<code>CompilerRun.Spec.CRSOptions</code> and may be specified
on the harness command-line
or in the <code>options</code> attribute of
<code><compile></code>.
In the <code>options</code> attribute, each argument is comma-delimited,
so an option with an argument would look like
<code><compile options="-source,1.4" ...></code>.
If options collide, duplicates
can be resolved using option dominance (below).
</td></tr>
<tr><td><u>compiler selectors</u>:
Use <code>-ajc</code> or <code>-eclipse</code> to select the old
(ajc 1.0) or new (eajc 1.1) compilers.
Note that the old compiler is not
available in the CVS source tree at eclipse.org.
Use <code>-ajdeCompiler</code> to run a wrapper around the
AJDE interface
and <code>-ajctaskCompiler</code> to run a wrapper around the
AjcTask (Ant task) interface.
</td></tr>
<tr><td><u>option dominance <code>[-|!|^]</code></u>:
Some tests require or prohibit certain options;
likewise, sometime you want to force all tests
run with or without an option specified on the command-line,
regardless of its setting in the <code><compile options=".." ...></code>
attribute.
For this reason an option may be specified in the options attribute
or on the harness command-line as
<code>-option</code>,
<code>!option</code>, or
<code>^option</code>.
<ul>
<li><u>- set</u>: If the leading character of an option is "-", then it is set unless forced-off.</li>
<li><u>^ force-off</u>: If the leading character of an option is "^", then it is forced off.
Any other matching option will be removed.</li>
<li><u>! force-on</u>: If the leading character of an option is "!", then it is forced on.
Any other non-force-on matching option will be removed.</li>
<li><u>force conflict</u>: If there are two matching force-on options, the test is skipped.</li>
<li><u>related options</u>: Two options match if they are the same or
if they are in the same family. For example, <code>-ajc</code> and
<code>eclipse</code> are both compiler, and <code>-source 1.4</code>
and <code>-source 1.3</code> are both source.
<br/>
</li>
</ul>
</td></tr>
<tr><td><u>auto-skip</u>: After combining global and local options, there may be
conflicting or impossible options, which cause the test to be skipped:
<ul>
<li><u>semantic conflicts</u>: two options may conflict in meaning
- e.g., <code>-lenient</code> and <code>-strict</code></li>
<li><u>impossible option</u>: It may not be possible in the current configuration to
implement an option - e.g., <code>-usejavac</code> or <code>-eclipse</code>
when javac or the eclipse implementation is not on the classpath
<br/></li>
</ul>
</td></tr>
<tr><td><u>source searching</u>: Given <code>-seek:{literal}</code>,
as a side-effect,
<code>CompilerRun</code> will search source files for {literal},
emitting for each instance an INFO message of the form:
<tt>found: {file}:{line}:{column}</tt>
(Note that the harness does not display INFO messages unless <tt>-verboseHarness</tt>
or <tt>-loud</tt> is used.)
</td></tr>
<tr><td rowspan="2" valign="top">JavaRun
<p>Options and forking</p></td>
<td><u>options</u>: options specified in the test are passed
to the main method as they would be on the command-line.
No options passed to the harness are passed through to
the main class.
</td></tr>
<tr><td><u>forking</u>:
Forking is useful to run in a different version of Java
than can be supported by the harness (i.e., some 1.1 flavor);
it's very time-consuming otherwise.
Currently forking is only controllable through system properties
of the invoking vm (defined in JavaRun.java):
<ul>
<li><u>javarun.fork</u>: anything to run in a new vm.
</li>
<li><u>javarun.java</u>: path to the java executable to run
(suffix included). If not supplied, the harness tries to
find the java that invoked the harness.
</li>
<li><u>javarun.java.home</u>: the value of the JAVA_HOME
environment variable, if it needs to be set.
</li>
<li><u>javarun.bootclasspath</u>: this is prepended to the
run classpath. Multiple entries must be separated by
the system-dependent path separator.
</li>
<li><u>javarun.vmargs</u>: this is added to the fork command-line
right after java. Multiple entries must be separated by a comma
(and the whole thing should be one parameter), e.g.,
<code>-Djavarun.vmargs=-Dname=value,-Dname2="value 2"</code>
</li>
</ul>
</td></tr>
</table>
<br/>
Following are some sample configurations:
<ul>
<li><code>java {harness} -hideStreams {suiteFile}</code>
<p>Use this to output only a 1-line summary of the test results
(tests skipped, incomplete, failed, passed).<br/></p>
</li>
<li><code>java {harness} -hideStreams -traceTestsMin {suiteFile} > results.txt</code>
<p>This writes to result.txt one line [PASS|FAIL] per test, plus a
1-line summary of the test results.<br/></p>
</li>
<li><code>java {harness} -logFail {suiteFile} -ajctestTitleFailList=results.txt</code>
<p>This re-runs any test that failed from the "results.txt" run,
verbosely logging any fails.<br/></p>
</li>
<li><code>java {harness} -hideStreams -logMinFail {suiteFile}</code>
<p>Use this when running tests mainly to see if they pass or
if the failure messages are typically enough information
to indicate why the test is failing. It produces only minimal
output for failed tests.<br/></p>
</li>
<li><code>java {harness} -hideStreams -verboseHarness -logFail {suiteFile}</code>
<p>When it's not clear at first glance why a test is failing, before
looking at the test code you can run it and print any harness or test
setup failures and all the associated messages from the test components.<br/></p>
</li>
<li><code>java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile}</code>
<p>Because of the trailing '-' on two of the options,
this would do four complete runs with the old (Ajc 1.0) compiler: one with
no options, one with -lenient, one with -Xlint, and one with both.<br/></p>
</li>
<li><code>java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile}</code>
<p>Run any tests associated with bugs 101 and 102, with and without -Xlint,
forcing off -usejavac and forcing the use of the new eclipse-based compiler.<br/></p>
</li>
</ul>
If you have a set of options you use often, you can define a single-word
option alias for it; see <code>Harness.optionAliases</code>.
<br/><u>Configuration</u>: Most tests use the library jars in
<code>modules/lib/test</code>, defined in
<code>org.aspectj.testing.harness.bridge.Globals</code>.
Normally the harness finds these by relative path
<code>../lib/tests/*.jar</code>, which works whenever the tests are
run from a peer module directory. When running tests elsewhere,
define the environment variable <code>harness.libdir</code> - e.g.,
<pre>
$ cd aspectj/tests
$ java -Dharness.libdir=../modules/lib/test ...
</pre>
<br/><u>Forking:</u>:
The harness must be run in a compiler-compatible VM, and the
compiler steps run in-process.
However, the java steps can be run in forked mode, which is useful
when compiling for a VM which can't run the compiler.
To compile for a different target VM could require
setting the options for bootclasspath, target, and source.
To run the harness so that any <run.. tasks run in a
separate vm, do something like this:
<pre>
java -Djavarun.java=d:\jdk1.1.8\bin\java.exe \
-Djavarun.bootclasspath=d:\jdk1.1.8\lib\classes.zip \
-Djavarun.java.home=d:\jdk1.1.8 \
-Djavarun.fork=true \
-jar ../aj-build/jars/testing-drivers-all.jar \
ajcTests.xml -logFail
</pre>
Here <code>CompilerRun</code> would add the bootclasspath as such when compiling.
JavaRun would fork using the 1.1 vm and prepend the bootclasspath
to the classpath, with an effect like these commands
(ignoring the line splitting in the classpath):
<pre>
set JAVA_HOME=d:\jdk1.1.8
d:\jdk1.1.8\bin\java.exe \
-classpath "d:\jdk1.1.8\lib\classes.zip;
d:\aspectj-src\lib\test\testing-client.jar;
d:\aspectj-src\lib\test\aspectjrt.jar;
c:\TEMP\sandbox7wers\classes"
{mainClass} {option..}
</pre>
</body>
</html>
|
