From 31b7d84b2b4dc4ba29088947d8872027830e8fe4 Mon Sep 17 00:00:00 2001 From: wisberg Date: Mon, 16 Dec 2002 21:46:19 +0000 Subject: initial version --- .../src/org/aspectj/testing/drivers/package.html | 194 +++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 testing-drivers/src/org/aspectj/testing/drivers/package.html (limited to 'testing-drivers/src/org/aspectj/testing/drivers/package.html') diff --git a/testing-drivers/src/org/aspectj/testing/drivers/package.html b/testing-drivers/src/org/aspectj/testing/drivers/package.html new file mode 100644 index 000000000..8a098f798 --- /dev/null +++ b/testing-drivers/src/org/aspectj/testing/drivers/package.html @@ -0,0 +1,194 @@ + + +These harnesses drive ajc compiler tests. Following is information on how to use them. +

+ +Use Harness.main(String[]) from the command-line, +or Harness.getHarness() programmatically. +Harness.runMain(String[]) takes arguments that +each component in the chain may accept and interpret. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Component Options
Harness +
suite files, harness verbosity, temp files, option variants +
suite files: ajcTest-compliant .txt or .xml files are accepted. + +
-verboseHarness, -quietHarness: + Log accepted options and skipped tests, + or do not print even info messages. +
-keepTemp: Normally the harness saves temp files until + the end of the run, and deletes them. If you abort the run or specify + -keepTemp, then temporary (sandbox) directories will remain for analysis. +
*- variants: Options with a trailing "-" cause two sets of + option lists to be produced, one with and one without the corresponding + option. E.g., "-usejavac-" will run the suite twice, once with and + once without the "-usejavac" flag. + That means if you use this on each of three options, you will + get 8 variant sets (1 with no options, 1 with all 3 options, + 3 with 2 options, and 3 with 1 option). +
FeatureHarness +
output and logging options +
tracing: + -progressDots will print "." for every passed + test completed and "!" for every test completed but not passed. + -traceTests will print a one-line summary for each test + of the time and space taken and whether the test passed. + -traceTestsMin will print only the test and whether it passed. + -baseline is an alias for + -traceTestsMin + -hideStreams and + !eclipse, used to emit tests results in a form + comparable by org.aspectj.testing.util.TestDiffs. +
output: -hide{Compiler|Run}Streams will prevent output and + error streams from being printed to System.err and System.out, + optionally only for run or compile steps. +
logging: + Log variants take the form -log{Min|Xml}[Fail|Pass|All]. + The suffix {All|Pass|Fail} selects all tests or only passing or failing tests. + The infix {Min} means to log with minimal information, typically only any + fail messages. + The infix {Xml} means to log the XML form of the test definition, so that + you can inspect the input or re-run arbitrary tests. (For the latter, consider + also using keywords, under -ajctestsRequireKeywords=... below.) + Finally, the experimental option -XlogPublicType will + log the XML test definition for + any test run that emits any ERROR messages containing the text "public type". +
interaction of output streams and logging: + Streams will be emitted before the test is logged, unless streams are hidden. + When logging in normal (non-Min or -XML) form, the log will emit the streams + with the test report, so e.g., you can use -hideStreams -logFail to + hide streams for passing tests but emit them for failing tests. +
AjcTest +
selection options for keywords, bugID (PR) +
keywords: -ajctest[Require|Skip]Keywords=one{,two} + will either require or skip tests that have the specified keywords. +
Bugs: -ajctestPR=101{,102} + will run only tests that are associated with one of the bug id's listed. +
CompilerRun +
compiler options and side-effects +
supported options: Options given on the command-line have + the same meaning as the options in the test specification. + Only one-word options are supported; for this reason, -source 1.4 is + specified as -source14 and converted by CompilerRun back + to -source 1.4. Unsupported options include + -argfile, + -d, + -classpath, + -outjar, and + -sourceroot. +
compiler selectors: Use -ajc or -eclipse to select the old + (ajc 1.0) or new (eajc 1.1) compilers. +
option dominance -!^: + Some tests require or prohibit certain options; + likewise, sometime the person running the tests wants to require that all tests + run with or without an option specified on the command-line. CompilerRun supports encodings and + conflict resolution for these, so an option may be specified as + -option, + !option, or + ^option. +
+
- set: If the leading character of an option is "-", then it is set unless forced-off.
+
^ force-off: If the leading character of an option is "^", then it is forced off.
+
! force-on: If the leading character of an option is "!", then it is forced on.
+
force conflict: If local and global forces oppose, the test is skipped.
+
compiler conflicts: compiler selectors may be forced (e.g., + !ajc or ^eclipse), and conflicts are resolved + in the same way, so a local-global set conflict resolves in favor of the + global specification. (e.g., local -eclipse and global + -ajc will use the ajc compiler)
+
+
auto-skip: After collation of global and local options, there may be + conflicting or impossible options, which cause the test to be skipped: +
+
semantic conflicts: two options may conflict in meaning + - e.g., -lenient and -strict
+
impossible option: It may not be possible in the current configuration to + implement an option - e.g., -usejavac or -eclipse + when javac or the eclipse implementation is not on the classpath
+
+
source searching: Given -seek:{literal}, + as a side-effect, + CompilerRun will search source files for {literal}, + emitting for each instance an INFO message of the form: + found: {file}:{line}:{column} + (Note that the harness does not display INFO messages unless -verboseHarness + or -loud is used.) +
+

Component	Options
Harness + suite files, harness verbosity, temp files, option variants +
suite files: ajcTest-compliant .txt or .xml files are accepted. + +
`-verboseHarness`, `-quietHarness`: + Log accepted options and skipped tests, + or do not print even info messages. +
`-keepTemp`: Normally the harness saves temp files until + the end of the run, and deletes them. If you abort the run or specify + `-keepTemp`, then temporary (sandbox) directories will remain for analysis. +
*- variants: Options with a trailing "-" cause two sets of + option lists to be produced, one with and one without the corresponding + option. E.g., "-usejavac-" will run the suite twice, once with and + once without the "-usejavac" flag. + That means if you use this on each of three options, you will + get 8 variant sets (1 with no options, 1 with all 3 options, + 3 with 2 options, and 3 with 1 option). +
FeatureHarness + output and logging options +
tracing: + `-progressDots` will print "." for every passed + test completed and "!" for every test completed but not passed. + `-traceTests` will print a one-line summary for each test + of the time and space taken and whether the test passed. + `-traceTestsMin` will print only the test and whether it passed. + `-baseline` is an alias for + `-traceTestsMin` + `-hideStreams` and + `!eclipse`, used to emit tests results in a form + comparable by `org.aspectj.testing.util.TestDiffs`. +
output: `-hide{Compiler\|Run}Streams` will prevent output and + error streams from being printed to System.err and System.out, + optionally only for run or compile steps. +
logging: + Log variants take the form `-log{Min\|Xml}[Fail\|Pass\|All]`. + The suffix {All\|Pass\|Fail} selects all tests or only passing or failing tests. + The infix {Min} means to log with minimal information, typically only any + fail messages. + The infix {Xml} means to log the XML form of the test definition, so that + you can inspect the input or re-run arbitrary tests. (For the latter, consider + also using keywords, under `-ajctestsRequireKeywords=...` below.) + Finally, the experimental option `-XlogPublicType` will + log the XML test definition for + any test run that emits any ERROR messages containing the text "public type". +
interaction of output streams and logging: + Streams will be emitted before the test is logged, unless streams are hidden. + When logging in normal (non-Min or -XML) form, the log will emit the streams + with the test report, so e.g., you can use -hideStreams -logFail to + hide streams for passing tests but emit them for failing tests. +
AjcTest + selection options for keywords, bugID (PR) +
keywords: `-ajctest[Require\|Skip]Keywords=one{,two}` + will either require or skip tests that have the specified keywords. +
Bugs: `-ajctestPR=101{,102}` + will run only tests that are associated with one of the bug id's listed. +
CompilerRun + compiler options and side-effects +
supported options: Options given on the command-line have + the same meaning as the options in the test specification. + Only one-word options are supported; for this reason, `-source 1.4` is + specified as `-source14` and converted by CompilerRun back + to `-source 1.4`. Unsupported options include + `-argfile`, + `-d`, + `-classpath`, + `-outjar`, and + `-sourceroot`. +
compiler selectors: Use `-ajc` or `-eclipse` to select the old + (ajc 1.0) or new (eajc 1.1) compilers. +
option dominance `-!^`: + Some tests require or prohibit certain options; + likewise, sometime the person running the tests wants to require that all tests + run with or without an option specified on the command-line. CompilerRun supports encodings and + conflict resolution for these, so an option may be specified as + `-option`, + `!option`, or + `^option`. + + - set: If the leading character of an option is "-", then it is set unless forced-off. + ^ force-off: If the leading character of an option is "^", then it is forced off. + ! force-on: If the leading character of an option is "!", then it is forced on. + force conflict: If local and global forces oppose, the test is skipped. + compiler conflicts: compiler selectors may be forced (e.g., + `!ajc` or `^eclipse`), and conflicts are resolved + in the same way, so a local-global set conflict resolves in favor of the + global specification. (e.g., local `-eclipse` and global + `-ajc` will use the ajc compiler) + +
auto-skip: After collation of global and local options, there may be + conflicting or impossible options, which cause the test to be skipped: + + semantic conflicts: two options may conflict in meaning + - e.g., `-lenient` and `-strict` + impossible option: It may not be possible in the current configuration to + implement an option - e.g., `-usejavac` or `-eclipse` + when javac or the eclipse implementation is not on the classpath + +
source searching: Given `-seek:{literal}`, + as a side-effect, + CompilerRun will search source files for {literal}, + emitting for each instance an INFO message of the form: + `found: {file}:{line}:{column}` + (Note that the harness does not display INFO messages unless `-verboseHarness` + or `-loud` is used.) +

+Following are some sample configurations: +

java {harness} -hideStreams {suiteFile} +
Use this to output only a 1-line summary of the test results + (tests skipped, incomplete, failed, passed).
+
java {harness} -hideStreams -logMinFail {suiteFile} +
Use this when running tests mainly to see if they pass or + if the failure messages are typically enough information + to indicate why the test is failing. It produces only minimal + output for failed tests.
+
java {harness} -hideStreams -verboseHarness -logFail {suiteFile} +
When it's not clear at first glance why a test is failing, before + looking at the test code you can run it and print any harness or test + setup failures and all the associated messages from the test components.
+
java {harness} -hideStreams -usejavac- -ajc -Xlint- {suiteFile} +
This would do four complete runs with the old (Ajc 1.0) compiler: one with + no options, one with -lenient, one with -Xlint, and one with both.
+
java {harness} --ajctestPR=101,102 -Xlint- ^usejavac !eclipse {suiteFile} +
Run any tests associated with bugs 101 and 102, with and without -Xlint, + forcing off -usejavac and forcing the use of the new eclipse-based compiler.
+

+ +If you have a set of options you use often, you can define a single-word +option alias for it; see Harness.optionAliases. + +

Configuration: most tests use the library jars in +aspectj/modules/lib/test, defined in +org.aspectj.testing.harness.bridge.Globals. +Normally the harness finds these by relative path +../lib/tests/*.jar, which works whenever the tests are +run from a peer module directory. When running tests elsewhere, +define the environment variable harness.libdir - e.g., +

+    $ cd aspectj/tests
+    $ java -Dharness.libdir=../modules/lib/test -jar eajctesting.jar ajcTests.xml
+

+ + + \ No newline at end of file -- cgit v1.2.3