1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
|
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
<title>How to: Generate the Parser</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<table border=0 cellspacing=5 cellpadding=2 width="100%" >
<tr>
<td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
JDT Core / HowTo: Generate the Parser </font></b></td>
</tr>
<tr>
<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
<td WIDTH="98%"><b>Where to get the parser generator</b><br>
<blockquote>The parser files and resources are automatically generated using the LPG parser generator. This tools has
been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://www-124.ibm.com/developerworks/projects/jikes/">link</A>.
The latest tool is provided in source format. We don't provide any help for compiling these
source files. Refer to the link above if you have trouble to get binaries.
<br>Our grammar is generated using the version 2.30 of LPG. If newer versions fail to generate resources from our
grammar, please send request to the Jikes Parser Generator team.
</blockquote>
</p>
</td>
</tr>
<tr>
<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
<td WIDTH="98%"><b>Where to get the grammar</b><br>
<blockquote>The latest grammar is always located in the <code>grammar()</code> method of the <code>Parser</code> class. Go to the org.eclipse.jdt.core plugins
directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
<font color="#3366FF">jdtcoresrc.zip</font> file. Then search for the <font color="#3366FF">Parser.java</font> inside the
folder <font color="#3366FF">org\eclipse\jdt\internal\compiler\parser\</font>. You need to look for the method named
<code>grammar()</code>. Then copy its contents from:
<PRE>--main options
%options ACTION, AN=JavaAction.java, GP=java,
....
$end
-- need a carriage return after the $end
</PRE>
into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
</blockquote>
</p>
</td>
</tr>
<tr>
<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
<td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
<blockquote>
Assuming, the LPG executable (<code>lpg.exe</code> or <code>jikespg.exe</code>) is located inside <code>d:\lpg</code>.
<ol>
<li>First in a console, run:
<PRE>
d:
cd \temp
d:\lpg\lpg.exe java.g
</PRE>
</li>
<li>You will get an output that looks like this:
<PRE>
LPG Parser Generator (V2.30) Tue Apr 02 12:49:13 2002
%OPTIONS ACTION, AN=JavaAction.java, GP=java,
%OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
%OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE=TIME ,
%OPTIONS ERROR_MAPS
%OPTIONS first follow
%OPTIONS TRACE=FULL ,
%OPTIONS VERBOSE
Options in effect:
ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLIC
DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=OPTIMIZED NONT-CHECK ORMARK=|
OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE NOSCOPES NOSHIFT-DEFAULT
SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=TIME TRACE=FU
VERBOSE WARNINGS XREF
This grammar is LALR(1).
Number of Terminals: 105
Number of Nonterminals: 202
Number of Productions: 437
Number of Single Productions: 162
Number of Items: 1265
Number of States: 591
Number of Shift actions: 3482
Number of Goto actions: 4061
Number of Shift/Reduce actions: 369
Number of Goto/Reduce actions: 687
Number of Reduce actions: 7736
Number of Shift-Reduce conflicts: 0
Number of Reduce-Reduce conflicts: 0
Number of Reductions saved by default: 4913
Reallocating storage for TIME table, adding 3603 entries
Length of Check table: 16836
Length of Action table: 16608
Number of entries in Action Table: 12013
Percentage of increase: 38.2%
Highest symbol in Check Table: 307
Storage Required for Tables: 66888 Bytes, 66K
Storage Required for Rules: 1308 Bytes
Actions in Compressed Tables:
Number of Shifts: 3482
Number of Shift/Reduces: 369
Number of Gotos: 4061
Number of Goto/Reduces: 687
Number of Reduces: 2823
Number of Defaults: 390
Error maps storage:
Storage required for ACTION_SYMBOLS_BASE map: 1182 Bytes
Storage required for ACTION_SYMBOLS_RANGE map: 1007 Bytes
Storage required for NACTION_SYMBOLS_BASE map: 1182 Bytes
Storage required for NACTION_SYMBOLS_RANGE map: 630 Bytes
Storage required for SYMBOL_INDEX map: 616 Bytes
Storage required for STRING_BUFFER map: 4652 Bytes
***Warning: Base Check vector contains value > 127. 16-bit words used.
***Warning: Terminal symbol > 127. 16-bit words used.
Escaped symbol $eof is an invalid C variable.
Escaped symbol $error is an invalid C variable.
</PRE>
It can be quite different if the output changed since the version 2.30 of lpg. The important part is:<br>
<blockquote><b>This grammar is LALR(1).</b></blockquote>
This creates in the current directory some java source files and information files.
<blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
<tr>
<th VALIGN=top align=left>java.l</th>
<td VALIGN=top>Information generated by lpg/jikespg. Enumarate all the states created for the automaton, etc.</td>
</tr>
<tr>
<th VALIGN=top align=left>JavaAction.java</th>
<td>It contains the method consumeRule(int) of the class org.eclipse.jdt.internal.compiler.parser.Parser that handles all semantic actions dispatches.</td>
</tr>
<tr>
<th VALIGN=top align=left>javahdr.java</th>
<td>You don't need this file. It is actually empty.</td>
</tr>
<tr>
<th VALIGN=top align=left>javadcl.java</th>
<td>This files is used to generate the resources files.</td>
</tr>
<tr>
<th VALIGN=top align=left>javasym.java</th>
<td>This is the contents of the class org.eclipse.jdt.core.compiler.ITerminalSymbols. You need to replace:<br>
<ul>
<li>TokenName$eof with TokenNameEOF</li>
<li>TokenName$error with TokenNameERROR</li>
</ul></td>
</tr>
<tr>
<th VALIGN=top align=left>javadef.java</th>
<td>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
</tr>
<tr>
<th VALIGN=top align=left>javaprs.java</th>
<td>You don't need this file. Its contents is already inlined in the Parser class.</td>
</tr>
</table></blockquote>
</li>
<li><blockquote>Now we need to update the different classes and resource files.
</blockquote>
<ol>
<li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
</li>
<li>The definition of the Parser needs to be updated with two tables from <font color="#3366FF">javadcl.java</font>. Those are <font color="#3366FF">rhs[]</font> and <font color="#3366FF">name[]</font>.
The following entries in name[] need to be replaced:
<ul>
<li><font color="#3366FF">$eof</font> with <font color="#3366FF">UNEXPECTED_EOF</font></li>
<li><font color="#3366FF">$error</font> with <font color="#3366FF">"Invalid Character"</font></li>
</ul>
The previous definition of name[] will guide you.
</li>
<li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
<li>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.TerminalSymbols. You need to replace:<br>
<ul>
<li><font color="#3366FF">TokenName$eof</font> with <font color="#3366FF">TokenNameEOF</font></li>
<li><font color="#3366FF">TokenName$error</font> with <font color="#3366FF">TokenNameERROR</font></li>
</ul>
</li>
<li>The last step is to update the resource files:<br>
Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
Then run the following command-line:
<PRE>
D:\temp>java -classpath jdtcore.jar;. UpdateParserFiles javadcl.java
</PRE>
Once this done, you will end up with 5 new files inside d:\temp. They are called parser<n>.rsc, with n equals to 1..5.
All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
the new parser.
</li>
</ol>
</li>
</ol>
<P>
<b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
files are updated and recompiled before you run it. </blockquote>
</P>
</blockquote>
</td>
</tr>
</table>
</body>
</html>
|
