Developing Formula Evaluation

diff --git a/src/documentation/content/xdocs/hssf/eval-devguide.xml b/src/documentation/content/xdocs/hssf/eval-devguide.xml new file mode 100644 index 0000000000..994ca7a54a --- /dev/null +++ b/src/documentation/content/xdocs/hssf/eval-devguide.xml @@ -0,0 +1,184 @@ + + + + + +

+ Developing Formula Evaluation + + + +

+ +

Introduction +

This document is for developers wishing to contribute to the + FormulaEvaluator API functionality.

Currently, contribution is desired for implementing the standard MS + excel functions. Place holder classes for these have been created, + contributors only need to insert implementation for the + individual "evaluate()" methods that do the actual evaluation.

Overview of FormulaEvaluator +

Briefly, a formula string (along with the sheet and workbook that + form the context in which the formula is evaluated) is first parsed + into RPN tokens using the FormulaParser class in POI-HSSF main. + (If you dont know what RPN tokens are, now is a good time to + read + this.) +

The big picture +

RPN tokens are mapped to Eval classes. (Class hierarchy for the Evals + is best understood if you view the class diagram in a class diagram + viewer.) Depending on the type of RPN token (also called as Ptgs + henceforth since that is what the FormulaParser calls the classes) a + specific type of Eval wrapper is constructed to wrap the RPN token and + is pushed on the stack.... UNLESS the Ptg is an OperationPtg. If it is an + OperationPtg, an OperationEval instance is created for the specific + type of OperationPtg. And depending on how many operands it takes, + that many Evals are popped of the stack and passed in an array to + the OperationEval instance's evaluate method which returns an Eval + of subtype ValueEval.Thus an operation in the formula is evaluated.

+ An Eval is of subinterface ValueEval or OperationEval. + Operands are always ValueEvals, Operations are always OperationEvals. +

OperationEval.evaluate(Eval[]) returns an Eval which is supposed + to be of type ValueEval (actually since ValueEval is an interface, + the return value is instance of one of the implementations of + ValueEval). The valueEval resulting from evaluate() is pushed on the + stack and the next RPN token is evaluated.... this continues till + eventually there are no more RPN tokens at which point, if the formula + string was correctly parsed, there should be just one Eval on the + stack - which contains the result of evaluating the formula.

Ofcourse I glossed over the details of how AreaPtg and ReferencePtg + are handled a little differently, but the code should be self + explanatory for that. Very briefly, the cells included in AreaPtg and + RefPtg are examined and their values are populated in individual + ValueEval objects which are set into the AreaEval and RefEval (ok, + since AreaEval and RefEval are interfaces, the implementations of + AreaEval and RefEval - but you'll figure all that out from the code)

OperationEvals for the standard operators have been implemented and + basic testing has been done

FunctionEval and FuncVarEval +

FunctionEval is an abstract super class of FuncVarEval. The reason for this is that in the FormulaParser Ptg classes, there are two Ptgs, FuncPtg and FuncVarPtg. In my tests, I did not see FuncPtg being used so there is no corresponding FuncEval right now. But in case the need arises for a FuncVal class, FuncEval and FuncVarEval need to be isolated with a common interface/abstract class, hence FunctionEval.

FunctionEval also contains the mapping of which function class maps to which function index. This mapping has been done for all the functions, so all you really have to do is implement the evaluate method in the function class that has not already been implemented. The Function indexes are defined in AbstractFunctionPtg class in POI main.

Walkthrough of an "evaluate()" implementation. +

So here is the fun part - lets walk through the implementation of the excel + function... AVERAGE()

The Code + +public Eval evaluate(Eval[] operands) { + double d = 0; + int count = 0; + ValueEval retval = null; + for (int i = 0, iSize = operands.length; i < iSize; i++) { + if (operands[i] == null) continue; + if (operands[i] instanceof AreaEval) { + AreaEval ap = (AreaEval) operands[i]; + Object[] values = ap.getValues(); + for (int j = 0, jSize = values.length; j < jSize; j++) { + if (values[j] == null) continue; + if (values[j] instanceof NumberEval) { + //inside areas, ignore bools + d += ((NumberEval) values[j]).getNumberValue(); + count++; + } + else if (values[j] instanceof RefEval) { + RefEval re = (RefEval) values[j]; + ValueEval ve = re.getInnerValueEval(); + if (ve != null && ve instanceof NumberEval) { + d += ((NumberEval) ve).getNumberValue(); + count++; + } + } + } + } + else if (operands[i] instanceof NumericValueEval) { + // for direct operands evaluate bools + NumericValueEval np = (NumericValueEval) operands[i]; + d += np.getNumberValue(); + count++; + } + else if (operands[i] instanceof RefEval) { + RefEval re = (RefEval) operands[i]; + ValueEval ve = re.getInnerValueEval(); + if (ve instanceof NumberEval) { + //if it is a reference, ignore bools + NumberEval ne = (NumberEval) ve; + d += ne.getNumberValue(); + count++; + } + } + } + + if (retval == null) { + retval = (Double.isNaN(d)) ? + (ValueEval) ErrorEval.ERROR_503 : new NumberEval(d/count); + } + return retval; +} + +

Implementation Details +

The implementation of the AVERAGE function lives in package + o.a.p.hssf.record.formula.functions named Average.java. + (Every excel function has a corresponding java source file + in the above package)
If you open the file for a function thats not yet implemented, you will see one un-implemented method: + public Eval evaluate(Eval[] operands) {}
Since the excel Average() function can take 1 or more operands, we iterate over all operands that are passed in the evaluate method: + for (int i=0, iSize=operands.length; i<iSize; i++) {...}
inside the loop, you will do the following +
1. Do a null check: if (operands[i] == null) continue;
2. Figure out the actual subtype of ValueEval that the operands + implements. The possible types that you will encounter in an + evaluate() are: NumberEval, BoolEval, StringEval, ErrorEval, + AreaEval, RefEval, BlankEval.
3. Implement the function. See the next section for some + caveats on implementing the Excel semantics.
+
Finally before returning the NumberEval wrapping the double value that + you computed, do one final check to see if the double is a NaN, + if it is return ErrorEval.ERROR_503 (see the javadoc in ErrorEval.java + for description of error codes - it is html so you might as well + generate the javadocs)

Modelling Excel Semantics +

Strings are ignored. Booleans are ignored!!! (damn Oo.o! I was almost misled here - nevermind). Actually here's the info on Bools: + if you have formula: "=TRUE+1", it evaluates to 2. + So also, when you use TRUE like this: "=SUM(1,TRUE)", you see the result is: 2. + So TRUE means 1 when doing numeric calculations, right? + Wrong! + Because when you use TRUE in referenced cells with arithmetic functions, it evaluates to blank - meaning it is not evaluated - as if it was string or a blank cell. + eg. "=SUM(1,A1)" when A1 is TRUE evaluates to 1. + So you have to do this kind of check for every possible data type as a function argument for any function before you understand the behaviour of the function. The operands can be entered in excel as comma separated or as a region specified like: A2:D4. Regions are treated as a single token by the parser hence we have AreaEval which stores the ValueEval at each cell in a region in a 1D array. So in our function if the operand is of type AreaEval we need to get the array of ValueEvals in the region of the AreaEval and iterate over each of them as if each of them were individual operands to the AVERAGE function. +

Thus, since sometimes, Excel treats + Booleans as the numbers 0 and 1 (for F and T respectively). + Hence BoolEval and NumberEval both implement a common interface: + NumericValueEval (since numbers and bools are also valid string + values, they also implement StringValueEval interface which is + also implemented by StringEval).

+ The ValueEval inside an AreaEval can be one of: + NumberEval, BoolEval, StringEval, ErrorEval, BlankEval. + So you must handle each of these cases. + Similarly, RefEvals have a property: innerValueEval that returns the ValueEval at the referenced cell. The ValueEval inside a RefEval can be one of: NumberEval, BoolEval, StringEval, ErrorEval, BlankEval. So you must handle each of these cases - see how excel treats each one of them. +

+ +

Testing Framework + TODO! FormulaEval comes with a testing framework, where you add + formula's and their expected values to an Excel sheet, and the test code + automatically validates them. Since this is still in flux, the docs + will be put online once the system is stable +

+ + \ No newline at end of file diff --git a/src/documentation/content/xdocs/hssf/eval.xml b/src/documentation/content/xdocs/hssf/eval.xml new file mode 100644 index 0000000000..acf88b421a --- /dev/null +++ b/src/documentation/content/xdocs/hssf/eval.xml @@ -0,0 +1,151 @@ + + + + + +

+ Formula Evaluation + + + +

+ +

Introduction +

The POI formula evaluation code enables you to calculate the result of + formulas in Excels sheets read-in, or created in POI. This document explains + how to use the API to evaluate your formulas. +

+ This code currently lives in Bugzilla as + + bug 34828 . It is expected to land in POI CVS in the scratchpad + area soon. + +

Status +

The code currently provides implementations for all the arithmatic operators. + It also provides implementations for about 30 built in + functions in Excel. The framework however makes is easy to add + implementation of new functions. See the Formula + evaluation development guide for details.

Note that user-defined functions are not supported, and is not likely to done + any time soon... at least, not till there is a VB implementation in Java! +

User API How-TO +

The following code demonstrates how to use the HSSFFormulaEvaluator + in the context of other POI excel reading code. +

There are two ways in which you can use the HSSFFormulaEvalutator API.

Using HSSFFormulaEvaluator.<strong>evaluate</strong>(HSSFCell cell) + +FileInputStream fis = new FileInputStream("c:/temp/test.xls"); +HSSFWorkbook wb = new HSSFWorkbook(fis); +HSSFSheet sheet = wb.getSheetAt(0); +HSSFFormulaEvaluator evaluator = new HSSFFormulaEvaluator(sheet, wb); + +// suppose your formula is in B3 +CellReference cellReference = new CellReference("B3"); +HSSFRow row = sheet.getRow(cellReference.getRow()); +HSSFCell cell = row.getCell(cellReference.getCol()); +String formulaString = c.getCellFormula(); +HSSFFormulaEvaluator.CellValue cellValue = + evaluator.evaluate(formulaString); + +switch (cellValue.getCellType()) { + case HSSFCell.CELL_TYPE_BOOLEAN: + System.out.println(cellValue.getBooleanCellValue()); + break; + case HSSFCell.CELL_TYPE_NUMERIC: + System.out.println(cellValue.getNumberCellValue()); + break; + case HSSFCell.CELL_TYPE_STRING: + System.out.println(cellValue.getStringCellValue()); + break; + case HSSFCell.CELL_TYPE_BLANK: + break; + case HSSFCell.CELL_TYPE_ERROR: + break; + + // CELL_TYPE_FORMULA will never happen + case HSSFCell.CELL_TYPE_FORMULA: + break; +} + +

Thus using the retrieved value (of type + HSSFFormulaEvaluator.CellValue - a nested class) returned + by HSSFFormulaEvaluator is similar to using a HSSFCell object + containing the value of the formula evaluation. CellValue is + a simple value object and does not maintain reference + to the original cell. +

+ +

Using HSSFFormulaEvaluator.<strong>evaluateInCell</strong>(HSSFCell cell) + + +FileInputStream fis = new FileInputStream("c:/temp/test.xls"); +HSSFWorkbook wb = new HSSFWorkbook(fis); +HSSFSheet sheet = wb.getSheetAt(0); +HSSFFormulaEvaluator evaluator = new HSSFFormulaEvaluator(sheet, wb); + +// suppose your formula is in B3 +CellReference cellReference = new CellReference("B3"); +HSSFRow row = sheet.getRow(cellReference.getRow()); +HSSFCell cell = row.getCell(cellReference.getCol()); +String formulaString = c.getCellFormula(); + +if (cell!=null) { + switch (evaluator.evaluateInCell(cell).getCellType()) { + case HSSFCell.CELL_TYPE_BOOLEAN: + System.out.println(cell.getBooleanCellValue()); + break; + case HSSFCell.CELL_TYPE_NUMERIC: + System.out.println(cell.getNumberCellValue()); + break; + case HSSFCell.CELL_TYPE_STRING: + System.out.println(cell.getStringCellValue()); + break; + case HSSFCell.CELL_TYPE_BLANK: + break; + case HSSFCell.CELL_TYPE_ERROR: + System.out.println(cell.getErrorCellValue()); + break; + + // CELL_TYPE_FORMULA will never occur + case HSSFCell.CELL_TYPE_FORMULA: + break; + } +} + + +

+ +

Performance Notes +

Generally you should have to create only one HSSFFormulaEvaluator + instance per sheet, but there really is no overhead in creating + multiple HSSFFormulaEvaluators per sheet other than that of the + HSSFFormulaEvaluator object creation. +
Also note that HSSFFormulaEvaluator maintains a reference to + the sheet and workbook, so ensure that the evaluator instance + is available for garbage collection when you are done with it + (in other words don't maintain long lived reference to + HSSFFormulaEvaluator if you don't really need to - unless + all references to the sheet and workbook are removed, these + don't get garbage collected and continue to occupy potentially + large amounts of memory). +
CellValue instances however do not maintain reference to the + HSSFCell or the sheet or workbook, so these can be long-lived + objects without any adverse effect on performance. +

+ +