This extension is an interface to the basic calculator function for Arbitrary Precision Mathematics.

1 Overcoming Javascript numeric precision issues

Floating-point numbers are represented as binary (base 2) fractions. Regrettably, most decimal fractions cannot be represented exactly as binary fractions. The decimal floating-point numbers you enter are only approximated by the binary floating-point numbers actually stored in the machine. That being said, you'll see that floating-point arithmetic is NOT 100% accurate.

Copy
0.2 + 0.1
0.30000000000000004
 
0.3 - 0.1
0.19999999999999998
 
1111.11+1111.11+1111.11+1111.11+1111.11
5555.549999999999

To solve this problem, we need an alternative to Javascript numbers. That's why we have a BigDecimal Number Object and library, to address this issue. It provides arbitrary-precision decimal arithmetic.

1.1 Decimal data type

Primitive numeric types are useful for storing single values in memory. But when dealing with calculation using double and float types, there is a problems with the rounding. It happens because memory representation doesn't map exactly to the value.

To solve this issues you can use immutable, arbitrary-precision signed decimal numbers. A BigDecimal consists of an arbitrary precision integer unscaled value and a 32-bit integer scale. If zero or positive, the scale is the number of digits to the right of the decimal point. If negative, the unscaled value of the number is multiplied by ten to the power of the negation of the scale. The value of the number represented by the BigDecimal is therefore (unscaledValue × 10-scale). The BigDecimal class provides operations for arithmetic, scale manipulation, rounding, comparison, hashing, and format conversion. The toString() method provides a canonical representation of a BigDecimal.

1.2 Decimals created by application

In the following example we can see two implementations of the factorial function.

  • First uses the native JavaScript numbers. We can see data precision is lost whe number increases.
  • Second used multiplications using Ax.math.bc.mul function that takes two numnbers and returns a Decimal data type as result.

Copy
<script>
    function factorialDouble(num)
    {
        return (num === 0) ? 1 : num * factorialDouble( num - 1 )
    }
    
    function factorialDecimal(num)
    {
        return (num === 0) ? 1 : Ax.math.bc.mul(num, factorialDecimal( num - 1 ))
    }
    
    console.log(Ax.text.String.format("%3s %24s %s", "Num", "double", "Decimal"));
    for (var idx = 1; idx <= 50; idx++)
    	console.log(Ax.text.String.format("%3s %24s %s", idx, factorialDouble(idx), factorialDecimal(idx)));
</script>
Num                   double Decimal
  1                        1 1.00
  2                        2 2.000
  3                        6 6.0000
  4                       24 24.00000
  5                      120 120.000000
  6                      720 720.0000000
  7                     5040 5040.00000000
  8                    40320 40320.000000000
  9                   362880 362880.0000000000
 10                  3628800 3628800.00000000000
 11                 39916800 39916800.000000000000
 12                479001600 479001600.0000000000000
 13              6.2270208E9 6227020800.00000000000000
 14            8.71782912E10 87178291200.000000000000000
 15           1.307674368E12 1307674368000.0000000000000000
 16          2.0922789888E13 20922789888000.00000000000000000
 17         3.55687428096E14 355687428096000.000000000000000000
 18        6.402373705728E15 6402373705728000.0000000000000000000
 19      1.21645100408832E17 121645100408832000.00000000000000000000
 20      2.43290200817664E18 2432902008176640000.000000000000000000000
 21     5.109094217170944E19 51090942171709440000.0000000000000000000000
 22    1.1240007277776077E21 1124000727777607680000.00000000000000000000000
 23     2.585201673888498E22 25852016738884976640000.000000000000000000000000
 24     6.204484017332394E23 620448401733239439360000.0000000000000000000000000
 25    1.5511210043330986E25 15511210043330985984000000.00000000000000000000000000
 26    4.0329146112660565E26 403291461126605635584000000.000000000000000000000000000
 27    1.0888869450418352E28 10888869450418352160768000000.0000000000000000000000000000
 28    3.0488834461171384E29 304888344611713860501504000000.00000000000000000000000000000
 29     8.841761993739701E30 8841761993739701954543616000000.000000000000000000000000000000
 30    2.6525285981219103E32 265252859812191058636308480000000.0000000000000000000000000000000
 31     8.222838654177922E33 8222838654177922817725562880000000.00000000000000000000000000000000
 32     2.631308369336935E35 263130836933693530167218012160000000.000000000000000000000000000000000
 33     8.683317618811886E36 8683317618811886495518194401280000000.0000000000000000000000000000000000
 34    2.9523279903960412E38 295232799039604140847618609643520000000.00000000000000000000000000000000000
 35    1.0333147966386144E40 10333147966386144929666651337523200000000.000000000000000000000000000000000000
 36     3.719933267899012E41 371993326789901217467999448150835200000000.0000000000000000000000000000000000000
 37    1.3763753091226343E43 13763753091226345046315979581580902400000000.00000000000000000000000000000000000000
 38      5.23022617466601E44 523022617466601111760007224100074291200000000.000000000000000000000000000000000000000
 39    2.0397882081197442E46 20397882081197443358640281739902897356800000000.0000000000000000000000000000000000000000
 40     8.159152832478977E47 815915283247897734345611269596115894272000000000.00000000000000000000000000000000000000000
 41    3.3452526613163803E49 33452526613163807108170062053440751665152000000000.000000000000000000000000000000000000000000
 42    1.4050061177528798E51 1405006117752879898543142606244511569936384000000000.0000000000000000000000000000000000000000000
 43     6.041526306337383E52 60415263063373835637355132068513997507264512000000000.00000000000000000000000000000000000000000000
 44    2.6582715747884485E54 2658271574788448768043625811014615890319638528000000000.000000000000000000000000000000000000000000000
 45    1.1962222086548019E56 119622220865480194561963161495657715064383733760000000000.0000000000000000000000000000000000000000000000
 46    5.5026221598120885E57 5502622159812088949850305428800254892961651752960000000000.00000000000000000000000000000000000000000000000
 47    2.5862324151116818E59 258623241511168180642964355153611979969197632389120000000000.000000000000000000000000000000000000000000000000
 48    1.2413915592536073E61 12413915592536072670862289047373375038521486354677760000000000.0000000000000000000000000000000000000000000000000
 49     6.082818640342675E62 608281864034267560872252163321295376887552831379210240000000000.00000000000000000000000000000000000000000000000000
 50    3.0414093201713376E64 30414093201713378043612608166064768844377641568960512000000000000.000000000000000000000000000000000000000000000000000

2 Decimal returned from database

Databases have a special type called DECIMAL to perform abitrary precision calculations. When returning data of such type from database to application, data is mapped to class BigDecimal.

2.1 Getting count(*)

A typical operation that returns a BigDecimal in most databases is when selecting a COUNT(*).

  • executeGet, automatically detects for BigDecimal and converts to int/long
  • executeGetInt, returns an int
  • executeGetLong, returns a long

Notice that the previous functions prevents the need to use BigDecimal comparision to test for COUNT(*).

2.2 Dealing with bigdecimals

Copy
<script>
    var row = Ax.db.executeQuery(`
    SELECT CAST(0.02 AS DECIMAL) a, CAST(0.03 AS DECIMAL) b
      FROM systables 
     WHERE tabid = 1
     `).toOne();
    
    // A: using javascript double precision aritmetic
    console.log("Double precision");
    console.log(0.02 - 0.03);
    console.log(row.a - row.b);
    
    // B: using BigDecimal aritmetic
    console.log("BigDecimal precision");
    console.log(new Ax.sql.Decimal("0.02").subtract(new Ax.sql.Decimal("0.03")));
    console.log(row.a.subtract(row.b));
</script>
Double precision
-0.009999999999999998
-0.009999999999999998
BigDecimal precision
-0.01
-0.01

Notice that Decimal types returned from database must be operated using its own function (add, subtract, div, mul, etc). If not, they are converted to double before operating.

3 BC functions

The Ax.math.bc package contains a group of functions to perform BigDecimal operations.

Return Method Description
Creation
BigDecimal of(String number) Translates the string representation of a BigDecimal into a BigDecimal. The string representation consists of an optional sign, '+' ( '\u002B') or '-' ('\u002D'), followed by a sequence of zero or more decimal digits ("the integer"), optionally followed by a fraction, optionally followed by an exponent.
BigDecimal of(Number number) Translates a double into a BigDecimal which is the exact decimal representation of the double's binary floating-point value. The scale of the returned BigDecimal is the smallest value such that (10scale × val) is an integer.
add, sub, mul, div
BigDecimal add(Number n1, Number n2, ...) Returns a BigDecimal whose value is (this + augend), and whose scale is max(this.scale(), augend.scale()).
BigDecimal sub(Number n1, Number n2, ...) Returns a BigDecimal whose value is (this - subtrahend), and whose scale is max(this.scale(), subtrahend.scale()).
BigDecimal mul(Number n1, Number n2, ...) Returns a BigDecimal whose value is (this × multiplicand), and whose scale is (this.scale() + multiplicand.scale()).
BigDecimal div(Number n1, Number n2) Returns a BigDecimal whose value is (this / divisor), and whose preferred scale is (this.scale() - divisor.scale()); if the exact quotient cannot be represented (because it has a non-terminating decimal expansion) an ArithmeticException is thrown.
BigDecimal div(Number n1, Number n2, int scale) Returns a BigDecimal whose value is (this / divisor), and whose preferred scale is (this.scale() - divisor.scale()); if the exact quotient cannot be represented (because it has a non-terminating decimal expansion) an ArithmeticException is thrown.
BigDecimal div(Number n1, Number n2, int scale, String roundingMode) Returns a BigDecimal whose value is (this / divisor), and whose preferred scale is (this.scale() - divisor.scale()); if the exact quotient cannot be represented (because it has a non-terminating decimal expansion) an ArithmeticException is thrown.
abs, pow, round, scale
BigDecimal abs(Number n1) Returns a BigDecimal whose value is the absolute value of this BigDecimal, and whose scale is this.scale().
BigDecimal pow(Number n1, int n2) Returns a BigDecimal whose value is (thisn), The power is computed exactly, to unlimited precision.
BigDecimal round(Number n1, int precision) Returns a BigDecimal rounded according to the MathContext settings. If the precision setting is 0 then no rounding takes place
BigDecimal scale(Number n1, int scale, int roundingMode) Returns a BigDecimal whose scale is the specified value, and whose value is numerically equal to this BigDecimal's. Throws an ArithmeticException if this is not possible.

This call is typically used to increase the scale, in which case it is guaranteed that there exists a BigDecimal of the specified scale and the correct value. The call can also be used to reduce the scale if the caller knows that the BigDecimal has sufficiently many zeros at the end of its fractional part (i.e., factors of ten in its integer value) to allow for the rescaling without changing its value.

int getScale(Number n1) Returns the scale of this BigDecimal. If zero or positive, the scale is the number of digits to the right of the decimal point. If negative, the unscaled value of the number is multiplied by ten to the power of the negation of the scale. For example, a scale of -3 means the unscaled value is multiplied by 1000.
comparision
boolean isZero(Number n1) Returns a comparision between the number and BigDecimal.ZERO.
boolean equals(Number n1, Number n2) Compares this BigDecimal with the specified Object for equality. Unlike compareTo, this method considers two BigDecimal objects equal only if they are equal in value and scale (thus 2.0 is not equal to 2.00 when compared by this method).
int compareTo(Number n1, Number n2) Compares this BigDecimal with the specified BigDecimal. Two BigDecimal objects that are equal in value but have a different scale (like 2.0 and 2.00) are considered equal by this method. This method is provided in preference to individual methods for each of the six boolean comparison operators (<, ==, >, >=, !=, <=). The suggested idiom for performing these comparisons is: (x.compareTo(y) <op> 0), where <op> is one of the six comparison operators.
Copy
<script>

    console.log(Ax.math.bc.add(2, 2));
    console.log(Ax.math.bc.add(2, 2, 3));
    
    console.log(Ax.math.bc.mul(2, 2));
    console.log(Ax.math.bc.mul(2, 2, 3));

</script>
4
7
4
12

3.1 Comparing numbers

Be carefull when comparing BigDecimal to a javascript number. You can use

  • equals - takes scale into account
  • compareTo - does not take scale but returns 0, 1 or -1 accoding if number is equal, greater or lesser
  • isZero - compares if number is zero

equals
Expression Result
Ax.math.bc.equals(new Ax.math.BigDecimal(0), 0.00) false
Ax.math.bc.equals(new Ax.math.BigDecimal(0.0), 0.00) false
Ax.math.bc.equals(new Ax.math.BigDecimal(0.00), 0.00) false
Ax.math.bc.equals(new Ax.math.BigDecimal(0), 0) true
Ax.math.bc.equals(new Ax.math.BigDecimal(0.00), new Ax.math.BigDecimal(0.00)) true

Why the first cases fails ? You are passing 0.00 as right side element. This value, a javascript double, is passed to the equals function and it creates a BigDecimal using the double toString() representation that will generate it.

Unlike compareTo, equals considers two BigDecimal objects equal only if they are equal in value and scale (thus 2.0 is not equal to 2.00 when compared by this method).

compareTo
Expression Result
Ax.math.bc.compareTo(new Ax.math.BigDecimal(0), 0.00) 0
Ax.math.bc.compareTo(new Ax.math.BigDecimal(0.0), 0.00) 0
Ax.math.bc.compareTo(new Ax.math.BigDecimal(0.00), 0.00) 0
Ax.math.bc.compareTo(new Ax.math.BigDecimal(1), 0.00) 1
Ax.math.bc.compareTo(new Ax.math.BigDecimal(0), 1.00) -1
isZero
Expression Result
return Ax.math.bc.isZero(new Ax.math.BigDecimal(0)) true
return Ax.math.bc.isZero(new Ax.math.BigDecimal(0.00)) true

4 BigDecimal tutorial

4.1 Primer on Financial Issues

Currency calculations require precision to a specific degree, such as two digits after the decimal for most currencies. They also require a specific type of rounding behavior, such as always rounding up in the case of taxes.

4.1.1 Tax example

For example, suppose we have a product which costs 10.00 in a given currency and the local sales tax is 0.0825, or 8.25%. If we work it out on paper, the tax amount is,

Copy
10.00 * 0.0825 = 0.825

Because our precision for the currency is two digits after the decimal, we need to round the 0.825 figure. Also, because this is a tax, it is good practice to always round up to the next highest cent. That way when the accounts are balanced at the end of the day, we never find ourselves underpaying taxes.

Copy
0.825 -> 0.83

And so the total we charge to the customer is 10.83 in the local currency and pay 0.83 to the tax collector. Note that if we sold 1000 of these, we would have overpaid the collector by this much,

Copy
1000 * (0.83 - 0.825) = 5.00

4.1.2 Quantity example

Another important issue is where to do the rounding in a given computation. Suppose we sold Liquid Nitrogen at 0.528361 per liter. A customer comes in and buys 100.00 liters, so we write out the total price,

Copy
100.0 * 0.528361 = 52.8361

Because this isn't a tax, we can round this either up or down at our discretion. Suppose we round according to standard rounding rules: If the next significant digit is less than 5, then round down. Otherwise round up. This gives us a figure of 52.84 for the final price.

Now suppose we want to give a promotional discount of 5% off the entire purchase. Do we apply this discount on the 52.8361 figure or the 52.84 figure? What's the difference?

Copy
Calculation 1: 52.8361 * 0.95 = 50.194295 = 50.19
Calculation 2: 52.84 * 0.95 = 50.198 = 50.20

Note that we rounded the final figure by using the standard rounding rule.

See how there's a difference of one cent between the two figures? The old code never bothered to consider rounding, so it always did computations as in Calculation 1. But in the new code we always round before applying promotions, taxes, and so on, just like in Calculation 2. This is one of the main reasons for the one cent error.

4.2 Introducing BigDecimal

From the examples in the previous section, it should be clear that we need two things:

  • Ability to specify a scale, which represents the number of digits after the decimal place
  • Ability to specify a rounding method

4.2.1 Creating a BigDecimal

You can get a BigDecimal from a double, however it is a good idea to use the string constructor. Let's see why.

Copy
<script>
    console.log(Ax.math.bc.of("1.4"));
    console.log(Ax.math.bc.of(1.4));
</script>
1.4
1.399999999999999911182158029987476766109466552734375

So we need to be carefull when using Javascript numbers in our operations if they are related to the previous issues.

4.2.2 Rounding and Scaling

To set the number of digits after the decimal, use the setScale(scale) method. However, it is good practice to also specify the rounding mode along with the scale by using setScale(scale, roundingMode). The rounding mode specifies how to round the number.

Why do we also want to specify the rounding mode? Let's use the BD of 1.5 from above as an example,

Copy
<script>
    var bd = Ax.math.bc.of(1.4);
    bd.setScale(1); // This is an incorrect syntax of setScale()
</script>
java.lang.ArithmeticException: Rounding necessary

It throws the exception because it does not know how to round 1.399999 .... So it is a good idea to always use setScale(scale, roundingMode).


There are eight choices for rounding mode:

Copy
<script>
    console.log(Ax.math.bc.RoundingMode);
    console.log("------------------------------");

    var bd = Ax.math.bc.of(1.4);
    console.log("Number original =" + bd);
    console.log("Number scaled  4=" + bd.setScale(4, "HALF_UP"));
    console.log("Number scaled 18=" + bd.setScale(18, "HALF_UP"));
</script>
[DOWN, FLOOR, UNNECESSARY, CEILING, HALF_EVEN, UP, HALF_UP, HALF_DOWN]
------------------------------
Number original =1.399999999999999911182158029987476766109466552734375
Number scaled  4=1.4000
Number scaled 18=1.399999999999999912
Copy
<script>
    function round(bd, rounding)
    {
        var res = bd.setScale(2, rounding);
        console.log(Ax.text.String.format("%6s %6s %s", bd, res,rounding));
        return res;
    }
    
    for (var mode of Ax.math.bc.RoundingMode) {
        if (mode == Ax.math.bc.RoundingMode.UNNECESSARY)
            continue;
        round(Ax.math.bc.of("0.333"), mode);
        round(Ax.math.bc.of("-0.333"), mode);
    }
</script>
0.333   0.33 DOWN
-0.333  -0.33 DOWN
 0.333   0.33 FLOOR
-0.333  -0.34 FLOOR
 0.333   0.34 CEILING
-0.333  -0.33 CEILING
 0.333   0.33 HALF_EVEN
-0.333  -0.33 HALF_EVEN
 0.333   0.34 UP
-0.333  -0.34 UP
 0.333   0.33 HALF_UP
-0.333  -0.33 HALF_UP
 0.333   0.33 HALF_DOWN
-0.333  -0.33 HALF_DOWN

The following table shows results of scaling a value to tow deciml places on each mode

Mode Description 0.333 -0.333
CEILING Ceiling function 0.34 -0.33
UP .. 0.34 -0.34
DOWN Round towards zero 0.33 -0.33
FLOOR Floor function 0.33 -0.34
HALF_UP Round up if decimal >= .5 0.33 -0.33
HALF_DOWN Round up if decimal > .5 0.33 -0.33
HALF_EVEN Round half even will round as normal. However, when the rounding digit is 5, it will round down if the digit to the left of the 5 is even and up otherwise 0.33 -0.33
UNNECESSARY When you need to use one of the methods that requires input of a rounding mode but you know the result won't need to be rounded. RoundingNeccesary RoundingNeccesary

4.3 BigDecimal Immutability

BigDecimal numbers are immutable. What that means is that if you create a new BD with value "2.00", that object will remain "2.00" and can never be changed.

So how do we do math then?

The methods .add(), .multiply(), and so on all return a new BigDecimal instance with new value containing the result. For example, when you want to keep a running total of the order amount,

Copy
<script>
    var amount = new Ax.math.BigDecimal("2");
    var other  = new Ax.math.BigDecimal("3");
    amount = Ax.math.bc.add(amount,  other);
    console.log(amount);
</script>

4.4 Arithmetic and precission

Some considerations about precission should be taken when using BigDecimal arithmetic operations.

  • In divide operation (div), by default value returned has a minimum scale of 16 decimals. If operands has no scale defined (for example using strings or float native classes), a scale of 16 decimals is used to avoid exception in operations returning period decimal.
  • In multiply operation (mul) scales of both operads are added to get the result decimal precission.
Copy
<script>
    var div_floats = Ax.math.bc.div(1,  3);
    console.log(div_floats);
    var div_scaled = Ax.math.bc.div(new Ax.math.BigDecimal(1).setScale(4),  new Ax.math.BigDecimal(3).setScale(4));
    console.log(div_scaled);
    var mul_scaled = Ax.math.bc.mul("0",  new Ax.math.BigDecimal(2.15).setScale(8));
    console.log(mul_scaled);
</script>

You cannot know in advance the real precission of float numbers. That's why it's recommended to use always create bigdecimals from string representation of numbers.

Copy
<script>
    var d1 = Ax.math.bc.of(1.23);
    var d2 = Ax.math.bc.of(2.15);
    console.log("d1=" + d1);
    console.log("d2=" + d2);

    var mul_float = Ax.math.bc.mul(d1,  d2);
    console.log("result=" + mul_float);

    console.log("============================");

    var d3 = Ax.math.bc.of("1.23");
    var d4 = Ax.math.bc.of("2.15");
    console.log("d3=" + d3);
    console.log("d4=" + d4);

    var mul_scaled = Ax.math.bc.mul(d3,  d4);
    console.log("result=" + mul_scaled);

</script>
d1=1.229999999999999982236431605997495353221893310546875
d2=2.149999999999999911182158029987476766109466552734375
result=2.644499999999999852562382329779213009463524919562673323457130565572459346412870218046009540557861328125
============================
d3=1.23
d4=2.15
result=2.6445

5 Nashor vs Graal script engines

Nashorn and Graal have some differences when treating BigDecimal comparisons.

  • Nashorn, succesfully compares BigDecimal to int values
  • Graal, fails to compare BigDecimal to int values
Operation Nashorn Graal
BigDecimal(0) == 0 true false
BigDecimal(1) == 1 true false

It's better to use always BigDecimal comparison functions to ensure application runs in both engines.