spark: decimal arithmetic ignores the overflow=ERROR option

[alexandrefimov]

When the Spark consumer imports decimal add or multiply with the explicit overflow=ERROR option, overflow returns null if spark.sql.ansi.enabled=false. The behavior follows the Spark setting instead of the option carried by the Substrait invocation.

To reproduce

Reproduced at fff639064df794840db36fffcd881c09100e23df with Spark 3.5.4, Scala 2.12 and JDK 17. This example calls the expression importer directly:

import io.substrait.expression.Expression;
import io.substrait.expression.ExpressionCreator;
import io.substrait.expression.FunctionOption;
import io.substrait.extension.DefaultExtensionCatalog;
import io.substrait.spark.SparkExtension$;
import io.substrait.spark.expression.ToScalarFunction$;
import io.substrait.spark.expression.ToSparkExpression;
import io.substrait.type.TypeCreator;
import io.substrait.util.EmptyVisitationContext;
import java.math.BigDecimal;
import org.apache.spark.sql.internal.SQLConf;

public class SparkOverflowError {
  public static void main(String[] args) {
    SQLConf.get().setConfString("spark.sql.ansi.enabled", "false");
    var add = DefaultExtensionCatalog.DEFAULT_COLLECTION.scalarFunctions().stream()
        .filter(f -> f.key().equals("add:dec_dec")).findFirst().orElseThrow();
    var call = Expression.ScalarFunctionInvocation.builder()
        .declaration(add)
        .outputType(TypeCreator.REQUIRED.decimal(38, 0))
        .addOptions(FunctionOption.builder().name("overflow").addValues("ERROR").build())
        .addArguments(
            ExpressionCreator.decimal(false,
                new BigDecimal("99999999999999999999999999999999999999"), 38, 0),
            ExpressionCreator.decimal(false, BigDecimal.ONE, 1, 0))
        .build();
    var converter = new ToSparkExpression(
        ToScalarFunction$.MODULE$.apply(SparkExtension$.MODULE$.SparkScalarFunctions()),
        scala.Option.empty());
    var converted = call.accept(converter, EmptyVisitationContext.INSTANCE);
    System.out.println(converted.eval(null));
  }
}

Output:

null

Both inputs fit their declared decimal types. Their sum is 10^38, which does not fit decimal(38,0). I expected an arithmetic error, or rejection of the unsupported option during conversion. Changing the ANSI setting to true makes the same invocation throw SparkArithmeticException.

The example needs the core and Spark module runtime classpath. To use the corpus compilation helper, point SUBSTRAIT_JAVA_DIR at a checkout of the revision above and set the path to JDK 17:

export SUBSTRAIT_JAVA_DIR=/absolute/path/to/substrait-java
export JAVA17_HOME=/absolute/path/to/jdk17
export JAVA_HOME="$JAVA17_HOME"
git clone https://github.com/alexandrefimov/substrait-conformance-cases.git conformance-cases
git -C conformance-cases checkout b9751cfbce80ceff1c5aea0bd1fc373e3d157ce7
cd conformance-cases

Save the source above as probe/SparkOverflowError.java, then run:

bash probe/spark_run.sh SparkOverflowError

The same checkout also provides python3 probe/spark_function_options.py: eight evaluated cases covering add and multiply, both ANSI settings, and safe-value controls. Both overflowing ERROR cases return null with ANSI disabled; all four safe controls pass. These tests evaluate literal expressions, without starting a Spark session.

Expected behavior and conversion path

The v0.103.0 option contract allows consumer choice when an option is omitted, but an explicit option must be honored or the plan rejected. The decimal extension lists ERROR among the overflow choices.

The option is still present in the ScalarFunctionInvocation passed to the converter. ToSparkExpression.visit passes only the declaration key and arguments to getSparkExpressionFromSubstraitFunc; it does not pass the invocation options. The resulting Spark arithmetic expression uses the ANSI setting.

This report concerns Spark scalar-function option handling. The Isthmus option-loss report #1173 and the Spark decimal-division precision report #1292 cover different paths and behavior.

[alexandrefimov]

Rechecked the eight expression cases on Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with substrait-java fff639064df794840db36fffcd881c09100e23df and JDK 17. On every version, overflowing add and multiply with explicit overflow=ERROR still return null with ANSI disabled and raise an arithmetic error with ANSI enabled. All four safe-value controls pass.

Spark 4.2.0 also reproduces this on current Java main, ee9f3d20a6a21bc248ee4bc7addd72b440515106. Spark 4 enables ANSI by default, so both modes were set explicitly rather than relying on the runtime default.

These runs rebuilt unchanged consumer sources with the requested Spark dependencies: the 3.5 module for Spark 3.5.9, and the Scala 2.13 / 4.0 module for all three Spark 4 releases.