spark: decimal divide import changes the declared result precision

[alexandrefimov]

The Spark converter accepts a standard divide:dec_dec expression declared as decimal(21,8) and converts it to a Catalyst expression whose type is decimal(17,8).

Reproduced on substrait-java fff639064df794840db36fffcd881c09100e23df (0.103.0), Spark 3.5.4 and JDK 17. The operands are decimal(10,2) and decimal(5,1).

To reproduce

Save this as SparkDecimalDivide.java in a substrait-java checkout:

import io.substrait.expression.ExpressionCreator;
import io.substrait.extension.DefaultExtensionCatalog;
import io.substrait.spark.SparkExtension$;
import io.substrait.spark.expression.ToScalarFunction$;
import io.substrait.spark.expression.ToSparkExpression;
import io.substrait.type.TypeCreator;
import io.substrait.util.EmptyVisitationContext;
import java.math.BigDecimal;

public class SparkDecimalDivide {
  public static void main(String[] args) {
    var converter = new ToSparkExpression(
        ToScalarFunction$.MODULE$.apply(SparkExtension$.MODULE$.SparkScalarFunctions()),
        scala.Option.empty());
    var divide = DefaultExtensionCatalog.DEFAULT_COLLECTION.scalarFunctions().stream()
        .filter(f -> f.key().equals("divide:dec_dec")).findFirst().orElseThrow();
    var expression = ExpressionCreator.scalarFunction(
        divide, TypeCreator.NULLABLE.decimal(21, 8),
        ExpressionCreator.decimal(true, new BigDecimal("1.00"), 10, 2),
        ExpressionCreator.decimal(true, new BigDecimal("3.0"), 5, 1));
    var converted = expression.accept(converter, EmptyVisitationContext.INSTANCE);
    System.out.println(converted.dataType().simpleString());
  }
}

Use JDK 17 and the Spark module's runtime classpath. To obtain it, save this as spark-cp.gradle in the checkout:

gradle.afterProject { p ->
  if (p.path == ':spark:spark-3.5_2.12') {
    p.tasks.register('printReproClasspath') {
      dependsOn(p.sourceSets.main.runtimeClasspath)
      doLast { println 'REPRO_CP=' + p.sourceSets.main.runtimeClasspath.asPath }
    }
  }
}

Then run from the checkout root:

SPARK_CP="$(./gradlew -q -I spark-cp.gradle :spark:spark-3.5_2.12:printReproClasspath | sed -n 's/^REPRO_CP=//p')"
javac -cp "$SPARK_CP" SparkDecimalDivide.java
java -cp ".:$SPARK_CP" SparkDecimalDivide

The program prints decimal(17,8).

Expected behavior

The returned expression should preserve the standard function's declared type, or the converter should report that it cannot represent that contract.

The v0.103.0 decimal extension computes scale = max(6, 2 + 5 + 1) = 8 and precision = 10 - 2 + 5 + 8 = 21. Spark's division rule uses the second operand's scale in the precision calculation, giving 10 - 2 + 1 + 8 = 17. The converter maps divide directly to Catalyst Divide.

This report concerns the accepted expression's schema; it does not claim an incorrect numeric result. Aligning the standard extension formula, if desired, is a separate specification decision.

Optional: three minimal table-read plans reproduce the same result through ToLogicalPlan, with nullable operands. Addition and multiplication are passing controls at decimal(11,2) and decimal(16,3). Run them with python3 probe/structural_cases.py spark using the setup notes.

[alexandrefimov]

Rechecked the three table-read cases on Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 with substrait-java fff639064df794840db36fffcd881c09100e23df and JDK 17. With ANSI explicitly disabled, division still produces decimal(17,8) instead of the declared decimal(21,8) on every version. The addition and multiplication controls still produce decimal(11,2) and decimal(16,3).

Spark 4.2.0 also reproduces this on current Java main, ee9f3d20a6a21bc248ee4bc7addd72b440515106.

These runs rebuilt unchanged consumer sources with the requested Spark dependencies: the 3.5 module for Spark 3.5.9, and the Scala 2.13 / 4.0 module for all three Spark 4 releases. This confirms the schema mismatch on newer runtimes; it does not claim a numeric result error.