This repository was archived by the owner on Feb 27, 2025. It is now read-only.
Description While upgrading the pyspark version of sparkpools to 3.4 in Synapse Studio. We faced this error
An error occurred while calling o6063.save.\n: java.lang.NoSuchMethodError: 'org.apache.spark.sql.types.StructType org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(java.sql.ResultSet, org.apache.spark.sql.jdbc.JdbcDialect, boolean)'\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)\n\tat com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)\n\tat com.microsoft.sqlserver.jdbc.spark.DefaultSource.createRelation(DefaultSource.scala:66)\n\tat
We have updgraded the pyspark(3.4.0) and delta-spark(2.4.0) version in requirement.txt file also but issue is same.
Reactions are currently unavailable
While upgrading the pyspark version of sparkpools to 3.4 in Synapse Studio. We faced this error
An error occurred while calling o6063.save.\n: java.lang.NoSuchMethodError: 'org.apache.spark.sql.types.StructType org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(java.sql.ResultSet, org.apache.spark.sql.jdbc.JdbcDialect, boolean)'\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)\n\tat com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)\n\tat com.microsoft.sqlserver.jdbc.spark.DefaultSource.createRelation(DefaultSource.scala:66)\n\tatWe have updgraded the pyspark(3.4.0) and delta-spark(2.4.0) version in requirement.txt file also but issue is same.