Databricks: support CREATE TABLE USING, MAP<K, V> columns and LONG as BIGINT - #2425
Open
shuvamk wants to merge 1 commit into
Open
Databricks: support CREATE TABLE USING, MAP<K, V> columns and LONG as BIGINT#2425shuvamk wants to merge 1 commit into
shuvamk wants to merge 1 commit into
Conversation
… BIGINT
Databricks SQL is built on Spark SQL, but DatabricksDialect never got
three of the flags SparkSqlDialect sets, so these fail on Databricks and
parse on Spark:
CREATE TABLE t (id BIGINT) USING DELTA
CREATE TABLE t (m MAP<STRING, INT>)
with "Expected: end of statement, found: USING at Line: 1, Column: 28"
and "Expected: ',' or ')' after column definition, found: < at Line: 1,
Column: 22".
CREATE TABLE t (id LONG) parses, but builds DataType::Custom("LONG")
instead of DataType::BigInt(None). Databricks documents the type as
{ BIGINT | LONG }.
Set supports_create_table_using, supports_long_type_as_bigint and
supports_map_literal_with_angle_brackets on DatabricksDialect, mirroring
src/dialect/spark.rs. No parser change and no other dialect is affected.
Regression tests in tests/sqlparser_databricks.rs; all three fail with
src/dialect/databricks.rs reverted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LucaCappelletti94
suggested changes
Aug 5, 2026
|
|
||
| databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET"); | ||
|
|
||
| assert!(all_dialects_where(|d| !d.supports_create_table_using()) |
Contributor
There was a problem hiding this comment.
Consider adding the analogous assert also for the other flags you have set
|
|
||
| databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET"); | ||
|
|
||
| assert!(all_dialects_where(|d| !d.supports_create_table_using()) |
Contributor
There was a problem hiding this comment.
Also, I believe that since this assertion regards all dialects except databricks, it should not be in the databricks tests, but in common.
| fn supports_long_type_as_bigint(&self) -> bool { | ||
| true | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
While you are adding support for Spark flags in Databricks, I believe you should also add supports_pipe_operator and parse_infix for DIV.
| #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] | ||
| pub struct DatabricksDialect; | ||
|
|
||
| impl Dialect for DatabricksDialect { |
Contributor
There was a problem hiding this comment.
I believe there is some argument to be made that, since several of these properties need to be kept aligned between Spark and Databricks, it may be desirable to call directly Spark methods in these methods instead of duplicating the scalar value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Databricks SQL is built on Spark SQL, but
DatabricksDialectnever got three of the flagsSparkSqlDialectsets. These fail onDatabricksDialectand parse onSparkSqlDialect:with
Expected: end of statement, found: USING at Line: 1, Column: 28andExpected: ',' or ')' after column definition, found: < at Line: 1, Column: 22.CREATE TABLE t (id LONG)parses, but builds aDataType::CustomholdingLONGinstead ofDataType::BigInt(None). Databricks gives the type as{ BIGINT | LONG }— https://docs.databricks.com/aws/en/sql/language-manual/data-types/bigint-typeThe fix sets those three flags on
DatabricksDialect, mirroringsrc/dialect/spark.rs. No parser change, no other dialect affected.Tests are in
tests/sqlparser_databricks.rsand all three fail withsrc/dialect/databricks.rsreverted. TheAGENTS.mdpre-commit checks are clean locally.