Skip to content

fix(spark): spark sql union recursion#466

Open
ChaconneLuo wants to merge 1 commit intoDTStack:mainfrom
ChaconneLuo:bugfix/spark_sql_union_recursion
Open

fix(spark): spark sql union recursion#466
ChaconneLuo wants to merge 1 commit intoDTStack:mainfrom
ChaconneLuo:bugfix/spark_sql_union_recursion

Conversation

@ChaconneLuo
Copy link

@ChaconneLuo ChaconneLuo commented Mar 18, 2026

现象：在spark sql中，如果在with中遇到大量union all，queryTerm中会由于包含存在重复的规则，预测路径出现指数级增长，导致validate卡死。

复现SQL：

WITH t1 AS (
    SELECT 1
    UNION ALL SELECT 2
    UNION ALL SELECT 3
    UNION ALL SELECT 4
    UNION ALL SELECT 5
    UNION ALL SELECT 6
    UNION ALL SELECT 7
    UNION ALL SELECT 8
    UNION ALL SELECT 9
    UNION ALL SELECT 10
    UNION ALL SELECT 11
    UNION ALL SELECT 12
    UNION ALL SELECT 13
    UNION ALL SELECT 14
    UNION ALL SELECT 15
    UNION ALL SELECT 16
    UNION ALL SELECT 17
    UNION ALL SELECT 18
    UNION ALL SELECT 19
    UNION ALL SELECT 20
    UNION ALL SELECT 21
    UNION ALL SELECT 22
    UNION ALL SELECT 23
    UNION ALL SELECT 24
    UNION ALL SELECT 25
    UNION ALL SELECT 26
    UNION ALL SELECT 27
	UNION ALL SELECT 28
	UNION ALL SELECT 29
	UNION ALL SELECT 30
), //  extra  comma
SELECT * FROM t1

单测结果：
image

@ChaconneLuo ChaconneLuo force-pushed the bugfix/spark_sql_union_recursion branch from c9187a7 to a98e4d9 Compare March 18, 2026 08:02
ChaconneLuo
ChaconneLuo commented Mar 23, 2026
View reviewed changes
src/grammar/spark/SparkSqlParser.g4

queryTerm
: queryPrimary
| left=queryTerm operator=(KW_INTERSECT | KW_UNION | KW_EXCEPT | KW_MINUS) setQuantifier? right=queryTerm
Copy link
Author

@ChaconneLuo ChaconneLuo Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

此条规则在spark最新稳定版本中，是依据SQL标准做的优先级区分，此移除规则在spark中是默认关闭的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ChaconneLuo