🦅 Falcon: Enterprise-Grade Text-to-SQL Benchmark 🦅 Falcon：企业级 Text-to-SQL 基准测试

🏆 Leaderboard 🏆 排行榜

Current evaluation results on the Falcon benchmark (500 Questions).
Sorted by Execution Accuracy (EX). "VES" denotes Valid Executable SQL rate.

Falcon 基准测试（500题）的当前评估结果。
按 执行准确率 (Execution Accuracy, EX) 排序。"VES" 代表有效可执行 SQL 比率。

📖 Introduction 📖 项目简介

Falcon is a continuously evolving, high-quality benchmark designed to bridge the gap between academic Text-to-SQL datasets and real-world enterprise requirements. Unlike traditional benchmarks, Falcon focuses on MaxCompute/Hive dialects and stresses models with complex SQL patterns and linguistic ambiguities common in production environments.

Falcon 是一个持续迭代的高质量基准测试，旨在弥合学术界 Text-to-SQL 数据集与真实企业需求之间的差距。与传统基准测试不同，Falcon 专注于 MaxCompute/Hive 方言，并着重测试模型在生产环境中常见的复杂 SQL 模式和语言歧义处理能力。

📂 Dataset Structure 📂 数据集结构

The Falcon benchmark is split into a Development Set (with ground truth) and a Test Set (blind). The repository structure is designed for easy integration with evaluation pipelines.

Falcon 基准测试分为 开发集 (Development Set)（包含标准答案）和 测试集 (Test Set)（盲测）。仓库结构设计便于与评估流程集成。

FALCON/
├── dev_data/                   # Development Set (开发集)
│   ├── dev.json                # Questions, SQL, and Execution Results
│   ├── tables.json             # Schema definitions (PK/FK/Columns)
│   └── dev_databases/          # SQLite/CSV source files for execution
│
├── test_data/                  # Test Set (测试集)
│   ├── test.json               # Questions ONLY (Ground truth hidden)
│   ├── tables.json             # Schema definitions
│   └── test_databases/         # SQLite/CSV source files
│
├── simple_agent/               # [NEW] Lightweight Evaluation Scripts (轻量级评估脚本)
│   ├── comparator.py           # SQL execution result comparator
│   ├── utils.py                # Utilities for SQL extraction
│   └── simple_benchmark.py     # Main script to run dev/test evaluation
│
├── submission/                 # [NEW] Submission Helpers & Examples (提交辅助工具)
│   ├── example_submission_csv/ # Example CSV files for leaderboard
│   ├── example_submission_sql/ # Example SQL files for leaderboard
│   └── format_submission.py    # Helper to convert DB-GPT Excel output
│
└── README.md

🚀 Getting Started 🚀 快速开始

We provide two methods for evaluating your models on the Falcon benchmark.

我们提供两种方法来在 Falcon 基准上评估您的模型。

# For Development Set (开发集)
python simple_benchmark.py dev

# For Test Set (测试集)
python simple_benchmark.py test

python submission/format_submission.py \
--input <dbgpt_output.xlsx> \
--output submission.zip

📝 Citation 📝 引用

If you use Falcon in your research or development, please cite our paper:

如果您在研究或开发中使用了 Falcon，请引用我们的论文：

@article{falcon2025,
  title={Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation},
  author={Luo, Wenzhen and Guan, Wei and Yao, Yifan and Pan, Yimin and Wang, Feng and Yu, Zhipeng and Wen, Zhe and Chen, Liang and Zhuang, Yihong},
  journal={arXiv preprint arXiv:2510.24762},
  year={2025},
  url={https://arxiv.org/abs/2510.24762}
}