This article offers valuable insights into the accuracy of AI models in generating SQL queries from data models, which is crucial for understanding the practical implications and challenges of AI tooling in analytics and data engineering.

• — Substantial accuracy (94-95%) can be achieved in AI analytics using simpler data models without a semantic layer, challenging the need for complex predefined metrics.
• — BIRD benchmark's strict evaluation criteria can misrepresent model performance, with 49 errors found in the training dataset alone, indicating a need for more robust scoring methodologies.
• — LLM-enhanced reviews can significantly improve answer quality by allowing models to adapt interpretations, thereby preventing penalization for correct yet non-standard SQL outputs.