Solving NL2SQL
Finetuned Models vs. Frontier Models
A key decision is whether the Large Language Model used to generate SQL queries should be a general-purpose larger proprietary model prompted with schema information or a smaller model finetuned specifically on queries for the target schema.
We tested this and our Benchmarks show that a smaller fine-tuned model performs on par or even outperforms significantly larger general-purpose models on domain-specific SQL Generation:
Evaluation Details
70 Samples distinct from Training Dataset (~350 samples)
Each Sample categorized by Query Complexity and domain (what kind of data was accessed)
Metric used: Semantic Query Similarity of generated query with handcrafted ground-truth query (This is computed by comparing the database execution plans of both queries in detail)
The finetuned model receives: the user request
The larger LLMs receive: the user request + semantically enriched schema description (e.g. with explanations of database concepts) + few-shot examples of NL Query -> SQL Mappings from the training dataset
Overall it seems like our finetuned model bpro-suites-nl2sql-3B performs similar to larger open weight models like DeepSeek or Qwen and larger proprietary models like the non-reasoning models of the GPT5 family.
On evaluation requests that require more complex SQL queries to satisfy (e.g. Multiple Joins, CTEs, Subqueries) the finetuned model starts to pay off, as its generated queries are much more aligned with the target schema:
With 3 Billion Parameters our model is roughly 300 times smaller than the larger LLMs it is compared with here.
This circumstance is reinforced by breaking down the evaluation results by data domain. While larger models can generalize well to easier requests revolving around asset configuration and current data, they struggle with generalizing to more complex historical aggregations (trend) as these require to also take the asset hierarchy into account to first retrieve the relevant assets in the query and then their timeseries data:
Reasoning Models of the GPT-5. or Claude Opus Family were not evaluated. As low latency was a key requirement.
Context Engineering
Finetuning on schema-specific NL-> SQL pairs teach a model the structure of the database (tables, columns, joins, efficient query patterns), domain-specific conventions (e.g. how attributes, subtypes, and units are organized) and how to map common natural language phrasings to SQL constructs optimized for the specific Database in question. However it does not teach:
Which specific assets exist in a given project
What a user means by "the building" in their specific context
Whether a particular asset type actually has data for a requested metric
How to resolve ambiguity when multiple valid interpretations exist
That is why a model that can generate queries for a specific schema very well is only part of a user facing solution that allows to communicate to the database through natural language.
Semantic Layer
A semantic layer is often described as an abstraction layer between the user and the database that makes it easier to formulate queries and understand the semantic meaning of the data. Here it represents an approach to bridge the gap that cannot be solved by fine-tuning an LLM alone.
The way users understand their data and its structure through a user interface is often very different from how that data is ultimately stored in a database. The greater this difference, the more difficult it becomes to connect a request based on assumptions about one structure with the reality of the other.
Ontologies are a powerful tool for describing and understanding semantic structures. They allow us to define relationships between concepts and their properties. In the context of BuildingPro Suites, this means describing how portfolios are structured and how they can be structured. At the center of this model are assets, which can take various forms such as sensors, locations, buildings, or specific areas. How BPro Suites covers ontologies therefore describes how these assets can be modeled and how they relate to one another.
Within this context, the semantic layer is used to connect natural language queries with the concepts and structure defined by this ontology. Its purpose is to dynamically contextualize SQL generation as accurately as possible, effectively linking what a user is asking for in natural language with how the underlying data actually exists.
The semantic layer helps the LLM better understand what the user intends to retrieve, reducing ambiguity during query generation. For example t can clarify which assets a user wants to query before SQL generation takes place. In addition, it helps users understand how their request is interpreted by the system and how they might refine it to achieve better results.
Example Flow:
Link the natural language query to concepts and structures within the ontology
Extract relationships between these concepts (e.g. located in)
Use predefined rules for concepts and their relationships to map them to actual data instances.
Contextualize SQL generation based on the identified concepts and their corresponding database instances.
The core idea is that the semantic layer serves as a bridge between the natural language query and the database by leveraging concepts and relationships defined in the ontology. Rather than translating a natural language request directly into SQL, the system first attempts to map the request to concepts that users already know and understand because they are visible in the user interface.
Ambiguities, arguably the biggest challenge in natural language querying, can therefore be resolved by linking user requests directly to actual data entities. In the context of BuildingPro Suites these would most commonly be assets and asset types.
Key Takeaways
LLMs are often only as powerful as the context they are given.
LLMs are rarely the complete solution on their own and typically require surrounding systems that guide and support them when solving more complex tasks.
Small Models geared towardsa specific task can outperform much larger, more powerful models
Finetuning can work very well for optimizing performance on a specific database schema, but it requires dataset updates and retraining whenever significant schema changes occur.
Since generated SQL queries should ultimately reflect the user's intent, it makes sense to involve the user in the process and base the interpretation on concepts that the user understands best
Last updated