H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables

About

H-STAR revolutionizes tabular reasoning by combining textual and symbolic approaches for precise and flexible analysis. It first extracts relevant table sections using a ‘multi-view’ column extraction followed by row extraction, then adapts reasoning strategies based on question type—using semantic reasoning for lookups and complex queries while integrating symbolic reasoning for calculations and logic. Extensive evaluations show H-STAR surpasses state-of-the-art methods in tabular QA and fact verification.

The Need for a Hybrid Approach

Existing approaches that rely solely on either semantic/symbolic reasoning often face challenges due to the intricate interplay of numerical, temporal, and textual data, combined with complex table structures. While textual reasoning excels in natural language understanding, it frequently misinterprets table structures and struggles with quantitative reasoning. Conversely, SQL-based approaches are proficient in quantitative problem-solving but falter when handling noisy or unstructured inputs.

H-STAR Overview

H-STAR enhances tabular reasoning through a two-stage process: table extraction and adaptive reasoning. It first employs a multi-view approach to extract relevant table sections, identifying columns from both the original and transposed table. The original table is then filtered to only contain the data from the selected columns before passing on to the row selection. This method ensures precise table extraction minimizing hallucinations. In the second stage, H-STAR dynamically adapts its reasoning based on the question type. It relies on semantic reasoning for lookups, common-sense, and lexical queries, while integrating symbolic (SQL-based) reasoning for mathematical and logical tasks. By combining these strengths, H-STAR surpasses traditional LLM-based text-only and SQL-only methods, achieving more accurate and flexible table-based reasoning.

Results

H-STAR framework outperforms previous state-of-the-art methods on both WikiTQ and TabFact across five Large Language Models!

H-STAR's Efficiency

H-STAR framework reduces the number of table cells efficiently compared to other methods that use table extraction. Furthermore, H-STAR outperforms all baselines on longer tables containing more than 4000 tokens.

Performance on Longer Tables

Method	Small	Medium	Large
BINDER	56.54	25.13	6.41
DATER	62.50	42.34	34.62
Chain-of-Table	68.13	52.25	44.87
TabSQLify	68.15	57.91	52.34
H-STAR	71.64	65.20	64.84

Error Analysis

H-STAR excels in table extraction and reasoning, showing fewer errors in column/row extraction and better overall performance compared to TabSQLify and BINDER, with higher reasoning errors reflecting its strong table retrieval.

Acknowledgement

This implementation is based on Binding Language Models in Symbolic Languages. The work has also benefitted from TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition. Thanks to the author for releasing the code.

Citation

Please cite our paper as below.

@article{abhyankar2024h,
            title= "H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables",
            author="Abhyankar, Nikhil and 
            Gupta, Vivek and 
            Roth, Dan and 
            Reddy, Chandan K",
            journal={arXiv preprint arXiv:2407.05952},
            year={2024}
          }