BI-Agent 및 BI-Bench: 엔드투엔드 비즈니스 인텔리전스 자동화를 향하여
이 논문은 사용자가 데이터 준비 과정 없이 비즈니스 질문에 직접 답할 수 있는 end-to-end Business Intelligence(BI) 자동화를 연구합니다. 이를 위해 실제 BI 프로젝트에서 질문과 정답 쌍을 추출하여 구축한 최초의 벤치마크인 BI-Bench를 제안합니다. 실험 결과, 최신 frontier LLM들도 end-to-end BI 작업에서는 여전히 성능이 미흡함을 확인했습니다.
Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorl