DBQR-QA
A Question-Answering Dataset on a Hybrid of Database Querying and Reasoning
Submission starts 23 September 2024
A SHARED TASK
TLDR
Generate programs to answer a series of questions.
WHAT YOU GET
400 questions, ten per conversation, with tables (Pandas Data Frame) queried from our graph database.
YOUR MODEL GENERATES
Programs answering each question using our pre-defined (Python) or any custom functions in any language.
WHAT WE EVALUATE
The answer your model generated (number, text, set, or table).
Test the models' abilities to memorize and adapt to the changing logic throughout a conversation.
QUESTION 1
What is the ratio of combined revenues reported by companies in the retail industry with liabilities between 100M and 500M every year during 2017 and 2020 to those with liabilities between 300M and 1B during the same years?
QUESTION 4
If I removed the top three companies by combined deferred tax liabilities from the first group, what would be the ratio?
QUESTION 4
If I removed the top three companies by combined deferred tax liabilities from the first group, what would be the ratio?
Evaluation
Three-step automatic and manual evaluation.
AUTOMATIC
Evaluate using our Python evaluation script offline (unlimited) or online (up to 20 times per day).
GPT-4O
Close-to-human evaluation accuracy. Use our prompt (unlimited) or evaluate online (once a day).
MANUAL
One week to check your answers with all labels manually. Start after the system submission deadline.
Dataset Statistics
Five subsets of questions by type and complexity.
100 ×
S
: Simple query with specific companies
View
100 ×
C
: Complex query with unspecified companies
View
50 ×
T
: Reasoning steps requiring multiple tables
View
100 ×
H
: Multiple-hop query
View
50 ×
I
: Instruction QA
View
Examples and practice data are now available.
See the quick-start page for more information.
Stages
Practice
50 questions
Training
200 questions
Blind test
150 questions
Workshop
@COLING 2025 in Abu Dhabi, UAE
19-20 January 2025
Important Dates
AoE time zone
Done
Practice set release 2 Sep 2024
In progress
Training set release 23 Sep 2024
Pending
Blind test set release 23 Oct 2024
Pending
Submission deadline 30 Oct 2024
Pending
Manual evaluation deadline 7 Nov 2024
Pending
Release of results 12 Nov 2024
Pending
Paper submission deadline 25 Nov 2024
Pending
Notification of acceptance 5 Dec 2024
Pending
Camera-ready deadline 13 Dec 2024
Get Ready
Check back soon for further updates or subscribe to our mailing list.
Copyright © 2024-2025 R. Nararatwong et al. The images are generated by Dall-E · 3.
This website has been designed using resources from Flaticon.com. See credit page for links to materials used.
This study is partially based on the results obtained from a project JPNP20006,
commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
DBQR-QA
Quick Start
Downloads
Participate
Code
Issues
Paper
Contact
DBQR-QA
menu