Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study Ziyu Yao 1 , Yu Su 1 , Huan Sun 1 , Wen-tau Yih 2* {yao.470, su.809, sun.397}@osu.edu [email protected]1 The Ohio State University 2 Facebook AI Research, Seattle Abstract As a promising paradigm, interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the re- sults. In this paper, we propose a new, unified formulation of the interactive semantic parsing problem, where the goal is to design a model- based intelligent agent. The agent maintains its own state as the current predicted semantic parse, decides whether and where human inter- vention is needed, and generates a clarification question in natural language. A key part of the agent is a world model: it takes a percept (ei- ther an initial question or subsequent feedback from the user) and transitions to a new state. We then propose a simple yet remarkably ef- fective instantiation of our framework, demon- strated on two text-to-SQL datasets (WikiSQL and Spider) with different state-of-the-art base semantic parsers. Compared to an existing in- teractive semantic parsing approach that treats the base parser as a black box, our approach solicits less user feedback but yields higher run-time accuracy. 1 1 Introduction Natural language interfaces that allow users to query data and invoke services without program- ming have been identified as a key application of semantic parsing (Berant et al., 2013; Thomason et al., 2015; Dong and Lapata, 2016; Zhong et al., 2017; Campagna et al., 2017; Su et al., 2017). However, existing semantic parsing technologies often fall short when deployed in practice, facing several challenges: (1) user utterances can be in- herently ambiguous or vague, making it difficult to get the correct result in one shot, (2) the ac- curacy of state-of-the-art semantic parsers are still not high enough for real use, and (3) it is hard for * Work started while at AI2 1 Code available at https://github.com/ sunlab-osu/MISP. User Intent Actuator Error Detector Action World Model State Percept Question MISP Agent Environment Figure 1: Model-based Interactive Semantic Parsing (MISP) framework. users to validate the semantic parsing results, es- pecially with mainstream neural network models that are known for the lack of interpretability. In response to these challenges, interactive se- mantic parsing has been proposed recently as a practical solution, which includes human users in the loop to resolve utterance ambiguity, boost sys- tem accuracy, and improve user confidence via human-machine collaboration (Li and Jagadish, 2014; He et al., 2016; Chaurasia and Mooney, 2017; Su et al., 2018; Gur et al., 2018; Yao et al., 2019). For example, Gur et al. (2018) built the DialSQL system to detect errors in a generated SQL query and request user selection on alter- native options via dialogues. Similarly, Chaura- sia and Mooney (2017) and Yao et al. (2019) en- abled semantic parsers to ask users clarification questions while generating an If-Then program. Su et al. (2018) showed that users overwhelm- ingly preferred an interactive system over the non- interactive counterpart for natural language inter- faces to web APIs. While these recent studies suc- cessfully demonstrated the value of interactive se- mantic parsing in practice, they are often bound to a certain type of formal language or dataset, and the designs are thus ad-hoc and not easily gen- eralizable. For example, DialSQL only applies
Transcript
Model-based Interactive Semantic Parsing:A Unified Framework and A Text-to-SQL Case Study
1The Ohio State University2Facebook AI Research, Seattle
Abstract
As a promising paradigm, interactive semanticparsing has shown to improve both semanticparsing accuracy and user confidence in the re-sults. In this paper, we propose a new, unifiedformulation of the interactive semantic parsingproblem, where the goal is to design a model-based intelligent agent. The agent maintainsits own state as the current predicted semanticparse, decides whether and where human inter-vention is needed, and generates a clarificationquestion in natural language. A key part of theagent is a world model: it takes a percept (ei-ther an initial question or subsequent feedbackfrom the user) and transitions to a new state.We then propose a simple yet remarkably ef-fective instantiation of our framework, demon-strated on two text-to-SQL datasets (WikiSQLand Spider) with different state-of-the-art basesemantic parsers. Compared to an existing in-teractive semantic parsing approach that treatsthe base parser as a black box, our approachsolicits less user feedback but yields higherrun-time accuracy.1
1 Introduction
Natural language interfaces that allow users toquery data and invoke services without program-ming have been identified as a key application ofsemantic parsing (Berant et al., 2013; Thomasonet al., 2015; Dong and Lapata, 2016; Zhong et al.,2017; Campagna et al., 2017; Su et al., 2017).However, existing semantic parsing technologiesoften fall short when deployed in practice, facingseveral challenges: (1) user utterances can be in-herently ambiguous or vague, making it difficultto get the correct result in one shot, (2) the ac-curacy of state-of-the-art semantic parsers are stillnot high enough for real use, and (3) it is hard for
∗Work started while at AI21Code available at https://github.com/
users to validate the semantic parsing results, es-pecially with mainstream neural network modelsthat are known for the lack of interpretability.
In response to these challenges, interactive se-mantic parsing has been proposed recently as apractical solution, which includes human users inthe loop to resolve utterance ambiguity, boost sys-tem accuracy, and improve user confidence viahuman-machine collaboration (Li and Jagadish,2014; He et al., 2016; Chaurasia and Mooney,2017; Su et al., 2018; Gur et al., 2018; Yao et al.,2019). For example, Gur et al. (2018) built theDialSQL system to detect errors in a generatedSQL query and request user selection on alter-native options via dialogues. Similarly, Chaura-sia and Mooney (2017) and Yao et al. (2019) en-abled semantic parsers to ask users clarificationquestions while generating an If-Then program.Su et al. (2018) showed that users overwhelm-ingly preferred an interactive system over the non-interactive counterpart for natural language inter-faces to web APIs. While these recent studies suc-cessfully demonstrated the value of interactive se-mantic parsing in practice, they are often bound toa certain type of formal language or dataset, andthe designs are thus ad-hoc and not easily gen-eralizable. For example, DialSQL only applies
to SQL queries on the WikiSQL dataset (Zhonget al., 2017), and it is non-trivial to extend it toother formal languages (e.g., λ-calculus) or evenjust to more complex SQL queries beyond the tem-plates used to construct the dataset.
Aiming to develop a general principle forbuilding interactive semantic parsing systems, inthis work we propose model-based interactivesemantic parsing (MISP), where the goal is to de-sign a model-based intelligent agent (Russell andNorvig, 2009) that can interact with users to com-plete a semantic parsing task. Taking an utter-ance (e.g., a natural language question) as input,the agent forms the semantic parse (e.g., a SQLquery) in steps, potentially soliciting user feed-back in some steps to correct parsing errors. Asillustrated in Figure 1, a MISP agent maintains itsstate as the current semantic parse and, via an er-ror detector, decides whether and where humanintervention is needed (the action). This action isperformed by a question generator (the actuator),which generates and presents to the user a human-understandable question. A core component ofthe agent is a world model (Ha and Schmidhu-ber, 2018) (hence model-based), which incorpo-rates user feedback from the environment and tran-sitions to a new agent state (e.g., an updated se-mantic parse). This process repeats until a termi-nal state is reached. Such a design endows a MISPagent with three crucial properties of interactivesemantic parsing: (1) being introspective of thereasoning process and knowing when it may needhuman supervision, (2) being able to solicit userfeedback in a human-friendly way, and (3) beingable to incorporate user feedback (through statetransitions controlled by the world model).
The MISP framework provides several advan-tages for designing an interactive semantic parsercompared to the existing ad-hoc studies. For in-stance, the whole problem is conceptually reducedto building three key components (i.e., the worldmodel, the error detector, and the actuator), andcan be handled and improved separately. Whileeach component may need to be tailored to thespecific task, the general framework remains un-changed. In addition, the formulation of a model-based intelligent agent can facilitate the applica-tion of other machine learning techniques like re-inforcement learning.
To better demonstrate the advantages of theMISP framework, we propose a simple yet re-
markably effective instantiation for the text-to-SQL task. We show the effectiveness of theframework based on three base semantic parsers(SQLNet, SQLova and SyntaxSQLNet) and twodatasets (WikiSQL and Spider). We empiricallyverified that with a small amount of targeted, test-time user feedback, interactive semantic parsersimprove the accuracy by 10% to 15% absolute.Compared to an existing interactive semantic pars-ing system, DialSQL (Gur et al., 2018), our ap-proach, despite its much simpler yet more generalsystem design, achieves better parsing accuracy byasking only half as many questions.
2 Background & Related Work
Semantic Parsing. Mapping natural language ut-terances to their formal semantic representations,semantic parsing has a wide range of applications,including question answering (Berant et al., 2013;Dong and Lapata, 2016; Finegan-Dollak et al.,2018), robot navigation (Artzi and Zettlemoyer,2013; Thomason et al., 2015) and Web API call-ing (Quirk et al., 2015; Su et al., 2018). The targetapplication in this work is text-to-SQL, which hasbeen popularized by the WikiSQL dataset (Zhonget al., 2017). One of the top-performing modelson WikiSQL is SQLNet (Xu et al., 2017), whichleverages the pre-defined SQL grammar sketcheson WikiSQL and solves the SQL generation prob-lem via “slot filling.” By augmenting SQLNet witha table-aware BERT encoder (Devlin et al., 2019)and by revising the value prediction in WHEREclauses, SQLova (Hwang et al., 2019) advancesfurther the state of the art. Contrast to WikiSQL,the recently released Spider dataset (Yu et al.,2018c) focuses on complex SQL queries contain-ing multiple keywords (e.g., GROUP BY) and mayjoin multiple tables. To handle such complexity,Yu et al. (2018b) proposed SyntaxSQLNet, a syn-tax tree network with modular decoders, whichgenerates a SQL query by recursively calling amodule following the SQL syntax. However, be-cause of the more realistic and challenging settingin Spider, it only achieves 20% in accuracy.
We experiment our MISP framework with theaforementioned three semantic parsers on bothWikiSQL and Spider. The design of MISP allowsnaturally integrating them as the base parser. Forexample, when SQLNet fills a sequence of slotsto produce a SQL query, a “state” in MISP corre-sponds to a partially generated SQL query and it
transitions as SQLNet fills the next slot.
Interactive Semantic Parsing. To enhance pars-ing accuracy and user confidence in practicalapplications, interactive semantic parsing hasemerged as a promising solution (Li and Jagadish,2014; He et al., 2016; Chaurasia and Mooney,2017; Su et al., 2018; Gur et al., 2018; Yao et al.,2019). Despite their effectiveness, existing solu-tions are somewhat ad-hoc and bound to a spe-cific formal language and dataset. For example,DialSQL (Gur et al., 2018) is curated for Wik-iSQL, where SQL queries all follow the sameand given grammar sketch. Similarly, (Yao et al.,2019) relies on a pre-defined two-level hierar-chy among components in an If-Then programand cannot generalize to formal languages witha deeper structure. In contrast, MISP aims for ageneral design principle by explicitly identifyingand decoupling important components, such as er-ror detector, question generator and world model.It also attempts to integrate and leverage a strongbase semantic parser, and transforms it to a naturalinteractive semantic parsing system, which sub-stantially reduces the engineering cost.
3 Model-based Interactive SemanticParsing
We now discuss the MISP framework (Figure 1)in more detail. Specifically, we highlight the func-tion of each major building block and the relation-ships among them, and leave the description of aconcrete embodiment to Section 4.
Environment. The environment consists of a userwith a certain intent, which corresponds to a se-mantic parse that the user expects the agent to pro-duce. Based on this intent, the user gives an initialnatural language utterance u0 to start a semanticparsing session and responds to any clarificationquestion from the agent with feedback ut at inter-action turn t.
Agent State. The agent state s is an agent’s inter-nal interpretation of the environment based on allthe available information. A straightforward de-sign of the agent state is as the currently predictedsemantic parse. It can also be endowed with metainformation of the parsing process such as predic-tion probability or uncertainty to facilitate errordetection.
World Model. A key component of a MISP agentis its world model (Ha and Schmidhuber, 2018),
which compresses the historical percepts through-out the interaction and predicts the future based onthe agent’s knowledge of the world. More specif-ically, it models the transition of the agent state,p(st+1|st, ut), where ut is the user feedback atstep t and st+1 is the new state. The transitioncan be deterministic or stochastic.
Error Detector. A MISP agent introspects itsstate and decides whether and where human inter-vention is needed. The error detector serves thisrole. Given the current state st (optionally the en-tire interaction history) and a set of terminal states,it decides on an action at: If the agent is at a ter-minal state, it terminates the session, executes thesemantic parse, and returns the execution resultsto the user; otherwise, it determines a span in thecurrent semantic parse that is likely erroneous andpasses it, along with necessary context informa-tion needed to make sense of the error span, to theactuator.
Actuator. An actuator has a user-facing interfaceand realizes an agent’s actions in a user-friendlyway. In practice, it can be a natural language gen-erator (NLG) (He et al., 2016; Gur et al., 2018;Yao et al., 2019) or an intuitive graphical user in-terface (Su et al., 2018; Berant et al., 2019), or thetwo combined.
4 MISP-SQL: An Instantiation of MISPfor Text-to-SQL
Under the MISP framework, we design an inter-active semantic parsing system (Figure 2), namedMISP-SQL, for the task of text-to-SQL transla-tion. MISP-SQL assumes a base text-to-SQLparser and leverages it to design the world modeland the error detector. The world model is es-sentially a wrapper that takes the user input andchanges the behavior of the base semantic parser(e.g., by changing the probability distribution orremoving certain prediction paths). The error de-tector makes decisions based on the uncertainty ofthe predictions: if the parser is uncertain about aprediction, it is more likely to be an error. The ac-tuator is a template-based natural language ques-tion generator developed for the general SQL lan-guage. Figure 2 shows an example of the MISP-SQL agent.
4.1 Agent StateFor ease of discussion, we assume the base parsergenerates the SQL query by predicting a sequence
of SQL components,2 as in many state-of-the-art systems (Xu et al., 2017; Wang et al., 2018;Yu et al., 2018a; Hwang et al., 2019). Agentstate st is thus defined as a partial SQL query,i.e., st={o1, o2, ..., ot}, where ot is the predictedSQL component at time step t, such as SELECTplace in Figure 2. What constitutes a SQL com-ponent is often defined differently in different se-mantic parsers, but typically dictated by the SQLsyntax. To support introspection and error detec-tion, each prediction is associated with its uncer-tainty, which is discussed next.
4.2 Error Detector
The error detector in MISP-SQL is introspectiveand greedy. It is introspective because it examinesthe uncertainty of the predictions as opposed to thepredictions themselves. On the other hand, it isgreedy because its decisions are solely based onthe last prediction ot instead of the entire state st.
We experiment with two uncertainty measures,based on the probability of ot estimated by thebase semantic parser, as well as its standard de-viation under Bayesian dropout (Gal and Ghahra-mani, 2016), respectively.
Probability-based Uncertainty. Intuitively if thebase semantic parser gives a low probability to thetop prediction at a step, it is likely uncertain aboutthe prediction. Specifically, we say a prediction otneeds user clarification if its probability is lowerthan a threshold p∗, i.e.,
p(ot) < p∗.
This strategy is shown to be strong in detect-ing misclassified and out-of-distribution examples(Hendrycks and Gimpel, 2017).
Dropout-based Uncertainty. Dropout (Srivas-tava et al., 2014) has been used as a Bayesianapproximation for estimating model uncertainty(Gal and Ghahramani, 2016) in several tasks(Dong et al., 2018; Siddhant and Lipton, 2018;Xiao and Wang, 2019). Different from its standardapplication to prevent models from overfitting intraining time, we use it at test time to measuremodel uncertainty, similar to (Dong et al., 2018).The intuition is that if the probability on a pre-diction varies dramatically (as measured by the
2In practice this assumption may not be necessary as longas there is a reasonable way to chunk the semantic parse tocalculate uncertainty and formulate clarification questions.
Figure 2: MISP-SQL Agent. The base semantic parserincrementally parses the user question (Step 1) into aSQL query by first selecting a column from the table(Step 2). This partial parse is examined by the errordetector (Step 3), who determines that the predictionis incorrect (because the uncertainty is high) and trig-gers the actuator to ask the user a clarification question(Step 4). The user feedback is then incorporated intothe world model (Step 5) to update the agent state. Ifthe prediction was correct, Step 2 would be repeated tocontinue the parsing.
standard deviation) across different perturbationsunder dropout, the model is likely uncertain aboutit. Specifically, the uncertainty on prediction ot iscalculated as:
STDDEV{p(ot|Wi)}Ni=1,
where Wi is the parameters of the base seman-tic parser under the i-th dropout perturbation, andthe uncertainty score is the standard deviation ofthe prediction probabilities overN random passes.We say ot needs user clarification if its uncertaintyscore is greater than a threshold s∗.
Terminal State. The only terminal state is whenthe base semantic parser indicates end of parsing.
4.3 Actuator: An NL GeneratorThe MISP-SQL agent performs its action (e.g.,validating the column “place”) via asking users bi-nary questions, hence the actuator is a natural lan-guage generator (NLG). Although there has beenwork on describing a SQL query with an NL state-ment (Koutrika et al., 2010; Ngonga Ngomo et al.,2013; Iyer et al., 2016; Xu et al., 2018), few workstudies generating questions about a certain SQLcomponent in a systematic way.
Inspired by (Koutrika et al., 2010; Wang et al.,2015), we define a rule-based NLG, which con-sists of a seed lexicon and a grammar for de-riving questions. Table 1 shows rules covering
[Lexicon]is greater than|equals to|is less than → OP[>|=|<]
sum of values in|average value in|number of|minimum value in|maximum value in → AGG[sum|avg|count|min|max]
[Grammar]“col” → COL[col]
Does the system need to return information about COL[col] ? → Q[col‖SELECT agg? col]Does the system need to return AGG[agg] COL[col] ? → Q[agg‖SELECT agg col]
Does the system need to return a value after any mathematical calculations on COL[col] ? → Q[agg=None‖SELECT col]Does the system need to consider any conditions about COL[col] ? → Q[col‖WHERE col op val]
The system considers the following condition: COL[col] OP[op] a value. Is this condition correct? → Q[op‖WHERE col op val]The system considers the following condition: COL[col] OP[op] val. Is this condition correct? → Q[val‖WHERE col op val]
Table 1: Domain-general lexicon and grammar for NL generation in MISP-SQL (illustrated for WikiSQL; a morecomprehensive grammar for Spider can be found in Appendix A).
SQL queries on WikiSQL (Zhong et al., 2017).The seed lexicon defines NL descriptions for basicSQL elements in the form of “n→ t[p]”, where nis an NL phrase, t is a pre-defined syntactic cat-egory and p is either an aggregator (e.g., avg) oran operator (e.g.,>). For example, “is greater than→ OP[>]” specifies a phrase “is greater than” todescribe the operator “>”. In MISP-SQL, we con-sider four syntactic categories: AGG for aggrega-tors, OP for operators, COL for columns and Q forgenerated questions. However, it can be extendedwith more lexicon entries and grammar rules toaccommodate more complex SQL in Spider (Yuet al., 2018c), which we show in Appendix A.
The grammar defines rules to derive questions.Each column is described by itself (i.e., the col-umn name). Rules associated with each Q-typeditem “Q[v‖Clause]” constructs an NL questionasking about v in Clause. The Clause is thenecessary context to formulate meaningful ques-tions. Figure 3 shows a derivation example. Notethat, both the lexicon and the grammar in our sys-tem are domain-agnostic in the sense that it isnot specific to any database. Therefore, it can bereused for new domains in the future. Database-specific rules, such as naming each column witha more canonical phrase (rather than the columnname), are also possible.
4.4 World Model
The agent incorporates user feedback and updatesits state with a world model. Different from Di-alSQL which trains an additional neural network,the MISP-SQL agent directly employs the base se-mantic parser to transition states, which saves ad-ditional training efforts.
As introduced in Section 4.3, the agent raises abinary question to the user about a predicted SQLcomponent ot. Therefore, the received user feed-
Figure 3: Deriving an NL question about the aggregatormax in the clause “SELECT max(age)” from the rootedQ-typed item .
back either confirms the prediction or negates it.In the former case, the state is updated by proceed-ing to the next decoding step, i.e., st+1={o1, ...,ot, ot+1}, where ot+1 is the predicted next compo-nent and st+1 shows the updated partial parse. Inthe latter case, the user feedback is incorporated toconstrain the search space of the base parser (i.e.,forbidding the parser from making the same wrongprediction), based on which the parser refreshesits prediction and forms a new state st+1={o1, ...,ot−1, ot+1}, where ot+1 is a predicted alternativeto replace ot. To avoid being trapped in a largesearch space, for each SQL component, we con-sider a maximum number of K alternatives (inaddition to the original prediction) to solicit userfeedback on.
5 Experiments
We apply our approach to the task of mapping nat-ural language questions to SQL queries. In thissection, we first describe the basic setup, includ-ing the datasets and the base semantic parsers, fol-lowed by the system results on both simulated andreal users.
5.1 Experimental SetupWe evaluate our proposed MISP-SQL agent onWikiSQL (Zhong et al., 2017), which contains80,654 hand-annotated pairs of 〈NL question,
Table 2: Simulation evaluation of MISP-SQL (based on SQLNet or SQLova) on WikiSQL Test set. “MISP-SQLp∗=X” denotes our agent with probability-based error detection (threshold at X). “MISP-SQLUnlimitK” denotesa variant that asks questions for every component, with up to K + 1 questions per component.
SQL query〉, distributed across 24,241 tables fromWikipedia. Our experiments follow the same datasplit as in (Zhong et al., 2017).
We experiment MISP-SQL with two base se-mantic parsers: SQLNet (Xu et al., 2017) andSQLova (Hwang et al., 2019). Unlike in Dial-SQL’s evaluation (Gur et al., 2018), we do notchoose Seq2SQL (Zhong et al., 2017) as a baseparser but SQLova instead, because it achievessimilar performance as SQLNet while SQLovais currently the best open-sourced model on Wik-iSQL, which can give us a more comprehensiveevaluation. For each of the two base semanticparsers, we test our agent with two kinds of er-ror detectors, based on prediction probability andBayesian dropout, respectively (Section 4.2). Wetune the threshold p∗ within 0.5 ∼ 0.95 and s∗
within 0.01 ∼ 0.2. Particularly for uncertainty-based detection measured by Bayesian dropout,the number of passesN is set to 10, with a dropoutrate 0.1. The dropout layers are applied at thesame positions as when each semantic parser istrained. When the agent interacts with users, themaximum number of alternative options (in addi-tion to the original prediction) per component, K,is set to 3. If the user negates all the K + 1 pre-dicted candidates, the agent will keep the originalprediction, as in (Gur et al., 2018).
5.2 Simulation EvaluationIn simulation evaluation, each agent interacts witha simulated user, who gives a yes/no answer basedon the ground-truth SQL query. If the agent failsto correct its predictions in three consecutive in-teraction turns, the user will leave the interactionearly and the agent has to finish the remaining gen-eration without further help from the user.
Overall Comparison. We first compare MISP-SQL with the two base semantic parsers without
interactions in Table 2. For SQLNet, we also com-pare our system with the reported performance ofDialSQL (Gur et al., 2018, Table 4). However,since DialSQL is not open-sourced and it is noteasy to reproduce it, we are unable to adapt it toSQLova for more comparisons. Following (Xuet al., 2017; Hwang et al., 2019), we evaluate theSQL query match accuracy (“Accqm”, after con-verting the query into its canonical form) and theexecution accuracy (“Accex”) of each agent. “Avg.#q” denotes the average number of questions perquery. For any base parser, MISP-SQL improvestheir performance by interacting with users. Par-ticularly for SQLNet, MISP-SQL outperforms theDialSQL system with only half the number ofquestions (1.104 vs. 2.4), and has a much simplerdesign without the need of training an extra model(besides training the base parser, which DialSQLneeds to do as well). Our agent can even boost thestrong performance of SQLova from 85% to 94%in execution accuracy, with merely 0.773 ques-tions per query.
We also present an “upper-bounded” accuracyof our agent, when it does not adopt any errordetector and asks questions about every compo-nent with at most 10 (“MISP-SQLUnlimit10”) or 3(“MISP-SQLUnlimit3”) alternatives. Interestingly,even for the weaker SQLNet parser, most true pre-dictions have already been contained within thetop 10 options (giving 0.932 query match accu-racy). When equipped with the stronger SQLovaparser, the agent has a potential to boost the ex-ecution accuracy to around 100% by consideringonly the top 3 options of every prediction. Thecomplete results can be found in Appendix B.
Error Detector Comparison. We then comparethe probability-based and dropout-based error de-tectors in Figure 4, where each marker indicates
Figure 4: Comparison of probability- and dropout-based error detection.
the agent’s accuracy and the average number ofquestions it needs under a certain error detec-tion threshold. Consistently for both SQLNet andSQLova, the probability-based error detector canachieve the same accuracy with a lower numberof questions than the dropout-based detector. Itis also observed that this difference is greater interms of query match accuracy, around 0.15 ∼0.25 for SQLNet and 0.1 ∼ 0.15 for SQLova.A more direct comparison of various settings un-der the same average number of questions can befound in Appendix C.
To better understand how each kind of error de-tectors works, we investigate the portion of ques-tions that each detector spends on right predictions(denoted as “Qr”). An ideal system should askfewer questions on right predictions while iden-tify more truly incorrect predictions to fix the mis-takes. We present the question distributions of thevarious systems in Table 3. One important conclu-sion drawn from this table is that probability-basederror detection is much more effective on identi-fying incorrect predictions. Consider the systemusing probability threshold 0.5 for error detection(i.e., “p∗=0.5”) and the one using dropout-basederror detector with a threshold 0.2 (i.e., “s∗=0.2”)on SQLNet. When both systems ask around thesame number of questions during the interaction,the former spends only 16.9% of unnecessaryquestions on correct predictions (Qr), while thelatter asks twice amount of them (32.1%). Sim-ilar situations are also observed for SQLova. Itis also notable that, when the probability thresh-old is lower (which results in a fewer total num-ber of questions), the portion of questions on rightactions drops significantly (e.g., from 23.0% to16.9% when the threshold changes from 0.8 to 0.5on SQLNet). However, this portion remains al-most unchanged for dropout-based error detection.
Table 3: Portion of interaction questions on right pre-dictions (Qr%) for each agent setting on WikiSQL Devset (smaller is better). “p∗/s∗=X” denotes our agentwith probability/dropout-based error detection (thresh-old at X).
5.3 Extend to Complex SQL GenerationA remarkable characteristic of MISP-SQL is itsgeneralizability, as it makes the best use of thebase semantic parser and requires no extra modeltraining. To verify it, we further experimentMISP-SQL on the more complex text-to-SQLdataset “Spider” (Yu et al., 2018c). The datasetconsists of 10,181 questions on multi-domaindatabases, where SQL queries can contain com-plex keywords such as GROUP BY or join severaltables. We extend the NLG lexicon and gram-mar (Section 4.3) to accommodate this complex-ity, with details shown in Appendix A.
We adopt SyntaxSQLNet (Yu et al., 2018b) asthe base parser.3 In our experiments, we followthe same database split as in (Yu et al., 2018c) andreport the Exact Matching accuracy (“Accem”) onDev set.4 Other experimental setups remain thesame as when evaluating MISP-SQL on WikiSQL.Table 4 shows the results.
We first observe that, via interactions with sim-ulated users, MISP-SQL improves SyntaxSQL-Net by 10% accuracy with reasonably 3 questionsper query. However, we also realize that, un-like on WikiSQL, in this setting, the probability-based error detector requires more questions thanthe Bayesian uncertainty-based detector. This canbe explained by the inferior performance of thebase SyntaxSQLNet parser (merely 20% accuracywithout interaction). In fact, the portion of ques-tions that the probability-based detector spendson right predictions (Qr) is still half of that thedropout-based detector asks (12.8% vs. 24.8%).However, it wastes around 60% of questions onunsolvable wrong predictions. This typically hap-
3We chose SyntaxSQLNet because it was the best modelby the paper submission time. In principle, our frameworkcan also be applied to more sophisticated parsers such as (Bo-gin et al., 2019; Guo et al., 2019).
4We do not report results on Spider test set since it is notpublicly available.
Table 4: Simulation evaluation of MISP-SQL (built onSyntaxSQLNet) on Spider Dev set.
pens when the base parser is not strong enough,i.e., cannot rank the true option close to the top,or when there are unsolved wrong precedent pre-dictions (e.g., in “WHERE col op val”, when colis wrong, whatever op/val following it is wrong).This issue can be alleviated when more advancedbase parsers are adopted in the future.
5.4 Human Evaluation
We further conduct human user study to evalu-ate the MISP-SQL agent. Our evaluation settinglargely follows Gur et al. (2018). For each base se-mantic parser, we randomly sample 100 examplesfrom the corresponding dataset (either WikiSQLTest set or Spider Dev set) and ask three humanevaluators, who are graduate students with onlyrudimentary knowledge of SQL based on our sur-vey, to work on each example and then report theaveraged results. We present to the evaluators theinitial natural language question and allow themto view the table headers to better understand thequestion intent. On Spider, we also show the nameof the database tables. We select error detectorsbased on the simulation results: For SQLNet andSQLova, we equip the agent with a probability-based error detector (threshold at 0.95); for Syn-taxSQLNet, we choose a Bayesian uncertainty-based error detector (threshold at 0.03). As in thesimulation evaluation, we cannot directly comparewith DialSQL in human evaluation because thecode is not yet publicly available.
Table 5 shows the results. In all settings,MISP-SQL improves the base parser’s perfor-mance, demonstrating the benefit of involving hu-man interaction. However, we also notice that thegain is not as large as in simulation, especiallyon SQLova. Through interviews with the humanevaluators, we found that the major reason is thatthey sometimes had difficulties understanding thetrue intent of some test questions that are ambigu-
Table 5: Human evaluation on 100 random examplesfor MISP-SQL agents based on SQLNet, SQLova andSyntaxSQLNet, respectively.
ous, vague, or contain entities they are not familiarwith. We believe this reflects a general challengeof setting up human evaluation for semantic pars-ing that is close to the real application setting, andthus set forth the following discussion.
5.5 Discussion on Future Human Evaluation
Most human evaluation studies for (interactive)semantic parsers so far (Chaurasia and Mooney,2017; Gur et al., 2018; Su et al., 2018; Yao et al.,2019) use pre-existing test questions (e.g., fromdatasets like WikiSQL). However, this introducesan undesired discrepancy, that is, human evalua-tors may not necessarily be able to understand thetrue intent of the given questions in an faithfulway, especially when the question is ambiguous,vague, or containing unfamiliar entities.
This discrepancy is clearly manifested in ourhuman evaluation with SQLova (Table 5). Whenthe base parser is strong, many of the remainingincorrectly parsed questions are challenging notonly for the base parser but also for human evalu-ators. We manually examined the situations whereevaluators made a different choice than the sim-ulator and found that 80% of such choices hap-pened when the initial question is ambiguous orthe gold SQL annotation is wrong. For exam-ple, for the question “name the city for kanjiza”it is unlikely for human evaluators to know that“kanjiza” is an “Urban Settlement” without look-ing at the table content or knowing the specificbackground knowledge beforehand. This issue hasalso been reported as the main limitation to fur-ther improve SQLova (Hwang et al., 2019), which
could in principle be resolved by human interac-tions if the users have a clear and consistent intentin mind. To verify this, we conduct an additionalexperiment with SQLova where human evaluatorscan view the table content as well as the gold SQLquery before starting the interaction to better un-derstand the true intent (denoted as “w/ full info”in Table 5). As expected, the MISP-SQL agentperforms much better (close to simulation) whenusers know what they are asking. It further con-firms that a non-negligible part of the accuracy gapbetween simulation and human evaluation is dueto human evaluators not fully understanding thequestion intent and giving false feedback.
To alleviate this discrepancy, a common prac-tice is to show human evaluators the schema of theunderlying database, as Gur et al. (2018) and wedid (Section 5.4), but it is still insufficient, espe-cially for entity-related issues (e.g., “kanjiza”). Onthe other hand, while exposing human evaluatorsto table content helps resolve the entity-related is-sues, it is likely to introduce undesired biases in fa-vor of the system under test (i.e., “overexposure”),since human evaluators may then be able to givemore informative feedback than real users.
To further reduce the discrepancy between hu-man evaluation and real use cases, one possiblesolution is to ask human evaluators to come upwith questions from scratch (instead of using pre-existing test questions), which guarantees intentunderstanding. While this solution may still re-quire exposure of table content to evaluators (suchthat they can have some sense of each table at-tribute), overexposure can be mitigated by show-ing them only part (e.g., just a few rows) of thetable content, similar to the annotation strategyby Zhong et al. (2017). Furthermore, the reducedcontrollability on the complexity of the evaluator-composed questions can be compensated by con-ducting human evaluation in a larger scale. Weplan to explore this setting in future work.
6 Conclusion and Future Work
This work proposes a new and unified frameworkfor the interactive semantic parsing task, namedMISP, and instantiates it successfully on the text-to-SQL task. We outline several future directionsto further improve MISP-SQL and develop MISPsystems for other semantic parsing tasks:
Improving Agent Components. The flexibilityof MISP allows improving on each agent compo-
nent separately. Take the error detector for exam-ple. One can augment the probability-based er-ror detector in MISP-SQL with probability cali-bration, which has been shown useful in align-ing model confidence with its reliability (Guoet al., 2017). One can also use learning-basedapproaches, such as a reinforced decision policy(Yao et al., 2019), to increase the rate of identify-ing wrong and solvable predictions.
Lifelong Learning for Semantic Parsing.Learning from user feedback is a promisingsolution for lifelong semantic parser improvement(Iyer et al., 2017; Padmakumar et al., 2017;Labutov et al., 2018). However, this may leadto a non-stationary environment (e.g., changingstate transition) from the perspective of the agent,making its training (e.g., error detector learning)unstable. In the context of dialog systems, Pad-makumar et al. (2017) suggests that this effect canbe mitigated by jointly updating the dialog policyand the semantic parser batchwisely. We leaveexploring this aspect in our task to future work.
Scaling Up. It is important for MISP agents toscale to larger backend data sources (e.g., knowl-edge bases like Freebase or Wikidata). To this end,one can improve MISP from at least three aspects:(1) using more intelligent interaction designs (e.g.,free-form text as user feedback) to speed up thehypothesis space searching globally, (2) strength-ening the world model to nail down a smaller setof plausible hypotheses based on both the initialquestion and user feedback, and (3) training theagent to learn to improve the parsing accuracywhile minimizing the number of required humaninterventions over time.
Acknowledgments
This research was sponsored in part by the ArmyResearch Office under cooperative agreementsW911NF-17-1-0412, NSF Grant IIS1815674, Fu-jitsu gift grant, and Ohio Supercomputer Center(Center, 1987). The views and conclusions con-tained herein are those of the authors and shouldnot be interpreted as representing the official poli-cies, either expressed or implied, of the Army Re-search Office or the U.S. Government. The U.S.Government is authorized to reproduce and dis-tribute reprints for Government purposes notwith-standing any copyright notice herein.
ReferencesYoav Artzi and Luke Zettlemoyer. 2013. Weakly su-
pervised learning of semantic parsers for mappinginstructions to actions. Transactions of the Associa-tion for Computational Linguistics, 1:49–62.
Jonathan Berant, Andrew Chou, Roy Frostig, and PercyLiang. 2013. Semantic parsing on freebase fromquestion-answer pairs. In Proceedings of the 2013Conference on Empirical Methods in Natural Lan-guage Processing, pages 1533–1544.
Jonathan Berant, Daniel Deutch, Amir Globerson,Tova Milo, and Tomer Wolfson. 2019. Explainingqueries over web tables to non-experts. In Proceed-ings of the 35th IEEE International Conference onData Engineering (ICDE).
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019.Representing schema structure with graph neuralnetworks for text-to-SQL parsing. In Proceedingsof the 57th Annual Meeting of the Association forComputational Linguistics, pages 4560–4565.
Giovanni Campagna, Rakesh Ramesh, Silei Xu,Michael Fischer, and Monica S Lam. 2017. Al-mond: The architecture of an open, crowdsourced,privacy-preserving, programmable virtual assistant.In Proceedings of the 26th International Conferenceon World Wide Web, pages 341–350. InternationalWorld Wide Web Conferences Steering Committee.
Shobhit Chaurasia and Raymond J Mooney. 2017. Di-alog for language to code. In Proceedings of theEighth International Joint Conference on NaturalLanguage Processing (Volume 2: Short Papers),pages 175–180.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186.
Li Dong and Mirella Lapata. 2016. Language to logi-cal form with neural attention. In Proceedings of the54th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), vol-ume 1, pages 33–43.
Li Dong, Chris Quirk, and Mirella Lapata. 2018. Con-fidence modeling for neural semantic parsing. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), volume 1, pages 743–753.
Catherine Finegan-Dollak, Jonathan K Kummerfeld,Li Zhang, Karthik Ramanathan, Sesh Sadasivam,
Rui Zhang, and Dragomir Radev. 2018. Improvingtext-to-SQL evaluation methodology. In Proceed-ings of the 56th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 351–360.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout asa Bayesian approximation: Representing model un-certainty in deep learning. In international confer-ence on machine learning, pages 1050–1059.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein-berger. 2017. On calibration of modern neural net-works. In Proceedings of the 34th InternationalConference on Machine Learning-Volume 70, pages1321–1330. JMLR. org.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao,Jian-Guang Lou, Ting Liu, and Dongmei Zhang.2019. Towards complex text-to-SQL in cross-domain database with intermediate representation.In Proceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics, pages4524–4535.
Izzeddin Gur, Semih Yavuz, Yu Su, and Xifeng Yan.2018. DialSQL: Dialogue based structured querygeneration. In Proceedings of the 56th Annual Meet-ing of the Association for Computational Linguistics(Volume 1: Long Papers), pages 1339–1349.
David Ha and Jurgen Schmidhuber. 2018. World mod-els. ArXiv preprint arXiv:1803.10122.
Luheng He, Julian Michael, Mike Lewis, and LukeZettlemoyer. 2016. Human-in-the-loop parsing. InProceedings of the 2016 Conference on EmpiricalMethods in Natural Language Processing, pages2337–2342.
Dan Hendrycks and Kevin Gimpel. 2017. A baselinefor detecting misclassified and out-of-distributionexamples in neural networks. In Proceedings ofInternational Conference on Learning Representa-tions.
Wonseok Hwang, Jinyeung Yim, Seunghyun Park, andMinjoon Seo. 2019. A comprehensive explorationon WikiSQL with table-aware word contextualiza-tion. ArXiv preprint arXiv:1902.01069.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung,Jayant Krishnamurthy, and Luke Zettlemoyer. 2017.Learning a neural semantic parser from user feed-back. In Proceedings of the 55th Annual Meeting ofthe Association for Computational Linguistics (Vol-ume 1: Long Papers), pages 963–973.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, andLuke Zettlemoyer. 2016. Summarizing source codeusing a neural attention model. In Proceedings ofthe 54th Annual Meeting of the Association for Com-putational Linguistics (Volume 1: Long Papers),volume 1, pages 2073–2083.
Georgia Koutrika, Alkis Simitsis, and Yannis E Ioan-nidis. 2010. Explaining structured queries in naturallanguage. In 2010 IEEE 26th International Confer-ence on Data Engineering (ICDE 2010), pages 333–344. IEEE.
Igor Labutov, Bishan Yang, and Tom Mitchell. 2018.Learning to learn semantic parsers from natural lan-guage supervision. In Proceedings of the 2018 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 1676–1690.
Fei Li and HV Jagadish. 2014. Constructing an in-teractive natural language interface for relationaldatabases. Proceedings of the VLDB Endowment,8(1):73–84.
Axel-Cyrille Ngonga Ngomo, Lorenz Buhmann,Christina Unger, Jens Lehmann, and Daniel Ger-ber. 2013. Sorry, I don’t speak SPARQL: translatingSPARQL queries into natural language. In Proceed-ings of the 22nd international conference on WorldWide Web, pages 977–988. ACM.
Aishwarya Padmakumar, Jesse Thomason, and Ray-mond J Mooney. 2017. Integrated learning of dialogstrategies and semantic parsing. In Proceedings ofthe 15th Conference of the European Chapter of theAssociation for Computational Linguistics: Volume1, Long Papers, pages 547–557.
Chris Quirk, Raymond Mooney, and Michel Galley.2015. Language to code: Learning semantic parsersfor If-This-Then-That recipes. In Proceedings of the53rd Annual Meeting of the Association for Compu-tational Linguistics and the 7th International JointConference on Natural Language Processing (Vol-ume 1: Long Papers), volume 1, pages 878–888.
Stuart Russell and Peter Norvig. 2009. Artificial intel-ligence: A modern approach.
Aditya Siddhant and Zachary C Lipton. 2018. DeepBayesian active learning for natural language pro-cessing: Results of a large-scale empirical study.In Proceedings of the 2018 Conference on Empiri-cal Methods in Natural Language Processing, pages2904–2909.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,Ilya Sutskever, and Ruslan Salakhutdinov. 2014.Dropout: a simple way to prevent neural networksfrom overfitting. The Journal of Machine LearningResearch, 15(1):1929–1958.
Yu Su, Ahmed Hassan Awadallah, Madian Khabsa,Patrick Pantel, Michael Gamon, and Mark Encar-nacion. 2017. Building natural language interfacesto Web APIs. In Proceedings of the 2017 ACMon Conference on Information and Knowledge Man-agement, pages 177–186. ACM.
Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, andRyen W White. 2018. Natural language interfaceswith fine-grained user interaction: A case study onweb APIs. In Proceedings of the International ACM
SIGIR Conference on Research and Development inInformation Retrieval.
Jesse Thomason, Shiqi Zhang, Raymond J Mooney,and Peter Stone. 2015. Learning to interpret nat-ural language commands through human-robot dia-log. In Twenty-Fourth International Joint Confer-ence on Artificial Intelligence.
Chenglong Wang, Marc Brockschmidt, and RishabhSingh. 2018. Pointing out SQL queries from text.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015.Building a semantic parser overnight. In Proceed-ings of the 53rd Annual Meeting of the Associationfor Computational Linguistics and the 7th Interna-tional Joint Conference on Natural Language Pro-cessing (Volume 1: Long Papers), volume 1, pages1332–1342.
Yijun Xiao and William Yang Wang. 2019. Quanti-fying uncertainties in natural language processingtasks. In Proceedings of the AAAI Conference onArtificial Intelligence, volume 33, pages 7322–7329.
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng,and Vadim Sheinin. 2018. SQL-to-text generationwith graph-to-sequence model. In Proceedings ofthe 2018 Conference on Empirical Methods in Nat-ural Language Processing, pages 931–936.
Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQL-Net: Generating structured queries from naturallanguage without reinforcement learning. ArXivpreprint arXiv:1711.04436.
Ziyu Yao, Xiujun Li, Jianfeng Gao, Brian Sadler, andHuan Sun. 2019. Interactive semantic parsing for if-then recipes via hierarchical reinforcement learning.In Proceedings of the AAAI Conference on ArtificialIntelligence, volume 33, pages 2547–2554.
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, andDragomir Radev. 2018a. TypeSQL: Knowledge-based type-aware neural text-to-SQL generation. InProceedings of the 2018 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,Volume 2 (Short Papers), pages 588–594.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang,Dongxu Wang, Zifan Li, and Dragomir Radev.2018b. SyntaxSQLNet: Syntax tree networks forcomplex and cross-domain text-to-SQL task. InProceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing, pages1653–1663.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga,Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn-ing Yao, Shanelle Roman, et al. 2018c. Spider:A large-scale human-labeled dataset for complexand cross-domain semantic parsing and text-to-SQLtask. In Proceedings of the 2018 Conference on Em-pirical Methods in Natural Language Processing,pages 3911–3921.
Victor Zhong, Caiming Xiong, and Richard Socher.2017. Seq2SQL: Generating structured queriesfrom natural language using reinforcement learning.ArXiv preprint arXiv:1709.00103.
A Extension to Complex SQL
Table 8 shows the extended lexicon entries andgrammar rules in NLG for applying our MISP-SQL agent to generate more complex SQLqueries, such as those on Spider (Yu et al., 2018c).In this dataset, a SQL query can associate withmultiple tables. Therefore, we name a column bycombining the column name with its table name(i.e., “col” in table “tab”→ COL[col (table tab)]).For simplicity, we omit “(table tab)” when refer-ring to a column col in the grammar.
B Simulation Evaluation Results
The complete simulation experiment resultsof MISP-SQL agents (based on SQLNet andSQLova) are shown in Table 6 & 7.
C Error Detector Comparison
As a supplementary experiment to Figure 4, in thissection, we show the performance of different er-ror detectors under the same average number ofquestions (“target budget”). Specifically, for eachbase semantic parser and each kind of error de-tector, we tune its decision threshold (i.e., p∗ ands∗) such that the resulting average number of ques-tions (“actual budget”) is as close to the target aspossible. In practice, we relax the actual budget tobe within ±0.015 of the target budget, which em-pirically leads to merely negligible variance. Theresults are shown in Table 9-10 for SQLNet andTable 11-12 for SQLova.
Table 7: Simulation evaluation of MISP-SQL (basedon SQLova) on WikiSQL Test set.
[Lexicon]is greater than (or equivalent to)|equals to|is less than (or equivalent to)|does not equal to → OP[>(=)|=|<(=)|! =]
is IN|is NOT IN|follows a pattern like|is between → OP[in|not in|like|between]sum of values in|average value in|number of|minimum value in|maximum value in → AGG[sum|avg|count|min|max]
in descending order (and limited to top N)|in ascending order (and limited to top N) → ORDER[desc(limit N)|asc(limit N)]
[Grammar](R1) “col” in table “tab” → COL[col (table tab)](R2) Does the system need to return information about COL[col] ? → Q[col‖SELECT agg? col](R3) Does the system need to return AGG[agg] COL[col] ? → Q[agg‖SELECT agg col](R4) Does the system need to return a value after any mathematical calculations on COL[col] ? → Q[agg=None‖SELECT agg col](R5) Does the system need to consider any conditions about COL[col] ? → Q[col‖WHERE col op val](R6) The system considers the following condition: COL[col] OP[op] a given literal value. Is this condition correct? →
Q[terminal‖WHERE col op terminal](R7) The system considers the following condition: COL[col] OP[op] a value to be calculated. Is this condition correct? →
Q[root‖WHERE col op root](R8) Do the conditions about COL[coli] and COL[colj] hold at the same time? → Q[AND‖WHERE coli .. AND colj ..](R9) Do the conditions about COL[coli] and COL[colj] hold alternatively? → Q[OR‖WHERE coli .. OR colj ..](R10) Does the system need to group items in table tab based on COL[col] before doing any mathematical calculations? →
Q[col‖GROUP BY col](R11) Given that the system groups items in table tabg based on COL[colg] before doing any mathematical calculations,
does the system need to consider any conditions about COL[col] ? → Q[col‖GROUP BY colg HAVING agg? col](R12) Given that the system groups items in table tabg based on COL[colg] before doing any mathematical calculations,
does the system need to consider any conditions about AGG[agg] COL[col] ? → Q[agg‖GROUP BY colg HAVING agg col](R13) Given that the system groups items in table tabg based on COL[colg] before doing any mathematical calculations, does the system need to
consider a value after any mathematical calculations on COL[col] ? → Q[agg=None‖GROUP BY colg HAVING agg col](R14) The system groups items in table tabg based on COL[colg] before doing any mathematical calculations, then considers the following
condition: COL[col] OP[op] a value. Is this condition correct? → Q[op‖GROUP BY colg HAVING agg? col op val](R15) Given that the system groups items in table tabg based on COL[colg] before doing any mathematical calculations, does it need to
consider any conditions? → Q[NONE HAVING ‖GROUP BY colg NONE HAVING](R16) Does the system need to order results based on COL[col] ? → Q[col‖ORDER BY agg? col](R17) Does the system need to order results based on AGG[agg] COL[col] ? → Q[agg‖ORDER BY agg col](R18) Does the system need to order results based on a value after any mathematical calculations on COL[col] ? →
Q[agg=None‖ORDER BY agg col](R19) Given that the system orders the results based on (AGG[agg]) COL[col], does it need to be ORDER[od] ? →
Q[od‖ORDER BY agg? col od]
Table 8: Extended lexicon and grammar for MISP-SQL NLG module to handle complex SQL on Spider.
