Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

An FSM is a model of discrete computation applicable to dialogues. In computer science, an FSM is a special case of a Turing machine that reads but does not write on the tape.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context. The act aims to get the other partner to do or understand something. This understanding then forms the context within which the other partner takes their turn.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

Dialogue is about the organization of communication as a series of turns between communication partners. The core elements of dialogue are communication turns, the communication context, and turn interpretation. Dialogue interaction includes speech-based and graphical interactions. Dialogue can be understood as computation, goal-directed action, communication, or embodied action.

Kirsh argued that we are not just passively reacting to computer-generated options. If we look at interaction at a higher level, beyond a single action, we see that users are also actively influencing their environments. Users are 'architects' of their environments, as Kirsh put it.

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option. In practice, we do not always know what the options mean or even what options are available. Kirsh argued that users need to actively explore interfaces to become aware of the available functions and how they work.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative. Unlike in the case of an FSM, the computing system can take action without a command from the user; the initiative is mixed.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

Communication repair refers to the 'work of restoring shared understanding' when conversational partners misunderstand each other.

Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

Human–machine interaction, according to Suchman, is similar to but different from human–human dialogue. It is similar in the sense that people pursue a shared understanding: They actively work to make themselves understood. It is different in the sense that the communication abilities of computers are limited, which requires humans to adapt.

FSMs are an effective way to describe how dialogue is structured; however, they are limited to memory-free dialogue.

An FSM is a model of discrete computation applicable to dialogues.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

Dialogue can be described using models of computation from computer science. Such models include finite state machines (FSMs), pushdown automata, and Petri nets.

Affordance, which we discussed in Chapter 3, refers to how well users can interpret what actions are possible with a widget. Visibility is a handy related concept in design that underlies direct manipulation interfaces.

Mapping and feedback are crucial concepts for understanding a computer's turn.

Gulf of evaluation: This gulf refers to knowing how a perceived change in the computer has moved it closer to the intended goal state.

Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer.

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation. These two concepts describe inferential breakpoints for users seeking to express their intentions and interpret feedback from the system, respectively.

A significant early theory of dialogue interaction is the seven-stage model of Norman [600]. It considers interaction as goal-directed, turn-based dialogue.

both the computer and the user may have initiative. For example, a pop-up window can be presented to confirm a risky selection. When there is a misunderstanding about the context of the dialogue, errors may happen, and the partners must recover from them.

both the computer and the human participate in establishing a shared context. The computer does not simply receive a message; it also communicates the effects of that message. Therefore, the design of feedback, affordances, and cues is central to dialogue-based interaction.

in dialogue, acts of communication are conditional: The meaning of a turn depends on the communication context.

the concepts of dialogue are applicable across modalities.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context.

Dialogue can be understood as computation, goal-directed action, communication, or embodied action. Each perspective provides specific methods for the analysis and design of dialogue.

Dialogue interaction includes speech-based and graphical interactions.

The core elements of dialogue are communication turns, the communication context, and turn interpretation.

Dialogue is about the organization of communication as a series of turns between communication partners.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context. The act aims to get the other partner to do or understand something. This understanding then forms the context within which the other partner takes their turn.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

The theory of task–technology fit (TTF) can illuminate what users consider useful and how this affects their decision to adopt a particular technology. TTF refers to the ability of technology to support a task [197]. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

Users actively repurpose tools to make them more personally usable and relevant. Design should support such repurposing. For example, Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use.

One prominent definition of accessibility is given by ISO 9241-171, which defines it as 'the usability of a product, service, environment or facility by people with the widest range of capabilities.'

The second dimension, social acceptability, concerns whether interactions map well to the social norms and roles in the settings where they occur.

Acceptability has two main dimensions [591]. The first dimension, practical acceptability, includes costs, the reliability of the interactive system, and its compatibility with other systems. The perceptions of utility and usability may also influence the judgment of practical acceptability.

usability is multidimensional. This means that in most settings, a valid characterization of usability will need to employ several dimensions and measures.

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

ISO 9241-11 definition... defines usability as the 'extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.'

One shorthand way of expressing this is that utility is 'whether the functionality of a system in principle can do what is needed' [591, p. 25]. In practice, whether people can do anything concerns—among other things—usability.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user. Ideally, the user can use the tool without unnecessary effort so that the use is direct, transparent, and unnoticeable.

Usability is one of the best predictors of users' willingness to adopt software. For example, the User Burden Scale is a questionnaire for measuring the felt burden in software use [806]. It consists of six subscales: difficulty of use, physical burden, time and social burden, mental and emotional burden, privacy burden, and financial burden.

However, self-attention alone is permutation-invariant, i.e., if we reorder the rows of X, then the mechanism has no built-in sense of which token came first. Since word order matters, we must inject positional information. We often add a position vector pt to the token embedding: h(0)_t = e(xt) + pt One classical choice for the positional encoding is called the sinusoidal positional encoding. pt[2k] = sin(t / 10000^{2k/d}), pt[2k+1] = cos(t / 10000^{2k/d}) The sinusoidal features give each position a distinct geometric signature across many frequencies. Nearby positions have related encodings while distant positions remain distinguishable. This lets the network reason about relative offsets.

For example, in control-theoretical analysis, we see the interaction as dynamically changing states.

For example, in interaction-as-rationality, courses of action emerge as a joint function of the user's goals and capabilities and the properties of the environment.

For example, statistical models, such as Fitts' law, describe a relationship that is considered a statistical determination.

For example, interaction-as-tool-use focuses on this idea (Chapter 19). Tools change people and their activities; in turn, this changes the tools.

An interactive AI system can possess an objective function it pursues when acting, for example, when correcting a character it flags as a typo.

In the case of particle physics, the propositions that make up a theory may concern the nature and behavior of particles.

HCI theories contain statements that link humans and technology and possibly some outcomes (e.g., poor usability, high user experience).

Text entry can also be seen as a task where different subtasks are shared between the human and the computer (Chapter 20).

One example is autocorrect, which automatically corrects typing errors while the user is typing. Another example is the use of word predictions, which allow the user to select a word from a set of word suggestions instead of typing out the word in full.

For example, text entry methods such as eye typing are designed to allow nonspeaking users with motor disabilities to enter text using their eye movements only.

Text entry is also a good example of tool use (Chapter 19). A text entry method is a tool that allows the user to communicate with someone or something, typically other people or a service, using asynchronous text messages and longer documents.

A fundamental goal of interaction is to provide users with tools that allow them to achieve goals that they would otherwise not have been able to achieve.

One goal of interaction is communicating intentions, for example, issuing commands to a computer, selecting graphical elements in image processing software, or entering text.

Accessibility is an extension of usability to ensure that as many people as possible find a tool easy to operate, regardless of individual capabilities.

Usability refers to the ease with which a tool is operated.

Utility captures how well a tool supports users in achieving their goals.

A model is a formally expressed set of propositions that follows some axiomatic system, such as algebra, logic, or a programming language.

The explanatory power of a theory refers to the empirical accuracy and coverage of the explanations offered by the theory.

A latent factor is something that affects observations about interaction without being directly observable.

Finally, in quantitative determination, interaction is described as a continuous unfolding of states.

In structural determination, the end results of the interaction are jointly determined by multiple causes that make up the whole.

In statistical determination, there is a stochastic relationship between the two entities.

In mechanical determination, an antecedent determines a consequent.

Propositions characterize entities and link them to other entities, some of which are conceptual.

In teleological determination, goals or purposes determine interaction in some way.

What happens in interaction is mutually determined by the human and the computer. In other words, what happens in interaction cannot be attributed solely to the human or the computer—the two must be considered together.

"Average movement time can be predicted as linear regression to the index of difficulty."

"The difficulty of selecting a target is proportional to its distance and inversely proportional to its width (index of difficulty)."

"If the user tries to increase speed, accuracy will be compromised, and vice versa: An increase in accuracy reduces speed."

For example, they can talk about information, difficulty, working memory, and so on.

A proposition is a claim about the world.

Phenomena in interaction are emergent, that is, they are not attributable to the user or to the computer alone.

Interaction also occurs in different contexts, including work, leisure, and in-between contexts such as commuting.

Interaction is a dynamic phenomenon that unfolds over time as users and computers influence each other.

What happens in interaction cannot be attributed solely to the human or the computer—the two must be considered together.

It has been used to describe individuals, groups, and communities using computers.

Interaction is a core notion in HCI and refers to the mutual influence between people and computers.

Pressing a button takes about a hundred milliseconds; adopting an information system in a large organization can easily take months.

We have used it to discuss various applications, from a user typing on a smartphone to a team of information workers communicating via email.

For example, a photocopier can automatically sort and collate copies.

Such points about the origins of data and the processes of their collection are a key factor in civic text visualization. Indeed, a shift to emphasizing paradata can help draw attention to the representativeness of data.

On the other side of this spectrum, at the detail level, articulating nuanced information present in raw text data can enable civic leaders to peruse and sublimate critical insights.

In contrast, we could consider designing explicitly for multiple users. Doing so requires more than designing for different levels of expertise (see the following subsection for more on expertise) or designing for collaborative use, though both those things may be valuable in their own right. Rather, this dimension encourages accounting for the different types of relationalities that users may have with a system [cf. BB17].

Civic text visualizations similarly designed to foreground interpretation could help make clearer who is making these interpretive decisions, thereby highlighting the lack of neutrality and objectivity in data [DK20].

work on visualization evaluation [SP06; IZCC08; LBI*12] has emphasized the importance of close attention to the various contexts in which a visualization will be applied.

It is informative to contrast this analytic emphasis with other evolving discourses in information visualization. The prior work reviewed above illustrates a few alternative orientations, including rhetoric [HD11], feminism [DK16; DK20], ethics [Cor19], and others [DFCC13; VW08].

work in the digital humanities often explicitly emphasizes the interpretation both of texts themselves and of computational analysis thereof [Ram03; Joc13; Und14; BSM*20].

For instance, CommunityPulse provides a scaffolding for multifaceted public input analysis using visualizations [JHSM21], and MultiConVis enables multilevel exploration and analysis of threaded conversations [HC16b].

For example, CommunityPulse [JHSM21] uses common, simple visualizations and iconography, such as bar charts and emojis, to provide overviews of people's emotions towards civic agendas and ideas. Similarly, ConsiderIt [KMF*12b] uses bar charts to visualize people's stance towards ballot measures.

For instance, visual analytic systems such as MultiConVis [HC16b] use multiple connected views to enable analysts to filter and explore text data at multiple levels.

Tools such as ConsiderIt [KMF*12b] or Opinion Space [FBRG10] are designed specifically for the public. In contrast, tools such as CommunityPulse [JHSM21] or CommunityClick [JKW*21] are focused more on supporting community leaders and decision makers.

For example, MultiConVis [HC16b] makes prescriptive statements not only as to the sentimental valence of individual conversations but also as to the topics that each conversation is about. Similarly, ConsiderIt [KMF*12b] asks participants to place individual statements as either supporting or opposing a given ballot proposition.

Consider how systems such as MutiConVis [HC16b] and CommunityClick [JKW*21] provide visual representations to help the viewer understand the structure and content of conversations.

tools such as ConsiderIt [KMF*12b] and CommunityPulse [JHSM21] prominently feature specific comments from members of the public (i.e., the data).

Some tools provide both computational and visualization features. For instance, CommunityPulse provides a scaffolding for multifaceted public input analysis using visualizations [JHSM21], and MultiConVis enables multilevel exploration and analysis of threaded conversations [HC16b].

Researchers in HCI and digital civics have begun to explore methods to improve the analysis capabilities of visual analytics tools [JHSM21; MJS20b]. Although the broader community of visualization researchers acknowledges the importance of designing for varied levels of expertise [Mun14; GTS10; SNHS13], existing work on text analytics in general, as well as civic text visualizations in particular, focuses research efforts towards designing for analysts. Less effort has been put on designing and developing text visualization for non-experts—people who are not trained in or have had limited exposure to visualization and analytics.

Improving the public input process has become an important goal in the field of digital civics [MNC*19; VCL*16; OW15]. To that end, researchers and practitioners have developed a variety of systems for, e.g., sharing public opinions [FBRG10], building consensus [KMF*12a; ZNB15], summarizing public input [19], or identifying people's priorities, reflections, and hidden insights [JHSM21].

Previous work has introduced several online engagement platforms to enable the public to asynchronously provide their comments, ideas, and feedback around civic issues [19; 20b; MJN*18]. These engagement tools have used micro-tasks [MJN*18], visualizations [19], and forum-like discussions [20b] to engage disconnected and disenfranchised populations [MNC*19]. Others have proposed technologies to promote in-person engagement of reticent participants during town halls [JKW*21] and public meetings [LLS] using clicker-like devices.

Despite their central importance in the civic engagement process, members of the general public are not necessarily involved in the analysis process. Hence, they are often left out of the loop when designing civic text visualizations—their requirements, aptitudes, knowledge, etc. are not given central consideration. Integrating participatory approaches in civic text visualization could pave the way not only for more inclusive analysis but also for leveraging the general public's knowledge to gather richer insights.

social dynamics, such as shyness and tendency to avoid confrontation with dominant personalities can also hinder opinion sharing in town halls by favoring privileged individuals who are comfortable or trained to take part in contentious public discussions [27, 127].

town halls inadvertently cater to a small number of privileged individuals, and silent participants often become disengaged despite physically attending the meetings [61]. Due to the lack of inclusivity, the outcome of such meetings often tends to feel unjust and opaque for the general public [39, 54].

designing communitysourcing technologies to include marginalized opinions and amplify participation alone may not be enough to solve inequality of sharing opinions in the civic domain [26, 126]. Despite the success of previous works [25, 53, 90], technology is rarely integrated with existing manual practices and follow-ups of engagements between government officials and community members are seldom propagated to the community.

Marginalization can be broadly defined as the exclusion of a population from mainstream social, economic, cultural, or political life [58], which still stands as a barrier to inclusive participation in the civic domain [48, 94]. Researchers in HCI and CSCW have explored various communitysourcing approaches to include marginalized populations in community activities, proceedings, and designs [48, 53, 81, 93, 132].

To increase broader civic participation, researchers in HCI have proposed both online [4, 5, 7, 81, 93] and face-to-face [21, 80, 91, 125] technological interventions that use the communitysourcing approach.

Prior investigations by Bryan [29] and Gastil [56] showed a steady decline in civic participation in town halls due to the growing disconnect between local government and community members and the decline in social capital [43, 111, 113]. Despite the introduction of online methods to increase public engagement in the last decade [4, 5, 7, 37, 81, 93], government officials continue to prefer face-to-face meetings to engage the community in the decision-making process [32, 52, 94].

To reengage disconnected, reticent, or disenfranchised community members, researchers in HCI and digital civics have offered novel strategies and technological interventions to increase engagement [60, 62, 94, 107, 130].

Bryan [29] and Gastil [56] investigated the state of town halls and demonstrated a steady decline in civic participation due to the growing disconnect between local government and the community.

Traditional community consultation methods, such as town halls, public forums, and workshops are the modus operandi for public engagement [52, 94]. For fair and impartial civic decision-making, the inclusivity of community members' feedback is paramount [60, 94, 126]. However, traditional methods rarely provide opportunities for inclusive public participation [30, 87, 95].

Murphy used such systems to promote democracy and community partnerships [103]. Similarly, Boulianne et al. deployed clicker devices in contentious public discussions about climate change to gauge public opinions [25]. Bergstrom et al. used a single button device where the attendees anonymously voted (agree/disagree) on issues during the meeting. They showed that back-channel voting helped underrepresented users get more involved in the meeting [22].

As evidenced by numerous studies on statistical cognition (Kline, 2004; Beyth-Marom et al, 2008), even trained scientists have a hard time interpreting p-values, which frequently leads to misleading or incorrect conclusions.

few researchers can resist the temptation to conclude that there is no effect, a common fallacy called "accepting the null" which had frequently led to misleading or wrong scientific conclusions (Dienes, 2014, p.1).

Again, p is the probability of seeing results as extreme (or more extreme) as those actually observed if the null hypothesis were true. So p is computed under the assumption that the null hypothesis is true. Yet it is common for researchers, teachers and even textbooks to think of p as the probability of the null hypothesis being true (or equivalently, of the results being due to chance), an error called the "fallacy of the transposed conditional" (Haller and Krauss, 2002; Cohen, 1994, p.999).

Many researchers fail to appreciate that p-values are unreliable and vary widely across replications.

Providing non-misleading interpretations of figures with confidence intervals requires judgment, and no mechanical decision procedure can carry out this job better than a thoughtful investigator.

NHST as it is carried out today consists of this incoherent mix of Fisher and Neyman–Pearson methods (Gigerenzer, 2004).

p-values give a seductive illusion of certainty and truth (Cumming, 2012, Chap. 1). The sacred α = .05 criterion amplifies this illusion, since results end up being either "significant" or "non-significant".

Estimation seems much more likely to promote clear statistical thinking.

Decades spent educating researchers have had little or no influence on beliefs and practice (Schmidt and Hunter, 1997, pp.20–22).

NHST has been severely criticized for more than 50 years by end users to whom fair statistical communication matters.

This assessment raises two issues. First, it is arbitrary. If 10 of the 15 CIs included the predicted values, would the results also support the theory, or instead refute it? If one instead used 99% CIs, would positive results for 12 of the 15 predictions be enough to support the theory? This arbitrariness arises because CIs offer no principled method for generating an inference regarding the theory.

two out of three necessary conditions for testing theory are missing.

To illustrate this point Oakes posed a series of true/false questions regarding the interpretation of p-vales to seventy experienced researchers and discovered that only two had a sound understanding of the underlying concept of significance [25].

failure to check assumptions about the data required by particular tests, over-testing and using inappropriate tests

abusing statistical tests, making illogical arguments as a result of tests, deriving inappropriate conclusions from nonsignificant results, and confusing the size of p-values with effect sizes.

This approach, fiercely promoted by Fisher in the 1930's [9], has become the gold standard in many disciplines including quantitative evaluations in HCI. However, the approach is rather counter-intuitive; many researchers misinterpret the meaning of the p-value.

We found that using MINE directly gave identical performance when the task was nontrivial, but became very unstable if the target was easy to predict from the context (e.g., when predicting a single step in the future and the target overlaps with the context).

We note that better [49, 27] results have been published on these target datasets, by transfer learning from a different source task.

We also found that not all the information encoded is linearly accessible. When we used a single hidden layer instead the accuracy increases from 64.6 to 72.5, which is closer to the accuracy of the fully supervised model.

For lasertag_three_opponents_small, contrastive loss does not help nor hurt. We suspect that this is due to the task design, which does not require memory and thus yields a purely reactive policy.

Although this is a standard transfer learning benchmark, we found that models that learn better relationships in the childeren books did not necessarily perform better on the target tasks (which are very different: movie reviews etc).

We found that more advanced sentence encoders did not significantly improve the results, which may be due to the simplicity of the transfer tasks (e.g., in MPQA most datapoints consists of one or a few words), and the fact that bag-of-words models usually perform well on many NLP tasks [48].

It is important to note that the window size (maximum context size for the GRU) has a big impact on the performance, and longer segments would give better results. Our model had a maximum of 20480 timesteps to process, which is slightly longer than a second.

Interestingly, CPCs capture both speaker identity and speech contents, as demonstrated by the good accuracies attained with a simple linear classifier, which also gets close to the oracle, fully supervised networks.

Figure 6 shows that for 4 out of the 5 games performance of the agent improves significantly with the contrastive loss after training on 1 billion frames.

CPC 76.9 80.1 91.2 87.7 96.8

CPC 73.6

CPC 48.7

Despite being relatively domain agnostic, CPCs improve upon state-of-the-art by 9% absolute in top-1 accuracy, and 4% absolute in top-5 accuracy.

We also found that not all the information encoded is linearly accessible. When we used a single hidden layer instead the accuracy increases from 64.6 to 72.5, which is closer to the accuracy of the fully supervised model.

Are the following two answers to my question Q semantically equivalent?\n\nQ: ${THE_QUESTION}\nA1: ${GOLD_ANSWER}\nA2: ${PRED_ANSWER}\n\nPlease answer with a single word, either "Yes." or "No.", and explain your reasoning.

Provide your best guess for the following question, and describe how likely it is that your guess is correct as one of the following expressions: ${EXPRESSION_LIST}. Give ONLY the guess and your confidence, no other words or explanation. For example:\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\nConfidence: <description of confidence, without any extra commentary whatsoever; just a short phrase!>\n\nThe question is: ${THE_QUESTION}

Provide your ${k} best guesses and the probability that each is correct (0.0 to 1.0) for the following question. Give ONLY the guesses and probabilities, no other words or explanation. For example:\n\nG1: <first most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nP1: <the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!>

Each linguistic likelihood expression is mapped to a probability using responses from a human survey on social media with 123 respondents (Fagen-Ulmschneider, 2023). Ling. 1S-opt. uses a held out set of calibration questions and answers to compute the average accuracy for each likelihood expression, using these 'optimized' values instead.

Finally, our study is limited to short-form question-answering; future work should extend this analysis to longer-form generation settings.

While our work demonstrates a promising new approach to generating calibrated confidences through verbalization, there are limitations that could be addressed in future work. First, our experiments are focused on factual recall-oriented problems, and the extent to which our observations would hold for reasoning-heavy settings is an interesting open question.

the 1-stage and 2-stage verbalized numerical confidence prompts sometimes differ drastically in the calibration of their confidences. How can we reduce sensitivity of a model's calibration to the prompt?

Provide your best guess and the probability that it is correct (0.0 to 1.0) for the following question. Give ONLY the guess and probability, no other words or explanation. For example:\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\n Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!>\n\nThe question is: ${THE_QUESTION}

Provide your ${k} best guesses and the probability that each is correct (0.0 to 1.0) for the following question. Give ONLY the guesses and probabilities, no other words or explanation.

Provide your best guess for the following question, and describe how likely it is that your guess is correct as one of the following expressions: ${EXPRESSION_LIST}. Give ONLY the guess and your confidence, no other words or explanation.

To fit the temperature that is used to compute ECE-t and BS-t we split our total data into 5 folds. For each fold, we use it once to fit a temperature and evaluate metrics on the remaining folds. We find that fitting the temperature on 20% of the data yields relatively stable temperatures across folds.

To avoid excessive false negatives in our correctness computation as a result of exact-match evaluation, we use either GPT-4 or GPT-3.5 to evaluate whether a response is essentially equivalent to the ground truth answer.

We sample 1000 questions from the validation split of TriviaQA (rc.web.nocontext) and SciQ and all 817 questions from the validation split of TruthfulQA (generation) for our experiments.

Ling. 1S-opt. 0.056 0.051 0.088 0.927 0.028 0.052 0.172 0.828 0.082 0.105 0.212 0.632

Verb. 1S top-4 0.041 0.039 0.081 0.959 0.056 0.059 0.185 0.815 0.198 0.144 0.245 0.619

Ling. 1S-opt. 0.058 0.066 0.135 0.878 0.064 0.068 0.220 0.674 0.125 0.165 0.270 0.492

Verb. 1S top-4 0.054 0.057 0.144 0.896 0.065 0.051 0.209 0.763 0.203 0.189 0.284 0.455

Additionally, the lack of technical details available for many state-of-the-art closed RLHF-LMs may limit our ability to understand what factors enable a model to verbalize well-calibrated confidences and differences in this ability across different models.

With Llama2-70B-Chat, verbalized calibration provides improvement over conditional probabilities across some metrics, but the improvement is much less consistent compared to GPT-* and Claude-*.

The verbal calibration of the open source model Llama-2-70b-chat is generally weaker than that of closed source models but still demonstrates improvement over its conditional probabilities by some metrics, and does so most clearly on TruthfulQA.

Chain-of-thought prompting does not improve verbalized calibration

Among the methods for verbalizing probabilities directly, we observe that generating and evaluating multiple hypotheses improves calibration (see Figure 1), similarly to humans (Lord et al., 1985), and corroborating a similar finding in LMs (Kadavath et al., 2022).

Ling. 1S-opt. 0.060 0.070 0.151 0.874 0.049 0.056 0.214 0.738 0.099 0.130 0.266 0.446

the solution is not to reform p-values or to replace them with some other statistical summary or threshold, but rather to move toward a greater acceptance of uncertainty and embracing of variation.

What, then, can and should be done? I agree with the ASA statement's final paragraph, which emphasizes the importance of design, understanding, and context—and I would also add measurement to that list.

the psychology research community has been strongly questioning the value of NHST in psychology for some years now [6] and calling for a more meaningful reporting of statistical inference based on effect sizes, confidence intervals and Bayesian reasoning [9].

Similarly, if the significance level is set at 0.05, then this is the probability of the data occurring by chance when there is no experimental effect, namely one in twenty times. The more tests that are done on a particular dataset, the more likely it is that some chance variation will be extreme enough to seem like significance.

Violation of the assumptions of any statistical test can produce p values that bear little relation to the actual probabilities of outcomes and hence comparison to the significance level of 0.05 is meaningless.

for an analysis to be sound, it is necessary that in the tests performed the probabilities of outcomes are accurately reflected in the p values produced by the tests. If this is not the case, then the NHST argument form is severely weakened.

NHST is the most commonly encountered form of statistical inference and is what is usually associated with producing a null hypothesis, then testing it to give some statistic such as a t value, and then turning the statistic into a p value.

properly reported non-significant results can help future researchers to provide estimates of effect sizes and associated confidence intervals.

calling for a more meaningful reporting of statistical inference based on effect sizes, confidence intervals and Bayesian reasoning [9].

P8 said: "as I get familiar with this system [C3], I feel more skilled" to use the highlighted and grayed phrases.

It is worth noting that P3 and P8 mentioned feeling more comfortable with the more familiar visualization in C1 and C2 during their first impression of the conditions.

The inclusion of counterfactuals often resulted in a substantial increase in precision, indicating that the models were better able to correctly classify relevant instances while reducing false positives.

By visualizing these consistent pattern rules, users may be better understanding the behavior of the model through inference projection.

Mocha addresses two seemingly contradictory objectives: (1) generating labeled data that diversifies the training dataset to aid the model's learning, and (2) maintaining structural consistency across the batches of data presented to users to support their cognitive processes.

The results of our study indicate that participants spent significantly less time annotating batches of counterfactuals when they were rendered according to SAT compared to other conditions i.e., supporting the participants' selective focus on the varying phrases, rather than phrases that stay consistent.

From a cognitive perspective, the theme color aligns with the human's (theorized) structural mapping engine [27] by making relational discrepancies between the original and counterfactual examples more explicit.

Estes and Hasson [17] highlights the significant role of bringing salience to 'non-alignable' differences.

The last two prior works also combine Variation Theory (VT) and SAT together, as we did (i.e., a corollary of SAT referred to as Analogical Transfer/Learning Theory).

Estes and Hasson [17] highlights the significant role of bringing salience to "non-alignable" differences.

Estes and Hasson [17] argue that while alignable differences can be more straightforward and easier for comparison, non-alignable differences can also provide key information that might otherwise remain overlooked.

By incorporating theories such as Structural Alignment Theory and Variation Theory, it aims to support the learning of both the human and the model.

This symbiotic relationship stems from the fact that Structural Alignment Theory (SAT) enhances the salience of differences, while the way we used Variation Theory (VT) to generate contradicting examples across the boundaries of labels ensures that these differences are conceptually informative.

It states that understanding and sensemaking involve mapping the relationships between elements, especially in complex and ambiguous tasks.

Structural Alignment Theory states that humans naturally look for structural mapping between representations of objects to help them understand, compare, and infer relationships between said objects.

According to Variation Theory, learners better understand concepts by observing variations along critical features (dimensions of variation) that define that concept and, separately, observing variations along superficial features that do not define that concept—all while other features, when possible, are held constant.

Mohamed et al. (2020) put forth the idea of dismantling power assymmetries to resist data colonialism.

Couldry and Mejias (2019) propose 'data colonialism' as a new form of colonialism to make sense of the use of large amounts of data by a small group of corporate and government actors.

Taken together, these findings almost unanimously show that, on average, AI-supported writing decreases but does not eliminate writer's feelings of ownership, underscoring the need for a larger theory of AI participation in the creative process.

This can be understood through the frame of precarious work [5]; as writers feel that their work is increasingly precarious, the power differential between themselves and the organizations seeking to train LLMs grows larger.