We chose OpenAI's ChatGPT Canvas as a baseline for five reasons: (i) it is a popular, commercially available tool, hence it is likely familiar to users; (ii) it provides a document editing view, where users can select text and ask GPT to rewrite it, or chat with an AI to make global edits; (iii) it employs a similar class of model (GPT-4o); (iv) it supports similar editing features as SemanticCommit like inline text selection, conflict highlighting, and a diff view, while adding free-form editing; and (v) similar interfaces like Anthropic Artifacts tended to rewrite the specification entirely, and did not offer Canvas's "diff" view to allow for a fair comparison.

With participant consent, we recorded audio and screen-casts, and participants were encouraged to think aloud.

Four coauthors created the evals, and two coauthors manually double-checked all conflicts, a process that took several days.

We ran one pilot study with five users of our card-based interface, and a second with four users of a revised interface.

Our explorations went through substantial iterations and prompt prototyping over a period of eight months, evolving in response to two pilot studies and progressing from a card-based interface to a list of texts.

We iterated on prompts using ChainForge [5] by setting up an evaluation pipeline against our datasets, which allowed us to observe the effects of prompt changes and model choices.

To measure statistical significance, we used Mann–Whitney–Wilcoxon tests and report the p-values.

For qualitative analysis, the first author performed open coding on participant responses and audio transcripts to identify themes, which were used to interpret the qualitative results.

In the post-task surveys, we collected self-reported NASA Task Load Index (TLX) scores, Likert-scale ratings for ease of use, and responses on how well the AI helped participants identify, understand, and resolve semantic conflicts.

Each condition had a time limit of 15 minutes, after which the participant completed a post-task survey.

Before each task, participants received a tutorial on the assigned tool and were given five minutes to explore it using a test document.

Both the order of task assignment and tool assignment were counterbalanced and randomly assigned.

We conducted a controlled within-subjects study with mixed methods, comparing SemanticCommit with a baseline interface.

We run end-to-end on our four eval datasets using GPT-4o and GPT-4o-mini and report the mean ± stddev for accuracy, precision, recall, and F1 scores for the three approaches in Figure 5.

We compare our end-to-end system against two simpler methods: (i) DropAllDocs, which adds all documents to the context for conflict classification; and (ii) InkSync [56] which generates a JSON list of string-replace operations.

In order to minimize relevance assessment issues, we apply a PageRank-based relevance ranking over the KG, akin to HippoRAG [36].

We implement the back-end using a knowledge-graph (KG) RAG architecture [36] consisting of two phases: pre-processing and inference.

Through a within-subjects study with 12 participants comparing SemanticCommit to a chat-with-document baseline (OpenAI Canvas), we find differences in workflow: half of our participants adopted a workflow of impact analysis when using SemanticCommit, where they would first flag conflicts without AI revisions then resolve conflicts locally, despite having access to a global revision feature.

We implement the back-end using a knowledge-graph (KG) RAG architecture [36] consisting of two phases: pre-processing and inference.

In order to minimize relevance assessment issues, we apply a PageRank-based relevance ranking over the KG, akin to HippoRAG [36].

We compare our end-to-end system against two simpler methods: (i) DropAllDocs, which adds all documents to the context for conflict classification; and (ii) InkSync [56] which generates a JSON list of string-replace operations.

We run end-to-end on our four eval datasets using GPT-4o and GPT-4o-mini and report the mean ± stddev for accuracy, precision, recall, and F1 scores for the three approaches in Figure 5.

We conducted a controlled within-subjects study with mixed methods, comparing SemanticCommit with a baseline interface.

Both the order of task assignment and tool assignment were counterbalanced and randomly assigned.

Before each task, participants received a tutorial on the assigned tool and were given five minutes to explore it using a test document.

Each condition had a time limit of 15 minutes, after which the participant completed a post-task survey.

In the post-task surveys, we collected self-reported NASA Task Load Index (TLX) scores, Likert-scale ratings for ease of use, and responses on how well the AI helped participants identify, understand, and resolve semantic conflicts.

For qualitative analysis, the first author performed open coding on participant responses and audio transcripts to identify themes, which were used to interpret the qualitative results.

To measure statistical significance, we used Mann–Whitney–Wilcoxon tests and report the p-values.

We iterated on prompts using ChainForge [5] by setting up an evaluation pipeline against our datasets, which allowed us to observe the effects of prompt changes and model choices.

Our explorations went through substantial iterations and prompt prototyping over a period of eight months, evolving in response to two pilot studies and progressing from a card-based interface to a list of texts.

We ran one pilot study with five users of our card-based interface, and a second with four users of a revised interface.

Four coauthors created the evals, and two coauthors manually double-checked all conflicts, a process that took several days.

With participant consent, we recorded audio and screen-casts, and participants were encouraged to think aloud.

These semantic conflicts require dedicated support to detect, visualize, and resolve. Semantic conflict resolution interfaces must go beyond visualizing what changes were made, to what changes could be made, where they should be made, and what the effects might be. This resembles feedforward: affordances that help the user foresee the impact of an action [67, 93].

Today with LLMs, we are less limited by this constraint, and solutions to the problem of human-machine communication might be better found in cybernetics theory [9] than static formalism.

This reflects the principle of feedforward [67, 93] in communication theory—"a needed prescription or plan for a feedback, to which the actual feedback may or may not confirm" [79]—where a communicator provides "the context of what one was planning to talk about" [64, p. 179-80] in order to "pre-test the impact of [its output]" on the listener [34, p. 65].

We consider common sequences of chunk roles to be alignable structures that could be used to support users in identifying structural similarities and differences across sentences in different abstracts, in line with Structure-Mapping Theory [17].

Like prior Structural Mapping Theory (SMT)-informed work in text corpora representation, AbstractExplorer's features have enabled some users to see more of both the overview and the details at the same time, facilitating abstraction without losing context.

Our work demonstrates that designs informed by Structure-Mapping Theory can support users in navigating, making use of, and engaging with variation present in information.

According to SMT, this generalization depends on most documents having some shared implicit structure.

This ordering prioritizes dominant structural patterns (largest groups first) while exposing fine-grained variations (via length-sorted triplets), mirroring how humans compare sentences, if SMT is an accurate description in this domain of comparative close reading.

Structural mappings between objects are part of the cognitive process of comparison according to the Structure-Mapping Theory [17], and juxtaposition can facilitate humans in recognizing particular possible structural mappings between objects [75].

In SMT terminology, rendering and arranging according to corresponding chunks reify "commonalities in structure," while variation within corresponding chunks are "alignable differences" that users are predicted to notice.

The prior SMT-informed tools in Section 2.3 for both code and natural language corpora suggest that the cognitive process of comparing texts may be no exception to the cognitive processes SMT predicts.

SMT posits that visual alignment helps people perceive relational similarities and differences more clearly, thereby improving their ability to make meaningful comparisons and understand underlying patterns [28, 38, 47].

SMT provides a framework for understanding how humans compare two or more objects by finding common structural alignments between objects.

Structural Mapping Theory (SMT) is a long-standing well-vetted theory from Cognitive Science that describes how humans attend to and try to compare objects by finding mental representations of them that can be structurally mapped to each other (analogies).

This SMT-informed approach, which AbstractExplorer shares, tries to give this mental machinery "a leg up," letting users perhaps skip some steps by accepting reified cross-document relationships identified by the computer.

The human perceptual, comparative mental machinery that SMT describes is part of what enables humans to form more abstract structured mental models from concrete examples, among other critical knowledge tasks.

These examples of text-centric lossless techniques do not abstract away or summarize; they strategically re-organize and re-render the existing text to help enhance readers' own perceptual cognition, informed by Structural Mapping Theory (SMT) [17].

Theory (SMT) to facilitate seeing both the overview and the details at the same time, facilitating abstraction without losing context.

We process this data in a three-stage pipeline (Figure 6). In the first stage, Sentence Segmentation and Categorization, abstracts are split into individual sentences using the NLTK package, and each sentence is classified into one of the five pre-defined aspects as listed in Section 4.1.1. Classification is performed by prompting an LLM (see prompt used in Appendix D.1) with the sentence and its full abstract.

Then, we segment sentences within each aspect into grammar-preserving chunks (see prompt used in Appendix D.2). This results in grammatically coherent chunks that are the basis of structure patterns. After identifying chunk boundaries, we again prompt an LLM to generate labels for chunks in a human-in-the-loop approach: starting from an initial set of labels for chunk roles, when a new label is generated, a researcher from the research team examines the new label and merges it with existing labels if appropriate, controlling for the total number of labels.

In this study, we allowed participants to experience views of same-aspect sentences (Section 4.1.1) with different combinations of highlighting, ordering, and alignment (as described in Section 4.1.2 and Section 4.1.4) enabled or not, in order to understand which and/or what combinations most effectively supported users' ability to skim and read laterally across documents.

Inspired by GP-TSM [24], AbstractExplorer first segments sentences into grammar-preserving chunks—segments that respect grammatical boundaries, i.e., an LLM judges that the sentence can be truncated at that chunk boundary without breaking the grammatical integrity of the preceding text. Each chunk is then classified by an LLM as having one of nine pre-defined roles, each of which has its own assigned color.

AbstractExplorer classifies sentences into five pre-defined aspects common in CHI abstracts: Problem Domain, Gaps in Prior Work, Methodology/Contribution, Results/Findings, and Discussion/Conclusion.

We conducted a qualitative analysis of user study transcripts and survey responses using a Grounded Theory approach [8]. First, the lead researcher collected a list of participants' behaviors, approaches, reflections on their experience, and feedback about the interface. The researcher then systematically coded this data, revisiting the data multiples times and refining the codes to ensure consistency and coherence. Through this process, high-level themes were identified and organized using affinity diagramming. Once the thematic structure was finalized, the researcher gathered supporting evidence for each theme and synthesized the findings, which were reviewed by the research team to ensure agreement on the results.

Activity log data, which revealed how participants actually used the interface, echoed the above findings. According to the log data, participants spent most of their reading time (66.31%) with vertical alignment on the second element in structure pairs, followed by alignment on the first element (29.19%), and left-justified alignment (5.13%). Highlighting usage showed a similar preference: 91.13% of time with all chunks highlighted, 8.25% with partial highlighting, and minimal time (0.63%) without highlights.

In this section, we present findings on how AbstractExplorer supports comparative close reading at scale by integrating quantitative survey responses and log data with qualitative analysis of transcripts and open-ended responses. The qualitative analysis process is described in detail in Appendix H.

Throughout the two tasks, we also collected detailed interaction logs including counts of user-defined aspects created, duration of highlighting usage, and time allocation across the three possible alignment options.

Both gaze data and the semi-structured interviews revealed that lower NFC participants were more willing to be guided by the three features and took advantage of them consciously.

Using a two-tailed Mann-Whitney U Test, we found that participants who reported their lowest perceived cognitive load when all three features were enabled had significantly lower NFC than participants who reported their lowest cognitive load level when skimming with no features enabled—in the baseline interface (p=0.03).

The raw NASA-TLX score is the sum of all 6 NASA-TLX questions after reversing the appropriate questions.

To compute a participant's NFC score, we averaged their response to the six questions, each ranging from 1 to 7, after reversing the appropriate questions.

For simplicity of analysis, we denote participants with NFC scores above the overall participants' median NFC of 5.42 (IQR = 0.583) as higher NFC, and lower NFC otherwise.

To contrast participants' gaze patterns in each condition, we used a Tobii Pro Spark eye-tracker placed below the desktop monitor used by all subjects; Tobii Pro Lab software recorded each participant's gaze over time in each condition.

We collected 80 sentences from our abstracts dataset labeled by our system as "Methodology/Contribution." Participants viewed the same 80 sentences in each condition—often with a different subset of sentences initially visible due to ordering changes—but only had two minutes to look at them in each condition.

After obtaining an expanded set of high-level chunk labels, we assign them to each of the sentence chunks by using LLMs in a multiclass classification few-shot learning task, with the initial labels and assignment as examples (see prompt used in Appendix D.3).

Then, we segment sentences within each aspect into grammarpreserving chunks (see prompt used in Appendix D.2). This results in grammatically coherent chunks that are the basis of structure patterns. After identifying chunk boundaries, we again prompt an LLM to generate labels for chunks in a human-in-the-loop approach: starting from an initial set of labels for chunk roles, when a new label is generated, a researcher from the research team examines the new label and merges it with existing labels if appropriate, controlling for the total number of labels.

We process this data in a three-stage pipeline (Figure 6). In the first stage, Sentence Segmentation and Categorization, abstracts are split into individual sentences using the NLTK package, and each sentence is classified into one of the five pre-defined aspects as listed in Section 4.1.1. Classification is performed by prompting an LLM (see prompt used in Appendix D.1) with the sentence and its full abstract.

After the interviews, we analyzed the data using the process described in Appendix B

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.