Once the thematic structure was finalized, the researcher gathered supporting evidence for each theme and synthesized the findings, which were reviewed by the research team to ensure agreement on the results.

Through this process, high-level themes were identified and organized using affinity diagramming.

The researcher then systematically coded this data, revisiting the data multiples times and refining the codes to ensure consistency and coherence.

First, the lead researcher collected a list of participants' behaviors, approaches, reflections on their experience, and feedback about the interface.

We conducted a qualitative analysis of user study transcripts and survey responses using a Grounded Theory approach [8].

13 indicated they would use it regularly if made available, two would use it situationally, and only one preferred traditional interfaces.

Though user-defined aspects received comparatively lower ratings (M=4.81, STD=1.80) than other features, they were still rated as useful by most participants (Figure 12).

Each participant generated at least one user-defined aspect, averaging 2.25 (STD=1.57) per participant, with heavier use during the second task (focused reading), demonstrating active engagement with this feature.

Only three out of sixteen participants reported such issues.

Two participants (P4, P14) reported that recognizing these common patterns helped them write their own abstracts for HCI venues.

participants rated the interface as very helpful for recognizing common patterns in CHI abstracts (M=5.81, STD=1.17) (Figure 11) and discovered interesting patterns in both content and writing style (P2, P4, P8, P14).

Participants also rated the interface as very helpful for discovering otherwise-missed papers (M=5.81, STD=1.22).

As reflected in Figure 11, participants highlighted how the interface scaled up the number of papers they could review (P2, P4, P9, P11, P16), finding the interface to be highly helpful for exploring a greater number (M=6.19, STD=0.91) and broader range (M=5.94, STD=1.18) of paper abstracts along a given aspect than they typically would.

Left-justified alignment received more mixed ratings (M=4.25, STD=1.84), demonstrating participants' ability to differentiate between different alignment options and preference for SMT-inspired vertical alignment.

Vertical alignment by the first element (M=5.63, STD=1.09) received slightly lower but still favorable usefulness rating.

Vertical alignment by the second element (M=6.19, STD=0.83) and chunk highlighting (M=6.06, STD=1.34) were the top two most useful features rated by participants.

Notably, since participants were required to try all alignment and highlighting options during warmup exercises, their sustained use of SMT-inspired vertical alignment and chunk highlighting features after experimenting with alternatives provided strong evidence that they found these features helpful for their tasks.

Highlighting usage showed a similar preference: 91.13% of time with all chunks highlighted, 8.25% with partial highlighting, and minimal time (0.63%) without highlights.

According to the log data, participants spent most of their reading time (66.31%) with vertical alignment on the second element in structure pairs, followed by alignment on the first element (29.19%), and left-justified alignment (5.13%).

Activity log data, which revealed how participants actually used the interface, echoed the above findings.

More than half of the participants noted that AbstractExplorer increased their reading or skimming efficiency (P1, P4, P6, P9, P10, P11, P14, P15, P16).

Participants did not find the task too hurried or rushed (M=3.50, STD=1.59), and despite the novel presentation style and feature-rich design of the interface, reported low levels of negative effect such as insecurity, discouragement, irritation, stress, and annoyance (M=2.56, STD=1.55).

According to NASA-TLX responses (Figure 10), participants perceived the reading task as mentally demanding (M=4.88, STD=1.26)—consistent with their reported effort levels (M=4.75, STD=1.18)—but they also reported comparable levels of perceived success in accomplishing the task (M=4.63, STD=1.26).

Overall, participants were positive about AbstractExplorer and its features, mentioning that they made the task easier or led to a better experience in one way or another (15 out of 16 participants), highlighting the benefit for more efficient skimming (9 participants) and comparison of abstracts (5 participants), which allowed them to read abstracts at scale (5 participants), although they also pointed out issues such as misclassified aspects and/or chunks (3 participants), readability issues (4 participants), and steeper learning curves with the user-defined feature (3 participants).

The qualitative analysis process is described in detail in Appendix H.

In this section, we present findings on how AbstractExplorer supports comparative close reading at scale by integrating quantitative survey responses and log data with qualitative analysis of transcripts and open-ended responses.

Throughout the two tasks, we also collected detailed interaction logs including counts of user-defined aspects created, duration of highlighting usage, and time allocation across the three possible alignment options.

Each study took approximately 60-75 minutes, and participants received $25 (USD) via digital payment (Zelle or Venmo) as compensation.

All studies were conducted remotely via Zoom and facilitated by the first author.

The average Need for Cognition (NCS-6) score [42] was 5.55, with a standard deviation of 0.59, indicating that participants generally had a moderate to high tendency to engage in and enjoy thinking, with relatively low variability across the group.

Participants were roughly evenly split between lower and higher familiarity when asked to rate their familiarity with CHI or other major HCI confererences, i.e., UIST or CSCW, on a 1-7 scale.

Additionally, six participants were actively preparing manuscripts for submission to CHI 2025 or a similar HCI venue.

The group had a roughly balanced mix of individuals with varying levels of experience in HCI, with eight participants having attended CHI or a similar HCI conference; three had specifically attended CHI 2024.

The majority (nine) were PhD students, five were Master's students, one was an industry researcher, and one was academic research staff.

Twelve of the 16 participants were between the ages of 25 and 34, while the remaining four were younger (18–24).

We recruited 16 participants (9 male, 7 female) from various universities across the USA through mailing lists and social media posts.

These two gaze plots show how the reading/skimming behavior of a lower NFC participant (P4) changed when role highlighting was added to the role ordering and alignment features: from more horizontal (within a sentence) to more vertical (across corresponding chunks of sentences across different abstracts).

Both gaze data and the semi-structured interviews revealed that lower NFC participants were more willing to be guided by the three features and took advantage of them consciously.

Meanwhile, higher NFC participants often skimmed from left to right, line by line, regardless of what features were available to them (e.g., P16, whose gaze plots are shown in Figure 31 in Appendix G).

These participants also often verbally described in their interviews how they appreciated the colors guiding them in where and how to skim.

Figure 9 shows a typical lower-NFC participant reading down, across documents rather than left to right within a single document's sentence when all three features are working together; this is a radically different reading pattern.

Lower NFC participants were generally guided by emergent visual patterns created by the interactions between features, especially blocks of color spanning multiple sentences created when all three features are turned on.

Using a two-tailed Mann-Whitney U Test, we found that participants who reported their lowest perceived cognitive load when all three features were enabled had significantly lower NFC than participants who reported their lowest cognitive load level when skimming with no features enabled—in the baseline interface (p=0.03).

Figure 8 shows that most lower-NFC participants reported their lowest cognitive load when all features were enabled, while most higher-NFC participants reported their lowest cognitive load when no features were enabled (baseline).

Participants with lower NFC more frequently preferred and experienced less cognitive load when skimming with all the features enabled.

Figures in Appendix G show the distribution of participants' NFC scores as a function of the conditions in which they reported the least cognitive load (Figure 29) and which they preferred most (Figure 30).

To compute a participant's NFC score, we averaged their response to the six questions, each ranging from 1 to 7, after reversing the appropriate questions.

For simplicity of analysis, we denote participants with NFC scores above the overall participants' median NFC of 5.42 (IQR = 0.583) as higher NFC, and lower NFC otherwise.

No participant reported their lowest Raw NASA-TLX score when using the without-alignment condition.

Only one participant each reported their lowest raw NASA-TLX score when working with the without-highlighting and without-ordering ablation conditions, respectively.

Specifically, 11 participants reported their lowest raw NASA-TLX scores in the all-three-features condition, and a different 11 participants reported their lowest raw NASA-TLX scores in the baseline condition.

One participant each liked the without-highlighting and without-ordering ablation conditions most, respectively.

The remaining participants were roughly evenly split between the no-features baseline (6 participants) and the without-alignment ablation condition (5 participants).

The most popular condition had all three features enabled, i.e., 11 out of 24 participants (≈50%) preferred Figure 7C, as shown in the "Preferred" columns of Table 1.

Table 1: For each condition, we counted the number of participants who preferred it most, and then show the distribution of participants' NFC scores for that group. We do the same for reported lowest cognitive load.

The study concluded with a 15-minute semi-structured interview.

After each condition, they filled out the NASA-TLX questionnaire [27].

In each condition, the participants' task was to "skim" the 80 sentences for two minutes and then verbally answer the question What do you think this collection of text is about?

As a final onboarding step, the eye tracker was calibrated to better track their eyes using its standard software.

After signing the consent form, they filled out a questionnaire to determine their Need for Cognition (NFC) levels via the NCS-6 survey [42].

The study used a balanced design: each of the three ablation conditions—where one of the three features was disabled—was assigned to an equal number of participants.

All participants experienced the baseline, i.e., all features turned off (Figure 7A), the condition with all three features enabled (Figure 7C), and one of the three feature-ablation conditions illustrated in Figure 7B.

To contrast participants' gaze patterns in each condition, we used a Tobii Pro Spark eye-tracker placed below the desktop monitor used by all subjects; Tobii Pro Lab software recorded each participant's gaze over time in each condition.

Participants viewed the same 80 sentences in each condition—often with a different subset of sentences initially visible due to ordering changes—but only had two minutes to look at them in each condition.

We collected 80 sentences from our abstracts dataset labeled by our system as "Methodology/Contribution."

Each study took 20–30 minutes and participants received $20 USD via digital payment as compensation.

Participants were fluent in English and over 18 years of age.

We recruited 24 participants (15 female, 7 male, and 2 non-binary; all undergraduate students) from Harvard University via mailing lists.

A summative study (N=16) describes how these features support users in familiarizing themselves with a corpus of paper abstracts from a single large conference with over 1000 papers.

An ablation study (N=24) validated that these features work best together.

The research team then met, discussed the analysis outcomes, and finalized themes over three sessions.

Using the codebook, the two coders analyzed six randomly selected participant data each.

The two coders then met together to cross-check, resolve coding conflicts, and consolidate the codes into a codebook across two sessions.

The inter-coder reliability between the coders was 0.88 using Krippendorff's alpha [37].

First, two members of the research team independently coded four (25% of collected data) randomly chosen participant data to generate low-level codes.

We analyzed the interview data using thematic analysis [1].

We transcribed the audio using OpenAI's Whisper automatic speech recognition system and anonymized the transcript before analysis.

Interviews were video and audio recorded.

While English was the primary language of research for all participants, three mentioned being most comfortable with languages other than English.

All of them except one had published manuscripts in conferences or journals in their respective research fields, and the remaining participant was finalizing a manuscript for submission.

Three participants had been active researchers for 1-5 years, six participants had been active for 6-10 years, and three participants had been active researchers for more than 10 years.

Two were 18–24 years old, 7 were 26–35, and 3 were 36–45.

Demographically, seven with he/him pronouns, four with she/her pronouns, and one who did not disclose.

We recruited 12 active researchers by word of mouth followed by snowball sampling [53].

Participants frequently mentioned difficulties with search, e.g., needing to try searching "at least 10 times with different keywords to find the initial set of relevant papers" (P8).

After the interviews, we analyzed the data using the process described in Appendix B

Participants were compensated with $15 USD.

Sessions, which were held on Zoom, lasted 55 minutes on average.

we conducted a formative interview study with 12 active researchers (see Appendix A for participant information).

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (n=70) and qualitative (n=30) studies with healthcare experts.

We conducted a qualitative study with 16 blind and visually impaired (BI) developers.

We conducted role-playing exercises with 24 US journalists.

We conducted a study with 32 blind SR users.

We conducted a collaborative, user-centered design study with a team of scientific researchers.

We conducted quantitative (N=79) and qualitative (N=93) studies with healthcare experts.

We conducted a qualitative study with 35 blind and visually impaired (VI) Developers.

We conducted a study with 32 blind US users.

We conducted a study with 12 blind SR users.