- Jun 2023
Examples include press releases, short reports, and analysis plans — documents that were reported as realistic for the type of writing these professionals engaged in as part of their work.
Have in mind the genres tested.
Looking from a perspective of "how might we use such tools in UX" we're better served by looking at documents that UX generates through the lens of identifying parallels to the study's findings for business documents.
To use AI to generate drafts, we'll want to look at AI tools built into design tools UXers use to create drafts. Those tools are under development but still developing.
I want to point out that it’s disappointingly rare for UX studies to assess the quality of the work produced with the tool that’s being studied. Output is, after all, the goal of much computer use, and the quality of this output is an essential element in judging the user interface.
Agreed, much of user testing focuses on the experience of the user when using the tool and depends upon users speaking to voluntarily making the quality of output part of their evaluation of the tool.
A possible counter: UX research professionals are not subject matter experts in the work deliverables generated in using the tool and not equipped to assess the quality of those deliverables. But that doesn't address the need the author calls out. Solving this will depend on the domain in which the tool is used and connecting with SMEs that can evaluate output in that domain.
In educational technology, we measure the quality of output by measuring the growth in students' mastery against state standards as they use the tool. This is why reporting has become the key feature for EdTech products.
the estimates of how users divided their times between different stages of document generation were based on self-reported numbers
The numbers for how users divided their time may not be reliable as they're self-reported.
Still leaves me curious about the accuracy of reported brainstorming time.
the productivity and quality improvements are likely due to a switch in the business professionals’ time allocation: less time spent on cranking out initial draft text and more time spent polishing the final result.
This points to AI providing the best time savings in draft generation, which fits with the idea of having the AI generate the drafts based on the professional's queries.
For UX designers, this points to AI in a design tool being most useful when it generates drafts (sketches) that the designer then revises. Where UX deliverables don't compare easily to written deliverables is the contextual factors that influence the design, like style guides or design systems. Design too AI assistants don't yet factor those in, though it seems likely it will, if provided style guides and design systems in a format it can read.
Given a draft of sufficient quality that it doesn't require longer to revise than a draft the designer would create on their own, getting additional time to refine sounds great.
I'm not sure what to make of the reduced time to brainstorm when using AI. Without additional information, it's hard not to assume that the AI tool may be influencing the direction of brainstorming as professionals think through the queries they'll use to get the AI to generate the most useful draft possible.
We assume the AI will generate what a human collaborator might generate given the prompt.
Mistaken human assumptions that AI will generate what a human would given the same prompt are reinforced by claims by those selling AI tools that such tools "understand human language." We don't actually know that AI understands, just that it provides a result that we can interpret as understanding (with the help of our cognitive biases).
This claim to understanding is especially misleading for neural network-based AI. We don't know how neural networks think. With older Lisp based AI we could at least trace through the code to see how the AI thinks.
we can improve AI interfaces by enabling conversational interactions that can let users establish common ground/shared semantics with the AI, and that provide repair mechanisms when such shared semantics are missing.
By providing interfaces to AI tools that help us duplicate the aligning, clarifying, and iterating behaviors that we perform with human collaborators we can increase the sense that users can predict what results the AI will provide in subsequent iterations. This will remove the frustration of working with a collaborator that doesn't understand you.
Collaborating with another human is better than working with generative AI in part because conversation allows us to establish common ground, build shared semantics and engage in repair strategies when something is ambiguous.
Collaborating with humans beats collaborating with AI because we can sync up our mental models, clarify ambiguity, and iterate.
Current AI tools are limited in the methods they make available to perform these tasks.
finding effective prompts is so difficult that there are websites and forums dedicated to collecting and sharing prompts (e.g. PromptHero, Arthub.ai, Reddit/StableDiffusion). There are also marketplaces for buying and selling prompts (e.g. PromptBase). And there is a cottage industry of research papers on prompt engineering.
Natural language alone is a poor interface for creating an effective prompt. So bad that communities and businesses are surfacing to help people create effective prompts.