Log Studies collect the most natural observations of people as they use systems in whatever ways they typically do, uninfl uenced by experimenters or observers. As the amount of log data that can be collected increases, log studies include many different kinds of people, from all over the world, doing many different kinds of tasks. However, because of the way log data is gathered, much less is known about the people being observed, their intentions or goals, or the contexts in which the observed behaviors occur. Observational log studies allow researchers to form an abstract picture of behavior with an existing system, whereas experimental log stud-ies enable comparisons of two or more systems.
Benefits of log studies:
• Complement other types of lab/field studies
• Provide a portrait of uncensored behavior
• Easy to capture at scale
Disadvantages of log studies:
• Lack of demographic data
• Non-random sampling bias
• Provide info on what people are doing but not their "motivations, success or satisfaction"
• Can lack needed context (software version, what is displayed on screen, etc.)
Ways to mitigate: Collecting, Cleaning and Using Log Data section
