An “attention map” of each prediction shows the important data points considered by the models as they make that prediction.

Koo's discovery makes it possible to peek inside the black box and identify some key features that lead to the computer's decision-making process.