Jun 12, 2023
Systematic review of user experience and trust improvement evaluations on healthcare oriented explainable artificial intelligence
- Samantha Cramer1,
- Elina Gastreich-de-Llanes1,
- Jonas Bäcker1,
- Robin Manz1,
- Philip Meyer1,
- Dominik Müller1,2,
- Ludwig Christian Hinske1,
- Johannes Raffler1,
- Iñaki Soto-Rey1
- 1Digital Medicine, University Hospital Augsburg, Augsburg, Germany;
- 2IT‐Infrastructure for Translational Medical Research, University of Augsburg, Augsburg, Germany
Protocol Citation: Samantha Cramer, Elina Gastreich-de-Llanes, Jonas Bäcker, Robin Manz, Philip Meyer, Dominik Müller, Ludwig Christian Hinske, Johannes Raffler, Iñaki Soto-Rey 2023. Systematic review of user experience and trust improvement evaluations on healthcare oriented explainable artificial intelligence. protocols.io https://dx.doi.org/10.17504/protocols.io.bp2l69omzlqe/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: May 31, 2023
Last Modified: June 12, 2023
Protocol Integer ID: 82696
Keywords: Artificial Intelligence, Usability Testing, Trust, Explainable Artificial Intelligence, XAI, AI
Funders Acknowledgements:
Medical Faculty of University Augsburg
Abstract
Many different AI systems have been developed in recent years. The field of explainable AI is designed to shed light on the black box modles of the various AIs. However, the explainable AI is rarely developed in relation to the needs of the users, but is based on the technical side that the AI structure provides. Our goal is to explore the various scientific researches that have been conducted on explainable AI in terms of user satisfaction. This goal is to be achieved through a systematic review based on the PRISMA guidelines.
Expected result
Our hypothesis is that XAI is rarely evaluated based on usability/user experience but mostly based on its functionality. Furthermore, we aim to find the most used methods for usability/user experience evaluations.
Guidelines
PRISMA guidlines.
PRISMA_2020_checklist.pdf PRISMA_2020_flow_diagram_new_SRs_v1.pdf Before start
Before starting the Systematic Review we studied the PRISMA guidlines, in order to proceed and complete the Systematic Review according to these guidlines.
Background
Background
The reviews subjective is to identify the amount of evaluations done on the usability, user satisfaction, user experience and trust of XAI. The domain of interest is the medical field.
Methodology - Literature Search
Methodology - Literature Search
General search query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence") AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND ("Evaluation")
Search restrictions: publication between 01.01.2017-31.12.2022
Data extraction from the following data bases on 02.05.2023, with the following results:
PubMed:
Query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND "Evaluation"
Formats saved: PubMed
Results: 17
PubMed Central (PMC):
Query: ("XAI" OR "X-AI" OR ("Explainable" AND "AI") OR ("Explainable" AND "artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND "Evaluation"
Formats saved: MEDLINE
Results: 2996
Science Direct:
Query: ("XAI" OR "X-AI" OR ("Explainable AI") OR ("Explainable artificial intelligence")) AND ("Usability" OR "satisfaction" OR "experience" OR "trust") AND ("Evaluation")
Formats saved: RIS
Results: 1296
Google Scholar:
Website:
using the Website for a quick overview and the Query: find article... with all words: “Evaluation” ; …and one of the words: “XAI” “X AI” “Explainable AI” “Explainable artificial intelligence”
Words contained in Title.
Results: 83
Problem: articles can’t be extracted from google scholar, therefor the Literature Search for this databank was done using the following software:
Software
Harzing's Publish or Perish
NAME
Tarma Software Research Ltd
DEVELOPER
Due to Publish and Perish set up extraction of max. 1000 results possible.
Publish or Perish search:
Query in Title words: (“XAI" OR “X-AI” OR “Explainable AI” OR "Explainable artificial intelligence”) AND (“Evaluation”)
Citation records included
Result: 80 records
Formats saved: csv ; bibtex
filter out the results included in the last part of the query: AND (“Usability” OR “satisfaction” OR “experience” OR “trust”) by looking for the keywords in the .csv file
Results: 6
Extracted Literature was imported into Civati (version 6), excluding Duplicates during import process.
Software
Citavi
NAME
Swiss Academic Software GmbH
DEVELOPER
Fulltexts were collected for results imported into Citavi, using the "find Fulltext"-function of Citavi as well as manual searches in the web. The searches resulted in:
- 3869 results including fulltext
- 395 no fulltext
Methodology - Screening
Methodology - Screening
Inclusion:
- health care information systems
- scientific evaluation in terms of usability, user satisfaction, user experience or trust and XAI
- scienticfic publication (poster/abstract/article/book)
Exclusion:
- non health care publications
- non english publications
- non scientific publications
The screening process was done individually by two seperate reviewers. Individual screening results were then compared.
Disclaimer: The results are at risk of bias due to the subjective opinion of the reviewers, which we aim to avoid by reviewing the papers individually. Furthermore, risk of bias exists, due to the fact we might not find all fitting publications with our search query, even though we have developed it to the best of our knowledge.
Protocol references
Databases: