publications

2023

1. IJHCI ’23

   Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction

   Andrew Silva, Mariah Schrum, Erin Hedlund-Botti, and 2 more authors

   International Journal of Human–Computer Interaction, 2023

   Abs Bib

   Intelligent agents must be able to communicate intentions and explain their decision-making processes to build trust, foster confidence, and improve human-agent team dynamics. Recognizing this need, academia and industry are rapidly proposing new ideas, methods, and frameworks to aid in the design of more explainable AI. Yet, there remains no standardized metric or experimental protocol for benchmarking new methods, leaving researchers to rely on their own intuition or ad hoc methods for assessing new concepts. In this work, we present the first comprehensive (n = 286) user study testing a wide range of approaches for explainable machine learning, including feature importance, probability scores, decision trees, counterfactual reasoning, natural language explanations, and case-based reasoning, as well as a baseline condition with no explanations. We provide the first large-scale empirical evidence of the effects of explainability on human-agent teaming. Our results will help to guide the future of explainability research by highlighting the benefits of counterfactual explanations and the shortcomings of confidence scores for explainability. We also propose a novel questionnaire to measure explainability with human participants, inspired by relevant prior work and correlated with human-agent teaming metrics.

   @article{IJHCI2023,
     author = {Silva, Andrew and Schrum, Mariah and Hedlund-Botti, Erin and Gopalan, Nakul and Gombolay, Matthew},
     title = {Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction},
     journal = {International Journal of Human–Computer Interaction},
     volume = {39},
     number = {7},
     pages = {1390-1404},
     year = {2023},
     publisher = {Taylor & Francis},
     doi = {10.1080/10447318.2022.2101698},
     url = {https://doi.org/10.1080/10447318.2022.2101698},
     eprint = {https://doi.org/10.1080/10447318.2022.2101698}
   }

2. THRI ’23

   Concerning Trends in Likert Scale Usage in Human-Robot Interaction: Towards Improving Best Practices

   Mariah Schrum, Muyleng Ghuy, Erin Hedlund-Botti, and 3 more authors

   J. Hum.-Robot Interact., Apr 2023

   Abs Bib

   As robots become more prevalent, the importance of the field of human-robot interaction (HRI) grows accordingly. As such, we should endeavor to employ the best statistical practices in HRI research. Likert scales are commonly used metrics in HRI to measure perceptions and attitudes. Due to misinformation or honest mistakes, many HRI researchers do not adopt best practices when analyzing Likert data. We conduct a review of psychometric literature to determine the current standard for Likert scale design and analysis. Next, we conduct a survey of five years of the International Conference on Human-Robot Interaction (HRIc) (2016 through 2020) and report on incorrect statistical practices and design of Likert scales [1, 2, 3, 5, 7]. During these years, only 4 of the 144 papers applied proper statistical testing to correctly designed Likert scales. We additionally conduct a survey of best practices across several venues and provide a comparative analysis to determine how Likert practices differ across the field of Human-robot Interaction. We find that a venue’s impact score negatively correlates with number of Likert-related errors and acceptance rate, and total number of papers accepted per venue positively correlates with the number of errors. We also find statistically significant differences between venues for the frequency of misnomer and design errors. Our analysis suggests there are areas for meaningful improvement in the design and testing of Likert scales. Based on our findings, we provide guidelines and a tutorial for researchers for developing and analyzing Likert scales and associated data. We also detail a list of recommendations to improve the accuracy of conclusions drawn from Likert data.

   @article{THRI2023,
     author = {Schrum, Mariah and Ghuy, Muyleng and Hedlund-Botti, Erin and Natarajan, Manisha and Johnson, Michael and Gombolay, Matthew},
     title = {Concerning Trends in Likert Scale Usage in Human-Robot Interaction: Towards Improving Best Practices},
     year = {2023},
     issue_date = {September 2023},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     volume = {12},
     number = {3},
     url = {https://doi.org/10.1145/3572784},
     doi = {10.1145/3572784},
     journal = {J. Hum.-Robot Interact.},
     month = apr,
     articleno = {33},
     numpages = {32},
     keywords = {Likert scales, Metrics for HRI, statistical practices}
   }

3. RSS ’23

   Investigating the Impact of Experience on a User’s Ability to Perform Hierarchical Abstraction

   Nina Marie Moorman, Nakul Gopalan, Aman Singh, and 5 more authors

   In Robotics: Science and Systems, Apr 2023

   Bib

   @inproceedings{RSS2023,
     title = {Investigating the Impact of Experience on a User's Ability to Perform Hierarchical Abstraction},
     author = {Moorman, Nina Marie and Gopalan, Nakul and Singh, Aman and Hedlund-Botti, Erin and Schrum, Mariah L and Yang, Chuxuan and Seelam, Lakshmi and Gombolay, Matthew},
     booktitle = {Robotics: Science and Systems},
     year = {2023},
     url = {https://openreview.net/forum?id=kXy0HAnah0-}
   }

4. HRI ’23

   The Effect of Robot Skill Level and Communication in Rapid, Proximate Human-Robot Collaboration

   Kin Man Lee, Arjun Krishna, Zulfiqar Zaidi, and 5 more authors

   In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Mar 2023

   Abs Bib

   As high-speed, agile robots become more commonplace, these robots will have the potential to better aid and collaborate with humans. However, due to the increased agility and functionality of these robots, close collaboration with humans can create safety concerns that alter team dynamics and degrade task performance. In this work, we aim to enable the deployment of safe and trustworthy agile robots that operate in proximity with humans. We do so by 1) Proposing a novel human-robot doubles table tennis scenario to serve as a testbed for studying agile, proximate human-robot collaboration and 2) Conducting a user-study to understand how attributes of the robot (e.g., robot competency or capacity to communicate) impact team dynamics, perceived safety, and perceived trust, and how these latent factors affect human-robot collaboration (HRC) performance. We find that robot competency significantly increases perceived trust (p < .001), extending skill-to-trust assessments in prior studies to agile, proximate HRC. Furthermore, interestingly, we find that when the robot vocalizes its intention to perform a task, it results in a significant decrease in team performance (p = .037) and perceived safety of the system (p = .009).

   @inproceedings{HRI2023b,
     author = {Lee, Kin Man and Krishna, Arjun and Zaidi, Zulfiqar and Paleja, Rohan and Chen, Letian and Hedlund-Botti, Erin and Schrum, Mariah and Gombolay, Matthew},
     title = {The Effect of Robot Skill Level and Communication in Rapid, Proximate Human-Robot Collaboration},
     year = {2023},
     month = mar,
     isbn = {9781450399647},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3568162.3577002},
     doi = {10.1145/3568162.3577002},
     booktitle = {Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction},
     pages = {261–270},
     numpages = {10},
     keywords = {perceived safety, agile task, trust, collaborative robot, varied skill, voice communication},
     location = {Stockholm, Sweden},
     series = {HRI '23}
   }

5. HRI Pioneers ’23

   Investigating Learning from Demonstration in Imperfect and Real World Scenarios

   Erin Hedlund-Botti, and Matthew C. Gombolay

   In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Mar 2023

   Abs Bib

   As the world’s population is aging and there are growing shortages of caregivers, research into assistive robots is increasingly important. Due to differing needs and preferences, which may change over time, end-users will need to be able to communicate their preferences to a robot. Learning from Demonstration (LfD) is one method that enables non-expert users to program robots. While a powerful tool, prior research in LfD has made assumptions that break down in real-world scenarios. In this work, we investigate how to learn from suboptimal and heterogeneous demonstrators, how users react to failure with LfD, and the feasibility of LfD with a target population of older adults.

   @inproceedings{HRIPioneers,
     author = {Hedlund-Botti, Erin and Gombolay, Matthew C.},
     title = {Investigating Learning from Demonstration in Imperfect and Real World Scenarios},
     year = {2023},
     month = mar,
     isbn = {9781450399708},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3568294.3579980},
     doi = {10.1145/3568294.3579980},
     booktitle = {Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction},
     pages = {769–771},
     numpages = {3},
     keywords = {human-robot interaction, personalization, learning from demonstration, assistive robots, user studies},
     location = {Stockholm, Sweden},
     series = {HRI '23}
   }

6. CoRL ’22

   Reciprocal MIND MELD: Improving Learning From Demonstration via Personalized, Reciprocal Teaching

   Mariah Schrum, Erin Hedlund-Botti, and Matthew Gombolay

   In Proceedings of The 6th Conference on Robot Learning, Dec 2023

   Abs Bib PDF

   Endowing robots with the ability to learn novel tasks via demonstrations will increase the accessibility of robots for non-expert, non-roboticists. However, research has shown that humans can be poor teachers, making it difficult for robots to effectively learn from humans. If the robot could instruct humans how to provide better demonstrations, then humans might be able to effectively teach a broader range of novel, out-of-distribution tasks. In this work, we introduce Reciprocal MIND MELD, a framework in which the robot learns the way in which a demonstrator is suboptimal and utilizes this information to provide feedback to the demonstrator to improve upon their demonstrations. We additionally develop an Embedding Predictor Network which learns to predict the demonstrator’s suboptimality online without the need for optimal labels. In a series of human-subject experiments in a driving simulator domain, we demonstrate that robotic feedback can effectively improve human demonstrations in two dimensions of suboptimality (p < .001) and that robotic feedback translates into better learning outcomes for a robotic agent on novel tasks (p = .045).

   @inproceedings{CoRL22,
     title = {Reciprocal MIND MELD: Improving Learning From Demonstration via Personalized, Reciprocal Teaching},
     author = {Schrum, Mariah and Hedlund-Botti, Erin and Gombolay, Matthew},
     booktitle = {Proceedings of The 6th Conference on Robot Learning},
     pages = {956--966},
     year = {2023},
     editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff},
     volume = {205},
     series = {Proceedings of Machine Learning Research},
     month = dec,
     publisher = {PMLR},
     url = {https://proceedings.mlr.press/v205/schrum23a.html},
   }

7. HRI ’23

   Impacts of Robot Learning on User Attitude and Behavior

   Nina Moorman, Erin Hedlund-Botti, Mariah Schrum, and 2 more authors

   In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Mar 2023

   Abs Bib

   With an aging population and a growing shortage of caregivers, the need for in-home robots is increasing. However, it is intractable for robots to have all functionalities pre-programmed prior to deployment. Instead, it is more realistic for robots to engage in supplemental, on-site learning about the user’s needs and preferences. Such learning may occur in the presence of or involve the user. We investigate the impacts on end-users of in situ robot learning through a series of human-subjects experiments. We examine how different learning methods influence both in-person and remote participants’ perceptions of the robot. While we find that the degree of user involvement in the robot’s learning method impacts perceived anthropomorphism (p=.001), we find that it is the participants’ perceived success of the robot that impacts the participants’ trust in (p<.001) and perceived usability of the robot (p<.001) rather than the robot’s learning method. Therefore, when presenting robot learning, the performance of the learning method appears more important than the degree of user involvement in the learning. Furthermore, we find that the physical presence of the robot impacts perceived safety (p<.001), trust (p<.001), and usability (p<.014). Thus, for tabletop manipulation tasks, researchers should consider the impact of physical presence on experiment participants.

   @inproceedings{HRI2023,
     author = {Moorman, Nina and Hedlund-Botti, Erin and Schrum, Mariah and Natarajan, Manisha and Gombolay, Matthew C.},
     title = {Impacts of Robot Learning on User Attitude and Behavior},
     year = {2023},
     month = mar,
     isbn = {9781450399647},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3568162.3576996},
     doi = {10.1145/3568162.3576996},
     booktitle = {Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction},
     pages = {534–543},
     numpages = {10},
     keywords = {learning from demonstration, reinforcement learning, trust, robot learning, human-robot interaction, care robotics, robotics},
     location = {Stockholm, Sweden},
     series = {HRI '23}
   }

2022

1. HRI ’22

   MIND MELD: Personalized Meta-Learning for Robot-Centric Imitation Learning

   Mariah Schrum*, Erin Hedlund-Botti*, Nina Moorman, and 1 more author

   In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Mar 2022

   *co-first authors. Best Technical Paper Award

   Abs Bib

   Learning from demonstration (LfD) techniques seek to enable users without computer programming experience to teach robots novel tasks. There are generally two types of LfD: human- and robot-centric. While human-centric learning is intuitive, human centric learning suffers from performance degradation due to covariate shift. Robot-centric approaches, such as Dataset Aggregation (DAgger), address covariate shift but can struggle to learn from suboptimal human teachers. To create a more human-aware version of robot-centric LfD, we present Mutual Information-driven Meta-learning from Demonstration (MIND MELD). MIND MELD meta-learns a mapping from suboptimal and heterogeneous human feedback to optimal labels, thereby improving the learning signal for robot-centric LfD. The key to our approach is learning an informative personalized em-bedding using mutual information maximization via variational inference. The embedding then informs a mapping from human provided labels to optimal labels. We evaluate our framework in a human-subjects experiment, demonstrating that our approach improves corrective labels provided by human demonstrators. Our framework outperforms baselines in terms of ability to reach the goal (p<.001), average distance from the goal (p=.006), and various subjective ratings (p=.008).

   @inproceedings{HRI2022,
     author = {Schrum*, Mariah and Hedlund-Botti*, Erin and Moorman, Nina and Gombolay, Matthew},
     booktitle = {2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)},
     title = {MIND MELD: Personalized Meta-Learning for Robot-Centric Imitation Learning},
     year = {2022},
     volume = {},
     number = {},
     pages = {157-165},
     month = mar,
     doi = {10.1109/HRI53351.2022.9889616},
     note = {*co-first authors.  Best Technical Paper Award}
   }

2021

1. HRI ’21

   The Effects of a Robot’s Performance on Human Teachers for Learning from Demonstration Tasks

   Erin Hedlund, Michael Johnson, and Matthew Gombolay

   In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, Mar 2021

   Abs Bib

   Learning from Demonstration (LfD) algorithms seek to enable end-users to teach robots new skills through human demonstration of a task. Previous studies have analyzed how robot failure affects human trust, but not in the context of the human teaching the robot. In this paper, we investigate how human teachers react to robot failure in an LfD setting. We conduct a study in which participants teach a robot how to complete three tasks, using one of three instruction methods, while the robot is pre-programmed to either succeed or fail at the task. We find that when the robot fails, people trust the robot less (p < .001) and themselves less (p=.004) and they believe that others will trust them less (p < .001). Human teachers also have a lower impression of the robot and themselves (p < .001) and found the task more difficult when the robot fails (p < .001$). Motion capture was found to be a less difficult instruction method than teleoperation (p=.016), while kinesthetic teaching gave the teachers the lowest impression of themselves compared to teleoperation (p=.017) and motion capture (p < .001). Importantly, a mediation analysis showed that people’s trust in themselves is heavily mediated by what they think that others – including the robot – think of them (p < .001). These results provide valuable insights to improving the human-robot relationship for LfD.

   @inproceedings{HRI2021,
     author = {Hedlund, Erin and Johnson, Michael and Gombolay, Matthew},
     title = {The Effects of a Robot's Performance on Human Teachers for Learning from Demonstration Tasks},
     year = {2021},
     isbn = {9781450382892},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3434073.3444664},
     doi = {10.1145/3434073.3444664},
     booktitle = {Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction},
     pages = {207–215},
     numpages = {9},
     month = mar,
     keywords = {human-robot interaction, learning from demonstration, trust},
     location = {Boulder, CO, USA},
     series = {HRI '21},
     dimensions = {true},
   }