The key to STEMtastic robot tutors? Dialogue.

Heather Pon-Barry, Clare Boothe Luce Assistant Professor of Computer Science, has received the National Science Foundation’s CAREER award, the most esteemed award for early-career faculty.

Recipients of the CAREER award demonstrate both the potential to serve as academic role models in research and education and the ability to lead advances in the mission of their organization. 

“I was excited and proud,” said Pon-Barry, who learned in December that she had been selected for the award. “I put in a tremendous amount of time and energy in preparing my CAREER proposal, and wrote the majority of it while caring for my 3-month old baby.” 

The CAREER award brings Pon-Barry an anticipated funding total of more than $500,000 over five years. Her research focuses on improving the capacity of robots to converse and collaborate with students in STEM learning. Ultimately she hopes to develop a socially intelligent robot tutor who engages students one-on-one through interaction, conversation and questions. 

Existing research on human-robot collaborative activities has largely focused on robots that respond to stand-alone questions or commands, Pon-Barry said. “Less attention has been given to robots that sustain longer conversational interactions.” 

Pon-Barry directs the Interactive Computing Research Lab, where she and students study spoken language processing in the context of human-robot interaction. Her work is at the intersection of artificial intelligence, computational linguistics, signal processing, emotion recognition and cognitive science. But at the core is conversational dialogue — and how to enable humanoid and service robots to engage in it with human users.

“Sometimes, it’s not what you say, but how you say it. In everyday communication, people use intonation, loudness and timing to convey emphasis and emotion — a layer of meaning beyond the semantic content of the words that are spoken,” said Pon-Barry. “For example, questions are frequently signaled by rising intonation. Affective and cognitive states such as annoyance, engagement, confidence and uncertainty might be conveyed through a combination of signals.”

In her lab, Pon-Barry develops technologies for social robots that analyze the acoustics of a person’s speech to augment traditional speech recognition. The aim? To teach computers to move beyond just the words a person says and incorporate other signals. It’s also to inform the design of intelligent, adaptive human-robot interactions. 

Pon-Barry envisions many potential benefits of such an educational robot, especially as it could reach a broad population of students — including women and other groups who are underrepresented in STEM. The robot, with its ability to analyze and respond to the subtleties of spoken language, may also be an effective tutor, she said. When students are prompted to talk through their reasoning and approaches, such as to mathematics and computer science problems, it deepens their understanding. 

The CAREER grant will fund the employment of a part-time lab manager, and undergraduate students who will work as research assistants on the human-robot dialogue project. Given the interdisciplinary nature of the research — which draws upon computer science, psychology, linguistics, statistics and more — Pon-Barry noted the project is “a great fit” for the many Mount Holyoke students who pursue interdisciplinary majors. 

The grant will also support the development of a new introductory seminar on talking robots, planned for the 2021-2022 academic year. It is specifically designed for students who may not yet see themselves as technically oriented — and to combat the misperception that computer science research is a grueling, lonely, competitive discipline. 

“Students will learn about the social impact of technology as well as a technical understanding of social robots,” said Pon-Barry. “Topics will include understanding the components of spoken dialogue systems and the challenges of situated human-robot dialogue. And we’ll compare popular media images of conversational robots to actual research systems.”

Grant funds will also be used to continue building a curriculum that trains peer mentors to create inclusive learning environments, an ongoing collaboration among Pon-Barry and professors Audrey St. John and Becky Wai-Ling Packard.

Pon-Barry has excellent company at Mount Holyoke: Over the past 20 years, nine faculty have received CAREER awards. Most recently, Kerstin Nordstrom, Clare Boothe Luce Assistant Professor of Physics, received the honor in April 2019 for her work in granular flow.