The key to STEMtastic robot tutors? Dialogue.

Heather Pon-Barry received the prestigious CAREER award for her work in educational — and socially intelligent — robots that collaborate with students in STEM learning.

By Emily Malloy

Heather Pon-Barry, Clare Boothe Luce Assistant Professor of Computer Science, has received the National Science Foundation’s CAREER award, the most esteemed award for early-career faculty.

Recipients of the CAREER award demonstrate both the potential to serve as academic role models in research and education and the ability to lead advances in the mission of their organization. 

“I was excited and proud,” said Pon-Barry, who learned in December that she had been selected for the award. “I put in a tremendous amount of time and energy in preparing my CAREER proposal, and wrote the majority of it while caring for my 3-month old baby.” 

The CAREER award brings Pon-Barry an anticipated funding total of more than $500,000 over five years. Her research focuses on improving the capacity of robots to converse and collaborate with students in STEM learning. Ultimately she hopes to develop a socially intelligent robot tutor who engages students one-on-one through interaction, conversation and questions. 

Existing research on human-robot collaborative activities has largely focussed on robots that respond to stand-alone questions or commands, Pon-Barry said. “Less attention has been given to robots that sustain longer conversational interactions.” 

Pon-Barry directs the Interactive Computing Research Lab, where she and students study spoken language processing in the context of human-robot interaction. Her work is at the intersection of artificial intelligence, computational linguistics, signal processing, emotion recognition and cognitive science. But at the core is conversational dialogue — and how to enable humanoid and service robots to engage in it with human users.

“Sometimes, it’s not what you say, but how you say it. In everyday communication, people use intonation, loudness and timing to convey emphasis and emotion — a layer of meaning beyond the semantic content of the words that are spoken,” said Pon-Barry. “For example, questions are frequently signaled by rising intonation. Affective and cognitive states such as annoyance, engagement, confidence and uncertainty might be conveyed through a combination of signals.”

In her lab, Pon-Barry develops technologies for social robots that analyze the acoustics of a person’s speech to augment traditional speech recognition. The aim? To teach computers to move beyond just the words a person says and incorporate other signals. It’s also to inform the design of intelligent, adaptive human-robot interactions. 

Pon-Barry envisions many potential benefits of such an educational robot, especially as it could reach a broad population of students — including women and other groups who are underrepresented in STEM. The robot, with its ability to analyze and respond to the subtleties of spoken language, may also be an effective tutor, she said. When students are prompted to talk through their reasoning and approaches, such as to mathematics and computer science problems, it deepens their understanding. 

The CAREER grant will fund the employment of a part-time lab manager, and undergraduate students who will work as research assistants on the human-robot dialogue project. Given the interdisciplinary nature of the research — which draws upon computer science, psychology, linguistics, statistics and more — Pon-Barry noted the project is “a great fit” for the many Mount Holyoke students who pursue interdisciplinary majors. 

The grant will also support the development of a new introductory seminar on talking robots, planned for the 2021-2022 academic year. It is specifically designed for students who may not yet see themselves as technically oriented — and to combat the misperception that computer science research is a grueling, lonely, competitive discipline. 

“Students will learn about the social impact of technology as well as a technical understanding of social robots,” said Pon-Barry. “Topics will include understanding the components of spoken dialogue systems and the challenges of situated human-robot dialogue. And we’ll compare popular media images of conversational robots to actual research systems.”

Grant funds will also be used to continue building a curriculum that trains peer mentors to create inclusive learning environments, an ongoing collaboration among Pon-Barry and professors Audrey St. John and Becky Wai-Ling Packard.

Pon-Barry has excellent company at Mount Holyoke: Over the past 20 years, nine faculty have received CAREER awards. Most recently, Kerstin Nordstrom, Clare Boothe Luce Assistant Professor of Physics, received the honor in April 2019 for her work in granular flow.

Related News

Darby Dyar, blond white woman in blue sweater seated at desk with three computer monitors and charts on the wall.

Life on Venus?

Mount Holyoke’s Darby Dyar talked to The Wall Street Journal about the recent discovery of phosphine gas in Venus’s atmosphere.

Photo of the field gate

Meet Mount Holyoke’s newest faculty 2020

Mount Holyoke’s newest faculty bring cross-disciplinary scholarship to the College’s thriving intellectual community.

This is a Zoom panel featuring Andre White and the other presenters and moderator for the “Beyond the PhD: Neuro Careers in Academia, Policy, and Industry” panel.

Black in Neuro Week takes on career advice

Mount Holyoke professor André White spoke on the importance of engaging in mentorship, developing networks and being a “beacon” for those who come next.

Nine faculty members have been granted tenure. Top row, l-r: Olivia Aguilar; Riché J. Daniel Barnes; Thomas Ciufo. Middle row, l-r: Naomi Darling; Nina Emery; Kerstin Nordstrom. Bottom row, l-r: Heather Pon-Barry; Jared Schwartzer; Jennifer Wallace Jacoby

Nine faculty receive tenure

The Mount Holyoke College Board of Trustees has voted to grant tenure to nine faculty members spanning disciplines from gender studies to physics. 

six adults posting on a deck overlooking a stream

Faculty Awards 2020

Mount Holyoke honored four professors for their scholarship and teaching at the annual Faculty Awards ceremony in early March, prior to the closing of campus. 

Find more stories >