Robots to the Rescue

Published by Penn Today

Authored by Erica K. Brockmeier

Photographed by Denise Henhoeffer

Earlier this semester, a group of Penn students, postdocs, and faculty travelled to an experimental mine near Pittsburgh to participate in the first round of the Subterranean (SubT) Challenge hosted by the Defense Advanced Research Projects Agency (DARPA). Led by Camillo J. Taylor, researchers partnered with Penn spinoff companies to build a team of robots to navigate and explore unknown underground environments. 

DARPA challenges are meant to inspire innovative solutions to complex problems, and the goal of SubT was to develop robots that could be sent into places too dangerous for humans, like a subway tunnel after a fire or a mine where workers are trapped. These robots could then send back reports on what it saw to a person who could remain in a safe location. 

Members of the Pennsylvania Laboratory for Underground Tunnel Operations team, PLUTO for short, had experience working with drones in challenging settings, but going underground meant that another type of platform was needed to carry heavy equipment while navigating narrow spaces. By combining Ghost Robotics legged “dog” platforms with Exyn technologies drones, PLUTO united prior areas of expertise with the benefits of a legged system. 

To traverse a mine successfully, robots need to be able to see what’s around them so they can avoid obstacles while also sharing what they see with human operators. Each dog carried a light detection and ranging device that uses laser light to create a 3D map, stereo RGB cameras to see artifacts, thermal cameras to detect heat signatures, and an onboard computer to process the data. 

PLUTO’s robots also need the ability to decide where to travel next, recognize artifacts, and relay information to other dogs and to the humans outside the mine. Programs developed by Ph.D. student Anthony Cowley were used to generate a map of the robot’s location based on images collected by the sensors, while artifacts like backpacks and phones were detected using a program developed by Ph.D. student Shreyas Shivakumar. 

Communication is particularly challenging underground since radio waves can’t travel through thick cave walls. PLUTO’s strategy, led by master’s student Fernando Cladera, was to create a “bucket brigade” system that allowed robots to share data with one another. That way, if a robot couldn’t get back to the entrance, the data it collected could still be relayed by other dogs to the base station. 

Bringing all of these abilities together requires high-level autonomy to allow the robots to plan their exploration strategies without direct human input. Ph.D. student Ian Miller led this effort and helped make sure that all the sensors, hardware, and algorithms worked together. 

PLUTO spent time at the Number 9 Coal Mine and Museum in Lansford, Pennsylvania, over the summer and at the Colorado School of Mines experimental mine earlier this year to see how their automated systems performed underground. Many months of preparation led up to the first round of the SubT challenge in August against ten other teams a test mine near Pittsburgh. 

Each team completed two mine courses and had two attempts through each course to find artifacts, ranging from backpacks, cell phones, fire extinguishers, and dummy personnel, with a one-hour time limit. None of the team members was allowed to travel inside the mine, and only Miller was allowed to interact with the robots while they were collecting data. 

While their robots did not spot as many artifacts as hoped, the PLUTO team is pleased with how the system performed in such challenging and unknown environments. Components that worked well include how the dogs detected and explored tunnels, their ability to recognize objects, and sharing data through the bucket brigade. Several detections were provided by “fallen” dogs, showing how their system was resilient even when individual robots took a stumble. 

Adarsh Kulkarni, a master’s student who also works at Ghost Robotics, says that he’s happy with the dogs’ mechanical stability and how well they performed even after multiple falls. “This was by far the hardest we’ve run the robots and the worst environments they’ve been in,” says Kulkarni. “The mine is its own beast.”

“They took a beating every single day, and they were working the next morning,” Shivakumar adds. “That’s really commendable.”

Although some of their design had been overfit to challenges specific to Number 9, including sensors designed for narrow, textured walls while SubT had wider, smooth walls, the experience showed the team first-hand how difficult it is to engineer robots destined for unknown environments and was a rare opportunity to test robots in a new setting. “It’s very different from the normal academic workflow,” says Miller. “Going from an algorithm to something that works in a place that you’ve never seen before are very different problems.”

This “last 10%” in robotics, making an automated system that is robust and reliable, is a challenge that is often solved through a combination of state-of-the-art technology and pragmatic perspectives. “Sometimes these systems are not novel per se, but what is novel is their implementation in an untested environment,” says Cladera. “The novelty is how to solve all these problems so that the robots are reliable in harsh environments.”

The team is still discussing their plans for future rounds of SubT, with the next round happening in February in an urban setting, meaning more human-made structures and shapes like sharp angles, smooth walls, and staircases. Regardless of what lies ahead, creating and sending robots into challenging real-world environments is fundamental for progress both at Penn and in the field of robotics as a whole, especially for a future where automated systems could be tasked with a wide range of challenging tasks from driving cars to searching for survivors. 

Researchers at Penn’s General Robotics, Automation, Sensing, and Perception Lab are well-placed to address these challenges, thanks in part to a culture that encourages collaboration and communication. “It’s something that we instill in everybody that walks into our facility,” says Taylor. “If you are not worried about having a project that fits only within your area of expertise, if you are willing to be broad about how you think, it enables you to do bigger things.”

The PLUTO team was supported by funding from the U.S. Defense Advanced Research Projects Agency. 

Camillo J. Taylor is a professor in the Department of Computer and Information Science and deputy director of the General Robotics, Automation, Sensing, and Perception Lab in the School of Engineering and Applied Science at the University of Pennsylvania. 

The Number 9 Coal Mine is a non-profit organization operated by the Panther Creek Valley Foundation. Zachary Petroski serves as the foundation’s president and director of the mine. 

Ghost Robotics was co-founded by Penn alumni Avik De and Gavin Kenneally. 

Read full article by Penn Today