Open Gym

Since it first became publicly available in 2023, LabGym has been downloaded more than 70,000 times. Researchers across five continents have used it to analyze behavior in animals ranging from lab mice to wild jaguars. Its origins, though, arose from a much smaller need: measuring the activities of fruit fly larvae. 

In 2019, Ye lab postdoctoral researcher Yujia Hu was studying how fruit fly larvae process harmful external stimuli to make behavioral decisions. The project required reviewing videos of the larvae and manually tabulating their activities. Hu estimates that it takes 30 to 60 minutes to analyze every 10-minute recording of larval behavior, and a typical study like this requires four to eight hours of recordings. 

When Hu submitted his findings for publication, a reviewer suggested using automated tools for additional analyses of larval behavior. 

“I tried a couple of tools, and at that time at least, they just were not useful or user-friendly,” recalls Hu, who is now a research associate at the Cleveland Clinic. 

“They required more coding knowledge, and they couldn’t classify behaviors at the level I needed.” Hu was able to publish the study without data from the automated tools, but he was left frustrated by the choice between time-consuming manual analysis or inadequate software options.

Then the COVID-19 pandemic hit. With most in-person research restricted at U-M, Hu was unable to come into the lab but still wanted to move his research forward in some way. 

“And I had this idea: Why not just develop a tool?” 

The neurobiologist taught himself to code using Python, a programming language, then collaborated with computer scientists to build LabGym. 

At its core, LabGym is an artificial intelligence software tool that is designed to “think” like a human to identify and quantify animal behavior in videos fed into the program. Researchers can upload recordings and define the precise behaviors they want to measure, all without any coding expertise. The program then uses deep learning to improve its ability to recognize and quantify those behaviors. 

“So, for example, a larva wouldn’t have to roll a certain number of degrees for the program to decide it was a roll. As the user, I could tell the program ‘these are examples of rolling,’ and the program will teach itself to analyze all the features and details in these videos to accurately detect rolling,” Hu explains. 

Training LabGym for a new scenario — to analyze a new species or detect a new behavior — can take anywhere from a few hours to a few days of hands-on time from the researcher, depending on the complexity of the data. After that one-time training, the program can comb through video footage independently, analyzing and quantifying animal behavior while freeing researchers to continue other work.

Once he could return to the lab full time, Hu began putting the program through various training trials with new fruit fly behavior scenarios. Around the same time, Carrie Ferrario, one of Ye’s neuroscience colleagues at the Medical School, was looking for a tool to accurately analyze videos from her rodent behavior research. 

Ye mentioned the program to Ferrario and Brendon Watson, another Medical School faculty member, and the three labs expanded their LabGym exercises to rats and mice. In February 2023, they published the first paper on LabGym, demonstrating that the program could efficiently detect and quantify behaviors across various animals. 

“As the project evolved, I realized that, because we had built the tool around deep learning, it didn’t have to be for just larva,” Hu recalls. “It could be trained to analyze other species, like mice, rats, monkeys or maybe humans.”

About 200 yards from Ye’s lab in the LSI, U-M School of Dentistry faculty member Joshua Emrick was encountering the same challenge that Hu had faced in 2019: the need to objectively assess animal behavior, without adequate automated systems to do so. 

Emrick is a sensory biologist and dentist who launched his lab in 2021. His team explores how nerves within the mouth, neck and head process sensory information and contribute to normal tissue function and pain perception. They were developing their first major study, examining the role of specialized neurons within the tooth, when Elizabeth Ronan joined the group as a postdoctoral researcher.

 “As the project came together, we had gotten some feedback that there was a lot of enthusiasm for this work, but also that, if we wanted to state that these neurons cause pain, we had to show it,” recalls Emrick, an assistant professor in the Department of Biologic and Materials Sciences & Prosthodontics. “And thinking about pain sensation and behavior in animal models, we wanted to take a different approach.”

In this case, Ronan knew about a promising option. She had completed her Ph.D. in a lab just one floor up from Ye in the LSI. As fellow trainees at the LSI, she and Hu had frequently discussed the development of LabGym, and she realized it may be relevant to her new project. 

Ronan contacted Hu and asked if he thought the program could be trained to detect and measure pain behavior. Together, the two groups began to define the specific postures and facial features that indicate pain, and then they started training LabGym.

The main output of this work was the discovery that these neurons don’t just sense pain, they use that information to induce jaw movements that protect the tooth from further damage. But beyond those findings, the effort also resulted in a novel approach to detecting and quantifying pain sensation in animal models. The two labs are now collaborating to further develop this approach. 

“It’s kind of an amazing illustration of both collaboration at U-M and the application of AI,” Emrick says. “The trouble with studying behavior is that it takes a really long time to evaluate, and there is still a high potential for individual bias. Here, we had ready access to LabGym, which addressed both challenges.” 

As more colleagues expressed interest in the program, Ye and his team began building more resources to support its adoption: user manuals, training videos, even an annual symposium with more than 150 participants each year. Now, what started as an effort to maximize efficiency in his own lab has helped Ye forge connections across a global network of animal behavior research powered by AI.

publication that mentioned LabGym. It was the first LabGym study to come from outside of U-M, and its authors were, at the time, completely unknown to Ye. 

The study’s senior author, animal behavior researcher Cédric Sueur, found out about LabGym through a graduate student in his lab, Théo Ardoin. Sueur’s research program at the University of Strasbourg in France aims to automate the recognition and analysis of complex behaviors in Japanese macaques using deep learning, with the dual goals of advancing fundamental primatology and developing accessible AI tools for behavioral research. After benchmarking available programs, Ardoin landed on LabGym. 

Sueur and Ardoin trained the program using only materials and tutorials the developers had posted online. Their goal was to determine whether LabGym could be applied to primates in the field, without needing advanced programming modifications. They found the software could accurately detect the behaviors they aimed to study (specifically, stone-handling behaviors of macaques in the wild) while significantly reducing the hands-on time for researchers. 

“Without AI assistance, analyzing continuous video data across individuals, seasons and years would require prohibitive human resources,” Sueur explains. “LabGym enables us to envision large-scale, long-term automated behavioral monitoring in wild primates — something that would be unrealistic using manual annotation alone.” 

Sueur’s publication led to a new international collaboration for Ye’s group, as they evolve the software to accommodate new use cases. One of the newer features enables the program to analyze individuals interacting in a group, which Ye and Sueur are using to study social roles in groups of macaques.

It’s a feature that has been particularly useful for other researchers who want to take LabGym out into the field. At the University of Aalborg in Denmark, for example, undergraduate students Laura Liv Nørgaard Larsen, Ninette Christensen and Trine Kristensen used it to track the activity of three jaguars housed together in captivity. The students and their mentor then applied that training data to videos of jaguars in the wild, to see if LabGym could adapt its training to a new setting. 

While LabGym did struggle a bit to accurately evaluate the animals in the wild, having never seen footage from that setting, the team was surprised by its ability to adjust to the new context without additional instruction.

“It was very amazing that researchers and students like us, who have not been taught anything yet about data science or machine learning, were able to pick up this program so quickly and actually get meaningful scientific results out of it,” Larsen says. 

Ye, Tomlinson and Hu hope to expand this usability enough that perhaps even non-scientists will find a use for the program. That’s where Punch comes in.

The team is now testing the program not on wellcontrolled videos recorded on stationary cameras, but on zoo visitors’ videos posted on YouTube and other social media channels — with widely varying perspectives, filming angles, lighting and orientation. 

Beyond deriving new behavioral insights from analyses of Punch’s interactions, they hope to extend the program’s adoption outside of traditional scientific settings, envisioning its use by citizen scientists, high school students or even animal enthusiasts who want to learn more about animal behavior. 

“We are optimizing its accessibility as well as its effectiveness for researchers,” Tomlinson says. “We hope to unlock this pipeline in a very real way that makes it easier for our current users to use it within research, but also apply it to new context and to new types of users.” 

“We have always thought that LabGym could have that kind of broad use because, after all, LabGym is here to democratize the use of AI for behavior analysis. And we’re really excited about that,” Ye adds. “At the end of the day, we have to ask ourselves: ‘Why are we doing this work?’ It’s because we love it, and we believe it can help the whole field of science.”