Ghost Dance: Building a virtual dance partner with inertial motion capture

Ghost Dance is a research project out of Lusófona University that sets out to explore the value of human connection and how it can be recreated with a virtual dance partner. Project directors Cecilia de Lima and Rui Antunes looked to inertial motion capture to create fluid motions that real-life dancers could perform alongside. 

Challenge: The Ghost Dance team needed to build a 3D model based on dancers' movements to test the relationship between virtual and real-life performers. Creating a realistic model posed a significant challenge. 

Solution: Inertial motion capture technology allowed dancers to record an entire routine with maximum comfort and accuracy. The result was a virtual partner with smooth motions, making for a natural performance with its real-life counterpart.

Key takeaways 

1. A diverse range of motion: Even complex movements in contemporary dance can be tracked with inertial sensors
2. Efficient recording: Entire choreographed performances were recorded quickly, to work around time restrictions in the dance studio
3. Comfortable wear: Dancers could perform unobstructed thanks to the adjustable nature of Xsens wearable technology
4. Fluid results: The virtual dance partner moved smoothly, creating a high sense of realism during the performance

Studying the importance of human touch

Before beginning the project, the Ghost Dance directors had two main goals. “The first was to understand the need for physical, bodily presence between humans,” says Rui Antunes, Director. “In the new era of technology, we seem to be growing further apart, and we wanted to see whether this had an impact on our wellbeing.” 

To explore this, the team planned to use VR goggles to compare dancing with a human partner and a virtual counterpart. “We wanted to find out what happens when a human body interacts with a virtual entity,” explains Cecilia de Lima, Director. “We knew it would make for a completely different experience, but we needed to find out how and why.” 

The other part of Cecilia and Rui’s project was to see if virtual dance partners could improve via machine learning. The digital counterparts were created from the dancer’s movements, but they wanted to see if they could develop their own routines based on the information provided.

To achieve these goals, extensive planning was put into place to find the right technology.

Building a virtual dance partner 

This complex project required not only a skilled team, but a set of specialized technology to record the dancers’ performances and transform them into a digital counterpart.

Dancers use movements that aren’t typical in day-to-day life, so it was crucial to have a motion capture system to accommodate that complexity. “The Xsens system could accurately record their movements for a perfect translation into a 3D model,” says Rui. Inertial sensors contain accelerometers and gyroscopes, tracking the joint rotation and velocity of the subject. This pinpoint precision allowed for lifelike movements in the virtual dance partner, making a more natural experience for the real-life dancers.

“Comfort is essential when dancing,” explains Cecilia. “We needed the motion capture technology to be unobtrusive to get the best results.” The system consists of 17 sensors, which all strap individually onto the body, making each one adjustable to the wearer’s preferences. Having this comfort led to more natural, fluid moves to create the virtual partner. 

Machine learning for virtual dance partners

After the dancers were recorded and the virtual partners were made, Cecilia and Rui needed to see whether they could develop from the base-level data. To do this, they created machine-learning algorithms to put the models to the test.

A machine cannot make subjective decisions, so the team found a solution to objectify and quantify the dancers’ movements. “The eight Laban efforts are commonly used in theatre and dance classes to expand students’ creativity,” says Cecilia. “These are basic movements, including punching, wringing, and pressing, that a robot could easily identify.”

From this information, Cecilia and Rui hope to develop a virtual dance partner that can make its own organic moves in time with a real-life dancer. “Inertial motion capture was the perfect choice for this type of research,” continues Rui. “We had a well-built model that was based on real movement, which provided a baseline for machine learning.”

“The research project was incredibly enlightening on the importance of human connection,” concludes Cecilia. “Virtual dance partners are not currently able to replace real ones.” Small sensations, such as breaths and footsteps, are vital when dancing cohesively with a partner. At this stage of technology development, these nuances are not available digitally. 

However, that’s not to say it won’t happen in the future. Ghost Dance proved that in the future, this could be a feasible alternative to dancing with a real counterpart. It seems increasingly possible when using inertial motion capture to create the basis of the digital partner.

Using Xsens meant that the dancers could perform freely, with no need to adjust choreography. This led to an optimal experience when dancing with a virtual partner. 

Learn more about Xsens motion capture.