Canon EOS cameras are widely used in multi-camera and 3D scanning solutions for photogrammetry. But TU Dortmund University is making the process of creating lifelike 3D models much faster with its EOS-based scanner – and taking avatars into more meaningful territory than the metaverse.

“My group has been working on virtual avatars since 2015,” explains Mario Botsch, a professor in the Computer Science Department.

“We can now put a person in a scanner and generate their digital twin, which is more or less photorealistic, in a very short space of time. And then we can put the people into a virtual reality, where they can look at themselves in a virtual mirror.”

The study group has been working on two applications for the virtual avatars – one of which is focused on sports training in virtual reality, and the other on a therapy for obese people using VR.

“In both settings, people stand in front of the virtual mirror as they are motion-tracked,” Mario says. “So whenever they move, their avatar in the mirror moves in the same way – just like in a real mirror.

“For the sports training, the participants run through various exercises and then receive feedback in the virtual mirror, where areas of the body in which improvements can be made are highlighted.

“In the obesity study, which is run in collaboration with University of Würzburg, we have used the virtual mirror to scale the body weight of a person’s avatar up, so they can see how much weight they’ve lost.

“To create the virtual avatars, we need a scanner,” Mario adds, “and for the scanner, we need cameras.”

The scope

The scanner itself is more or less a standard photogrammetry setup for scanning people, Mario explains. “We use 56 cameras positioned around the subject. The cameras are triggered simultaneously over a wired connection to take a photo, and we then use photogrammetry to generate a three-dimensional point cloud.”

The whole process is remarkably quick, taking less than ten minutes from the moment someone enters the scanner to having a fully animatable virtual avatar that can be put into virtual reality.

“The download from the cameras to the computer via USB takes about 40-50 seconds,” says Stephan Wenninger, a PhD student in Mario’s study group, who has been closely involved with the scanner build.

“We download the RAW data so that we are able to apply a universal white balance and then convert to JPEG before running the images though standard photogrammetry software. This gives us a 3D point cloud of the body surface, the 3D camera position and orientation of the cameras.”

It’s at this point that the team’s expertise starts to be demonstrated, Mario notes. “Going from a scan to a static avatar is easy, but going to something that you can animate is much harder.

“We want to generate a virtual avatar from the point cloud. What many people do at the stage is connect all the points to a triangle mesh. But then this mesh, like the avatar, has to move. So you need a skeleton, and you need to know how the skin should move when the bones move. You need controllers for facial animation, which is typically added later. So in the end, it can take up to a week to build an avatar.

“Instead, we have developed a statistical model of the human body that we trained from almost 2,000 scans – and it has all the controllers that we need.

“So we can change the body shape, we can change the skeleton angles, and we do this to the degree that the model closely resembles the point cloud. And then we transfer the photos to the texture of the model. This process is very fast.”

The imaging technology requirements included:

• Consistently high image quality that satisfies the requirements for photogrammetry.
• Remote control of multiple cameras over a wired USB connection.
• Cost-effective solution for installation in a multi-camera body scanner.

The solution

Mario and his team opted for the EOS 250D and 35mm prime lenses for the scanner at TU Dortmund University. Working with stable, controlled lighting meant that there was little requirement for a professional camera with exceptional low-light performance. The number of units that were required also underlined the need for a cost-effective model such as the EOS 250D.

The team’s decision to use the Canon EOS system can be traced back to Mario’s time as a post doc at Eidgenössische Technische Hochschule Zürich (ETH).

“Disney Research Studios is a research lab of the Disney company in Zurich that has close ties with ETH. They built a dedicated face scanner for which they used Canon cameras. I trusted them to have made a reasonable choice, so we bought the same!”

The scanner build took longer than expected, admits Mario. Prior to joining TU Dortmund University in 2020, he had constructed a similar scanner at Bielefeld University. Mario led the Computer Graphics Group here, where Stephan was also a student.

“That scanner was more messy and less professional – it was really obvious that it was a research prototype,” Mario says. “So we already knew more or less what we wanted to have for the scanner at TU. But, on and off, it still took us a year to construct.”

Stephan points out there are some areas of the scanner that would benefit from having more cameras. “We could then dedicate some to focusing closely on the face, for example. When we were in Bielefeld, we had a dedicated face scanner which consisted of eight cameras. That configuration gave really detailed results. But, for now, all of our cameras are basically the same distance away from the person.”

Taking photogrammetry further

Although the sports and obesity research projects have concluded, Mario suggests that the obesity project, in particular, has a lot of potential and will be applying for funding to continue the work. “There is no truly effective treatment for obesity,“ he says. “The only thing that really works is getting surgery to shrink your stomach. If we are able to help to make that sector better, through scanning people and virtual reality, I think it would be a worthwhile contribution.”

In the meantime, the scanning technology has been opened up to a wider audience at the university. It is part of the Hybrid Learning Centre, which is located in the central library.

“Every student at TU can get themselves scanned,” Mario says. “They can then use their scan in different ways – such as 3D printing a small figure of themselves, or as an avatar in a video game.

“We also want to evaluate learning situations in VR, something that was originally motivated by experiences during the pandemic. We want to see if having the ability to teach in virtual reality might be more successful and fun than remote learning via video conference.

The Computer Science Department is looking at further integrations of Canon EOS technology. With the assistance of Canon Germany, Mario and his group have acquired an EOS R5 C and RF 5.2mm F2.8L DUAL FISHEYE lens for a digital history project being run in collaboration with Bielefeld University.

“We’re analysing how historical events can be captured and communicated to people,” Mario explains. “The project is exploring how VR can bring history to life. The idea is that if you scale up from letters to pictures, and from pictures to videos, then the next step is from a video to a VR experience. And a good compromise between the two is stereoscopic video that you can experience in VR, but which is still very easy to capture.”

Mario explains that they are at the experimental stage, but he welcomes the ability to address technical questions about Canon’s VR System with Canon Germany. “Initially we didn’t know that such an opportunity existed,” he says, “and then we came into contact with Marcel Hess from Canon. Now that we know the possibility exists, we are grateful for this easy way of exchanging ideas with Canon and asking for help.”