As The Shape of Water has been showered with praise this fall — it sits at 95 percent fresh on Rotten Tomatoes and led all films with seven Golden Globe nominations — much of the focus has been on the practical magic conjured by director Guillermo del Toro and his on-set collaborators. Without paying close attention to a long block of names during the end credits, it would seem as if no digital effects work was done on the movie — which was the goal of the VFX team, no doubt, but also a major misnomer.
The Toronto-based digital effects shop Mr. X played a heavy role in the development, production, and post-production of The Shape of Water. The studio began its work nearly half a year before cameras rolled on del Toro's latest release, and had a hand in practically all aspects of its realization, from conjuring its 1960s period setting to bringing to life the Amazonian Fish Man character at the center of the film.
Del Toro, a lifelong monster movie aficionado and collector of props and art, has spent much time touting both the carefully crafted bodysuit made for the Fish Man, as well as the soulful and dedicated work of Doug Jones, the actor who wore that heavy latex-and-rubber. Yet while the veteran actor, renown for his decades of prosthetics and motion capture work, was the beating heart (and moving limbs) of the creature, Mr. X was crucial in making it human enough to work as the centerpiece of a love story.
"The suit was tight-fitting and the makeup was latex and rubber, about an inch thick," Dennis Berardi, the founder of Mr. X and visual effects supervisor on The Shape of Water, explained to SYFY WIRE in a recent chat. "That was going to present some challenges in terms of facial expressions, because whatever Doug did under that makeup wouldn't translate one-to-one to the surface of the makeup.
Much of the work that Berardi and his team executed was extremely nuanced, though that didn't make the task any more simple. Berardi detailed for SYFY WIRE the methodical approach that del Toro and Mr. X took to bringing a practical effect to life with a whole lot of digital help, and the new technology that required.
How did you go about replacing individual elements of the Fish Man's face?
Dennis Berardi: We made an exact digital duplicate of the creature. Very detailed digital version of the face and a full body version, which is a duplicate of the live action version. Guillermo told me from the beginning that he wanted the option to be able to use a digital version of the creature at any time in the movie, for any shot, including closeups. And what we ended up having to do was, as we suspected, the micro-expressions and the subtle facial movements were not coming through in the makeup. Aa brow movement, the mouth, the eyes.
Every time you see the creature, by the way, the eyes and brow area are always digital. That's the minimum work that we did on every shot. We performed the eyes and the eye movements, and the brow area. The makeup had eyes, most of the time didn't have them in the makeup because Doug couldn't see when he had the eyes in. They were this resin, plastic resin, static material that he couldn't see through. So he generally had the eyes out, and we added the eyes in digitally, and performed them.
And it wasn't just the eyes. It was the blinks, the nictitating membrane. All these kind of micro-expressions around the brow area. In some cases where the performance had to be bigger or more extreme, like where Strickland is torturing the creature with the cattle prod, and those facial expressions are very ... He's in anguish, he's in pain, he's frustrated, he's exhausted. Those were mostly complete head replacements. Where we did the face entirely digitally.
How involved was Doug in that post-production process?
We always tried to use Doug's performance as our inspiration. We captured various sort of static poses and expressions that became our inspiration for the animation. And he was a great partner. He was a great collaborator in that respect.
We had a capture system on set. And we would run Doug through, it's a tent, a system with 60 cameras in a tent. And we scanned him multiple times, both with makeup and without makeup. And when we did it without makeup we did it to capture these facial expressions, if you will. And that became the basis of our digital animation performance.
He would do the scene with Sally on set, in situation, on camera. And we usually had the eyes out. It's not a motion capture situation where you have a performer in a motion capture suit on set. No, it was Doug Jones acting as the Asset against Sally Hawkins in those scenes. And then we augmented from there, trying to retain as much of that performance as possible. Certainly, in all cases, doing the eyes. Sometimes doing the face. Sometimes doing the whole head.
So what does that mean exactly — what did you animate?
It was sort of a varied approach. On a shot-by-shot basis, we would basically triage the shot and we knew we had to add the eyes. We would go through an inventory of what else we needed to do. Okay, there's a fold in the suit here, we'll fix that fold. This is a tender expression, we should have an open brow and we should maybe let the mouth open a little bit, and maybe dilate the eyes a little bit, slow blink. And make our lists of digital performance animation tweaks that we would do. Of course, all under the guidance of Guillermo.
Was there ever an all-digital Fish Man?
It was a mix of both. At the end of the movie, where he takes her into the water and then does a deep dive and swims around her, there was no way to perform that. We had experimented with trying to get him on a wire rig, but because he had to swim, basically two laps around her, if you will, there was not a way to rig a cable system that could have him do that eloquently and powerfully and gracefully. We made the decision early to go all digital with that. And we hung Sally from a cable, and she just acted as if he was there. And then we added him in digitally. And that worked out nicely.
Then when you cut to the close-up, it's the same methodology, where Doug is really there, and we added the eyes in and the brows and facial expressions.
Are there any ways to make sure that there are no tells? Or something that you just pay specific attention to, that really is the thing that is the tell for people?
The big thing for us is that it became very obvious when we over-animated anything. And it started to look cartoony. And so this was all about subtlety. Micro-expressions. And not going over the top with big zany, big eyebrows and big cartoony facial expressions, and that kind of thing.
We looked just at normal people in close-ups, in various aspects, like in lovemaking or in torture scenes or in just daily discussion or dialogue. Or a tender moment, or an action moment. And but for big action moments where someone's jumping off a cliff, the amount of facial movement ... I mean, the face is very complicated. There are hundreds and hundreds of tiny muscles in the face. And we found that really it was these micro-movements and the dilations of the eyes and the blinks, and these sort of little movements that we all register and use to communicate with each other.
Were there any new technologies required?
There was a number of things. The first big one was sort of a custom shading and rendering algorithm that we developed to render the surface of the creature, and the skin of the creature, in a believable way. We introduced a technique called subsurface scattering, where light penetrates the surface of the skin and scatters under, subdermally.
That was new territory for us, and something that we did a lot of research on and developed our own proprietary rendering shader for it. That was new. And then the sheer volume of water work was something that we had to invent a new way of working for ourselves. Other companies are starting to do this as well, or have already done it, in some of the higher-end movies. As I said, water work is very computationally heavy. And sometimes it would take a week to turn around just a single take of a shot. After we make our creative choices on something, you know, kind of in a proxy, sort of low-res world.
We render it on various processing machines, rendering machines, some of these shots were so heavy and so long, it would take a week to render it across, you know, 10,000 CPUs. In the end, we had to beef up our rendering farm, and we had to come up with a new way to distribute these renderers across more and more workstations. And interactively, because we were concerned about not getting enough creative cycles in, to get to a level of creative approval for Guillermo, we had to come up with a way, before we did that final big render, which we did over 100 versions of the shot before we got it done.
Before we do all the final quality rendering, if you will, or the feature film quality rendering, we have to do quick interactive creative cycles for the director, so he can see what it's going to be. And Guillermo's amazing that way, because he's a great visualizer. You know, he can see the potential in something. And I would tell him, like, just concentrate on the physics, don't concentrate on the lighting.