More info i
Credit: Marvel Studios/Disney

The art (and science!) of making Korg and Hulk in Thor: Ragnarok

Contributed by
Mar 6, 2018, 2:00 PM EST

The great comic book movie revolution has been built on increasingly advanced special effects, with leaps in technology over the last two decades producing otherworldly locales and physics-defying action. To audiences, the expectation of hyper-real and detail-soaked fantasy worlds is almost a given, but even with the new technology, VFX work is increasingly complicated, requiring armies of artists and technicians working around the clock and across the world.

Jake Morrison has been a key player in the VFX industry from the start of the new era of blockbuster filmmaking, having worked in senior positions on Harry Potter and the Sorcerer's Stone and Sam Raimi's first Spider-Man movie. He most recently served as the VFX supervisor on Ant-Man and director Taika Waititi's Thor: Ragnarok, the most psychedelic Marvel adventure yet. He spoke with SYFY WIRE about his career and the work he's done on all of the Marvel movies, with a focus on the evolution of VFX, what stunts they still do in real life, and the development of the Hulk. Thor: Ragnarok is now available on home video.

How'd you get your start in the VFX industry?

Jake Morrison: I started off doing the sort of live projection visuals along with a live band. So, I suppose that's the sort of the non-commercial start of it. If I say the pure artistic side, what I mean is that you couldn't charge for it. And then I was working at a 35mm slide bureau that was just making the change to digital. I learned an early version of 3D Studio or something like that. Then that led me on to Power Animator.

Commercials in London was a good break, doing some relatively high profile stuff over there. And then moved to the States and then television, and it's just sort of steady progression up into feature film as a jobbing artist, working in the companies, doing all the bits and pieces, doing look development. At some point, you end up supervising and you've got a view of a sequence, and then you start building better machines. For Lord of the Rings 2, I was at Sony, so I built the compositing pipeline for Rivendell for how we would build the backgrounds and all that stuff.

The view gets a bit broader and broader and you start working with more and more people. That’s the sort of thing that leads you to something like Ragnarok, which, by the end of it, we had 18 visual effects vendors around the planet.

That’s an insane number of different teams working on one movie.

It was absolutely bonkers. Literally, starting from Germany and we'd follow the sun all the way until we ended up somewhere in the middle of Australia, and then we'd reset the next day. I think we had about 2700 visual effects shots on the show, at 98% visual effects on Ragnarok.

If you imagine the old trick where magicians used to spin a few plates and then they'd have to keep going around and spin the plate and then it would all balance. You're doing that, but 2700 times and across the planet. So, it's fun, it's good memory test, that for a while afterward there isn't a shot in the movie that I couldn't tell you the exact note history between the director and the studio and the editor and how we got there. And then there's this great moment after you're finished where gradually it starts to go away and you start remembering people's birthdays again and stuff like that.

And with a movie that has VFX in 98% of its shots, you don’t just work in post-production.

The other side of it is if you're on the production side as a supervisor, the first half of the picture, which is the prep and the shoot, actually involves almost the opposite of visual effects, because you are only dealing with the heads of department in the other departments.

We just all sit in these big meetings with the first AD, who's put the schedule together, and then you've got the costume designer, the special effects supervisor, the prop master and the production designer. You've got the script in front of you with the director, and you're trying to tease the vision. What is it that you see from this sequence? How do you want the audience to feel? And then it becomes a matter of just looking at each other and kind of going, "Well, that [stunt] happens in just one shot. So, it's actually not worth you trying to build a thing that would do it for real, because it's just one shot. So we're better off saving the resources for something else."

Because you want to do everything for real, but you look at the stunt coordinator and you're like, "Could you do this?" And if it’s "maybe not,” then you know that you're going to be doing digi-double takeover, or motion capture some bits of it. And then when we get into post, it all becomes about the edit and filling in the blue screens and making everything look, frankly, as if it was all shot with a single camera on location, even though on something like Thor, obviously it wasn't.

How do you decide what you want to shoot for real, and what you want to animate?

As much as you would like to shoot that stuff for real, or motion capture that stuff for real, there’s a certain point where the diminishing returns on trying to do this stuff, because, frankly, the really extreme stuff on wires tends to look like it's on wires. I mean, on pretty much most superhero films I work on nowadays, there's this counterintuitive thing where I'll be talking to a stunt coordinator and they'll go, "Well, this is the bit where so and so's gotta run and jump like 40 feet in the air” or whatever. And, I'm like, "Well, let's not even try to do it with wires.”

The thing is with wires, even with the most talented wire rigging operators, there's usually a moment where you can feel the wire takes over from gravity and it's a very hard blend to do and the days of being able to do like a hundred takes are gone in motion pictures. There’s an expectation and understanding of what you can do in post.

So, I say, "We know that Chris Hemsworth's gotta jump off the palace balcony, land on the bridge, and smash everything,” and they all just look at me like, "Apple box?" And I’m like, "Yeah." Jump off a three-foot tall blue box onto the ground and just look really freaking' cool when he lands is basically all that he has to do.

Because the stuff in the air, that's the stuff that digital can do really well, but a really good solid physical landing is a hard thing. That sort of physics is something that the animators struggle with. Never forget that the audience are the best observers of human behavior that's ever been made. This is why when you try and get close to real [CGI] humans that you run into this uncanny valley where it's close but it's not quite right because we're just hard-coded to be able to actually evaluate instantly if something frankly is a threat or not a threat or real or not real.

The idea of just somebody jumping off a box and landing and then standing up, that moment where you impact... The physics and the amount of muscle control and balance and tiny neuro-physics to go into that silly little moment are incredibly complicated and it would take super-computers to work out all the math of all that stuff. Or you could just have the dude jump off a box.

So it’s the up-close, heavy impact stuff that you wind up actually doing as a stunt.

Hitting balls, heavy impacts are always tough for some coordinators; a really messy fall, a messy landing is the thing to get. With Thor or even Ant-Man, actually, to some degree because his physics, even though they're crazy, is based on a person. It's somebody in that suit.

So if you were to run and bounce off a wall, what would most useful is literally to have a stunt player or a stunt motion capture person literally run and jump and just kick off the wall and you might just use that bit, the moment where you get close to the wall, your feet make contact, your legs compress, the knees bend, the thighs tense and. There's a sponge-y moment where you absorb that momentum and then you push back from that. That's the hard stuff to do. Honestly, the flying through the air and looking cool with your fist forward, we can do that.

What about The Hulk? He’s not springy, like Spider-Man, but you still motion capture him, obviously, since Mark Ruffalo isn’t a green giant.

We use as much of Mark as we possibly can, and we did that even with Korg, our seven-foot-six rock man. Let's pretend he weighs a ton, 'cause that would be probably about right. So even though that was Taika's performance and improv, we then slowed that down, we re-tugged at the performance. He should walk more slowly and steadily than Taika does because he's seven foot six, his stride is much bigger. If Taika walks with six paces to keep up with Chris, Korg should have four, something like that. So you do re-work this stuff to make sure it fits. If Taika bent his arm, scratched his chin, Korg can't do that easily because the size of the muscles in the forearm and the fact that he's made of rock means he can't compress the arm as much as Taika's elbow could.

So at that point, you have to make a choice: you slow that down a little bit because that makes it more weighty, you back time from the moment he actually has to touch the chin, or you might also pivot the shoulder joint up, lift the elbow up further so the elbow has less compression to do there and split the difference. So it's the essence of the performance and of course it's all Taika reading the lines and providing the core of the character, but there are certain things that you have to do as mo-cap and then mo-cap plus.

Credit: Marvel Studios

Visual effects is very much art, but as you mention there's also a science to it, a math to it because it's computers. You mentioned if Taika takes six paces, then Korg takes four to keep up with Thor. Is there a Korg calculation? Is there a set of numbers or data that you kind of plug into or is it all by feel?

With Korg it's kind of by feel. He's kind of a light character in spirit, so you don't want to weigh him down too much. Take the Hulk for example. We had a system on the set very specifically for this, so we had the motion capture volume built on set so Mark could perform, but we would be able to capture that information live and also feed it back to the video village and also feed it into the camera operator's view-piece.

So for example, if you got a scene with Chris and Mark together, you don't want the camera operator panning from Chris to Mark because Mark is significantly shorter than the eight-foot-six Hulk, so we had a representation live of an eight foot six Hulk so, instead of a pan across, we do a pan and a tilt up. So if you mention that as a basis for the technique, then imagine that he's walking. Now think about exactly that math where you say "If Mark takes six paces to go across the room and walks over to Chris, how many paces does the Hulk do?"

So we actually did that time, and I honestly I forget the exact metric, but what we would do is we would actually pace it out. If you can imagine, a target or an origin point where if Thor and Hulk were to end up toe to toe, that's your end point and that's kind of your origin and so if Mark started off four feet away, actually in the viewpoint, in the camera, the operator would see that the Hulk would start, say eight feet away, and then even though the camera operator is actually, if you look at the live feed, they're actually panning onto nothing.

They're tracking a character that doesn't exist. When they land on that spot, that origin point where the Hulk's got to go toe to toe with Thor, at that point, you actually have Mark and Chris in the same frame. But the motion capture cameras are actually capturing all the data and the facing camera is capturing it all for Mark locally. You're actually free to have the camera operator, the motion picture camera, really shoot the virtual scene even though you're shooting Chris for real.

And you have to have all this calculated before you even shoot it at all?

Oh yeah, definitely. In fact, we have a team in the back who literally were doing this math. That origin point that I'm talking about, that changed per shot. So literally, you're looking at it and you go "well, what's the focus of this shot is the beginning of the shot, the end of the shot and let's frame around that stuff." Because you want to make it all feel as organic as possible, but at the same time, there's also this heightened realism to everything, where you are dealing with a character who is two tons and is eight foot six, so you want to pre-think that you have to consider that stuff even as you're capturing the actor performing live and improvising.

Hulk Thor Ragnarok

Credit: Marvel Studios/Disney

There’s a Hulk calculus.

Yeah, there totally is. There's metrics for sure and again there's hard math to the whole thing, but then because it's an artistic endeavor, you shoot it all and then you throw some of that away and then you go with the feel of what's best for the scene.

Because it really is a blend. You start out with all the technical stuff, and it is hard to do technically, but really what it comes down to, it really is a bunch of artists. We probably had 3,000 separate artists working across the planet on this picture, so you really have to trust their instincts, it's not like there's a machine that makes the "Good Shot Button." It is down to every single jobbing artist doing what I used to do every single day, really living that shot because you're working on that particular shot for like two months.

For example, the reflection in the Hulk's eye or those moments where he will literally just look away and his eyes turn up slightly, we might take Mark's performance there, but then maybe it would be funny cause we're playing out the "Hulk Philosopher" beat a little bit more, maybe just have those eyes linger up a little bit longer and then maybe let's throw a little eye light in there just so the audience will read that moment. That’s literally somebody, in this case in Vancouver, literally placing a virtual light up, just off camera there shining down into his eyes just to get that little glint in there to draw the audience's attention.

It's entirely subjective and it's a million decisions that you make all the time to try and make this stuff look real, but also make sure that the audience actually feel the way that the director wants them to feel.

You worked on the first Avengers movie and that was Mark's first time as Hulk. What were you not able to do with Hulk then that you're able to do now?

Well, I think the Hulk's an entirely different creature from how he was back then. And I don't mean from a technology point of view, I mean from a character point of view. He was more of a dumb beast in that sense, with the "Hulk Smash" beat and then the "Puny God" beat. They're great, they're phenomenally funny beats, but he definitely was a brutish animal. Now, because he has to deliver these comedic lines and actually has to carry whole scenes, frankly, there's time where Chris is basically the foil of Mark's performance, which means the Hulk has to be able to deliver dialogue, has to be able to believably amuse the audience and not ever have them thinking that they're watching a visual effect.

So I would say the character has completely evolved and physically is much more like Mark now in the sense that we were able to rebuild Hulk from scratch. And facially, I think he's a lot more fleshy in the other films. There's a lot of his thicker lips, there's a much bigger ocular ridge of the eyes. But we genuinely went back to first principles on this one and found Ryan Meinerding, who is the Artist Supreme, the Visual Effects development leader at Marvel. We found some of his early sketches, 3D sketches of Hulk that he'd done for Avengers 1 that was a path we didn't go down on that one because the edict, the creator's choice was to make him more of a beast.

We actually thought there was more of Mark in there in some of these sketches, so we pulled those back forward into Ragnarok, so I think it's really, from a story point of view, the fact that Banner has spent two years locked in the back of Hulk's head has got to have a knock-on effect, some of him has got to leak through.