Tutorial: Dynamically changing facial normal maps in Unreal Engine 4

Hi!

magyar verzió

We've promised more tutorials, and we've specifically promised a tutorial on dynamic facial normal maps in UE4. So here it is! Before we begin...

It took a long time to get started with these, because we were busy with other works, our own projects, and with learning a LOT about the engine. So for now, we didn't really progress with the actual project of Lemniscate, but this recently acquired knowledge will speed us up when we do have time to progress.

Now, on to the tutorial.

***

To start out, you must have a finished face model with bone-driven morph targets. You'll have to decide which morphs should activate normal map changes. For example, Sal here has normal maps for raising each individual eyebrow, for frowning, for raising the inner portion of his eyebrows, and for opening his mouth. For a human, there could be additional wrinkles on the nose, under the eyes and such. Because Sal has fur, these normal maps are pretty subtle.
We will probably use a different technology for him, since we use NeoFur now for the fur effect, so this is only for demonstration.
We are using Blender and GIMP in this tutorial, but you can use any modeling and photo editing software you like. But Blender and GIMP are free, so no excuses!

First, you'll have to remove every part of the skeletal mesh that is irrelevant. The only parts to stay should be the parts with the material that will make use of the dynamic normal map (so, in this case, the face).

Duplicate the head. Make one for every morph target you want to influence the normals, and then apply the chosen morphs to the heads, one morph to one mesh.

Duplicate all of the heads again, make a new version that removes absolutely everything other than the parts where you want the normals to change (so the neck, back of the head, ears have to go). Then apply a multires modifier and set it to about x4, so that you can make details on the face with the sculpt tool.

Don't worry about anything else other than the wrinkles, because this normal map will be added to the default normal, so all the base normal map details will still be visible. It's better if you don't make overlapping wrinkles, because all of these will be combined into a single normal map later.

After you're done with sculpting, bake normal maps from each of the sculpted faces onto the original meshes (you can find our tutorial on normal map baking with Blender here, the only thing different here is using multires). Optionally you can also bake ambient occlusion maps for more illusion of depth, which we didn't, but you can operate them the same way as the normals.

After you're done baking all of them, combine them into a single file (this is the part where you can make adjustments if you have overlapping wrinkles!)

Next, you'll have to make masks for using this additional normal map. It's best if you use the image file's channels for separating different masks. Because a mask only uses greyscale values, you can combine up to 4 of them into a single image file using the red, green, blue and alpha channels. The ideal format to use for this is the Targa format (.tga), because it keeps the channels clean and separate. You can use GIMP or Photoshop to edit the channels individually, but you can probably use any other photo-manipulation software for this as well.

I've created six different wrinkles, so I will save them as two .tga files using the r, g, b channels. Use a black background. When editing the channels, you'll have to use white to fill the spaces where the individual wrinkled spots are.

These will indicate the areas where you want to add your new wrinkle normal map to the default one. After finishing, combine the channels into colored .tga files and save.

It's time to open up Unreal 4. 

Import the new normal map and the masks, and put them inside the character's material. Arrange the mask textures below each other and add a multiply node next to each of their channels. Connect scalar parameters to the other input of the multiply nodes and name them appropriately - for example, the parameter next to the mask controlling the wrinkles above the left eyebrow will be called BrowUp_L. Name all of them, and make sure not to change their default values from 0.

Create  LinearInterpolation nodes (Lerp nodes) for all of the masks, and chain them together using their A inputs. Create a three value texture sample node (hold down 3 and click with the left mouse button) and set the third value (blue) to 1 (so, 0,0,1) or just use an empty normal map. Connect this into the first Lerp node's A input.

Connect the multiply nodes to the Lerp nodes' alpha inputs, and our new normal map into the Lerp nodes' B inputs. The last Lerp node's output should go into a BlendAngleCorrectedNormals node's AdditionalNormal input, and the original normal map should be connected to the BaseNormal input. Connect the last output to the final material input node's Normal input.

You are done with the material editor, let's get started with preparing the skeleton!

These are bone driven morph targets. Currently there isn't a node for getting the value of a morph target in Blueprints (the 'get morph target' node doesn't return any other value other than 0, correct me if I'm wrong about this). So what we have to do is create sockets for each bone that drives a morph target that we want to add dynamic normal maps to. Then you should rotate the morph targets to such a position that you can easily remember and calculate with - you should choose a rotation axis (in my case, X) that matches with the rotation of the bone that drives the morph.

So, in an ideal case, when the morph target's value is 1, the bone's local X rotation is 90 degrees, and the socket's should read the same. When the value is -1, the rotation should read -90 degrees. This is important, because if the degrees are, for example, 90 for neutral, 0 for -1 and 180 for 1, then the 180 degrees will cause problems. In Unreal, if the degree would go above 180, it becomes -180, which is a nightmare when working with clamps. So, ideally just stick to -90, 0 and 90 degrees.

You can see the socket's local rotations in the Skeleton view if you select them while an animation is playing. You can also output them in a print node and check while playing, or right click on them in a blueprint, 'watch' them and the values will appear on the blueprint while the game is playing.

Let's move on to the animation blueprint that drives all of this!

It's pretty big, so you may have to download it to see it properly

You may get all of it by just looking at the picture, but I'll explain everything below.

You need to add this to the character's anim blueprint's event graph's Update Animation event. I've added a bool that can turn off the whole thing if it's not needed. Then, the blueprint makes sure that the variable holding the material of the face isn't empty; if it is, the blueprint will create a dynamic material out of the appropriate material and use that. (The dynamic part is important, parameters can only be updated in dynamic materials, and so far I've only managed to create them during runtime. So this part will happen when the anim blueprint starts updating.)

Then comes the math. I'll show you the easiest example: you want to figure out when to activate the normal map for wrinkling the forehead above the left brow. You'll have to get the appropriate morph target's socket's rotation first (make sure that you get the rotation in component space - the rotation compared to the original rotations of the socket, not in world space, because that rotation is compared to the world's rotation. You can get component space by using the Socket Transform node instead of the simple rotation node). In our case, when the morph target's value is 1, the socket's rotation is 90.

The material parameter's values are: 0 for invisible, 1 for completely visible, everything in between is partly visible, visibility depending on the number. For this reason, we have to assign 90 degrees to the number 1, and 0 degrees to the number 0. The easiest way to do this is using the 'normalize to range' node. This node lets you type in a minimum and maximum value, and outputs the position of the input number on this scale as a number from 0 to 1 (for example, on a range of 0 to 100, 87 would be 0,87).

After we got this number, we just have to set it as the scalar material parameter controlling the normal map's opacity, and we're done. It's a good idea to clamp the values of the socket rotations, because animations - obviously - are not clamped to a 90 degrees of movement, there can be slip-ups. (This is where the jump from 180 to -180 would make things hard: -180 is well below 0 or even -90, so it would instantly make the value go from 1 to 0, thus make the normalmap invisible, which is something that we don't want.)

A slightly more complicated version is if both the positive and negative morph target values are bound to normal maps. You'll have to do the first part the same way, but you have to get the absolute value of the rotation degrees of the socket (negative numbers become positive, positive numbers stay positive), and then normalize to range. The rest is the same.

The only other variation we're using is multiple morph targets triggering the same normal map. If you want some of the morphs to affect the normal map's opacity a little less then others, you'll have to multiply the output amount by some less-than-1 number, like 0.8. At the end, you'll have to add all the affecting morph's values together, then clamp the resulting number between 0 and 1. That's all there is to it, really.

For testing, I suggest you create a morph target test animation where you operate all of the morphs individually and in order. You'll be able to clearly see if all of the morphs are working. For me, there was a lot of trial and error involved, but you could minimize this by clearly noting and setting all the socket's rotations.

Subtle wrinkles on the forehead

Less subtle nasolabial folds

If you have any questions or suggestions, leave a comment below!

Tutorial: Baking normal (and other) maps in Blender

It's Geril. Hi, all!

As I looked back on our previous posts, I realized I promised to make more tutorials about UDK and game development in general. The reason why we didn't make any is because it's hard to teach what we are still learning. But I'd like to share something that I think not many people know.

Blender3D can bake many kinds of textures, so we don't need ZBrush, 3D Coat or Blacksmith to create materials (even tileable ones). But as Blender is a free software, we have to work twice as hard.

I'm trying to make this tutorial as in-depth and easy to understand as possible, because I tend to omit important parts. If I'm wrong about something, please tell me in the comments, you know that I'm still learning, too.

So, in this tutorial, we're going to show you how we create tileable materials; in this case, a cobblestone surface. This is the result, rendered in Unreal 4:

This tutorial is only for the normal, displacement and ambient occlusion maps. In theory, anything can be a color map, we're using one that is a general stone texture we use for many other things.

---

First, launch Blender (we're using 2.76 right now) and delete the starting box. Then add a Plane mesh using the Add menu [SHIFT + A] - this is going to be the surface to which we are projecting our texture. We're using the upper right area of the editor as a UV/Image editor, of course you can use any other area, too.

Press [TAB] to go into the plane's mesh edit mode that has all of the vertexes selected by default. Press [U] and select Unwrap from the menu - you just created a UV map for the plane. You can see it in the UV/Image editor area. This is where your texture will appear after baking.

In that area, create a new image in the image menu. Choose the right resolution for your texture, based on how important it is going to be in the game. We use 2048x2048. The color and name parameters are not important.

Exit the plane's edit mode [TAB], and add [SHIFT + A] an ico sphere (this will be the first stone). Push [G] and then [Z] to move it along the Z axis above the plane. Push [S] to re-size the sphere, shrink it to the size of the stone you want, relative to the texture's size (the plane).

Using the [NUM 7] key, you can switch to a top-down view, and using [NUM 5], you can change the perspective to orthographic (this is very important if you want to be precise!). Go into the sphere's edit mode [TAB], make sure all the vertexes are selected (if not, use [CTRL + L]).You can randomize the sphere by clicking randomize on the left toolbar. You can adjust it further in that menu, but I find it easier to just repeatedly push the randomize button until I am happy with it.

Open the specials menu with [W]. You'll find a "Smooth" option, with that, you can smooth your randomized sphere so it looks more like a rock. Once again, you can adjust it in the left-side menu, or just keep smoothing it over and over. With this method, you can randomize rock easily. (I add the other rock this way, too) Move your rock to one of the corners, I moved mine to the lower right. To move the mesh on the grid, hold down the [CTRL] key, that way you can align it perfectly with the corner.

Duplicate the rock with [SHIFT + D], and put a duplicate in each of the corners using the grid. Create some new ico spheres, and make new rocks from them using the previous method. Select some rock with area selection [C] (if you didn't select all the vertexes of a mesh with [C], push [CTRL+L] to select the remaining vertexes on them as well), and arrange them as a group, re-size them [S], rotate them ([R] to rotate on the current view's axis, [R] again if you want to rotate them on all the axes).

Duplicate them, and to make it more random, set the pivot point to Individual Origins, so that when you rotate or re-size them, they will all rotate and re-size individually (not as a group). Just make it seem as random as possible, customize them, but it's important that, except for the rocks in the corners, no other rock should touch the plane's edges.

Pick a group of rocks, preferably one that is roughly as wide as it is long. Place it to one edge of the plane, so that it touches the rock in the corner and goes off the edge a bit. After that, duplicate it and place the duplicate on the opposite edge, using the grid. The easiest way is, say you want to move them from the left side edge to the right side one. You'd use grab [G], and to move them on the X axis, push [X] after [G]. Then just type in "-2". This moves the group two units on the X axis - right onto the other edge of the plane. For the top and bottom edges, of course you'd use [Y] instead of [X], for the Y axis. Do this for every edge, the point is that you should fill all the opposing edges with the same models, so that in the end it becomes tileable. It's a little tricky and tedious, and requires a lot of patience, but if I could do it, you can, too.

After our rocky surface is done, and there are no holes remaining, exit edit mode [TAB] and move it to about one unit above the plane. Then duplicate it, and move the duplicate a little closer to the plane (it still shouldn't touch it!). You should also rotate the duplicate 90 degrees ([R], [Z], 90 - this shortcut rotates it on the Z axis 90 degrees) Another important thing is to set the rocks' flat shading to smooth on the left side toolbar.

Select the two rock objects and the plane, in this order; the plane should be the last one [SHIFT + Right Mouse Button] for multiple selection). In the Properties section on the right side, choose the photo icon - that is the Render tab. There you can find the Bake sub window, where you can adjust the bake settings. First, check "Selected to Active" and "Clear". Margin can be left on 16, but I've had bad experiences with margins and tiling, so I recommend setting it to about 4. Set Bake mode to Normals. Now let's bake!

Well now, if the rendered image doesn't look healthily blue, but rather greenish or orange, our Plane's surface points in the wrong direction. In this case, go to the plane's edit mode, select all of the vertexes, and look for a "Flip Direction" button on the right-side toolbar. That should flip the plane's direction. Select the rocks and the plane in the right order, and try again. It should look normal now (pun intended). If you're happy with it, you can save it in the UV/Image editor window with Save as Image, I recommend saving as either PNG or TGA. When you've saved it, let's go back to baking - select Displacement in the Bake menu, and Bake again.

You shouldn't have trouble with the plane's direction again if the normal map came out good. Just bake it and save it next to the normal map you created. After that, repeat the process for Ambient Occlusion as well. You can do a little touch-up in Photoshop or GIMP, like filling in holes between rocks, but be careful, and always keep the original file. You can set the contrast higher on the Displacement map though, it makes the material look deeper. (Don't mind the Hungarian stuff on the pic)

Let's import the pictures into Unreal 4. Select one of the imported pics, and make a material based on it. Put the remaining pictures in that material as well. Use something appropriate as a color texture. Using the color texture as a DetailNormal map is also a good idea. 

This is the last step, messing around with the material settings. It's a little hard to explain this in text, I hope that the picture is understandable. I'll try to write the instructions down as best as I can, though, because I'm not that lazy.

Connect the Displacement's red output to a BumpOffset node [B], then duplicate it and repeat. Create a TexCoord Node [U], and set it to U:2 and V:2. Connect this to the second BumpOffset Coords node's input. Connect the first BumpOffset node's output to the inputs of the textures we created. The second BumpOffset node (with the TexCoord) should be connected to the color texture, and its detailnormal map.

Multiply [M] the Displacement map with the rock texture. Connect this to Basic Color. If it's too dark, leave out the multiply node, but I think it improves the visuals quite a bit. Our normal map would be boring on its own, so you should also add the color map's detailnormal to it (multiplying its texture coord so it becomes more dense).

Create a BlendAngleCorrectedNormal node (that's a fancy way of saying MAKE A MORE DETAILED NORMAL) (okay.. there is a reason why it's called that, but.. let's just continue, okay?) Put this node next to the normal maps. Connect our baked Normal map to Base Normal, and the DetailNormal's to AdditionNormal. Connect this fancy node to the Normal input. And finally, connect the Ambient Occlusion's output to...... the Ambient Occlusion input. Yes, there is an easy step in this.

All that is left is to save the material and try it out. You should adjust the BumpOffset, Roughness and Specular, but I can't help you there, it all depends on what looks good with the color map and environment you are using.

---

This wraps up the tutorial. If you have any questions or requests, leave them in the comments below! I'll be happy to create a tutorial on anything, as long as I know what I'm doing.

Basic workflow: simple static object

In this post I'd like to show you our basic workflow.

Our primarily used software are Gimp, Blender and the UDK.
Gimp is a freely distributed image manipulation software that we use for making textures and drawing concept art. Blender is also free, it's an open-source 3D modeling software.
I'll show you how a simple object is made from scratch, all the way till it's usable in the engine.

In this case, we wanted to make a crate. We made the mesh in Blender first (although in some cases, it is recommended to make the textures first). We only use the necessary amount of polygons, because it's not an important object.

We then exported the UVs (the texture maps), and made the textures based on the UVs (though they are only placeholders right now). We used 3 kinds of textures in this case: a diffuse texture, a specular map, and a normal map.
I'll briefly explain what they do:

• The diffuse texture is everything you see without any "special effects", essentially a drawing that the model is wrapped in.
• The specular map determines what happens when the model receives light.
• The normal map makes the model look more detailed, for example, the wood grains on this crate are normal mapped, so they don't look like they are simply drawn on.

The texture resolution is 1024*1024p, and we try to use one texture for as many models as possible.

To import the model into the UDK, we usually export from Blender in FBX format. After the importing was done, we needed to make a material, using our previously made textures.

So then we had our crate model in the engine as a static mesh, placable on any map. We created LODs for the crate (so that it gets simpler the farther away it is from the player), and a collision box.

And that's how it looks in our test map.
The cel-shade effect is not made by the material, it is only a post-process effect that is placed on every object.

So this is how we make simple models in a nutshell. We also plan on making modeling and UDK tutorials sometime soon.

See you next time!