1. IntroductionThis is meant to be an advanced texturing guide for making mods with a similar level of detail to the original Smash 4 models. Normal mapping and ambient occlusion mapping are confusing at first, but they enable you to create very clean and detailed results.
This tutorial isn't software specific, but the more advanced techniques will require more advanced applications. You can use free applications just fine, but more advanced applications offer some advantages. Photoshop is ideal for editing textures compared to free alternatives. Blender is a free and fully featured 3d modeling package that can be used to bake normal and ambient occlusion maps, so you shouldn’t need paid software.
Smash Forge is a powerful modding application that can preview normal maps, ambient occlusion maps, and many other material settings. This is also the only way to accurately debug issues with normals, normal maps, tangents, and bitangents. You can find the latest official release here. You can find basic usage information for Smash Forge on the Forge wiki.
Surface normals, or simply normals, point perpendicular to the tangent plane at a point on a surface. This is a fancy way of saying what direction a face is pointing. Each vertex has its own normal. The above image shows the world space normals for Cloud's trophy model before and after applying normal maps. The normals are used to calculate the shading for diffuse, specular, reflection, fresnel, and many other shading techniques. This is why models with normal maps appear more detailed, even though the original geometry is unaffected.
2.1. Displaying Normals in Forge
Normals can be displayed in Forge by changing the render mode in the render settings. The "Normals" mode displays the world space normals remapped to from [-1,1] to [0 ,1] for easier visualization. In the above image, normal map rendering is enabled for the right half of the image. The added detail is very apparent when using this rendering mode.
2.1. Normal Maps
Normal maps use an image to change the normals of a model to simulate detail without increasing the number of polygons. Bumps, scales, and ridges can all be efficiently approximated using normal maps. The normal direction affects the lighting of a surface. Changing the apparent angle of a surface can affect how bright or dark it appears. Note that normal maps affect the lighting of an object but do not change the actual geometry. This means the outline of your model will not be changed.
The above screenshot illustrates the benefits of using a normal map. Samus does not have a very detailed model but appears very detailed in game thanks to a tangent space normal map. Normal maps mainly affect the appearance of specular highlights and cubemap reflections, so I have disabled the diffuse component for the comparison render.
2.1.1. Displaying Normal Maps in Smash ForgeSmash Forge can be used to visualize normal maps for your models. Under View>Render Settings>Debug Mode, select Normal Maps. This will display the normal maps for every model. If the model does not have a normal map, a blue and black checkerboard texture is used. If you enable material lighting and "Use Normal Maps," you can preview the effects of normal maps in real time in Forge's viewport. Always check to make sure the normal maps aren't messing up the shading for your models.
2.1.2. Normal Map Types
Normal maps use different colors to represent different directions for the surface normal. The colors used depends on the normal map type. Normals should be unit vectors, meaning they have a length of one (sqrt(X^2 + Y^2 + Z^2) = 1). This means that you shouldn’t edit the brightness of a normal map.
2.1.3. Tangent Space
Tangent space normal maps are predominately blue. These are the only types of normal maps I've seen in game. Tangent space maps differ from other maps in that the directions are always relative to the surface. Tangent space is used for deforming objects because the normal map will always produce the same effect regardless of the object’s orientation or deformations.
Blue (128,128,255) is the original direction of the surface, so the normal map will have no effect in that area. Combinations of red, green, and blue rotate the normal vector in different directions. The red channel affects the x direction, the green channel affects the y direction, and the blue channel affects the z direction (in or out of the surface). The blue channel tends to have less detail than the other channels.
2.1.4. Tangent Space Normal Map Calculations OverviewThe world space normals for the model are modified using the colors of the tangent space normal map. Mesh normals are per vertex, but normal maps are calculated per pixel, which allows for much more detail. The colors for the normal map are actually used as vectors (directions). The red, green, and blue channel correspond to the X, Y, and Z direction, respectively. The normals should be normalized before being used to prevent shading issues. The x, y, and z component of the normals have values between -1 and 1. Normal maps have values between 0 and 1, so the values need to be remapped before being used.
The tangent and bitangent vectors, which won't be covered in depth in this tutorial, are also necessary for calculating the new normal using matrix math. If the tangents and bitangents are messed up, the resulting normal will be as well.
2.2. Debugging Issues with NormalsIncorrect normals or normal maps can make models appear blocky, overly smooth, or even black. Reflection, specular, and fresnel will be more effected by poor normals than diffuse will. I'll cover common issues in the following sections. If you follow these steps in order, you should be able to fix your rendering issues.
2.2.1. 1. Mesh NormalsThe first thing to check is the normals for the mesh itself. This problem should also be apparent in the viewport of the 3d modeling program, such as 3ds Max. If the normals look fine in 3ds Max but look bad in game, you can use the Normals render mode.
The problem can usually be solved in Forge by right clicking a model or polygon and selecting Normals > Calculate or Normals > Smooth from the mesh list. For best results, fix the normals manually in a 3d modeling program. Use Normals, Tan, Bi-Tan (Float) for the dae import settings in Forge for best results. If the mesh normals look fine but you still have issues, move on to the next section.
2.2.2. 2. Normal Map RenderingTo determine if normal maps are the source of your shading issues, you can disable them in Forge in the render settings. By toggling normal map rendering on/off and looking at the normal and normal map rendering modes, you can figure out what part of the map is causing issues. If the normal map looks fine but you still have issues, move on to the next section.
2.2.3. 3. Tangents and Bitangents
The tangents and bitangents are used in calculating the effects of normal maps. If the tangents and/or bitangents are messed up, normal map rendering will also not work properly. You can check this by using the tangent and bitangent render modes. Correct tangents and bitangents should be very colorful with no black regions.
If the entire model is black for the Tangents or Bitangents rendering modes, you need to generate new tangents and bitangents. Forge will display a message if the wrong vert type is used.
In this image, you can clearly see the areas where the tangents and bitangents were not saved properly. This causes the mesh to render as black when using normal maps. Make sure the normals are fine in the 3d modeling application and that you are using the appropriate DAE import settings. Newer versions of Forge will not generate these black artifacts.
2.2.4. Mirrored Normal MapsNormal maps often make use of mirrored UVs to save texture resolution. When using mirrored normal maps, the bitangents should be the same color for each half. Previous versions of Forge did not generate correct bitangents for mirrored UVs, but this is fixed in recent versions. Be sure to split the faces of the mesh at UV seams to avoid smoothing the tangents/bitangents and introducing shading artifacts.
2.2.5. Normal Map Normalization
Mesh normals or the normals in normal maps only encode a direction, so the length is assumed to be 1. This means that the sqrt(x^2 + y^2 + z^2) = 1. If the normal map isn't normalized, the shading will look incorrect for the model.
In the above image, you can see what the resulting normals will look like if the normal map isn't normalized (in this case its too dark). This usually results from combining two normal maps in an image editor using the wrong blending mode. I've included information about accurately combining normal maps in Photoshop later in the tutorial. If your normal map is too dark or too bright, this can be easily fixed by normalizing the image. The xNormal plugin for photoshop can easily do this, for example. Even after normalizing the color channels, the normal map may still be too dark. Adjust the image until the base value is (127,127,255).
2.3. Editing Normal Maps
2.3.1. Object Space
Object Space normal maps are not relative like tangent space normal maps. Object space normals are based on the object's local space. Forward is always forward, left is always left, etc. The object can be rotated, but deformations will mess up the normals. Mirroring UVs also causes issues with these maps. Object space maps are very colorful, such as the above example from Diancie's Sun/Moon model. After further research it seems that Sm4sh does not use object space maps.
2.3.2. Converting Object Space Normal Maps to Tangent Space
Object space normal maps can have better performance and produce smoother results than tangent space normal maps. For characters with skeletons and deformations, tangent space normal maps will usually work better. You can easily convert an object space normal map using Handplane 3d, provided that you have the object space normal map and the original geometry with proper UVs.
2.3.3. Converting Object to Tangent Space with Handplane 3d
Handplane 3d can be installed from the link above. The software is free, so you don't have to donate. The main version is for baking normal maps, curvature maps, and other map types from high poly geometry. For converting object space to tangent space, download "Legacy_handplane_1.6_NotTheBaker.rar."
The above example screenshot for Diancie shows the basic settings for using the application. Pokemon Sun/Moon and X/Y use object space normal maps. The suffix and output destination will vary, of course. I've found the maya setting matches the channels of the in game normal maps. If you flip any of the channels, the normal maps will not appear correctly in game. Padding helps with texture seams, so leave this on.
2.3.4. Removing Detail from Normal MapsThe only way to get accurate normal maps for a model is to generate them. Editing existing normal maps is still helpful for adding details that would be difficult to model or removing existing detail.
Removing detail from a normal map can be somewhat tricky due to normal maps being composed of three channels. For tangent space normal maps, you can simply paint the neutral color (128,128,255) over areas you don’t want. This will remove the detail but may also negatively impact the shading of the model.
2.3.5. Removing Detail in Photoshop with Content Aware ToolsThe best method I’ve found for removing logos, textures, and designs you don’t want from the normal map is to use the content aware tools in Photoshop. The patch, healing brush, and spot healing brush can remove smaller details while still preserving the overall shading of the normal map. I like to work on a separate layer, so I can always undo the changes later. When using the healing brush or spot healing brush, be sure "content aware" is checked in the tool settings. Check "sample all layers" if you want to work on a separate layer.
2.4. Generating Normal Maps
2.4.1. Accurate Method - Baking Normal Maps in 3d Application
The best way to generate a normal map is using two models: your original model and a model with details you want to transfer. The application will calculate the differences between the two models and generate an accurate normal map. The application looks for differences between the two models with a search envelope that surrounds both meshes. This feature varies based on application. Search if your 3d application can “bake normal maps” to find out.
The source mesh usually has higher detail than the target mesh. You can use normal maps to make a low poly mesh look like a high poly mesh. In the above image, a low poly sphere is made to look much smoother by calculating a normal map from a higher poly sphere. Notice that the resulting low poly sphere appears much smoother, but the jagged outlines are still present. The normals only smoothed the lighting on the model and not the actual geometry.
Normal maps can also simulate other details that would be difficult to model, such as hair or scales. Models in Smash 4 use normal maps for smoothing and for added details. If your vertex edit or model import looks jagged, generating a normal map from a smooth, high poly mesh can help a lot with the lighting.
2.4.2. Baking Accurate Normal Maps with Handplane Baker
In order to generate accurate normal maps, you need to bake the detail from a high resolution mesh onto a low resolution mesh. Handplane Baker can be downloaded from the link above. As of writing this tutorial, the current version is 0.9. Handplane Baker can easily generate these maps, provided that you have a low resolution destination mesh with proper UVs and the detailed high resolution mesh. Check the box labeled "Tangent Space Normals" to generate a normal map. For the the normal map settings, the "Autodesk Maya 2013/2014" option should generate a normal map with proper color channels for Smash 4.
2.4.3. Hand Creating Normal Maps in 2D Application
Tangent space normal maps can be easily generated from height maps. Differences in height create differences in the normals of a surface. The nvidia plugin for photoshop has a normal map filter, which usually does a decent job. There are other stand alone applications as well.
The quality of the generated normal map depends on the quality of the heightmap. Use lighter colors for higher areas and black for darker colors. Black creates depressions, 50% grey is neutral, and white creates raised areas. Notice that that colors are different for the normals generated for the black rectangle compared to the white one. After creating the heightmap, use the plugin or application to create a normal map.
2.4.4. Combining Normal Maps in Photoshop
After creating a normal map, you will want to combine your generated normal maps with the existing one. There is an excellent script for achieving this in Photoshop posted on Polycount. The script uses more mathematically accurate methods than the common method of using the overlay blending mode. You can drag the sliders to control the relative intensity of each normal map layer. The link includes the download and the author's tutorial on how to use the script.
3. Ambient Occlusion Maps
The “ambient” part of ambient occlusion refers to indirect lighting. Direct lighting would be the light coming directly from a light source. Ambient occlusion affects what part of a surface are occluded, meaning the ambient light doesn’t reach these places. Areas underneath capes, hair, or even the cracks of a model would be partially occluded. Ambient occlusion can be simulated in real time or using pre generated ambient occlusion maps. Smash 4 uses the latter method.
Ambient Occlusion darkens the mesh to simulate the shadows in occluded areas. White represents the maximum lighting, and black represents the minimum amount of ambient lighting. The above comparison shows how ambient occlusion maps, especially when used in combination with normal maps, can add a lot of detail to your models. The diffuse and ambient occlusion maps are multiplied, making them darker. This result can be tinted or brightened by material data.
3.1. Displaying Ambient Occlusion Maps in Smash Forge
Ambient occlusion maps are always rendered in Smash Forge's viewport, so there is no way to turn this off. You can, however, preview just the ambient occlusion texture using the Ambient Occlusion render mode.
3.2. AO and Material Editing
This value can be found in the materials of many models in Sm4sh. AOMinGain controls the color of the ambient occlusion map. AOMinGain can only make the model brighter. This value can also be used to color the diffuse component of a model. Setting AOMinGain to (1.0,0.0,0.0) for example, will brighten just the red channel of the ao map. If you've baked an ambient occlusion map, leave AOMinGain at 0 to avoid making the model too bright. Baked ao maps are usually mostly white and don't need to be brightened.
The fourth value for AOMinGain controls the intensity of the AO blending for specular and reflections. A value of 0 is full blending, so reflections and specular will be multiplied by the AO map. Values of 1 or higher will not use blending.
3.3. Editing AO Maps
3.3.1. Removing Detail in Photoshop with Content Aware Tools
Ambient occlusion maps are usually found in the alpha channel of normal maps. This isn't always the case for certain materials, such as Corrin's hair. When loading your normal maps, never load the alpha channels as transparency, as this will prevent you from editing the ambient occlusion map or normal map correctly.
Ambient occlusion maps in Smash 4 are grayscale images, so editing is relatively straightforward. If the ambient occlusion map has very little detail, you can simply use the brush tool to paint out unwanted areas. If the map has more detail, such as scales, you will want to use the content aware tools in newer versions of Photoshop. The spot healing brush, healing brush, and patch tool can make removing detail relatively easy. In the above video, I use a combination of the spot healing brush to remove the design and then the healing brush to add the texture back to the image.
3.3.2. Editing Ambient Occlusion Maps in Gimp
Gimp is a free image editing application that can be used to edit normal and ambient occlusion maps. I highly suggest you don't use Paint.net for this. Open the DDS file for the normal map in Gimp. The image should be mostly blue and partially transparent. Normal maps aren't actually transparent, so we need to separate the color channels to see the AO map. In the top menu bar under Colors>Components>Decompose, select RGBA. This will separate the color channels into layers. Hide the RGB layers to reveal the AO map. When you're done editing the AO map, go to Colors>Components>Compose>RGBA and make sure the proper layers are selected for each channel.
4. Baking AO Maps
Ambient Occlusion maps can be generated using a 3d application. After assigning an ambient occlusion shader to the mesh, you can bake the render to a texture. This texture can serve as a base for your ambient occlusion map. Added details, such as scales, can be added from images or painted rather than baked from geometry.
4.1. Baking AO from High Poly in Handplane
Ambient occlusion maps can be used to simulate the detail of a high poly mesh on a low poly mesh. The low poly mesh is the default mesh for Rosalina's shoulders imported from the game files. The high resolution mesh was sculpted in Mudbox. When you enable "Ambient Occlusion" in the output settings, an AO map will be generated.
4.1.1. Ambient Occlusion Settings for Clean Maps in HandplaneAmbient occlusion is more complicated to configure than normal maps. Ambient occlusion casts rays on the surface to calculate shadowed areas. This produces accurate but often very noisy results. You can improve the "grainy" look of your AO maps by increasing the "Super Sample" setting to something like 4x. Under the ambient occlusion settings, try increasing the sample count until the baked map looks cleaner. The higher the samples, the smoother the result will be, but the longer the baking time. You can blur the map in an image editor to avoid using a super high sample count.
4.2. Baking Simple AO in XNormal
Handplane and most render engines offer ambient occlusion baking, but the results are very noisy and often have issues with low poly meshes and texture seams. Xnormal can bake AO from a high poly mesh, but it also has a useful tool for generating simple AO maps. The application is free and offers similar functionality to Handplane, but it isn't very efficient for most things.
Under Tools>Simpler ambient occlusion generator, select the source mesh and output settings for the AO map. Play around with Uniform, Cosine, or CosineSq and see what looks the best. If the map is too bright, change the gamma to 1. Edge padding helps prevent dark seams around UV borders. The above image shows a visual comparison of each method. Baking should only take a few seconds, depending on your computer.
4.3. Baking AO in Blender WIP
Ambient occlusion maps can be easily baked in Blender Cycles. Blender is a free and open source application, which can be downloaded from the official site.
4.3.1. Preparing Models for Baking
- Model is UV unwrapped
- No UVs are overlapping
- Appropriate material assigned to each mesh
- Correct surface normals to avoid shading artifacts
For baking low frequency detail like lighting information or ambient occlusion, an automatic UV mapping is ideal. This ensures each face takes up an appropriate amount of the texture and no UVs are overlapping. This can be done in Blender in the UV/Image Editor by selecting all the vertices in edit mode and selecting UV > Unwrap. After the unwrapping process, select UV > Pack Islands (ctrl + p) to ensure none of the UV islands are overlapping. Mirrored UVs will not bake properly for lighting and ambient occlusion.
4.3.2. Baking to Image Textures in CyclesIn order to bake a Cycles render to a texture, you first need to create an image texture. In the UV/Image Editor in Blender, create a new image with a sufficiently high resolution. Depending on the UV layout, you may want to bake in 4k or higher. In the material node editor, create a new image texture node and select the image you just created. This image texture node will need to be selected when you start the baking process. The node doesn't need to be connected to the material, however.
Most of the same render settings apply to the bake as well. Start with 1 sample to ensure the baking process is working correctly. If you're baking to vertex color later, high sample counts aren't necessary. Instead of pressing render, press "Bake" at the bottom of the Render tab. When the bake is complete, the image will show up in the UV/Image Editor. Use the textured view mode in the viewport or an emission material to ensure the texture was baked properly. Save the image to a png when the bake is complete.
4.3.3. Creating Clean BakesAmbient occlusion will be very noisy unless you use a very high sample count. Rather than increasing the sample count, its more effective to just use a blur or noise reduction filter in an image editing program. 64 samples is usually more than enough. Note that denoising in Blender 2.79 doesn't seem to work for texture bakes. If the model has visible seams when displaying the baked texture, try baking with a higher margin value, which expands the texture around the UV islands to eliminate seams.
4.4. Baking Textures to Vertex ColorThe results of rendering a model to a texture can also be baked into the model's vertex color. Keep in mind that the detail of the vertex color depends on the vertex count of the model. The low poly models used in Sm4sh won't have the vertex counts to bake high frequency detail. Vertex color is ideal for baking lighting or ambient occlusion information because vertex color doesn't rely on UVs. A brief overview of the process is as follows:
- Bake render to texture in Cycles
- Create a shadeless material in Blender Render with the baked texture
- Bake the texture to vertex color in Blender Render
- Export the model as DAE
Create a new Blender Render material for the mesh. Do not check use nodes. In the material options, check shadeless under shading options so no additional lighting information is baked. In the baking settings of the render menu, set the bake mode to Full Render and select bake to vertex color. Make sure that the mesh has a vertex color in the Data menu. The default name should be "Col."
After baking, you can check that the vertex color was baked properly by creating using material nodes for Blender Render or Blender Cycles.
4.5. Hand Creating Ambient Occlusion Maps
Ambient occlusion maps, similar to normal maps can be painted by hand or using grayscale images. In the above image, I painted the chest and jaw on top of Bowser’s default ambient occlusion maps. Paint white for fully exposed areas and black for less exposed areas. The cracks in Bowser’s scales, for example, are black to darken these areas. Creating a proper ambient occlusion map can help your textures “pop” and have much more detail.
5. ConclusionThis concludes the guide on normal maps and ambient occlusion maps. I still have a lot of information I would like to cover, so I’ll be updating this guide frequently. Below are some helpful links that go more in depth on how normal and ambient occlusion maps work.
5.1. Additional ReadingExplanation of Normal Maps | Explanation of Ambient Occlusion
Polycount Wiki page | Online Normal Map Creation