Sign In to the Console
Lumberyard
User Guide (Version 1.16)

AWS Documentation » Amazon Lumberyard » User Guide » Working with the Asset Pipeline and Asset Files » Shader Cache and Generation

Shader Cache and Generation

This section discusses the shader cache and how to generate shader cache .pak files.

Shader Cache

The shader cache stores a collection of parsed and precompiled shaders. Since the shader code is written with multiple definitions, Lumberyard can generate an enormous number of different shaders. Compiling shaders on demand at run time is only possible on the PC. On-demand shader compiling can cause freezes during the game play and uses extra memory. To reduce this overhead, all required shader combinations for a game are parsed, compiled, and stored in the shader cache.

The shader cache generally refers to the following files:

  • Shaders.pak – Contains the shader source files, which is everything inside the lumberyard_version\dev\Engine\Shaders\ directory, excluding EngineAssets.

    Note

    The actual shader source code (*.cfi and *.cfx) can be removed from this file for the final released version, and is not required anymore when the binary shaders are valid and available.

  • ShadersBin.pak – Contains the binary-parsed shader information of the shader source code.

  • ShaderCache.pak – Contains compiled shaders for all possible combinations that have been submitted to the remote shader compiler.

  • ShaderCacheStartup.pak – Contains a small subset of the shader cache that is used during game start. This file is loaded into memory for quicker start times, but is not required. This cache is often used to hold the minimum required set of shaders to show a loading screen so that the rest of the loading can occur.

ShaderCache.pak File Generation

Creating a ShaderCache.pak file consists of running the BuildShaderPak_DX11.bat batch script, which in turn runs ShaderCacheGen.exe to ensure the local cache directory contains all the shaders that are listed in the ShaderList.txt file. The BuildShaderPak_DX11.bat script packs the contents of the cache directory, creates a ShaderCache.zip file, and then renames the file to ShaderCache.pak.

You can find the ShaderList_platform.txt file either from the remote shader compiler server or from the Lumberyard Editor directory. This file contains the list of all shaders which ShaderCacheGen.exe uses to produce the shader combinations for your game.

When running Lumberyard Editor, individual shaders are created as you view them. As such, you do not strictly need a remote shader compiler server to test game release mode or test shader pack generation. You just need access to the ShaderList_platform.txt file that is created in the lumberyard_version\dev\cache\game_name\platform\user\cache\shaders directory when running Lumberyard Editor.

However, only the shaders you have viewed on your local computer while running Lumberyard Editor will be listed in the ShaderList_platform.txt file. For this reason, it is recommended that you use a remote shader compiler server if possible.

Note

During development time, when you run the game or Lumberyard Editor, and before the shaders are packed into shader cache .pak files, loose shader files are created in the following directory: lumberyard_version\Dev\Cache\your_game\platform\user\cache

The following sections detail the steps used to generate ShaderCache.pak files:

ShaderCacheGen.exe

Lumberyard ships with ShaderCacheGen.exe, which is located in the lumberyard_version\dev\Bin64 directory. ShaderCacheGen.exe is a stripped-down version of the Lumberyard game launcher without the render viewport, and is used to populate the local shader cache directory with all the shaders contained in the ShaderList.txt file.

When running ShaderCacheGen.exe, it first loads the ShaderCacheGen.cfg file, which you can customize to suit your needs.

If you customized Lumberyard in any way, you must build Lumberyard and your game using the all profile. This argument builds both ShaderCacheGen.exe (and ensures that it is up-to-date) and the game .dll files that are required.

Enter the following command for your version of Visual Studio:

lmbr_waf build_win_x64_vs2015_profile -p all --targets=CrySCompileServer

If you don’t want to (or cannot) build using the all profile, you can alternatively just build the game_and_engine spec and the shadercachegen spec.

Enter the following command for your version of Visual Studio:

lmbr_waf build_win_x64_vs2015_profile -p game_and_engine
lmbr_waf build_win_x64_vs2015_profile -p shadercachegen

Packing the Shader Cache Using a Batch File

The BuildShaderPak_DX11.bat file generates the ShaderCache.pak files, which are saved to the lumberyard_version\dev\build\platform\your_game directory. The batch file first calls the ShaderCacheGen.exe and then calls Tools\pakShaders.bat.

From a command line, navigate to the lumberyard_version\dev directory, and run the BuildShaderPak_DX11.bat, and specify the location to the ShaderList_platform.txt file.

Example 

F:\lumberyard_version\dev\BuildShaderPak_DX11.bat C:\shader_compiler_server\ShaderList_DX11.txt

Once the shader .pak files are created, you can move them as needed. For example, if you already built a release version of your game, you can place them with the rest of the .pak files.

When compiling shaders for your project, you can customize the BuildShaderPak_DX11.bat file as needed. The following is an excerpt from a sample .bat file: 

set SOURCESHADERLIST=%1
set GAMENAME=your_game_project
set DESTSHADERFOLDER=Cache\%GAMENAME%\PC\user\Cache\Shaders

set SHADERPLATFORM=PC
rem other available platforms are GL4 GLES3 METAL
rem if changing the above platform, also change the below directory name (D3D11, METAL, GL4, GLES3)
set SHADERFLAVOR=D3D11

Packing the Shader Cache Manually

If you want to use more complex build pipelines, you can pack the shader cache manually.

To pack the shader cache manually

  1. Run ShaderCacheGen.exe to generate the shader cache so you can pack it later.

  2. Zip all the shaders into a ShaderCache.zip file, then rename the file to ShaderCache.pak.

Each platform has different .pak files. See the following directory mapping for the PC platforms:

The PC platform should copy data from the following directories:

lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\D3D9\

lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\D3D10\

lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\D3D11\

lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\GL4\

into the following destination directories:

shaders\cache\D3D9\

shaders\cache\D3D10\

shaders\cache\D3D11\

shaders\cache\GL4\

ShaderCache.pak should contain everything from the previously listed subfolders.

ShadersBin.pak should contain only the *.cfxb and *.cfib files.

ShaderCacheStartup.pak should contain the following files: 

lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\lookupdata.bin -> Shadercache\<platform>\lookupdata.bin
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\CGPShader\FixedPipelineEmu* -> Shadercache\<platform>\CGPShader\FixedPipelineEmu*
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\CGPShader\Scaleform* -> Shadercache\<platform>\CGPShader\Scaleform*
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\CGPShader\Stereo* -> Shadercache\<platform>\CGPShader\Stereo*
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\CGVShader\FixedPipelineEmu* -> Shadercache\<platform>\CGVShader\FixedPipelineEmu*
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\CGVShader\Scaleform* -> Shadercache\<platform>\CGVShader\Scaleform*
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\CGVShader\Stereo* -> Shadercache\<platform>\CGVShader\Stereo*
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\lookupdata.bin -> Shaders\Cache\<platform>\lookupdata.bin
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\Common.cfib -> Shaders\Cache\<platform>\Common.cfib
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\fallback.cfxb -> Shaders\Cache\<platform>\fallback.cfxb
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\fixedpipelineemu.cfxb -> Shaders\Cache\<platform>\fixedpipelineemu.cfxb
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\FXConstantDefs.cfib -> Shaders\Cache\<platform>\FXConstantDefs.cfib
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\FXSamplerDefs.cfib -> Shaders\Cache\<platform>\FXSamplerDefs.cfib
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\FXSetupEnvVars.cfib -> Shaders\Cache\<platform>\FXSetupEnvVars.cfib
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\FXStreamDefs.cfib -> Shaders\Cache\<platform>\FXStreamDefs.cfib
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\scaleform.cfxb -> Shaders\Cache\<platform>\scaleform.cfxb
lumberyard_version\dev\cache\your_game\platform\user\shaders\cache\<platform>\stereo.cfxb -> Shaders\Cache\<platform>\stereo.cfxb

Build Platforms

You can find the build platform subfolders listed in the following table in the lumberyard_version\dev\Cache\your_game\platform\user\cache\shaders\ directory.

Build Platform Build Platform Subfolder
PC, DirectX 11 \D3D11
PC, OpenGL 4 \GL4