Shader Cache and Generation
This section discusses the shader cache and how to generate shader cache .pak
files.
Shader Cache
The shader cache stores a collection of parsed and precompiled shaders. Since the shader code is written with multiple definitions, Lumberyard can generate an enormous number of different shaders. Compiling shaders on demand at run time is only possible on the PC. On-demand shader compiling can cause freezes during the game play and uses extra memory. To reduce this overhead, all required shader combinations for a game are parsed, compiled, and stored in the shader cache.
The shader cache generally refers to the following files:
-
Shaders.pak
– Contains the shader source files, which is everything inside the
directory, excludinglumberyard_version
\dev\Engine\Shaders\EngineAssets
.Note The actual shader source code (
*.cfi
and*.cfx
) can be removed from this file for the final released version, and is not required anymore when the binary shaders are valid and available. -
ShadersBin.pak
– Contains the binary-parsed shader information of the shader source code. -
ShaderCache.pak
– Contains compiled shaders for all possible combinations that have been submitted to the remote shader compiler. -
ShaderCacheStartup.pak
– Contains a small subset of the shader cache that is used during game start. This file is loaded into memory for quicker start times, but is not required. This cache is often used to hold the minimum required set of shaders to show a loading screen so that the rest of the loading can occur.
ShaderCache.pak File Generation
Creating a ShaderCache.pak
file consists of running the
lmbr_pak_shaders.bat
batch script, which in turn runs
ShaderCacheGen.exe
to ensure the local cache directory contains
all the shaders that are listed in the ShaderList.txt
file. The
lmbr_pak_shaders.bat
script packs the contents of the cache
directory, creates a ShaderCache.zip
file, and then renames the
file to ShaderCache.pak
.
You can find the
ShaderList_
file either
from the remote shader compiler server or from the Lumberyard Editor directory. This
file
contains the list of all shaders which platform
.txtShaderCacheGen.exe
uses to
produce the shader combinations for your game.
When running Lumberyard Editor, individual shaders are created as you view them. As
such,
you do not strictly need a remote shader compiler server to test game release mode
or
test shader pack generation. You just need access to the
ShaderList_
file
that is created in the
platform
.txt
directory when running Lumberyard Editor.
lumberyard_version
\dev\cache\game_name
\platform
\user\cache\shaders
However, only the shaders you have viewed on your local computer while running
Lumberyard Editor will be listed in the
ShaderList_
file.
For this reason, it is recommended that you use a remote shader compiler server if
possible.
platform
.txt
During development time, when you run the game or Lumberyard Editor, and before the
shaders are packed into shader cache .pak
files, loose shader
files are created in the following directory:
lumberyard_version
\Dev\Cache\your_game
\platform
\user\cache
The following sections detail the steps used to generate
ShaderCache.pak
files:
ShaderCacheGen.exe
Lumberyard ships with ShaderCacheGen.exe
, which is located in the
directory. For macOS, the file is located in the lumberyard_version
\dev\Bin64\BinMac64
directory. ShaderCacheGen.exe
is a stripped-down version of the
Lumberyard game launcher without the render viewport, and is used to populate the
local
shader cache directory with all the shaders contained in the
ShaderList.txt
file.
Running the ShaderCacheGen.exe
will load the
ShaderCacheGen.cfg
file. This configuration file includes
the IP address and other settings for the remote shader compiler that generates the
shaders. The ShaderCacheGen.exe
must connect to the remote
shader compiler that's specified in the configuration file. The
ShaderCacheGen.exe
will not use the IP address and other
settings that are specified in the system configuration files.
If you customized Lumberyard in any way, you must build Lumberyard and your game using
the
all
profile. This argument builds both
ShaderCacheGen.exe
(and ensures that it is up-to-date) and
the game .dll
files that are required.
Enter the following command for your version of Visual Studio.
lmbr_waf build_win_x64_vs
2017
_profile -p all --targets=CrySCompileServer
If you don’t want to (or can't) build using the all
profile, you can
alternatively just build the game_and_engine
spec and the
shadercachegen
spec.
Enter the following command for your version of Visual Studio.
lmbr_waf build_win_x64_vs
2017
_profile -p game_and_engine lmbr_waf build_win_x64_vs2017
_profile -p shadercachegen
Packing the Shader Cache as Part of the Release Build
The ShaderCache.pak
files are generated for release builds as
part of the build process. To generate the files, the
ShaderCacheGen.exe
must be connected to the Remote Shader
Compiler. This must be the same Remote Shader Compiler that you used to generate the
shaders for your game. As long as this instance of the shader compiler is running,
the release build process can generate the necessary .pak
files for
your release build. If the shader compiler isn't running or reachable for some
reason, the release build process fails.
Packing the Shader Cache Using a Batch File
The lmbr_pak_shaders.bat
file generates the
ShaderCache.pak
files, which are saved to the
lumberyard_version\dev\build\
directory. The batch file first calls the platform
\your_game
ShaderCacheGen.exe
and then calls Tools\PakShaders\pak_shaders.py
.
From a command line, navigate to the
directory, and run the lumberyard_version
\devlmbr_pak_shaders.bat
, and specify the
location to the
ShaderList_
file.
platform
.txt
Example
F:\
lumberyard_version
\dev\lmbr_pak_shaders.batgame_project_name
D3D11 pc C:\shader_compiler_server
\ShaderList_DX11.txt
Once the shader .pak
files are created, you can move them as
needed. For example, if you already built a release version of your game, you can
place them with the rest of the .pak
files.
Packing the Shader Cache Manually
If you want to use more complex build pipelines, you can pack the shader cache manually.
To pack the shader cache manually
-
Run
ShaderCacheGen.exe
to generate the shader cache so you can pack it later. -
Zip all the shaders into a
ShaderCache.zip
file, then rename the file toShaderCache.pak
.
Each platform has different .pak
files. See the following
directory mapping for the PC platforms:
The PC platform should copy data from the following directory:
lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\D3D11\
into the following destination directory:
shaders\cache\D3D11\
ShaderCache.pak
should contain everything from the previously
listed subfolders.
ShadersBin.pak
should contain only the
*.cfxb
and *.cfib
files.
ShaderCacheStartup.pak
should contain the following
files:
lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\lookupdata.bin -> Shadercache\<platform>\lookupdata.binlumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\CGPShader\FixedPipelineEmu* -> Shadercache\<platform>\CGPShader\FixedPipelineEmu*lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\CGPShader\Scaleform* -> Shadercache\<platform>\CGPShader\Scaleform*lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\CGPShader\Stereo* -> Shadercache\<platform>\CGPShader\Stereo*lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\CGVShader\FixedPipelineEmu* -> Shadercache\<platform>\CGVShader\FixedPipelineEmu*lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\CGVShader\Scaleform* -> Shadercache\<platform>\CGVShader\Scaleform*lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\CGVShader\Stereo* -> Shadercache\<platform>\CGVShader\Stereo*lumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\lookupdata.bin -> Shaders\Cache\<platform>\lookupdata.binlumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\Common.cfib -> Shaders\Cache\<platform>\Common.cfiblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\fallback.cfxb -> Shaders\Cache\<platform>\fallback.cfxblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\fixedpipelineemu.cfxb -> Shaders\Cache\<platform>\fixedpipelineemu.cfxblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\FXConstantDefs.cfib -> Shaders\Cache\<platform>\FXConstantDefs.cfiblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\FXSamplerDefs.cfib -> Shaders\Cache\<platform>\FXSamplerDefs.cfiblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\FXSetupEnvVars.cfib -> Shaders\Cache\<platform>\FXSetupEnvVars.cfiblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\FXStreamDefs.cfib -> Shaders\Cache\<platform>\FXStreamDefs.cfiblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\scaleform.cfxb -> Shaders\Cache\<platform>\scaleform.cfxblumberyard_version
\dev\cache\your_game
\platform
\user\shaders\cache\<platform>\stereo.cfxb -> Shaders\Cache\<platform>\stereo.cfxb
Build Platforms
You can find the build platform subfolders listed in the following table in the
directory.
lumberyard_version
\dev\Cache\your_game
\platform
\user\cache\shaders\
Build Platform | Build Platform Subfolder |
---|---|
PC, DirectX 11 | \D3D11 |