Menu
Sign In to the Console
Lumberyard
User Guide (Version 1.14)

AWS Documentation » Amazon Lumberyard » User Guide » Working with Shaders and Materials » Shader Rendering System » Developing a Custom Shader » Remote Shader Compiler

Remote Shader Compiler

Lumberyard's remote shader compiler application provides a convenient way to compile shaders during development. You can install the shader compiler on a local network server that can communicate over TCP. The server receives the shader source file from a computer running Lumberyard, compiles it, and sends back the shader.

The remote shader compiler is used to store all the shader combinations that have been requested by the game so far. Ideally, during development, you set up a remote shader compiler server for all designers, developers, and testers to connect to while building the game. This builds a comprehensive shader list (ShaderList.txt), which you then use to generate the shaders and compress them into .pak files for use by the game.

You can also set up the shader compiler locally on a PC rather than setting up a central remote shader compile server.

Lumberyard also generates its own ShaderList.txt, which records all of the shader combinations requested on that particular instance of Lumberyard. Using Lumberyard's local ShaderList.txt for shader generation when preparing the game for release is sufficient if the game is being developed solely on one instance of Lumberyard. Otherwise, if a team is developing multiple levels or using a very large map, some shaders may be missed by using the ShaderList.txt in the Lumberyard application's file system.

Important

Ensure the server or computer that is running the remote shader compiler is in a controlled environment that restricts incoming network requests to only authorized and trusted users or devices. Do not run the remote shader compiler with escalated root, admin, or super-user privileges.

Running the Remote Shader Compiler

You can find the remote shader compiler at \Tools\CrySCompileServer\x64\profile\.

For Visual Studio 2017, use CrySCompileServer_vc141x64.exe.

For Visual Studio 2015, use CrySCompileServer_vc140x64.exe.

A configuration file is also available for configuring the TCP port that the server application will listen on.

You can launch the remote shader compiler by starting CrySCompileServer_vc140_or_141x64.exe manually. However, usually it makes sense to set it up as a service, so that it always starts with the operating system.

Since requests for shaders are executed in parallel, you may notice significant delays in acquiring shaders at runtime.

Remote Shader Compiler Configuration

You configure the remote shader compiler by creating or editing the config.ini file, located in \Tools\CrySCompileServer\x64\profile. To configure the remote shader compiler, edit the following parameters:

  • MailError - Set to an internal company e-mail address to which notifications about compilation errors will be sent. The cache \TempDir directory in which the binary shaders are stored once they get compiled needs to point to a valid absolute path - the default is C:\SHADER_CACHE.

  • The \TempDir cache directory in which the binary shaders are stored once they got compiled must point to a valid absolute path. The default is C:\SHADER_CACHE.

  • port - TCP port, which has to match the setting in the game system_platform_shader_version.cfg file. Some examples for this file: system_windows_pc.cfg, system_osx_metal.cfg, or system_android_es3.cfg.

  • MailServer - Your email server.

  • SCMailAddress - Email address used in the From field of the email sent by the remote shader compiler.

The completed config.ini file should look similar to this example: 

MailError = shadererror@your_company.tld
MailInterval = 1
port = 61453
TempDir = C:\SHADER_CACHE
MailServer = your_email_server
SCMailAddress = RemoteShaderCompiler@your_company.tld
PrintErrors = 1

Creating a Whitelist for the Remote Shader Compiler

You can use a whitelist to specify the IP addresses that are allowed to connect to your remote shader compiler. If a computer has an IP address that is not in the whitelist, the remote shader compiler will provide a message that an invalid computer tried to connect and then close the connection. This prevents data from being read or sent on the invalid connection.

To create a whitelist for the remote shader compiler, create or edit the config.ini in the same directory as the remote shader compiler executable. Add the following parameter to the file:

  • whitelist – Provide a comma separated list of IP addresses in CIDR format. The remote shader compiler uses this list to validate incoming connection requests. The remote shader compiler automatically adds the loopback IP address (127.0.0.1) and its own IP address.

The following example allows computers or devices with an IP address of 192.168.0.1 to connect to the remote shader compiler. 

whitelist=192.168.0.1

The following example allows computers or devices with an IP address of 192.168.0.* to connect to the remote shader compiler. The /24 specifies a net mask of 24-bits. If you specify /8, any address that starts with 192 is allowed, given only an 8-bit net mask. 

whitelist=192.168.0.1/24

Specific Platforms

In the root directory of the remote shader compiler, each supported platform has its own subfolder with additional subfolders for different version numbers. All paths follow this pattern: lumberyard_version\Tools\CrySCompileServer\Compiler\platform_folder\version.

You can find information about the path used by the remote shader compiler in the file ShaderCache.cpp, under the function mfGetShaderCompileFlags.

Lumberyard provides all appropriate shader compilers for you that match the code of that version. Just copy the entire \RemoteShaderCompiler directory and run the provided binary.

Shader Cache Lists

The cache subfolder of the remote shader compiler contains different text files of all the combinations requested so far by the game. These text files are named ShaderList_platform.txt (ShaderList_DX11.txt for example) and contain all the shader combinations that have ever been requested on a certain platform for any level. These files are important as the shader .pak files cannot be generated without them. You can find these files in \dev\cache\<project_name>\pc\user\cache\shaders.

The game submits the requests to the remote shader compiler either during actual gameplay or during loading phases, even when remote shader compiling itself is disabled. This is to ensure that all possible shader combinations are collected and that the shader caches, which are generated during the shader cache generation phase, are as complete as possible.

Game Development Configuration

Having a remote shader compiler server can provide a performance benefit as it caches the results and sends them out to team members instead of having to compile shaders each time. In addition, the server keeps track of all shaders used by all people, which can be valuable if you want to make a release build that includes all shaders.

You can set the following parameters in the system_platform_shader_version.cfg in Lumberyard's root directory. For example, system_android_es3.cfg or system_windows_pc.cfg.

Turning the Remote Shader Compiler On and Off

You can configure whether the game uses the remote shader compiler in the following console variable:

r_ShadersRemoteCompiler=1

If r_ShadersRemoteCompiler is set to 0, no remote shader compilation will be performed and Lumberyard will do local shader compilation instead.

Specifying the Remote Shader Compiler Location

When the remote shader compiler is enabled, the game needs the location of the remote shader compiler. To configure the IP address of the server, use the following console variable:

r_ShaderCompilerServer=IPv4_of_PC_running_the_RemoteShaderCompiler

Using the Remote Shader Compiler Locally

You can set r_ShaderCompilerServer=localhost if you are running on a PC and want to use the remote shader compiler locally.

Using Multiple Remote Shader Compilers

It is possible to specify more than one remote shader compiler, as shown in the following example. The IP addresses need to be separated by semicolons as shown:

r_ShaderCompilerServer=10.0.0.10;10.0.0.11

Note

It is not possible to use the network name of the server instead of the IP address, since no name resolving is performed.

Specifying a Port Number

If the remote shader compile server uses a user-defined port number as specified in the config.ini file, you can configure the port number with the following console variable:

r_ShaderCompilerPort=portnumber

Disabling Request Lines

Submitting request lines to the remote shader compiler can also be disabled with the following console variable. This is useful when experimenting with shaders and you don't want to have these combinations added to the shader cache:

r_shaderssubmitrequestline=0

Proxying Remote Requests

You can use the Asset Processor to proxy remote requests to the shader compiler server if a device cannot connect to the shader compiler server. In this case, set r_AssetProcessorShaderCompiler=1. Now whenever the game would have made a request directly to the shader compiler server, it instead submits the request to the Asset Processor (this can also be over a USB connection), which then forwards it to the shader compiler server.