Menu
Lumberyard
Developer Guide (Version 1.12)

Asset Builder API

You can use the asset builder API to develop a custom asset builder that creates your own asset types. Your asset builder can process any number of asset types, generate outputs, and return the results to the asset processor for further processing. This can be especially useful in a large project that has custom asset types.

Builder Modules

A builder module is a .dll module that contains a lifecycle component and one or more builders. The lifecycle component is derived from AZ::Component. The builders can be of any type and have no particular base class requirements.


        Builder module structure

The job of the lifecycle component is to register its builders during the call to Activate() and to make sure that resources that are no longer being used are removed in the calls to Deactivate and Destructor.

Creating a Builder Module

To create a builder module, you must perform the following steps.

  • Create the exported .dll entry point functions and invoke the REGISTER_ASSETBUILDER macro, which creates a forward declaration for the entry point functions.

  • Register your lifecycle component's Descriptor

  • Add your lifecycle component to the Builder entity

  • Register your builder instances when your lifecycle component's Activate() function is called

  • Shut down safely

Note

A complete example of a builder module is in the Lumberyard dev\Code\tools\AssetProcessor\Builders directory. We recommend that you follow the commented example as you read this documentation. The asset builder SDK is located in the Lumberyard directory \dev\Code\Tools\AssetProcessor\AssetBuilderSDK\AssetBuilderSDK.

Main Entry Point

The following code shows an example of a main.cpp file for an asset builder module.

#include <AssetBuilderSDK/AssetBuilderSDK.h> #include <AssetBuilderSDK/AssetBuilderBusses.h> // Use the following macro to register this module as an asset builder. // The macro creates forward declarations of all of the exported entry points for you. REGISTER_ASSETBUILDER void BuilderOnInit() { // Perform any initialization steps that you want here. For example, you might start a third party library. } void BuilderRegisterDescriptors() { // Register your lifecycle component types here. // You can register as many components as you want, but you need at least one component to handle the lifecycle. EBUS_EVENT(AssetBuilderSDK::AssetBuilderBus, RegisterComponentDescriptor, ExampleBuilder::BuilderPluginComponent::CreateDescriptor()); // You can also register other descriptors for other types of components that you might need. } void BuilderAddComponents(AZ::Entity* entity) { // You can attach any components that you want to this entity, including management components. This is your builder entity. // You need at least one component that is the lifecycle component. entity->CreateComponentIfReady<ExampleBuilder::BuilderPluginComponent>(); } void BuilderDestroy() { // By the time you leave this function, all memory must have been cleaned up and all objects destroyed. // If you have a persistent third party library, you could destroy it here. }

Lifecycle Component

The lifecycle component reflects the types that you want to serialize and registers the builder or builders in your module during its Activate() function.

The following shows example code for the lifecycle component.

//! This is an example of the lifecycle component that you must implement. //! You must have at least one component to handle your module's lifecycle. //! You can also make this component a builder by having it register itself as a builder and //! making it listen to the builder bus. In this example it is just a lifecycle component for the purposes of clarity. class BuilderPluginComponent : public AZ::Component { public: AZ_COMPONENT(BuilderPluginComponent, "{8872211E-F704-48A9-B7EB-7B80596D871D}") static void Reflect(AZ::ReflectContext* context); BuilderPluginComponent(); // Avoid initializing here. ////////////////////////////////////////////////////////////////////////// // AZ::Component virtual void Init(); // Create objects, allocate memory and initialize without reaching out to the outside world. virtual void Activate(); // Reach out to the outside world and connect to and register resources, etc. virtual void Deactivate(); // Unregister things, disconnect from the outside world. ////////////////////////////////////////////////////////////////////////// virtual ~BuilderPluginComponent(); // free memory and uninitialize yourself. private: ExampleBuilderWorker m_exampleBuilder; };

In the following example, the Activate() function registers a builder, creates a builder descriptor, and then provides the details for the builder.

void BuilderPluginComponent::Activate() { // Activate is where you perform registration with other objects and systems. // Register your builder here: AssetBuilderSDK::AssetBuilderDesc builderDescriptor; builderDescriptor.m_name = "Example Worker Builder"; builderDescriptor.m_patterns.push_back(AssetBuilderSDK::AssetBuilderPattern("*.example", AssetBuilderSDK::AssetBuilderPattern::PatternType::Wildcard)); builderDescriptor.m_createJobFunction = AZStd::bind(&ExampleBuilderWorker::CreateJobs, &m_exampleBuilder, AZStd::placeholders::_1, AZStd::placeholders::_2); builderDescriptor.m_processJobFunction = AZStd::bind(&ExampleBuilderWorker::ProcessJob, &m_exampleBuilder, AZStd::placeholders::_1, AZStd::placeholders::_2); // The above binds are an example of how to bind to a class. You could just use a set of functions instead; there is no particular requirement to use a class. builderDescriptor.m_busId = ExampleBuilderWorker::GetUUID(); // Shutdown is communicated on this bus address. m_exampleBuilder.BusConnect(builderDescriptor.m_busId); // You can use a global listener for shutdown instead of // for each builder; it's up to you. EBUS_EVENT(AssetBuilderSDK::AssetBuilderBus, RegisterBuilderInformation, builderDescriptor); }

Notes

  • The example calls an EBus to register the builder. After you register a builder, the builder receives requests for assets from its two registered callback functions.

  • If the application needs to shut down, the asset processor broadcasts the Shutdown() message on the builder bus using the address of the registered builder's UUID.

  • Your builders do not have to be more than functions that create jobs and then process those jobs. But if you want your builder to listen for Shutdown() messages, it must have a listener that connects to the bus.

Creating a Builder

Your next step is to create a builder. You can have any number of builders, or even all of your builders, inside your module. After registering your builders as described in the previous section, implement the two CreateJobFunction and ProcessJobFunction callbacks.

The following example code declares a builder class:

#include <AssetBuilderSDK/AssetBuilderSDK.h> class ExampleBuilderWorker : public AssetBuilderSDK::AssetBuilderCommandBus::Handler // This handler delivers the "shut down!" // message on another thread. { public: ExampleBuilderWorker(); ~ExampleBuilderWorker(); //! Asset Builder Callback Functions void CreateJobs(const AssetBuilderSDK::CreateJobsRequest& request, AssetBuilderSDK::CreateJobsResponse& response); void ProcessJob(const AssetBuilderSDK::ProcessJobRequest& request, AssetBuilderSDK::ProcessJobResponse& response); ////////////////////////////////////////////////////////////////////////// //!AssetBuilderSDK::AssetBuilderCommandBus interface void ShutDown() override; // When this is received, you must fail all existing jobs and return. ////////////////////////////////////////////////////////////////////////// static AZ::Uuid GetUUID(); private: bool m_isShuttingDown = false; };

The asset processor calls the Shutdown() function to signal a shutdown. At this point, the builder should stop all tasks and return control to the asset processor.

Notes

  • Failure to terminate promptly can cause a hang when the asset processor shuts down and restarts. The shutdown message comes from a thread other than the ProcessJob() thread.

  • The asset processor calls the CreateJobs(const CreateJobsRequest& request,CreateJobsResponse& response) function when it has jobs for the asset types that the builder processes. If no work is needed, you do not have to create jobs in response to CreateJobsRequest, but the behavior of your implementation should be consistent.

  • For the purpose of deleting stale products, the job that you spawn is compared with the jobs spawned in the last iteration that have the same input, operating system, and job key.

  • You do not have to check whether a job needs processing. Instead, at every iteration, emit all possible jobs for a particular input asset on a particular operating system.

  • In general, in the CreateJobs function, you create a job descriptor for each job that you want to emit, and then add the job to the list of job descriptors for the response.

The following code shows an example CreateJobs function.

// This function runs early in the file scanning pass. // This function should always create the same jobs, and should not check whether the job is up to date. void ExampleBuilderWorker::CreateJobs(const AssetBuilderSDK::CreateJobsRequest& request, AssetBuilderSDK::CreateJobsResponse& response) { // The following example creates one job descriptor for the PC operating system. // Normally, you create a job for each operating system that you can make assets for. if (request.m_platformFlags & AssetBuilderSDK::Platform_PC) { AssetBuilderSDK::JobDescriptor descriptor; descriptor.m_jobKey = "Compile Example"; descriptor.m_platform = AssetBuilderSDK::Platform_PC; // You can also place whatever parameters you want to save for later into this map: descriptor.m_jobParameters[AZ_CRC("hello")] = "World"; response.m_createJobOutputs.push_back(descriptor); response.m_result = AssetBuilderSDK::CreateJobsResultCode::Success; } }

How to Declare Source File Dependencies in CreateJobs

You can use the Builder SDK API to declare dependencies for a source file on other source files. These other files can be any file within the project directory or directories. They do not have to be source files consumed by a builder.

Declaring dependencies for a source file implies that the data in its output changes if the files that the source file depends on change. If any of the source dependency files are modified, the Asset Processor retriggers the CreateJobs and ProcessJobs sequence for those files. This causes the data to be recompiled by your builder when needed.

The Asset Processor recurses source file dependencies automatically. If the source files depended on emit their own dependencies when they are queried, you do not have to recurse in your own source files to the full tree. Just emit your local dependencies for each node in the tree, and the Asset Processor takes care of the rest.

To declare dependencies, add them during CreateJobs to m_sourceFileDependencyList in your CreateJobsResponse structure.

Metafiles do not have to be added as dependencies. Metafiles are a special case and cause your asset to rebuild automatically.

The SourceFileDependency structure contains m_sourceFileDependencyPath and m_sourceFileDependencyUUID. The builder must supply a value for only one of these fields. For example, if the UUID of the file to be added as a source dependency is known inside the CreateJobs method, the builder can just fill in the field m_sourceFileDependencyUUID. Otherwise the builder would need to fill in the field m_sourceFileDependencyPath with the appropriate source dependency file path.

It is important to note that the field m_sourceFileDependencyPath can take both absolute and relative file paths. If a relative path is specified, the appropriate overriding asset is used if present.

If the builder is populating the m_sourceFileDependencyPath field with a relative file path, then it has to be in relative to one of the watched directories. However, if both the source and the source dependency file exist in the same directory, you can provide just the filename without a path.

The following code snippet that shows how to use the Builder SDK API to add a source file dependency.

//! CreateJobsResponse contains job data that will be send by the builder to the assetProcessor in response to CreateJobsRequest struct CreateJobsResponse { CreateJobsResultCode m_result = CreateJobsResultCode::Failed; // The result code from the create jobs request AZStd::vector<SourceFileDependency> m_sourceFileDependencyList; // This is required for source files that want to declare dependencies on other source files. AZStd::vector<JobDescriptor> m_createJobOutputs; }; //! Source file dependency information that the builder will send to the assetprocessor //! It is important to note that the builder do not need to provide both the sourceFileDependencyUUID or sourceFileDependencyPath info to the asset processor, //! any one of them should be sufficient struct SourceFileDependency { //! Filepath on which the source file depends, it can be either be a relative or an absolute path. //! if it's relative, the asset processor will check every watch folder in the order specified in the assetprocessor config file until it finds that file. //! For example if the builder sends the sourcedependency info with sourceFileDependencyPath = "texture/blah.tiff" to the asset processor, //! it will check all watch folders for a file whose relative path with regard to it is "texture/blah.tiff". //! then "C:/dev/gamename/texture/blah.tiff" would be considered the source file dependency, if "C:/dev/gamename" is only watchfolder that contains such a file. //! You can also send absolute path to the asset processor in which case the asset processor //! will try to determine if there are any other file which overrides this file based on the watch folder order specified in the assetprocessor config file //! and if an overriding file is found, then that file will be considered as the source dependency. AZStd::string m_sourceFileDependencyPath; //! UUID of the file on which the source file depends AZ::Uuid m_sourceFileDependencyUUID = AZ::Uuid::CreateNull(); ... } void ExampleBuilderWorker::CreateJobs(const AssetBuilderSDK::CreateJobsRequest& request, AssetBuilderSDK::CreateJobsResponse& response) { //Looping over all the available operating systems to spawn jobs. Note that source dependencies are independent of jobs and operating systems. // This example shows how you would be able to declare source file dependencies on source files inside a builder and forward the info to the asset processor // Basically here we are creating source file dependencies as follows // the source file .../test.examplesource depends on the source file .../test.exampleinclude and // the source file .../test.exampleinclude depends on the source file .../common.exampleinclude // the source file .../common.exampleinclude depends on the non-source file .../common.examplefile // Important to note that both file extensions "exampleinclude" and "examplesource" are being handled by this builder. // Also important to note is that files with extension "exampleinclude" are not creating any jobdescriptor here, which imply they do not create any jobs. AZStd::string fullPath; AzFramework::StringFunc::Path::ConstructFull(request.m_watchFolder.c_str(), request.m_sourceFile.c_str(), fullPath, false); AzFramework::StringFunc::Path::Normalize(fullPath); AZStd::string relPath = request.m_sourceFile; AssetBuilderSDK::SourceFileDependency sourceFileDependencyInfo; // source files in this example generate dependencies and also generate a job to compile it for each operating system: if (AzFramework::StringFunc::Equal(ext.c_str(), "examplesource")) { AzFramework::StringFunc::Path::ReplaceExtension(relPath, "exampleinclude"); // declare and add the dependency on the .exampleinclude file: sourceFileDependencyInfo.m_sourceFileDependencyPath = relPath; response.m_sourceFileDependencyList.push_back(sourceFileDependencyInfo); // since we're a source file, we also add a job to do the actual compilation: for (size_t idx = 0; idx < request.GetEnabledPlatformsCount(); idx++) { AssetBuilderSDK::JobDescriptor descriptor; descriptor.m_jobKey = "Compile Example"; descriptor.m_platform = request.GetEnabledPlatformAt(idx); // you can also place whatever parameters you want to save for later into this map, and it will be available in your process function descriptor.m_jobParameters[AZ_CRC("hello", 0x3610a686)] = "World"; // add the job: response.m_createJobOutputs.push_back(descriptor); } } else if (AzFramework::StringFunc::Equal(ext.c_str(), "exampleinclude")) { // if we're processing an 'include file' then we still generate dependency information but emit no actual compilation jobs // similar to how CPP/H works. if (AzFramework::StringFunc::Find(request.m_sourceFile.c_str(), "common.exampleinclude") != AZStd::string::npos) { // Add any dependencies that common.exampleinclude would like to depend on here, we can also add a non source file as a dependency like we are doing here sourceFileDependencyInfo.m_sourceFileDependencyPath = "common.examplefile"; } else { AzFramework::StringFunc::Path::ReplaceFullName(fullPath, "common.exampleinclude"); // Assigning full path to sourceFileDependency path sourceFileDependencyInfo.m_sourceFileDependencyPath = fullPath; } response.m_sourceFileDependencyList.push_back(sourceFileDependencyInfo); } response.m_result = AssetBuilderSDK::CreateJobsResultCode::Success; }

CreateJobsRequest Helper Functions

CreateJobsRequest provides helper functions for operations related to the operating system. The following code shows the helper functions provided by CreateJobsRequest.

//! Enum used by the builder for sending operating system information. enum Platform: AZ::u32 { Platform_NONE = 0x00, Platform_PC = 0x01, Platform_ES3 = 0x02, Platform_IOS = 0x04, Platform_OSX = 0x08, //! If you add a new entry to this enum, you must also add it to AllPlatforms in order for the entry to be considered valid. AllPlatforms = Platform_PC | Platform_ES3 | Platform_IOS | Platform_OSX }; //! CreateJobsRequest contains input job data that is sent by the AssetProcessor to the builder to create jobs. struct CreateJobsRequest { ... //! Platform flags informs the builder about the operating systems in which the AssetProcessor is interested. int m_platformFlags; // Return the number of operating systems that are enabled for the source file. size_t GetEnabledPlatformsCount() const; // Return the enabled operating system by index. If no operating system is found, then Platform_NONE is returned. AssetBuilderSDK::Platform GetEnabledPlatformAt(size_t index) const; // Determine whether the inputted operating system is enabled. Returns true if enabled, otherwise false. bool IsPlatformEnabled(AZ::u32 platform) const; // Determine whether the inputted operating system is valid. Returns true if valid, otherwise false. bool IsPlatformValid(AZ::u32 platform) const; };
  • GetEnabledPlatformsCount() – For the specified flags, returns the number of operating systems that are enabled for the create jobs request.

  • IsPlatformEnabled(AZ::u32 platform) – For the specified operating system, returns whether it is enabled. True if the operating system is enabled, false otherwise.

  • IsPlatformValid(AZ::u32 platform) – For the specified value, checks whether it is a valid operating system value in the enum. True if the operating system value is valid, false otherwise.

ProcessJob

The asset processor calls the ProcessJob function when it has a job for the builder to begin processing:

ProcessJob(const AssetBuilderSDK::ProcessJobRequest& request, AssetBuilderSDK::ProcessJobResponse& response)

ProcessJob is given a job request that contains the full job descriptor that CreateJobs emitted, as well as additional information such as a temporary directory for it to work in.

This message is sent on a worker thread, so the builder must not spawn threads to do the work. Be careful not to interact with other threads during this call.

Warning

Do not alter files other than those in the temporary directory while ProcessJob is running. After your job indicates success, the asset processor copies your registered products to the asset cache, so be sure not to write to the cache. You can use the temporary directory in any way that you want.

After your builder has finished processing assets, your response structure should list all of the assets that you have created. Because only the assets that you list are added to the cache, you can use the temporary directory as a scratch space for processing.

The following code shows an example ProcessJob function.

// This function is called for jobs that need processing. // The request contains the CreateJobResponse you constructed earlier, including the // keys and values you placed into the hash table. void ExampleBuilderWorker::ProcessJob(const AssetBuilderSDK::ProcessJobRequest& request, AssetBuilderSDK::ProcessJobResponse& response) { // The following example shows how to listen for cancellation requests. You should listen for cancellation requests and then cancel work if possible. // You can derive from the job cancel listener and reimplement Cancel() if you need to do more processing like signaling a // semaphore or doing other threading work. AssetBuilderSDK::JobCancelListener jobCancelListener(request.m_jobId); AZ_TracePrintf(AssetBuilderSDK::InfoWindow, "Starting Job."); AZStd::string fileName; AzFramework::StringFunc::Path::GetFullFileName(request.m_fullPath.c_str(), fileName); AzFramework::StringFunc::Path::ReplaceExtension(fileName, "example1"); AZStd::string destPath; // Do all of your work inside the tempDirPath. // Do not write outside of this path AzFramework::StringFunc::Path::ConstructFull(request.m_tempDirPath.c_str(), fileName.c_str(), destPath, true); // Use AZ_TracePrintF to communicate job details. The logging system automatically places the // text in the appropriate log file and category. AZ::IO::LocalFileIO fileIO; if (!m_isShuttingDown && !jobCancelListener.IsCancelled() && fileIO.Copy(request.m_fullPath.c_str(), destPath.c_str()) == AZ::IO::ResultCode::Success) { // If assets were successfully built into the temporary directory, push them back into the response's product list. // The assets that you created in your temporary path can be specified using paths relative to the temporary path. // It is assumed that your code writes to the temporary path. AZStd::string relPath = destPath; response.m_resultCode = AssetBuilderSDK::ProcessJobResult_Success; AssetBuilderSDK::JobProduct jobProduct(fileName); response.m_outputProducts.push_back(jobProduct); } else { if (m_isShuttingDown) { AZ_TracePrintf(AssetBuilderSDK::ErrorWindow, "Cancelled job %s because shutdown was requested", request.m_fullPath.c_str()); response.m_resultCode = AssetBuilderSDK::ProcessJobResult_Cancelled; } else { AZ_TracePrintf(AssetBuilderSDK::ErrorWindow, "Error during processing job %s.", request.m_fullPath.c_str()); response.m_resultCode = AssetBuilderSDK::ProcessJobResult_Failed; } } }

JobCancelListener

Builders can use JobCancelListener to listen for job cancellation requests in their processJob method. You should listen for cancellation requests and then cancel work if possible. The address of this listener is the job ID of the job that can be found in processJobRequest. You can derive from the JobCancelListener and reimplement Cancel() if you must do additional processing like signaling a semaphore or other threading work. The following code example shows the use of JobCancelListener.

//! JobCancelListener can be used by builders in their processJob method to listen for job cancellation requests. //! The address of this listener is the job ID that can be found in the process job request. class JobCancelListener : public JobCommandBus::Handler { public: explicit JobCancelListener(AZ::u64 jobId); ~JobCancelListener() override; JobCancelListener(const JobCancelListener&) = delete; ////////////////////////////////////////////////////////////////////////// //JobCommandBus::Handler overrides //Note: This is called on a thread other than your processing job thread. //You can derive from JobCancelListener and reimplement Cancel if you need to do something special in order to cancel your job. void Cancel() override; /////////////////////////////////////////////////////////////////////// bool IsCancelled() const; private: AZStd::atomic_bool m_cancelled; };

Notes

  • So that critical files are not missed, the editor is blocked until all jobs are created. For this reason, you should execute the code in CreateJobs as quickly as possible. We recommend that your code do minimal work during CreateJobs and save the heavy processing work for ProcessJob.

  • In CreateJobs, you can place arbitrary key–value pairs into the descriptor's m_jobParameters field. They key–value pairs are copied back when ProcessJob executes, which removes the need for you to add them again.

  • All of the outputs for your job should be placed into your temporary workspace. However, if you just need to copy an existing file into the asset cache as part of your job, you can emit as a product the full absolute source path of the file without copying it into your temporary directory first. The asset processor then copies the file into the cache and registers it as part of the output of your job. All other files are moved from your temporary directory into the asset cache in an attempt to perform an atomic cache update in case your job succeeds.

Message Loggging

You can use BuilderLog(AZ:Uuid builderId, char* message, ...) to log any general builder related messages or errors. BuilderLog cannot be used during job processing; use it during startup, shutdown, or registration.

For job related messages, use AZ_TracePrintf(window, msg), AZ_Warning(...), AZ_Error(...), AZ_Assert(...), and so on. . The function automatically records the messages in the log file for the job.