Compute Farm Cloud Gem - Lumberyard User Guide

Compute Farm Cloud Gem

This cloud Gem is deprecated and no longer supported. Some functionality is broken because of its dependency on the Cloud Gem Portal, which was removed in Lumberyard 1.28. Complete documentation for this Gem from previous versions of Lumberyard can be found in the Documentation Archive.

The Compute Farm cloud gem is a tool for large-scale divide and conquer tasks that can be processed on a fleet of Amazon EC2 instances. This cloud gem is particularly suited for heavy-duty, computationally intensive Windows tasks like nav mesh generation, static lightmap baking, and terrain generation. In general, the Compute Farm cloud gem is designed for tasks that can be highly parallelized with recursive subdivision.

The Compute Farm cloud gem has the following three major aspects:

  1. Harness – The brain of the cloud gem that orchestrates the task. The harness is a Python script that runs on every Amazon EC2 instance. The script continuously polls the Amazon Simple Workflow Service (SWF) for decisions that need to be made and activities that must be run. You can customize the harness for your own purposes.

  2. Amazon Machine Image (AMI) – An image of the machine to do the task. The AMI must be prepared with all the necessary software to perform the task as well as a copy of the harness. The Compute Farm cloud gem includes a script to build the AMI.

  3. Launch Configuration and Auto Scaling Group – After the AMI is created, you must create an Amazon EC2 Launch Configuration and an Amazon EC2 Auto Scaling Group to control the fleet of Amazon EC2 instances that will run the AMI. You set the Auto Scaling Group to the desired number of Amazon EC2 instances.

Workflow Summary

The harness receives and processes events from Amazon SWF, and runs three categories of task: divide, build and merge.

The harness must be extended with Python scripts that perform these three tasks for the target. In the dictionary sorting example that is included with the cloud gem, these tasks are the following:

  • Divide a dictionary of words into multiple parts.

  • Build a subset of the dictionary by sorting it.

  • Merge the two sorted subsets back into one sorted subset.

For a high-level explanation and demonstration of the Compute Farm cloud gem for large scale terrain generation, see the 2018 GDC classroom session on YouTube. Developers who are interested in this particular implementation can reach out to us at lumberyard-feedback@amazon.com.

Account Limits

Your use of the Compute Farm cloud gem is subject to the limits on your AWS account. The following are the most relevant limits that you are likely to encounter:

To request limit increases, see the AWS Support Center.