Class: AWS::Flow::S3DataConverter

Inherits:
Object
  • Object
show all
Defined in:
aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb

Overview

Note:

The AWS Flow Framework for Ruby doesn't delete files from S3 to prevent loss of data. It is recommended that you use [Object Lifecycle Management][olm] in Amazon S3 to automatically delete files after a certain period.

S3DataConverter serializes/deserializes Ruby objects using YAMLDataConverter, storing the serialized data on Amazon S3. This data can exceed 32K (32,768) characters in size, providing a way to work past Amazon SWF's input/output data limit.

To activate it, set the AWS_SWF_BUCKET_NAME environment variable to the name of an Amazon S3 bucket to use to store workflow/activity data. The bucket will be created if necessary.

S3DataConverter caches data on the local system. The cached version of the file's data is used if it is found. Otherwise, the file is downloaded from Amazon S3 and then deserialized to a Ruby object.

S3DataConverter serializes Ruby objects using YAMLDataConverter and stores them in the Amazon S3 bucket specified by AWS_SWF_BUCKET_NAME, using a randomly-generated filename to identify the object's data.

[olm]: http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html

Class Attribute Summary (collapse)

Instance Attribute Summary (collapse)

Class Method Summary (collapse)

Instance Method Summary (collapse)

Constructor Details

- (S3DataConverter) initialize(bucket)

Note:

The default data converter specified by FlowConstants.default_data_converter will be used to serialize and deserialize the data. Ordinarily, this is YAMLDataConverter.

Initialize a new S3DataConverter, providing it the name of an Amazon S3 bucket that will be used to store serialized Ruby objects. The bucket will be created if it doesn't already exist.

Parameters:

  • bucket (String)

    The Amazon S3 bucket name to use for serialized data storage.



150
151
152
153
154
155
156
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 150

def initialize(bucket)
  @bucket = bucket
  @cache = S3Cache.new
  s3 = AWS::S3.new
  s3.buckets.create(bucket) unless s3.buckets[bucket].exists?
  @converter = FlowConstants.default_data_converter
end

Class Attribute Details

+ (Object) conv

Returns the value of attribute conv



122
123
124
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 122

def conv
  @conv
end

Instance Attribute Details

- (Object) bucket (readonly)

Returns the value of attribute bucket



119
120
121
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 119

def bucket
  @bucket
end

- (Object) cache (readonly)

Returns the value of attribute cache



119
120
121
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 119

def cache
  @cache
end

- (Object) converter (readonly)

Returns the value of attribute converter



119
120
121
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 119

def converter
  @converter
end

Class Method Details

+ (Object) converter

Ensures singleton



125
126
127
128
129
130
131
132
133
134
135
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 125

def converter
  return self.conv if self.conv
  name = ENV['AWS_SWF_BUCKET_NAME']
  if name.nil?
    raise "Need a valid S3 bucket name to initialize S3DataConverter."\
      " Please set the AWS_SWF_BUCKET_NAME environment variable with the"\
      " bucket name."
  end
  self.conv ||= self.new(name)
  return self.conv
end

Instance Method Details

- (String) dump(object)

Serializes a Ruby object into a string (by default, YAML). The resulting data, if > 32,768 (32K) characters, is uploaded to the bucket specified when the S3DataConverter was initialized and is copied to the local cache.

Parameters:

  • object (Object)

    The object to be serialized into a string. By default, the framework serializes the object into a YAML string using YAMLDataConverter.

Returns:

  • (String)

    The file's serialized data if the resulting data is < 32,768 (32K) characters.

    If the resulting data is > 32K, then the file is uploaded to S3 and a YAML string is returned that represents a hash of the following form:

    { s3_filename: <filename> }
    

    The returned filename is randomly-generated, and follows the form:

    rubyflow_data_<UUID>
    


182
183
184
185
186
187
188
189
190
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 182

def dump(object)
  string = @converter.dump(object)
  ret = string
  if string.size > 32768
    filename = put_to_s3(string)
    ret = @converter.dump({ s3_filename: filename })
  end
  ret
end

- (Object) load(source)

Deserializes a string into a Ruby object. If the deserialized string is a Ruby hash of the format: { s3_filename: <filename> }, then the local cache is searched for the file's data. If the file is not cached, then the file is downloaded from Amazon S3, deserialized, and its data is copied to the cache.

Parameters:

  • source (String)

    The source string that needs to be deserialized into a Ruby object. By default it expects the source to be a YAML string that was serialized using #dump.

Returns:

  • (Object)

    A Ruby object created by deserializing the YAML source string.



206
207
208
209
210
211
212
213
# File 'aws-flow-ruby/aws-flow/lib/aws/decider/data_converter.rb', line 206

def load(source)
  object = @converter.load(source)
  ret = object
  if object.is_a?(Hash) && object[:s3_filename]
    ret = @converter.load(get_from_s3(object[:s3_filename]))
  end
  ret
end