SageMakerTrainingSessionRuntime / Client / execute_sampling_operation

execute_sampling_operation

SageMakerTrainingSessionRuntime.Client.execute_sampling_operation(**kwargs)

Submits an asynchronous sampling operation to generate text or compute log-probabilities from the model associated with the specified training session. The session must be in Running or Blocked state. Use GetSamplingOperationResult to poll for the result.

See also: AWS API Documentation

Request Syntax

response = client.execute_sampling_operation(
    trainingSessionName='string',
    operationConfig={
        'sample': {
            'prompt': {
                'chunks': [
                    {
                        'tokens': [
                            123,
                        ],
                        'type': 'encoded_text'
                    },
                ]
            },
            'numSamples': 123,
            'samplingParams': {
                'maxTokens': 123,
                'temperature': 123.0,
                'topK': 123,
                'topP': 123.0,
                'seed': 123,
                'stop': [
                    'string',
                ]
            },
            'includePromptLogprobs': True|False,
            'topkPromptLogprobs': 123,
            'modelPackageArn': 'string'
        },
        'computeLogprobs': {
            'prompt': {
                'chunks': [
                    {
                        'tokens': [
                            123,
                        ],
                        'type': 'encoded_text'
                    },
                ]
            },
            'modelPackageArn': 'string'
        },
        'loadWeightsForSampling': {
            'modelPackageArn': 'string'
        }
    }
)
Parameters:

  • trainingSessionName (string) –

    [REQUIRED]

    The name of the training session to sample from.

  • operationConfig (dict) –

    [REQUIRED]

    The configuration for the sample operation. Specify either a sample configuration to generate text or a computeLogprobs configuration to compute token log-probabilities.

    Note

    This is a Tagged Union structure. Only one of the following top level keys can be set: sample, computeLogprobs, loadWeightsForSampling.

    • sample (dict) –

      The configuration for generating text samples from the model.

      • prompt (dict) – [REQUIRED]

        The input prompt to send to the model for text generation.

        • chunks (list) – [REQUIRED]

          The ordered list of encoded text chunks forming the input sequence.

          • (dict) –

            A chunk of pre-tokenized text represented as a list of token IDs.

            • tokens (list) – [REQUIRED]

              The token IDs representing the encoded text.

              • (integer) –

            • type (string) – [REQUIRED]

              The type of the chunk encoding.

      • numSamples (integer) – [REQUIRED]

        The number of independent samples to generate for the prompt.

      • samplingParams (dict) – [REQUIRED]

        The parameters controlling the sampling strategy.

        • maxTokens (integer) – [REQUIRED]

          The maximum number of tokens to generate per sample.

        • temperature (float) –

          The sampling temperature controlling randomness. Higher values produce more diverse outputs.

        • topK (integer) –

          The number of highest-probability vocabulary tokens to keep for top-k filtering. Set to -1 to disable.

        • topP (float) –

          The cumulative probability threshold for nucleus (top-p) sampling. Only tokens with cumulative probability below this threshold are considered.

        • seed (integer) –

          The random seed for reproducible sampling. When set, the same seed with the same input produces deterministic outputs.

        • stop (list) –

          The list of stop sequences that terminate generation when encountered in the output.

          • (string) –

      • includePromptLogprobs (boolean) –

        Specifies whether to include per-token log-probabilities for the input prompt in the response.

      • topkPromptLogprobs (integer) –

        The number of top-k log-probabilities to return per prompt token.

      • modelPackageArn (string) –

        The ARN of the model package whose checkpoint this sample should run against. Must be a valid SageMaker model package ARN in the format arn:aws:sagemaker:REGION:ACCOUNT:model-package/PACKAGE_NAME/VERSION. When absent, the sample runs against the session’s latest checkpoint.

    • computeLogprobs (dict) –

      The configuration for computing log-probabilities of prompt tokens without generating new text.

      • prompt (dict) – [REQUIRED]

        The input prompt whose token log-probabilities will be computed.

        • chunks (list) – [REQUIRED]

          The ordered list of encoded text chunks forming the input sequence.

          • (dict) –

            A chunk of pre-tokenized text represented as a list of token IDs.

            • tokens (list) – [REQUIRED]

              The token IDs representing the encoded text.

              • (integer) –

            • type (string) – [REQUIRED]

              The type of the chunk encoding.

      • modelPackageArn (string) –

        The ARN of the model package whose checkpoint this operation should run against. Must be a valid SageMaker model package ARN in the format arn:aws:sagemaker:REGION:ACCOUNT:model-package/PACKAGE_NAME/VERSION. When absent, the operation runs against the session’s latest checkpoint.

    • loadWeightsForSampling (dict) –

      The configuration for loading model weights for inference and sampling without restoring optimizer state.

      • modelPackageArn (string) – [REQUIRED]

        The ARN of the model package containing the weights to load for sampling. Must be a valid SageMaker model package ARN in the format arn:aws:sagemaker:REGION:ACCOUNT:model-package/PACKAGE_NAME/VERSION. The model package must belong to the same AWS account as the training session.

Return type:

dict

Returns:

Response Syntax

{
    'operationId': 'string'
}

Response Structure

  • (dict) –

    • operationId (string) –

      The unique identifier for the submitted sample operation. Use this identifier with GetSamplingOperationResult to retrieve the operation status and result.

Exceptions