Interface CfnDataSourcePropsMixin.IWebCrawlerConfigurationProperty

The configuration of web URLs that you want to crawl.

Namespace: Amazon.CDK.CfnPropertyMixins.AWS.Bedrock

Assembly: Amazon.CDK.CfnPropertyMixins.dll

Syntax (csharp)

public interface CfnDataSourcePropsMixin.IWebCrawlerConfigurationProperty

Syntax (vb)

Public Interface CfnDataSourcePropsMixin.IWebCrawlerConfigurationProperty

Remarks

You should be authorized to crawl the URLs.

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html

ExampleMetadata: fixture=_generated

Examples

// The code below shows an example of how to instantiate this type.
             // The values are placeholders you should change.
             using Amazon.CDK.CfnPropertyMixins.AWS.Bedrock;

             var webCrawlerConfigurationProperty = new WebCrawlerConfigurationProperty {
                 CrawlerLimits = new WebCrawlerLimitsProperty {
                     MaxPages = 123,
                     RateLimit = 123
                 },
                 ExclusionFilters = new [] { "exclusionFilters" },
                 InclusionFilters = new [] { "inclusionFilters" },
                 Scope = "scope",
                 UserAgent = "userAgent",
                 UserAgentHeader = "userAgentHeader"
             };

Synopsis

Properties

CrawlerLimits	The configuration of crawl limits for the web URLs.
ExclusionFilters	A list of one or more exclusion regular expression patterns to exclude certain URLs.
InclusionFilters	A list of one or more inclusion regular expression patterns to include certain URLs.
Scope	The scope of what is crawled for your URLs.
UserAgent	Returns the user agent suffix for your web crawler.
UserAgentHeader	A string used for identifying the crawler or bot when it accesses a web server.

Properties

CrawlerLimits

The configuration of crawl limits for the web URLs.

object? CrawlerLimits { get; }

Property Value

object

Remarks

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html#cfn-bedrock-datasource-webcrawlerconfiguration-crawlerlimits

Type union: either IResolvable or CfnDataSourcePropsMixin.IWebCrawlerLimitsProperty

ExclusionFilters

A list of one or more exclusion regular expression patterns to exclude certain URLs.

string[]? ExclusionFilters { get; }

Property Value

string[]

Remarks

If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html#cfn-bedrock-datasource-webcrawlerconfiguration-exclusionfilters

InclusionFilters

A list of one or more inclusion regular expression patterns to include certain URLs.

string[]? InclusionFilters { get; }

Property Value

string[]

Remarks

If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html#cfn-bedrock-datasource-webcrawlerconfiguration-inclusionfilters

Scope

The scope of what is crawled for your URLs.

string? Scope { get; }

Property Value

string

Remarks

You can choose to crawl only web pages that belong to the same host or primary domain. For example, only web pages that contain the seed URL "https://docs.aws.amazon.com/bedrock/latest/userguide/" and no other domains. You can choose to include sub domains in addition to the host or primary domain. For example, web pages that contain "aws.amazon.com" can also include sub domain "docs.aws.amazon.com".

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html#cfn-bedrock-datasource-webcrawlerconfiguration-scope

UserAgent

Returns the user agent suffix for your web crawler.

string? UserAgent { get; }

Property Value

string

Remarks

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html#cfn-bedrock-datasource-webcrawlerconfiguration-useragent

UserAgentHeader

A string used for identifying the crawler or bot when it accesses a web server.

string? UserAgentHeader { get; }

Property Value

string

Remarks

The user agent header value consists of the bedrockbot , UUID, and a user agent suffix for your crawler (if one is provided). By default, it is set to bedrockbot_UUID . You can optionally append a custom suffix to bedrockbot_UUID to allowlist a specific user agent permitted to access your source URLs.

See: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-bedrock-datasource-webcrawlerconfiguration.html#cfn-bedrock-datasource-webcrawlerconfiguration-useragentheader