AWS::Wisdom::KnowledgeBase WebCrawlerConfiguration
The configuration details for the web data source.
Syntax
To declare this entity in your AWS CloudFormation template, use the following syntax:
JSON
{ "CrawlerLimits" :
CrawlerLimits
, "ExclusionFilters" :[ String, ... ]
, "InclusionFilters" :[ String, ... ]
, "Scope" :String
, "UrlConfiguration" :UrlConfiguration
}
YAML
CrawlerLimits:
CrawlerLimits
ExclusionFilters:- String
InclusionFilters:- String
Scope:String
UrlConfiguration:UrlConfiguration
Properties
CrawlerLimits
-
The configuration of crawl limits for the web URLs.
Required: No
Type: CrawlerLimits
Update requires: Replacement
ExclusionFilters
-
A list of one or more exclusion regular expression patterns to exclude certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.
Required: No
Type: Array of String
Minimum:
1
Maximum:
25
Update requires: Replacement
InclusionFilters
-
A list of one or more inclusion regular expression patterns to include certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.
Required: No
Type: Array of String
Minimum:
1
Maximum:
25
Update requires: Replacement
Scope
-
The scope of what is crawled for your URLs. You can choose to crawl only web pages that belong to the same host or primary domain. For example, only web pages that contain the seed URL
https://docs.aws.amazon.com/bedrock/latest/userguide/
and no other domains. You can choose to include sub domains in addition to the host or primary domain. For example, web pages that containaws.amazon.com
can also include sub domaindocs.aws.amazon.com
.Required: No
Type: String
Allowed values:
HOST_ONLY | SUBDOMAINS
Update requires: Replacement
UrlConfiguration
-
The configuration of the URL/URLs for the web content that you want to crawl. You should be authorized to crawl the URLs.
Required: Yes
Type: UrlConfiguration
Update requires: Replacement