Configure Contrast Crawler (preview)

Note

This feature is in preview mode and not generally available to all users. For access to this feature, contact Contrast support.

Use the template in this topic to configure the Contrast Crawler.

Steps

Create a YAML file using this template:
```
user_api:
  enabled: true
  url: NEEDS_TO_BE_SET
  api_key: NEEDS_TO_BE_SET
  service_key: NEEDS_TO_BE_SET
  user_name: NEEDS_TO_BE_SET
  org_id: NEEDS_TO_BE_SET
application:
  name: NEEDS_TO_BE_SET
  url: NEEDS_TO_BE_SET
authentication:
  enabled: false
  auth_script_file: NEEDS_TO_BE_SET
  # keep_auth_script: false
  # no_logged_out_indicator: false
  # no_logged_in_indicator: false
# crawl_config:
  # headless: false
  # max_requests_per_crawl: 10
  # max_request_retries: 2
  # max_concurrency: 1
# debug:
  # auth_only: true   
  # single_crawl: true
  # unauth_only: true
  # just_url: http://localhost:8080/owners/1/edit
  # dont_enqueue: true
  # just_url_repeats: 200
  # dont_seed_crawler: true
```
- Settings in the template with a value of NEEDS_TO_BE_SET indicate a user-specific configuration that you need to provide. However, if you set either client_api.enabled or authentication.enabled to false, there's no need to specify values for the rest of the settings in those nodes.
- You can omit settings in the template that contain default values, if those values are acceptable. Some settings have no default values but are not required. You can choose to omit or provide values for these settings.
- The user_api and application root nodes are required. The Crawler uses these settings to retrieve route and vulnerability information from Contrast.
- The user_api node settings user API credentials, (which are different than the agent API credentials).
  The Crawler uses these settings to retrieve route and vulnerability information from Contrast. Alternatively, if you set client_api.enabled to false, the credentials are not required. Doing so means the Crawler is not be seeded with route URLs and the Crawler only makes requests for URLs that are discoverable from the application home page. For some applications, this may reduce the number of exercised routes.
- The application.name setting must match the name of the application as it is known in Contrast.
- The application.url setting is the base URL of the tested application, for example: http://localhost:8080.
- If your application does not need authentication, don't change the authentication setting. If you do need to crawl an authenticated application, use these settings:
```
authentication.enabled: true 

authentication.auth_script_file: <path to recorded authentication script>.js
```
  If you recorded an authentication script and want to continue to use it without being asked, set authentication.keep_auth_script to true. This setting is required for crawling an application in a CI environment since there is no user to respond to Crawler questions.
- The telemetry, crawl_config, and debug nodes are not required and may be omitted.
  If for some reason you want to change the value of settings in these nodes, be aware that the Crawler only recognizes child configuration nodes up until the first commented out configuration at the same level. For example, these settings prevent a second crawl:
```
debug:
  single_crawl: true
  # auth_only: true 
```
  While these settings have no effect on the second crawl:
```
debug:
  # auth_only: true   
  single_crawl: true
```

If needed, configure environment variables to override or supplement settings in the configuration file:

YAML setting	Environment variable
`user_api.url`	`CONTRAST__CRAWLER__USER_API__URL`
`user_api.api_key`	`CONTRAST__CRAWLER__USER_API__API_KEY`
`user_api.service_key`	`CONTRAST__CRAWLER__USER_API__SERVICE_KEY`
`user_api.user_name`	`CONTRAST__CRAWLER__USER_API__USER_NAME`
`user_api.org_id`	`CONTRAST__CRAWLER__USER_API__ORG_ID`
`application.name`	`CONTRAST__CRAWLER__APPLICATION__NAME`
`application.url`	`CONTRAST__CRAWLER__APPLICATION__URL`
`authentication.enabled`	`CONTRAST__CRAWLER__AUTHENTICATION__ENABLED`
`authentication.auth_script_file`	`CONTRAST__CRAWLER__AUTHENTICATION__AUTH_SCRIPT_FILE`
`authentication.keep_auth_script`	`CONTRAST__CRAWLER__AUTHENTICATION__KEEP_AUTH_SCRIPT`
`authentication.no_logged_out_indicator`	`CONTRAST__CRAWLER__AUTHENTICATION__NO_LOGGED_OUT_INDICATOR`
`authentication.no_logged_in_indicator`	`CONTRAST__CRAWLER__AUTHENTICATION__NO_LOGGED_IN_INDICATOR`

In this section:

Configure Contrast Crawler (preview)

Note

Steps

Search results