Example: Contrast Crawler in a GitHub CI workflow

name: Crawl
    steps:
      - name: ⬇️ Checkout code
      - name: 🛠️ Install node
      - name: 📦 Install App Dependencies
      - name: ⚙️ Instrument App with Assess
      - name: 🪛 Install Agent Configs
      - name: 🚀 Start the test app
        id: start-app
      - name: 🔭 Wait for it to be ready
      - name: 🧑‍💻 Make an App Request

      # The above steps "build" the application to test and 
      # run it with a Contrast agent attached. The details of 
      # starting the application with an agent will vary by 
      # language and build tools used.      
       
      # The step 'Make an App Request' is to ensure that the 
      # application and its discovered routes are avalable in 
      # the TeamServer instance.

      # If the application under test were not a node.js app
      # then to support the crawler application we would include 
      # something like this here:
      # - name: 🛠️ Install node
      #   uses: actions/setup-node@b39b52d1213e96004bfcb1c61a8a6fa8ab84f3e8  # v4.0.1
      #   with: 
      #     node-version: 18

      - name: 📦 Install the Contrast Crawler
        run: npm install -g @contrast/crawler

      # The crawler now includes a utility mode that can be used to
      # execute recorded Playwright scripts.  When launched this way 
      # the crawler executes the script but does not actually crawl the app.
      
      # NOTE: There is currently a deficiency in the crawler in that, when
      # Playwright records a script it sets `headless: false` which is not
      # compatible with runnning in a pipeine.  In this example, the recorded
      # script, juice-shop-prep.js, was manually edited to fix the problem.  
      # In a future release the crawler will handle this more gracefully.
      
      # The script specified, juice-shop-prep.js, registers a user
      # with the name and password expected by the authentication 
      # script.
      - name: 📝 Register a user
        run: npx @contrast/crawler ./contrast-crawler/resources/crawler.yaml script=./contrast-crawler/resources/juice-shop-prep.js
        env:
           CONTRAST__CRAWLER__USER_API__URL:          ${{ vars.CONTRAST__CRAWLER__USER_API__URL }}
           CONTRAST__CRAWLER__USER_API__ORG_ID:       ${{ vars.CONTRAST__CRAWLER__USER_API__ORG_ID }}
           CONTRAST__CRAWLER__USER_API__USER_NAME:    ${{ vars.CONTRAST__CRAWLER__USER_API__USER_NAME }}
           CONTRAST__CRAWLER__USER_API__API_KEY:      ${{ secrets.CONTRAST__CRAWLER__USER_API__API_KEY }}
           CONTRAST__CRAWLER__USER_API__SERVICE_KEY:  ${{ secrets.CONTRAST__CRAWLER__USER_API__SERVICE_KEY }}

      - name: 🕷️ Crawl the app
        run: npx @contrast/crawler ./contrast-crawler/resources/crawler.yaml
        env:
           CONTRAST__CRAWLER__USER_API__URL:          ${{ vars.CONTRAST__CRAWLER__USER_API__URL }}
           CONTRAST__CRAWLER__USER_API__ORG_ID:       ${{ vars.CONTRAST__CRAWLER__USER_API__ORG_ID }}
           CONTRAST__CRAWLER__USER_API__USER_NAME:    ${{ vars.CONTRAST__CRAWLER__USER_API__USER_NAME }}
           CONTRAST__CRAWLER__USER_API__API_KEY:      ${{ secrets.CONTRAST__CRAWLER__USER_API__API_KEY }}
           CONTRAST__CRAWLER__USER_API__SERVICE_KEY:  ${{ secrets.CONTRAST__CRAWLER__USER_API__SERVICE_KEY }}

      # The uploaded artifacts include debug level structured crawler logs
      - name: ⬆️ Upload crawl artifacts
        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3  # v4.3.1
        with:
          name: Crawl Artifacts
          path: storage/**/*
          retention-days: 14

      - name: 🪓 Kill the server
        run: |
          test_app_pid=${{ steps.start-app.outputs.test_app_pid }}
          echo "test_app_pid=$test_app_pid"
          echo "Killing server."
          kill $test_app_pid
        

This example configuration file (in a file called crawl.yaml) configures the Crawler to use the specified authentication script as is and suppresses the usual console prompts regarding what to do with the script. The configuration must specify an application name that matches the application name as it is reported in Contrast.

user_api:
  # read from repo secrets and set as env vars
  # CONTRAST__CRAWLER__USER_API__URL
  # CONTRAST__CRAWLER__USER_API__ORG_ID
  # CONTRAST__CRAWLER__USER_API__USER_NAME
  # CONTRAST__CRAWLER__USER_API__API_KEY
  # CONTRAST__CRAWLER__USER_API__SERVICE_KEY
application:
  name: crawled-juice-shop-ci
  url: http://localhost:3000
authentication:
  enabled: true
  auth_script_file: ./contrast-crawler/resources/juice-shop-auth.js
  no_logged_out_indicator: true
  keep_auth_script: true
crawl_config:
  max_concurrency: 10
  

This setting prevents the Crawler from asking how to handle the authentication script and waiting for user input.

authentication:
  keep_auth_script: true