This is a GitHub Action for running EvoMaster as part of CI.
For now, it is recommended to use this action only for black-box testing. Documentation and configuration settings for white-box testing will be done in the future.
Example of configuration:
- name: Build the Application
run: ... # this swill depend on your application
- name: Start Docker Compose
run: docker compose up -d --build
- uses: webfuzzing/evomaster-action@v1
with:
args: >-
--blackBox true
--bbSwaggerUrl http://localhost:8080/v3/api-docs
--maxTime 20s
--showProgress false
failOnErrors: "true"
- name: Stop Docker Compose
if: always()
run: docker compose down
How you want to run EvoMaster is CI is up to you, e.g., if on each commit, or only on specific branches, or only for PRs, etc. Then, there are four steps.
-
Need to build your application. This will be programming language and framework dependent.
-
The recommended way to start your application is with Docker Compose. This alo helps if you need to start databases (e.g., Postgres, MySQL and MongoDB).
-
Use
webfuzzing/evomaster-action@v1to run EvoMaster. The action also uploads the generated tests and the interactive report as a workflow artifact, so you can download them as a zip file. -
Shutdown the application using Docker Compose
This action has the following options:
-
failOnErrors: specify whether to fail the build if any fault was detected by EvoMaster.
-
args: command-line arguments given to EvoMaster. These are exactly the same you would use when running EvoMaster on your local machine.
-
outputFolder: folder where EvoMaster writes its output. Default is
generated_tests. If you use--outputFolderinargs, set this option to the same value. -
uploadArtifact: whether to upload the output folder as a workflow artifact. Default is
true. -
artifactName: name of the uploaded artifact. Default is
evomaster-results. -
annotations: whether to create annotations for the found fault categories. Default is
true. -
prComment: on pull requests, write a comment with a short summary of the results, and update it on each new run. Default is
false. This needspull-requests: writepermission in your workflow. On PRs from forks, the token is read-only, and the comment is skipped with a warning.
The job summary of the action shows an overview of the results, the found faults per category and endpoint, and the code of the generated tests.
This requires a version of EvoMaster that writes wfc-summary.cjs next to report.json (which is not done if --writeWFCReportExcludeWebApp true is used).
With older versions, only the number of found faults is reported.
In this example, we used the arguments:
-
--blackBox true: specifying we are doing black_box testing. for historical reasons, default testing mode in EvoMaster is white-box.
-
--bbSwaggerUrl http://localhost:8080/v3/api-docs: when testing a REST API, need to tell where to find the schema. In this example, it is provided by the API itself, which is listening on port 8080. Note, however, that Docker Compose needs to be configured to expose such port, so that EvoMaster can communicate with the API running inside Docker Compose.
-
--maxTime 20s: specify for how long to run EvoMaster. The longer you run it the better results you can expect. Ideally, try to run for at least 10 minutes, but better if for 1 hour, or more.
-
--showProgress false: a progress bar is useful when running EvoMaster on a local shell. But, on GitHub Actions, the line-overwrite feature does not work, and each update creates new lines. This can flood the logs of this action. If this is a problem, this option can be used to deactivate these logs.
At the time of writing, this is a first version of this new GitHub Action. Several new features will be added in the next coming months. If there is any important feature you think it is missing, please let us know by creating a new post on the Discussions page.
To test this action, this repository itself is configured to use the action on itself (see ci.yaml).
EvoMaster is run in --help mode (so there is no need to have any actual API to test), where an example report.json file, together with the wfc-summary.cjs script that EvoMaster writes next to it, is already added to the repository for testing.
The CI runs the action twice: once with wfc-summary.cjs (rich summary), and once without it (fallback for older EvoMaster versions).