Repository navigation
Expand file tree
/
Copy pathnotes_adding_suts.txt
More file actions
227 lines (193 loc) · 16.7 KB
/
Copy pathnotes_adding_suts.txt
File metadata and controls
227 lines (193 loc) · 16.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
==============================================
NOTES
Here some high level notes on how to add new SUTs to EMB.
As there are quite a few things to keep in mind, we list here the most important.
==============================================
- before adding a SUT, its repository needs to be cloned locally, and should make sure the application can be built and
run with no problem.
Then, need to be able to write an Embedded driver for it, and make sure that EM can work with it.
Only after this is verified, we can add an SUT to EMB.
Note that we have (tens of) thousands of files in EMB, and adding new stuff does have a toll on Git.
So should avoid cases of adding something and then remove them later for some issues.
- need to choose a NAME for the SUT. usually, it depends on the actual name from GitHub (eg some combination/adaptation
of the "author/name" GitHub identifier), but it does not necessarily mean it must be exactly the same.
Small modifications (especially if the name is generic or very long) can be done.
The reason is that NAME is used everywhere, including in the writing of the scientific articles.
You MUST discuss with supervisor before choosing the NAME (as the change of the name can impact thousands of files
if refactored).
This MUST be done before the first PR.
- if the NAME is composed of 2 or more words, use "-" to separate them.
- adding a new SUT should always be done on a branch of "develop", as needs to be reviewed.
- need to check which built tool is used (e.g., Maven or Gradle), and which LTS version of the JDK is used.
Currently, we support from 8 on, ie, 8, 11, 17 and 21.
If an API uses a non-LTS version, need to discuss/make a decision (eg, check a previous commit with latest LTS).
Based on this, the SUT will belong to a folder matching such settings, eg, "jdk_8_maven".
If such folder does not exist yet, it needs to be created and configured.
See pom.xml from the other existing folders, to use as starting point (ie, copy&paste&update)
- the SUT needs to be added under the folder "cs" (which stands for "case study"), and then under a subfolder matching
the type of the application, eg: "rest", "web", "graphql", "grpc" and "thrift".
Note: in the past we have used further subfolder structures like "artificial", "original" and "rpc".
Those are not really so needed anymore, so should avoid using them when adding new SUTs (but we are not going to
refactor EMB to removed them).
A special case is when an SUT is an API that also provide a GUI (eg Web browser frontend).
In those cases, for REST we used the label "rest-gui".
But we have not made a proper decision on how to deal with those cases yet.
So, if the SUT you add falls under such a case, need to discuss first with supervisors.
- when adding the SUT under "jdk_X_Y/cs/<type>" for first time, add original code of the SUT WITHOUT ANY MODIFICATION,
in a single commit, containing only that (ie, no custom modification, and no EM drivers).
This is to be able to easily trace the original version of the SUT.
Also, it is EXTREMELY DIFFICULT to review in PRs the EM drivers when you have as well thousands of SUTs files cluttering
the GitHub diff views.
THIS MEANS THERE IS THE NEED OF 2 DISTINCT PRs:
1) having the original SUTs files, and only those, ie nothing outside of "jdk_X_Y/cs/<type>/NAME"
2) having all custom modifications (if any needed), pom files and their modifications, and EM drivers (embedded and external)
- under no circumstance should add compiled files. Also large files should be avoided if they are not strictly necessary.
EMB is huge, and we should be careful of what we add, especially in terms of MBs size.
- note: there might be few special cases in which we need to do some minor modifications to the SUTs.
Those should be rare. Might need to add specific comments stating why the SUT was modified directly in the modified files.
A common example is parent pom declarations for SpringBoot applications, which mess up a lot of things (more on this later).
- when adding the SUT, some files should be skipped, like ".idea", ".gitignore", "mvnw", ".github", etc., as those can
mess up the configurations of EMB. In other words, we only need the files to be able to build an executable JAR file.
Other files could be added, as long as not too heavy (eg readme.md) or potentially messing up EMB (eg netsted .gitignore).
- Maven: when creating pom.xml tree-structures for "jdk_X_Y/cs/<type>", it makes sense to copy&paste from the other existing pom
files at same "depth", and modify names in <artifactId> accordingly.
Those names are hierarchical structured, starting with "evomaster-benchmark".
For example, the pom file for "jdk_17_maven/cs/grpc" would have artifactId "evomaster-benchmark-jdk17-cs-grpc",
whose parent in "jdk_17_maven/cs" would have artifactId "evomaster-benchmark-jdk17-cs".
The SUT should be buildable from "jdk_X_Y", ie, the tree-structure needs to be defined with the <module> directive
to specify which child subfolders with pom files represent sub-modules.
- Maven: if the SUT root pom.xml file refers to an external <parent>, eg "spring-boot-starter-parent" or "micronaut-parent",
in the parent declaration you might need to add <relativePath/> to avoid an inconsistent Maven structure.
- Gradle: should use same structure "jdk_X_Y/cs/<type>", but the build structure needs to be defined in settings.gradle.kts
- the README.md needs to be updated with info on the new SUT, under the proper group.
Need to specify the license, folder location on EMB, and original source repositories.
See other SUT entries to check exactly how to do it (especially how to add clickable links in Markdown).
- the content of folder "statistics" needs to be updated.
In particular, need to create a new entry for the API in "data.csv" file.
To count #Files, can use bash command
find . -name *.java | wc -l
assuming the project is in Java (.kt for Kotlin otherwise).
Note that this pick up everything, including tests, and automatically generated files from build (eg. for .proto)/
To count LOCs, can use
cat `find . -name *.java` | wc -l
- once "data.csv" file has been updated, need to recreate "table_emb.md" from "analyze.R" -> "markdown()" function.
- the drivers should be written under "jdk_X_Y/em/embedded" and "jdk_X_Y/em/external", following the same tree structure
of the SUT under "cs", eg, "jdk_17_maven/cs/grpc/NAME" would lead to embedded driver be under
"jdk_17_maven/em/embedded/grpc/NAME".
- Maven: the artifactId for the new modules follow the same convention of using same folder names separated with "-",
eg "evomaster-benchmark-jdk17-em-embedded-NAME" for an embedded driver under JDK 17.
- EMBEDDED: the driver should be called "EmbeddedEvoMasterController", under the package "em.embedded.X", where X is the
main package name of the SUT.
Should be under "src/main" and not "src/test".
The pom file needs to import 2 dependencies:
1) EM dependencies
2) SUT as a library
Regarding (1), this should be done by default in the pom file of "jdk_X_Y/cs".
If you make a new "jdk_X_Y", then you need to copy&paste from another top module (settings are the same).
Note, you might still want to override some dependencies in case of library conflicts, eg, you can resolve some of
those issues by redefining the dependency imports by using <exclusions> tags.
Regarding (2), this might be problematic if the default settings of the SUT do build a uber JAR that use a custom
classloader, ie, the typical case of SpringBoot applications (will need to check if same issues with other frameworks
such as Micronaut).
Solving this "might" (eg depending on SpringBoot version) require modifications to the pom.xml (in case of Maven) of the SUT.
In particular, might need to remove the <parent> declaration, and rather include it as a pom import
under the <dependencyManagement> (that would fix all version issues for dependencies, but not for plugins... which
would need to be fixed manually).
Then, in the plugin used for creating the uber JAR, need to make sure to still build the original JAR as well
(so need 2 different names, default based on module for original, and modified name for uber JAR).
Note: for the actual writing of the "EmbeddedEvoMasterController", you can see the EM documentation for it.
- EXTERNAL: first, need to make sure to build an uber JAR in the SUT.
It MUST BE in the form "NAME-sut.jar". Eg, for SpringBoot you can have a setting like:
<configuration>
<finalName>NAME</finalName>
<classifier>sut</classifier>
</configuration>
(btw, recall that NAME is just a placeholder for the actual name...)
The driver "ExternalEvoMasterController" needs to be under the package "em.external.PACKAGE", in similar way as
for embedded driver.
Unfortunately, we currently have no documentation on how to write External Drivers.
An important difference here is that we need to build an uber JAR of the driver itself, named NAME-evomaster-runner.jar.
This is done with the "maven-shade-plugin" plugin (can copy&paste an existing one, and update names in it).
Those JAR files are needed by "dist.py", discussed next, to be able to run experiments.
- Distribution: the "dist.py" file needs to be updated with the new API. Usually, this is just adding 2 lines, one in
which NAME-sut.jar is copied over into the "dist" folder, and one for NAME-evomaster-runner.jar
Note: dist.py can take a LOT of time... while debugging it, might want to temporarily disable (ie comment out) the
build of the other top modules
- If the SUT is a REST API, need to add its OpenAPI schema into "openapi" folder
- If REST APIs, must support BB experiments via Docker files. This done in files under "scripts/dockerize".
The entry point is "scripts/createDockerFiles.py": it reads "scripts/dockerize/data/sut.csv", and, for each SUT
marked with Dockerized=TRUE, it calls "scripts/dockerize/docker_generator.py".
For each such SUT, this generates 2 files under the "dockerfiles" folder (from the Jinja2 templates in
"scripts/dockerize/templates"):
1) "NAME.dockerfile": the Docker image for the SUT itself, which copies "NAME-sut.jar" (and "jacocoagent.jar")
from the "dist" folder, and starts the SUT with the given JVM/input parameters.
This means "dist.py" must already handle the SUT (see previous point on Distribution).
2) "NAME.yaml": a docker-compose file, including any extra service the SUT depends on (eg databases).
Note: the JDK version used to select the Docker base image is NOT in "sut.csv": it is read from the RUNTIME
column of "statistics/data.csv" (values like "JDK 8", "JDK 11", "JDK 17" and "JDK 21", mapped to Amazon Corretto
Alpine images). So, the entry in "statistics/data.csv" must be present and correct before generating the Docker files.
Note: the generated compose file also includes a "mitmproxy" service, which reverse-proxies (and logs) all the
HTTP traffic toward the SUT (whose service is named "sut-NAME"). It is mitmproxy, not the SUT, that exposes
port 8080 (overridable with ${HOST_PORT}) on the host.
Inside its container, the SUT MUST listen on port 8080 (this is hardcoded, eg in the mitmproxy configuration),
so use the SUT input parameters (eg "--server.port=8080") to enforce it.
To add a new SUT, a new row must be added to "scripts/dockerize/data/sut.csv". Its columns are:
* NAME: the name of the SUT. Must match exactly the NAME used in "statistics/data.csv" (otherwise the generator
stops with an error), as well as the "NAME-sut.jar" file created for the external driver.
* Dockerized: TRUE/FALSE, whether the Docker files should be generated for this SUT. Should be TRUE for all
REST APIs (needed for BB experiments).
* JVM_PARAMETERS: JVM options (eg "-Dkey=value" and "-Xmx" settings) placed BEFORE "-jar NAME-sut.jar" in the
ENTRYPOINT of the generated dockerfile. Can be left empty.
* INPUT_PARAMETERS: program arguments placed AFTER "-jar NAME-sut.jar" (eg "--server.port=8080
--spring.profiles.active=dev"). This is the typical place to force port 8080 and to point the SUT to its
services in the compose file (note: services are reachable by their compose service name as hostname, eg
"jdbc:postgresql://db:5432/..." when the database service is called "db"). Can be left empty.
* SWAGGER_URL: the URL on which the running container serves its OpenAPI/Swagger schema
(eg "http://localhost:8080/v3/api-docs"). Used to fetch the schema in BB experiments.
* TARGET_URL: the base URL of the running SUT, typically "http://localhost:8080".
* COPY_ADDITIONAL_FILES: TRUE/FALSE. If TRUE, ALL files inside the folder
"dockerfiles/additional_files/NAME" are copied (COPY) into the working directory of the Docker image.
This is for SUTs needing extra files at runtime next to the JAR (eg configuration files such as YAML configs).
Note the folder MUST exist and MUST have the same name as the SUT (NAME), otherwise the generation fails.
* DEPENDS_ON: semicolon-separated list of service names the SUT container must wait for (rendered as
"depends_on" with a healthcheck condition in the compose file), eg "db" for SUTs whose database defines a
health_check_command. Usually left empty ("").
* SERVICES: a JSON array (quoted as a single CSV cell, with doubled "" for inner quotes) describing the extra
services (databases, mock servers, etc.) to add in the docker-compose file. Each JSON object supports:
- "name": service name in the compose file (optional, defaults to "db"). This is the hostname the SUT
uses to reach it.
- "image_name": the Docker image, eg "postgres:13.13", "mongo:6.0", "mysql:8.0".
- "tmp_fs": path mounted as tmpfs, typically the data folder of the database (eg "/var/lib/postgresql/data"),
so that each run starts with a clean state. Empty string if not needed.
- "environment": semicolon-separated list of "KEY: value" environment entries for the service
(eg "POSTGRES_PASSWORD: password;POSTGRES_DB: foo"). Empty string if none.
- "volume": semicolon-separated list of "host_path:container_path" mounts. Host paths here point into
"./additional_files/NAME/..." (relative to the "dockerfiles" folder), eg to
provide DB init scripts ("/docker-entrypoint-initdb.d/...") or mock-server configs. Empty string if none.
- "port": optional "host:container" port mapping, if the service must be reachable from outside the
compose network (eg mock OAuth servers).
- "command": optional command overriding the image default (eg "mongod --replSet rs0 --bind_ip_all").
- "health_check_command": healthcheck command in Docker list form (eg ["CMD", "mysqladmin", "ping", "-h",
"localhost"]), or empty string "" for no healthcheck. Needed when the SUT uses DEPENDS_ON on this service.
Leave the whole cell empty if the SUT needs no extra services.
Note: if the SUT needs a mock OAuth2 server for authentication, that is defined here as well, as one more
service (see "familie-ba-sak" for an example using "mock-oauth2-server", with the exposed ${AUTH_PORT}).
- if the SUT needs any extra file inside Docker (either copied into the SUT image via COPY_ADDITIONAL_FILES, or
mounted as a volume of a service in SERVICES), such file must be placed under
"dockerfiles/additional_files/NAME", where NAME is exactly the same name of the SUT used in "sut.csv".
Note: this folder MUST be under "dockerfiles", as that is the only folder distributed by "wfd.dockerfile".
Paths pointing outside of it would break when WFD is distributed as a Docker image.
See the existing subfolders there (eg "blogapi", "pay-publicapi", "webgoat") for examples.
- after updating "sut.csv" (and "statistics/data.csv"), run "scripts/createDockerFiles.py" to (re)generate
"dockerfiles/NAME.dockerfile" and "dockerfiles/NAME.yaml", and commit them as well.
A good way to verify the setup is to build the SUT JARs (dist), start the compose file, and check that the
OpenAPI schema is reachable at SWAGGER_URL.
- The "jacoco.dockerfile" needs to be updated as well, with new "COPY" entries for each new "target/classes" folder.
Note that, in a multi-module maven project, not all modules need to be necessarily added.
Only all the modules involved in building the jar of the SUT must be added.
This requires to check how the SUT is built, and dependencies among its sub-modules.
Adding unrelated modules will negatively impact how coverage is measured with JaCoCo.
- The script `http-verbs.py` needs to be run to recompute `http-verbs.csv`.
Once done, the function `markdown()` in `http-verbs.R` needs to be executed to update `http-verbs.md`.
- Given a SUT called X, its image needs to be published with name:tag "webfuzzing/wfd-X:FINAL".
As it requires authentication on Docker Hub, this must be done manually by the architect, after X is added to WFD.