In this milestone, the focus was on testing and analysis of the source code, and integrating it with the existing pipeline. The following tasks were performed:
With Fuzzing, we wanted to the approach to be recreated as and when required, hence we used a pseudo-randomgenerator, with a seed value associated with each of the build. Thus it is easier for anyone, to get the same results in the test cases, and errors. Once the seed value is passed, we generate a probability of whether a line has to be modified, and then there is a probability that is associated with each of the fuzing operations. Once the fuzzing operation is completed, we push the code onto Jenkins Server for a build. Once the build is complete, we copy the test files fromsurefire-reports to a different folder, and then another fuzzing operation is queued up.
Once the hundred builds are built, and all the files collected we wrote an analysis script, that would be responsible for walking through the directories, and compiling information about each build. Some analysis of the errors generated, and the changes that caused the errors are listed below. This is not a comprehensive list, but something that we uncovered while going through the saved results.