Random Input and Automated Output Generation in Submitty
| Autor: | Evan Maicus, Barbara Cutler, Matthew Peveler, Drumil Patel |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Computer science
business.industry 05 social sciences 050301 education 020207 software engineering 02 engineering and technology Fuzz testing Set (abstract data type) ComputingMilieux_COMPUTERSANDEDUCATION 0202 electrical engineering electronic engineering information engineering Code (cryptography) Software engineering business 0503 education Codebase |
| Zdroj: | SIGCSE |
| DOI: | 10.1145/3328778.3372685 |
| Popis: | Fuzzing, testing a codebase against a set of randomly generated inputs, has become a promising model of testing across the industry due to its ability to reveal difficult to detect bugs. Separately, the use of randomized inputs when testing student code submissions removes the potential for student hard-coding behavior. Motivated by these factors, we present a solution for the automated generation of testcase inputs and expected outputs within Submitty, an open source automated grading system from Rensselaer Polytechnic Institute. We detail an enhanced workflow that allows instructors to provide our testing system with an assignment-specific input generation script and an assignment solution. The input generation script is run at student test-time, providing students with either entirely generated inputs or a combination of generated and hand-crafted testcases. The instructor solution is run against the same inputs to produce expected results. This model of testcase specification carries the benefit of simple regeneration of expected output files if an assignment's specification changes after submissions open or between semesters. We present preliminary results of the use of random input generation in our large introductory programming courses, and evaluate the ability of random inputs to curb student hardcoding behavior as it relates to an early submission incentive system, which grants students an extension for achieving a target assignment score early in the week an assignment is due. We examine random input generation's ability to reveal bugs in student submissions from previous semesters. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|