The proposed intermediate concept of replicability stems from the unique properties of computational experiments, i.e., that the measurement procedure/system, being virtual, is more easily portable, enabling inspection and exercise by others. Results that are not repeatable are rarely suitable for publication. Repeatability is something we expect of any well-controlled experiment. The concepts of repeatability and reproducibility are taken directly from the VIM. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently. The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. Replicability (Different team, same experimental setup )* For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts. The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. Reproducibility (Different team, different experimental setup )* For computational experiments, this means that a researcher can reliably repeat her own computation. The measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. Repeatability (Same team, same experimental setup) The following are inspired by the International Vocabulary for Metrology(VIM) see the Appendix for details. Because of this we find it necessary to define our terms. Unfortunately, the terminology in use has not been uniform. Specific labels, or badges, are proposed below.Ī variety of research communities have embraced the goal of reproducibility in experimental science. Authors who do agree to such additional review, and whose work meets established standards, will be rewarded with appropriate labeling both in the text of the article and in the metadata displayed in the ACM Digital Library. As a result, such review processes are encouraged, but remain completely optional for ACM journals and conferences, and when they are made available, it is recommended that participation by authors also be made optional. Of course, there remain many circumstances in which such enhanced review will be either infeasible or not possible. Here we provide terminology and standards for review processes of these types in order to promote a base level of uniformity which will enable labeling of successfully reviewed papers across ACM publications choosing to adopt such practices. And perhaps more importantly, well-formed and documented artifacts allow others to build directly upon the previous work through reuse and repurposing.Ī number of ACM conferences and journals have already instituted formal processes for artifact review. This also enables replication experiments to be performed, which, because they inevitably are done under slightly different conditions, serve to verify the robustness of the original results.
#Artifact definition software#
For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.Īdditional benefits ensue if the research artifacts are themselves made publically available so that any interested party may audit them. By "artifact" we mean a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. An intermediate approach is to require that artifacts associated with the work undergo a formal audit. An extreme approach would be to require completely independent reproduction of results as part of the refereeing process. Publishers can promote the integrity of the research ecosystem by developing review processes that increase the likelihood that results can be independently replicated and reproduced. A variety of recent studies, primarily in the biomedical field, have revealed that an uncomfortably large number of research results found in the literature fail this test, because of sloppy experimental methods, flawed statistical analyses, or in rare cases, fraud. (see: current version of Artifact Review and Badging v1.1)Īn experimental result is not fully established unless it can be independently reproduced.
0 kommentar(er)
