Description
Our vision is that journal publications will report simulation results and methods according to established criteria to enable archiving, reproducibility, and maximize scientific leverage. These criteria will be established by a partnership of journal editors, developers, and industrial experts.
Goals
Identify key journals (completed)
Obtain buy-in to strategic plan by key journal editors.
Identify editors for each of the key journals
- Contact the editors to inform them about what we are planning and ask about their existing criteria for simulation papers
- Develop and vet criteria with users and developers of simulations and associated methods.
Beginning with the feedback from the editors, exisiting literature, and our own experiences, draft a strawman set of criteria
- Solicit feedback on the strawman from practitioners
- Revise the strawman based on feedback
- Develop an agreement with journals to publicize validation criteria and to implement a policy for authors to adhere to these criteria prior to acceptance and publication of relevant articles in these journals.
Communicate the criteria to journal editors
- Publish the criteria at fluidproperties.org
- Develop a repository/database of refereed simulation results
- Define more carefully what types of simulation results to be incorporated and criteria to judge validity of results
- http://www.biosimgrid.org/ is an example of a simulation database
- Provide framework for researchers to submit results (check with Tom Allison who is doing this for kinetics)
- Monitor acceptance of criteria
Team Notes
Validation items
- The quality of the theory or model used: Explain clearly the assumptions used, how the results are obtained, and why the theory/model is appropriate.
- The accuracy of the force field: Specify functional forms and parameters or provide a complete reference. Are their any tests, current or prior, that justify use of the force field?
- Sampling, statistics, convergence: Specify all pertinent parameter values, explain how the sampling is done, provide an uncertainty analysis, and measures of convergence.
- Is the software correct? Explain why the software used is appropriate for the problem.
- Is the software used properly? Discuss/provide checks for physically consistent/sensible outputs. Are the parameters sensible? Are there any unphysical correlations?