One of the challenges in the SEAMS community has been the dearth of community resources (e.g., exemplars or model problems) that can be used to compare existing approaches or be built upon by other researchers. There is a community repository at http://self-adaptive.org/exemplars that contains some examples that have been discussed in prior SEAMS, but there is a need to populate this repository with additional examples.
To address this, SEAMS 2016 will haveĀ a special artifacts session, and soliciting short papers (6 pages) that describe such resources that could be used by researchers in our community. Resources of interest include:
- Exemplars, Case Studies, and Benchmarks, which are implementations of systems that can be used with multiple self-adaptive approaches.
- Model Problems, which are descriptions of problems that pose and highlight fundamental or characteristic challenges in this community, and that self-adaptive systems should address.
- Data repositories, which are data (e.g., logging data, system traces, survey raw data) useful in other studies.
- Frameworks, which are tools and services illustrating new approaches to self-adaptation that could be used by other researchers in different contexts.
Each paper should include a synopsis or problem description that is being addressed, a description of the context(s) in which the resource would be useful, a list of the challenges that it poses for self-adaptation, and examples of its use in at least one area of self-adaptive systems.
Accompanying each paper, a set of artifacts that could be used by the community should be included. To encourage this, submissions should also contain links to the artifacts being discussed, which will be downloaded by the evaluators. Papers and accompanying artifacts will be evaluated for suitability for being used by the wider community. Accepted papers and artifacts will be included in the proceedings, and authors will be given an opportunity to present at SEAMS.
Packaging Guidelines
When packaging your artifact, please keep in mind: a) how accessible you are making your artifact to other researchers, and b) the fact that the SEAMS evaluators will have very limited time in which to make an assessment of each artifact. The setup for your artifact should take less than 30 minutes or it is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it. To expedite the evaluation process, it may be appropriate to include a virtual machine image as part of your package.
You should make your artifact available as a single archive file using a widely available compressed archive format such as ZIP (.zip), tar and gzip (.tgz), or tar and bzip2 (.tbz2).
The archive must:
- be self-contained (with the exception of pointers to external tools or libraries; which we will not consider being part of the evaluated artifact, but which we will try to use when evaluating the artifact);
- contain a HTML file called index.html that fully describes the artifact and includes (relative) links to the files (included in the archive) that constitute the artifact;
- include a Getting Started Guide (a section within index.html, see below);
- include Step-by-Step Instructions (another section within index.html) for how you propose to evaluate your artifact;
- where appropriate, include descriptions of and links to files (included in the archive) that represent expected outputs (e.g., the log files expected to be generated by your tool on the given inputs).
The artifact may include, but is not limited to, code, executables, data, a virtual machine image, and documents. Please use open formats for documents and we prefer experimental data to be submitted in csv format.
Within your index.html, you must include a section with a basic Getting Started guide. Reviewers will follow all steps in the Getting Started guide at the beginning of the evaluation period and if necessary can relay questions if they run into difficulties. You should write your Getting Started guide to be as simple and straightforward as possible, and yet it should stress the key elements of your artifact. If well written, anyone who has successfully completed the Getting Started guide should not have any technical difficulties with the rest of your artifact. The Getting Started guide is your only opportunity to allow “debugging” of your artifact. Once the reviewers have completed this phase, there will be no further opportunity for interaction with you, the authors.
Selection Criteria
The artifact will be evaluated in relation to the expectations set by the paper. Thus, in addition to just running the artifact, the evaluators will read the paper and may try to tweak provided inputs and create new ones, to test the limits of the system.
Submission
Note that there can be different kinds of artifacts, such as software tools, data sets (such as raw data from experiments), services, or anything else that may be relevant for community use. You may want to include multiple artifacts (e.g., a tool and the data set used in its evaluation) in your submission. Please bundle everything into a single archive file. Papers submitted in this track should contain a link for accessing the artifacts. After acceptance, authors will be encouraged to add an entry to the relevant section of http://self-adaptive.org.
Please perform your artifact submission via EasyChair, following the submission guidelines above. The submissions deadlines for abstracts and papers also apply to artifact paper submissions.
1 comment for “Call for Artifacts”