Below is a brief guide to policies related to RTSS. The aim of this guide is to improve transparency, so that authors and participants in the conference can better understand how certain roles are selected and how the review process works. (Note this guide is an initial draft, it may not capture every aspect).
The TCRTS Executive Committee oversees RTSS. The Chair of the TCRTS Executive Committee serves for a period of two years, having served previously as Vice Chair. The Vice Chair, and hence the Chair, are elected via a vote of IEEE Computer Society members.
The Chair of the Executive Committee is responsible for inviting and appointing members of the Committee to serve for a period of two years. A list of the current members of the TCRTS Executive Committee is published on the TCRTS website: http://sites.ieee.org/tcrts/committees/
Selecting the Venue:
Each year a call is sent out inviting proposals to host RTSS in two years’ time. This call may indicate a preferred geographical region. The TCRTS Executive Committee reviews the proposals received and selects a shortlist to be considered and voted upon by all participants (i.e. those attending RTSS) during a TCRTS meeting held at the conference. The vote takes place after short presentations by the prospective local organisers.
Selecting the Program Chair and General Chair:
The Conference Planning Sub-Committee of the TCRTS Executive (http://sites.ieee.org/tcrts/committees/) proposes candidates for the next Program Chair of RTSS. The Program Chair is determined from the shortlist via a vote of the full TCRTS Executive Committee, with the exception of members with a Conflict of Interest (COI).
The Program Chair becomes the General Chair for the following year.
Selecting Track Chairs
Track Chairs are invited and appointed by the Program Chair in consultation with the General Chair and the Chair of the TCRTS Executive Committee.
Selecting the Program Committee
The Program Committee (PC) is selected by the Program Chair and the Track Chair(s).
PC members are permitted to serve for a maximum of two consecutive years on the PC, with exceptions permitted for well-founded reasons such as diversity, and to support minority groups. Note since Program / Track Chairs do not review papers, previous Chairs are not counted as serving on the PC for the purposes of this policy.
The objectives in choosing the composition of the PC are: to ensure that there is sufficient expertise in the relevant topic areas, to ensure that the PC has a suitably diverse demographic, and to ensure a degree of continuity. PC members are expected to have a high level of expertise and are chosen based on merit. PhD students are not selected as PC members.
The Chair of the TCRTS Executive keeps a document recording prior PC members who have not fulfilled their role well, for example by providing poor quality or late reviews, or by being absent from the physical PC meeting without good cause or notification. This document acts as a guide or caution regarding subsequent appointments to the PC; note, no-one is precluded, the list is advisory.
The Program Chair is not permitted to submit papers to the main conference (all other members of the organizing team, including Track Chair(s), General Chair, Local Organisers, PC members etc. are permitted to submit papers). Workshop/Brief Presentations/RTSS@Work Chairs are not permitted to submit papers to their own workshops/sessions.
The Review Process
The review process for RTSS is single blind, i.e. the reviewers are anonymous, but the papers include the authors’ names, and so the reviewers know who the authors are.
PC members bid for papers, indicating their reviewing preferences (yes, no, or maybe) and any Conflicts of Interest (COIs) – see later definition.
Authors of papers may optionally indicate PC members that they have a COI with that is outside of the normal scope of COIs (see later definition). This information, including proper reasons for the additional COIs must be communicated directly to the Program Chair in writing prior to the submission deadline. These PC members will not normally be eligible to review the authors’ paper(s). Note the final decision on whether to make this restriction rests with the Program Chair.
Once the bids have been completed, the conference management system assigns papers to PC members based on their preferences.
Each paper is reviewed by at least four PC members.
Once the initial reviews have been completed, PC members have the opportunity to look at the other reviews for papers they were allocated, and to discuss them online with the other reviewers. The reviews are then sent to the authors who have a short period of time to write a rebuttal. The purpose of the rebuttal is to answer questions that the reviewers may have raised and correct any specific misunderstandings in the reviews.
Prior to the PC meeting, the reviewers have the opportunity to read the rebuttal for each paper they reviewed and update their reviews and scores as appropriate. During the physical PC meeting, each paper that has at least one supporting reviewer is discussed. The discussion is initiated by one of the reviewers (appointed as the meta-reviewer for that paper by the Track Chair) who summarises the work, and the main review comments. The other reviewers then articulate their views. The meta-reviewer provides a summary of the PC discussion as feedback to the authors from the PC meeting inside the online review system (or indicates that no such discussion took place). Papers that are shepherded will have clear points set out which must be addressed in the revision. If these points are not satisfactorily resolved, then the paper will not be accepted.
PC members with a COI with a paper will not review or see the reviews for that paper, and they will not be party to the discussions about it during the PC meeting.
The review process for any papers where the Track Chair / Program Chair has a COI are handled by an alternate chair (often someone who has previously chaired the conference) outside of the conference management system. This ensures that the Track Chair / Program Chair has no access to the reviews. The alternate takes over the physical PC meeting for a brief time to cover the Chair’s COIs.
Note submitted papers undergo plagiarism checks and the authors’ names are checked against the IEEE prohibited authors list.
Outstanding Paper, Best Paper, and Best Student Paper Awards
During the PC meeting, reviewers may nominate papers which they consider as being of outstanding quality. After the PC meeting, the Program Chair (or Track Chair in the case of Program Chair COI) is responsible for selecting a small sub-committee to consider which of the nominated papers will receive Outstanding Paper Awards. (Typically 3-4 papers receive such awards each year). Those papers form the shortlist for the Best Paper and Best Student Paper Awards (exceptionally a wider set of papers may need to be examined for the latter award). The recipients of these awards are determined by the sub-committee on the basis of the final versions of the papers.
To be eligible for the Best Student Paper Award, a paper must have as its first author a person who was a student at the time of the submission deadline.
Best Presentation Award
The Program Chair is responsible for selecting a small sub-committee who will attend the conference sessions and determine the recipient of the Best Presentation Award.
Selecting other Organisational Roles
The Program Chair is responsible in consultation with the General Chair for selecting the Workshops, Artefact Evaluation, Brief Presentations, RTSS@Work, and Publication Chairs.
A call for proposals for workshops at RTSS is made each year. Workshops are selected by a small sub-committee comprising the Workshops Chair, Program Chair, and General Chair. The criteria for selecting workshops are set out in the call for proposals. The space available for workshops may be limited depending upon the venue for RTSS.
Conflicts of Interest (COIs)
For the purposes of reviewing papers, PC members have a conflict with a paper, if they and the authors:
(i) Had at any time a supervisor/PhD student relationship.
(ii) Are both from the same institution, or have worked at the same institution in the past 3 years.
(iii) Are currently working together on a research paper or project, or have done so during the past 3 years.
(iv) Are related, or are close personal friends.
(v) Are in some form of financial relationship, or have been at some point during the past 3 years.
(vi) Any other reason or circumstance that creates a risk that professional judgement may be unduly influenced.
Since the Track Chairs / Program Chairs do not themselves review papers, but rather facilitate discussion, point (iii) above is deemed to be a COI only if there is a substantial history of the Chair working with the authors.
RTSS is taking a number of actions on diversity and gender equality in 2018:
(i) The number of female researchers on the Program Committee for the main conference is 12 out of 51 or 23.5%, which is a much higher proportion than in previous years. (Note all invitations to join the PC were made on the basis of merit).
(ii) The conference is collecting and will publish statistics on diversity (gender and geographical location) relating to the main authors of accepted papers and submissions.