From b8e50e037b562365f94903b0f748b32553ad8b2b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Anders=20M=C3=B8ller?= Date: Fri, 10 Jul 2026 19:44:36 +0200 Subject: [PATCH 1/2] Updated ProChair, Review, and Diversity pages --- Resources/Guidelines/ProChair.md | 778 +++++++++++++++---------------- Resources/Policies/Diversity.md | 128 +++-- Resources/Policies/Review.md | 139 +++--- 3 files changed, 511 insertions(+), 534 deletions(-) diff --git a/Resources/Guidelines/ProChair.md b/Resources/Guidelines/ProChair.md index f6f42f69..3520f390 100644 --- a/Resources/Guidelines/ProChair.md +++ b/Resources/Guidelines/ProChair.md @@ -3,13 +3,14 @@ layout: default title: "Guidelines for the Program Chair of a SIGPLAN Event" --- - +These guidelines are primarily intended for Program Chairs of SIGPLAN-sponsored conferences. +For organizers of workshops and similar events, see [this page](/Resources/Guidelines/Workshops). -[N.b.: SIGPLAN rules are highlighted like -this to distinguish them from general recommendations and -discussion.] +This document uses "Program Committee (PC)" and "Program Chair (PCC)" +as generic terms. For PACMPL conferences, "Program Committee" +corresponds to "Review Committee," and the current Program Chair +normally serves as an Associate Editor, as described in the +[PACMPL procedures](https://sigplan.org/PACMPL/pacmpl_procedures.pdf). Executive Summary ----------------- @@ -24,67 +25,26 @@ skim the rest of this page and the following others: - SIGPLAN's [diversity policy](/Resources/Policies/Diversity) - SIGPLAN's [republication policy](/Resources/Policies/Republication) - SIGPLAN's [code of conduct policy](/Resources/Policies/CodeOfConduct/) -- SIGACCESS's [Guide to Creating a Conference Accessibility FAQ Page](http://www.sigaccess.org/welcome-to-sigaccess/resources/creating-a-conference-accessibility-faq-page/) + +Most importantly: + - For a SIGPLAN-sponsored event, seek approval for the Program Committee from the SIGPLAN Vice-Chair, *before* sending invitations - Consult with the conference Steering Committee before diverging from any existing conventions -- Make sure the Call for Papers makes potential authors aware of SIGPLAN's - [Copyright policies](https://authors.acm.org/main.html). -- Some special considerations for organizers of workshops and similar events - can be found on [this page](/Resources/Guidelines/Workshops). +- Make sure the Call for Papers makes potential authors aware of ACM's + [policy on authorship](https://www.acm.org/publications/policies/new-acm-policy-on-authorship). _Please help us keep this document (and the rest of this site) up to date:_ If you notice any errors, duplications, inconsistencies, etc., please inform the [SIGPLAN vice chair](mailto:vc_sigplan@acm.org). -Submission and Reviewing Policies ---------------------------------- - - - -The Program Chair (PCC) may not submit papers to the conference; SIGPLAN's -Executive Committee recommends that the General Chair also not be -allowed to submit papers. If Program Committee members are allowed to -submit papers, either their papers should be held to a higher standard -or other EC-approved measures should be taken, to avoid the appearance -of impropriety. - -Conferences that use full double blind reviewing and have program committees with at least fifty members need not hold PC papers to a higher standard, as the EC has judged that these are sufficient conditions to avoid the appearance of impropriety. Here full double blind means submissions are anonymous, references to prior work are in the third person, authors' identities are not revealed until a paper is conditionally or fully accepted, and author identities for rejected papers (that were not conditionally accepted) are never revealed. - -The PC Chair should make all program committee members and reviewers aware -of SIGPLAN's policies on -[reviewing of conference paper submissions](/Resources/Policies/Review) and -on [plagiarism](http://www.acm.org/publications/policies/plagiarism_policy). - -The conference steering committee should be consulted for any changes to the -accepted way of organizing and running the conference. - -You, the Program Chair, in consultation with the General Chair, should -decide on general policies relating to the program committee, such as: - -- Will you have a physical PC meeting? If so, when and where will it be - held? If not, will the PC discussion be a "synchronous" teleconference or - an extended discussion over email or moderated by some conference - management platform? (Whichever you choose, SIGPLAN expects all PC - members to participate fully in the PC meeting, except in unusual - circumstances.) -- Who pays for the travel expenses of the PC members? -- Who pays for the conference registration of the PC members? Typically, - the PC member is responsible for all of his/her expenses, including - registration. If a decision is made to provide funding for some/all of the - members, these expenses must of course be included in the budget. - -The previous year's Program Chair for the same conference is a good -resource. - Program Committee Selection --------------------------- -In consultation with the General Chair, +In consultation with the General Chair (and the rest of the Steering Committee), develop a list of potential Program Committee members. In formulating this list, be mindful of [SIGPLAN's diversity policy](/Resources/Policies/Diversity). Also, some people will decline the -invitation, so your list should includes sufficient alternates. The list +invitation, so your list should include sufficient alternates. The list should be finalized at least one year prior to the meeting. Download the [PC proposal @@ -93,415 +53,423 @@ template and enter the requested information for your proposed PC members and alternates. Send the completed spreadsheet to the SIGPLAN Vice Chair ([vc\_sigplan@acm.org](mailto:vc_sigplan@acm.org)) for comments and approval by the SIGPLAN Executive Committee. The approval process normally takes -less than two weeks. The same approval process applies to all other -reviewing committees for conferences such as ERC and EPC. - - +less than two weeks. If the conference has other reviewing committees +(for example, an External Review Committee), +the same approval process applies. -At the same time, make plans for the Program Committee meeting, so that +At the same time, make plans for the review-discussion and decision process so that these can be communicated to potential PC members when you invite them: -choose the location (if physical) and finalize the date(s). +finalize key dates for online discussion periods and any virtual or in-person meetings. After SIGPLAN approval is confirmed, invite the people on the list to serve. Explain what is expected of them and any relevant -[policy decisions](/Resources/Policies/Review), and give them the date(s) for -the PC meeting. If more people than expected decline and you run out of +[policy decisions](/Resources/Policies/Review), and give them relevant dates. +If more people than expected decline and you run out of alternates, send the SIGPLAN Vice Chair another set of candidates for approval. -Recommendations: +Recommendations: -- The PCC should strive to avoid the appearance of favoring current and former students, postdocs, colleagues, or collaborators for membership on the PC or ERC. -- The size of the PC should be such that, given the expected number of submissions, will yield a reasonable reviewing load per PC member. Exceeding 25 papers per PC members is strongly discouraged. -- If there is an ERC, the same diversity criteria listed above should apply +- The PCC should strive to avoid the appearance of favoring current and former students, postdocs, colleagues, or collaborators for membership on the PC. +- The PC should be large enough to keep the expected reviewing load reasonable. Assigning more than 15 papers to a PC member is discouraged. +- If there are other reviewing committees, the same diversity criteria listed above should apply to them, scaled to the size of each committee. - - -- PC members should generally not serve on the PC more often than every four years. Exceptions may be made to enable a small overlap (<10%) from year to year to provide continuity. It is advisable that no one should serve on the PC for more than two years in a row. -- The PCC of year X+1 can be invited to serve on the PC (with a lighter reviewing load) for year X as soon as he or she is selected. -- The PCC should strive to avoid selecting PC members that collaborate regularly with each other or with the PCC, or PC members from the same institution in the exact same area. -- For conferences, all PC members should have PhDs (or other appropriate qualifications) by the submission deadline of the conference. For affiliated workshops, the PC can include PhD students in their final year, as long as the total number does not exceed 10% (rounded up) the PC. +- PC members should generally not serve on the PC for the same conference more often than once every three years. A small overlap of less than 15% from one year to the next may be used to provide continuity, but no one should ordinarily serve for more than two consecutive years. +- The PCC of year X+1 can be invited to serve on the PC (possibly with a lighter reviewing load) for year X as soon as they are selected. +- The PCC should avoid selecting a committee that is overly concentrated + among frequent collaborators, close colleagues, or researchers from + the same institution or research group. Applicable conflict-of-interest + rules should be followed when determining institutional or + organizational conflicts. +- PC members should have appropriate research experience and qualifications for evaluating submissions in the conference's area. For affiliated workshops, suitably qualified senior PhD students may serve where appropriate. +PC Member Responsibilities +-------------------------- +When inviting PC members, clearly communicate the expected reviewing +load, deadlines, discussion responsibilities, and applicable policies. +A person who cannot make that commitment should decline the invitation +or promptly inform the PCC if circumstances change. + +PC members should personally read their assigned submissions, provide +constructive and sufficiently detailed reviews, declare their expertise +accurately, read the other reviews and any author response, and +participate actively in discussion and decision making. + +The PCC should monitor timely completion, review quality, participation +in discussion, and compliance with SIGPLAN's +[Conference Submission Review Policy](/Resources/Policies/Review/). +Questions about related publications, conflicts, reviewer conduct, or +the integrity of the review process should be referred to the PCC. Call for Papers --------------- Develop a call for papers. Note that you can produce a preliminary call without listing the -PC members. Make sure to put the call on the web site for the -conference (established by the General Chair). The web page -should refer prospective authors to SIGPLAN's -[Republication Policy](/Resources/Policies/Republication) -and to ACM's -[Policy and Procedures on Plagiarism](http://www.acm.org/publications/policies/plagiarism_policy). -Once the conference web page is in place, establish a link to the page +PC members. Make sure to put the call on the website for the +conference (established by the General Chair). The web page should refer +prospective authors to SIGPLAN's +[Republication Policy](/Resources/Policies/Republication), +[Conference Submission Review Policy](/Resources/Policies/Review/), +ACM's +[Policy on Authorship](https://www.acm.org/publications/policies/new-acm-policy-on-authorship), +and ACM's +[Publications Policies](https://www.acm.org/publications/policies). +Once the conference web page is in place, establish a link to it from the SIGPLAN conference page by contacting the SIGPLAN [Information Director](mailto:infodir_sigplan@acm.org?subject=Conference%20Information). -If the proceedings will be published by ACM, declare explicitly in the call for -papers that authors of accepted submissions will be required to choose one of -the following options: Author retains copyright of the work and grants ACM a -non-exclusive permission-to-publish license (and, optionally, licenses the work -with a [Creative Commons license](https://creativecommons.org)); Author retains -copyright of the work and grants ACM an exclusive permssion-to-publish license; -Author transfers copyright of the work to ACM. For more information, please see -[ACM Copyright -Policy](http://www.acm.org/publications/policies/copyright-policy) and [ACM -Author Rights](http://authors.acm.org/main.html). - -Proceedings published by ACM as an issue of PACMPL are Gold Open Access, and -the PACMPL editorial board recommends the first option (author retains -copyright of the work, licenses the work with a [Creative Commons -license](https://creativecommons.org), and grants -ACM a non-exclusive permission-to-publish license). -The PACM Steering Committee also requires conferences to collect statistics -about the review process---see [here](https://www.acm.org/publications/pacm/title-assessment) for details. - -If presentations are to be recorded, mention that presenters must sign a -permission form in order for the video to be released online. - -Mention that publication of auxiliary material is encouraged, and note the -associated copyright policy (see 'Auxiliary material', below). +For proceedings published by ACM, the call for papers should +explain ACM's open-access publication model and refer authors to ACM's +current +[Open Access information](https://authors.acm.org/open-access), +[Publication Rights and Licensing Policy](https://www.acm.org/publications/policies/publication-rights-and-licensing-policy), +and +[Author Rights and Responsibilities](https://authors.acm.org/author-resources/author-rights). +The call should make clear that publication charges may apply unless the +corresponding author is covered by an ACM Open institutional agreement or an applicable +waiver or discount. Because the rates and arrangements may change, link +to ACM's current information rather than stating specific charges. + +For proceedings published as an issue of a *Proceedings of the ACM* +journal, consult the journal's Editor-in-Chief regarding any additional +journal-specific requirements, including the collection of +review-process statistics required by the PACM Steering Committee. See +[PACM Title Assessment and EiC-Provided Statistics](https://www.acm.org/publications/pacm/title-assessment) +for details. + +If presentations are to be recorded or streamed, inform presenters of +the applicable consent or release requirements. + +Explain whether authors may submit auxiliary or supplementary material, +whether reviewers are expected or merely permitted to consider it, and +any applicable anonymity requirements. Auxiliary material may be +published; see "Auxiliary material" below for the associated policy. +If the conference has a separate artifact-evaluation process, the call +or conference web page should clearly distinguish that process from the +submission of supplementary material with the paper. The conference web page should include information about the various -benefits offered by the [SIGPLAN Professional Activities -Committee](/PAC) to SIGPLAN members attending SIGPLAN-sponsored -conferences. These benefits include travel and lodging assistance for -student authors and assistance for members who need a companion -care-provider in order to attend. - -Author Response ---------------- - -Many SIGPLAN conferences have included author response mechanisms as part of -the paper review process, and the general feedback concerning author -response has been positive (see -[Experiences with Author Response at PLDI and ICFP 2004](http://portalparts.acm.org/1060000/1052883/fm/frontmatter.pdf), -Kathleen Fisher and Craig Chambers, *SIGPLAN Notices*, Dec. 2004). -Program Chairs and Steering Committees are encouraged to consider -allowing author response or rebuttals as part of the paper review -process. +benefits offered by the +[SIGPLAN Professional Activities Committee](/PAC) +to participants in SIGPLAN-sponsored conferences. These include travel +support for eligible students and other participants, child-care travel +support, and companion travel grants. Submission and Review --------------------- -Set up the paper submission web site. Many services are available for this -purpose; most SIGPLAN conferences these days use -[HotCRP](https://hotcrp.com) (which is free for us because ACM has a bulk -deal). If you choose a different one, look for support for the [ABCD rating -scale][Nier00], author feedback, and double-blind reviewing. - -[Nier00]: http://www.iam.unibe.ch/~oscar/Champion/champion.pdf "Oscar Nierstrasz, _Identify the Champion_" - -Some advice: Many systems make it easy to include ratings on a -large number of different scales, but please don't do this. Most -PC chairs use only two scales: an overall rating and an -expertise rating. It also works best to have just two or three text -fields: one for a summary of the paper (this is optional), one for -comments for both the PC and the authors, and one for comments to -be seen by the PC only. The ABCD scale avoids ranking papers by -average: instead you can group papers by highest and lowest rating -(not a linear scale, since AD neither precedes nor follows BC). -For more details see: -[Conferences with improved management and style](http://portalparts.acm.org/1140000/1137933/fm/frontmatter.pdf), -Philip Wadler, _SIGPLAN Notices_, Feb. 2006. - - - -Before papers are received, develop an evaluation procedure. After the -submission deadline, send submitted papers and a description of the -evaluation procedure to the PC members. - -The PCC should identify another PC member (a “COI-PCC”) to handle papers for -which the Chair is in conflict. - - -PC Meeting ----------- - -Some SIGPLAN conferences have in-person PC meetings; others have moved to -virtual meetings (to reduce carbon footprint, wear and tear on PC members, -or both). - -For both kinds of meetings, PC members should be allowed to participate in -the discussion (both online and in-person) of papers that they didn't -review, and with which they have no conflicts. The PCC should strive to -ensure, prior to the PC meeting, that every competitive paper has at least -two PC reviews at least one, and preferably two expert reviews - -During the online discussion period (whether or not followed by an in-person -meeting), the PC should be allowed to see all papers with which they have no -conflicts, and participate in the discussion. Online discussion must be -actively facilitated by the PCC, and (if applicable) paper guardians. - -Before the PC meeting, decide on any policies that should be in effect, such as: -- What defines a conflict of interest? -- What happens if a PC member has a conflict of interest? -- What are the rules of confidentiality? -- Do you want to have a best paper award? - -### In-Person PC Meetings - -A physical PC meeting should be preceded by an extensive online discussion -period. The online discussion period should be a minimum of one week, but -two (or more) is strongly preferred. - -The designated COI-PCC (see COI policies) should handle papers the PCC is -conflicted with during the meeting. - -At the end of the PC meeting, the PCC may want to take a bit of time to plan the -sessions and the session chairs. - -It is strongly recommended that the PCC have an assistant during -the physical PC meeting, to take on certain administrative and logistical -tasks during the meeting and allow the PCC to concentrate on -leading the discussions of papers. Typical duties of the assistant would be -to keep track of the disposition of papers for the PC, interface with the -on-line review system, and record any relevant information about a -particular paper (e.g., paper Y paper is being shepherded by committee -member X, committee member X agrees to revise their on-line review, -etc.). Past PCCs have used graduate students or post-docs as -assistants, with good results. - -Any PC meeting expenses that are to be reimbursed by the conference should -be put in the conference budget. Send receipts to the ACM Representative to -SIGPLAN for reimbursement (the [SIGPLAN Vice -Chair](mailto:vc_sigplan@acm.org) can tell you who this is at the moment). - -Additional suggestions: -- Experience has shown that a two day PC meeting provides adequate time for deliberation while avoiding PC burnout. -- The GC may play the role of "COI-PCC". -- The PCC is encouraged to have one or two dedicated assistants to deal with timekeeping and bookkeeping in order to allow the PCC to focus entirely on the discussion rather than logistical concerns. A senior non-PC member such as the GC or next PCC can fulfill this role. -- Instead of considering the papers in order from highest-ranked to lowest, consider the papers in a quasi-random order (see article by Kathleen Fisher in SIGPLAN Notices, 46(4):17, April 2011). -- If the most positive reviewer for a paper is external, the chair may wish to ensure that the paper is discussed on day one of the PC meeting and the outcome summarized by the PCC so that the external reviewer can provide additional feedback prior to a final decision. -- The PCC is encouraged to use a multi-round discussion process for papers where an initial consensus is not apparent. -- The PCC can distinguish between contentious and marginal papers and table all marginal papers until the end of the meeting to allow them to be systematically considered in the context of the emergent acceptance standard. - -After the PC Meeting --------------------- +Set up the submission and review site sufficiently early to test the +submission form, review form, conflict declarations, anonymity settings, +reviewer assignments, author response, discussion, and decision +notifications. Most SIGPLAN conferences use +[HotCRP](https://hotcrp.com/), but the PCC may use another system that +supports the conference's reviewing model and applicable SIGPLAN +policies. + +Before submissions open, determine and publish the main elements of the +review process, including: + +* the reviewing model and anonymity requirements; +* the evaluation criteria; +* the treatment of supplementary material; +* whether and when author response will be permitted; +* whether papers may be conditionally accepted or invited for revision; +* the applicable conflict-of-interest rules; and +* any rules concerning the use of external reviewers. + +The review form should be kept reasonably simple, reflect the published +evaluation criteria, and explain its questions and rating scales clearly. +The PCC should avoid treating numerical scores or averages as substitutes +for substantive judgment and discussion. + +### Managing the Review Process + +Keeping the review form reasonably stable across editions can help +authors and reviewers calibrate and interpret its ratings. +One possible model is Nierstrasz's Identify the Champion scoring process, +which has stood the test of time. A typical review form in the spirit of +this process has (at least) the following fields: +- Paper score: + - A: Good paper. I will champion it during the discussion. + - B: OK paper, but I will not champion it. + - C: Weak paper, though I will not fight strongly against it. + - D: Serious problems. I will argue to reject this paper. +- Reviewer expertise: + - X: I am an expert in the subject area of this paper. + - Y: I am knowledgeable in the area, though not an expert. + - Z: I am not an expert. My evaluation is that of an informed outsider. +- Paper summary (reviewer’s synopsis provides a useful sanity check) +- Points in favor (encourage positive reviewing) +- Points against (be clear about principal objections) +- Detailed comments for authors (body of review) - - - -Send acceptance and rejection letters as soon as possible after the PC -meeting. Authors using LaTeX should be directed to the [template -developed by SIGPLAN](/Resources/Author). This template was designed to -allow more material in a paper while remaining easy on the eyes -(typically, a paper that requires 12 pages with the standard ACM -format requires 11 with the SIGPLAN format, and most readers find the -SIGPLAN style easier to read). +Additionally: +- The PCC should become familiar with the conference management system + and its facilities well before submissions open. +- A two-phase reviewing process may be used to reduce reviewing load, + provided that the early phase gives each submission an adequate expert + assessment and follows the review process described to authors. +- PC members should be strongly encouraged to submit reviews as they are completed; + this makes it easier for the PCC to monitor progress and identify problems early. +- The PCC should allow all reviewers to see the decisions for the papers + they reviewed before those decisions are communicated to authors. + +The PCC should ensure that each submission receives reviews of sufficient +quality and breadth. In particular, the PCC should identify papers for +which the assigned reviewers lack sufficient expertise, reviews are +incomplete or inadequate, or reviewers substantially disagree, and seek +additional reviews where appropriate. +External reviewers should be selected and managed in accordance with SIGPLAN's +[Conference Submission Review Policy](/Resources/Policies/Review/). + +### Guardians + +The PCC may assign a PC member as guardian for a submission. A guardian +is responsible for helping ensure that the paper is assessed carefully +and fairly. Depending on the conference's review process, the guardian's +role may include: + +* identifying gaps in reviewer expertise; +* checking that the reviews are sufficiently complete and constructive; +* facilitating discussion among the reviewers; and +* drafting or coordinating the author-visible decision rationale. + +The guardian does not replace the assigned reviewers and does not make +the acceptance decision. The PCC retains responsibility for ensuring +that the review process is fair and that the final decision is properly +supported. + +### Author Response + +Configure and announce the author-response process in accordance with +SIGPLAN's +[Conference Submission Review Policy](/Resources/Policies/Review/). +The PCC should make every effort to complete the initial reviews before +the response period, ensure that reviewers read and consider the +responses, and incorporate them into subsequent discussion and +decisions. + +### Discussion and Decisions + +The PCC should organize an active discussion period after the initial +reviews and any author response. Discussion may take place +asynchronously in the conference management system, in synchronous +online meetings, or through a combination of these. + +The PCC, and any designated discussion leads, should ensure that +reviewers read the other reviews and the author response, discuss +significant disagreements and outstanding questions, and assess papers +according to the published criteria. Concerns raised late in the process +should be communicated to the other reviewers and considered fairly. + +The PCC should make acceptance decisions based on the reviews, +discussion, author response, and published evaluation criteria. +Numerical scores and votes may inform a decision but should not determine +it mechanically. Before decisions are communicated to authors, each +decision should be supported by a clear author-visible rationale that +accurately reflects the main points of the discussion. + +### Conflicts + +The PCC should designate another qualified committee member, commonly +called the COI-PCC, to manage submissions for which the PCC has a +conflict of interest. The COI-PCC should have the authority and access +needed to manage reviewing, discussion, and decisions for those +submissions independently of the PCC. + +### Shepherding + +A conference may use shepherding or another conditional-acceptance +process when a paper can be accepted subject to clearly identified +revisions. The PCC should appoint a qualified reviewer or committee +member to oversee the revision. In consultation with the other +reviewers, the shepherd should provide the authors with a concise list +that clearly distinguishes changes required for acceptance from +optional suggestions. + +The shepherd should assess whether the revised paper adequately +addresses the required changes and report this assessment to the PCC. +The shepherd should not impose substantial new requirements that were +not part of the acceptance decision without consulting the PCC and the +other reviewers. The PCC retains responsibility for the final +acceptance decision. Managing the Proceedings ------------------------ -The proceedings service by Conference Publishing Consulting (ConfPub) -can be initiated once you decide -that you want formal proceedings. Please contact the -[ACM Publications Coordinator](mailto:griscti@acm.org), Adrienne Griscti, -to begin the process. ACM will provide ConfPub with pertinent -conference information. ConfPub will forward you a schedule and -instructions on how to proceed. - - -Draft a "Letter from the Program Chair," which should -include a description of the paper selection process (number of -papers submitted and accepted, etc.). Forward the letter to -ConfPub along with any other front matter (perhaps a "Letter from the -Conference Chair," a List of Reviewers, etc.) for inclusion -in the proceedings. ConfPub will format all front matter; their -schedule will include a deadline for front matter submission. They -will also compile the table of contents/author index, using the -schedule of talks/advanced program that you provide to determine -paper order. - -Accepted authors will submit their papers directly to the ConfPub -web site. Once ConfPub have collected all of the papers/copyright -forms and front matter, compiled the table of contents/author index -and paginated the volume, they will provide camera-ready copy for -your review. The proceedings will not be printed until this -has been approved. It is a good idea to send -the table of contents to the contributing authors to confirm the -spelling of their names and affiliations. You then will be asked to -provide conference shipping information and the number of proceedings -needed for the conference. - -At the Conference ------------------ - -At the meeting, give a short summary of the paper evaluation -process. Include the number of submissions, in what topical areas, -and the number of accepted papers. You might want to compare the -number of submissions and acceptances with the previous year (these -numbers should also be in the proceedings and the final report). -You should also explain the review process, e.g, how many PC -members reviewed each paper and the organization of the PC meeting. -You might want to talk about the geographic distribution of -submissions and any unusual trends. The summary should take about -15 minutes. - -After the Conference --------------------- - -Help the Conference Chair prepare the post-event report for SIGPLAN EC. - - -As former Program Chair, you may be invited to serve on the -conference's Steering Committee. For the formal -rules for your particular conference's steering committee, consult the relevant SIGPLAN -conference web page. Links to these pages can be be found -[here](/Conferences). - -The program chairs of the major SIGPLAN conferences (POPL, PLDI, -OOPSLA, and ICFP) are automatically on the selection committee for -the following year's -[SIGPLAN Programming Languages Achievement Award](/Awards/Achievement). - - -Artifacts ---------- - -ACM treats artifacts as "first-class" objects in the digital library, -that is, each artifact has its own DOI, meta data, and landing page. -AEC chairs should send a list of papers with accepted artifacts -and their badge assignment to ConfPub. -They will print the badges on the papers at a uniform position. -ConfPub also collects the meta data from the authors and provides -the data to ACM. - - -Auxiliary Material ------------------- - -The ACM Digital Library and ConfPub, who prepare many -ACM proceedings, are both set up to accept auxiliary material. The -SIGPLAN EC encourages SIGPLAN-sponsored conferences to accept such -material as part of the publication process. (Some conferences also -accept auxiliary material with submissions; this is an independent -issue.) - -Auxiliary material may include - -- a technical report with additional details -- source code for software -- proof scripts for automated theorem provers -- test data - -or anything else relevant. It is SIGPLAN policy to encourage authors to -publish adequate auxiliary material to enable others to reproduce their -work. Authors retain copyright of their auxiliary material, while copyright -of the paper can be either assigned to ACM or held by the authors. Referees -may examine auxiliary material, if it accompanies the submission, but this -should not be considered part of the review process. For an example of -auxiliary material in the Digital Library, see -[here](https://doi.org/10.1145/1462173.1462178) and click on -"Supplemental Material". - - - -Shepherding ------------ - - -Papers may be accepted conditionally, in which case, such papers require shepherds. The role of the shepherd is to compile a list of required changes to the paper that are required for the paper to be accepted, in consultation with all reviewers, and subject to the approval of the PCC. Authors are expected to modify their papers appropriately; in the event that the reviewers and the shepherds do not come to an agreement that a paper has been revised satisfactorily, the PCC should have the ultimate decision as to whether the paper will be accepted. - - -Additionally, the shepherds can ensure that outstanding work is not rejected on the basis of a problem that can easily be fixed with minor modifications to the paper. - +Most SIGPLAN-sponsored conferences use +[Conference Publishing Consulting](https://www.confpub.com/) +to manage the proceedings-production process. Contact ConfPub early and +agree on the production schedule, author instructions, required +metadata, and division of responsibilities. For proceedings published +as an issue of a Proceedings of the ACM journal, also coordinate with +the journal's Editor-in-Chief and follow any journal-specific +requirements. + +After acceptance decisions have been finalized, provide ConfPub with +complete and accurate metadata for all accepted contributions, +including titles, author names and affiliations, contact information, +submission identifiers, and contribution types. ConfPub will normally +use this information to initiate ACM's publication-rights process and +the production workflow. + +Authors must complete ACM's publication-rights process and provide their +final source material in accordance with the instructions supplied by +ConfPub. ACM currently uses +[The ACM Publishing System (TAPS)](https://authors.acm.org/proceedings/production-information/taps-for-conference-organizers) +to produce and validate the PDF and HTML versions of published +articles. Depending on the production workflow, authors or ConfPub may +submit and correct the source files in TAPS. ConfPub coordinates the +process, communicates instructions and deadlines to authors, monitors +progress, and helps resolve production problems. + +The PCC should provide authors with clear deadlines and instructions +for preparing their final versions, including the applicable ACM +template, page limits, citation style, accessibility requirements, +supplementary or auxiliary material, and treatment of third-party +material. Allow sufficient time for authors to address formatting, +metadata, rights, and TAPS validation problems. + +Before the proceedings are finalized, the PCC or a designated +proceedings chair should verify that all accepted contributions are +included and that their titles, author lists, ordering, and other +metadata are correct. Also check the front matter, table of contents, +and any required review-process statistics or journal-specific +information. Coordinate corrections with ConfPub rather than making +changes directly in ACM's production systems. + +### Artifacts + +ACM supports the evaluation and publication of research artifacts and +displays applicable artifact badges with the associated papers in the +ACM Digital Library. + +The Artifact Evaluation Committee chairs should provide ConfPub with the +final list of papers whose artifacts have received badges and the badge +assignments. ConfPub coordinates the inclusion of the badges in the +published papers and the collection of any required artifact metadata. + +The conference should give authors clear instructions concerning artifact +submission, evaluation, archival availability, licensing, and any +requirements for persistent identifiers or external repositories. + +### Auxiliary Material + +ACM proceedings may include auxiliary material associated with a +published paper, such as: + +- technical reports containing additional details; +- source code or software; +- proof scripts; +- data sets; and +- other material relevant to understanding, reproducing, or building on + the work. + +SIGPLAN encourages authors to make appropriate auxiliary material +available when it supports reproducibility and future research. The +conference should provide authors with clear instructions for submitting +such material through the applicable proceedings-production process. + +ACM uses the term *auxiliary material* for files that are provided as an +additional resource but are not reviewed as part of the published work. +Authors retain copyright in such material and grant ACM permission to +make it available through the ACM Digital Library. Authors should ensure +that they have the necessary rights to distribute all included content. + +Auxiliary material submitted for publication should be distinguished +from supplementary material submitted during paper review. The call for +papers should explain whether reviewers are expected or merely permitted +to consider supplementary material. Supplementary material must comply +with the same anonymity requirements as the paper. Distinguished Papers -------------------- -Up to 10% of the accepted papers may be designated for ACM SIGPLAN Distinguished Paper Awards. -- Nominations may be solicited from the PC and ERC, and will also include the top 10% papers in terms of numeric scores. PC/ERC members should not nominate papers by authors they are conflicted with. -- PC papers may be nominated. -- The PC/ERC (excluding the authors of any nominated papers) should vote on this set of nominated papers. Each PC member may cast as many votes as the target number of Distinguished Papers. The usual conflict-of-interest rules will apply (e.g., PC/ERC members cannot vote on papers with which they have a conflict of interest). -- The PCC and GC should decide which papers will receive the award. While the number of votes received by nominated papers is an important factor, the PCC and GC may exercise their judgment in making their decision. -PC Member Responsibilities --------------------------- +Up to 10% of the accepted papers may be selected for ACM SIGPLAN +Distinguished Paper Awards. The selection process should consider +relevance, originality, significance, and clarity, and should not rely +mechanically on numerical review scores. -PC members must commit to reading all of their assigned papers and writing their own reviews. PC members may also suggest additional reviewers, but they should not subcontract reading or review writing duties to others. The PCC should be consulted before additional reviewers are contacted to avoid conflicts and to ensure topical balance. +The PCC should establish a fair process for identifying and assessing +candidate papers. Nominations may be solicited from the PC and any +other reviewing committees, and the PCC may appoint a suitably qualified +subset of the committee to assess the candidates. Papers with PC members +among the authors remain eligible. -If there is an in-person PC meeting, then PC members must attend the PC meeting in-person. If the meeting is virtual, PC members must participate in the manner described in the invitation to join the PC. Invited members who cannot commit to attending or participating in the meeting in person in the way the PC chair has specified should decline the invitation. +The usual conflict-of-interest rules apply throughout the process. +Conflicted committee members must not nominate, assess, discuss, or vote +on the relevant papers. The PCC and General Chair have final +responsibility for the selection, unless the conference has an approved +alternative procedure. -_Recommendations:_ +At the Conference +----------------- -The PCC should consider whether to explicitly allow PC members to share and discuss papers with their own students (subject to the usual confidentiality and COI provisions), and to incorporate information from such discussions into their review. If this is permitted, the PC member should still be responsible for writing the review. When a student is involved in this capacity, the PCC should be informed so that the student receives appropriate credit. +Coordinate with the General Chair and other organizers on the PCC's +responsibilities during the conference. These may include introducing +the technical program, presenting paper or artifact awards, assisting +session chairs, and handling program changes or problems involving +presenters. + +Provide attendees with a brief summary of the submission and review +process, either during the conference or in the proceedings or +conference website. This will typically include the numbers of +submissions and accepted papers, the reviewing model, the number and +types of reviews obtained, and any other information useful for +understanding how the program was selected. Relevant comparisons with +previous years or aggregate statistics about submissions may also be +included. + +Ensure that session chairs have the information they need and know whom +to contact if a presenter is absent, a session runs late, or another +problem arises. Coordinate any changes to the published program with +the General Chair and web or publicity chairs. + +Be familiar with the conference's code-of-conduct and incident-response +arrangements, and promptly refer any incident to the appropriate +conference organizer. -PC members should consult the SIGPLAN Republication Policy. If a related version of the paper appeared in a workshop, PC members should take into account whether its call for papers stated that publication in the workshop is not intended to preclude later publication. +After the Conference +-------------------- +Prepare a post-event report for the conference Steering Committee, if +the conference has one, and otherwise for the +[SIGPLAN Executive Committee](/ContactUs). +The report should summarize the submission, review, +acceptance, and technical-program statistics, note any significant +issues that arose, and record recommendations for future editions of +the conference. -Recommendations for Reviewing Process Management ------------------------------------------------- +Prepare a brief handover for the next PCC, including the review-process +timeline, committee organization, relevant statistics, useful +procedures, and lessons learned. -Keeping the same reviewing form has benefit for authors and reviewers as they can more easily calibrate and interpret scores if the ratings are well understood. Nierstrasz' Identify the Champion scoring process has stood the test of time. A typical review form in the spirit of this process has (at least) the following fields: -- Paper score: - --A: Good paper. I will champion it at the PC meeting. - --B: OK paper, but I will not champion it. - --C: Weak paper, though I will not fight strongly against it. - --D: Serious problems. I will argue to reject this paper. -- Reviewer expertise: - --X: I am an expert in the subject area of this paper. - --Y: I am knowledgeable in the area, though not an expert. - --Z: I am not an expert. My evaluation is that of an informed outsider. -- Paper summary (reviewer’s synopsis provides a useful sanity check) -- Points in favor (encourage positive reviewing) -- Points against (be clear about principal objections) -- Detailed comments for authors (body of review) - -Additionally: -- It is useful for the PCC to acquaint themselves ahead of time with the conference management system (currently HotCRP), and the facilities it offers. -- A two-phase reviewing process can be used, in order to avoid PC overload, by identifying weaker papers earlier. -- PC members should be strongly encouraged to submit reviews as they are completed; this makes it easier for the PCC to monitor progress and identify problems early. -- The PCC may wish to identify major reviewer disagreements and papers without sufficient expertise and seek second opinions where necessary to help resolve the disagreements. -- The PCC should allow all reviewers to see decisions for the papers they have reviewed before decisions are publicly announced. +Complete any remaining proceedings, award, artifact-evaluation, or +journal-specific tasks, and ensure that outstanding author or reviewer +communications have been handled. +Former PCCs may also be invited to serve on the conference Steering +Committee or otherwise advise future organizers. Important Email Addresses ------------------------- _SIGPLAN Chair_: [chair\_sigplan@acm.org](mailto:chair_sigplan@acm.org) + _SIGPLAN Vice Chair_: [vc\_sigplan@acm.org](mailto:vc_sigplan@acm.org) + _SIGPLAN Information Director_: [infodir\_sigplan@acm.org](mailto:infodir_sigplan@acm.org?subject=Conference%20Information) See Also -------- -- ACM's *[How to Organize a Conference](http://www.acm.org/sigs/volunteer_resources/conference_manual/conference_manual-toc)* manual -- [ACM's How to Organize a Conference](http://www.acm.org/sigs/volunteer_resources/conference_manual/manual_read_me) -- SIGPLAN conference style [LaTeX class file](http://www.sigplan.org/sites/default/files/sigplanconf.cls), - [article template](http://www.sigplan.org/sites/default/files/sigplanconf-template.tex), and - [user guide](http://www.sigplan.org/sites/default/files/sigplanconf-guide.pdf) +- ACM's *[SIG Conference Planning Guide](https://www.acm.org/special-interest-groups/volunteer-resources/conference-planning)* +- ACM's [Primary Article Templates](https://www.acm.org/publications/proceedings-template) + and [LaTeX best practices for TAPS](https://www.acm.org/publications/taps/latex-best-practices) - [Author Information for SIGPLAN Conferences](/Resources/Author) - [Guidelines for Referees](/Resources/Guidelines/Referees) - -Some columns containing useful advice for conference organizers have -been published in _SIGPLAN Notices_ and elsewhere over the years: - - - -- [Experiences with Author Response at PLDI and ICFP 2004](http://portalparts.acm.org/1060000/1052883/fm/frontmatter.pdf), Kathleen Fisher and Craig Chambers, (*SIGPLAN Notices*, Dec. 2004) -- [Conferences with improved management and style](http://portalparts.acm.org/1140000/1137933/fm/frontmatter.pdf), Philip Wadler (*SIGPLAN Notices*, Feb. 2006) -- [SIGPLAN EC Activities: Vice Chair Report](http://portalparts.acm.org/1150000/1149982/fm/frontmatter.pdf), Kathleen Fisher (*SIGPLAN Notices*, May 2006) -- [A Report from the POPL 2007 Chairman](http://portalparts.acm.org/1230000/1229493/fm/frontmatter.pdf), Matthias Felleisen (*SIGPLAN Notices*, December 2006) -- [How to chair a conference](http://portalparts.acm.org/1990000/1988042/fm/frontmatter.pdf), Alex Aiken (*SIGPLAN Notices*, April 2011) -- [POPL'12 program chair's report: (or, how to run a medium-sized conference)](http://dl.acm.org/citation.cfm?id=2442779), Michael Hicks (*SIGPLAN Notices*, April 2012) -- [Artifact evaluation for software conferences](http://dl.acm.org/citation.cfm?id=2502518), Shriram Krishnamurthi (*SIGPLAN Notices*, April 2013) -- [The OOPSLA two-phase review process](http://dl.acm.org/citation.cfm?id=2641648), Cristina V. Lopes (*SIGPLAN Notices*, April 2014) -- [A Handbook for [ECOOP] PC Chairs](http://www.pps.univ-paris-diderot.fr/~gc/papers/pc-handbook.pdf), Giuseppe Castagna -- [Refreshing ATC – USENIX ATC 2019 Program Co-Chairs Message](https://www.usenix.org/sites/default/files/atc19_message.pdf), Dan - Tsafrir and Dahlia Malkhi diff --git a/Resources/Policies/Diversity.md b/Resources/Policies/Diversity.md index 47718261..e56d5a3d 100644 --- a/Resources/Policies/Diversity.md +++ b/Resources/Policies/Diversity.md @@ -2,54 +2,84 @@ layout: default title: "SIGPLAN Diversity Policy" --- -SIGPLAN values diversity in the composition of the program and -organizing committees for its meetings, recognizing that there is a -large pool of people qualified to serve on the committees. Each -year, meeting organizers must choose a subset of these qualified -people to serve on program committees. People chosen to serve on -such committees are seen as leaders in the community, and hence we -must take pains to ensure that they are chosen fairly and represent -the breadth of the community. Diversity should also be considered -with respect to the composition of previous program committees to -ensure fair representation over time. - -Factors that organizers should consider when forming program -committees from the pool of qualified candidates include: - - - - - -1. Coverage of meeting topics -2. Turnover: each year's committee should be substantially - distinct from previous years'. Some overlap to promote continuity - is permissible. -3. Diversity of institution: there should be no more than - two people from a given institution. - (Multiple sites for the same organization count as the same - "institution". For large PCs, this may be relaxed to "no more than 10%, - rounded to the nearest integer, from a given institution".) -4. Type of institution: Industry, Academia, Government Laboratory, - etc. -5. Years of experience: committees should be a mix of junior, - mid-level, and senior members of the community. (Workshop PCs may - include at most one PhD student in their final year; please include a - comment detailing their qualifications. Conference PC members should - all have PhDs, or other appropriate qualifications, by the submission deadline.) -6. Geographic distribution: members should come from all parts of - the world active in the given research area. -7. Gender. -8. Ethnic background. - -In addition, the PC Chair should strive to avoid the appearance of favoring -current and former students, postdocs, colleagues, or collaborators for -membership on the PC or ERC. - -Meeting organizers are required to submit their organizing and program -committees to the + +SIGPLAN values diversity in the composition of the organizing and +reviewing committees for its events. There is a broad pool of people +qualified to serve on these committees, but each year organizers must +select only a subset of them. Committee membership is a form of +professional recognition and leadership, so organizers must take care +to select members fairly and to represent the breadth of the relevant +community. + +Diversity should be considered both within the current committee and +across successive editions of an event, so that opportunities to serve +are distributed fairly over time. + +This policy applies to Program Committees and other reviewing +committees, including External Review Committees. The principles should +also be considered when forming organizing committees. Their +application may be scaled to the size, purpose, and expertise +requirements of each committee. + +When forming a reviewing committee, organizers should consider: + +1. **Coverage of the event's topics.** The committee should collectively + have the expertise needed to evaluate submissions across the scope of + the event. + +2. **Rotation and continuity.** Each year's committee should be + substantially distinct from those of previous years, while allowing + limited overlap to provide continuity. Members should generally not + serve on the same conference's Program Committee more often than once + every three years. Ordinarily, overlap from one year to the next + should be less than 15%, and no one should serve for more than two + consecutive years. + +3. **Institutional diversity.** Ordinarily, no more than two members + should come from the same institution. For large committees, this + limit may instead be interpreted as no more than 10% of the committee. + For the purpose of applying this limit and determining institutional + conflicts, a large organization with multiple campuses or + organizational units may be treated as comprising separate + institutions where those units are comparable in size and + organizational independence to typical academic institutions. + +4. **Type of institution.** The committee should, where relevant, + include members from academia, industry, government laboratories, + and other research organizations active in the area. + +5. **Career stage and experience.** The committee should include an + appropriate mix of junior, mid-career, and senior members of the + community. Conference Program Committee members should normally hold + a PhD or have comparable research qualifications by the submission + deadline. For affiliated workshops and similar events, suitably + qualified senior PhD students may serve where appropriate. + +6. **Geographic distribution.** Members should be drawn from the + regions of the world active in the relevant research area. + +7. **Diversity of backgrounds.** The committee should reflect diversity + in gender and in ethnic, cultural, and other backgrounds relevant to + representation in the community. + +These considerations should be applied holistically rather than as +independent quotas. Organizers should be prepared to explain +significant departures from them, particularly where a committee is +large enough that broader representation could reasonably have been +achieved. + +The Program Chair should also avoid selecting a committee that appears +to favor current or former students, postdoctoral researchers, +colleagues, or collaborators. The committee should not be overly +concentrated among frequent collaborators, close colleagues, or +members of the same institution or research group. + +Proposed reviewing committees for SIGPLAN-sponsored conferences and +SIGPLAN-approved workshops must be submitted to the [SIGPLAN Vice Chair](mailto:vc_sigplan@acm.org?subject=Program%20Committee%20Approval) -for approval by the Executive Committee, in the form of a +for approval by the SIGPLAN Executive Committee, using the [PC proposal spreadsheet](/Resources/Guidelines/sigplan-pc-proposal-template.xlsx). + +Further guidance on committee selection, approval, invitations, and +committee management is available in the +[Guidelines for Program Chairs](/Resources/Guidelines/ProChair/). diff --git a/Resources/Policies/Review.md b/Resources/Policies/Review.md index 244e7f79..dd1b0db7 100644 --- a/Resources/Policies/Review.md +++ b/Resources/Policies/Review.md @@ -1,38 +1,28 @@ --- layout: default -title: "Policies Regarding Review of Conference Papers" +title: "Conference Submission Review Policy" --- - - - **Confidentiality** - - -Papers under consideration for publication should not be distributed -or disclosed to any individual, except with the explicit authorization -of the program chair for purpose of review. Accepted papers are -considered unpublished until the day of the conference, and should not -be discussed outside the program committee without explicit permission -from the authors. All reviewers must be made aware that submissions -are unpublished works, and in no case should they duplicate them, pass -them on to other people, or discuss them with people other than the -members of the program committee. The program chair should keep a -record of the external reviewers used to review submissions. +Submissions must be treated as confidential and must not be distributed +or disclosed except to people explicitly authorized by the PCC for the +purpose of review. Reviewers must not copy submissions, pass them to +other people, or discuss them with anyone who is not authorized to +participate in the review process. Confidentiality continues until the +work has been publicly released by the authors or publisher. The PCC +must keep a record of all external reviewers. Reviewers grant paper authors permission to use the reviews however they wish. Reviews can also be analyzed under the auspices of SIGPLAN for research purposes such as evaluation of bias. -The title, authors, acceptance status, and other metadata (including -discussion summaries, if they exist) can be shared under the -discretion of the program chair, when there is an expectation of -legitimate benefit for other SIGPLAN processes (such as a related -review committee, or an artifact evaluation process). +Submission metadata, acceptance status, and decision summaries may be +shared with another authorized SIGPLAN process, such as a related review +committee or artifact-evaluation committee, when necessary for that +process. Only the information needed for the stated purpose should be +shared, and all recipients must observe the applicable confidentiality +requirements. **Conflicts** @@ -42,10 +32,7 @@ A conflict of interest is defined as a situation in which the reviewer can be viewed as being able to benefit personally from the outcome of a review, or in which the reviewer is not able to remain objective for personal reasons. If a conflict of interest exists, then the reviewer -should decline to review a paper. During the program committee -meeting, any committee member who has a conflict of interest regarding -a paper may be asked to leave the room for the duration of the -discussion. +should decline to review a paper. Among the situations in which conflicts are typically considered to exist are between an advisor and an advisee (Ph.D. and post-doc; @@ -55,9 +42,22 @@ companies or different locations of research institutes are considered to be the same institution; for two years after leaving an institution), between people with financial conflicts of interest, and between friends or relatives. If a reviewer feels unable to render an -objective judgement for any reason, he or she should notify the +objective judgment for any reason, they should notify the program chair. +If the PCC has a conflict of interest with a submission, the PCC must +not have access to or participate in its reviewing, discussion, or +decision. Another qualified committee member must manage the submission +independently. + + +**Use of generative AI or LLM systems** + +Please see the [FAQ for the general ACM policy](https://www.acm.org/publications/policies/peer-review-faq). +In particular: + +1. *Confidentiality of submissions*. Reviewers may not upload or paste a submission, or any parts of its content, into any generative AI or LLM system, or any other third-party service, that does not guarantee the confidentiality of the submitted material. +2. *Integrity of reviews*. Reviewers are expected to personally read, evaluate, and form judgments about submissions. AI tools should not be used as substitutes for the reviewer's expertise or judgment. AI may be used to correct spelling, grammar, and presentation, and to improve the clarity of reviewer-written text (provided that confidentiality is ensured). Reviewers remain fully responsible for the contents of their reviews, including any text that has been edited or generated with AI assistance. **Contacts** @@ -67,7 +67,6 @@ papers. Any interaction should go through the program chair. Review Process -------------- - **Author Anonymity** SIGPLAN conferences should implement double-blind reviewing. The primary goal of DBR @@ -77,85 +76,65 @@ learning their identity: 1. Submissions should not have author names, and references to previous work must be in the third person. 2. Authors are not required to "hide" their submissions: they can put them on web pages and give talks about them. -3. Reviewers should notify the PCC if anonymity is an obstacle to sound and objective reviewing of a paper. - -Given the use of DBR, the PCC must vet any external reviewers suggested by a PC member. - -_Recommendations:_ +3. Reviewers should notify the PCC if anonymity is an obstacle to sound and objective reviewing of a paper. Provision (2) above is intended to prevent DBR from inhibiting normal dissemination of scientific ideas. Authors should not, however, take it as a license to explicitly lobby the PC or likely reviewers on behalf of their work. While SIGPLAN will not attempt to codify or police such behavior, PC chairs may wish to remind authors of their obligation to live up to the spirit as well as to the letter of DBR. A major point of contention concerns posting research results in social media or wide-reach mailing lists. PC Chairs should carefully consider to what extent such publicity conforms to their implementation of DBR and inform authors accordingly. The two variants of DBR in current SIGPLAN use are “lightweight DBR” and “full DBR”. Under lightweight DBR, author names are revealed to reviewers after the initial review is submitted and before paper deliberations. In full DBR, author names are not revealed until the end of deliberations, and possibly only for accepted papers. These variants (and their many variations in implementation) reflect different models of the threat of reviewing bias and the benefit of knowing author identity. Lightweight DBR is based on the belief that authorship information should not bias “first impressions” of the paper’s quality, but the reviewers’ informed technical opinion will be unaffected, and even improved in depth, if author identities are later revealed. Full DBR rejects all authorship information as a bias factor. - -**Guardians** - -Guardians may be used at the discretion of the program chair. A guardian is a PC member assigned responsibility for ensuring that a paper is carefully and fairly assessed. Concretely, a guardian’s role may include: -Assisting the PCC in identifying gaps in reviewer expertise. -Ensuring that all reviews are of an appropriate standard. -Facilitating online discussions. -Writing the author-visible summary of the decision rationale. - - **Expert and External Reviews** -SIGPLAN conferences should strive to identify expert reviewers for every submission, insofar as possible. An "expert" is a reviewer who is very well versed and current in related work in the field. Authors gain confidence in the outcome of review decisions when expert reviewers are involved. That said, well-informed, but non-expert reviews also play a significant role in acceptance decisions: they represent the majority of the future readership of a paper, and involving them mitigates against topical balkanization. - -_Recommendation:_ - -Several successful formulas for obtaining expert reviews have been used in the past: +SIGPLAN conferences should strive to ensure that every submission is +reviewed by people with sufficient relevant expertise. Review teams may +combine complementary areas of expertise, and well-informed non-expert +reviews can also provide a valuable perspective representative of the +paper's broader readership. -- The PCC designates and announces an External Review Committee (ERC) in the CFP. -- The chair (sometimes in consultation with the PC) identifies a roster of external reviewers after submissions have closed. -- The paper’s "guardian" PC member (if applicable) identifies external reviewers, in consultation with the PCC. +External reviewers must be vetted, authorized, and recorded by the PCC. +They should be encouraged to participate in discussion of the papers +assigned to them, but must not have access to unrelated submissions or +discussions. -The PCC should exercise some judgment in assessing expertise ratings. Some PC members consider themselves experts on most topics, while other, more modest, PC members almost never declare themselves an expert. In the end, the PCC should use his/her judgment in deciding whether the expertise on a given paper is sufficient. +**Submission of Supplementary Material** -External reviewers should be encouraged to participate in online discussion of the papers they've been assigned; however, they should not see unrelated online PC discussions. - -**Recommendations for Submission of Supplementary Material** - -Authors can be allowed to submit supplementary material (examples, proofs, software, datasets, etc.) at the time of submission. The PC is allowed, but not required, to consult this material. Supplementary material should be made anonymous if possible and reasonable. For example, proofs and datasets should always be anonymized, while source code may be too difficult to reasonably anonymize. +Conferences may allow authors to submit supplementary material (examples, proofs, software, datasets, etc.) at the time of submission. +The PC may, but is not required to, consider this material. +Supplementary material should be anonymized for double-blind reviewing. **Evaluation Criteria and Acceptance Ratio** Conferences should strive to accept all high-quality submissions, and avoid numerical limits on acceptable papers. -S -_Recommendation:_ - The PC should focus its deliberations on: --- whether there is a genuine research contribution which may include a new insightful evaluation of previous work --- whether the approach is fundamentally sound --- whether the community will benefit from reading the paper --- whether the paper (and supporting material) contain sufficient information for others to reproduce and build on the results + - whether there is a genuine research contribution which may include a new insightful evaluation of previous work + - whether the approach is fundamentally sound + - whether the community will benefit from reading the paper + - whether the paper (and supporting material) contain sufficient information for others to reproduce and build on the results - The committee should lean toward accepting papers that are controversial, that is, ones that, after discussion, still have both a strong advocate and a strong detractor. - The committee should lean toward accepting papers that explicate their results clearly. - **PC Submissions** -PC submissions can be problematic to manage, but, on balance, the benefits of allowing PC submissions exceed the costs. Conferences should allow PC submissions, but disallow submissions by the GC and PCC. +PC submissions can be problematic to manage, but, on balance, the benefits of allowing PC submissions exceed the costs, so conferences should allow PC submissions. - - -PC members should not be allowed to review or discuss other PC papers; hence all reviews of PC submissions must be external. The PCC will designate the roster of external reviewers for PC papers, manage the review process, and make the final acceptance decision, in consultation with the reviewers. SIGPLAN requires that PC papers be held to a higher standard than other papers. - -Acceptance decisions for PC papers will be announced at the same time as other author decisions are announced; i.e., the PC will not be aware of any PC paper decisions until the PC meeting is complete. - -_Recommendation:_ +If Program Committee members are allowed to submit papers, either their papers should be held to a higher standard or other EC-approved measures should be taken, to avoid the appearance of impropriety. +Conferences that use full double-blind reviewing and have program committees with at least fifty members need not hold PC papers to a higher standard, as the EC has judged that these are sufficient conditions to avoid the appearance of impropriety. The policy on PC submission should be clearly explained to candidate PC members when their participation on the PC is solicited. -An External Program Committee (EPC) can be used to evaluate PC submissions. The EPC can be a subset of the ERC, or a separate, non-overlapping committee, or “EPC” can be used as a replacement term (or synonym) for the ERC. +Acceptance decisions for PC papers should be communicated at the same time as decisions for other submissions. -**Author Response Recommendations** +The Program Chair (PCC) may not submit papers to the conference; SIGPLAN’s Executive Committee recommends that the General Chair (GC) also not be allowed to submit papers. -Recent SIGPLAN conferences typically allow author responses to reviewers. Authors should be allowed to read all reviews that are available at the time of the author response period. The PCC should make every effort to complete all initial reviews before the author-response period. Additional reviews may be solicited after the author response period and it may not be possible for the authors to see these reviews before final decisions are made by the PC. Ideally, authors should be allowed at least four days to respond. +**Author Response** + +SIGPLAN conferences typically allow author responses to reviewers. Authors should be allowed to read all reviews that are available at the time of the author response period. The PCC should make every effort to complete all initial reviews before the author-response period. Additional reviews may be solicited after the author response period and it may not be possible for the authors to see these reviews before final decisions are made by the PC. Ideally, authors should be allowed at least four days to respond. Additionally: -- Authors strongly prefer seeing the scores at author response time. The PCC should consider making the scores available as well as the reviews. -- Authors should be strongly encouraged to be brief. + +- Authors strongly prefer seeing the overall merit score at author response time. The PCC should consider making the scores available as well as the reviews. +- The PCC may choose whether to make reviewer expertise scores visible to authors. The main purpose of these scores is to help the PCC assess whether the reviewers have sufficient expertise, and keeping them internal may help focus author responses on the substance of the reviews. +- Authors should be strongly encouraged to be brief. - PC chairs may choose to state that the PC is not obliged to read or respond to rebuttals beyond a certain designated length. - The PCC may choose to allow authors to provide feedback on review quality, e.g., via a simple rating scale, in addition to rebutting the content of the review. From bb305efae8e08e4e746bd6411e6a0b111d0202f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Anders=20M=C3=B8ller?= Date: Tue, 8 Sep 2026 16:48:28 +0200 Subject: [PATCH 2/2] Updated ProChair and Review after comments --- Resources/Guidelines/ProChair.md | 63 ++++++++++++++++++-------------- Resources/Policies/Review.md | 7 +++- 2 files changed, 41 insertions(+), 29 deletions(-) diff --git a/Resources/Guidelines/ProChair.md b/Resources/Guidelines/ProChair.md index 3520f390..e8f3c8ee 100644 --- a/Resources/Guidelines/ProChair.md +++ b/Resources/Guidelines/ProChair.md @@ -142,13 +142,15 @@ for details. If presentations are to be recorded or streamed, inform presenters of the applicable consent or release requirements. -Explain whether authors may submit auxiliary or supplementary material, +Explain whether authors may submit supplementary material for reviewers +to consult (see the [Review policy](/Resources/Policies/Review)), whether reviewers are expected or merely permitted to consider it, and -any applicable anonymity requirements. Auxiliary material may be -published; see "Auxiliary material" below for the associated policy. -If the conference has a separate artifact-evaluation process, the call -or conference web page should clearly distinguish that process from the -submission of supplementary material with the paper. +any applicable anonymity requirements. Such material may later be +published alongside the accepted paper; see "Published Supplementary +Material" below. If the conference has a separate +artifact-evaluation process, the call or conference web page should +clearly distinguish that process from the submission of supplementary +material with the paper. The conference web page should include information about the various benefits offered by the @@ -189,13 +191,20 @@ for substantive judgment and discussion. Keeping the review form reasonably stable across editions can help authors and reviewers calibrate and interpret its ratings. One possible model is Nierstrasz's Identify the Champion scoring process, -which has stood the test of time. A typical review form in the spirit of -this process has (at least) the following fields: +which has stood the test of time. The essence of the process is to ask +each reviewer whether they would champion the paper during discussion, +without prescribing a specific scale. One example review form in this +spirit has (at least) the following fields: - Paper score: - A: Good paper. I will champion it during the discussion. - B: OK paper, but I will not champion it. - C: Weak paper, though I will not fight strongly against it. - D: Serious problems. I will argue to reject this paper. + + Reviewers sometimes gravitate toward middle scores; the PCC should + remind the PC that the scale is meant to be used across its full + range, and that a paper's score should reflect the reviewer's honest + willingness (or reluctance) to champion it. - Reviewer expertise: - X: I am an expert in the subject area of this paper. - Y: I am knowledgeable in the area, though not an expert. @@ -224,12 +233,17 @@ additional reviews where appropriate. External reviewers should be selected and managed in accordance with SIGPLAN's [Conference Submission Review Policy](/Resources/Policies/Review/). -### Guardians +### Associate Chairs and Guardians + +Some large conferences divide the Program Committee into groups, each +led by an Associate Chair who chairs that group much as the PCC chairs +the PC as a whole. Associate Chairs are sometimes called "Area Chairs" +when the groups are organized by topic of expertise. -The PCC may assign a PC member as guardian for a submission. A guardian -is responsible for helping ensure that the paper is assessed carefully -and fairly. Depending on the conference's review process, the guardian's -role may include: +The PCC may also assign a PC member as guardian for a submission. A +guardian is responsible for helping ensure that the paper is assessed +carefully and fairly. Depending on the conference's review process, the +guardian's role may include: * identifying gaps in reviewer expertise; * checking that the reviews are sufficiently complete and constructive; @@ -328,7 +342,7 @@ progress, and helps resolve production problems. The PCC should provide authors with clear deadlines and instructions for preparing their final versions, including the applicable ACM template, page limits, citation style, accessibility requirements, -supplementary or auxiliary material, and treatment of third-party +supplementary material, and treatment of third-party material. Allow sufficient time for authors to address formatting, metadata, rights, and TAPS validation problems. @@ -355,9 +369,9 @@ The conference should give authors clear instructions concerning artifact submission, evaluation, archival availability, licensing, and any requirements for persistent identifiers or external repositories. -### Auxiliary Material +### Published Supplementary Material -ACM proceedings may include auxiliary material associated with a +ACM proceedings may include supplementary material associated with a published paper, such as: - technical reports containing additional details; @@ -367,22 +381,15 @@ published paper, such as: - other material relevant to understanding, reproducing, or building on the work. -SIGPLAN encourages authors to make appropriate auxiliary material +SIGPLAN encourages authors to make appropriate supplementary material available when it supports reproducibility and future research. The conference should provide authors with clear instructions for submitting such material through the applicable proceedings-production process. -ACM uses the term *auxiliary material* for files that are provided as an -additional resource but are not reviewed as part of the published work. -Authors retain copyright in such material and grant ACM permission to -make it available through the ACM Digital Library. Authors should ensure -that they have the necessary rights to distribute all included content. - -Auxiliary material submitted for publication should be distinguished -from supplementary material submitted during paper review. The call for -papers should explain whether reviewers are expected or merely permitted -to consider supplementary material. Supplementary material must comply -with the same anonymity requirements as the paper. +This published, post-acceptance supplementary material may be distinct +from material that authors may submit alongside a paper at submission time +for reviewers to consult, which has its own anonymity and consideration +rules; see the [Review policy](/Resources/Policies/Review) for that. Distinguished Papers -------------------- diff --git a/Resources/Policies/Review.md b/Resources/Policies/Review.md index dd1b0db7..2b58ce4e 100644 --- a/Resources/Policies/Review.md +++ b/Resources/Policies/Review.md @@ -90,6 +90,11 @@ combine complementary areas of expertise, and well-informed non-expert reviews can also provide a valuable perspective representative of the paper's broader readership. +The PCC should exercise judgment when interpreting reviewers' +self-reported expertise ratings, since PC members calibrate these +ratings differently: some consider themselves experts on most topics, +while others rarely do. + External reviewers must be vetted, authorized, and recorded by the PCC. They should be encouraged to participate in discussion of the papers assigned to them, but must not have access to unrelated submissions or @@ -108,7 +113,7 @@ Conferences should strive to accept all high-quality submissions, and avoid nume - The PC should focus its deliberations on: - whether there is a genuine research contribution which may include a new insightful evaluation of previous work - whether the approach is fundamentally sound - - whether the community will benefit from reading the paper + - whether the results are likely to be of interest to a sufficiently broad part of the community - whether the paper (and supporting material) contain sufficient information for others to reproduce and build on the results - The committee should lean toward accepting papers that are controversial, that is, ones that, after discussion, still have both a strong advocate and a strong detractor. - The committee should lean toward accepting papers that explicate their results clearly.