What is the purpose of review process in academic conferences? Reviewing the papers and then deciding whether to accept the paper for the conference. The order is important. Nevertheless, my experience has been that reviewing and judgment on a paper run in parallel and the review is often influenced by the decision, as the reviewer tries to justify their decision in the review.
It was a pleasant surprise then that when I was on the Shadow Program Committee of IEEE S&P 2021 the order was maintained: review first and decide later. The chairs–Reza Shokri and Yuan Tian–emphasized that the reviewers review the paper as objectively as possible rather than make a decision on the paper while they review. You can read more about the process here.
Things only got better from there on. How often have you received reviews where the reviewers contradict each other and you wondered why they did not discuss with each other? We had this issue covered. Once the reviews were submitted, the reviewers discussed the paper and each other’s reviews to clarify their understanding of the paper. Then, the discussion lead wrote a meta-review for the paper.
I found that these discussions worked well when all the reviewers were actively involved in the process and were eager to understand the paper and each other’s reviews better. Some reviewers updated their individual reviews to reflect their improved understanding. Unfortunately, for a couple of papers, some of the reviewers either did not contribute to the discussion at all or chipped in sporadically. These reviewers, in most cases, also wrote reviews that were short and nonconstructive. They would have benefited the most had they engaged more in the discussions.
And yes, we did decide on the papers afterwards. Apart from a few papers where the reviewers strongly disagreed with each other, the decision making was rather straight-forward as the prior discussions were helpful in bringing about a good level of common understanding of the paper.
Would this process work with program committees of other conferences? There are a couple of factors that need to be considered. Will all the reviewers submit their reviews on time? Will they engage actively in the discussions? Late reviews will make it harder to kick-start the discussion and passive reviewers will prevent the discussion phase from being useful.