1、1How to write a reviewToby WalshDepartment of Computer ScienceUniversity of Yorkwww.cs.york.ac.uk/tw/phd2Outlinew What is a review?w Why should you review?w How do you review a paper?w What not to do?w What are the dilemmas?w Case study3What is a review?w “Something that will ruin your day” Alan Bun
2、dynEven if it is good!w The stamp of scientific qualityw Feedback from your peersnFuture directions?Prof. Alan Bundy4What is a review not?w Acceptance/rejectionnEditors/Program committees accept/rejectnYou recommend!w A place for bias, prejudice, personal animosity, nThough it often appears to be so
3、5Why should you review?w Youd much rather enjoy PaphosnAnd so would I!w But science would grind to a haltnNot immediately, of course6Reasons to revieww Dutyw Fairnessn2-3 reviews written/ paper writtenw Promotionw EducationnGood reviewers write good papers?7Bad reasons to revieww To settle old score
4、sw To advance your own theories/hinder rivalsw To get latest resultsnUnpublished papers are strictly confidential8How do you review?w Read the paperw Read the review formnUseful dimensions to look atnNovelty, Clarity, Importance, Timeliness w Read the paperw Wait a few daysw Read paperw Write review
5、Everyone has their own method9How do you review?w Put yourself in authors shoesnThink how you would like to read this revieww Offer constructive criticismnDont just tell them something is inadequate!nTell them how they might fix it10What not to do?w Miss the deadlinenWe all hate late reviewsw Displa
6、y partiality, bias, animosity, w Destructively criticizenAlways work out what they would need to do to fix problems 11Collect and share reviewsw Learn if others agree with your opinionsw Thicken your skinClearly, the author fails to understand the work of Walsh in this area Since they mention no rel
7、ated work, this paper cannot be original This idea is too simple not to exist already This work is good but I dont understand why Bundy hasnt done this already?12Ethical dilemmasw You are working on the same problemnTalk to Editor/Program Chairw You already reviewed and rejected papernLook for chang
8、es13Ethical dilemmasw This journal submission already appeared at a conferencenDoes it extend previous appearance?w An almost identical paper already appearednUnless it was at a workshop, inform Editor/Chair14Case studyw “Stochastic Constraint Programming”nBy Toby WalshnBe frank, the feedback is goo
9、d!w What do you think?15What did reviewer 1 think?w Appears to like itw Main criticisms:nRelationship to influence diagramsnAlgorithm performancenPhase transition too preliminary“The paper reads well. I have a number ofremarks though. First, from a probabilistic reasoning viewpoint, I wonder about t
10、he relationship between this framework and influence diagrams (or decisiondiagrams). It appears to me that what you have defined here is very closely related Second, from a constraint satisfaction viewpoint: you gave us noindication of how well the different algorithms you presented work in practice
11、. Third, I think the discussion on phase transition cannot be left at this level. It is not surprising that we have a phase transition here, but what is interesting is the nature of this transition I think this topic is too serious to sum it upin a small Section it deserves a dedicated and more thor
12、ough treatment. I would have preferred to see this space dedicated to experimental results on the performance of presented algorithms”16What did reviewer 2 think?w Appears to like itnVery relevantnModerately significant/originalnGood readability/Englishw Minimal commentsTotal of their written commen
13、ts:“It would be nice to include the exact syntax of one SCprogram,as accepted by your system (?), say, for the example of Section 3.”17What did reviewer 3 think?w Again appears to like itnVery relevant, very original, moderately significantw Main criticismsnRelationship to influence diagramsnPhase t
14、ransition too preliminary“Clearly, one could just add constraints to influence diagram representation and extend algorithms to exploitthem (my preferred approach) but the approach here isstill very valid and could motivate researchers in MDPsand influence diagrams to treat constraints as special cre
15、atures so that their special algorithmscan be exploited.I think the experimental portion of the paper should have been to compare the performance of the algorithms with the performance of traditional MDPs or influence diagram algorithms applied to this class of problems and I speculate that gain can
16、 be shown.I dont find the phase transition experiments of much value at this stage. So, there may be a phase transition,so what?I recommend that the author will carefully analyze theirmodel against standard influnce diagrams or factored MDPs and discuss the pros and cons.”18What did the IJCAI PC thi
17、nk?w Paper was rejectednAlong with 75% of the other submissionsnA less good paper (my and reviewers opinions) was accepted!w Some compensationn$150,000 to be preciseIJCAI 2001 logo19Conclusionsw Reviewing can be rewardingnBoth to authors and to reviewersw Be constructivenThink how you would react to the revieww Take on board your reviewsnReviewers hate most being ignored!
