About
ICSERGen
is a program that generates an automatic C (reject) review for
Computer Science conference papers, complete with missing related work,
clarification of terms, complaints about terminology, and, of
course,
the mandatory not-so-subtle disclosure of the reviewer's
identity. ICSERGen works for any paper
that is ACM material, but is particularly tuned for Software
research. The goal here is to
make fun of the situation, so all reviews use the same template.
ICSERGen
does content profiling of the papers, and uses that to query the ACM
Digital Library and the Encyclopedia Britannica. One possible purpose of this program is to remind you of related work that you might not think of citing.
Another purpose is to prepare you for the upcoming C review that the
paper you just submitted will most likely get.
I
strongly discourage conference reviewers from using this program for
generating their reviews at the very last minute. YOU WILL BE CAUGHT.
|
Generate
an Incredible C Review
|
The Incredible C Review Template
Creating Incredible
C reviews is really easy. Many conference reviewers do it manually,
which is a sligthly time-consuming thing to do. Therefore,
ICSERGen automates the process using the following popular template:
a) General related
work missing. Just mention something on the top of your head, possibly
well-known papers that you think are related to the paper in question.
ICSERGen uses the ACM Digital Library - you can do that too.
b) Find a quote from the paper and say that it's not clear what the authors mean.
c) Pick a few keywords that seem to be important in the paper, and complain that they are being misused.
d) Point the
authors to your own work. No Incredible C review will be
incredible without this! You can be more or less subtle. ICSERGen is
not subtle.
e) Pretend that you
tried to understand the technical content by saying that the paper has
no technical details, and that you don't understand how X relates to Y.
f) Include the following sentence verbatim: "The paper does not provide enough details for the work to be reproducible."
g) Just to confuse the authors even more, add something purely non-sensical that sounds like you know something that they don't.
h) Complain about some figure, if there are any.
i) Finish by stating your disappointment with the paper.
That's it.
|
Code
Here it is.
It works like this. First the paper is profiled by counting the
frequency of the words. Noise words such as "the", "more", etc. (about
100 of them) are ignored. It turns out that the top 5 or 6 words are a
very accurate signature of the paper's content. Those words are then used to
query several services on the web, such as the ACM Digital Library and
the Encyclopedia Britannica. (Sniffing through the results is pure
hacking fun!) Those words are also used to search for
good quotes in the paper and for generally conveying that this
"reviewer" understands what the paper is about.
|
Motivation - or The Unbearable Lightness of CS Conferences
The majority of reviewers that I interacted with
over
the years, both in program committees that I've been and in
conferences to which I submitted papers,
are reasonable people who try hard to be objective and to
justify
their ratings of the papers. A few, though, are not, and they produce terribly sloppy reviews. There are mainly
two situations behind these poor-quality reviews: one is when the committee
member outsources the review to an inexperienced person,
usually without double checking the result; the other is when the
committee member feels hurt
that his/her work wasn't cited, and, uncapable of overcoming that,
decides to kill the paper. In this last case, the identity of the
reviewer usually emerges through sentences such as "you didn't cite X
and Y," when those citations would be possible but not essential (or would be plain farfetched!).
Reviews like these are unbearably light, and they put into question the whole process of paper selection of a conference.
While bad reviewers can't be totally blamed for their sloppiness
-- they're possibly overcommitted by having to review 100
conference/workshop papers per year! -- it's time to start
acknowledging that, when reviews like these go through the process
unnoticed, there is a problem here, somewhere.
|
Credits
ICSERGen wouldn't
have happened without the spark of inspiration provided by the
anonymous reviewer at ICSE 2007 who provided us a true perl of a C review (or was it a D?). To thank him, here is the Incredible C review of our paper, generated by ICSERGen (compare it with his original review).
ICSERGen also took inspiration from SCIGen. To thank them, here is an Incredible C review generated by ICSERGen of a paper
that was generated by SCIGen. If we continue improving our tools, we'll
eventually be able to remove people from the loop altogether.
:-)
|
ICSERGen was written, and is maintained by me, Crista Lopes.
If you encounter any problems, please let me know. If you're intrigued
with the related work retrieval part, you may want to check this other tool.
|