CISC 879 : Software Support for Multicore Architectures
John CavazosDept of Computer & Information Sciences
University of Delaware
www.cis.udel.edu/~cavazos/cisc879
Lecture 8How to Read, Critique, Write, and Present a Research Paper
CISC 879 : Software Support for Multicore Architectures
Lecture 8: Overview
• Effective and Efficient Reading• Good and Bad Critical Reviewing • How to Write a Paper• How to Present a Paper
CISC 879 : Software Support for Multicore Architectures
What do I mean by “Read?”
• Effective and efficient reading• Do not waste your time on irrelevant papers• Maximize your time on relevant papers
• The Whys, Whats, and Hows of Reading• Critical Reading
• Essential to writing a “good” critical reviewing
CISC 879 : Software Support for Multicore Architectures
Problems with Some Papers
• Solving problems not applicable to an actual need• CS has not adopted Scientific Method• Least Publishable Unit (LPU)• Never clearly contrasting to related work• Hard to Read
CISC 879 : Software Support for Multicore Architectures
Questions to Ask
• What are the motivations?• What is solution and how is it evaluated?
• What is your analysis of problem, solution, and evaluation?
• What are the major results?
• Correct, new, clearly presented, worth publishing
• What are contributions?
• Are there any questions not answered?
Reference: http://www-cse.ucsd.edu/users/wgg/CSE210/howtoread.html
CISC 879 : Software Support for Multicore Architectures
Efficient Reading
• Preparation• Quiet place and note-taking material
• Read for Breadth• Read title and abstract• Skim intro, headers, tables, graphs, and conclusions• Look for primary contributions
• Read in Depth• Challenge arguments, assumptions, evaluations, and
conclusions
• Fallacy: Must read from beginning to end
Reference: www.cs.columbia.edu/~hgs/netbib/efficientReading.pdf
CISC 879 : Software Support for Multicore Architectures
Lecture 8: Overview
• Effective and Efficient Reading• Good and Bad Critical Reviewing • How to Write a Paper• How to Present a Paper
CISC 879 : Software Support for Multicore Architectures
Anatomy of Good Review
• Summary of main points of paper (3 to 5 sentences)• Evaluation with regard to validity and significance• Evaluation quality of work• Overall recommendation and thorough justification
Reference: http://www.cs.utexas.edu/users/mckinley/notes/reviewing-smith.pdf
Evaluation of novelty, significance, correctness, readability.
CISC 879 : Software Support for Multicore Architectures
Bad Critical Reviewing
• Seek to find all flaws• Criticize paper in a negative way
• Use words like “Idiot” and “Trash”
• Criticize paper with little comment• After all, they know it’s bad
• Criticize/accept all papers• Be known as person that accepts/rejects everything!
Reference: http://www.cs.utexas.edu/users/mckinley/notes/reviewing.html
CISC 879 : Software Support for Multicore Architectures
Good Critical Reviewing
• Can Always be Positive• Think face-to-face constructive criticism
• Consider Recently-Published New Directions• Explain your Decision
• No explanation will carry little weight
• Decide which paper is best• Not which paper least worthy of rejection
Reference: http://www.cs.utexas.edu/users/mckinley/notes/reviewing.html
CISC 879 : Software Support for Multicore Architectures
Bad Review Example
This paper is concerned with blah blah blah. The authors develop a method to automatically generate blah blah blah. The solution is based on genetic algorithms and evolution-based methodology.
This is a solid paper with respect to methodology and the experiments performed. My score is relatively low because I am not particularly excited by genetic, evolutionary algorithms.
Not a Reasonable Justification!
CISC 879 : Software Support for Multicore Architectures
Good Review Example
I like this work a lot.
There are two things that would substantially improve it:
1) Investigation of the reasons why blah blah blah. Is there room for more improvement here, perhaps combining the two in some way? If not, what does this say about what technique we should apply, if forced to choose?
2) Quantitative evaluation of the predictive power of blah blah blah. You seem to have a lot of the necessary data (ie, you can compare the results of blah blah blah from Fig 1).
Starts positive, then suggests how to “substantially improve paper.”
Reviewer justifies remark!
CISC 879 : Software Support for Multicore Architectures
Lecture 8: Overview
• Effective and Efficient Reading• Good and Bad Critical Reviewing • How to Write a Paper• How to Present a Paper
CISC 879 : Software Support for Multicore Architectures
Getting a Paper Accepted
• Follow the Rules!
• paper format, length, on time, etc.
• Spell and Grammar Check
• Write, Read, Edit, Repeat (many times!)
• Have colleagues review
• Look at good (award) papers as examples
• After-the-fact outline (check flow of paper)
CISC 879 : Software Support for Multicore Architectures
Getting a Paper Accepted
• Write to the reviewer
• Quality must be recognized quickly
• What are contributions?
• Is paper stimulating?
• Is paper relevant?
• Have a planned organization of paper
• Enough details to reproduce experiments
CISC 879 : Software Support for Multicore Architectures
Anatomy of a Good Paper
• Abstract
• State the problem and why its important
• Briefly describe solution
• Implication of solution
• Introduction
• Describe problem and motivate importance
• Describe solution and significant contributions
• Describe your solution
• Convince reader this could solve problem
• Implementation details
CISC 879 : Software Support for Multicore Architectures
Anatomy of a Good Paper
• Defence of your solution
• Experimental methodology and results
• Convince reader you solved problem
• State reasonable counterexamples
• Related work
• Convince reader novelty of your solution
• Conclusion
• Summary of results and significance of contributions
CISC 879 : Software Support for Multicore Architectures
Icing on the Cake
• Attention Grabber Sentence
• Great Introduction!
• Easy to read
• More time you spend editing, less time reader has to spend
• Good looking graphs
• Can understand (paper) from reading graph captions
CISC 879 : Software Support for Multicore Architectures
Lecture 8: Overview
• Effective and Efficient Reading• Good and Bad Critical Reviewing • How to Write a Paper• How to Present a Paper
CISC 879 : Software Support for Multicore Architectures
What is Your Talk is about
• The Paper = The Beef
• The Talk = Beef Advertisement
• Do not confuse the two!
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
CISC 879 : Software Support for Multicore Architectures
Your Talk is NOT About:
• To impress your audience with your brainpower• To tell them all you know on the topic• To present all the technical details
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
CISC 879 : Software Support for Multicore Architectures
Your audience
• Have never heard of the topic• Just had lunch and ready to nap
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
Your mission is
WAKE THEM UP
and make them glad you did.
CISC 879 : Software Support for Multicore Architectures
What’s in your Talk
• Motivation (20%)• 2 minutes to engage audience• Why should I listen?• What is the problem?• Why is it an interesting solution?
• Key Idea (80%)• Be specific• Organize talk around specific goal• Ruthlessly prune everything else!
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
CISC 879 : Software Support for Multicore Architectures
Tips for a Good Talk
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
• Narrow, deep rather than wide, shallow• Avoid shallow overview talks• Cut to the chase: technical “meat”
• Examples are your main weapon!• To motivate talk• To convey intuition• To illustrate key idea
CISC 879 : Software Support for Multicore Architectures
What to leave out
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
• Related Work• Know it, but don’t elaborate on it
• Technical Detail• Extensive formulas and code• Dense slides will put audience to sleep!• Present specific aspects; refer to paper for details
• Do not apologize• “I didn’t have time to prepare the talk”• “I don’t feel properly prepared to give this talk”
CISC 879 : Software Support for Multicore Architectures
Final Tips
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
• Be Enthusiastic• Practice• Memorize first few sentences• Move around, use arms• Speak to someone at back of room• Identify nodders and speak to them• Watch for questions
CISC 879 : Software Support for Multicore Architectures
What NOT to Do
Reference: http://research.microsoft.com/~simonpj/papers/giving-a-talk/giving-a-talk.pdf.gz
• Use slides with lots of text• Use small fonts• Reveal one point at a time• Use lots of animation and color• Go over your time limit• Stand in front of projected material• Stand with back to audience• Point to laptop
CISC 879 : Software Support for Multicore Architectures
Project 1 Specific
• Due March 11th (next Tuesday!)• 25% of grade is checkpoint
• 10 minute discussion of status• Easy points!
• Link to Template for Project Report on Website• ACM Conference format style
CISC 879 : Software Support for Multicore Architectures
Experiment Example 1
Performance Normalized to -O0
0
0.5
1
1.5
2
2.5
3
3.5
fallbackSort fallbackSimpleSort mainGtU fallbackQSort3 generateMTFValues BZ2_decompress
Top 5 Hot Functions
O1
O2
O3
O4
O5
Top 5 important functions reported by gprof. This graph shows the performance results of each function
compiled using different optimization levels of XLC normalized to using no optimizations (-O0).
CISC 879 : Software Support for Multicore Architectures
Experiment Example 2
Performance results for top 4 important functions and total benchmark. This graph shows the results of each function
and the whole benchmark compiled using different optimization levels of GCC and XLC normalized to using
GCC with no optimizations (-O0).
Performance of GCC versus XLC
0
0.5
1
1.5
2
2.5
3
3.5
fallbackSort fallbackSimpleSort mainGtU fallbackQSort3 BENCHMARK
Hot Methods and Total
Normalized to GCC(-O0)
GCC (O1)
GCC (O2)
GCC(O3)
XLC(00)
XLC(01)
XLC(O2)
XLC(O3)
XLC(O4)
XLC(O5)