COMMENTS ON PEER REVIEW AND RATING OF NRAO OBSERVING PROPOSALS Frederic R. Schwab, Dana S. Balser, and Gareth C. Hunt March 10, 2015 1. Introduction Recently we were asked to comment on the NRAO proposal peer review process, and to focus primarily on the scoring procedure reviewers are instructed to follow and the algorithm used for aggregation of the reviewers’ scores. In particular, we were asked to comment on the robustness of the score aggregation process—for example, sensitivity to imbalance between reviewers’ score distributions—and to compare our proposal scoring system with those used by other institutions and other observatories (e.g., ESO, Arecibo, HST, etc.) in order to provide assurance that our practice is not out of the mainstream. We begin by summarizing, in Section 2, the mechanics of the current review process for standard GBT, VLA, and VLBA proposals. There we describe the instructions provided to reviewers for scoring, the score normalization and averaging method that is used, and the peculiarities (particular the imbalance) of the distributions of reviewers’ observed raw score distributions for recent proposal review cycles. We continue with a discussion of possible deficiencies in our proposal scoring system and suggest a few simple remedies. In Section 3 we compare the NRAO process with what is in use at other observatories and at two federal agencies, NSF, and NIH. In Section 4 we discuss ranking, rating, and score aggregation methods that are based on pairwise score comparisons. In Section 5, we discuss two other methods that have appeared in recent literature, and in Section 6 we call attention to a recent paper advocating a distributed approach to telescope proposal peer review. Conclusions are given in Section 7, along with a few more comments and suggestions. 2. Description of the Review Process The NRAO Web pages give a useful overview of the review process, one intended both for reviewers and proposers. 1 Also, a comprehensive description of the gritty, technical details of the process is given in a memorandum by Bryan Butler [1], titled “Requirements for the PST for the New NRAO Proposal Evaluation and Time Allocation Process”, dated October 13, 2010. We include that memorandum here as Appendix A. Most of the details there are still current. This process pertains to proposals for use of North American NRAO facilities—the GBT, VLA, and VLBA. 1 See https://science.nrao.edu/observing/proposal-types . 1 TTA Report 1
Transcript
COMMENTS ON PEER REVIEW AND
RATING OF NRAO OBSERVING PROPOSALS
Frederic R. Schwab, Dana S. Balser, and Gareth C. Hunt
March 10, 2015
1. Introduction
Recently we were asked to comment on the NRAO proposal peer review process, and tofocus primarily on the scoring procedure reviewers are instructed to follow and the algorithmused for aggregation of the reviewers’ scores. In particular, we were asked to comment on therobustness of the score aggregation process—for example, sensitivity to imbalance betweenreviewers’ score distributions—and to compare our proposal scoring system with those usedby other institutions and other observatories (e.g., ESO, Arecibo, HST, etc.) in order toprovide assurance that our practice is not out of the mainstream.
We begin by summarizing, in Section 2, the mechanics of the current review process forstandard GBT, VLA, and VLBA proposals. There we describe the instructions provided toreviewers for scoring, the score normalization and averaging method that is used, and thepeculiarities (particular the imbalance) of the distributions of reviewers’ observed raw scoredistributions for recent proposal review cycles. We continue with a discussion of possibledeficiencies in our proposal scoring system and suggest a few simple remedies. In Section 3we compare the NRAO process with what is in use at other observatories and at two federalagencies, NSF, and NIH.
In Section 4 we discuss ranking, rating, and score aggregation methods that are based onpairwise score comparisons. In Section 5, we discuss two other methods that have appearedin recent literature, and in Section 6 we call attention to a recent paper advocating adistributed approach to telescope proposal peer review.
Conclusions are given in Section 7, along with a few more comments and suggestions.
2. Description of the Review Process
The NRAO Web pages give a useful overview of the review process, one intended bothfor reviewers and proposers.1
Also, a comprehensive description of the gritty, technical details of the process is givenin a memorandum by Bryan Butler [1], titled “Requirements for the PST for the NewNRAO Proposal Evaluation and Time Allocation Process”, dated October 13, 2010. Weinclude that memorandum here as Appendix A. Most of the details there are still current.This process pertains to proposals for use of North American NRAO facilities—the GBT,VLA, and VLBA.
2.1. Overall Summary. Each observing proposal pertains to one of five instrumentalcategories:
• GBT — Green Bank Telescope;• VLA — Very Large Array;• VLBA — Very Long Baseline Array;• HSA — High-Sensitivity Array (utilizing the VLBA plus one or more among the
following: GBT, Effelsberg 100-m, Arecibo, the phased VLA); or• GMVA — Global 3 mm VLBI Array (utilizing the VLBA plus the GBT, Effelsberg,
Pico Veleta, Plateau de Bure, Onsala, Yebes, and Metsaehovi radio telescopes).
And each proposal is assigned to one of the eight science categories which are denoted bythe acronyms defined below:
• AGN — Active Galactic Nuclei;• EGS — Extragalactic Structure;• ETP — Energetic Transients and Pulsars;• HIZ — High Redshift and Source Surveys;• ISM — Interstellar Medium;• NGA — Normal Galaxies;• SFM — Star Formation; and• SSP — Sun, Stars, Planets, and Planetary Systems.
Associated with each science category is a six-member Science Review Panel (SRP). Eachpanel member is an expert in the subject discipline. One member is designated as the panelchairperson. Each panel member—except the chair—submits a score for each proposal,unless that member declares a conflict of interest. If one or more of those members areineligible, then the panel chair does submit a score—except in cases where the chair alsohas a conflict of interest. (The procedure for identifying conflicts of interest is specified in§3.12 of Appendix A.)
The HSA and GMVA proposals are—like the GBT, VLA, and standalone VLBAproposals—reviewed by the SRPs. The GMVA proposals are, however, also reviewed by aEuropean review panel and their time allocation is determined by a different process thanthat of the standard NRAO TAC. The time allocation process used for HSA proposals alsois different from the standard one.
There are two calls for proposals per year, for two six-month semesters, denoted Aand B. The nominal start dates for observing within a given semester is February 1 forSemester A and August 1 for Semester B. The proposal deadline for Semester A is thefirst of August of the preceding calendar year, and the deadline for Semester B is the firstof February (the precise date depends on whether the first day of the month falls on aweekend). Proposal cycle semesters are designated by three-character alphanumeric stringsof the form 12B, 13A, . . . to denote “2012 Semester B”, “2013 Semester A”, etc.
Review criteria which panel members are asked to consider are described in detailon the NRAO Science Review Panel Web page.2 Specific review criteria include scientificmerit, justification for any extra resources that are requested (as for GMVA and HSA pro-posals), qualifications of the project team, their publication record from any past proposalsubmissions, the possibility of acquiring more appropriate data than requested (say, from anexisting data archive), amount of resources requested vis-a-vis telescope time pressure, andstudent status, when relevant (e.g., whether a sidelight or a main focus of thesis research).
After all panel members have submitted their independently derived proposal scores,the entire SRP meets by teleconference for discussion, debate, and reconsideration of the
scoring. During this stage of the process individual reviewers do not modify their scores,but the aggregate scores (and hence the aggregate rank order of preference from the panel)are subject to modification. The panel chair always has final say. Further details can befound in [1]. The final aggregate scores and rank order of proposals then are submitted tothe Time Allocation Committee (TAC). The TAC then merges the rank orderings from alleight panels.3
2.2. Scoring, and Score Aggregation. As noted above, proposal scoring within theScience Review Panel is done independently by panel members—without consultation—prior to a group meeting of the SRP. The permissible range of scores, from best to worst,is 0.1 to 9.9, in steps of 0.1.4
After scores have been submitted, reviewers’ score distributions each are shifted tohave a common mean (equal to five) and linearly scaled to have a standard deviation oftwo. The scores, so normalized (or “standardized”), then are arithmetically averaged toobtain an aggregate score for each proposal.
Note that some of the standardized scores may fall outside of the range [0.1, 9.9] (e.g.,if the raw distribution has a smaller standard deviation than 2.0 and a significant positiveor negative skew); this happens rather infrequently.5 It may also happen that the aggregatescore falls outside of the range [0.1, 9.9]. (However this has not occurred in the case of anyof the averaged scores (1546 total) for proposal cycles 12B, 13A, 13B, and 14A.)
2.3. Observed Distributions of Reviewers’ Raw Scores. It is interesting and instruc-tive to note the diversity among reviewers’ raw score distributions. As one example, Table 1shows the raw scores submitted by the Cycle 13B, AGN panel reviewers, versus proposalID number.6 Reviewer 170’s scores all are integer values. Reviewer 136’s are integer andhalf-integer values. The other reviewers discriminate at the finest permissible granularityof 0.1. Reviewer 170 has ties in every row of the table—two-way ties at three scores, andmulti-way ties in each other row. Reviewer 149 is the most discriminating, with only five(two-way) ties. Reviewer 151 is very lenient, compared to the others—with 31 out of 51scores ≤ 1.0.
As to the degree of consensus among panel members in this example, let us considerproposal 8166: this proposal is rated last by two (out of five) reviewers (136 and 170), andsecond- or third-to-last by the three others (149, 150, and 151). Here we see a high degreeof consensus. There is much less of a consensus regarding proposal 8225: Reviewer 125rates it first, Reviewer 151 rates it in a two-way tie for last place, two others place it intheir third quartiles, and one in their second quartile.
Histograms of reviewers’ scores for this example are shown in Figure 1, along withsome descriptive statistics. Among the six reviewers, the mean score varies between 1.57and 4.57, and the median between 1.0 and 4.5. Reviewer 151, with a mean assigned score of1.57 (and median of 1.0), is very lenient (as noted before). The scales of these score distri-butions, as measured by their standard deviations, vary between 1.60 and 2.75. (The scaledmedian absolute deviation about the median value (MAD)—perhaps a better estimate of
3More details are given at https://safe.nrao.edu/wiki/bin/view/Software/PSTReviewCookbook .4It seems peculiar that the range is not [0,10] rather than [0.1,9.9]. I believe originally the scale 1 to 9.
was in use. Then someone questioned, “Why not 0 to 10?” Someone replied that the code would alreadywork for 0 to 10—except that 0 was not possible because it was used as a flag to indicate a conflict of
interest; so 0.1 and 9.9 (for the sake of symmetry) were adopted as limits. A better choice would have beento indicate a conflict of interest via a an alphabetic character, a NaN flag, or somesuch. — F.R.S.
5In the four most recent proposal cycles (12B, 13A, 13B, and 14A) this happens for 47 out of 1546raw scores (3.04% of the total).
6These proposal ID numbers have been randomly assigned, for reasons of confidentiality.
3
0 2 4 6 8 100.00
0.05
0.10
0.15
Reviewer 125
0 2 4 6 8 100.000.050.100.150.200.250.300.35
Reviewer 136
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
Reviewer 149
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
Reviewer 150
0 2 4 6 8 100.00.10.20.30.40.50.6
Reviewer 151
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
0.5
Reviewer 170
AGN Panel Raw Score Distributions, Cycle 13B H50 proposalsL
nr Mean Std Dev Skewness Kurtosis Median H-L Loc Scaled MAD Sn Qn
Figure 1. Histograms of Cycle 13B AGN proposal raw scores, by reviewer. The ordinate in each caserepresents the probability density.
the “typical” scale of a distribution such as Reviewer 151’s—varies between 0.82 and 2.74.)Three of the distributions (those of Reviewers 136, 150, and 151) show significant positiveskew.
Figures showing the raw score distributions from each of the eight review panels andthe four most recent proposal cycles (12B, 13A, 13B, and 14A) are given in Appendix B(Figures B-1 through B-8). Like Figure 1, these show for each panel/cycle pair (in columns 1through 6): the number of scores submitted by the given reviewer, the mean raw score,standard deviation, skewness, kurtosis, and median score. Columns 7 through 10 give a fewancillary statistics; these are defined in §2.4.
The reviewers’ raw score distributions shown in Appendix B are typically clusteredtoward the low end of the score range (i.e., higher ratings), and they often are positivelyskewed, with a few poor scores in the right-tail region. The average mean is 4.06 ± 0.85,the mean standard deviation is 1.85± 0.50, the mean skewness is 0.39± 0.48, and the meankurtosis is 2.63 ± 0.98. (For reference, the kurtosis of Gaussian, or normal distributionis equal to three; and for a normal distribution truncated at ±2.5σ about the mean thekurtosis is 2.6242.) Histograms of these statistics are shown in Figure 2. Table 2 liststhe number of proposals, by panel, for the four most recent proposal cycles. The typicalproposal is rated by five reviewers; out of the 1529 non-GMVA proposals, 17 were rated byonly two reviewers, 91 by three reviewers, 446 by four reviewers, 971 by five reviewers, andfour by six reviewers (the corresponding percentages are 1.1, 6.0, 29.2, 63.5, and 0.3).
Sometimes a reviewer’s score distribution is representative of a total ordering, or com-plete ranking of proposals, with no ties present. And sometimes reviewers appear to gradeon a curve. On the AGN Panel, Cycle 14A (see Figure B-1), Reviewer 149 has a scoredistribution which is exactly symmetric and contains no ties—in ascending order the 51
Figure 2. This Figure shows histograms of the mean, standard deviation, skewness, and kurtosis statisticsof all the reviewers’ raw score distributions combined for proposal semesters 12B, 13A, 13B, and 14A. Theaverage mean is 4.06 ± 0.85, the mean standard deviation is 1.85 ± 0.50, the mean skewness is 0.39 ± 0.48,and the mean kurtosis is 2.63± 0.98. Note that the distribution of the means is skewed to the left and thatof the skewnesses is skewed to the right.
scores are: a few scores separated in steps of 0.5, then a few separated by 0.3, then by0.2, then a bunch separated by 0.1, then the reverse of all that. Another example is ETPPanel Cycle 14A, Reviewer 114 (see Figure B-3). This score distribution, consisting of 21scores, is an exact fit (by Cramer–von Mises and Anderson–Darling tests) to a triangulardistribution over the range [0.5, 9.5]
2.4. Methodological Tools. In this section a few special methodological tools, suchas ancillary statistical measures, kernel density estimates, and “smooth” histograms aredescribed.
2.4.1. Hodges–Lehmann estimator. In the case of a skewed or fat-tailed distribution, thesample median might be viewed as better representative of the “most typical” value of anunderlying (or parent) distribution than the sample mean. A similar estimator, also basedon order statistics, is the so-called Hodges–Lehmann estimator of location [2]. This has asimple definition: it is the median of all the pairwise averages of x1, . . . , xn, i.e.,
med1≤i,j≤n
xi + xj
2. (1)
For independent samples from a continuous symmetric distribution, the H–L estimator is,in fact, an estimator of the population median. For fat-tailed distributions especially, ithas the advantage that it generally converges faster to the population median than doesthe sample median (i.e., it is often a more efficient estimator of the population medianthan the sample median itself). Also, the H–L estimator has a smooth influence function—i.e., if a sample value is varied continuously, this estimate of distribution centrality varies
Table 2. Number of proposals, by panel, for recent review cycles 12B, 13A, 13B, and 14A. A total of1546 proposals underwent review: 357 for GBT, 960 for VLA, 212 for VLBA (including HSA), and 17 forGMVA time. The GMVA proposals are also reviewed by a European review panel; the final ranking for thiscategory of proposals, and the telescope time allocation, are done outside of the normal SRP and TAC.
continuously, unlike the sample median. Thus it can be viewed as a smooth version of themedian (see Rousseeuw and Croux (1993) [3]). Figure 3 (left) shows the distribution ofthe ratio of the mean score to H–L location estimate, including all reviewers’ raw scoredistributions for the four most recent proposal cycles.
2.4.2. Alternative measures of scale (or dispersion). Also for many non-Gaussian (e.g.,fat-tailed, skewed, or outlier-contaminated distributions) the standard deviation is an over-estimate of the “typical” dispersion of the distribution. The median absolute deviationabout the median value (the so-called MAD), scaled by a constant c, is commonly usedas an alternative measure of scale.7 However, the scaled MAD is only 37% efficient in thecase of the Gaussian distribution. Rousseeuw and Croux (1993) [3] define two alternativeauxiliary estimates of scale, which they term Sn and Qn, that achieve higher efficiency (58%and 81%, respectively) at the Gaussian distribution. Both of these are defined in terms of
7The value c = 1.4826 is chosen to make the scaled MAD asymptotically agree with the standarddeviation in the case of a Gaussian distribution.
7
0.90 0.95 1.00 1.05 1.10 1.15 1.20 1.250
5
10
15
20
Mean�H-L Location
0.6 0.8 1.0 1.2 1.4 1.6 1.80.0
0.5
1.0
1.5
2.0
2.5
3.0
Std. Dev.�Qn
Figure 3. At left is shown a histogram of the ratio of mean score to Hodges–Lehmann location estimate,including all reviewers’ raw score distributions for the four most recent proposal semesters. At right isshown the histogram of ratios of standard deviation to the Qn estimator of scale. The mean values of thesedistributions are 1.024 ± 0.043 and 1.035 ± 0.202, respectively.
order statistics on the set of pairwise absolute differences, {|xi − xj | ; i < j}.8,9 Like theMAD, the influence function for Sn has discontinuities—whereas the influence function forQn is smooth. Figure 3 (right) shows the distribution of the ratio of the standard deviationto Qn estimate of scale, including all reviewers’ raw score distributions for the four mostrecent proposal cycles.
2.4.3. Kernel density estimates and “smooth histograms”. We show in Figure 1 and Appen-dices B and C, in addition to the traditional histogram, a smooth curve representing someputative, underlying probability density function (PDF), for each of the given raw scoredistributions. These smooth curves are based on the so-called kernel density estimator, asdescribed by Silverman (1986) [4]. The method is easy to describe: Given data samplesx1, x2, . . . , xn, place a unit mass at the abscissa of each data sample, convolve the resultingsum of δ-distributions with a smooth, non-negative kernel (e.g., a Gaussian) of appropri-ately chosen width, and normalize so that the resulting smooth function integrates to unity.Expressed mathematically, the kernel density estimate is given by
f(x) =1
nh
n∑
i=1
K
(x − xi
h
), (2)
where K is the kernel function and h is the bandwidth.The plot of a kernel density estimate is sometimes referred to as a “smooth histogram”.
This is the terminology used in Mathematica. For the curves shown in this memorandum weuse the Mathematica implementation of kernel density estimates, with a Gaussian kernel.The kernel bandwidth is chosen adaptively. For bandwidth selection we choose, variously, amethod due to Scott—or another by Silverman—both are implemented within Mathematica.And for the traditional histogram, we select a bin width commensurate with the bandwidthof the adaptive kernel density estimate.
The smooth histogram resulting from a kernel density estimator has a major advantageover its traditional counterpart, in that the traditional histogram can be overly sensitive to
8Sn is defined via Sn = c lomedi{himedj |xi − xj |}, where lomed is the ⌊(n + 1)/2⌋th order statistic,himed is the (⌊n/2⌋ + 1)st order statistic, ⌊·⌋ denotes the greatest integer (or floor) function, and c =1.1926. Additionally, a bias correction is applied which, in the case of Gaussian-distributed data, makesthis estimator unbiased for small sample sizes.
9Qn is defined via the kth order statistic of pairwise differences: Qn = d {|xi − xj | ; i < j}(k), where
k =`
h
2
´
, h = ⌊n/2⌋ + 1, and d = 1.0483. A bias correction is applied, as in the case of Sn.
8
choice of origin—or choice of bin locations, in general.10 The smooth histogram is sensitiveto neither. The most powerful modern methods for tests of multi-modality are based onkernel density estimates [4]. One disadvantage of smooth histograms is that long-taileddistributional detail may not show up as well as with a traditional histogram (though adata-adaptive variable bandwidth choice, dependent on local density, may overcome thisdisadvantage).
2.4.4. Quantitative comparison of score distributions. Since the end product of panel reviewis a rank-ordered list of proposals based on final scores, quantitative rank-order comparisonsof score distributions (e.g., one reviewer vs. another, or one scoring method vs. another)are very apt. For lists of equal length, the Kendall τ and Spearman ρ measures of rankcorrelation are commonly used. Definitions for these vary slightly; we use the tie-correctedversions that are implemented in the standard Mathematica library. The ρ and τ valuescan vary over the range [−1, 1] (+1 if the rank orderings are identical, −1 if they are thereverse of each other, 0 for no association).
Langville and Meyer [5] introduce a modified Kendall τ which can be used for compar-ison of partial lists (e.g., top-k lists). The so-called Spearman weighted footrule is definedanalogously to Spearman’s ρ, but assigns higher weight to discrepancies at higher rankorders.
2.5. Comments and Recommendations. In this section we offer a few suggestions forminor modifications of the proposal scoring system.
1. Adjustment of scale of reviewers’ score distributions. In Appendix B we saw that thereviewers’ raw score distributions often are skewed. For such distributions, the standarddeviation tends to be an over-estimate of the typical spread between scores. Perhaps use ofthe scaled MAD, Sn, or Qn should be used for normalization of the score distributions, inlieu of the standard deviation. We would suggest the use of Qn, which is based on pairwisedifferences of scores, has high efficiency and a smooth influence function, and—unlike thestandard deviation or the MAD—is a scale estimate which does not depend on a priorestimate of location (e.g., mean or median).
If this change were made, however, there would be more occurrences of scores outsidethe range [0, 10] because the influence of scores in the tail of a narrow, skewed, raw scoredistribution would increase. One might prefer to truncate the distribution of normalizedscores to the [0, 10] range.
2. Shift of the reviewers’ score distributions. Likewise, we might consider use of the medianor the H–L location estimate of each reviewer’s score distribution—rather than the samplemean—for alignment of reviewers’ score distributions in the score normalization process.But we believe the effect would be relatively minor compared with that of Suggestion #1,given the nature of the ratio distributions shown in Figure 3.
3. Use of data from all cycles for score normalization. Each reviewer ordinarily serves fortwo years (four consecutive semesters). In Appendix B we see—for a given reviewer—afair degree of consistency between score distributions from consecutive semesters. Hencewe might consider normalizing over previous semesters, as well as the immediate one, forcalibrating reviewers’ scores. However, if reviewers were at some point to be given revisedscoring guidelines, this idea would certainly be ruled out (except for following semesters).
4. Normalization of panel chair’s score distribution. One should note that, because thepanel chair votes only when another member of the panel is conflicted, there often will be
10This can be especially the case if the data are quantized. (As they are, in our case.)
9
a paucity of scores from the panel chair, and therefore a relatively poorer calibration thanfor other reviewers. Incorporation of Suggestion #3 could help somewhat.
5. Should panel chairs always vote (except when conflicted)? If the panel chairs were tovote on all their panel’s proposals, then—obviously—there would be better calibration fortheir scores, and, typically, five or six reviewers per proposal, rather then four or five. Therationale for the current scheme is probably (1) that the panel chair should not have theadditional burden, beyond his or her organizational duties, of scoring every proposal, and(2) possibly, that the chair should not be given additional power. Whatever the rationale,we feel it ought to be elucidated somewhere.
6. Reviewer effort expended in resolving minor score differences. Some reviewers, in spiteof having a large number of proposals to review (50 or 60+ in some cases), produce rawscore distributions which are either completely free of tie scores—equivalent to a totalrank ordering of the proposals—or very nearly free of ties. It almost surely takes iterationto achieve this level of discrimination, and we do not believe it is worth the effort todiscriminate this finely between similarly ranked proposals. Reviewers perhaps should beinstructed not to worry about minor score differences—that differences with respect to theother panelists’ scores will likely overweigh the fine tuning of one’s own scores.
7. Point scale. Along this same vein of reasoning: if the point scale had coarsergranularity—e.g., permitting only integer, or integer and half-integer scores—then reviewerswould not be able to discriminate so finely as now between similarly ranked proposals,11
and this restriction could possibly reduce the reviewers’ exertion of effort, lessening theirburden.
8. Rating one’s own competence to review a given proposal. In some review systems,particularly for journal or conference paper refereeing, reviewers are asked to rate their ownlevel of expertise in scoring the assigned paper or proposal (perhaps on a coarse scale ofone to three). See Haenni (2008), [6]. These self-confidence levels can then be factored in,via appropriate weighting, when computing an aggregate score for each proposal. At firstglance, since we have eight expert SRP panels rather than one; our expectation might bethat every panel member is fully qualified to review every proposal. But perhaps this ideais worth consideration.
This could alleviate some of the burden on reviewers by relieving them of the need toagonize over score decisions in sub-specialty areas that they are not totally familiar with.It could lessen their embarrassment in the face-to-face panel review if they have been toosqueezed for time to fully research every proposal. Also it could alleviate, somewhat, theneed for score modifications in the SRP group meeting—and thus make the process moreeffective and objective, overall.
9. Grading on a curve. The current normalization scheme does not modify the shape of areviewer’s score distribution (recall that it only shifts and linearly stretches or shrinks it).A simple alternative would be to grade on a curve: i.e., to modify each reviewer’s scoredistribution to match the percentiles of some target distribution, say, a truncated normaldistribution with a mean of five and standard deviation equal to two.
Another possibility would to choose as the target distribution the mean raw scoredistribution, averaging over all reviewers in the panel. That way, the scores of a “typical”reviewer (say, with a mean near 4.0, a standard deviation near 1.85, the typical positiveskew, and few outliers) would be minimally altered. (See Section 5, below.)
11For example, if the scores were restricted to half-integer scores in the range [.5, 9.5], then there wouldbe only 19 distinct possible raw scores, as opposed to 99.
10
2.6. Discussion. At a recent NRAO scientific staff meeting, concerns were raised aboutthe appropriateness of proposal review combining all three instrumental categories (GBT,VLA, VLBA), as opposed to having separate review panels for each of these categories.The consolidated review is likely more economical than the alternative, and it is more inaccord with the prevailing “One Observatory” philosophy. But one might argue that eachinstrument ‘deserves’ its own dedicated review structure.12 A specific concern that wasraised was that, given the premier capabilities of the GBT for pulsar studies, meritoriousproposals for pulsar studies using unique capabilities of the VLA might be unfairly out-competed by GBT proposals in the consolidated review (within the ETP panel). In light ofthis we have taken a detailed look at the distribution of normalized scores, by instrument, forthe four most recent review cycles. Histograms of these distributions, along with summarystatistics, are shown in Appendix C. The cumulative distributions are shown in Appendix D,together with distributional two-sample test statistics (Kolmogorov–Smirnov, Cramer–vonMises, and Anderson–Darling P -values) showing pairwise comparisons: GBT vs. VLA, GBTvs. VLBA, and VLA vs. VLBA.
In Appendix E we compare, for each of the SRPs, the initial rank-order aggregatepreferences vs. the rank order after SRP meeting score adjustments. (Since only the ag-gregated scores are adjusted, as opposed to individual reviewers’ scores, c.d.f. comparisonslike those of Appendix D are not possible.) We were surprised by the large number of scoreadjustments and rather extreme rank-order excursions which are seen in some cases (e.g.,NGA panel, semester 13A).
3. Comparison with Procedures Used Elsewhere
[Note: We apologize that this section is incomplete. We will try to make an updatedversion available.]
European Southern Observatory. An ESO working group undertook a review of their pro-posal selection process in 2012. The group report, and an accompanying study of thegrowth of observing programs at ESO, were published in the December 2012 issue of theESO Messenger [15, 16]. Their review system includes thirteen panels, with six memberseach, to cover four science categories: Cosmology (three panels); Galaxies and Galactic Nu-clei (two panels); ISM, Star Formation, and Planetary Systems (four panels); and StellarEvolution (four panels). Proposals for all the ESO telescopes, at Paranal and LaSilla, aswell as APEX, are reviewed by the same panels. They review typically 1000 proposals persemester, with an average of approximately 70–80 proposals per panel. Their review processis structured similarly to ours. The rating scale is 1 to 5 (low is best), with a granularityof 0.1. The bottom 30% of proposals are “triaged” (i.e., not considered in the post-ratingpanel discussion).13
Additional details are given in [17]. One notable differences from the NRAO reviewprocess is that, in committee, revised scores for proposals are submitted by each reviewer,and this is done by formal written ballot. These scores then are averaged to arrive atthe group consensus (in contrast to the procedure in the NRAO SRPs, where there is noprescribed procedure for score adjustment). Also, apparently, separate votes are taken pertelescope.
ALMA. For ALMA proposals there are five science categories (Cosmology and the high-zuniverse; Galaxies and galactic nuclei; ISM, star formation and astrochemistry; Circum-
12Similar concerns have been raised concerning ESO proposal review. See [15].13However, according to [17] there is a mechanism by which a panel may request that a triaged
proposal be “resurrected”.
11
stellar disks, exoplanets, and solar system; and Stellar evolution and the Sun) [18]. Thereare eleven review panels (two for category 1; three each for categories 2 and 3; two forcategory 4; and one for category 5), with seven members per panel. Initially each proposalis rated by four reviewers. The range of scores is 1 to 10 (low score is best). Each reviewer’sraw score distribution is normalized to a common mean and variance. The last 30% aretriaged, and not considered further by the review committee. Otherwise, the initial scoresare taken as recommendations only. A final rank-ordering of proposals is arrived at byconsensus of the panel members.
Arecibo Observatory. Arecibo has a similar panel structure and similar review procedure.They use a scale of 1 to 9; high numerical score is best.
Hubble Space Telescope.
NOAO.
National Science Foundation and National Institutes of Health. Proposal review proce-dures in use at NSF are not uniform across the various divisions of the agency, accordingto Robert L. Dickman and William E. Howard III (private communication). Based oninformation from the Web, it appears that relatively more uniform procedures have beenadopted by NIH than by NSF.
According to Hal R. Arkes (2003) [20], in 1994 the GAO issued a report with anevaluation of the review procedures at NSF and NIH. With regard to NSF, the authorsof this report “were concerned that the stated criteria by which NSF proposals should beevaluated were not, in fact, the only criteria by which such proposals were evaluated. . . .[and] the GAO ‘stated that we found that unwritten or informal criteria were used by panelsat all three agencies’.” In the 1994–1995 time period, according to Arkes, both NSF andNIH were revamping their review processes, and he was involved with both efforts.
Arkes chose to examine one of the grant review processes, in which each reviewer (outof four, total) was asked to submit numerical scores on each of four specific criteria, and alsoto submit an overall score. He found, by regression analysis (for 70 proposals), that, whilethree out of four reviewers’ overall scores agreed with their scores on the individual criteria(R2 values between 0.80 and 0.95, accounting for a large proportion of the variance in theoverall ratings), one reviewer was not consistently using these four criteria in generating anoverall rating (R2 = 0.28).
Apparently, in the mid-90s NSF and NIH review panels were not generally using scorenormalization. Arkes suggested using z-scores, i.e., standardizing to normal distributionwith a mean of zero and a standard deviation equal to one. This is entirely equivalent toour normalization procedure.
His second suggestion was that proposals be “triaged” before panel discussion (similarlyto the ESO procedure), i.e., to generate “cut-off” scores, so that panelists would not needto discuss proposals that had no chance of funding.
His final suggestion was to use what he termed “disaggregated” reviewers’ ratings—i.e., have the reviewers score only on the the individual criteria, and not submit an overallscore—the argument being that this would make it more likely that solely official criteriawould be used.
The NSF rejected all three of his recommendations (including score normalization).NIH around this time period utilized a 150-point scoring system. Arkes cites psycho-
logical research studies which he said show that “if points on rating scales extend beyondapproximate seven”, rater reliability either drops or fails to increase. Other consultants, hesays, were also recommending that the scale be trimmed.
The only recommendation that NIH adopted was to explicitly request the reviewers12
to rate by each criterion. And they considered score normalization to be too difficult toimplement.
From current Web documentation we find that NIH now asks reviews to score on ascale of 1 to 9. They must provide both an overall impact score and scores on, typically,five specified criteria. The guidelines state specifically that the impact score is not intendedto be an average of criterion scores. Reviewers may modify their initial scores during thepanel review meeting.
4. Rating Aggregation Methods Based on Pairwise Score Comparisons
There is a great deal of current interest in rating and ranking methods, and a large,rapidly growing literature. A comprehensive survey on the subject is given by Amy N. Lang-ville and Carl D. Meyer in a book titled Who’s #1? The Science of Rating and Ranking,published by Princeton University Press in 2012 [5]. There the concentration is on algebraicand graph-theoretic methods (rather than classical or Bayesian statistical methods) whichare widely used today in fields such as sports team ranking, e-commerce (e.g., Amazon,Netflix, most major retailers), and information search and retrieval (e.g., Google).14 Algo-rithmic techniques based on the same theory as these can be used for aggregating the scoresof journal referees or proposal reviewers. On pp. 179–181 of Langville and Meyer there isan aside titled “Ranking NSF Proposals”. Below we will show the result of applying theirsuggested method to the AGN Panel, Cycle 13B reviewers’ scores.
The method they propose is based on the Perron–Frobenius theorem (which dates backto 1907–1912) on the eigenvalues of real, square, non-negative, irreducible15 matrices. Thefoundations for methods of this type were established around the early 1950s (see [7]): byJohn R. Seeley (1949); by T.-H. Wei (1952), a student of the famous statistician MauriceKendall at Cambridge University, in a Ph. D. dissertation titled The Algebraic Foundationsof Ranking Theory; and by Kendall (1955). The idea is that given n entities to compare andan n×n matrix M expressing by how much entity i is favored over entity j (or vice-versa),for all pairs i and j, the normalized eigenvector corresponding to the dominant eigenvalueof M provides the correct ranking. So, for each of m reviewers one can construct a squarematrix Mk of pairwise score differences; i.e., in row i column j of the matrix one has thescore differential |s(i) − s(j)|, if proposal i is rated above proposal j, and 0 otherwise.One normalizes each Mk by dividing by the sum of all entries. (It does not matter if thereviewer has not scored all proposals.) Then one forms the average of these matrices andfind the dominant eigenvector. Assuming the matrix is irreducible (which it will be unlessthe review assignments are inadequate) the Perron–Frobenius theorem guarantees that theelements of the dominant eigenvector will be non-negative. These elements then representthe aggregated scores of the reviewers, according to the theory of Wei et al.
Results using the Langville–Meyer method. Figure 4 shows a comparison between our usualmean standardized scores and the scores obtained using the Langville–Meyer formulationof the method above. The comparison is by means of bipartite score plots. Proposal IDnumbers are shown along the vertical axes. In general, the quartile memberships agreefairly well, but we were surprised by the number of relatively large jumps in rank (which weobserved also for other cycle/panel pairs). In the case of Proposal 8051, we see a jump fromthe third quartile to the first, and for Proposal 7756 we see a similarly large jump—bothof these proposals were rated by only three reviewers. In the first quartile, Proposal 7972
14Google’s CEO Larry Page (a computer scientist) has a net worth of around $31 billion, largely dueto the success of his PageRank algorithm which is at the heart of the Google search engine.
15The definition is somewhat technical.
13
jumps eight places higher in ranking—this proposal was rated by only two reviewers. ForCycle 13B, AGN, the distribution number of reviewers per proposal is as follows: just oneproposal had only two reviewers, three proposals had just three reviewers; seven proposalshad four reviewers, and thirty-nine proposals had five reviewers. The percentages are:2 reviewers, 2%; 3 reviewers, 6%; 4 reviewers, 14%; and 5 reviewers, 78%.
Results using a method due to Gleich and Lim. Another approach to the aggregation ofratings or scores is described by David Gleich and Lek-Heng Lim in [8] and Jiang et al. [9].We summarize the Gleich and Lim process as follows: They begin by noting a connectionto skew-symmetric matrices.16 Given a column vector of n scores s = (s1, . . . , sn)T , thematrix Y of pairwise score differences Yij = si − sj is skew-symmetric. Assuming Y 6= 0,Y is of rank two, since it can be written in the form
Y = s eT − e sT , (3)
where e is a column vector of n ones. Suppose one were given a measured version Y of thatmatrix, contaminated by noise and perhaps missing some elements. In that case, one could
solve for a low-rank, skew-symmetric approximation to Y which is, in some well-defined
sense,17 closest to Y. (We denote the kth individual reviewer’s pairwise score differencematrix by Y(k).)
This is an example of the so-called matrix completion problem, which is a bit of ahot topic these days, as it arises in contexts such as compressive sensing. Algorithms formatrix completion are discussed by Gleich and Lim in [8, Section 3]. They treat the specialproblem of matrix completion restricted to the class of skew-symmetric matrices. Supposewithin a panel we have m reviewers of n proposals, whose scores—or ratings—are given by
an m × n matrix R. We then can form an n × n matrix Y whose elements represent thearithmetic means of reviewers’ pairwise score differences, i.e.,
Yij =
∑m
k=1(Rki − Rkj)
# {k |both Rki and Rkj exist}, (4)
where #{·} denotes the cardinality of the given set,18 and in the case of a denominator
equal to 0, we set Yij = 0. In our case, Y is not an exact pairwise difference matrix becausenot all m reviewers review all proposals.
In Section 3.1 of their paper, Gleich and Lim use the singular value projection algorithm
of Jain et al. [10], to find a rank-2 skew-symmetric approximation Y which is nearest to
16An n × n real matrix M is skew-symmetric if M = −MT.17As measured, say, by a matrix norm.18Gleich and Lim include a few other possibilities: Rather than the arithmetic mean of score differ-
ences, one might choose the (log) geometric mean of score ratios; i.e.,
bYij =
Pmk=1(log Rki − log Rkj)
#˘
k |both Rki and Rkj exist¯ ;
or binary comparison, in which case, Y(k)i,j = sign(Rkj − Rki) and
bYi,j = Probk
(Rki > Rkj) − Probk
(Rki < Rkj) ;
or the logarithmic odds ratio, with bYij = logProbk (Rki ≥ Rkj)
Probk (Rki ≤ Rkj).
14
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
7679
77097731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
79938010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
82258231
9.34758
2.97524
9.9
0.1
MStd Langville-Meyer
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
7679
77097731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
79938010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
82258231
9.34758
2.97524
9.9
0.1
MStd Langville-Meyer
Figure 4. Comparison between mean standardized scores and scores obtained using the Langville–Meyeralgorithm [5, pp. 179 ff.] for the Cycle 13B, AGN Panel. In the plot at left, the vertical scale is linear, fromminimum score (top) to maximum (bottom). (The Langville–Meyer dominant eigenvector scores have beenscaled to the range [0.1, 9.9].) At right, the score distances are ignored; i.e., the comparison is solely byrank order. The first quartile scores are shown in green, the second in blue, etc. Dashed lines correspond
to inter-quartile jumps.
15
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
79727992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
9.34758
2.97524
9.9
0.1
MStd Gleich-Lim
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
9.34758
2.97524
9.9
0.1
MStd Gleich-Lim
Figure 5. Comparison between mean standardized scores and scores obtained using the Gleich–Lim algo-rithm [8] for the Cycle 13B, AGN Panel. Here we see essentially identical memberships in the first quartile
(the only exception being proposals 7972 and 8178, which stay very close to the quartile boundary), andwe see identical memberships in the fourth quartile. There are two pairs of exchanges between the secondand third quartiles.
16
Y in the sense of the so-called nuclear norm. The nuclear norm is the matrix equivalentof the discrete ℓ1 vector norm. It is also known as the trace norm, or in physics a Ky-Fannorm. For a general real matrix A, the nuclear norm is simply the sum of the singularvalues of A; the singular values are equal to the square roots of the non-zero eigenvalues of
AT A. Given the minimum norm solution Y, the aggregate scores from the algorithm are
s = (1/n)Ye (which we rescale to cover the range [0,10]). A Matlab implementation of thisalgorithm can be found among the research codes available at David Gleich’s home page.19
We used our own Mathematica implementation.
Figure 5 shows a comparison between the mean standardized scores and scores obtainedusing the Gleich–Lim algorithm for the Cycle 13B, AGN Panel. Here we see rather lessextreme differences than in the comparison with Langville–Meyer scores (Fig. 4).
5. A Quadratic Programming Method and a Probabilistic
Score Normalization Algorithm for Score Aggregation
In this section we briefly describe two approaches from the recent literature on cali-brating and aggregating reviewers’ scores. These algorithms assume that high scores arebest, so we would need to additively invert the input raw scores (s 7→ 10 − s), and invertthe output aggregate scores as well.
5.1. The Method of Roos et al. Two interesting papers on calibrating the scores ofbiased reviewers were published in 2011 and 2012 by Roos et al., [11, 12]. These authorsfirst describe a standard linear modeling approach, referred to in the statistical literature,as two-way cross-classification in the analysis of variance (ANOVA), that can be solved bylinear least squares if all reviewers review all proposals. The score model
yij = µ + αi + βij + ǫij (5)
is additive, consisting of the overall mean score µ, the mean difference αi between thescores of reviewer i and µ, the mean difference βj between the scores of proposal j andµ, and a random error ǫij . The errors are assumed to be independent and identicallydistributed (i.i.d.). The solution parameters are the αi, which can be thought of as reviewerlenience parameters, and the βj , which are estimates of intrinsic proposal quality. Forranking n proposals it suffices to have estimates of the quality difference with respect toone chosen proposal, say βi−β1. For the imbalanced case, in which reviewers score differingsubsets of all proposals, a constrained least-squares solver is required, with the constraints∑
i αi = 0 and∑
j βj = 0. This algorithm is inadequate for our purposes, because it doesnot include a multiplicative scale factor.
The authors next describe a nonlinear model
yij = µ + γi(αi + βij + ǫi,j) (6)
which does include scale factors, the γi. With the substitution γi = 1/γi, the least-squaresobjective function becomes
Figure 6. Comparison between mean standardized scores and scores obtained using the Roos quadraticprogramming algorithm [11] for the Cycle 13B, AGN Panel.
18
Defining a vector x = (β1, . . . , βn, γ1, . . . , γm, α1, . . . , αm), one ends up with the quadraticprogramming problem:
minimize1
2xT Qx
subject to Ax ≥ b ,(8)
where the n × n matrix Q and the constraint matrix A both derive from Equation 7 andthe condition that 1
m
∑γi = 1. The solution can be obtained using existing solvers for
bound-constrained quadratic programming. Roos et al. used the Matlab MINQ; we usedthe Mathematica FindMinimum (which is actually a general-purpose solver). The solutionis the maximum likelihood estimate if the ǫij are i.i.d. Gaussian. A sample result is shownin Figure 6.
.
5.2. Grading on a Curve: The Method of Fernandez, Vallet, andCastells. Fernandez et al. in 2006 published a paper [13] on the topic of probabilisticscore normalization for rank aggregation. Their method consists in precisely matching thepercentage points of each reviewer’s raw score distribution to those of some common targetdistribution, then arithmetically averaging the scores so obtained. Thus it can be thoughtof as an extreme form of “grading on a curve”.
In their own application, in information retrieval, these authors appear to have specificgrounds to favor one target distribution over another, which are not relevant to our ownapplication. However, it occurred to us that it might be interesting in our application toapply this method, choosing as target distribution the mean distribution of raw scores,averaging over all reviewers within the given SRP for the given semester. Our rationale isthat, in this case, the “typical” reviewer would see relatively less difference with respectto his or her initial scores.20 Another thought would be to choose as target distributiona parametrized, bounded distribution (covering the range [0,10]) with similar first fourmoments (mean, standard deviation, skewness, and kurtosis) to the mean score distributionshown in our Figure 2.
The mathematics of this method can be described succinctly: Let F denote the cu-mulative distribution (c.d.f.) of the target distribution, let F (−1) denote the inverse c.d.f.,and let Fr denote the empirical c.d.f. of the reviewer’s raw score distribution. Then thetransformation from raw score to normalized score is
snormalized = F (−1)(Fr(sraw)
). (9)
A sample result is shown in Figure 7.
6. A Proposal by Merrifield and Saari for Distributed Peer Review
We would like to call attention to a 2009 paper by Michael Merrifield and DonaldSaari [14] who advocate an alternative, distributed approach to peer review of telescopeproposals. (Merrifield has served as a member of the ESO proposal review committee.)Their approach would spread the task of proposal review across the user community byrequiring each PI to review a certain number, m, of other proposals. That number mightbe m = 10, for example. (For each additional proposal with the same PI, he or she wouldbe given another m review assignments.) Conflicts of interest would be declared, as usual,in which case alternate assignments would be made. The major advantages are:
(1) that no one would be burdened by the task of reviewing a very large number ofproposals;
20And be less likely to grumble.
19
(2) that the model is scalable: if the number of proposals increases, the number ofreview assignments, per reviewer, does not;
(3) that each proposal would be reviewed by the same number of reviewers—conflictsof interest would not reduce that number; and
(4) that instrument-by-instrument proposal review (GBT, VLA, and VLBA/HSA, eachseparately) would be more affordable than under the traditional review panel model.
Each PI would be required to perform his or her full list of assignments; failure to do sowould result in disqualification of the PI’s own proposal(s). Thus, the workload wouldbe distributed evenly, and, as the authors point out “there is a disincentive to taking thelottery-ticket approach to telescope applications.” And the views of the entire communitywould be taken into account. Consensus rank-order preferences would be assigned using arank-aggregation of the type discussed in Section 4 of this report. Various safeguards wouldbe built-in, in order to award good refereeing.
Brinks et al. [15] report that the ESO working group on ran a test on the method.NSF is sponsoring a pilot study within their Sensors and Sensing Systems program. ThePI on that study, George Hazelrigg, an NSF official, reports that the response to the initialcall for proposals was well-received (private communication).
7. Discussion
With regard to proposal scoring and score aggregation we conclude that:
(1) We are not out of the mainstream, with respect to the procedures used by otherobservatories and at the federal science agencies; however,
(2) Our score aggregation procedure is behind the current state of the art, as exempli-fied by the modern rating and ranking theories—developed by mathematicians andcomputer scientists—that are widely used in the corporate world.
In Section 2.5 we offered suggestions for minor modifications of the current score normal-ization and aggregation procedure. We believe these suggestions should be given carefulconsideration. However, we do not strongly advocate that the alternative score-aggregationprocedures discussed in Section 4 should be adopted. This is because the practical differ-ence from these might well be “in the noise,” in comparison to the score adjustments thatare made in the SRP Panel meetings. (We do note, however, that any of the alternativemethods would be an easy plug-in replacement for the current score-aggregation module.)
On the other hand, if the External Review Committee were to recommend that a higherdegree of reliance be placed on the initial, independently derived SRP scores, then thesemore “sophisticated” score-aggregation methods should be considered.
20
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
9.34758
2.97524
7.79674
1.14767
MStd MPSN
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
7679
7709
7731
7756
7768
7770
7789
7827
7833
7859
7867
7874
7880
7897
7903
7905
7924
7939
7943
7956
7965
7972
7992
7993
8010
8015
8025
8040
8051
8055
8061
8096
8108
8112
8123
8133
8136
8138
8161
8163
8164
8166
8170
8172
8176
8178
8193
8219
8225
8231
9.34758
2.97524
7.79674
1.14767
MStd MPSN
Figure 7. Comparison between mean standardized scores and scores obtained using the algorithm of
Section 5.2 (Fernandez et al. [13], “grading on a curve”) for the Semester 13B, AGN Panel proposals. Thetarget distribution is the distribution of mean scores including all panel members. There are no inter-quartilejumps.
21
References
[1] Bryan Butler, “Requirements for the PST for the new NRAO proposal evaluation and time allocationprocess”, Version 2.10, NRAO, October 13, 2010; included here as Appendix A.
[2] J. L. Hodges, Jr. and E. L. Lehmann, “Estimates of location based on ranks tests”, Ann. Math. Stat.,Vol. 34, No. 2, 1963, pp. 598–611.
[3] Peter J. Rousseeuw and Christophe Croux, “Alternatives to the median absolute deviation”, J. Amer.Stat. Assoc., Vol. 88, No. 424, 1273–1283.
[4] Bernard W. Silverman, Density Estimation for Statistics and Data Analysis, Monographs on Statisticsand Applied Probability 26, Chapman & Hall/CRC, 1986.
[5] Amy N. Langville and Carl D. Meyer, Who’s #1? The Science of Rating and Ranking, Princeton
University Press, 2012.[6] Rolf Haenni, “Aggregating referee scores: an algebraic approach”, preprint, 2008; available on the Web
at: www.iam.unibe.ch/ run/papers/haenni08e.pdf .[7] Sebastiano Vigna, “Spectral Ranking”, preprint Nov. 8, 2013; see arXiv:0912.0238v13 .[8] David F. Gleich and Lek-Heng Lim, “Rank aggregation via nuclear norm minimization”, preprint,
Feb. 23, 2011; see arXiv:1102.4821v1 .[9] Xiaoye Jiang, Lek-Heng Lim, Yuan Yao, and Yinyu Ye, “Part 1: Rank aggregation via Hodge Theory”,
NIPS Workshop on Advances in Ranking, Neural Information Processing Systems Foundation, Waikiki,
HI, Dec. 2009; also, arXiv:0811.1067v2 .[10] Raghu Meka, Prateek Jain, and Inderjit S. Dhilon, “Guaranteed rank minimization via singular value
projection”, preprint, 2009, arXiv:0909.5457 .[11] Magnus Roos, Jorg Rothe, and Bjorn Scheuermann, “How to calibrate the scores of biased review-
ers by quadratic programming”, in Proceedings of the Twenty-Fifth AAAI Conference on ArtificialIntelligence, 2011, pp. 255–260, Association for the Advancement of Artificial Intelligence.
[12] Magnus Roos, Jorg Rothe, Joachim Rudolph, Bjorn Scheuermann, and Dietrich Stoyan, “A statisticalapproach to calibrating the scores of biased reviewers: the linear vs. the nonlinear model”, in Sixth
Multidisciplinary Workshop on Advances in Preference Handling, 2012.[13] Miriam Fernandez, David Vallet, and Pablo Castells, “Probabilistic score normalization for rank aggre-
gation”, in Advances in Information Retrieval: 28th European Conference on IR Research, ECIR2006,Eds. M. Laimas et al., Lecture Notes in Computer Science, Vol. 2936, 2006, Springer Berlin/Heidelberg,pp. 553–556.
[14] Michael Merrifield and Donald Saari, “Telescope time without tears: a distributed approach to peerreview”, Astronomy and Geophysics, Vol. 50, Issue 4, Aug. 2009, pp. 4.16–4.20.
[15] Elias Brinks, Bruno Leibundgut, and Gautier Mathys, “Report of the ESO OPC Working Group”,
European Southern Observatory, The Messenger, Vol. 150, Dec. 2012, pp. 21–25.[16] Ferdinando Patat and Gaitee Hussain, “Growth of observing programs at ESO”, European Southern
Observatory, The Messenger, Vol. 150, Dec. 2012, pp. 17–20.[17] European Southern Observatory, ESO Period 90: A step-by-step guide for OPC & Panel members;
available on the Web at www.vt-2004.org/public/about-eso/.../opc/docs/P90 step-by-step.pdf .[18] Francoise Combes, “ALMA Proposal Review”, Observatoire de Paris, slide presentation dated 12 No-
vember 2013; see www.asa2013.sciencesconf.org/file/56031 .[19] National Science Foundation, “Dear Colleague Letter: Information to Principal Investigators (PIs)
Planning to Submit Proposals to the Sensors and Sensing Systems (SSS) Program October 1, 2013,
Deadline,” www.nsf.gov/publications/pub summ.jsp?ods key=nsf13096 .[20] Hal R. Arkes, “The non-use of psychological research at two federal agencies”, Psychological Sci.,
Vol. 14, 2003, pp. 1–6.[21] Neill Reid, “Behind the TAC Process”, Space Science Telescope Institute, Feb. 13, 2014.
22
Appendix A
23
Appendix A
24
Appendix A
25
Appendix A
26
Appendix A
27
Appendix A
28
Appendix B. Reviewers’ Raw Score Distributions
for Proposal Cycles 12B, 13A, 13B, and 14A
29
Appendix B
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
Reviewer 2
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
Reviewer 7
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
Reviewer 8
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
0.30
Reviewer 26
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
Reviewer 125
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
Reviewer 133
AGN Panel Raw Score Distributions, Cycle 12B H55 proposalsL
nr Mean Std Dev Skewness Kurtosis Median H-L Loc Scaled MAD Sn Qn
Figure B-1. This figure shows AGN panel reviewers’ raw score distributions for proposal cycles 12B, 13A,13B, and 14B. The mean, standard deviation, skewness, and kurtosis are given in tabular form, as are
the sample median and the Hodges–Lehmann robust/resistant estimates of distribution centrality. Besidesthe standard deviation, three other estimates of distribution scale are shown: the (scaled) median absolutedeviation (MAD) about the median, and the Sn and Qn scale estimators of Rousseeuw and Croux. GMVAproposal scores are excluded from these distributions. The ordinate in each case represents the probabilitydensity. (Continued on next page.)
30
Appendix B
0 2 4 6 8 100.00
0.05
0.10
0.15
Reviewer 125
0 2 4 6 8 100.000.050.100.150.200.250.300.35
Reviewer 136
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
Reviewer 149
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
Reviewer 150
0 2 4 6 8 100.00.10.20.30.40.50.6
Reviewer 151
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
0.5
Reviewer 170
AGN Panel Raw Score Distributions, Cycle 13B H50 proposalsL
nr Mean Std Dev Skewness Kurtosis Median H-L Loc Scaled MAD Sn Qn
Figure C-1. This figure shows plots of the distribution of AGN panel normalized scores, by instrumentalcategory (GBT, VLA, and VLBA/HSA), together with summary statistics, for proposal review cycles 12B,13A, 13B, and 14A. GMVA scores have been excluded from these distributions (and excluded from the
score normalization).
47
Appendix C
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
EGS Panel, Cycle 12B
0 2 4 6 8 1005
10152025
GBT, n = 64
0 2 4 6 8 1005
10152025
VLA, n = 85
0 2 4 6 8 100123456
VLBA�HSA, n = 15
Mean Std Dev Skewness Kurtosis Median H-L Loc Scaled MAD Sn Qn
Figure D-1. The plots above show the empirical cumulative distribution functions of AGN panel nor-malized scores—categorized by instrument (GBT, blue; VLA, green; and VLBA/HSA, red)—for proposalreview cycles 12B, 13A, 13B, and 14A. The tabulated statistics represent probabilities for the hypothesis
of identical distributions for GBT versus VLA scores, GBT versus VLBA/HSA scores, and VLA versusVLBA/HSA, according to three standard statistical tests. GMVA scores have been excluded.
56
Appendix D
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
EGS Panel, Cycle 12B CDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
EGS Panel, Cycle 13A CDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
EGS Panel, Cycle 13B CDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
EGS Panel, Cycle 14A CDFs
Comparison of EGS Panel GBT HblueL and VLA HgreenL Normalized Score Distributions
Figure E-1. Comparison of AGN Panel initial rank-order preference vs. rank-order after SRP meeting
score adjustments, for semesters 12B through 14A (left to right). The first quartile is shown in light green,the second in light blue, etc. Dashed lines indicate inter-quartile jumps. The table shows the total numberof proposals per semester, the number of scores that were adjusted, the percentage that were adjusted, andthe Kendall τ and Spearman ρ measures of rank-correlation between the initial aggregate scores and thefinal, adjusted scores.
65
Appendix E
6342
6417
6421
6431
6439
6441
6458
6489
6515
6518
6522
6548
6555
6560
6568
6571
6580
6601
6647
6650
6667
6668
6700
6707
6783
6799
6805
6816
6837
6853
6855
6865
6877
6342
6417
6421
6431
6439
6441
6458
6489
6515
6518
6522
6548
6555
6560
6568
6571
6580
6601
6647
6650
6667
6668
6700
6707
6783
6799
6805
6816
6837
6853
6855
6865
6877
8.46667
2.89947
8.47
2.9
MStd TAC
6401
6988
6990
6998
7006
7007
7034
7046
7063
7070
7071
7075
7076
7089
7140
7159
7160
7164
7166
7198
7210
7217
7224
7238
7268
7302
7313
7326
7366
7388
7392
7420
7434
7436
7447
7472
7474
7477
7480
7504
7515
7530
7538
7540
7547
7556
7567
7610
6401
6988
6990
6998
7006
7007
7034
7046
7063
7070
7071
7075
7076
7089
7140
7159
7160
7164
7166
7198
7210
7217
7224
7238
7268
7302
7313
7326
7366
7388
7392
7420
7434
7436
7447
7472
7474
7477
7480
7504
7515
7530
7538
7540
7547
7556
7567
7610
7.90915
1.77027
7.91
1.77
MStd TAC
7719
7729
7743
7754
7773
7787
7790
7823
7830
7855
7871
7922
7930
7931
7953
7973
7978
7994
7999
8004
8014
8026
8027
8030
8035
8060
8072
8104
8141
8162
8174
8175
8191
7719
7729
7743
7754
7773
7787
7790
7823
7830
7855
7871
7922
7930
7931
7953
7973
7978
7994
7999
8004
8014
8026
8027
8030
8035
8060
8072
8104
8141
8162
8174
8175
8191
8.32458
2.70189
8.32
2.7
MStd TAC
7950
8358
8392
8405
8423
84538465
8468
8483
8495
8498
8500
8507
8547
8552
8558
8559
8560
8591
8595
8613
8619
8622
8631
8633
8634
8663
8666
8680
8696
8700
8705
8715
8727
8738
8740
8744
8746
8753
87598761
8793
8798
8815
8817
8864
8879
8889
8934
8937
8950
8951
8979
89828989
8992
8994
9002
9003
7950
8358
8392
8405
8423
84538465
8468
8483
8495
8498
8500
8507
8547
8552
8558
8559
8560
8591
8595
8613
8619
8622
8631
8633
8634
8663
8666
8680
8696
8700
8705
8715
8727
8738
8740
8744
8746
8753
87598761
8793
8798
8815
8817
8864
8879
8889
8934
8937
8950
8951
8979
8982
8989
8992
8994
9002
9003
9.20168
2.38068
9.2
2.38068
MStd TAC
EGS,12B EGS,13A EGS,13B EGS,14A
ð of proposals 33 48 33 59ð of score adjustments 5 4 2 15
Figure E-8. Like Figure E-1, but comparing SSP panel initial and final rank-order preferences.
72
Appendix F
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
Score, s
Frac
tion
Adj. Linearized Score CDFs, Cycle 12B
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
Score, s
Frac
tion
Adj. Linearized Score CDFs, Cycle 13A
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
Score, s
Frac
tion
Adj. Linearized Score CDFs, Cycle 13B
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
Score, s
Frac
tion
Adj. Linearized Score CDFs, Cycle 14A
Figure F-1. These plots show the cumulative distributions of adjusted, linearized SRP aggregate scores,by instrument. GBT proposal scores are shown in blue, VLA scores in green, and VLBA/HSA scores in red.
73
Appendix F
Adjusted, Linearized Score CDFs, AGN Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, AGN
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, AGN
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, AGN
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, AGN
Adjusted, Linearized Score CDFs, EGS Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, EGS
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, EGS
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, EGS
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, EGS
Figure F-2. The cumulative distributions of adjusted, linearized SRP aggregate scores, by instrument, forthe AGN and EGS panls. (Continued on next page.)
74
Appendix F
Adjusted, Linearized Score CDFs, ETP Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, ETP
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, ETP
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, ETP
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, ETP
Adjusted, Linearized Score CDFs, HIZ Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, HIZ
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, HIZ
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, HIZ
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, HIZ
Figure F-2 (Continued). The cumulative distributions of adjusted, linearized SRP aggregate scores, byinstrument, for the ETP and HIZ panels. (Continued on next page.)
75
Appendix F
Adjusted, Linearized Score CDFs, ISM Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, ISM
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, ISM
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, ISM
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, ISM
Adjusted, Linearized Score CDFs, NGA Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, NGA
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, NGA
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, NGA
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, NGA
Figure F-2 (Continued). The cumulative distributions of adjusted, linearized SRP aggregate scores, byinstrument, for the ISM and NGA panels. (Continued on next page.)
76
Appendix F
Adjusted, Linearized Score CDFs, SFM Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, SFM
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, SFM
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, SFM
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, SFM
Adjusted, Linearized Score CDFs, SSP Panel
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
12B, SSP
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13A, SSP
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
13B, SSP
0 2 4 6 8 10
0.0
0.2
0.4
0.6
0.8
1.0
14A, SSP
Figure F-2 (Continued). The cumulative distributions of adjusted, linearized SRP aggregate scores, byinstrument, for the SFM and SSP panels. (Continued on next page.)
77
Appendix F
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
12B, ETP Pulsar Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
12B, ETP Pulsar Proposal CDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13A, ETP Pulsar Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13A, ETP Pulsar Proposal CDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13B, ETP Pulsar Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13B, ETP Pulsar Proposal CDFs
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
0.30
14A, ETP Pulsar Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
14A, ETP Pulsar Proposal CDFs
Figure F-3. Comparison of linearized, adjusted score distributions for ETP pulsar proposals, by instru-ment. GBT proposal scores are shown in blue, VLA scores in green, and VLBA/HSA scores in red. Comparewith Figure F-4.
78
Appendix F
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
12B, ETP Triggered Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
12B, ETP Triggered Proposal CDFs
0 2 4 6 8 100.00
0.05
0.10
0.15
0.20
0.25
0.30
13A, ETP Triggered Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13A, ETP Triggered Proposal CDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13B, ETP Triggered Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
13B, ETP Triggered Proposal CDFs
0 2 4 6 8 100.0
0.1
0.2
0.3
0.4
0.5
14A, ETP Triggered Proposal PDFs
0 2 4 6 8 100.0
0.2
0.4
0.6
0.8
1.0
14A, ETP Triggered Proposal CDFs
Figure F-4. Comparison of linearized, adjusted score distributions for ETP triggered proposals, by in-strument. GBT proposal scores are shown in blue, VLA scores in green, and VLBA/HSA scores in red.Compare with Figure F-3.
Figure F-5. The table above shows modified Borda counts for each reviewer in each of the eight SRPs,
for semesters 12B, 13A, 13B, and 14A. This is for comparison of individual reviewers’ preliminary scoreswith the SRP average, normalized preliminary scores. These correspond to the Qprelim values in Table 1,Line 1 of Neill Reid’s Memorandum [21]. (We cannot compute the Qfinal scores, as he has done, becauseour panelists do not individually submit revised proposal scores; but rather they agree by acclamation ona consensus score for each proposal.) Also shown is a smooth histogram of the entire list of Qinitial values.The mean value, 0.605 ± 0.075, is very close to that of [21].
