It's not that simple

A take-off of John Dupuis's blog post that looks at Thomson's Nobel Prize predictions (which are based on citation counts) and how often they are wrong:

While citation frequency could be very important when it comes to applying for grants and tenure-track positions, some people might also claim that it has a direct effect on Noble Prize eligibility. Thomson Reuters, an information company created by Thomson Corporation, attempts to predict the Nobel Prize winners annually. Interesting enough, this goes back to 1989. Every year, they develop the list of Citation Laureates for sciences and economics winners. According to Thomson Reuters, there is a strong correlation between citations in the literature and the receipt of prestigious awards, such as the Nobel Prize. Based on this hypothesis, their success rate has been approximately 20% since 2002. It is noteworthy that this success rate is not based on each given year. What basically they have done is to predict who would win the prize someday. The successful prediction table is available here.

They claim that they own their success to the methodology they apply to develop Citation Laureates. This method is based on citation frequency (derived from Web of Science, a brand of Thompson Corporation) and Pedlebury's methodology. Based on Pendlebury's method, the top 0.1% papers of each scientific field are being selected. Then, using quantitative data analysis the annual predictions of the most influential researchers in each field will be identified. Unfortunately, this methodology is not transparent enough to be reproducible by others (1). In addition, one could argue that a strong correlation that they have been referring to is not convincing unless they present statistical analysis and numbers. Also, it is obvious that being in the top 0.1% in a scientific field is significant and it is a given for almost anyone who could win any prize. The pool that they are selecting their candidates from is large specially when they rate their success based not on each given year but all years since 2002. This approach increases the probability of being right even if the success is purely random.

John Dupuis has been trying to convince Thomson Reuters to admit to the fallacy of their predictions. He very well argues that when measuring the true impact of a person’s career, citation is only a small factor. Citation counts and the bar by which a researcher is measured is a wrong practice and it should not be advertised. This is a poor marketing strategy by Thomson and they will only damage their own credibility.

 In order to identify individual merits, citation count does not suffice nor does the Journal Citation Reports and the Impact Factor. Multiple metrics and other ranking algorithms, such as Faculty of 1000, Mendeley, and eigenfactor, should be considered specially by someone who is new and not familiar with the field. While each system has its own strengths and limitation, the combination of all might (only might) help to provide a better evaluation method. As some people say religious and politics do not make a good mix, I believe neither do science and business.