Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.stsci.edu/~carolc/publications/ScienceNewsMetric.pdf Дата изменения: Sat Nov 15 06:26:55 2008 Дата индексирования: Fri Feb 11 02:28:42 2011 Кодировка: Поисковые слова: п п п п п п п п п п п п п п п п п п п

THE SCIENCE NEWS METRICS

CAROL A. CHRISTIAN

Space Telescope Science Institute Homewood Campus 3700 San Martin Drive Baltimore MD 21218, USA
carolc@stsci.edu AND GREG DAVIDSON

Northrup Grumman Space and Technology One Space Park Drive Redondo Beach CA 90278, USA
greg.davidson@ngc.com

Abstract. Scientists, observatories, academic institutions and funding agencies persistently review the usefulness and productivity of investment in scientific research. The Science News Metrics was created over 10 years ago to review NASA's performance in this arena. The metric has been useful for many years as one facet in measuring the scientific discovery productivity of NASA-funded missions. The metric is computed independently of the agency and has been compiled in a consistent manner. Examination of the metric yields year-by-year insight into NASA science successes in a world wide context. The metric has shown that NASA's contribution to worldwide top science news stories has been approximately 5% overall with the Hubble Space Telescope dominating the performance.

1. Introduction The Office of Space Science (OSS) of the US National Aeronautics and Space Administration (NASA), recently absorbed into NASA's Science Mission Directorate, is responsible for developing a space science program with a primary ob jective to accomplish fundamental science. Ultimately, like many science organizations, NASA's success is measured in part by the achievement of scientific insights relative to the cost. NASA OSS consid-


2

CAROL A. CHRISTIAN AND GREG DAVIDSON

ers scientific potential and output of missions in strategic planning and is held accountable for the associated costs. Independent measures of scientific accomplishments have been an integral part of this strategy. While fundamental science is the primary ob jective of the Space Science program, it is also among the most difficult of outcomes to measure. In the early 1990's, one of us (Davidson) led an effort at NASA to identify science metrics. At the most fundamental level, a good metric is something that you can count which is correlated with what you want. Desirable secondary characteristics include ease of data collection, precision, and low levels of bias. The group at NASA explored both prospective measures (planned capabilities in terms of angular resolution, spectral resolution, sensitivity, and time resolution, all as a function of wavelength coverage) as well as retrospective measures (quantity of data, number of observations, and bibliometric measures such as number of refereed papers or frequently cited refereed papers). Two retrospective metrics were selected from this analysis as being particularly well-suited to address NASA strategic planning needs. Key advantages of these metrics included relative ease of collection, independence of NASA, correlation with other, more complex metrics (particularly the citation bibliometrics) and ability to communicate results in a meaningful way to policy makers and to the public. These metrics are not perfect surrogate measures of all aspects of scientific performance, but they do provide important insights into fundamental scientific performance. The first measure, colloquially referred to as the "Science News Metric" is based on the annual listing of "most important stories" in the journal, Science News. This listing has a 31-year history and is published at the end of each calendar year. The Science News Metric essentially tracks "what's hot" in science on a year-by-year basis. The second metric formulated for OSS is the "Textbook Metric". This measure is an attempt to penetrate how the "hot" science topics of a single year get incorporated in the body of knowledge. The purpose is to understand OSS's capture of "intellectual market share" (what percentage of textbook material is based on OSS contributions) in the long term as well as overall growth of knowledge about astronomy. This metric will not be discussed herein. A similar metric was discussed by Christian (2004) in evaluating the impact of the Hubble Space Telescope mission. 2. A Description of the Science News Metric Science News is published weekly. It summarizes scientific findings in fields as diverse archeology, biomedicine, chemistry, mathematics, psychology, space science, and technology. Science News captures the essence of ref-


SCIENCE NEWS METRICS

3

ereed scientific publications in a digest form. Since 1973, Science News has published an annual list highlighting over 100 (usually 125-170) of the "most important stories" in science. Each year, the Science News Metric is compiled (by Davidson) to track those stories and estimate the OSS contribution to each. The metric represents the scientific or technical accomplishments for each year and from this the performance of Space Science funded by NASA can be compared over to all other "world-class" science in fields as diverse as archeology to biomedicine.
2.1. CALCULATION

The Science News Metric is calculated as follows: 1. All the "most important stories" are screened. Those that are not based on discoveries (data collected and scientific inferences made) or technological accomplishments are eliminated, usually about 15% of the total. 2. One point is awarded for each "most important story". In most cases, the discovery is due to collaborative efforts, and so credit is apportioned among the groups (foreign, ground-based astronomers, etc.) referenced by Science News as being responsible for the discovery. 3. NASA "points" are compared against points for all other scientific discoveries to establish NASA Space Science as a percentage of world science. The method allows comparison of NASA with other federal agencies (e.g., National Science Foundation). For example, in 2002, one of the top stories involved Dark Matter. In fact, Science News had three separate articles on this particular sub ject. The contributing NASA missions were the Wilkinson Microwave Anisotropy Probe (WMAP), the Hubble Space Telescope (HST), the High Energy Astronomical Observatory (HEAO), and the Sloan Digital Sky Survey (SDSS), but with different contributions (WMAP being the most significant followed by HST). The discovery was definitely based on data and research results, and involved more that one agency. WMAP was given 0.38 points, HST given 0.25 points, HEAO given 0.03 points and the SDSS given 0.05 points. The total number of points allocated is less than 1.0 because non-space facilities contributed to this story.
2.2. STRENGTHS AND WEAKNESSES

The strengths of the Science News Metric for the first, that the data used to generate this metric have pendently of NASA. The news stories are selected by dent entity. The research and assessment following

purpose intended are, been determined indea completely indepenthe publication of the


4

CAROL A. CHRISTIAN AND GREG DAVIDSON

year-end issue of Science News in December is conducted autonomously. Additionally, this metric provides data on individual missions as a function of time, thus yielding insight to support life-cycle cost trade-offs. One correlation with bibliometrics that was surprising to some in the early 1990's was that 30%-40% of science return occurred after the completion of prime mission lifetime. A parallel study of bibliometrics on IUE and OAO showed that peak publication of papers was 4-6 years after launch, and that peak publication of frequently cited papers (>5 citations/year) occurred 5-7 years after launch. Note there is some randomness in the metric due to the incidence of discoveries in a given calendar period. As an example, two black hole discoveries may share one point in one year. If the results had been split between years, the stories might have been attributed to more than one point. As an additional caveat, it is recognized that the Science News Metric is based on the journal Science News, and not as rigorous as a refereed scientific journal. One can argue that the number of "most important stories" from a mission as highlighted by a commercial journal is not necessarily correlated with the scientific value of that mission. What researchers value as the most important advances in their discipline may loosely correlate with what Science News reports, but the correspondence is not necessarily one-to-one. Also different science stories may involve varied levels of effort. For example, the importance of the single finding of the origin of the universe based on the Cosmic Background Explorer (COBE) mission data and analysis took considerable time and effort. In some sense the Science News Metric can capture the importance and level of effort in such situations because, for example, the COBE result was so significant that the story re-emerged repeatedly in subsequent years as a "most important story". Alternatively, attempts at using refereed publications on a year-to-year basis to evaluate the impact of specific discoveries and research results is time consuming and problematic. Usually refereed publications take considerable time to reach publication and an important sub ject may take years to accumulate the representative articles that would indicate "importance". Using a "sub ject citation index" if it existed as such would also be difficult in that it takes several years for a specific set of results to be assimilated by the scientific community and then referenced. It follows that refereed publications and citations can be useful for retrospectives, but as timely measures of productivity they are risky. The importance of the Science News Metric should be understood. As one facet of the accomplishments that NASA uses to formally report activity under the Government Performance and Results Act (GPRA), this indicator clearly is taken as a serious measure. The metric can have some


SCIENCE NEWS METRICS

5

deliberate influence on the funding for missions and specific lines of scientific inquiry. Certainly missions that appear favorably with high Science News Metrics use the metrics as a leverage point in arguing for continuation or augmentation. 3. Science News Metrics
3.1. TOP MISSIONS

Table 1 exhibits the 2004 Science News Metric results for the top scoring missions. The data are the accumulated points for all NASA missions during the period 1973 to 2004. The points accumulated through 2003 also are shown in the third column entitled "Points 03". It can be seen from this table, and has been borne out by previous analyzes, that the Hubble Space Telescope has been the single most productive mission NASA has supported. Other missions have had significant stories, surprisingly persistent over a long period of time, for example Voyager. This table cannot capture the whole picture of science productivity however. Stories such as the discovery of evidence for water on Mars and the success of the Mars rovers Opportunity and Spirit in 2004 are undisputed, but the accumulated points for those missions in one year do not pro ject those individual missions into the top 25 consistently generating significant results.
3.2. 2004 TOP STORIES

Another cut at the data includes the top stories listed by Science News that contain results attributable to NASA missions. These results are exhibited in Table 2. This table contains the story description (Column 5) and the fractional points (Column 1) attributable to a specific NASA mission or program (Column 4). The NASA Center (for example Jet Propulsion Laboratory = JPL or Goddard Space Flight Center = GSFC) are designated in the third column. The Sponsor column refers to an agency (such as European Space Agency = ESA or the US Department of Defense = DoD) or a branch of NASA as it existed in 2004. The designation "S" indicates Space Science, "Y" indicates Earth Science, "M/U" indicates the Human Space Flight enterprise and "R" indicates Aerospace and Technology. It can be seen that many stories are split between missions and facilities, demonstrating the multi-instrument, multi-wavelength nature of space science investigations.


6

CAROL A. CHRISTIAN AND GREG DAVIDSON

TABLE 1. Top 25 Most Productive Space Programs (1973-2004) Points show the cumulative number of discoveries between 1973 and 2004 or 2003 Points 04 1 2 3 4 5 6 7 8 9 10 11 11 11 11 15 16 17 17 19 19 21 22 23 24 25 53.0 15.7 15.2 11.2 9.8 9. 9. 7. 7. 5. 5. 5. 5. 5. 5. 5. 4. 4. 4. 4. 4. 4. 4. 3. 3. 5 2 7 5 8 5 5 5 5 3 2 7 7 6 6 4 2 0 8 7 Points 03 50.2 15.7 15.2 11.2 9.8 9. 9. 7. 7. 5. 5. 5. 5. 5. 5. 5. 4. 4. 4. 4. 4. 4. 4. 3. 3. 5 2 7 5 8 5 5 5 5 3 2 7 7 6 6 4 2 0 8 7 Program Hubble Space Telescope Voyager Viking Galileo Apollo, Skylab, Apollo Telescope Mount, Apollo-Soyuz Space Shuttle (HEDS and Microgravity) Gamma Ray Observatory Mars Global Surveyor Chandra X-ray Observatory Salyut NOAA Satellites Pioneer 10/11 Pioneer-Venus Physics and Astronomy Rockets and Balloons ISTP (SOHO, WIND, Polar) ROSAT Nimbus 4-7 Solar Maximum Mission (SMM) Cosmic Background Explorer (COBE) Infrared Astronomy Satellite (IRAS) Mariner 9/10 Venera Magellan Astromaterials High Energy Astrophysics Observatories (HEAO)

3.3. NASA PERFORMANCE

Table 3 compares NASA to a few other agencies represented in the news stories of 2004. The NASA enterprises are broken out, illustrating that Space Science has carried the bulk of the important stories for the agency. It also emphasizes the valuable investment the Office of Space Science has made in public information and outreach based on solid scientific research. Figure 1 exhibits the 31-year history of the NASA's contribution to the top stories, where the total for NASA is exhibited as well as the individual branches of NASA. The mid-1980's dip is associated with (a) the relatively weak scientific payoff from large investments in shuttle-based instruments,


SCIENCE NEWS METRICS

7

TABLE 2. 2004 Most Important Stories, sorted by mission. Note: Center refers to the institution responsible for building or operating the spacecraft making the discovery; in most cases the science investigations are done by scientists elsewhere in the other community or at other Centers. No. stories 0.8 0.2 1 0.65 0.05 0.3 1 1 0.5 0.9 0.5 Sp onsor S ESA S S S S S S S S S Center JPL ESA JPL HQ JPL GSFC GSFC JPL GSFC JPL GSFC Program MER Mars Express Cassini grants SST HST HST Stardust HST SST HST Discovery Discovery of evidence of past water on Mars Evidence of methane on Mars New measurements at Saturn and new high-resolution images Discovery of Sedna Discovery of Sedna Discovery of Sedna Hubble Ultra-Deep Field shows some of the earliest galaxies Highest resolution images ever taken of a comet Discovery of the earliest known galaxies Youngest star ever detected Dark energy is found to be uniformly spread across the universe Planetary debris disks found around Sun-like stars Planetary debris disks found around Sun-like stars Wind "highways" carrying spores and vegetation bits account for similarity of plant species on islands thousands of kilometers apart First flight of X-43a scramjet First flight of X043a scramjet Frost flowers provide a source of ozone-destroying bromine Iron Oxide concretions in Utah are similar to those on Mars Satellite InSAR imagery shows geomorphological changes associated with past underground nuclear tests

0.5 0.5 1

S S Y

GSFC JPL JPL

HST SST QuikSCAT

0.5 0.5 0.5 0.5 1

R R DoD S ESA

LARC DFRC DoD JPL ESA

Hyper-X Hyper-X DMSP MER ERS


8

CAROL A. CHRISTIAN AND GREG DAVIDSON

Table 2 (continued). No. stories 0.5 Sp onsor Y Center JPL Program GRACE Discovery Demonstration of gravitational frame dragging and test of general relativity Demonstration of gravitational frame-dragging and test of general relativity Demonstration of gravitational frame dragging and test of general relativity Discovery of the youngest planet known and organic compounds in a space region with potential for planets Detection of three of the lightest known planets Detection of three of the lightest known planets A class of proteins seem to trigger muscle atrophy The onset of the El Nino ~ phenomenon can be forecast as much as two years in advance The onset of the El Nino ~ phenomenon can be forecast as much as two years in advance The onset of the El Nino ~ phenomenon can be forecast as much as two years in advance Oxygen was present in small quantities on the Earth's surface 2.32 billion years ago, 100 million years earlier than expected

0.25

Y

GSFC

LAGEOS

0.25

ESA

ESA

LAGEOS

0.25

S

JPL

SST

0.1 0.02 0.5 0.125

S S M/U Y

HQ GSFC HQ JPL

Grants HST NSBRI QuikSCAT

0.25

Y

JPL

TOPEX

0.125

Japan

ADEOS

0.5

Y

ARC

Astrobiology

plus (b) the launch slips of the ma jor planned missions of the 1980's (Hubble, Galileo, Magellan, COBE, GRO) due to both developmental issues and the 3.5 year shuttle launch hiatus after Challenger was lost. At the end of the decade and into the 1990's, NASA productivity soars as returns come from both from those 1980's legacy missions as well new missions starting development in the 1990's. Figure 2 exhibits the cumulative percent contri-


SCIENCE NEWS METRICS

9

TABLE 3. NASA Enterprise Performance (1973-2004). Points 2004 5074 308.9 228.2 47.9 19.5 13.0 7.6 12.6 30.5 5.6 366.0 Percent 2004 100 6. 4. 0. 0. 0. 0. 09 50 94 38 26 15 Points 2003 4865 296.2 220.1 44.8 19.0 12.3 7.1 12.6 29.0 5.6 350.5 Percent 2003 100 6. 4. 0. 0. 0. 0. 09 52 92 39 25 15 Total "Most Important" stories in Science News NASA Space Science Earth Science Human Exploration Aeronautics/Technology U.S Department of Defense (Space) Soviet Union/Russia (Space) European Space Agency (and member states) Japan (Space) Total Space Activities

0.25 0.60 0.11 7.21

0.26 0.60 0.12 7.20

butions of individual missions, where it is seen that HST has contributed most significantly over time. 4. Synopsis The Science News Metrics, created under the auspices of NASA, have been instrumental for many years as one tool for determining the scientific discovery productivity of missions. The metric has merit because it relies on autonomous evaluation of data compiled independent of the agency. The metric has been compiled in a consistent manner for over 12 years and is valuable as a yearly probe of mission productivity and also draws strength as a multiple year measure of NASA science successes. The Science News Metrics fit well into the year-by-year analysis and reporting by NASA for internal purposes as well as to the federal government. The metric has shown that NASA's contribution to world wide top science news stories has been approximately 5% overall with the Hubble Space Telescope dominating the performance. Other measures, such as the number of scientific refereed publications based on mission data, also provide useful retrospectives on productivity. Publication history is a measure not only of new scientific insights but also of the longevity and integrity of specific results over periods of time


Figure 1. Thirty-one year history of NASA's contribution to Science News top stories.


Figure 2. Individual NASA mission contribution to Science News top stories.


12

CAROL A. CHRISTIAN AND GREG DAVIDSON

longer than a year. Other metrics serve different purposes as well, and in combination can give a variety of perspectives on the success of NASA scientific endeavors within the agency and compared to other organizations. Acknowledgements This work is supported by a contract, NAS5-26555, to the Association of Universities for Research in Astronomy, Inc. for the operation of the Hubble Space Telescope at Space Telescope Science Institute. Additional support for the annual production of the Science News Metric, performed by Luke Sollitt and Rick Ohlemacher, has been provided by contract SV3-73015, to Northrop Grumman Space Technology. References
1. Christian, C.A. 2004, The Public Impact of HST: A Case Study, in Organizations and Strategies in Astronomy ­ Vol. 5, Ed. A. Heck, Kluwer Acad. Publ., Dordrecht, 203-216.