Withholding Hubble Space Telescope Data

Good:
Actually, it is a very good thing; it improves the quality of the knowledge received from the data.

Note: This is a long article; the longest Bad Astronomy to date. This particular issue is one near and dear to me, so please bear with me!

1. People simply being curious as to why HST data are withheld from public view for a year,

2. People that think a NASA conspiracy is doing this, and

3. People that feel that as taxpayers they have a right to immediate access to the data.

Why withhold the data at all?
Answering that first question pretty much takes care of the others, so let's tackle it immediately. When data are received by a team of scientists using HST, it is standard operating procedure for them to have sole access to the data for a period of one year. To see why, it helps to know just how scientists get access to the telescope.

Hubble is an extremely sensitive instrument, and is capable of taking data that cannot be taken from the ground. For those reasons alone, HST is a very popular instrument for astronomers. There is only so much time HST can spend collecting data in a year, and so the available time on the 'scope is quite limited. Every year, the people at the Space Telescope Science Institute (STScI) announce on behalf of NASA that they are accepting proposals for research projects using HST. The response is always overwhelming; they receive proposals for five times as much time as HST has! So 80% of the proposals must be eliminated. This makes for pretty tough competition among astronomers to get time on HST. Since the 'scope is so expensive to operate, only the very best proposals will get accepted to run. This ensures that we get the best use of our money from HST.

Not surprisingly then, preparing a proposal to get time on HST is a long, arduous process. You must show rigorously why these observations you propose cannot be done from the ground, and why they will help the astronomy community better understand the objects you are observing. You must itemize each expenditure and defend it. Note that in general a lot of the funding goes towards supporting a graduate student, who will help reduce and analyze the data (the idea being that the student will go on to become a productive scientist as well). There is a lot of math and science that goes into a proposal, and the whole process can take weeks or months to prepare! That is time the scientist is not doing other research; they are making a tradeoff between doing work on data now and doing work with HST data later.

But suppose that you have created a proposal and it has been successfully run on HST. They send you the data; it may be several gigabytes of images or spectra, with as much as ten times as much data in the form of calibration data (telescope sensitivities, camera performances, etc.). It is a daunting amount of information. It can take months just to get it organized and ready to analyze.

Now suppose again that the folks at STScI simply take your data and immediately dump it on the Web for anyone to retrieve. There may be someone else out there better prepared than you to process the data. They may scoop you on your own data! You may think that perhaps that other team deserves to analyze the data if they are better prepared for it, but that is short range thinking; if there were no guarantee that you get exclusive rights to the data for a year, there is no incentive to spend all that time getting a proposal ready. In my personal opinion, there is always someone out there better prepared than you are to look at data, but how else will you ever learn how to do the best you can? If only one group ever gets the data no one benefits at all.

This is why the data are held back from the astronomy public for a year. After all the time preparing to get the data, it is only fair to the team that made the proposal to give them enough time to work on it. A year is a good compromise between giving them all the time they need and getting the data out there quickly for others to see. A year is actually a pretty short period of time: I have some observations that are now 7 years old that I have not completely analyzed yet! However, if someone else wants that data, they are free for the taking; since the data are older than one year, all anyone needs to do is ask STScI for it. That gives me incentive to get it done, and it also gives me incentive to do it right: someone else out there may be looking at the same data I am, and so I have to make doubly sure that my analyses are correct. Also remember that scientists ask for and receive grants from STScI to work on analyzing the data. If someone else scoops them, that money is wasted. So it also makes good monetary sense to let the proposing team keep the data rights for a year.

[Note added September 5, 2001: Incidentally, Bad Reader Tuomas Koivula from Finland emailed me with the analogy that HST data is like having a patent; you have rights to the material, but only temporarily. Eventually, that time is up and the patent becomes public. Mind you, people can get patents for ideas on which they worked while employed by the government as well, so the analogy is a pretty good one.]

Early Release of Data/Images
The team of people (or single person, of course) that got the data always has the option to release it earlier than one year. The Eagle Nebula images were released early, for example. It is standard procedure for data taken with new instruments to be released early as well. So-called "Early Release Observations", or ERO's, were taken when the corrective optics were installed in 1994, and more are planned for STIS and NICMOS in the coming months. Some of these ERO's are very exciting projects which will be jumped on by other astronomers, rest assured. For two whole years the research I did for my Ph.D. was based on a series of ERO's when HST was first launched. Also, some data are released immediately, like the amazing Hubble Deep Field.

An aside here: when you see the images released from STScI on the news, or in a newspaper, or even on the net, you are not seeing the actual data but the data converted to an image readable by computers (like JPG or GIF images). These formats cannot convey the detailed information in the actual data, and so cannot be used scientifically; you need the actual data numbers to produce any kind of real science. As someone who has tried to produce a nice picture from data, trust me here; it's hard! It takes a lot of time and effort to get the color balance right, to highlight the features you want to see, and then to put it in a format that everyone can read. And hardcopies still don't look as good. Unfortunately, STScI does not have the money or staff to produce nice pictures of all the data HST takes. It is up to the scientists themselves to produce those images, and so many images never make it to the public eye. Even so, there are many many spectacular images that are made public. I have been told that HST cost far less than a single B2 bomber, so even making a hardcopy of one HST image gives you more return on your tax dollar than you put in.

What about a conspiracy?
Most of the conspiracy theories I have seen on the 'net deal with images taken by HST of the bright comet Hale-Bopp. People ask why these images are not public; but now we see that if they are older than a year they are public. The conspiracy has been amplified somewhat because no images have been released for some time. This is because HST has a very stringent restriction that it cannot look at any objects closer to the Sun than 45 degrees. This restriction has only been lifted once, to my knowledge, to look at Venus, and that proposal was one of the toughest graded by STScI. Hale-Bopp has been closer than 45 degrees from the Sun for quite some time, and so of course HST observations stopped. This is why we haven't seen anything from HST in some time.

Do people have a right to the data?
The last problem is people that feel they have a right to the data. This may surprise you, but I agree with that idea. HST is a scientific instrument that is paid for by the people and is generating answers (and questions) that are philosophical in their ramifications, and people have a right to know about it. Where I disagree with these people is to when everyone has a right to the data. I feel that the one year proprietary period is a Good Thing. It makes sure that the data are analyzed as well as possible. And at the end of that period, any piece of data HST has taken becomes freely available.

Think of it another way: your tax dollars also pay for top-secret military projects, but these are kept from you. Tax returns are also paid for by tax dollars. Do you have the right to see your neighbor's returns? Now, these examples are more along the lines of actual secrets, while HST has no secrets. But the argument is simply that your tax dollars do not give you the right to see everything they pay for. A better argument here would be to say that in the end, HST is for everyone, and so you have the right to the data. And you do have access to the data, but only after the one year proprietary period is over, for the reasons I outlined above. Remember, no one "owns" the data; the proposers simply have one year to process them.

Why are astronomers so special anyway?
These people may also feel that scientists are given special treatment to the data; in a sense they are placed in an ivory tower, doling out the data to the public in dribs and drabs. This is a delicate point: it does take a lot of training to understand just how to analyze and interpret the data. We are trained over years to understand just what goes into the data and how best to squeeze as much as we can out of it. It can take years of graduate education and experience to know how to interpret images. That is yet another reason why the data are released. It gives other scientists a chance to take a look themselves. Believe me, there are plenty of arguments back and forth between astronomers about conclusions based on HST data. Two people analyzing the same data can come to very different conclusions. That's what science is all about!

So, can you as a member of the public get your hands on the data? Not easily! The data are stored and maintained in an archive by STScI, which spends a lot of time and money keeping it up and supporting it. The format of the data, as stated above, is not easily interpretable by someone without a lot of special software, and of course it really does take years of training to know what to do with the data. So access is not simple to come by unless you can show a definite need for the actual data. Of course, it isn't for any other type of data either; given an x-ray, for example, how would you examine it? Well, you might simply want it as a nifty picture. That's fine, and STScI has many of HST's images online which you can download. Another good site is the HST Nuggets page which has lots of images as well.

My philosophy on all this
In my mind, this situation is ironic: people seem to be accusing astronomers of covering up data or withholding it because they look down their noses at the public. Believe me, that would never happen. I have never in my life met an astronomer who wouldn't talk at length about their research to anyone willing to listen. The reason we do this is because we love it. When the Soviet Union collapsed and the money for funding ran out, the scientists there still kept up their research even though they were making less money in a month than a minimum wage employee here makes in a day. When funding runs out here, astronomers do everything they can to keep the projects going. Look at amateurs: they pay thousands of dollars for equipment and spend cold, sleepless nights scanning the skies. Why? Because they love doing it.

I think of myself as a professional and an amateur. I get paid to work with astronomy and I am damn lucky to be. But I also do it for free, in my backyard. I have a small 10" Newtonian that I take out on occasion and look at the Moon, or try to find a few nebulae or galaxies in my heavily light-polluted sky. The need to do this is part of my soul. The need to help others understand it all is also a fundamental part of me, or else I wouldn't keep these web pages updated during my spare time. I don't get paid for this; I just want others to feel what I feel when I look at the images and ponder their implications. I think that most astronomers would agree with me as well.