Документ взят из кэша поисковой машины. Адрес оригинального документа : http://lnfm1.sai.msu.ru/~math/docs/ivoa_interop2006.pdf Дата изменения: Fri Sep 22 16:14:31 2006 Дата индексирования: Mon Oct 1 23:00:53 2012 Кодировка: Поисковые слова: europa

Storing and accessing the largest astronomical catalogues with the SAI CAS project

S. Koposov1,2,3, O. Bartunov2, S. Karpov4 et al2.
Russian Virtual Observatory Max Planck Institute for Astronomy, Heidelberg, Germany 2 Sternberg Astronomical Institute, Moscow, Russia 3 Institute of Astronomy, Cambridge, UK 4 Special Astrophysical Observatory, Nijniy Arhyz, Russia
1

IVOA Interop. meeting & Small projects meeting Moscow 2006

http://vo.astronet.ru http://vo.astronet.ru/preview/cas


Outline Existing data-centers, and catalogue access providers. Why we want to build our own Catalogue Access Service The requirements for our CAS system The importance of the Database for the CAS (PostgreSQL & Q3C) The technical realization of CAS What has been already done Example of work TODO




Storing and Accessing Astronomical Catalogues Existing projects


CDS OpenSkyQuery CASjobs with SDSS

They are all really great! But there are problems: Not OpenSource No good API to work with Not prepared for the work with VO No possibility of combining private datasets with large catalogues (cross-matches, etc...)


Requirements for the CAS Relational storage of the Data The ability to attach different metadata to the data (including different VO attributes). That metadata should be also stored in the relational database. OpenSource API allowing to work with it and allowing put miscellaneous WS on top of it. Good architecture (separation of the DB, WS software and frontend). The support of spatial queries and cross-matches. Easy uploading, editing of the catalogues (through the WS). The support for separate user storage area, allowing to work with all datasets.



Importance of the Database for the Catalogue Storage & Access System



Large Data volumes Spatial searches Cross matches

PostgreSQL + Q3C (Quad Tree Cube)


Q3C sky indexing scheme for PostgreSQL
C plugin for PostgreSQL Support of cone-searches, ellipse-searches, polygonal searches Circle & ellipse cross-matches (with constant cross-match radius and variable cross-match radius). Do not require additional columns in the table ­ only one index. Easy queries: SELECT * FROM table WHERE q3c_radial_query(ra, dec, 0.001) SELECT * FROM a, b WHERE q3c_join(a.ra, a.dec, b.ra, b.dec, 0.001) Extremely fast ( cone-searches on catalogues with ~ 109 objects take ~ 1 second, cross-matches of 2MASS with USNOB1 takes ~ 16 hours)


Q3C can be downloaded from http://q3c.sourceforge.net/ Ref: Koposov, Bartunov, ASPC 2006, vol. 351, p. 735


Structure of the metadata storage in the DB



Technical implementation of CAS.
The application seWver rS WS (Apache Tomcat, Apache Axis)
WS SOAP S OAP

JDBC

The Database server (PostgreSQL + Q3C)

The user

HT T P

The frontend (PHP, Apache, Aladin...)


Currently working services ConeSearch with output in different formats (VOTable, CSV) through HTTP GET query, but also through the SOAP WS Cross-match with user's VOTable The miscellaneous WebServices retrieving the information about the catalogues, their info, contents etc. The WS of new catalogue creation Some authorization WS are beeing built right now Basic SkyNode in testing regime



Currently loaded catalogues


USNO-A2.0 (5x108 objects) USNO-B1.0 ( 109 objects) NOMAD catalogue ( > 109 objects) 2MASS PSC (2.5x108 objects) 2MASS XSC (2x106 objects) Tycho 2 UCAC 2 GSC 1.2 2XMM SDSS DR5 is beeing downloaded


Small demonstration of capabilities of SAI CAS


TODO Enable the full SkyNode (it requires only wrapping of SkyNode WS calls around already existing WS API of SAI CAS. (the Basic Skynode will be available before ADASS). We are finishing the WCS matching web-service using the catalogues to WCS calibrate user images. The user's storage and upload of personal catalogues. There is a working prototype, but the authorization mechanisms are still under development. Start to enable more VO stuff on top of CAS (registry, SkyNode etc.)



Conclusions We presented the new Catalogue Access system the first large project made by Russian Virtual Observatory. It already has a collection of the largest existing astronomical catalogues You can work with it using the URLs: http://vo.astronet.ru http://vo.astronet.ru/cas/preview