 IKI - MSR Research Workshop Space Research Institute (IKI), Moscow, Russia 11 June, 2009 |
|
Speaker: Mikhail Zhizhin, IKI RAS and GC RAS New technologies of data streaming and data storage in a parallel cluster of MS SQL databases, when combined with the parallel data mining and visualization algorithms inside the same MS Windows HPC cluster, will provide a powerful tool for interactive knowledge discovery for environmental research in the fields of global change, remote sensing and geophysics. This technology eventually will result in affordable petabyte-scale data center to store and mine scientific data, which will not only host a set of the most demanded environmental databases, but it will provide to authorized users a long term space to store, index and cross-correlate their data sets. We use VxOware search engine to manage metadata in federation of scientific virtual observatories. We have developed a universal Active Storage for multidimensional numeric arrays in UNIDATA Common Data Model (CDM) on parallel relational database cluster. Together with NOAA and MSR we have developed Environmental Scenario Search Engine (ESSE) for distributed scientific data mining in data arrays and streams with fuzzy logic. For the high-resolution visualization of the geo-referenced data on the MS Virtual Earth background we have a tiled-display video wall with a Windows-based interactive MultiViewer browser, designed together with the Computer Science Department of Moscow State University. We have extended OGSA-DAI grid and web service framework with a set of specific data processing and mining activities for CDM data resources. We prove the concept of the transparent data cube for the largest datasets from meteorology, global change and space physics within the MSR-funded CLIVT project. |