Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.sai.msu.su/~megera/postgres/talks/scienceDB.pdf Äàòà èçìåíåíèÿ: Fri Apr 23 23:14:46 2010 Äàòà èíäåêñèðîâàíèÿ: Sun Sep 12 01:10:56 2010 Êîäèðîâêà: Ïîèñêîâûå ñëîâà: âå÷íûé êàëåíäàðü

, -









­ ­

-- -- . -- CEO, TO, CIO-senior information officer NSF-CDI (Cyber-enabled Discovery and Innovation)


­ ­

.


--

Web Services
UDDI, WSDL,SOAP

!


WWW
URI,HTML,HTTP TEXT

Semantic Web
RDF,RDF(s),OWL

Email:

@address, text, smtp


R D BM S

XML D B

GRID

ACID ColumnarDB NoSQL EC2 S3 BASE StreamDB UtilityComputin CAP (key,value) SciDB COA I aaS BPELWS URI CloudDB SaaS SOAP Web2.0 OWL Science 2.0 U D DI REST RD F W S DL WS R D FS HTTP Semantic Web SOA WOA
SMTP

GoogleApp

g

MapReduce

XML








eScience -- -
­





e- -- , ,
­

(. , , , , , ,





eScience -- LHC: 50+ , 200+ ,
­ ­ ­



-- Grid (Open Grid) -- VO (Virtual Observatory) --





«» :
­

, ...



: !
«Early Science»



/:
­

,


! VLDB -> XLDB
Very Large Extremely Large XXX Tb XXX Pb


!





­ ­ ­ ­

()



«Sensor-centric» science ! --





- (1570-1601) ~ 500Kb SDSS -- 2007 3 Tb () -- 15 Tb LSST --
­ ­ ­ ­







8.4 , 3.2 Gpx CCD 49 . , 2.8 30 Tb/night, 100 Tflop 10 : 60 Pb raw data, -- 30 Pb 15 Pb , 100K CPU 200 ~ 40



LHC -- Large Hadron Collider
­ ­



: (, SN 1987A) - ,
!

( )








?


«» (raw) -- (Nikon's raw 12/14 bit). ! (cooking) «» -- (Capture NX) , .
­



:

--

-c

cd

­
­

:
, CCD LHC Raw converter: Capture NX, NX2


?


-
­

-- (, ,...) -- % rdbms -- (, , ) . Jpg (8-bit) www.flickr.com

­ ­ ­ ­


?


, , , , , (, OS, software) (design, ) , !!!









!


Climategate !
http://wattsupwiththat.com/2009/12/08/the-smoking-gun-at-darwin-zero/





(purl) ?
­ ­ ­ ­

, , , Oops! This link appears to be broken. « » ....?page=237



?
­ ­ ­





?
­

«» ! , - .



?
­



(data provenance, lineage)
­ ­

, ?


?


, , , , , (, OS, software) (design, ) ,








?



-- CPU, hostname - , provenance

p://wattsupwiththat.files.wordpress.com/2009/12/darwin_zero8.png" ttsupwiththat.files.wordpress.com/2009/12/darwin_zero8.png">









, WORM. . - , (error bar) , ,









- «sensor-centric science»






SN 2008fv NGC 3147 -- 2d ( , )



'' RDBMS , !



I 0 0 0 ... 1 ... 3 3 2 ... 0 1 2 ... J V 0.02 0.01 0.002 ... 0.5 ... 0.02 ! , .

SELECT A1.I, A1.J, AVG(A2.V) FROM Observtion A1, Observation A2 WHERE A2.I BETWEEN A1.I ­ 1 AND A1.I + 1 AND A2.J BETWEEN A1.J ­ 1 AND A1.J + 1 GROUP BY A1.I, A1.J;





RDBMS <- system R
­ ­ ­

1 $$$$ (mainframe) , «» , RAM, : -




­ ­ ­





RDBMS
­ ­ ­ ­ ­

(, GiST,GIN PostgreSQL) (shared-nothing, middleware) Cloud Computing -- PostgreSQL Plus + Elastra -> Amazon WS FPGA OLAP
­

GreenPlum, AsterData -- MapReduce


Yahoo Everest


!
­ ­ ­

Yahoo Everest (2008) 10Pb 2010 PostgreSQL +


() .......





- Hadoop





RDBMS , (ACID A . . D) ,
­



(StreamDB) -- !



XML -- NoSQL (~40 !) - (key,value), no join. ACID->BASE




NoSQL databases (wikipedia)
Document store Lotus Notes CouchDB MongoDB Apache Jackrabbit Colayer XML databases o MarkLogic Server o eXist Graph * Neo4j * AllegroGraph Tabular * BigTable * Mnesia * Hbase * Hypertable * * * * * * Key/value store on disk * * * * * * * Tuple space Memcachedb Redis SimpleDB flare Tokyo Cabinet BigTable Eventuallyconsistent keyvalue store * Dynamo * Cassandra * Project Voldemort Ordered key-value store * N MD B * Luxio * Memcachedb * Berkeley DB Object database * Db4o * InterSystems CachÈ * Objectivity/DB * ZODB

Key/value cache in RAM * memcached * Velocity * Redis





- (-store, Vertica, BigTable, Cassandra....)
­ ­ ­ ­ ­ ­ ­

, . , OLAP «» -- undo (*), redo ( RDBMS Btree )





-
­ ­ ­ ­ ­ ­

GRID Shared-nothing , TimeTravel -- , , 2PC





, x (xPb)
­

BaBar, LHC, LCLS, PanSTARRS Ebay(xPb, Terradata, GreenPlum), WalMart (xPb, Oracle), SDSS (xTb MS SQL), Genome ATT, Google, Yahoo, Amazon, Facebook




­



Home-grown (xPb)
­



!






­ ­ ­

Shared-nothing parallel database c Aster Data, Vertica, ParAccel, Greenplum, Neteza, Teradata




­




RDBMS
­ ­ ­ ­

+ , + - overhead (ACID) - (middleware) + native + -



(key, value)
­ ­ ­




:
­ ­ ­

«1st class citizen» Provenance


XLDB


XLDB -- - , , , + (MIT, Yahoo, Microsoft, IBM, BEA) + EBay
­

3 : 2007,2008,2009



«» - , ,


SciDB.org


SciDB - ! LSST -- --

ages/faculty/stonebraker.jpg" alt="MIT Adjunct Professor Michael Stonebraker" height="400" hspace="10"

­ ­ ­

Ingres,Postgres, Illustra,StreamDB, Vertica, VoltDB


!


Data cleaning (cooking) Feature extraction Data mining Data sharing


Scidb


, open-source , OLTP, ACID BASE -- , . , ,










Scidb
(R, Matlab, IDL), (C++, Python)


No overwrite storage, (named versions), - ,