Документ взят из кэша поисковой машины. Адрес оригинального документа : http://acat02.sinp.msu.ru/presentations/kryukov/acat02-020625.pdf Дата изменения: Tue Jun 25 14:23:48 2002 Дата индексирования: Mon Oct 1 20:29:42 2012 Кодировка:

Implementation of remote job submission over GRID with IMPALA/BOSS CMS MC production tools*

A. U. A. S. M.

Edunov, JINR, Dubna, Gasparini, INFN, Padua University and INFN Padua Kryukov, SINP Moscow State University, Moscow, Lacaprara, INFN, Padua University and INFN Padua Verlato, INFN, Padua University and INFN Padua

ACAT-2002, 24-28 June

A.Kryukov, SINP MSU


Outline



Introduction



Computational model



IMPALA/BOSS overview



Integration of IMPALA/BOSS with GRID



DOLLY: The package to clone of CMS Production jobs



Conclusions and results
A.Kryukov, SINP MSU

ACAT-2002, 24-28 June


Introduction



CMS MC Production: SPRING-2002

Ў

Tools:

ў

IMPALA

ў

BOSS

Ў

Stages

ў

CMKIN

ў

CMSIM

ў

OOHit formatting

ў

Digitization A.Kryukov, SINP MSU

ACAT-2002, 24-28 June


Computational Model
FZ ORCA ooHITS

Pythia

NTPL

GEANT3

Data_set_1 ObjDB FZ Signal ObjDB ORCA Digi Pile_up A.Kryukov, SINP MSU ORCA ooHITS

Data_set_2

Pythia

NTPL

GEANT3

ObjDB

ROOT, NTPL

ORCA User analysis

ACAT-2002, 24-28 June


IMPALA/BOSS overview
Local Files to Process Objectivity DB Runs to Process

Input Parameters Input Parameters (Obtained with wget) (Stored in IMPALA)

Parameters Interface Input Source Discovery File Based Tracking

Declare Script Tracking Interface MySQL Based Tracking

Create Script

Submit Script

Pre Process

Run Job Script Batch System

Post Process A.Kryukov, SINP MSU

ACAT-2002, 24-28 June


IMPALA/BOSS integration with grid
Environment Jobs

Reference DB in CERN

UI

IMPALA

BOSS MySQL DB

Batch Manager CE
NFS

WN1 Wnn

WN2

Job Executer CMKIN A.Kryukov, SINP MSU

ACAT-2002, 24-28 June


IMPALA/BOSS integration with GRID (continue)
Environment Jobs

CERN RefDB

UI MySQL DB

Dolly

BOSS

Recource Broker Gate Keeper Batch Manager CE GRID
NFS

Job

WN1

WN2

Job Executer Wnn IMPALA CMKIN

ACAT-2002, 24-28 June

A.Kryukov, SINP MSU


DOLLY: The package to clone CMS production jobs



DOLLY_CE

Ў

Modified IMPALA

ў

DOLLY_CE.patch

Ў

DollyEnv.sh



DOLLY_UI

Ў

Modified GetXXX_fromRefDB.sh

Ў

dolly

Ў

.dollyrc

Ў

Dolly-service
A.Kryukov, SINP MSU

ACAT-2002, 24-28 June


DOLLY: The package to clone CMS production jobs (continue)



Three steps:

Ў

Installation:

ў

Just unpack: tar xzvf dolly-xxxxxx.tgz

Ў

Initialization working directory:

ў

Create working directory

ў

Modify of dolly-env and run it

ў

Run dolly-init command

Ў

Running
A.Kryukov, SINP MSU

ў

Dolly commands.

ACAT-2002, 24-28 June


DOLLY: The package to clone CMS production jobs (continue)



Usage (briefly)

dolly job_type [-debug]

ў

job_type is kin, sim, oohit. Oodigi in progress.

Dolly-service utilitie for reinitialized some date.

ў

dolly-service job_type [-clean] [-r run_number] [-j job_number] [-s set_name] [-o newowner_number]

Some additional utilities for retrieving data and monitoring jobs.

ACAT-2002, 24-28 June

A.Kryukov, SINP MSU


Conclusions and results



There was designed and development new scheme of integration of IMPALA/BOSS production tools with GRID

Ў

Minimal modification in IMPALA.

Ў

Standard BOSS (version 3.x).



The set of scripts (DOLLY) to clone the CMS production jobs on the UI side was developed.



Two steps of production (CMKIN and CMSIN) was tested completely.
A.Kryukov, SINP MSU



The oodigi formating step is under development

ACAT-2002, 24-28 June


Conclusions and results (continue)

ё

IMPALA problems

Ў

Single user application

ў

Require the write access to impala directories

ў

Use fixed names for temporally files (like /tmp/xxxx)

Ў

Two ways to pass initial data and parameters.

ў

Through the files (pythia cards, ...)

ў

Through global environment

ё

BOSS
A.Kryukov, SINP MSU

Ў

Direct access to MySQL DB in the world.

ACAT-2002, 24-28 June


JDL file
#

###########################################################

###########################################################

#

# ----- kin_2006.jdl ----#

#

###########################################################

###########################################################

Executable = "kin_2006.sh";

StdOutput = "kin_2006.out";

StdError = "kin_2006.err";

InputSandbox = {"kin_2006.sh","kin_2006.tgz"};

OutputSandbox = {"kin_2006.out","kin_2006.err","kin_2006_out.tgz"};

ACAT-2002, 24-28 June

A.Kryukov, SINP MSU


Script file

###########################################################

###########################################################

#

#

# ----- kin_2006.sh -----

#

#

###########################################################

###########################################################

tar xfz kin_2006.tgz

export defaultCMKINNumberOfEventsUI=10

export CMS_ProdData=CMS_ProdData

./DollyCMKINJob

tar cfz kin_2006_out.tgz *

ACAT-2002, 24-28 June

A.Kryukov, SINP MSU


Job file

#!/bin/sh

#Globals:

export DATA_SET_NAME=PD_MB_test

export REGIONAL_CENTER=PD_INFN_TEST

export DATA_SET_DIR=`pwd`/${CMS_ProdData}/tracking/$REGIONAL_CENTER/cmkin/$DATA_SET_NAME

export RUN_DATA_DIR=`pwd`/${CMS_ProdData}/rundata

export IMPALA_ROOT=/home/impalabo/cms/impala

export PROD_RESOURCES=/home/impalabo/cms/impala/scripts

export ROOT_OF_WORK_PATH=`pwd`/${CMS_ProdData}

export PATH=$PATH:$PROD_RESOURCES

if [ -z $defaultCMKINNumberOfEventsUI ]; then

export defaultCMKINNumberOfEventsUI=500

fi

export EDGUpdate=true

mkdir -p $DATA_SET_DIR

DeclareCMKINJobs.sh -v $DATA_SET_NAME 1

CreateCMKINJobs.sh -v $DATA_SET_NAME 1

RunJob.sh -v -j cmkin $DATA_SET_NAME 1