Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.eso.org/~qc/tqs/pyqc/python_2.6.2.html
Дата изменения: Fri Dec 18 13:22:23 2009
Дата индексирования: Thu Apr 8 11:01:03 2010
Кодировка:
Поисковые слова: п п п п п п п п п п п п п п п п п п
pyQC: Python 2.6.2

Common Trending and QC tools:
Documentation
tqs = Trending and Quality Control System
*make printable   new: see also:
 

A Radical Update of Python on the DFO machines. Python 2.6.2 + most up-to-date modules (December 2009)

vimos_stdimg.tar scripts for QC on VIMOS imaging standard star fields (July 2009)

qc_imadisp.py (May 2009, V1.2 with symbol overplotting July 2009)

release of V1.1 for qclib and plot modules (March 2009)

HAWK-I example scripts (March 2009)

repository of example and tutorial scripts (December 2009)

qc_detlin.py (February 2009)

- description of library QClib
- pyQC script structure
- step-by-step installation tutorial
- repository of example and tutorial scripts
[ used databases ] databases none
[ used dfos tools ] dfos tools processQC, writeQC, scoreQC
[ output used by ] output used by certifyProducts
[ upload/download ] upload/download none

Python Update for DFO (python 2.6.2 + newest modules)

The current installation of python and its associated modules on our DFO machines is extremely old (i.e. python 2.4.2 was released in Sept. 2005). Since that time python has undergone a number of significant changes. More importantly, the modules used by the DFO installation are extremely out-dated. This means that many new and useful features are unavailable, and that those that do run are far from optimal in terms of processing time. This was the motivation for installing python version 2.6.2 (April 2009) and the newest versions of the modules.

However, in order to support python routines historical to DFO (i.e. qc1Ingest, cdbIngest, cdbQuery, dfscheck, etc.) requires that we maintain the old modules and python intrinsic to the DFO installation (python 2.4 with numarry). However, since we also want to make use of the large number of improvements in speed and features (numpy, scipy, matplotlib, etc.) that have been created since python 2.4, we need to have these two versions coexist. Therefore, the new python will be installed in a separate directory, and cleverly addressed within our new routines.

 

Installation:

The new python executable lives in: /qcdp/bin/python

The new modules are located at: /qcdp/lib/python2.6/site-packages

Making it work:

1. First modify the PYTHONPATH in your .qcrc file to contain the following:

export PATH=/opsw/util/python/bin:$PATH
export PYTHONPATH=${PYTHONPATH}:/opsw/packages:/opsw/util/python2.4.2.1/lib/python2.4/site-packages
export PYTHONPATH=${PYTHONPATH}:/qcdp/lib/python2.6/site-packages

This works because the first python path points to the old installation. If you are running a
python script that requires the new installation, it will first look in the old path, not find it,
and then look at the new site packages. This allows the old python routines to work along side the
new ones.

2. Each of your python scripts that require the new installation should begin with:

#!/qcdp/bin/python

Python scripts using the old DFO installation will begin with:

#!/usr/bin/env python or,
#!/scisoft/bin/python

3. Here is a summary of potential changes that may exist in your python routines:

Point of Change

Old DFO Python Installation (2.4)

New Python Installation (2.6.2)

top line of your python script

 #!/scisoft/bin/python or #!/usr/bin/env python #!/qcdp/bin/python
modules import numarray import numpy (numarray is no longer supported)
array functions 1 (for example:) x = zeros(N, Float32)
x = zeros(N, Int)		 x = zeros(N, float)
x = zeros(N, int)
array functions 2 (for example:)

min_x = min(x)
max_x = max(x)
sum_x = sum(x, axis=1)
etc.

min_x = x.min()
max_x = x.max()
sum_x = x.sum()
etc.

array functions 3 (for example:)

semilogy(x, where(hist>0, hist, 1), linestyle='steps', linewidth=1, color='blue')


In the old system, any use of a logarithmic function required that you first filter out any negative values.

semilogy(x, hist, linestyle='steps', linewidth=1, color='blue')

 

In the new system, negative values are automatically filtered
out of the input . . . nice!
array methods (for example:)

x.getflat()
x.stddev()
x.nelements()

 x.ravel()
x.std()
x.size
matplotlib/pylab color definitions: character strings instead of floats plot(..., color=0.00) plot(..., color='0.00')
scipy optimize package must be imported import scipy import scipy
import scipy.optimize

 

The new modules that have been installed include:

python 2.6.2
scipy
numpy
matplotlib
asciidata
dateutil
matplotlib
pyfits
pylab

 

Improvements to Expect when Using the New Python Installation:

Using the new DFO python 2.6.2 installation give you the most up-to-date version of python and, more importantly, its modules. As a result, there are now a very large number of extremely useful additions and debugging that have vastly improved the standard modules. This is particularly true forr matplotlib, numpy, scipy, and pylab.

Finally, when running your scripts using the new installation, you will find a huge increase in performance. This is mainly due to inprovements within numpy (versus the older numarray), but are also due to improvents in matplotlib (in particular, a speed enhancement in creating hardcopies of the QC plots) and scipy. On average, my QC python routines now run a remarkable six times faster!

Here is the proof. Below are the summaries of from the AB Monitor for 2009-11-08. The first figure shows the execution times for the old python installation, while the second figure shows the execution times based on scripts running the python 2.6.2 installation with the new modules. Summing up the total QC processing times, the latter is 6.1 times faster.

 

2009-11-08 Processed with the old DFO python installation:

2d gauss fit Total times: 62.5 + 160.8 minutes

 

2009-11-08 Processed with the new DFO python installation:

2d gauss fit Total times: 77.0 + 26.2 minutes

 

 

 

Send comments to <mneeser@eso.org>
Last update: December 18, 2009