Common Trending and QC tools:
Documentation
|
| tqs
= Trending and Quality Control System |
Python Update for DFO (python 2.6.2 + newest modules)
The current installation of python and its associated modules on our DFO machines is extremely old (i.e. python 2.4.2 was released in Sept. 2005). Since that time python has undergone a number of significant changes. More importantly, the modules used by the DFO installation are extremely out-dated. This means that many new and useful features are unavailable, and that those that do run are far from optimal in terms of processing time. This was the motivation for installing python version 2.6.2 (April 2009) and the newest versions of the modules.
However, in order to support python routines historical to DFO (i.e. qc1Ingest, cdbIngest, cdbQuery, dfscheck, etc.) requires that we maintain the old modules and python intrinsic to the DFO installation (python 2.4 with numarry). However, since we also want to make use of the large number of improvements in speed and features (numpy, scipy, matplotlib, etc.) that have been created since python 2.4, we need to have these two versions coexist. Therefore, the new python will be installed in a separate directory, and cleverly addressed within our new routines.
Installation:
The new python executable lives in: /qcdp/bin/python
The new modules are located at: /qcdp/lib/python2.6/site-packages
Making it work:
1. First modify the PYTHONPATH in your .qcrc file to contain the following:
export PATH=/opsw/util/python/bin:$PATH
export PYTHONPATH=${PYTHONPATH}:/opsw/packages:/opsw/util/python2.4.2.1/lib/python2.4/site-packages
export PYTHONPATH=${PYTHONPATH}:/qcdp/lib/python2.6/site-packages
This works because the first python path points to the old installation. If you are running a
python script that requires the new installation, it will first look in the old path, not find it,
and then look at the new site packages. This allows the old python routines to work along side the
new ones.
2. Each of your python scripts that require the new installation should begin with:
#!/qcdp/bin/python
Python scripts using the old DFO installation will begin with:
#!/usr/bin/env python or,
#!/scisoft/bin/python
3. Here is a summary of potential changes that may exist in your python routines:
Point of Change |
Old DFO Python Installation (2.4) |
New Python Installation (2.6.2) |
top line of your python script |
#!/scisoft/bin/python or #!/usr/bin/env python |
#!/qcdp/bin/python |
| modules |
import numarray |
import numpy (numarray is no longer supported) |
| array functions 1 (for example:) |
x = zeros(N, Float32)
x = zeros(N, Int) |
x = zeros(N, float)
x = zeros(N, int) |
| array functions 2 (for example:) |
min_x = min(x)
max_x = max(x)
sum_x = sum(x, axis=1)
etc.
|
min_x = x.min()
max_x = x.max()
sum_x = x.sum()
etc.
|
| array functions 3 (for example:) |
semilogy(x, where(hist>0, hist, 1), linestyle='steps', linewidth=1, color='blue')
In the old system, any use of a logarithmic function required that you first filter out any negative values.
|
semilogy(x, hist, linestyle='steps', linewidth=1, color='blue')
In the new system, negative values are automatically filtered
out of the input . . . nice! |
| array methods (for example:) |
x.getflat()
x.stddev()
x.nelements() |
x.ravel()
x.std()
x.size |
| matplotlib/pylab color definitions: character strings instead of floats |
plot(..., color=0.00) |
plot(..., color='0.00') |
| scipy optimize package must be imported |
import scipy |
import scipy
import scipy.optimize |
The new modules that have been installed include:
python 2.6.2
scipy
numpy
matplotlib
asciidata
dateutil
matplotlib
pyfits
pylab
Improvements to Expect when Using the New Python Installation:
Using the new DFO python 2.6.2 installation give you the most up-to-date version of python and, more importantly, its modules. As a result, there are now a very large number of extremely useful additions and debugging that have vastly improved the standard modules. This is particularly true forr matplotlib, numpy, scipy, and pylab.
Finally, when running your scripts using the new installation, you will find a huge increase in performance. This is mainly due to inprovements within numpy (versus the older numarray), but are also due to improvents in matplotlib (in particular, a speed enhancement in creating hardcopies of the QC plots) and scipy. On average, my QC python routines now run a remarkable six times faster!
Here is the proof. Below are the summaries of from the AB Monitor for 2009-11-08. The first figure shows the execution times for the old python installation, while the second figure shows the execution times based on scripts running the python 2.6.2 installation with the new modules. Summing up the total QC processing times, the latter is 6.1 times faster.
 |
Total times: 62.5 + 160.8 minutes
 |
Total times: 77.0 + 26.2 minutes
|
Last update:
December 18, 2009
|
make
printable ![[ used databases ]](/%7Eqc/dfos/database.png "[ used databases ]"){kind=small}
databases ![[ used dfos tools ]](/%7Eqc/dfos/tool_icon.gif "[ used dfos tools ]"){kind=icon}
dfos tools ![[ output used by ]](/images/text.gif "[ output used by ]"){kind=small}
output used by![[ upload/download ]](/%7Eqc/dfos/download-page-blue.gif "[ upload/download ]"){kind=small}
upload/download