PyTangoArchiving Recipes

Intended audience: developers, Programming language: python

by Sergi Rubio

PyTangoArchiving is the python API for Tango Archiving.

This package allows to:

• Integrate Hdb and Snap archiving with other python/PyTango tools.

• Start/Stop Archiving devices in the appropiated order.

• Increase the capabilities of configuration and diagnostic.

• Import/Export .csv and .xml files between the archiving and the database.

Don’t edit this wiki directly, the source for this documentation is available at PyTangoArchiving UserGuide

Installing PyTangoArchiving:

Repository is available on sourceforge:

$ svn co https://svn.code.sf.net/p/tango-cs/code/archiving/tool/PyTangoArchiving/trunk

Dependencies:

1. Tango Java Archiving, ArchivingRoot from sourceforge,

2. PyTango

3. python-mysql

4. Taurus (optional)

5. fandango:

$ svn co https://svn.code.sf.net/p/tango-cs/code/share/fandango/trunk/fandango fandango

Setup:

• Follow Tango Java Archiving installation document to setup Java Archivers and Extractors.

• Some of the most common installation issues are solved in several topics in  Tango forums (search for Tdb/Hdb/Snap Archivers):

  • tango-archiving,

  • installation-of-hdb-and-tdb-in-linux.

• Install PyTango and MySQL-python using their own setup.py scripts.

• fandango, and PyTangoArchiving parent folders must be added to your PYTHONPATH environment variable.

• Although Java Extractors may be used, it is recommended to configure direct MySQL access for PyTangoArchiving

Accessing MySQL:

Although not needed, I recommend you to create a new MySQL user for data querying:

$ mysql -u hdbmanager -p hdb

$ GRANT USAGE ON hdb.* TO 'user'@'localhost' IDENTIFIED BY '**********';
$ GRANT USAGE ON hdb.* TO 'user'@'%' IDENTIFIED BY '**********';
$ GRANT SELECT ON hdb.* TO 'user'@'localhost';
$ GRANT SELECT ON hdb.* TO 'user'@'%';

$ mysql -u tdbmanager -p tdb

$ GRANT USAGE ON tdb.* TO 'user'@'localhost' IDENTIFIED BY '**********';
$ GRANT USAGE ON tdb.* TO 'user'@'%' IDENTIFIED BY '**********';
$ GRANT SELECT ON tdb.* TO 'user'@'localhost';
$ GRANT SELECT ON tdb.* TO 'user'@'%';

Check in a python shell that your able to access the database:

import PyTangoArchiving

PyTangoArchiving.Reader(db='hdb',config='user:password@hostname')

Then configure the Hdb/Tdb Extractor class properties to use this user/password for querying:

import PyTango

PyTango.Database().put_class_property('HdbExtractor',{'DbConfig':'user:password@hostname'})

PyTango.Database().put_class_property('TdbExtractor',{'DbConfig':'user:password@hostname'})

You can test now access from a Reader (see recipes below) object or from a taurustrend/ArchivingBrowser UI (Taurus required):

python PyTangoArchiving/widget/ArchivingBrowser.py

Download

Download PyTangoArchiving from sourceforge:

svn co https://svn.code.sf.net/p/tango-cs/code/archiving/tool/PyTangoArchiving/trunk

Submodules

• api,

  • getting servers/devices/instances implied in the archiving system and allowing

• historic,

  • configuration and reading of historic data

• snap,

  • configuration and reading of snapshot data,

• xml,

  • conversion between xml and csv files

• scripts,

  • configuration scripts

• reader,

  • providing the useful Reader and ReaderProcess objects to retrieve archived data

General usage

In all these examples you can use hdb or tdb just replacing one by the other

Get archived values for an attribute

The reader object provides a fast access to archived values

In [9]: import PyTangoArchiving
In [10]: rd = PyTangoArchiving.Reader('hdb')
In [11]: rd.get_attribute_values('expchan/eh_emet02_ctrl/3/value','2013-03-20 10:00','2013-03-20 11:00')
Out[11]:
[(1363770788.0, 5.79643e-14),
 (1363770848.0, 5.72968e-14),
 (1363770908.0, 5.7621e-14),
 (1363770968.0, 6.46782e-14),
 ...

Start/Stop/Check attributes

You must create an Archiving api object and pass to it the list of attributes with its archiving config:

import PyTangoArchiving
hdb = PyTangoArchiving.ArchivingAPI('hdb')
attrs = ['expchan/eh_emet03_ctrl/3/value','expchan/eh_emet03_ctrl/4/value']

#Archive every 15 seconds if change> +/-1.0, else every 300 seconds
modes = {'MODE_A': [15000.0, 1.0, 1.0], 'MODE_P': [300000.0]}

#If you omit the modes argument then archiving will be every 60s
hdb.start_archiving(attrs, modes)

hdb.load_last_values(attrs)
{'expchan/eh_emet02_ctrl/3/value': [[datetime.datetime(2013, 3, 20, 11, 38, 9),
    7.27081e-14]],
    'expchan/eh_emet02_ctrl/4/value': [[datetime.datetime(2013, 3, 20, 11, 39),
    -3.78655e-08]]
}

hdb.stop_archiving(attrs)

Loading a .CSV file into Archiving

The .csv file must have a shape like this one (any row starting with ‘#’ is ignored):

Host  Device  Attribute   Type    ArchivingMode   Periode >15  MinRange    MaxRange

#This header lines are mandatory!!!
@LABEL  Unique ID
@AUTHOR Who?
@DATE   When?
@DESCRIPTION    What?

#host   domain/family/member    attribute   HDB/TDB/STOP    periodic/absolute/relative

cdi0404 LI/DI/BPM-ACQ-01    @DEFAULT        periodic    300
                            ADCChannelAPeak HDB absolute    15  1   1
                                            TDB absolute    5   1   1
                            ADCChannelBPeak HDB absolute    15  1   1
                                            TDB absolute    5   1   1
                            ADCChannelCPeak HDB absolute    15  1   1
                                            TDB absolute    5   1   1
                            ADCChannelDPeak HDB absolute    15  1   1
                                            TDB absolute    5   1   1

The command to insert it is:

import PyTangoArchiving
PyTangoArchiving.LoadArchivingConfiguration('/...fbecheri_20130319.csv','hdb',launch=True)

There are some arguments to modify Loading behavior.

launch:

if not explicitly True then archiving is not triggered, it just verifies that format of the file is Ok and attributes are available

force:

if False the loading will stop at first error, if True then it tries all attributes even if some failed

overwrite:

if False attributes already archived will be skipped.

Checking the status of the archiving

hdb = PyTangoArchiving.ArchivingAPI('hdb')
hdb.load_last_values()
filter = "/" #Put here whatever you want to filter the attribute names
lates = [a for a in hdb if filter in a and hdb[a].archiver and hdb[a].modes.get('MODE_P') and hdb[a].last_date<(time.time()-(3600+1e-3*hdb[a].modes['MODE_P'][0]))]

#Get the list of attributes that cannot be read from the control system (ask system responsibles)
unav = [a for a in lates if not fandango.device.check_attribute(a,timeout=6*3600)]
#Get the list of attributes that are not being archived
lates = sorted(l for l in lates if l not in unav)
#Get the list of archivers not running properly
bad_archs = [a for a,v in hdb.check_archivers().items() if not v]

#Restarting the archivers/attributes that failed
bads = [l for l in lates if hdb[l] not in bad_archs]
astor = fandango.Astor()
astor.load_from_devs_list(bad_archs)
astor.restart_servers()
hdb.restart_archiving(bads)

Restart of the whole archiving system

admin@archiving:> archiving_service.py stop-all
...
admin@archiving:> archiving_service.py start-all
...
admin@archiving:> archiving_service.py status

#see archiving_service.py help for other usages

Using the Python API

Start/Stop of an small (<10) list of attributes

#Stopping ...
api.stop_archiving(['bo/va/dac/input','bo/va/dac/settings'])

#Starting with periodic=60s ; relative=15s if +/-1% change
api.start_archiving(['bo/va/dac/input','bo/va/dac/settings'],{'MODE_P':[60000],'MODE_R':[15000,1,1]})

#Restarting and keeping actual configuration

attr_name = 'bo/va/dac/input'
api.start_archiving([attr_name],api.attributes[attr_name].extractModeString())

Checking if a list of attributes is archived

In [16]: hdb = PyTangoArchiving.api('hdb')
In [17]: sorted([(a,hdb.load_last_values(a)) for a in hdb if a.startswith('bl04')])
Out[17]:
[('bl/va/elotech-01/output_1',
  [[datetime.datetime(2010, 7, 2, 15, 53), 6.0]]),
 ('bl/va/elotech-01/output_2',
  [[datetime.datetime(2010, 7, 2, 15, 53, 11), 0.0]]),
 ('bl/va/elotech-01/output_3',
  [[datetime.datetime(2010, 7, 2, 15, 53, 23), 14.0]]),
 ('bl/va/elotech-01/output_4',
  [[datetime.datetime(2010, 7, 2, 15, 52, 40), 20.0]]),
...

Getting information about attributes archived

Getting the total number of attributes:

import PyTangoArchiving
api = PyTangoArchiving.ArchivingAPI('hdb')
len(api.attributes) #All the attributes in history
len([a for a in api.attributes.values() if a.archiving_mode]) #Attributes configured

Getting the configuration of attribute(s):

#Getting as string
modes = api.attributes['rs/da/bpm-07/CompensateTune'].archiving_mode

#Getting it as a dict
api.attributes['sr/da/bpm-07/CompensateTune'].extractModeString()

#OR
PyTangoArchiving.utils.modes_to_dict(modes)

Getting the list of attributes not updated in the last hour:

failed = sorted(api.get_attribute_failed(3600).keys())

Getting values for an attribute:

import PyTangoArchiving,time

reader = PyTangoArchiving.Reader() #An HDB Reader object using HdbExtractors
#OR
reader = PyTangoArchiving.Reader(db='hdb',config='pim:pam@pum') #An HDB reader accessing to MySQL

attr = 'bo04/va/ipct-05/state'
dates = time.time()-5*24*3600,time.time() #5days
values = reader.get_attribute_values(attr,*dates) #it returns a list of (epoch,value) tuples

Exporting values from a list of attributes as a text (csv / ascii) file

from PyTangoArchiving import Reader
rd = Reader(db='hdb') #If HdbExtractor.DbConfig property is set one argument is enough
attrs = [
         'bl11-ncd/vc/eps-plc-01/pt100_1',
         'bl11-ncd/vc/eps-plc-01/pt100_2',
        ]

#If you ignore text argument you will get lists of values, if text=True then you get a tabulated file.
ascii_values = rd.get_attributes_values(attrs,
                      start_date='2010-10-22',stop_date='2010-10-23',
                      correlate=True,text=True)

print ascii_values

#Save it as .csv if you want ...
open('myfile.csv','w').write(ascii_values)

Filtering State changes for a device

import PyTangoArchiving as pta
rd = pta.Reader('hdb','...:...@...')
vals = rd.get_attribute_values('bo02/va/ipct-02/state','2010-05-01 00:00:00','2010-07-13 00:00:00')
bads = []
for i,v in enumerate(vals[1:]):
    if v[1]!=vals[i-1][1]:
        bads.append((v[0],vals[i-1][1],v[1]))
report = [(time.ctime(v[0]),str(PyTango.DevState.values[int(v[1])] if v[1] is not None else 'None'),str(PyTango.DevState.values[int(v[2])] if v[2] is not None else 'None')) for v in bads]

report =
[('Sat May  1 00:07:03 2010', 'UNKNOWN', 'ON'),
...

Getting a table with last values for all attributes of a same device

hours = 1
device = 'bo/va/ipct-05'
attrs = [a for a in reader.get_attributes() if a.lower().startswith(device)]
vars = dict([(attr,reader.get_attribute_values(attr,time.time()-hours*3600)) for attr in attrs])
table = [[time.ctime(t0)]+
         [([v for t,v in var if t<=t0] or [None])[-1] for attr,var in sorted(vars.items())]
        for t0,v0 in vars.values()[0]]
print('\n'.join(
      ['\t'.join(['date','time']+[k.lower().replace(device,'') for k in sorted(vars.keys())])]+
      ['\t'.join([str(s) for s in t]) for t in table]))

Using CSV files

Loading an HDB/TDB configuration file

Create dedicated archivers first

If you want to use this option it will require some RAM resources in the host machine (64MbRAM/250Attributes) and installing the ALBA-Archiving bliss package.

from PyTangoArchiving.files import DedicateArchiversFromConfiguration
DedicateArchiversFromConfiguration('LX_I_Archiving.csv','hdb',launch=True)

TDB Archiving works different as it shouldn’t be working on diskless machines, using instead a centralized host for all archiver devices.

DedicateArchiversFromConfiguration('LX_I_Archiving.csv','tdb',centralized='archiving01',launch=True)

Loading the .csv files

All the needed code to do it is:

import PyTangoArchiving

#With launch=False this function will do a full check of the attributes and print the results
PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','hdb',launch=False)

#With launch=True configuration will be recorded and archiving started
PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','hdb',launch=True)

#To force archiving of all not-failed attributes
PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','hdb',launch=True,force=True)

#Starting archiving in TDB mode (kept 5 days only)
PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','tdb',launch=True,force=True)

Note

You must take in account the following conditions:

• Names of attributes must match the NAME, not the LABEL! (that’s a common mistake)

• Devices providing the attributes must be running when you setup archiving.

• Regular expressions are NOT ALLOWED (I know previous releases allowed it, but never worked really well)

filtering a list of CSV configurations / attributes to load

You can use GetConfigFiles and filters/exclude to select a predefined list of attributes

import PyTangoArchiving as pta

filters = {'name':".*"}
exclude = {'name':"(s.*bpm.*)|(s10.*rf.*)|(s14.*rf.*)"}

#TDB
confs = pta.GetConfigFiles(mask='.*(RF|VC).*')
for target in confs:
    pta.LoadArchivingConfiguration(target,launch=True,force=True,overwrite=True,dedicated=False,schema='tdb',filters=filters,exclude=exclude)

#HDB
confs = pta.GetConfigFiles(mask='.*BO.*(RF|VC).*')
for target in confs:
    pta.LoadArchivingConfiguration(target,launch=True,force=True,overwrite=True,dedicated=True,schema='hdb',filters=filters,exclude=exclude)

Comparing a CSV file with the actual configuration

import PyTangoArchiving
api = PyTangoArchiving.ArchivingAPI('hdb')
config = PyTangoArchiving.ParseCSV('Archiving_RF_.csv')

for attr,conf in config.items():
    if attr not in api.attributes or not api.attributes[attr].archiving_mode:
        print '%s not archived!' % attr
    elif PyTangoArchiving.utils.modes_to_string(api.check_modes(conf['modes']))!=api.attributes[attr].archiving_mode:
        print '%s: %s != %s' %(attr,PyTangoArchiving.utils.modes_to_string(api.check_modes(conf['modes'])),api.attributes[attr].archiving_mode)

Checking and restarting a known system from a .csv

import PyTangoArchiving.files as ptaf
borf = '/data/Archiving/BO_20100603_v2.csv'
config = ptaf.ParseCSV(borf)
import PyTangoArchiving.utils as ptau
hdb = PyTangoArchiving.ArchivingAPI('hdb')

missing = [
    'bo/ra/fim-01/remotealarm',
    'bo/ra/fim-01/rfdet1',
    'bo/ra/fim-01/rfdet2',
    'bo/ra/fim-01/arcdet5',
    'bo/ra/fim-01/rfdet3',
    'bo/ra/fim-01/arcdet3',
    'bo/ra/fim-01/arcdet2',
    'bo/ra/fim-01/vacuum']

ptau.check_attribute('bo/ra/fim-01/remotealarm')
missing = 'bo/ra/fim-01/arcdet4|bo/ra/fim-01/remotealarm|bo/ra/fim-01/rfdet1|bo/ra/fim-01/rfdet2|bo/ra/fim-01/arcdet5|bo/ra/fim-01/rfdet3|bo/ra/fim-01/arcdet3|bo/ra/fim-01/arcdet2|bo/ra/fim-01/vacuum'

ptaf.LoadArchivingConfiguration(borf,filters={'name':missing},launch=True)
ptaf.LoadArchivingConfiguration(borf,filters={'name':'bo/ra/eps-plc.*'},stop=True,force=True)
ptaf.LoadArchivingConfiguration(borf,filters={'name':'bo/ra/eps-plc.*'},launch=True,force=True)

rfplc = ptaf.ParseCSV(borf,filters={'name':'bo/ra/eps-.*'})
stats = ptaf.CheckArchivingConfiguration(borf,period=300)