|
|
2008 Summer School |
VO Client side Integration: IDL (C. Miller 09/04/08)
http://www.ctio.noao.edu/~chrism/VOlib
In this lesson, we discuss how
astronomers can integrate VO services into their day-to-day research
activities using IDL.
NOTE: GDL (an
open-source version of IDL) does not support many of the features
discussed here.
Contents:
VOTable
Reader
readvot.pro
Effective
Translation of VOTable Data to other Data types
Other Techniques for
Working with VOTables
Utilizing
REST-based Services
Cone
Searches and Catalogs
SIAP
Queries and Images
Utilizing
SOAP-based Services
Registry
Calls
Sesame
Name Resolver
SkyPortal
Calls
Reading VOTables
The most common type of data output you will find in VO services is the VOTable format. It is a requirement of Cone Search and SIAP query returns. Similarly, WESIX returns only a VOTable and SkyPortal can return x,y,z. Thus, from whatever research environment you use, the ability to read VOTables is very important. In most cases, XML/VOTables can be re-formatted into a more "comfortable" format (e.g., ascii, FITs, etc). But the importance of having a near seamless connection between VOTable outputs and variables within your research environment cannot be over-emphasized. A fairly inefficient way to do one's research is to be continually exiting a research environment (e.g., Python, IDL, SM) in order to convert VOTables. A much preferred technique is to work with a converter or parser within your environment.
VOlib
contains readvot.pro, a DOM-based VOTable reader:
READVOT.PRO
Give it a try:
IDL> readvot, 'nvoss2008/idl/VOlib_0.3/data/xmm1_votable.xml',str 3 tables found. Table number 1 was read in. You can specify another with table = x. IDL> help, /str,str ** Structure <8347804>, 15 tags, length=92, data length=92, refs=1: CREATED STRING '2005-09-02' UNIQUE_ID LONG 38426 NAME STRING Array[1] RA DOUBLE 34.252984 DEC DOUBLE -5.0352980 TIME DOUBLE 51756.944 EP_FLUX FLOAT 1.54315e-14 EP_FLUX_ERROR FLOAT 4.09244e-15 PN_FLUX FLOAT 1.54315e-14 PN_FLUX_ERROR FLOAT 4.09244e-15 M1_FLUX FLOAT -999.900 M1_FLUX_ERROR FLOAT -999.900 M2_FLUX FLOAT -999.900 M2_FLUX_ERROR FLOAT -999.900 SEARCH_OFFSET DOUBLE 2.1590000 IDL> readvot, 'nvoss2008/idl/VOlib_0.3/xmm1_votable.xml',str, table=2 IDL> help, /str,str ** Structure <82adc3c>, 12 tags, length=108, data length=108, refs=1: CREATED STRING '2005-09-02' UNIQUE_ID LONG 191 NAME STRING Array[1] RA DOUBLE 34.100000 DEC DOUBLE -5.0000000 PROPOSAL_TYPE STRING Array[1] PRIORITY STRING Array[1] PNO LONG 11237 EXPOSURE LONG 50000 PI_LNAME STRING Array[1] PI_FNAME STRING Array[1] SEARCH_OFFSET DOUBLE 9.5630000
Using REST-based Webservices
Cone Searches and SIAP queries
are REST web-services (i.e., a POST) and their results are simple
XML files. These are easily obtained through spawning a "wget"
or via port streaming (?). For both of these web-services, the
required inputs are POSITION, SIZE and URL of the cone search or
SIAP query.
Points to note: spawning a wget require write
privileges in some working directory to save the XML file and then a
VOTable parser to read it in. Connecting to a port requires:
Cone Searches: CONECALL.PRO
Fill a structure with the
data returned from some Cone Search call:
IDL> conecall, 202.8, -1.72, 0.1, str=str, $
url = "http://heasarc.gsfc.nasa.gov/cgi-bin/vo/cone/coneGet.pl?table=xmmssc&"
IDL> help, /str,str
** Structure <1de3514>, 15 tags, length=124, data length=124, refs=1:
CREATED STRING '2008-09-04'
UNIQUE_ID LONG 188432
NAME STRING '2XMM J133110.8-014345'
RA DOUBLE 202.79512
DEC DOUBLE -1.7292320
TIME DOUBLE 52119.536
EP_8_FLUX DOUBLE 2.6054200e-12
EP_8_FLUX_ERROR DOUBLE 1.6796500e-13
PN_8_FLUX DOUBLE 2.3502500e-12
PN_8_FLUX_ERROR DOUBLE 2.3507900e-13
M1_8_FLUX DOUBLE 2.8773600e-12
M1_8_FLUX_ERROR DOUBLE 3.4881100e-13
M2_8_FLUX DOUBLE 2.8663400e-12
M2_8_FLUX_ERROR DOUBLE 3.3093600e-13
SEARCH_OFFSET DOUBLE 0.62700000
SIAP Queries
Once the SIAP call is
made, the URL is provided. the procedure SIAPCALL will automatically
download the image for you (whether a fits, gzipped fits, jpg, or
gif). WEBGET is part of the ASTROLIB libraries and works only on
FITs images (or text files), but not XML.
NOTE: MacOSX comes
standard with WGET v1.6. SIAPCALL needs version
WGET 1.8+ to work properly.
IDL> siapcall, str=str, 217.5,35.0, 0.05, url="http://archive.noao.edu/nvo/sim/voquery.php", /metadata
IDL> help, /str,str
** Structure <2157c04>, 20 tags, length=240, data length=240, refs=1:
CREATED STRING '2008-09-04'
OBJECT STRING 'NDWFS J1428+3456 Bootes R-band'
SURVEY STRING 'NOAO Deep Wide Field'
INSTRUMENT STRING 'NOAO.NDWFSJ1428P3456_R_03_SIGM'
MJDOBS DOUBLE 0.0000000
RA DOUBLE 217.21600
DEC DOUBLE 34.957700
NAXES LONG 2
NAXIS LONG64 Array[2]
SCALE DOUBLE Array[2]
FORMAT STRING 'image/fits'
ACCESSREFERENCE STRING 'http://archive.noao.edu/nvo/sim/cutout.php?recid=NDWFSJ1428p3456_R_03_sigm&ra=217.5&dec=35&rawidth=0.02&decwidth=0.02&prefix=NDWFSJ1428p3456_R_03_sigma'
REFERENCEFRAME STRING 'ICRS'
EQUINOX DOUBLE 2000.0000
CPROJECTION DOUBLE Array[1]
CREFPIXEL DOUBLE Array[2]
CREFVALUE DOUBLE Array[2]
CDMATRIX DOUBLE Array[4]
FILESIZE LONG 158400
PIXFLAGS STRING 'C'
IDL> a = webget(str[0].accessreference)
IDL> print, a.imageheader
IDL> atv, a.image
or
IDL> siapcall, str=str2, 217.5,35.0, 0.02, url="http://archive.noao.edu/nvo/sim/voquery.php", root='ndwfs'
IDL> image = readfits('ndwfs_image0.fits',0,hdr)
IDL> hprint, hdr
IDL> atv, imageUsing SOAP-based Webservices
Examples of current SOAP-based VO services include the JHU NVO Registry, SkyPortal, and WESIX. For legacy software that has a Java-bridge (i.e., the ability to make calls to external Java classes), the easiest way to go is to use the Service WSDL to generate the Java stubs (wsdl2java) and classes (javac) locally, and then call those classes from within your legacy software. Different legacy software have different ways of "importing" external classes. It is extremely helpful to have a minimally useful Service provided client. Additionally, some thought into how the results will be useful to the user is required. In most cases, I have found that filling a structure with all of the VOTable (XML returned file) does the trick.
Registry Calls
So how do I find the Cone
Search servers and SIAP servers (etc) to run CONECALL and SIAPCALL?
Use CALL_REGISTRY.
SIAP queries, Cone Searches, Open Sky Query
calls, all require service (or resource) discovery. This is
typically done through a Registry call.
How might you call
it?
Click Here
How does it work? Use the JAVA
classes RegistryLocator.class and the regService methods
(getResgitrySoap, queryRegistry, etc).
The QUERY is made to be as
general as possible.
regService = OBJ_NEW('IDLJavaObject$ORG_US_VO_WWW_REGISTRY_LOCATOR', 'org.us_vo.www.RegistryLocator')
regInterface = regService->getRegistrySoap()
query = "Title like '%" + keyword + "%' or " + $
"ShortName like '%" + keyword + "%' or " + $
"Subject like '%" + keyword + "%' or " + $
"Type like '%" + keyword + "%' or " + $
"Description like '%" + keyword + "%' or " + $
"ServiceType like '%" + keyword + "%' or " + $
"Identifier like '%" + keyword + "%'"CONEs, SIAPs, and SKYNODEs are treated "special".
IF keyword_set(cone) THEN query = "ServiceType like '%CONE%' and (" + query + ")"
IF keyword_set(skynode) THEN query = "ServiceType like '%SKYNODE%' and (" + query + ")"
IF keyword_set(siap) THEN query = "(ServiceType like '%SIAP%' or ServiceType like '%SIAP;%ARCHIVE%') and (" + query + ")"
callit = regInterface->queryRegistry(query)
results = callit->getSimpleResource()Data is in the object "results", which is a SimpleResource. Create a structure and fill it:
str = create_struct('Title', ' ', 'URL', ' ', 'Type', ' ', 'ShortName', ' ', $
'ID', ' ', 'Desc', ' ', 'ServiceType', ' ', 'Coverage', ' ', $
'Subjects', ' ')
str = replicate(str, n_elements(results))
FOR I = 0, n_elements(results) -1 DO BEGIN
str[I].Title = results[I]->getTitle()
str[I].URL = results[I]->getServiceURL()
str[I].Type = results[I]->getType()
str[I].ShortName = results[I]->getShortName()
str[I].ID = results[I]->getIdentifier()
str[I].Desc = results[I]->getDescription()
str[I].ServiceType = results[I]->getServiceType()
str[I].Coverage = results[I]->getCoverageSpatial()
Let's give it a
try:
IDL> call_registry, 'ROSAT',/SIAP, str=str
IDL> help, /str,str
** Structure <14a5116c>, 9 tags, length=108, data length=108, refs=1:
TITLE STRING '
ROSAT All Sky Survey
'
URL STRING '
http://skyview.gsfc.nasa.gov/cgi-bin/vo/sia.pl?survey=rass-cnt&
'
TYPE STRING ' Archive '
SHORTNAME STRING '
ROSAT/RASS
'
ID STRING 'ivo://nasa.heasarc/skyview/rass'
DESC STRING '
The ROSAT All-Sky X-ray Survey was obtained during 1990/1991 using the ROSAT Position Sens'...
SERVICETYPE STRING 'SIAP'
COVERAGE STRING '<region xsi:type="AllSky" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.ivo'...
SUBJECTS STRING 'surveys'
IDL> call_registry, 'XMM',/CONE, str=str
IDL> print, n_elements(str)
36
IDL> help, /str,str[10]
** Structure <1daa134>, 9 tags, length=108, data length=108, refs=2:
TITLE STRING 'XMM-Newton Bright Serendipitous Survey'
URL STRING '
http://heasarc.gsfc.nasa.gov/cgi-bin/vo/cone/coneGet.pl?table=xmmbss&
'
TYPE STRING 'Catalog'
SHORTNAME STRING 'XMM BSS'
ID STRING 'ivo://nasa.heasarc/xmmbss'
DESC STRING '
We present here the XMM-Newton Bright Serendipitous Survey,
composed of two flux-limited samples: the XMM-Newton Bright Source
Sample (BSS, hereafter) and t'...
SERVICETYPE STRING 'CONE'
COVERAGE STRING 'All-sky<regionOfRegard xmlns="http://www.ivoa.net/xml/VODataService/v0.5">0.0166666666666666667</regionOfRegard>'
SUBJECTS STRING 'Survey Source'
IDL> print, str[10].url
http://heasarc.gsfc.nasa.gov/cgi-bin/vo/cone/coneGet.pl?table=xmmssc&Sesame Name Resolver
Sesame is a name resolver for astronomy. The user specifies the name of an object, and Sesame responds with coordinates.
The call to the service is straightforward:
result = addr ->sesame(name, "pi", 1, "N")
In the above call, name specifies the object name being queried
(e.g., "m31"), while "pi" tells the service to
respond in plain text.
1 signifies to return all ALIASes and "N"
meands search NED. These options are futher explained at the Sesame
website.
An example call looks like this:
IDL> resolve_name, "m31", str=str % Compiled module: RESOLVE_NAME. % Compiled module: STRSPLIT. % Compiled module: APPEND_ARR. % Compiled module: CREATE_STRUCT. % Compiled module: REPCHR. % Compiled module: TEMP_M31X. IDL> help, /str, str ** Structure M31, 31 tags, length=356, data length=356: TYPE STRING 'LINER' RA DOUBLE 10.684625 DEC DOUBLE 41.269278 RA_ERR DOUBLE 10800.000 DEC_ERR DOUBLE 10800.000 ALIAS0 STRING '0 2C 56' ALIAS1 STRING '2C 56' ALIAS2 STRING 'DA 21' ALIAS3 STRING '[DGW65] 4' ALIAS4 STRING 'GIN 801' ALIAS5 STRING 'IRAS F00400+4059' ALIAS6 STRING 'IRAS 00400+4059' ALIAS7 STRING 'IRC +40013' ALIAS8 STRING 'K79 1C' ALIAS9 STRING 'LEDA 2557' ALIAS10 STRING 'M 31' ALIAS11 STRING '[M98c] 004000.1+405943' ALIAS12 STRING 'MCG+07-02-016' ALIAS13 STRING 'NAME ANDROMEDA NEBULA' ALIAS14 STRING 'NAME ANDROMEDA' ALIAS15 STRING 'NAME ANDROMEDA Galaxy' ALIAS16 STRING 'NGC 224' ALIAS17 STRING 'RAFGL 104' ALIAS18 STRING 'UGC 454' ALIAS19 STRING '[VV2000c] J004244.3+411610' ALIAS20 STRING '[VV2003c] J004244.3+411610' ALIAS21 STRING '[VV2006] J004244.3+411610' ALIAS22 STRING '[VV98c] J004245.1+411622' ALIAS23 STRING 'XSS J00425+4102' ALIAS24 STRING 'Z 535 - 17' ALIAS25 STRING 'Z 0040.0+4100
SkyPortal Calls
The Open Sky Query (OSQ--via
the SkyPortal) allows for SQL-like queries of more than 20
astronomical catalogs. OSQ also will cross-match multiple catalogs
for you.
SKYPORTAL client needs a position, a search
radius (in arcminutes) and possibly a SQL query:
qry = " SELECT o.objId, o.ra,o.dec, o.type,t.objId,t.j_m,o.z " + $
" FROM SDSSDR2:PhotoPrimary o, " + $
" TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<" + strtrim(string(chisqArr),2) + " " + " AND o.type = 3 " + $
" AND Region('Circle J2000 " + strtrim(string(raArr,format='(f10.3)'),2) + " " + strtrim(string(decArr, format='(f10.3)'),2) + $
" " + strtrim(string(srArr, format='(i2)'),2) + "') "The SKYCLIENT then parses the VOTable internal and places the data into a structure.
IDL> skyclient, 217.5,35.0,10 ,str = dat; Defaults to whatever is uncommented in the MAKE_ADQL.PRO
% Compiled module: MAKE_ADQL.
SELECT o.objId, o.ra,o.dec, o.type,t.objId,t.j_m,o.z FROM SDSSDR2:PhotoPrimary o,
TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<2.50000 AND o.type = 3 AND Region('Circle J2000 180.000 0.000 1')
IDL> help, /str, str
** Structure <520790>, 8 tags, length=48, data length=48, refs=1:
SDSSDR2_OBJID LONG64 588848899912695818** Structure <1ddbb54>, 8 tags, length=48, data length=48, refs=1:
SDSSDR2_OBJID LONG64 588848899912695818
SDSSDR2_RA DOUBLE 179.98346
SDSSDR2_DEC DOUBLE -0.0016030098
SDSSDR2_TYPE LONG 3
TWOMASS_OBJID LONG 1016752918
TWOMASS_J_M FLOAT 16.7000
SDSSDR2_Z FLOAT 17.6322
CHISQ DOUBLE 0.088211060
IDL> image = mrdfits('ndwfs_image0.fits',0,hdr)
IDL> adxy, hdr, dat.sdss_ra,dat.sdss_dec,x,y
IDL> atv, image, head=hdr
IDL> atvplot, x,y,color='red' ,psym= 2