103.3. TAP image access via the ObsCore table

103.3. TAP image access via the ObsCore table¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: medium
LSST Science Pipelines version: r30.0.9
Last verified to run: 2026-07-20
Repository: github.com/lsst/tutorial-notebooks

Learning objective: How to use the ObsCore table and the Table Access Protocol (TAP) service to access image data.

LSST data products: deep_coadd

Packages: lsst.rsp, lsst.afw.display

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

TAP provides standardized access to catalog data for discovery, search, and retrieval. Full documentation for TAP is provided by the International Virtual Observatory Alliance (IVOA).

ADQL (Astronomy Data Query Language) is similar to SQL (Structured Query Langage). The documentation for ADQL includes more information about syntax and keywords.

Not all ADQL functionality is supported yet in the preview-era RSP, such as the INTERSECTS operator.

When to use the TAP service.

Use the TAP service to find, retrieve, and display whole images, based on their spatial, temporal, and spectral coverage, and their LSST image subtype. TAP is good for quick access to images' pixel and header data, for display and examination by eye. TAP can be used to obtain the access_url to pass to the image cutout service.

For science cases that involve source detection or image analysis with the LSST Science Pipelines, the Butler should instead be used to find and retrieve images.

Related tutorials: Other 100-level tutorials demonstrate how to access images via other services, such as SIA and the Butler, and how to use the image cutout service. The 100-level tutorials on image display have more on using Firefly. The 200-level tutorials on coadded and visit images have more on these data products (e.g., the pixel data and metadata).

1.1. DAL service errors¶

The following error message might be encountered.

DALServiceError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Typically this is fixed by rerunning the cell with the TAP service fetch statement, or by clearing the notebook, restarting the kernel, and re-executing the cells.

1.2. Import packages¶

Import numpy.

Import pyvo packages for working with Datalinker.

Import LSST Science Pipelines packages for image display, image type conversion, and utilities for remote data access.

In [1]:

import numpy as np
from pyvo.dal.adhoc import DatalinkResults

import lsst.afw.display as afwDisplay
from lsst.images.serialization import read_archive
from lsst.rsp import RSPDiscovery
from lsst.rsp.utils import get_pyvo_auth

1.3. Define parameters and functions¶

Instantiate the TAP service.

In [2]:

discovery = RSPDiscovery("dp2")
service = discovery.get_tap_client()

Define search coordinates right ascension (target_ra) and declination (target_dec), in degrees, to use in all queries. These coordinates are near the center of the ECDFS field.

In [3]:

target_ra = 53.000
target_dec = -28.015

Set afwDisplay to use Firefly, and define afw_display to show images in frame 1.

In [4]:

afwDisplay.setDefaultBackend("firefly")
afw_display = afwDisplay.Display(frame=1)

2. The ObsCore table¶

The ObsCore table schema conforms to the IVOA (International Virtual Observatory Alliance) standards.

The LSST ObsCore table is essentially a view into the images stored in the LSST's data Butler.

Retrieve ObsCore schema: the names of the columns and their data types, descriptions and units, and display it. Notice that the ObsCore table is in the ivoa collection, not the DP1 collection like the data tables.

In [5]:

query = "SELECT column_name, datatype, description, unit " \
        "FROM tap_schema.columns " \
        "WHERE table_name = 'ivoa.ObsCore'"
results = service.search(query).to_table()
results

Out[5]:

Table length=36

column_name	datatype	description	unit
str64	str64	str512	str64
dataproduct_type	char	Data product (file content) primary type
dataproduct_subtype	char	Data product specific type
obs_title	char	Brief description of dataset in free format
facility_name	char	The name of the facility, telescope, or space craft used for the observation
calib_level	int	Calibration level of the observation: in {0, 1, 2, 3, 4}
target_name	char	Object of interest
obs_id	char	Internal ID given by the ObsTAP service
obs_collection	char	Name of the data collection
obs_publisher_did	char	ID for the Dataset given by the publisher
access_url	char	URL used to access dataset
access_format	char	Content format of the dataset
s_ra	double	Central Spatial Position in ICRS; Right ascension	deg
s_dec	double	Central Spatial Position in ICRS; Declination	deg
s_fov	double	Estimated size of the covered region as the diameter of a containing circle	deg
s_region	char	Sky region covered by the data product (expressed in ICRS frame)
s_resolution	double	Spatial resolution of data as FWHM of PSF	arcsec
s_xel1	long	Number of elements along the first coordinate of the spatial axis
s_xel2	long	Number of elements along the second coordinate of the spatial axis
t_xel	long	Number of elements along the time axis
t_min	double	Start time in MJD	d
t_max	double	Stop time in MJD	d
t_exptime	double	Total exposure time	s
t_resolution	double	Temporal resolution FWHM	s
em_xel	long	Number of elements along the spectral axis
em_min	double	start in spectral coordinates	m
em_max	double	stop in spectral coordinates	m
em_res_power	double	Value of the resolving power along the spectral axis (R)
o_ucd	char	Nature of the observable axis
pol_xel	long	Number of elements along the polarization axis
instrument_name	char	The name of the instrument used for the observation
lsst_visit	long	Identifier for a specific LSSTCam pointing
lsst_detector	long	Identifier for CCD within the LSSTCam focal plane
lsst_tract	long	Upper level of LSST coadd skymap hierarchy
lsst_patch	long	Lower level of LSST coadd skymap hierarchy
lsst_band	char	Abstract filter band designation
lsst_filter	char	Physical filter designation from the LSSTCam filter set

In [6]:

del query, results

3. Query for images¶

It is recommended to tightly constrain image queries, so that they return only the image data products needed for a given scientific analysis.

The main constraints on ObsCore TAP queries are:

• image region overlaps point
• image type (calibration level: raw, processed, coadded)
• band (filter, spectral coverage)
• time (date and time of acquisition)

3.1. Position¶

Query by position.

To query only by position is not recommended because it would return every image that overlaps that region, including raw, direct, difference, and coadded images. Typically, for scientific analyses, a specific image type or band is desired and should be included in the query, as demonstrated in following sections.

Notice: Both DP1 and DP2 image metadata are stored in the ObsCore table, so the obs_collection must be specified as LSST.DP2 or else matching DP1 images will be returned.

In [7]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        AND obs_collection = 'LSST.DP2'
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.0,-28.015), s_region) = 1
        AND obs_collection = 'LSST.DP2'

Run the TAP query.

In [8]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED

12

Option to show the results table.

In [9]:

# results

Print the unique data product subtypes and how many of each were returned by the query.

In [10]:

values, counts = np.unique(results['dataproduct_subtype'],
                           return_counts=True)
for value, count in zip(values, counts):
    print(value, count)

lsst.deep_coadd 12

In [11]:

job.delete()
del query, results, values, counts

3.2. Image type¶

Constrain the query by image type.

The calibration levels and data product subtypes are:

• 1 : raw
• 2 : visit_image
• 3 : deep_coadd, template_coadd, difference_image

With the TAP service, constraining on calib_level is not necessary.

For example, query for deep_coadd images by requiring that the dataproduct_subtype = lsst.deep_coadd.

In [12]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        AND dataproduct_subtype='lsst.deep_coadd'
        AND obs_collection = 'LSST.DP2'
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.0,-28.015), s_region) = 1
        AND dataproduct_subtype='lsst.deep_coadd'
        AND obs_collection = 'LSST.DP2'

In [13]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED

12

In [14]:

job.delete()
del query, results

3.3. Band¶

Query by band.

Add a constraint that the lsst_band be "r".

In [15]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        AND dataproduct_subtype='lsst.deep_coadd'
        AND lsst_band = 'r'
        AND obs_collection = 'LSST.DP2'
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.0,-28.015), s_region) = 1
        AND dataproduct_subtype='lsst.deep_coadd'
        AND lsst_band = 'r'
        AND obs_collection = 'LSST.DP2'

In [16]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result()
print(len(results))

Job phase is COMPLETED

2

In [17]:

job.delete()
del query

4. Retrieve and display an image¶

Extract the access URL from the first row of the results as datalink_url.

Retrieve the datalink VOTable document as dl_result. At this point the service requires authorization credentials, which are passed with the utility function get_pyvo_auth() from the lsst.rsp package.

Get the URL for the specific image as image_url.

In [18]:

dl_result = discovery.get_datalink_results(results[0])
datalink_url = results['access_url'][0]
image_url = dl_result.getrecord(0).get('access_url')

Option to display the URLs.

In [19]:

# datalink_url

In [20]:

# dl_result

In [21]:

# image_url

Read the entire image into an CellCoadd object. The CellCoadd format is designed to work with seamlessly with afw.display and so is recommended for all Rubin coadd images.

In [22]:

deep_coadd = read_archive(image_url)

Display the exposure in the Firefly tab.

In [23]:

afw_display.image(deep_coadd)

Set the mask transparency to 100% transparent.

In [24]:

afw_display.setMaskTransparency(100)

In [ ]: