104.3. Image queries with the Butler

104.3. Image queries with the Butler¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: large
LSST Science Pipelines version: r30.0.10
Last verified to run: 2026-07-23
Repository: github.com/lsst/tutorial-notebooks

Learning objective: How to query and retrieve image data with the Butler.

LSST data products: deep_coadd

Packages: lsst.daf.butler

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

The Butler is LSST Science Pipelines middleware for managing, reading, and writing datasets.

As the interface between the pipelines and the data, it is often referred to as "middleware".

Butler-related documentation:

• pipelines middleware Frequently Asked Questions
• Butler python module documentation
• Butler query expressions and operators

This tutorial demonstrates the individual components of Butler queries and how to compose expressions using the allowed operators.

Related tutorials: The earlier 100-level Butler tutorials in this series show how to retrieve data with the Butler and how to explore and discover the dataset types and their properties.

1.1. Import packages¶

Import the butler module from the lsst.daf package, and the display module from the lsst.afw package (for image display).

In [1]:

from lsst.daf.butler import Butler
import lsst.afw.display as afwDisplay
import lsst.sphgeom as sphgeom
import lsst.geom as geom

1.2. Define parameters¶

Create an instance of the Butler with the repository and collection for DP2, and assert that it exists.

In [2]:

butler = Butler("dp2", collections="dp2")
assert butler is not None

Set afwDisplay to use Firefly.

In [3]:

afwDisplay.setDefaultBackend("firefly")

Define inputs to use in all query demonstrations (these are arbitrary, just for the purposes of this tutorial).

Coordinates: Use coordinates RA, Dec = $53.000, -28.015$ deg, which are near the center of the Extended Chandra Deep Field South (ECDFS).

In [4]:

ra = 53.000
dec = -28.015

Region: Define a circle with a radius of 2 deg, centered on the coordinates.

In [5]:

region = sphgeom.Region.from_ivoa_pos("CIRCLE 53.000 -28.015 2.0")

Tract and patch: Define the coordinates as a point, and use the skymap to get the tract and patch which covers that point.

In [6]:

point = geom.SpherePoint(ra*geom.degrees, dec*geom.degrees)
skymap = butler.get("skyMap", skymap="lsst_cells_v2")
tract = skymap.findTract(point).tract_id
patch = skymap.findTract(point).findPatch(point).getSequentialIndex()
print(tract, patch)

5063 25

Filter: Use the $r$-band.

In [7]:

band = 'r'

2. Query formation¶

A call to the Butler's query_datasets function requires at least a dataset_type (e.g., image type, such as visit_image or deep_coadd) and a where statement (a string expression resembling an SQL WHERE clause). The where statement can use bind parameters. The order_by and limit parameters are optional.

butler.query_datasets(<dataset_type>,
                      where=<query>,
                      bind=<bind_dictionary>,
                      order_by=<dimension_list>,
                      limit=<integer>)

The query_datasets function returns the dataset references for all data that meet the query constraints. The dataset reference can then be used with the butler.get function to retrieve the data itself.

Note that there is also a find_dataset function, however, it is not for querying the Butler. Rather, it can be used to retrieve the dataset reference when a dataId is already known (documentation for find_dataset).

2.1. Dataset type¶

Dataset types for DP2 processed images:

• deep_coadd

Print the information for each dataset type.

In [8]:

image_types = ['deep_coadd']

for itype in image_types:
    print('')
    print(itype)
    print(butler.get_dataset_type(itype))
    print('required: ', butler.get_dataset_type(itype).dimensions.required)

deep_coadd
DatasetType('deep_coadd', {band, skymap, tract, patch}, CellCoadd)
required:  {band, skymap, tract, patch}

2.2. Where statement¶

The where statement can be used to place constraints on the dimensions and fields of a given dataset type.

For a deep_coadd, the dimensions are sky location (coordinate, or patch and tract) and filter.

This cell provides the option to print the dimensions and schema for the deep_coadd dataset type.

In [9]:

# dataset_type = butler.get_dataset_type('deep_coadd')
# for dimension in dataset_type.dimensions.data_coordinate_keys:
#     print('dimension = ', dimension)
#     print(butler.dimensions[dimension].schema)
#     print(' ')

Execute a query for data with dataset type deep_coadd that were obtained in the $r$-band and overlap the defined coordinates.

In [10]:

query = f"band.name = '{band}' AND \
patch.region OVERLAPS POINT({ra}, {dec})"
print(query)

dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query)
print(len(dataset_refs))

del query, dataset_refs

band.name = 'r' AND patch.region OVERLAPS POINT(53.0, -28.015)

2

2.3. Bind parameters¶

Recreate the same query as above, but in the query statement use bind parameters (placeholders) and define the list bind_params to hold the values.

In [11]:

query = "band.name = :band AND " \
        "patch.region OVERLAPS POINT(:ra, :dec)"
print(query)

bind_params = {"band": band, "ra": ra, "dec": dec}
print(bind_params)

band.name = :band AND patch.region OVERLAPS POINT(:ra, :dec)
{'band': 'r', 'ra': 53.0, 'dec': -28.015}

In [12]:

dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params)
print(len(dataset_refs))
del dataset_refs

2

2.4. Order by¶

The order_by parameter accepts a list of strings specifying the sort parameters. Results can be sorted by dimension name (e.g., band, tract, patch) and dimension name and schema field (e.g., band.name).

In [13]:

dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params,
                                     order_by=["patch"])
print(dataset_refs[0])
del dataset_refs

deep_coadd@{band: 'r', skymap: 'lsst_cells_v2', tract: 5063, patch: 15} [sc=CellCoadd] (run=LSSTCam/runs/DRP/DP2/v30_0_8/DM-55060/deep_coadd_rewrite/20260618T031503Z id=019ed8bf-de86-7b2c-9fde-c601f3d6d29c)

Use a - in front of the string to reverse sort.

In [14]:

dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params,
                                     order_by=["-patch"])
print(dataset_refs[0])
del dataset_refs

deep_coadd@{band: 'r', skymap: 'lsst_cells_v2', tract: 5063, patch: 25} [sc=CellCoadd] (run=LSSTCam/runs/DRP/DP2/v30_0_8/DM-55060/deep_coadd_rewrite/20260618T031503Z id=019ed8bf-ec7a-7f61-8d0c-5ed3bdd9ba80)

3. Spatial queries for images¶

3.1. Overlaps point¶

Execute a query for data with dataset type deep_coadd that overlap the defined coordinates. Twelve results are returned because two of the skymap patches overlap these coordinates, and there are six LSST filters.

In [15]:

query = "patch.region OVERLAPS POINT(:ra, :dec)"
bind_params = {"ra": ra, "dec": dec}
dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params)
print(len(dataset_refs))
del query, dataset_refs

12

3.2. Overlaps region¶

Execute a query for data with dataset type deep_coadd that overlap the defined region. Since the region is larger, significantly more images are returned. For this example, use bind parameters for the query constraints.

In [16]:

query = "patch.region OVERLAPS :region"
bind_params = {'region': region}
dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params)
print(len(dataset_refs))
del dataset_refs

3715

In [17]:

print(region)

Circle([0.5312971403291302, 0.7050551188155045, -0.4697027018339827], 0.03490658503988659)

3.2.1. Limit¶

Use the limit parameter to limit the number of results. This is recommended when testing queries. To test the limit, redefine the query radius to 10 degrees.

Execute the query from the last section for data with dataset type deep_coadd that overlap the defined coordinates. Since the region is larger, significantly more deep coadd image patches are returned. By using limit=10, force the query to stop at 10 results.

In [18]:

dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params,
                                     limit=10)
print(len(dataset_refs))
del dataset_refs

10

Use a - in front of the limit value to have the query return a warning if there were more results that were not returned.

In [19]:

dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query,
                                     bind=bind_params,
                                     limit=-10)
print(len(dataset_refs))
del dataset_refs

lsst.daf.butler._butler WARNING: More datasets are available than the requested limit of 10.

10

In [20]:

del query, bind_params

4. Query for value in list¶

Create a query with an IN statement to find data with particular values in a list.

4.1. Patches of a given deep_coadd¶

For tract 5063, query for the deep_coadd corresponding to patches 3, 4, and 5.

In [21]:

query = f"tract.id = 5063 AND patch IN (3, 4, 5) AND skymap='lsst_cells_v2'"
print(query)
dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query)
print(len(dataset_refs))

tract.id = 5063 AND patch IN (3, 4, 5) AND skymap='lsst_cells_v2'
18

Print the dataId for the returned dataset references.

In [22]:

for ref in dataset_refs:
    print(ref.dataId)

{band: 'g', skymap: 'lsst_cells_v2', tract: 5063, patch: 5}
{band: 'g', skymap: 'lsst_cells_v2', tract: 5063, patch: 4}
{band: 'y', skymap: 'lsst_cells_v2', tract: 5063, patch: 5}
{band: 'u', skymap: 'lsst_cells_v2', tract: 5063, patch: 5}
{band: 'g', skymap: 'lsst_cells_v2', tract: 5063, patch: 3}
{band: 'z', skymap: 'lsst_cells_v2', tract: 5063, patch: 5}
{band: 'i', skymap: 'lsst_cells_v2', tract: 5063, patch: 5}
{band: 'i', skymap: 'lsst_cells_v2', tract: 5063, patch: 3}
{band: 'u', skymap: 'lsst_cells_v2', tract: 5063, patch: 4}
{band: 'z', skymap: 'lsst_cells_v2', tract: 5063, patch: 3}
{band: 'y', skymap: 'lsst_cells_v2', tract: 5063, patch: 4}
{band: 'z', skymap: 'lsst_cells_v2', tract: 5063, patch: 4}
{band: 'y', skymap: 'lsst_cells_v2', tract: 5063, patch: 3}
{band: 'i', skymap: 'lsst_cells_v2', tract: 5063, patch: 4}
{band: 'r', skymap: 'lsst_cells_v2', tract: 5063, patch: 4}
{band: 'r', skymap: 'lsst_cells_v2', tract: 5063, patch: 3}
{band: 'r', skymap: 'lsst_cells_v2', tract: 5063, patch: 5}
{band: 'u', skymap: 'lsst_cells_v2', tract: 5063, patch: 3}

In [23]:

del query, dataset_refs

5. Retrieve data from the Butler¶

As a reminder, the resulting dataset reference from query_datasets can be used to retrieve data from the Butler.

For tract 5063, query for the $r$-band deep_coadd image for patch 4.

In [24]:

query = f"tract.id = 5063 AND patch = 4 AND skymap='lsst_cells_v2' AND band='r'"
dataset_refs = butler.query_datasets("deep_coadd",
                                     where=query)
assert len(dataset_refs) == 1

Retrieve the deep_coadd.

In [25]:

deep_coadd = butler.get(dataset_refs[0])

Display the image in Firefly.

Define afw_display to show images in frame 1 (the Firefly window will open in a new tab).

In [26]:

afw_display = afwDisplay.Display(frame=1)

Display the image and turn the mask off.

In [27]:

afw_display.image(deep_coadd)
afw_display.setMaskTransparency(100)

In [28]:

del query, dataset_refs, deep_coadd