201.05. DiaSource table

201.5. DiaSource table¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: Large
LSST Science Pipelines version: r30.0.10
Last verified to run: 2026-07-22
Repository: github.com/lsst/tutorial-notebooks
DOI: 10.11578/rubin/dc.20250909.20

Learning objective: To understand the contents of the DiaSource table and how to access it.

LSST data products: DiaSource

Packages: lsst.rsp, lsst.daf.butler

Credit: Originally developed by the Rubin Community Science team with feedback from Eric Bellm, Federica Bianco, and Tatiana Cuellar. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

The DiaSource table contains measurements for positive- and negative-flux sources detected in the difference images with signal-to-noise ratio $\geq5$.

• TAP table name: dp2.DiaSource
• butler table name: dia_source
• columns: 92
• rows: 1,000,825,975

Related tutorials: The TAP and butler data access services are demonstrated in the 100-level "How to" tutorials. There are 200-level tutorials on difference_images and DiaObjects.

1.1. Import packages¶

Import standard python packages numpy, matplotlib and astropy.

From the lsst package, import modules for the TAP service, the butler, and plotting.

In [1]:

import re
import numpy as np
import matplotlib.pyplot as plt
from lsst.rsp import RSPDiscovery
from lsst.daf.butler import Butler
from lsst.utils.plotting import (get_multiband_plot_colors,
                                 get_multiband_plot_symbols)

1.2. Define parameters and functions¶

Create an instance of the TAP service, and assert that it exists.

In [2]:

discovery = RSPDiscovery("dp2")
tap_service = discovery.get_tap_client()

Create an instance of the Rubin data butler, and assert that it exists.

In [3]:

butler = Butler("dp2", collections="dp2")

Define the colors and symbols to represent the LSST filters in plots.

In [4]:

filter_colors = get_multiband_plot_colors()
filter_names = filter_colors.keys()
filter_symbols = get_multiband_plot_symbols()

2. Schema (columns)¶

To browse the table schema visit the Rubin schema browser, or use the TAP service via the Portal Aspect or as demonstrated in Section 2.1.

2.1. Retrieve table schema¶

To retrieve the table schema, define a query for the schema columns of the DiaSource table and run the query job.

In [5]:

query = "SELECT column_name, datatype, description, unit " \
        "FROM tap_schema.columns " \
        "WHERE table_name = 'dp2.DiaSource'"
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'

Job phase is COMPLETED

Retrieve the query results and display them as an astropy table with the to_table() attribute.

In [6]:

results = job.fetch_result().to_table()
results

Out[6]:

Table length=90

column_name	datatype	description	unit
str64	str64	str512	str64
diaSourceId	long	Unique identifier of this DiaSource.
visit	long	Id of the visit where this diaSource was measured.
detector	short	Id of the detector where this diaSource was measured.
diaObjectId	long	Id of the diaObject this source was associated with, if any. If not, it is set to 0
ssObjectId	long	Id of the ssObject this source was associated with, if any. If not, it is set to 0
parentDiaSourceId	long	Id of the parent diaSource this diaSource has been deblended from, if any.
midpointMjdTai	double	Effective mid-visit time for this diaSource, expressed as Modified Julian Date, International Atomic Time.	d
ra	double	Right ascension coordinate of the center of this diaSource.	deg
dec	double	Declination coordinate of the center of this diaSource.	deg
raErr	float	Angular error in right ascension (∂RA*cos(Dec)).	deg
decErr	float	Error in declination.	deg
ra_dec_Cov	float	Covariance between right ascension (RA*cos(Dec)) and declination.	deg**2
x	double	Unweighted first moment centroid, overall centroid.	pixel
xErr	float	1-sigma uncertainty on x position.	pixel
y	double	Unweighted first moment centroid, overall centroid.	pixel
yErr	float	1-sigma uncertainty on y position.	pixel
centroid_flag	boolean	General centroid algorithm failure flag; set if anything went wrong when fitting the centroid. Another centroid flag field should also be set to provide more information.
apFlux	float	Flux in a 12 pixel radius aperture on the difference image.	nJy
apFluxErr	float	Estimated uncertainty of apFlux.	nJy
apFlux_flag	boolean	General aperture flux algorithm failure flag; set if anything went wrong when measuring aperture fluxes. Another apFlux flag field should also be set to provide more information.
apFlux_flag_apertureTruncated	boolean	Aperture did not fit within measurement image.
isNegative	boolean	Source was detected as significantly negative.
...	...	...	...
bboxSize	long	Size of the square bounding box that fully contains the detection footprint.	pixel
pixelFlags	boolean	General pixel flags failure; set if anything went wrong when setting pixels flags from this footprint's mask. This implies that some pixelFlags for this source may be incorrectly set to False.
pixelFlags_bad	boolean	Bad pixel in the DiaSource footprint.
pixelFlags_cr	boolean	Cosmic ray in the DiaSource footprint.
pixelFlags_crCenter	boolean	Cosmic ray in the 3x3 region around the centroid.
pixelFlags_edge	boolean	Some of the source footprint is outside usable exposure region (masked EDGE or centroid off image).
pixelFlags_nodata	boolean	NO_DATA pixel in the source footprint.
pixelFlags_nodataCenter	boolean	NO_DATA pixel in the 3x3 region around the centroid.
pixelFlags_interpolated	boolean	Interpolated pixel in the DiaSource footprint.
pixelFlags_interpolatedCenter	boolean	Interpolated pixel in the 3x3 region around the centroid.
pixelFlags_offimage	boolean	DiaSource center is off image.
pixelFlags_saturated	boolean	Saturated pixel in the DiaSource footprint.
pixelFlags_saturatedCenter	boolean	Saturated pixel in the 3x3 region around the centroid.
pixelFlags_suspect	boolean	DiaSource's footprint includes suspect pixels.
pixelFlags_suspectCenter	boolean	Suspect pixel in the 3x3 region around the centroid.
pixelFlags_streak	boolean	Streak in the DiaSource footprint.
pixelFlags_streakCenter	boolean	Streak in the 3x3 region around the centroid.
pixelFlags_injected	boolean	Injection in the DiaSource footprint.
pixelFlags_injectedCenter	boolean	Injection in the 3x3 region around the centroid.
pixelFlags_injected_template	boolean	Template injection in the DiaSource footprint.
pixelFlags_injected_templateCenter	boolean	Template injection in the 3x3 region around the centroid.
glint_trail	boolean	This flag is set if the source is part of a glint trail.

The table displayed above has been truncated.

Option to print every column name as a list.

In [7]:

# for col in results['column_name']:
#     print(col)

Option to use the regular expressions package re to search for column names that contain the string temp.

In [8]:

# temp = 'Err'
# temp = 'Flux'
# temp = 'psf'
# for col in results['column_name']:
#     if re.search(temp, col):
#         print(col)

Delete the job, but not the results.

In [9]:

del query
job.delete()

2.2. Key columns¶

Of the 92 columns of the DiaSource table, a $\sim$dozen are the most commonly used.

2.2.1. Identifiers¶

A long integer that uniquely identifies each row of the DiaSource table.

• diaSourceId

The long integers that identify the associated diaObject or ssObject; only one of these two will be populated.

• diaObjectId
• ssObjectId

2.2.2. Visit information¶

The visit is a long integer that uniquely identifies the visit the DiaSources were detected. The detector is which of the 9 LSSTComCam detectors the DiaSources were detected on. Together, the visit and detector correspond to a processed visit_image.

• visit
• detector

The filter band used for the observation of the DiaSource.

• band

Effective mid-exposure time for visit where the DiaSource was detected, expressed as Modified Julian Date, International Atomic Time.

• midpointMjdTai

2.2.3. Coordinates¶

The sky coordinates in decimal degrees for each DiaSource:

• ra, raErr
• dec, decErr

2.2.4. Flux measurements and flags¶

Fluxes are provided in nanoJanskys, which is preferred for difference-image photometry over magnitudes since negative flux measurements would be omitted when converting to magnitudes.

PSF fluxes

A forced fit of the Point Spread Function (PSF) at the diaSource's coordinates in each difference image. PSF fluxes are best to use for point-like sources.

• psfFlux, psfFluxErr

Note that in the DiaSource table, the psfFlux column contains fluxes measured on the difference image. In other tables psfFlux is a flux on the direct (science) image.

Other fluxes

• apFlux: fluxes in a 12 pixel radius aperture on the difference_image.
• scienceFlux: Forced PSF flux measured on the direct visit_image.

Flags

A variety of flags indicating whether pixels that are saturated, or affected by cosmic rays, contributed to the diaSource's measurements.

• pixelFlags_*

The flux (and shape) measurements mentioned above have associated flag columns suffixed with _flag.

• _flag

A boolean flag indicating whether or not the diaSource is classified as a dipole: , a likely image-differencing artifact that appears as offset positive and negative lobes. Setting this to False will filter out poorly subtracted stars.

• isDipole

2.2.5. Reliability score¶

The purpose of the reliability score in the DiaSource table is to provide a threshold measure to improve the purity of DIA detections.

For DP2, the reliability score accidentally used the v0.1 model, the same model that was used for DP1. Unfortunately this means that none of the advances made on reliability scoring that are documented in DMTN-337 were applied to DP2. As no reliability-based cuts were made to the catalog contents this ultimately has not impacted the DiaSource table, except that the reliability scores are not as informative as they could be.

• reliability

2.3. Descriptions and units¶

For a subset of the key columns show the table of their descriptions and units.

In [10]:

col_list = set(['diaSourceId', 'diaObjectId', 'ssObjectId', 'band', 'visit',
                'midpointMjdTai', 'ra', 'dec', 'psfFlux', 'psfFluxErr', 'isDipole'])
tx = [i for i, item in enumerate(results['column_name']) if item in col_list]
results[tx]

Out[10]:

Table length=11

column_name	datatype	description	unit
str64	str64	str512	str64
diaSourceId	long	Unique identifier of this DiaSource.
visit	long	Id of the visit where this diaSource was measured.
diaObjectId	long	Id of the diaObject this source was associated with, if any. If not, it is set to 0
ssObjectId	long	Id of the ssObject this source was associated with, if any. If not, it is set to 0
midpointMjdTai	double	Effective mid-visit time for this diaSource, expressed as Modified Julian Date, International Atomic Time.	d
ra	double	Right ascension coordinate of the center of this diaSource.	deg
dec	double	Declination coordinate of the center of this diaSource.	deg
psfFlux	float	Flux for Point Source model. Note this actually measures the flux difference between the template and the visit image.	nJy
psfFluxErr	float	Uncertainty of psfFlux.	nJy
band	char	Filter band this source was observed with.
isDipole	boolean	Source well fit by a dipole.

Clean up.

In [11]:

del col_list, tx, results

3. Data access¶

The DiaSource table is available via the TAP service and the butler.

Recommended access method: TAP.

3.1. Advisory: avoid full-table queries¶

Avoid full-table queries. It is ideal to constrain queries by diaObjectId or ssObjectId, or by visit/detector.

Most queries on the DiaSource table should be for all difference-image detections associated with a given diaObject, ssObject, or visit and detector. For the question of "did Rubin ever detect a time-domain object at this RA, Dec?", query the DiaObject table, then get all diaSources associated with that diaObject with a separate query (or use a table join, as in S.3.3.2.).

3.2. Advisory: not recommended for light curves¶

The ForcedSourceOnDiaObject table is recommended for light curves.

The DiaSource table contains measurements of difference-image sources with $\geq5\sigma$ detections, which means that lower-flux variability will not be captured. The ForcedSourceOnDiaObject table, which contains forced PSF photometry for all difference images as well as non-difference (i.e. visit) images, is recommended to generate complete variable and transient object light curves, with a data point for every time the coordinates were observed.

3.3. TAP (Table Access Protocol)¶

The DiaSource table is stored in Qserv and accessible via the TAP services using ADQL queries.

Including spatial constraints: Qserv stores catalog data sharded by coordinate (RA, Dec), so ADQL query statements that include constraints by coordinate do not requre a whole-catalog search and are typically faster (and can be much faster) than ADQL query statements which only include constraints for other columns. Use either an ADQL cone or polygon search for faster queries (do not use WHERE ... BETWEEN statements to set boundaries on RA and Dec).

3.3.1. Demo query¶

Define a query to return the "key columns" from Section 2.3.

Impose visit and detector constraints: create a query to return all diaSources detected in the difference image corresponding to detector number 86, visit number 2026010500067. This is not a special image, just a random example.

In [12]:

visit = 2026010500067
detector = 86

In [13]:

query = """SELECT diaSourceId, diaObjectId, ssObjectId, band, visit, detector,
        midpointMjdTai, ra, dec, psfFlux, psfFluxErr, isDipole
        FROM dp2.DiaSource
        WHERE visit = {} AND detector = {}""".format(visit, detector)
print(query)

SELECT diaSourceId, diaObjectId, ssObjectId, band, visit, detector,
        midpointMjdTai, ra, dec, psfFlux, psfFluxErr, isDipole
        FROM dp2.DiaSource
        WHERE visit = 2026010500067 AND detector = 86

In [14]:

job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

Job phase is COMPLETED

Fetch the results as an astropy table.

In [15]:

assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

23

Option to display the table.

In [16]:

# results

Plot the coordinates for each diaSource detection, and the histogram of psfFlux values.

In [17]:

fig, ax = plt.subplots(1, 2, figsize=(10, 5))
tx1 = np.where((results['psfFlux'] > 0.0) & (~results['isDipole']))[0]
tx2 = np.where((results['psfFlux'] < 0.0) & (~results['isDipole']))[0]
tx3 = np.where((results['psfFlux'] > 0.0) & (results['isDipole']))[0]
tx4 = np.where((results['psfFlux'] < 0.0) & (results['isDipole']))[0]
ax[0].plot(results['ra'][tx1], results['dec'][tx1], 'o', fillstyle='full', color='white',
           ms=7, mew=1, mec='black', alpha=0.5, label='positive')
ax[0].plot(results['ra'][tx2], results['dec'][tx2], 's', fillstyle='full', color='black',
           ms=7, mew=1, mec='black', alpha=0.5, label='negative')
ax[0].plot(results['ra'][tx3], results['dec'][tx3], 'o', fillstyle='right', color='black',
           ms=7, mew=1, mec='black', alpha=0.5, label='positive; dipole')
ax[0].plot(results['ra'][tx4], results['dec'][tx4], 's', fillstyle='left', color='black',
           ms=7, mew=1, mec='black', alpha=0.5, label='negative; dipole')
ax[0].set_xlabel('RA [arcsec]')
ax[0].set_ylabel('Dec [arcsec]')
ax[0].legend(loc='lower left')
bar_data = [results['psfFlux'][tx1], results['psfFlux'][tx2],
            results['psfFlux'][tx3], results['psfFlux'][tx4]]
bar_colors = ['white', 'black', 'lightgrey', 'grey']
bar_labels = ['positive', 'negative', 'positive; dipole', 'negative; dipole']
ax[1].hist(bar_data, bins=20, stacked=True, color=bar_colors, edgecolor='black', label=bar_labels)
ax[1].set_xlabel('psfFlux [nJy]')
ax[1].set_ylabel('Number')
ax[1].legend(loc='upper left')
plt.tight_layout()
plt.show()
del tx1, tx2, tx3, tx4

Figure 1: At left, the RA and Dec for all detected diaSources in the chosen visit+detector, with symbol type and color fill representing positive or negative flux, and whether the source exhibited a dipole morphology. At right, a stacked bar chart of the difference-image PSF flux measurements for detected diaSources.

Clean up.

In [18]:

job.delete()
del query, results

3.3.2. Joinable tables¶

The DiaSource table can be joined to the DiaObject table on the column containing the unique diaObjectId, and with the SSObject table on ssObjectId.

DiaObject Table

The DiaObject table contains statistical information from DiaSource properties associated with the DiaObject.

As an example, query for all the diaSources (difference-image detections) associated with all diaObjects within 5 arcseconds of the target coordinates (e.g., a potential host galaxy of interest). Return the diaSourceId, psfFlux, and band from the DiaSource table, and the diaObjectId and the number of detections nDiaSources from the DiaObject table.

In [19]:

ra_targ = 151.591891
dec_targ = 1.159049
search_radius = np.round(5./3600.0, 5)
query = """SELECT dias.diaSourceId, dias.psfFlux, dias.band,
        diao.diaObjectId, diao.nDiaSources
        FROM dp2.DiaSource AS dias
        JOIN dp2.DiaObject AS diao ON dias.diaObjectId = diao.diaObjectId
        WHERE CONTAINS(POINT('ICRS', diao.ra, diao.dec),
        CIRCLE('ICRS', {}, {}, {})) = 1""".format(ra_targ, dec_targ, search_radius)
print(query)

SELECT dias.diaSourceId, dias.psfFlux, dias.band,
        diao.diaObjectId, diao.nDiaSources
        FROM dp2.DiaSource AS dias
        JOIN dp2.DiaObject AS diao ON dias.diaObjectId = diao.diaObjectId
        WHERE CONTAINS(POINT('ICRS', diao.ra, diao.dec),
        CIRCLE('ICRS', 151.591891, 1.159049, 0.00139)) = 1

In [20]:

job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

Job phase is COMPLETED

Fetch the results as an astropy table.

In [21]:

assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()

Option to display the results.

In [22]:

# results

For each unique diaObject within the search radius, print the number of rows (detections) returned from the DiaSource table.

In [23]:

values, counts = np.unique(results['diaObjectId'], return_counts=True)
for value, count in zip(values, counts):
    print(value, count)

786352299365629972 201
786352299365630545 1
786352299365631347 1
786352299365631545 1
786352299365632044 1

Print the number of detections per band.

In [24]:

use_diaObjectId = 786352299365629972
print("For diaObjectId: ", use_diaObjectId)

tx = np.where(results['diaObjectId'] == 786352299365629972)[0]
print('Total number of detections: ', results['nDiaSources'][tx[0]])

string_out = 'Number of detections per filter:  '
for f, filt in enumerate(filter_names):
    fx = np.where(results['band'][tx] == filt)[0]
    string_out += filt + ': ' + str(len(fx)) + '   '
    del fx
print(string_out)
del use_diaObjectId, tx, string_out

For diaObjectId:  786352299365629972
Total number of detections:  201
Number of detections per filter:  u: 24   g: 19   r: 67   i: 61   z: 27   y: 3

In [25]:

del query, results

SSObject Table

The DiaSource table can also be joined to the SSObject table on the column containing the unique ssObjectId.

As an example, query for all the diaSources (difference-image detections) associated with the ssObject with the designation (name) "2001 HL69". Return the diaSourceId, psfFlux, and band from the DiaSource table, and the ssObjectId, the nObs (number of observations), and the designation from the SSObject table.

In [26]:

use_designation = '2001 HL69'
query = """SELECT dias.diaSourceId, dias.psfFlux, dias.band,
        sso.ssObjectId, sso.nObs, sso.designation
        FROM dp2.DiaSource AS dias
        JOIN dp2.SSObject AS sso ON dias.ssObjectId = sso.ssObjectId
        WHERE sso.designation = '{}'""".format(use_designation)
print(query)

SELECT dias.diaSourceId, dias.psfFlux, dias.band,
        sso.ssObjectId, sso.nObs, sso.designation
        FROM dp2.DiaSource AS dias
        JOIN dp2.SSObject AS sso ON dias.ssObjectId = sso.ssObjectId
        WHERE sso.designation = '2001 HL69'

In [27]:

job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED
587

Show that only one unique ssObject was returned (the one requested by name).

In [28]:

print(np.unique(results['ssObjectId'])[0])
print(np.unique(results['designation'])[0])

21163611476343116
2001 HL69

Print the number of detections per band.

In [29]:

string_out = 'Number of detections per filter:  '
for f, filt in enumerate(filter_names):
    fx = np.where(results['band'] == filt)[0]
    string_out += filt + ': ' + str(len(fx)) + '   '
    del fx
print(string_out)
del string_out

print('Total number of detections: ', results['nObs'][0])

Number of detections per filter:  u: 26   g: 187   r: 232   i: 142   z: 0   y: 0
Total number of detections:  587

Clean up.

In [30]:

job.delete()
del query, results

3.4. Butler¶

TAP is the recommended way to access the diaSource table, but the butler is a convenient way to retrive all diaSources detected in a given tract.

Show that the dimension for the dia_source table is just the skymap's tract, and that it is required.

In [31]:

butler.get_dataset_type('dia_source')

Out[31]:

DatasetType('dia_source', {skymap, tract}, ArrowAstropy)

In [32]:

butler.get_dataset_type('dia_source').dimensions.required

Out[32]:

{skymap, tract}

3.4.1. Demo query¶

Retrieve the dataset references (refs) for dia_source tables for tracts that overlap the coordinates near the center of the COSMOS field. There will be only 1.

In [33]:

ra_center = 150.1
dec_center = 2.1
refs = butler.query_datasets("dia_source",
                             where="tract.region OVERLAPS POINT(:ra, :dec)",
                             bind={"ra": ra_center, "dec": dec_center},
                             order_by='tract')
print(len(refs))

1

Define a subset of columns to retrieve from the butler, and retrieve the table data.

In [34]:

col_list = set(['diaSourceId', 'band', 'midpointMjdTai',
                'ra', 'dec', 'psfFlux', 'psfFluxErr'])
results = butler.get(refs[0], parameters={'columns': col_list})
print(len(results))

891511

Subset the results to r-band fluxes between 20 and 25 mag.

In [35]:

tx = np.where((results['psfFlux'] > 360.0)
              & (results['psfFlux'] <= 36000.0)
              & (results['band'] == 'r'))[0]
print(len(tx))

144088

As an example, for this subset plot the sky coordinates, the flux vs. uncertainty, and the histogram of detection dates.

In [36]:

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(10, 3))
ax1.plot(results['ra'][tx], results['dec'][tx],
         'o', ms=2, mew=0, alpha=0.1, color='grey')
ax1.set_xlabel('Right Ascension')
ax1.set_ylabel('Declination')
ax2.plot(results['psfFlux'][tx], results['psfFluxErr'][tx],
         'o', ms=2, mew=0, alpha=0.1, color='grey')
ax2.set_xlabel('PSF flux [nJy]')
ax2.set_ylabel('Error')
ax3.hist(results['midpointMjdTai'][tx], bins=100, color='grey')
ax3.set_xlabel('MJD')
ax3.set_ylabel('Number of detections')
plt.tight_layout()
plt.show()

Figure 2: At left, the RA vs. Dec of diaSources as grey overlapping points, appearing darker for coordinates with many detections over time. In the center, the difference-image PSF flux vs. the flux error, for all detections in all difference-images that overlapped this tract of sky. At right, the histogram of detection dates show that most observations of the COSMOS field were done in a batch, then after a big seasonal gap, a few more were acquired.

In [ ]: