201.06. ForcedSourceOnDiaObject table

201.6. Forced Source on DiaObject table¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: Large
LSST Science Pipelines version: r30.0.10
Last verified to run: 2026-07-22
Repository: github.com/lsst/tutorial-notebooks
DOI: 10.11578/rubin/dc.20250909.20

Learning objective: To understand the contents of the ForcedSourceOnDiaObject table and how to access it.

LSST data products: ForcedSourceOnDiaObject

Packages: lsst.rsp, lsst.daf.butler

Credit: Originally developed by the Rubin Community Science team. This notebook also utilizes a transient detection from LSSTComCam first identified by Dan Taranu. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

The ForcedSourceOnDiaObject table contains forced PSF measurements in individual visit_images and difference_images at the sky coordinates of all DiaObjects.

DiaObjects are the union set of all diaSources detected in difference_images.

It is recommended to use the ForcedSourceOnDiaObject table for light curves.

• TAP table name: dp2.ForcedSourceOnDiaObject
• butler table name: dia_object_forced_source
• columns: 26
• rows: 24,259,599,059

Related tutorials: The TAP and butler data access services are demonstrated in the 100-level "How to" tutorials. There are 200-level tutorials on difference_images and the DiaObject table.

1.1. Import packages¶

Import standard python packages numpy, matplotlib and astropy.

From the lsst package, import modules for the TAP service, the butler, and plotting.

In [1]:

import re
import numpy as np
import matplotlib.pyplot as plt
from lsst.rsp import RSPDiscovery
from lsst.daf.butler import Butler
from lsst.utils.plotting import (get_multiband_plot_colors,
                                 get_multiband_plot_symbols,
                                 get_multiband_plot_linestyles)

1.2. Define parameters and functions¶

Create an instance of the TAP service, and assert that it exists.

In [2]:

discovery = RSPDiscovery("dp2")
tap_service = discovery.get_tap_client()

Create an instance of the Rubin data butler, and assert that it exists.

In [3]:

butler = Butler("dp2", collections="dp2")

Define the colors and symbols to represent the LSST filters in plots.

In [4]:

filter_colors = get_multiband_plot_colors()
filter_names = filter_colors.keys()
filter_symbols = get_multiband_plot_symbols()
filter_linestyles = get_multiband_plot_linestyles()

2. Schema (columns)¶

To browse the table schema visit the Rubin schema browser, or use the TAP service via the Portal Aspect or as demonstrated in Section 2.1.

2.1. Retrieve table schema¶

To retrieve the table schema, define a query for the schema columns of the ForcedSourceOnDiaObject table and run the query job.

In [5]:

query = "SELECT column_name, datatype, description, unit " \
        "FROM tap_schema.columns " \
        "WHERE table_name = 'dp2.ForcedSourceOnDiaObject'"
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'

Job phase is COMPLETED

Retrieve the query results and display them as an astropy table with the to_table() attribute.

In [6]:

results = job.fetch_result().to_table()
results

Out[6]:

Table length=26

column_name	datatype	description	unit
str64	str64	str512	str64
band	char	Abstract filter that is not associated with a particular instrument
detector	short	Id of the detector where this forced source was measured.
diaObjectId	long	Id of the DiaObject that this DiaForcedSource was associated with.
diff_PixelFlags_nodataCenter	boolean	Source center is outside usable region on image difference (masked NO_DATA)
invalidPsfFlag	boolean	Forced source has an invalid PSF.
parentObjectId	long	Unique ObjectId of the parent of the ObjectId in context of the deblender.
patch	long	Skymap patch ID
pixelFlags_bad	boolean	Bad pixel in the Source footprint
pixelFlags_cr	boolean	Cosmic ray in the Source footprint
pixelFlags_crCenter	boolean	Cosmic ray in the Source center
pixelFlags_edge	boolean	Source is on the edge of an exposure region (masked EDGE)
pixelFlags_interpolated	boolean	Interpolated pixel in the Source footprint
pixelFlags_interpolatedCenter	boolean	Interpolated pixel in the Source center
pixelFlags_nodata	boolean	Source is outside usable exposure region (masked NO_DATA)
pixelFlags_saturated	boolean	Saturated pixel in the Source footprint
pixelFlags_saturatedCenter	boolean	Saturated pixel in the Source center
pixelFlags_suspect	boolean	Sources footprint includes suspect pixels
pixelFlags_suspectCenter	boolean	Sources center is close to suspect pixels
psfDiffFlux	float	Flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfDiffFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfDiffFlux_flag	boolean	Failure to derive linear least-squares fit of psf model forced on the image difference
psfFlux	float	Flux derived from linear least-squares fit of psf model forced on the calexp	nJy
psfFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the calexp	nJy
psfFlux_flag	boolean	Failure to derive linear least-squares fit of psf model forced on the calexp
tract	long	Skymap tract ID
visit	long	Id of the visit where this forced source was measured.

The table displayed above has been truncated.

Option to print every column name as a list.

In [7]:

# for col in results['column_name']:
#     print(col)

Option to use the regular expressions package re to search for column names that contain the string temp.

In [8]:

# temp = 'Err'
# temp = 'Flux'
# temp = 'psf'
# for col in results['column_name']:
#     if re.search(temp, col):
#         print(col)

Delete the job, but not the results.

In [9]:

del query
job.delete()

2.2. Key columns¶

The few most commonly used columns of the ForcedSourceOnDiaObject table.

Notice: Coordinates are not included in the ForcedSourceOnDiaObject table because all measurements are made at the exact same coordinates: those of the associated diaObject.

2.2.1. Identifier¶

The diaObjectId that identifies the row of the DiaObject table associated with the ForcedSourceOnDiaObject entry.

• diaObjectId

2.2.2. Observation metadata¶

The band, visit, and detector in which the ForcedSourceOnDiaObject was measured.

• band, visit, detector

2.2.3. Fluxes and flags¶

A forced fit of the Point Spread Function (PSF) at the object's coordinates in each individual visit_image and difference_image.

• psfFlux, psfFluxErr
• psfDiffFlux, psfDiffFluxErr

Forced fluxes on difference images can be negative.

It is not recommended to convert forced fluxes to magnitudes, but if they are positive, the fluxes ($f$) are in nanojanskies and the corresponding AB magnitude ($m$) is: $m = -2.5\log(f) + 31.4$.

Flags

A variety of flags indicating whether pixels that are saturated, or affected by cosmic rays, contributed to the source's measurements.

• pixelFlags_*

2.3. Descriptions and units¶

For a subset of the key columns show the table of their descriptions and units.

In [10]:

col_list = set(['diaObjectId', 'band', 'visit', 'detector',
                'psfFlux', 'psfFluxErr', 'psfDiffFlux', 'psfDiffFluxErr'])
tx = [i for i, item in enumerate(results['column_name']) if item in col_list]
results[tx]

Out[10]:

Table length=8

column_name	datatype	description	unit
str64	str64	str512	str64
band	char	Abstract filter that is not associated with a particular instrument
detector	short	Id of the detector where this forced source was measured.
diaObjectId	long	Id of the DiaObject that this DiaForcedSource was associated with.
psfDiffFlux	float	Flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfDiffFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfFlux	float	Flux derived from linear least-squares fit of psf model forced on the calexp	nJy
psfFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the calexp	nJy
visit	long	Id of the visit where this forced source was measured.

Clean up.

In [11]:

del col_list, tx, results

3. Data access¶

The ForcedSourceOnDiaObject table is available via the TAP service and the butler.

Recommended access method: TAP.

3.1. Advisory: avoid full-table queries¶

Avoid full-table queries. Query by diaObjectId or visit identifiers.

The ForcedSourceOnDiaObject table is a large inclusive union set of forced measurements made in all the visit and difference images at the locations of all diaObjects.

There are two main use-cases for queries on the ForcedSourceOnDiaObject table.

Case A: Return all forced photometry for a given DiaObject.

• Step 1. Query the DiaObject table to get the diaObjectId.
• Step 2. Retrieve all forced photometry by diaObjectId.

Case B: Return all forced photometry measurements from a given visit_ or difference_image.

• Step 1. Query the Visit table to get the visit identifier.
• Step 2. Retrieve all forced photometry by visit.

These two steps can be performed simultaneously with table joins as shown in Section 3.2.2.

3.2. TAP (Table Access Protocol)¶

The ForcedSourceOnDiaObject table is stored in Qserv and accessible via the TAP services using ADQL queries.

3.2.1. Demo query¶

For coordinates of interest (e.g., a known transient location), query the DiaObject table with a search radius of 1 arcsecond. This is an example of Case A, above.

In [12]:

ra_targ = 151.592579
dec_targ = 1.158333
search_radius = 1.0/3600.0
query = """SELECT diaObjectId, ra, dec
        FROM dp2.DiaObject
        WHERE CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', {}, {}, {})) = 1
        """.format(ra_targ, dec_targ, search_radius)

In [13]:

job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()

Job phase is COMPLETED

Assert that there was only one spatial match in the DiaObject table, and print the diaObjectId.

In [14]:

assert len(results) == 1
print(results['diaObjectId'][0])

786352299365629972

Retrieve the forced photometry for this diaObjectId.

In [15]:

query = """SELECT diaObjectId, band, visit, detector,
        psfFlux, psfFluxErr, psfDiffFlux, psfDiffFluxErr
        FROM dp2.ForcedSourceOnDiaObject
        WHERE diaObjectId = {}""".format(results['diaObjectId'][0])
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print('Number of forced photometry data points retrieved: ', len(results))

Job phase is COMPLETED
Number of forced photometry data points retrieved:  295

Option to display the table.

In [16]:

# results

As an example, calculate and print the minimum, maximum, mean and standard deviation in PSF flux in the difference_images per filter. Negative fluxes can indicate that the transient was in the reference template used for the image subtraction.

In [17]:

print('%8s %8s %8s %8s %8s %8s' % ('filter', 'number', 'min', 'max', 'mean', 'std'))
for filt in filter_names:
    fx = np.where(results['band'] == filt)[0]
    print('%8s %8i %8.1f %8.1f %8.1f %8.1f' %
          (filt, len(fx),
           np.min(results['psfDiffFlux'][fx]), np.max(results['psfDiffFlux'][fx]),
           np.mean(results['psfDiffFlux'][fx]), np.std(results['psfDiffFlux'][fx])))

  filter   number      min      max     mean      std
       u       42  -6961.7  10097.8   -664.8   5078.9
       g       35 -30664.9  22065.3   -166.6  15793.8
       r       76 -66059.0 111824.0  41157.9  54591.6
       i       66 -66221.2  66221.5   4722.0  46830.3
       z       45 -142311.0  59404.6   4780.0  50497.7
       y       31 -137202.0   2805.0  -9686.1  27405.8

Clean up.

In [18]:

job.delete()
del query, results

3.2.2. Joinable tables¶

The ForcedSourceOnDiaObject table can be joined to the Visit table on column visit and the DiaObject table on column diaObjectId. The Visit table contains information about the observation, such as the date and time. The DiaObject table contains measurement statistics made on the difference_images.

To plot a light curve, join the ForcedSourceOnDiaObject and Visit tables. The following query is a join of the two queries in S.3.2.1, and adds a join to the Visit table to retrieve the modified julian date (MJD) at the midpoint of the exposure (of the visit): expMidptMJD.

In [19]:

ra_targ = 151.592579
dec_targ = 1.158333
search_radius = 1.0/3600.0
query = """SELECT fsodo.band, fsodo.psfDiffFlux, fsodo.psfDiffFluxErr, v.expMidptMJD
        FROM dp2.ForcedSourceOnDiaObject AS fsodo
        JOIN dp2.Visit AS v ON fsodo.visit = v.visit
        JOIN dp2.DiaObject AS diao ON diao.diaObjectId = fsodo.diaObjectId
        WHERE CONTAINS(POINT('ICRS', diao.ra, diao.dec), CIRCLE('ICRS', {}, {}, {})) = 1
        """.format(ra_targ, dec_targ, search_radius)
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED
295

Option to display the results table.

In [20]:

# results

Plot the forced photometry light curve for this object.

In [21]:

fig, ax = plt.subplots(1, 2, figsize=(8, 3))
for f, filt in enumerate(filter_names):
    fx = np.where(results['band'] == filt)[0]
    if len(fx) > 0:
        ax[0].plot(results['expMidptMJD'][fx], results['psfDiffFlux'][fx],
                   filter_symbols[filt], ms=8, mew=0, alpha=0.5,
                   color=filter_colors[filt], label=filt)
        ax[1].plot(results['expMidptMJD'][fx], results['psfDiffFlux'][fx],
                   filter_symbols[filt], ms=8, mew=0, alpha=0.5, color=filter_colors[filt])
ax[0].set_xlabel('MJD')
ax[0].set_ylabel('PSF Diff Flux')
ax[0].legend(loc='upper right', ncol=2)
ax[1].set_xlabel('MJD')
ax[1].set_ylabel('PSF Diff Flux')
ax[1].set_xlim([60809, 60827])
plt.tight_layout()
plt.show()

Figure 1: The PSF forced photometry difference-image light curve: for all dates (left) and a zoom-in on the sequence of earliest dates (right).

The flux errors are retrieved in the query, but not plotted as they're too small to be seen on the flux light curve. Print the mean flux error to show that it is much smaller than the flux values.

In [22]:

print(np.mean(results['psfDiffFluxErr']))

735.7309322033898

Clean up.

In [23]:

job.delete()
del query, results

3.4. Butler¶

TAP is the recommended way to access the source table, but the butler can also be used.

Show that the dimensions for the dia_object_forced_source table are the skymap's tract and patch, and that both are required.

In [24]:

butler.get_dataset_type('dia_object_forced_source')

Out[24]:

DatasetType('dia_object_forced_source', {skymap, tract, patch}, ArrowAstropy)

In [25]:

butler.get_dataset_type('dia_object_forced_source').dimensions.required

Out[25]:

{skymap, tract, patch}

3.4.1. Demo query¶

Query the Butler for the dia_object_forced_source tables for patches that overlap the coordinates of the targeted object, and show that there is only one overlapping patch.

In [26]:

ra_targ = 151.592579
dec_targ = 1.158333
refs = butler.query_datasets("dia_object_forced_source",
                             where="patch.region OVERLAPS POINT(:ra, :dec)",
                             bind={"ra": ra_targ, "dec": dec_targ})
print(len(refs))

1

Print the dataId for the one dataset ref.

In [27]:

refs[0].dataId

Out[27]:

{skymap: 'lsst_cells_v2', tract: 9571, patch: 72}

Define the columns to retrieve from the Butler-based table.

In [28]:

col_list = ['diaObjectId', 'band', 'psfDiffFlux', 'psfDiffFluxErr']

Notice: table joins to also retrieve the MJD are not possible with the Butler like they are with TAP, making Butler-based queries of ForcedSourceOnDiaObject not as useful for light curves. Furthermore, Butler queries with small spatial constraints or specific object identifiers cannot be made - only full tables are returned. The TAP services is better for individual light curves.

Get the data from the butler for the patch in the list of returned dataset refs, and print the table length.

In [29]:

results = butler.get(refs[0], parameters={'columns': col_list})
print(len(results))

692675

Option to display the results.

In [30]:

# results

Without a small constraint on the region as in Section 3.2.1, data for many more unique objects has been returned.

Print the number of unique diaObjects for which forced photometry was returned by this Butler query.

In [31]:

values = np.unique(results['diaObjectId'])
print('Number of unique diaObjects in table: ', len(values))
del values

Number of unique diaObjects in table:  1861

Check that the transient is in the results table, and print the same table of psfDiffFlux statistics as in Section 3.2.1.

In [32]:

tx = np.where(results['diaObjectId'] == 786352299365629972)[0]
print('Number of forced sources for the diaObject: ', len(tx))

print('%8s %8s %8s %8s %8s %8s' % ('filter', 'number', 'min', 'max', 'mean', 'std'))
for filt in filter_names:
    fx = np.where(results['band'][tx] == filt)[0]
    print('%8s %8i %8.1f %8.1f %8.1f %8.1f' %
          (filt, len(fx),
           np.min(results['psfDiffFlux'][tx[fx]]),
           np.max(results['psfDiffFlux'][tx[fx]]),
           np.mean(results['psfDiffFlux'][tx[fx]]),
           np.std(results['psfDiffFlux'][tx[fx]])))
    del fx
del tx

Number of forced sources for the diaObject:  295
  filter   number      min      max     mean      std
       u       42  -6961.7  10097.8   -664.8   5078.9
       g       35 -30664.9  22065.3   -166.6  15793.8
       r       76 -66059.0 111824.0  41157.9  54591.6
       i       66 -66221.2  66221.5   4722.0  46830.3
       z       45 -142310.7  59404.6   4780.0  50497.7
       y       31 -137202.2   2805.0  -9686.1  27405.8

In [33]:

del refs, results

In [ ]: