201.03. ForcedSource table

201.3. Forced Source table¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: large
LSST Science Pipelines version: r30.0.10
Last verified to run: 2026-07-22
Repository: github.com/lsst/tutorial-notebooks

Learning objective: To understand the contents of the ForcedSource table and how to access it.

LSST data products: ForcedSource

Packages: lsst.rsp, lsst.daf.butler

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

The ForcedSource table contains forced PSF measurements in individual visit_images and difference_images at the sky coordinates of all Objects.

Objects are the union set of all sources detected in the deep_coadd images or in the individual visit images (even those detected only once in one filter).

It is recommended to use the Forced Source and Difference Image Analysis (DIA) tables for light curves.

• TAP table name: dp2.ForcedSource
• Butler table name: object_forced_source
• columns: 28
• rows: 44,727,948,693

Related tutorials: The TAP and Butler data access services are demonstrated in the 100-level "How to" tutorials. There are 200-level tutorials on visit_images, difference_images and the Object table.

1.1. Import packages¶

Import standard python packages re, numpy, and matplotlib.

From the lsst package, import modules for RSPDiscovery, the Butler, and plotting.

In [1]:

import re
import numpy as np
import matplotlib.pyplot as plt
from lsst.rsp import RSPDiscovery
from lsst.daf.butler import Butler
from lsst.utils.plotting import (get_multiband_plot_colors,
                                 get_multiband_plot_symbols,
                                 get_multiband_plot_linestyles)

1.2. Define parameters and functions¶

Instantiate RSPDiscovery with the DP2 release.

In [2]:

discovery = RSPDiscovery("dp2")

Create an instance of the TAP service.

In [3]:

tap_service = discovery.get_tap_client()

Create an instance of the Rubin data Butler.

In [4]:

butler = Butler('dp2', collections="dp2")

Define the colors and symbols to represent the LSST filters in plots.

In [5]:

filter_names = ['u', 'g', 'r', 'i', 'z', 'y']
filter_colors = get_multiband_plot_colors()
filter_symbols = get_multiband_plot_symbols()
filter_linestyles = get_multiband_plot_linestyles()

2. Schema (columns)¶

To browse the table schema visit the Rubin schema browser, or use the TAP service via the Portal Aspect or as demonstrated in Section 2.1.

2.1. Retrieve table schema¶

To retrieve the table schema, define a query for the schema columns of the ForcedSource table and run the query job.

In [6]:

query = "SELECT column_name, datatype, description, unit " \
        "FROM tap_schema.columns " \
        "WHERE table_name = 'dp2.ForcedSource'"
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

Job phase is COMPLETED

Retrieve the query results and display them as an astropy table with the to_table() attribute.

In [7]:

assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
results

Out[7]:

Table length=26

column_name	datatype	description	unit
str64	str64	str512	str64
band	char	Abstract filter that is not associated with a particular instrument
detector	short	Id of the detector where this source was measured.
diff_PixelFlags_nodataCenter	boolean	Source center is outside usable region on image difference (masked NO_DATA)
invalidPsfFlag	boolean	Source has an invalid PSF.
objectId	long	Unique Object ID. Primary Key of the Object Table
parentObjectId	long	Unique ObjectId of the parent of the ObjectId in context of the deblender.
patch	long	Skymap patch ID
pixelFlags_bad	boolean	Bad pixel in the Source footprint
pixelFlags_cr	boolean	Cosmic ray in the Source footprint
pixelFlags_crCenter	boolean	Cosmic ray in the Source center
pixelFlags_edge	boolean	Source is on the edge of an exposure region (masked EDGE)
pixelFlags_interpolated	boolean	Interpolated pixel in the Source footprint
pixelFlags_interpolatedCenter	boolean	Interpolated pixel in the Source center
pixelFlags_nodata	boolean	Source is outside usable exposure region (masked NO_DATA)
pixelFlags_saturated	boolean	Saturated pixel in the Source footprint
pixelFlags_saturatedCenter	boolean	Saturated pixel in the Source center
pixelFlags_suspect	boolean	Sources footprint includes suspect pixels
pixelFlags_suspectCenter	boolean	Sources center is close to suspect pixels
psfDiffFlux	float	Flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfDiffFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfDiffFlux_flag	boolean	Failure to derive linear least-squares fit of psf model forced on the image difference
psfFlux	float	Flux derived from linear least-squares fit of psf model forced on the calexp	nJy
psfFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the calexp	nJy
psfFlux_flag	boolean	Failure to derive linear least-squares fit of psf model forced on the calexp
tract	long	Skymap tract ID
visit	long	Id of the visit where this source was measured.

The table displayed above has been truncated.

Option to print every column name as a list.

In [8]:

# for col in results['column_name']:
#     print(col)

Option to use the regular expressions package re to search for column names that contain the string temp.

In [9]:

# temp = 'Err'
# temp = 'Flux'
# temp = 'psf'
# for col in results['column_name']:
#     if re.search(temp, col):
#         print(col)

Delete the job, but not the results.

In [10]:

del query
job.delete()

2.2. Key columns¶

The few most commonly used columns of the ForcedSource table.

2.2.1. Object Id¶

The objectId that identifies the row of the Object table associated with the ForcedSource.

• objectId

2.2.2. Observation¶

The band, visit, and detector in which the ForcedSource was measured.

• band, visit, detector

2.2.3. Photometry¶

A forced fit of the Point Spread Function (PSF) at the object's coordinates in each individual visit_image and difference_image.

• psfFlux, psfFluxErr
• psfDiffFlux, psfDiffFluxErr

Forced fluxes can be negative.

Forced PSF photometry on an image in which the background is slightly oversubtracted at the coordinates can be negative. Photometry on differences can be negative as well.

It is not recommended to convert forced fluxes to magnitudes, but if they are positive, the fluxes ($f$) are in nanoJanskys and the corresponding AB magnitude ($m$) is:

$m = -2.5\log(f) + 31.4$

2.2.4. Flags¶

A variety of flags indicating whether pixels that are saturated, or affected by cosmic rays, contributed to the source's measurements.

• pixelFlags_*

2.3. Descriptions and units¶

For a subset of the key columns show the table of their descriptions and units.

In [11]:

col_list = set(['objectId', 'band', 'visit', 'detector',
                'psfFlux', 'psfFluxErr', 'psfDiffFlux', 'psfDiffFluxErr'])
tx = [i for i, item in enumerate(results['column_name']) if item in col_list]
results[tx]

Out[11]:

Table length=8

column_name	datatype	description	unit
str64	str64	str512	str64
band	char	Abstract filter that is not associated with a particular instrument
detector	short	Id of the detector where this source was measured.
objectId	long	Unique Object ID. Primary Key of the Object Table
psfDiffFlux	float	Flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfDiffFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the image difference	nJy
psfFlux	float	Flux derived from linear least-squares fit of psf model forced on the calexp	nJy
psfFluxErr	float	Uncertainty on the flux derived from linear least-squares fit of psf model forced on the calexp	nJy
visit	long	Id of the visit where this source was measured.

Clean up.

In [12]:

del col_list, tx, results

3. Data access¶

The ForcedSource table is available via the TAP service and the Butler.

Recommended access method: TAP.

3.1. Advisory: retrieve data by objectId¶

Avoid full-table queries. Query by objectId or visit on the ForcedSource table.

The ForcedSource table is a large, inclusive union set of forced measurements made in all the visit and difference images at the locations of all Objects.

There are two main use-cases for queries on the ForcedSource table.

Case A: Return all forced photometry for a given Object.

• Step 1. Query the Object table to get the objectId.
• Step 2. Retrieve all forced photometry by objectId.

Case B: Return all forced photometry measurements from a given visit_image.

• Step 1. Query the Visit table to get the visit identifier.
• Step 2. Retrieve all forced photometry by visit.

These two steps can be performed simultaneously with table joins as shown in Section 3.2.2.

The future data release ForcedSource tables will contain trillions of rows. It is important to develop good habits in making efficient queries.

3.2. TAP (Table Access Protocol)¶

The ForcedSource table is stored in Qserv and accessible via the TAP services using ADQL queries.

3.2.1. Demo query¶

For this example, start with the coordinates of a known RR Lyrae variable star in the DP2.

• RA = 226.7244283498404
• Dec = -40.30405425755786

First get the objectId for this target. Search the Object table with a radius of 2 arcseconds.

In [13]:

obj_ra = 226.7244283498404
obj_dec = -40.30405425755786

query = "SELECT objectId, coord_ra, coord_dec " \
        "FROM dp2.Object " \
        "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), " \
        f"CIRCLE('ICRS', {obj_ra}, {obj_dec}, 0.00056)) = 1"
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

Job phase is COMPLETED

In [14]:

assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

1

In [15]:

results

Out[15]:

Table length=1

objectId	coord_ra	coord_dec
	deg	deg
int64	float64	float64
743582465276261182	226.72442217825807	-40.30405896997592

Then retrieve the forced photometry for that objectId.

In [16]:

query = "SELECT objectId, band, visit, detector, " \
        "psfFlux, psfFluxErr, psfDiffFlux, psfDiffFluxErr " \
        "FROM dp2.ForcedSource " \
        "WHERE objectId = " + str(results['objectId'][0])
print(query)
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

SELECT objectId, band, visit, detector, psfFlux, psfFluxErr, psfDiffFlux, psfDiffFluxErr FROM dp2.ForcedSource WHERE objectId = 743582465276261182

Job phase is COMPLETED

In [17]:

assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

887

Option to display the table.

In [18]:

# results

As an example, calculate and print the minimum, mean, maximum, and standard deviation in PSF flux in the visit_images per filter.

In [19]:

for filt in filter_names:
    fx = np.where(results['band'] == filt)[0]
    print('%s %3i %8.1f %8.1f %8.1f %8.1f' %
          (filt, len(fx),
           np.min(results['psfFlux'][fx]), np.mean(results['psfFlux'][fx]),
           np.max(results['psfFlux'][fx]), np.std(results['psfFlux'][fx])))

u 150    263.8  51372.2 114398.0  17173.4
g 261  91853.0 128989.2 277972.0  45645.0
r 164   1076.6 155040.9 214659.0  30697.0
i 171   3004.3 190003.8 284502.0  45528.0
z  95 149424.0 210833.5 269934.0  31401.6
y  46 156495.0 218527.8 275605.0  29148.9

Clean up.

In [20]:

job.delete()
del query, results

3.2.2. Joinable tables¶

The ForcedSource table can be joined to:

• the Visit table on column visit
• the Object table on column objectId

The Visit table contains information about the observation, such as the date and time.

The Object table contains measurements made on the deep_coadd image.

To plot a light curve, join the ForcedSource and Visit tables.

To the query used in Section 3.2.1, add a table join to the Visit table and retrieve the column expMidptMJD. Change the WHERE statement to select all rows of the ForcedSource catalog for the objectId identified in the previous section. This query would also work the same with a CONTAINS(POINT(), CIRCLE()) statement to do a spatial query of the Object table.

In [21]:

query = "SELECT fs.objectId, fs.band, fs.visit, fs.detector, " \
        "fs.psfFlux, fs.psfFluxErr, fs.psfDiffFlux, fs.psfDiffFluxErr, "\
        "v.expMidptMJD, " \
        "o.u_psfFlux, o.g_psfFlux, o.r_psfFlux, o.i_psfFlux, o.z_psfFlux, o.y_psfFlux " \
        "FROM dp2.ForcedSource AS fs " \
        "JOIN dp2.Visit AS v ON fs.visit = v.visit " \
        "JOIN dp2.Object AS o ON o.objectId = fs.objectId " \
        "WHERE o.objectId = 743582465276261182"
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

Job phase is COMPLETED

Fetch the results as an astropy table.

In [22]:

assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

887

Option to display the results table.

In [23]:

# results

Plot the light curve for this object, as measured in the visit_image and difference_image.

In [24]:

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))
for filt in filter_names:
    fx = np.where(results['band'] == filt)[0]
    if len(fx) > 0:
        ax1.axhline(results[filt+'_psfFlux'][fx[0]], color=filter_colors[filt],
                    ls=filter_linestyles[filt], alpha=0.2, label=filt)
        ax1.plot(results['expMidptMJD'][fx],
                 results['psfFlux'][fx],
                 filter_symbols[filt], ms=5, mew=0, alpha=0.7,
                 color=filter_colors[filt])
        ax2.plot(results['expMidptMJD'][fx],
                 results['psfDiffFlux'][fx],
                 filter_symbols[filt], ms=5, mew=0, alpha=0.7,
                 color=filter_colors[filt], label=filt)
ax1.set_xlabel('MJD')
ax1.set_ylabel('PSF Flux')
ax1.legend(loc='upper right', ncol=2)
ax2.set_xlabel('MJD')
ax2.set_ylabel('PSF Diff Flux')
ax2.legend(loc='upper right', ncol=2)
plt.tight_layout()
plt.show()

Figure 1: At left, the PSF forced photometry light curve on the visit_images for the object. Horizontal lines mark the object's flux in the deep_coadd image. At right, the PSF forced photometry light curve on the difference_images for the object.

Clean up.

In [25]:

job.delete()
del query, results

3.4. Butler¶

TAP is the recommended way to access the forced source table, but the Butler can also be used.

Show that the dimensions for the object_forced_source table are the skymap's tract and patch, and that both are required.

In [26]:

butler.get_dataset_type('object_forced_source')

Out[26]:

DatasetType('object_forced_source', {skymap, tract, patch}, ArrowAstropy)

In [27]:

butler.get_dataset_type('object_forced_source').dimensions.required

Out[27]:

{skymap, tract, patch}

3.4.1. Demo query¶

Retrieve all dataset_refs for object_forced_source tables for patches that overlap the coordinates that were also used in Section 3.2.

In [28]:

ra = obj_ra
dec = obj_dec

refs = butler.query_datasets("object_forced_source",
                             where="patch.region OVERLAPS POINT(:ra, :dec)",
                             bind={"ra": ra, "dec": dec})

Print the number of dataset references returned.

In [29]:

print(len(refs))

2

Print the dataIds of the refs.

In [30]:

for ref in refs:
    print(ref.dataId)

{skymap: 'lsst_cells_v2', tract: 3346, patch: 80}
{skymap: 'lsst_cells_v2', tract: 3347, patch: 89}

Define the columns to retrieve.

In [31]:

col_list = ['objectId', 'coord_ra', 'coord_dec',
            'band', 'psfFlux', 'psfFluxErr']

Get the data from the Butler for the second patch in the list of returned dataset refs.

In [32]:

results = butler.get(refs[1],
                     parameters={'columns': col_list})
print(len(results))

3958686

Option to display the results.

In [33]:

# results

Without a small constraint on the region as in Section 3.2.1, data for many more unique objects has been returned.

In [34]:

values = np.unique(results['objectId'])
print(len(values))
del values

4471

Check that the variable star object is in the results table.

In [35]:

tx = np.where(results['objectId'] == 743582465276261182)[0]
print(len(tx))

887

Print the same table of psfFlux statistics as in Section 3.2.1.

In [36]:

for filt in filter_names:
    fx = np.where(results['band'][tx] == filt)[0]
    print('%s %3i %8.1f %8.1f %8.1f %8.1f' %
          (filt, len(fx),
           np.min(results['psfFlux'][tx[fx]]),
           np.mean(results['psfFlux'][tx[fx]]),
           np.max(results['psfFlux'][tx[fx]]),
           np.std(results['psfFlux'][tx[fx]])))

u 150    263.8  51372.2 114397.7  17173.4
g 261  91853.0 128989.2 277972.0  45645.0
r 164   1076.6 155041.0 214659.0  30697.1
i 171   3004.3 190003.8 284502.4  45528.1
z  95 149423.9 210833.5 269933.8  31401.6
y  46 156495.0 218527.8 275605.1  29148.9

In [37]:

del ra, dec, refs, col_list
del results, tx, fx

In [ ]: