102.6. Catalog access with LSDB#

102_6_Catalog_access_with_LSDB

102.6. Catalog access with LSDB¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: Large
LSST Science Pipelines version: v30.0.10
Last verified to run: 2026-07-27
Repository: github.com/lsst/tutorial-notebooks

Learning objective: How to access Rubin data in LSDB format.

LSST data products: Object, DiaObject, DiaSource, ForcedSource, and ForcedSourceOnDiaObject

Packages: lsdb

Credit: Originally developed by the Rubin Community Science team with input from Neven Caplar, Tianqing Zhang, and Olivia Lynn. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

LSDB is an open-source Python framework that enables fast all-sky cross-matching, bulk application of user-defined functions, and simplified analysis of time-domain (light curve) data. Built on top of Dask, LSDB uses the HATS (Hierarchical Adaptive Tiling Scheme) data format (HEALPix-sharded Parquet), which provides an efficient, spatially-indexed format for storing catalog data.

LSDB also hosts many other datasets stored in the HATS format beyond the DP2 catalogs shown in this tutorial, but the DP2 HATS catalogs are only available on the Rubin Science Platform (RSP). Find the full list of LSDB-hosted datasets at data.lsdb.io.

Note: This notebook is intended only as a simple tutorial on LSDB DP2 catalogs. For more detailed examples and advanced use cases, see the full set of LSDB tutorials at LSDB tutorials.

References:

Descriptions of LSDB-formatted Rubin data: https://data.lsdb.io/
LSDB documentation: docs.lsdb.io
Working with Rubin Data using LSDB
LSDB hackathon at the Rubin Community Workshop 2025

Related tutorials: The 200-level tutorials on the Object, DiaObject, DiaSource, ForcedSource, and ForcedSourceOnDiaObject catalogs. The 300-level tutorial on how to access photometric redshifts in LSDB-formatted files.

1.1. Import packages¶

Import the LSDB package to work with LSDB-formatted files, upath for handling local and remote file paths uniformly, and matplotlib.pyplot for visualization.

In [1]:

import lsdb
import astropy.units as u
from astropy.coordinates import SkyCoord
from upath import UPath
import matplotlib.pyplot as plt
from lsst.utils.plotting import (get_multiband_plot_colors,
                                 get_multiband_plot_symbols)

Define the filter names, colors, and symbols to use when plotting.

In [2]:

filter_colors = get_multiband_plot_colors()
filter_names = filter_colors.keys()
filter_symbols = get_multiband_plot_symbols()

Set the base path to the LSDB-formatted DP2 data on the RSP.

In [3]:

base_path = UPath("/rubin/lsdb_data/dp2")

2. Access the LSDB DP2 catalogs¶

The LSDB DP2 read-only catalogs available on the Rubin Science Platform at data.lsst.cloud are located in the directory /rubin/lsdb_data.

Two of them are HATS collections (a main catalog packaged together with a margin cache and ID-based index tables):

object_collection: the Object table, with the forced photometry nested in an objectForcedSource column
dia_object_collection: the DiaObject table, with diaSource and diaObjectForcedSource nested columns

The third is a standalone catalog:

object_photoz: the photometric redshift estimates table (see Section 2.3, below)

Note that, unlike DP1 in the Rubin Science Platform, there is no object_collection_lite catalog for DP2.

2.1. object_collection¶

This LSDB-formatted catalog is the same as the DP2 Object table but with additional columns, <f>_psfMag and <f>_psfMagErr, which are the corresponding <f>_psfFlux columns converted to magnitudes (for each filter, <f>, in $ugrizy$).

Column descriptions for the DP2 tables.

Nested columns also have additional columns such as psfMag and psfMagErr (see section 2.1.2 below).

2.1.1. Load and display the catalog¶

In [4]:

object_cat = lsdb.open_catalog(base_path / "object_collection")

In [5]:

object_cat

Out[5]:

lsdb Catalog object_lc:

	coord_dec	coord_decErr	coord_ra	coord_raErr	g_psfFlux	g_psfFluxErr	g_psfMag	g_psfMagErr	i_psfFlux	i_psfFluxErr	i_psfMag	i_psfMagErr	objectId	patch	r_psfFlux	r_psfFluxErr	r_psfMag	r_psfMagErr	refBand	shape_flag	shape_xx	shape_xy	shape_yy	tract	u_psfFlux	u_psfFluxErr	u_psfMag	u_psfMagErr	y_psfFlux	y_psfFluxErr	y_psfMag	y_psfMagErr	z_psfFlux	z_psfFluxErr	z_psfMag	z_psfMagErr	objectForcedSource
npartitions=8840
Order: 8, Pixel: 10240	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	string[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	nested<band: [string], coord_dec: [double], co...
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 5, Pixel: 12283	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 5, Pixel: 12284	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...

37 out of 1261 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

This catalog has an estimated size of 1.5 TB

Lazily loaded catalogs: note the message under the displayed table above, that all of the columns have been loaded "lazily". This is always the default for LSDB catalogs, and it means that only the metadata is loaded at first. This way, LSDB can plan how tasks will be executed in the future without actually doing any computation.

Order is the HEALPix resolution, and Pixel is the HEALPix index of the specific sky patch.

Optional: uncomment the following cell and press "tab" to browse the available methods on an LSDB catalog like object_cat.

In [6]:

# object_cat.

Additionally, a table of catalog methods and descriptions can be viewed on the Catalog page in the LSDB API Reference.

2.1.2. Show column names¶

Display the subset of 37 columns that are lazily loaded by default.

In [7]:

object_cat.columns

Out[7]:

Index(['coord_dec', 'coord_decErr', 'coord_ra', 'coord_raErr', 'g_psfFlux',
       'g_psfFluxErr', 'g_psfMag', 'g_psfMagErr', 'i_psfFlux', 'i_psfFluxErr',
       'i_psfMag', 'i_psfMagErr', 'objectId', 'patch', 'r_psfFlux',
       'r_psfFluxErr', 'r_psfMag', 'r_psfMagErr', 'refBand', 'shape_flag',
       'shape_xx', 'shape_xy', 'shape_yy', 'tract', 'u_psfFlux',
       'u_psfFluxErr', 'u_psfMag', 'u_psfMagErr', 'y_psfFlux', 'y_psfFluxErr',
       'y_psfMag', 'y_psfMagErr', 'z_psfFlux', 'z_psfFluxErr', 'z_psfMag',
       'z_psfMagErr', 'objectForcedSource'],
      dtype='object')

Optional: uncomment the cell below to display the names of a larger subset of the 1261 columns from the Object catalog.

In [8]:

# object_cat.all_columns

Search for column names that contain a string, such as psfMag (i.e., the columns that contain the PSF fluxes converted to magnitudes).

In [9]:

search_string = 'psfMag'
for col in object_cat.all_columns:
    if search_string in col:
        print(col)

u_psfMag
u_psfMagErr
g_psfMag
g_psfMagErr
r_psfMag
r_psfMagErr
i_psfMag
i_psfMagErr
z_psfMag
z_psfMagErr
y_psfMag
y_psfMagErr

Load only selected columns

In [10]:

use_columns = ['coord_dec', 'coord_decErr', 'coord_ra', 'coord_raErr',
               'g_psfFlux', 'g_psfFluxErr', 'g_psfMag', 'g_psfMagErr']

In [11]:

object_cat_selected_columns = lsdb.open_catalog(base_path / "object_collection",
                                                columns=use_columns)

In [12]:

object_cat_selected_columns.columns

Out[12]:

Index(['coord_dec', 'coord_decErr', 'coord_ra', 'coord_raErr', 'g_psfFlux',
       'g_psfFluxErr', 'g_psfMag', 'g_psfMagErr'],
      dtype='object')

2.1.3. Execute a cone search¶

Cone searches are supported and defined by a center (ra, dec), in degrees, and a radius r, in arcseconds. Execute a cone search on the object catalog using the coordinates (in degrees) of the Extended Chandra Deep Field South (ECDFS) field, with a radius of 0.1 deg.

In [13]:

ra_ecdfs = 53.16
dec_ecdfs = -28.10

In [14]:

object_cat_ecdfs = object_cat.cone_search(ra=ra_ecdfs, dec=dec_ecdfs,
                                          radius_arcsec=0.1 * 3600.0)

This table contains only 2 partitions, compared to the 8,840 described in Section 2.1.1, due to the 0.1-degree spatial restriction.

In [15]:

object_cat_ecdfs

Out[15]:

lsdb Catalog object_lc:

	coord_dec	coord_decErr	coord_ra	coord_raErr	g_psfFlux	g_psfFluxErr	g_psfMag	g_psfMagErr	i_psfFlux	i_psfFluxErr	i_psfMag	i_psfMagErr	objectId	patch	r_psfFlux	r_psfFluxErr	r_psfMag	r_psfMagErr	refBand	shape_flag	shape_xx	shape_xy	shape_yy	tract	u_psfFlux	u_psfFluxErr	u_psfMag	u_psfMagErr	y_psfFlux	y_psfFluxErr	y_psfMag	y_psfMagErr	z_psfFlux	z_psfFluxErr	z_psfMag	z_psfMagErr	objectForcedSource
npartitions=2
Order: 7, Pixel: 143740	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	string[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	nested<band: [string], coord_dec: [double], co...
Order: 7, Pixel: 143742	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...

37 out of 1261 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

This catalog has an estimated size of 330.0 MB

In [16]:

object_cat_ecdfs.head(3)

Out[16]:

coord_dec

coord_decErr

coord_ra

coord_raErr

g_psfFlux

g_psfFluxErr

g_psfMag

g_psfMagErr

i_psfFlux

i_psfFluxErr

i_psfMag

i_psfMagErr

objectId

patch

r_psfFlux

r_psfFluxErr

r_psfMag

r_psfMagErr

refBand

shape_flag

shape_xx

shape_xy

shape_yy

tract

u_psfFlux

u_psfFluxErr

u_psfMag

u_psfMagErr

y_psfFlux

y_psfFluxErr

y_psfMag

y_psfMagErr

z_psfFlux

z_psfFluxErr

z_psfMag

z_psfMagErr

objectForcedSource

2528713655233233018

-28.197422

0.000059

53.185212

0.000060

256.604614

48.610924

25.376839

0.208196

179.061340

51.245422

25.767496

0.319652

755368886328655590

4

209.812103

43.972267

25.595423

0.230970

u

False

6.486713

3.180672

3.125711

5063

497.289703

149.751480

24.658476

0.337412

834.247314

766.329834

24.096764

1.715369

175.250366

108.331894

25.790854

0.783916

band	coord_dec	...	psfMagErr	visit
g	-28.197422	...	0.523769	2025071100809
+263 rows	...	...	...	...

2528713666530162584

-28.196704

0.000034

53.188282

0.000050

359.658905

46.214226

25.010273

0.140287

925.936401

73.264442

23.983547

0.086089

755368886328655582

4

944.091980

200.538177

23.962463

0.234191

y

True

12.425080

-0.936475

11.900757

5063

648.193054

150.862839

24.370739

0.257415

6487.844238

773.468933

21.869749

0.130058

band	coord_dec	...	psfMagErr	visit
g	-28.196704	...	0.008833	2025071100809
+260 rows	...	...	...	...

2528713666580786025

-28.196037

53.188218

5844.588867

58.671440

21.983114

0.010900

6426.307129

78.089821

21.880096

0.013194

755368886328655577

4

y

True

40.866325

35.522091

207.306259

5063

1456.542236

151.620728

23.491692

0.113432

18976.408203

776.328491

20.704466

0.044442

5951.000977

237.922333

21.963526

0.043431

band	coord_dec	...	psfMagErr	visit
g	-28.196037	...	0.010402	2025071100809
+260 rows	...	...	...	...

3 rows x 37 columns

In [17]:

object_cat_ecdfs.columns

Out[17]:

Index(['coord_dec', 'coord_decErr', 'coord_ra', 'coord_raErr', 'g_psfFlux',
       'g_psfFluxErr', 'g_psfMag', 'g_psfMagErr', 'i_psfFlux', 'i_psfFluxErr',
       'i_psfMag', 'i_psfMagErr', 'objectId', 'patch', 'r_psfFlux',
       'r_psfFluxErr', 'r_psfMag', 'r_psfMagErr', 'refBand', 'shape_flag',
       'shape_xx', 'shape_xy', 'shape_yy', 'tract', 'u_psfFlux',
       'u_psfFluxErr', 'u_psfMag', 'u_psfMagErr', 'y_psfFlux', 'y_psfFluxErr',
       'y_psfMag', 'y_psfMagErr', 'z_psfFlux', 'z_psfFluxErr', 'z_psfMag',
       'z_psfMagErr', 'objectForcedSource'],
      dtype='object')

Visualize the object distribution in the region.

In [18]:

plt.figure(figsize=(7, 5))
plt.hist2d(object_cat_ecdfs['coord_ra'], object_cat_ecdfs['coord_dec'],
           bins=50, cmap='viridis')
plt.colorbar(label="Number of Objects")
plt.xlabel("Right Ascension [deg]")
plt.ylabel("Declination [deg]")
plt.title("Sky Distribution of Objects")
plt.show()

No description has been provided for this image

Figure 1: A 2-dimensional distribution (heatmap) of the number of objects across the sky, as returned by the cone search centered on the ECDFS field.

2.1.4. Execute a query on column values¶

It is possible to filter LSDB catalogs using the .query() method.
The query expression is written as a string and follows the same syntax as Pandas .query(), which supports a subset of Python expressions for filtering DataFrames.

Select only objects with an $r$-band PSF magnitude between 16 and 24 mag.

Apply the query to the cone-search result from Section 2.1.3 rather than to the full catalog: parts of the DP2 footprint have no $r$-band coverage (all-NaN r_psfMag), so a .head() on a full-catalog query could scan through very many partitions before finding matching rows.

In [19]:

object_cat_mag_range = object_cat_ecdfs.query("r_psfMag > 16 and r_psfMag < 24")
object_cat_mag_range

Out[19]:

lsdb Catalog object_lc:

	coord_dec	coord_decErr	coord_ra	coord_raErr	g_psfFlux	g_psfFluxErr	g_psfMag	g_psfMagErr	i_psfFlux	i_psfFluxErr	i_psfMag	i_psfMagErr	objectId	patch	r_psfFlux	r_psfFluxErr	r_psfMag	r_psfMagErr	refBand	shape_flag	shape_xx	shape_xy	shape_yy	tract	u_psfFlux	u_psfFluxErr	u_psfMag	u_psfMagErr	y_psfFlux	y_psfFluxErr	y_psfMag	y_psfMagErr	z_psfFlux	z_psfFluxErr	z_psfMag	z_psfMagErr	objectForcedSource
npartitions=2
Order: 7, Pixel: 143740	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	string[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	nested<band: [string], coord_dec: [double], co...
Order: 7, Pixel: 143742	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...

37 out of 1261 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

This catalog has an estimated size of 330.0 MB

Use the .head() method to quickly inspect a few rows to check that the query worked as expected.

In [20]:

object_cat_mag_range.head(3)

Out[20]:

coord_dec

coord_decErr

coord_ra

coord_raErr

g_psfFlux

g_psfFluxErr

g_psfMag

g_psfMagErr

i_psfFlux

i_psfFluxErr

i_psfMag

i_psfMagErr

objectId

patch

r_psfFlux

r_psfFluxErr

r_psfMag

r_psfMagErr

refBand

shape_flag

shape_xx

shape_xy

shape_yy

tract

u_psfFlux

u_psfFluxErr

u_psfMag

u_psfMagErr

y_psfFlux

y_psfFluxErr

y_psfMag

y_psfMagErr

z_psfFlux

z_psfFluxErr

z_psfMag

z_psfMagErr

objectForcedSource

2528713666530162584

-28.196704

0.000034

53.188282

0.000050

359.658905

46.214226

25.010273

0.140287

925.936401

73.264442

23.983547

0.086089

755368886328655582

4

944.091980

200.538177

23.962463

0.234191

y

True

12.425080

-0.936475

11.900757

5063

648.193054

150.862839

24.370739

0.257415

6487.844238

773.468933

21.869749

0.130058

band	coord_dec	...	psfMagErr	visit
g	-28.196704	...	0.008833	2025071100809
+260 rows	...	...	...	...

2528713666708563438

-28.196761

0.000089

53.185950

0.000135

8897.466797

66.224953

21.526834

0.008081

4488.951660

84.682243

22.269638

0.020484

755368886328655578

4

4544.383789

81.056999

22.256313

0.019368

y

True

14.019876

8.406229

29.313395

5063

0.000000

153.701736

75.945480

7242.791992

775.388245

21.750235

0.116682

4033.910156

222.457184

22.385685

0.059936

band	coord_dec	...	psfMagErr	visit
g	-28.196761	...	0.006172	2025071100809
+264 rows	...	...	...	...

2528713666826076703

-28.196150

53.185886

585.894104

38.473053

24.480452

0.071398

1998.790649

68.416359

23.148081

0.037178

755368886328655580

4

2089.498291

218.568161

23.099895

0.113988

y

True

19.019653

-7.058270

18.340597

5063

146.896271

150.645676

25.982473

9176.551758

773.523926

21.493301

0.091738

band	coord_dec	...	psfMagErr	visit
g	-28.19615	...	0.006695	2025071100809
+264 rows	...	...	...	...

3 rows x 37 columns

2.1.5. Access nested light curves¶

Some LSDB catalogs have "nested" columns. These are columns which, instead of containing an array of data, contain a table.

The LSDB documentation contains more information on working with nested columns and time series data in LSDB format.

In the object_collection catalog, forced photometry is available in a nested column named objectForcedSource.

The subcolumns of objectForcedSource are a subset of the DP2 ForcedSource table columns, plus two additional subcolumns: psfMag and psfMagErr, the psfFlux and psfFluxErr columns converted to magnitudes.

Column descriptions for the DP2 ForcedSource table.

Discover which columns are nested.

In [21]:

object_cat.nested_columns

Out[21]:

['objectForcedSource']

Option to display the subcolumns in the nested column.

In [22]:

# object_cat["objectForcedSource"].dtype.column_dtypes

Extract and plot a light curve for a random object¶

Select a random object in the ECDFS region by its objectId.

Because DP2 is much larger than DP1 (about 783 million objects in 108,825 partitions), filtering by ID with .query() would scan the full catalog and take a very long time. Instead, use the id_search method, which uses the objectId index table that is packaged with the collection to read only the relevant partitions.

In [23]:

random_object = object_cat.id_search(values={"objectId": 755369573523416769})

Searching for objectId=755369573523416769

Option to show the row of the object_cat for this random object.

In [24]:

# random_object.head(1)

Extract just the objectForcedSource for this random object, use the compute method to convert it to a Pandas DataFrame (df), and extract the light curve (lc).

In [25]:

random_object_fs = random_object['objectForcedSource']
random_object_df = random_object_fs.compute()
random_object_lc = random_object_df.iloc[0]

Option to display the light curve as a table.

In [26]:

# random_object_lc

Plot the forced photometry light curve for this random object.

In [27]:

fig = plt.figure(figsize=(6, 4))
for f, filt in enumerate(filter_names):
    tx = (random_object_lc['band'] == filt)
    plt.plot(random_object_lc['midpointMjdTai'][tx], random_object_lc['psfMag'][tx],
             filter_symbols[filt], ms=5, mew=0, alpha=0.5, color=filter_colors[filt], label=filt)
plt.ylim([22, 15])
plt.legend(loc='lower left', ncol=3)
plt.xlabel('MJD')
plt.ylabel('PSF Magnitude')
plt.title('Nested Forced Photometry Light Curve')
plt.show()

Figure 2: The forced photometry light curve of a random object in the ECDFS field, extracted from a nested column. This object does not appear to be particularly variable with time.

Clean up.

In [28]:

del object_cat_selected_columns, object_cat_ecdfs, object_cat_mag_range
del random_object, random_object_fs, random_object_df, random_object_lc

2.2. dia_object_collection¶

This LSDB-formatted catalog is the same as the DP2 DiaObject table, with the DiaSource and ForcedSourceOnDiaObject records nested per object.

Column descriptions for the DP2 DiaObject table.

As with the DP2 object_collection, a subset of columns is lazily loaded by default: dec, diaObjectId, nDiaSources, ra, and tract, plus the nested light-curve columns diaSource and diaObjectForcedSource (each with a subset of their subcolumns). To load a different set of columns, pass the columns argument to open_catalog.

In [29]:

dia_object_cat = lsdb.open_catalog(base_path / "dia_object_collection")

2.2.1. Access nested light curves¶

Check which columns are nested.

In [30]:

dia_object_cat.nested_columns

Out[30]:

['diaObjectForcedSource', 'diaSource']

The subcolumns in these nested columns are mostly the same as the DP2 DiaSource and ForcedSourceOnDiaObject tables.

Column descriptions for the DP2 DiaSource table.

Column descriptions for the DP2 ForcedSourceOnDiaObject table.

There are four additional subcolumns:

psfMag and psfMagErr (in both nested columns)
scienceMag and scienceMagErr (in the diaSource nested column only)

Warning: For both nested columns, the psfMag column has been calculated from the psfFlux column, but they are not the same: in the DiaSource table psfFlux is the PSF fit flux in the difference image, but in the ForcedSourceOnDiaObject table psfFlux is the PSF forced photometry flux in the direct (or science) image. Fluxes measured on a difference image will be negative when the object is brighter in the template image than in the direct (science) image, and so typically, difference-image fluxes are not converted to magnitudes.

Option to display the subcolumns in the nested columns.

In [31]:

# dia_object_cat["diaSource"].dtype.column_dtypes

In [32]:

# dia_object_cat["diaObjectForcedSource"].dtype.column_dtypes

Extract and plot a light curve for a supernova-like transient¶

The transients detected in DP2 include named, publicly reported events. AT 2025wab is a probable supernova in the ECDFS region, discovered by the BlackGEM survey on 2025-07-11 (MJD 60867).

Find the DP2 counterpart of AT 2025wab with a small-radius cone search at its TNS-reported coordinates.

In [33]:

ra_at2025wab = 53.311667
dec_at2025wab = -29.588856
transient = dia_object_cat.cone_search(ra=ra_at2025wab, dec=dec_at2025wab, radius_arcsec=2.0)
transient.compute(progress_bar=False)[["diaObjectId", "ra", "dec", "nDiaSources"]]

Out[33]:

	diaObjectId	ra	dec	nDiaSources
_healpix_29
2528502114738878730	753899182879670304	53.311638	-29.588794	103

1 rows × 4 columns

The cone search returns exactly one DIA object, diaObjectId 753899182879670304, with over a hundred difference-image detections spanning both DP2 observing seasons of the ECDFS field.

Note: When the diaObjectId is already known, id_search (as in Section 2.1.5) is the most efficient way to retrieve it, as it uses the diaObjectId index table that is packaged with the collection.

Extract the light curve from the diaSource and diaObjectForcedSource columns.

In [34]:

transient_ds = transient["diaSource"]
transient_ds_df = transient_ds.compute()
transient_ds_lc = transient_ds_df.iloc[0]

transient_fs = transient['diaObjectForcedSource']
transient_fs_df = transient_fs.compute()
transient_fs_lc = transient_fs_df.iloc[0]

Compare light curves that use the forced difference-image flux, the detected difference-image flux, and the detected difference-image flux converted to a magnitude.

This will illustrate why it is recommended to use forced photometry fluxes for difference-image light curves.

In [35]:

fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(6, 8))
for f, filt in enumerate(filter_names):
    tx1 = transient_fs_lc["band"] == filt
    tx2 = transient_ds_lc["band"] == filt
    ax1.plot(transient_fs_lc['midpointMjdTai'][tx1]-60000, transient_fs_lc['psfDiffFlux'][tx1],
             filter_symbols[filt], ms=5, mew=0, alpha=0.5, color=filter_colors[filt], label=filt)
    ax2.plot(transient_ds_lc['midpointMjdTai'][tx2]-60000, transient_ds_lc['psfFlux'][tx2],
             filter_symbols[filt], ms=5, mew=0, alpha=0.5, color=filter_colors[filt], label=filt)
    ax3.plot(transient_ds_lc['midpointMjdTai'][tx2]-60000, transient_ds_lc['psfMag'][tx2],
             filter_symbols[filt], ms=5, mew=0, alpha=0.5, color=filter_colors[filt], label=filt)
    del tx1, tx2
ax1.set_xlim([860, 1055])
ax2.set_xlim([860, 1055])
ax3.set_xlim([860, 1055])
ax1.set_ylim([-30000, 62000])
ax2.set_ylim([-30000, 62000])
ax3.set_ylim([23.0, 19.2])
ax1.set_xlabel('MJD-60000')
ax2.set_xlabel('MJD-60000')
ax3.set_xlabel('MJD-60000')
ax1.set_ylabel('forced PSF Diff Flux [nJy]')
ax2.set_ylabel('diaSource PSF Flux [nJy]')
ax3.set_ylabel('diaSource PSF Mag')
ax2.legend(loc='upper right', ncol=2)
plt.suptitle('AT 2025wab', y=0.995)
plt.tight_layout()
plt.show()

Figure 3: Top: the forced PSF photometry on the difference image has light curve points from every observation, including the late epochs when the transient had faded. Middle: the diaSource PSF-fit photometry only has light curve points when the transient was detected with a signal-to-noise ratio $>$ 5, positive or negative, in the difference image. Note the negative-flux detections at late times, when the transient was dimmer than it appears in the $r$- and $z$-band template images. Bottom: converting difference-image fluxes to magnitudes means the observations where the difference-image flux is negative are lost.

Clean up.

In [36]:

del dia_object_cat, transient
del transient_ds, transient_ds_df, transient_ds_lc
del transient_fs, transient_fs_df, transient_fs_lc

2.3. object_photoz¶

This dataset is provided by the Rubin Commissioning Science Unit for photometric redshift.

Note: These estimates and their error bars are only very lightly validated. Treat them as provisional and use the results with caution.

The object_photoz table contains four point estimate columns per algorithm, following the format: {pz_algorithm_name}_z_{point_estimate_type}, where:

pz_algorithm_name ∈ ['bpz', 'dnf', 'fzboost', 'gpz', 'knn', 'tpz']
point_estimate_type ∈ ['mode', 'mean', 'median', 'best']
- (where eg, bpz_z_best would be bpz's recommended point estimate)

The object_photoz table also contains 1σ and 2σ credible interval bound columns per algorithm, following a similar format, {pz_algorithm_name}_z_{interval_bound}, where:

pz_algorithm_name ∈ ['bpz', 'dnf', 'fzboost', 'gpz', 'knn', 'tpz']
interval_bound ∈ ['err68_high', 'err68_low', 'err95_high', 'err95_low']

In addition, the table provides columns objectId, ra, and dec.

Differences from DP1 include the removal of algorithms cmnn and lephare, the addition of point estimate zbest, and the addition of ra and dec columns.

As mentioned in Section 1, use of the object_photoz table is demonstrated in more detail in the 300-level tutorials.

Load the photo-z table.

In [37]:

pz_cat = lsdb.open_catalog(base_path / "object_photoz")

Display the results.

In [38]:

pz_cat

Out[38]:

lsdb Catalog object_photoz:

	objectId	ra	dec	bpz_z_best	dnf_z_best	fzboost_z_best	gpz_z_best	knn_z_best	tpz_z_best
npartitions=1356
Order: 3, Pixel: 265	int64[pyarrow]	double[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]
...	...	...	...	...	...	...	...	...	...
Order: 4, Pixel: 3070	...	...	...	...	...	...	...	...	...
Order: 6, Pixel: 49136	...	...	...	...	...	...	...	...	...

9 out of 51 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

This catalog has an estimated size of 33.7 GB

Clean up.

In [39]:

del pz_cat

3. Visualize the LSDB sky partitions¶

LSDB catalogs are divided into partitions, which reflect how the LSDB-formatted files are stored. Each partition contains approximately the same number of objects, so partitions are not equal-area regions of the sky. The HATS partitioning scheme assigns smaller partitions to dense regions (for example, the Galactic bulge) and larger partitions to sparse regions, ensuring balanced row counts across files.

The plot_pixels method of a catalog object visualizes these partitions. The result is not a science-driven sky coverage map but a display of the polygonal partition boundaries. Pixel colors represent pixel sizes, with smaller pixels corresponding to regions of higher source density.

Warning: the following code cell may produce a UserWarning about small HEALPix pixels, which is OK to ignore.

In [40]:

fig = object_cat.plot_pixels(plot_title="Object Sky Partition Map")

Figure 4: An all-sky map showing the HEALPix partitions for the LSDB-formatted object_collection catalog.

Define a field of view (fov) and center in order to zoom-in on the partitions for the object_collection catalog, and re-create the plot.

Warning: The same UserWarning will show about small HEALPix pixels.

In [41]:

fov = (100 * u.deg, 120 * u.deg)
center = SkyCoord(70 * u.deg, -30 * u.deg)
fig = object_cat.plot_pixels(fov=fov, center=center,
                             plot_title="Object Sky Partition Map")

Figure 5: A zoomed-in version of Figure 4.

Clean up.

In [42]:

del object_cat

4. Learn more about LSDB¶

As mentioned in Section 1, this notebook is intended only as a simple tutorial on LSDB DP2 catalogs. For more detailed examples and advanced use cases, see the full set of LSDB tutorials.