301.4. Scheduler information

301.4. Scheduler information for visits¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 2
Container Size: Large
LSST Science Pipelines version: r30.0.10
Last verified to run: 2026-08-11
Repository: github.com/lsst/tutorial-notebooks
DOI: 10.11578/rubin/dc.20250909.20

Learning objective: How to extract scheduler information from the visit dimension records in the Butler.

LSST data products: visit dimension records

Packages: lsst.daf.butler

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

In the Early Data Preview 2 (EDP2) Visit table (schema) does not include scheduler information such as the target name, observation reason, or science program. These values were omitted partially because they were not standardized during commissioning, not used consistently throughout the commissioning period, and are thus of limited use for EDP2.

Warning: for EDP2, when "all the visits of a particular small area or deep drilling commissioning field" are desired, it is recommended to follow the guidance in tutorial "301.1. DP2 overview" and search for visits in a given region by coordinates and offsets. Note also that these "commissioning fields" were defined only for the commissioning (pre-LSST) era; with the exception of the deep drilling fields, the LSST does not have defined fields.

However, because some limited and non-standardized scheduler information does exist and is accessible via the Butler dimension records, in order to assuage curiosity and provide guidance, this tutorial demonstrates how to retrieve and interpret the scheduler-related information.

Related tutorials: The 100-level tutorials for how to use the RSP's butler and TAP services, the 200-level tutorials for details on the visit table, and the 301-series tutorials that demonstrate how best to use the observational metadata.

1.1. Import packages¶

Import pandas, numpy, and matplotlib; skyproj for plotting; and the Rubin packages for data access with the Butler and TAP services.

In [1]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
import skyproj
from lsst.daf.butler import Butler
from lsst.rsp import RSPDiscovery

1.2. Define parameters and functions¶

Instantiate the Butler and then the TAP service.

In [2]:

butler = Butler("dp2", collections="dp2")

In [3]:

discovery = RSPDiscovery("dp2")
tap_service = discovery.get_tap_client()

Define a function to truncate a colormap. This can be used to cut off, e.g., the darkest or lightest colors, to help with data visibility.

In [4]:

def truncate_colormap(cmap, minval=0.0, maxval=1.0, n=256):
    """
    Customize a matplotlib colormap.

    Parameters
    ----------
    cmap: the matplotlib colormap
    minval: desired lower bound of colormap
    maxval: desired upper bound of colormap
    n: number of intervals (colormap resolution)

    Returns
    -------
    new_cmap: the truncated colormap
    """

    new_cmap = mcolors.LinearSegmentedColormap.from_list(
        'trunc({n},{a:.2f},{b:.2f})'.format(n=cmap.name, a=minval, b=maxval),
        cmap(np.linspace(minval, maxval, n)))
    return new_cmap

2. Retrieve visit information¶

2.1. Visit table (TAP)¶

Using the TAP service, retrieve the five columns for all rows of the Visit table: the visit identifier, the coordinates RA and Dec in degrees, the band, and the modified julian date at the midpoint of the exposure. Store the results in tap_results_df as a pandas dataframe.

In [5]:

query = """SELECT visit, ra, dec, band, expMidptMJD
        FROM dp2.Visit"""
job = tap_service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
assert job.phase == 'COMPLETED'
tap_results_df = job.fetch_result().to_table().to_pandas()
print('Number of Visit table rows: ', len(tap_results_df))

Number of Visit table rows:  28698

2.2. Visit dimension records¶

Within the Butler, "dimensions" are properties of a dataset type that can be constrained in a Butler query, such as instrument, tract, patch, visit, and detector. The "dimension records" are all the unique values for a given dimension - for example, all visits - and related qualities for that particular unique value, such as the visit properties.

2.2.1. For specific visits¶

Visit dimension records can be retrieved from the Butler by individual visit identifier number. Retrieve the records for three randomly selected visits by their visit identifier.

In [6]:

visits_list = [2025042800283, 2025042800284, 2025042800285]
visits_string = "(" + ", ".join(str(value) for value in visits_list) + ")"
query = "visit.id IN " + visits_string
visit_dimrecs = butler.query_dimension_records("visit", limit=None, where=query)

print(query)
print('Number of visit dimension records: ', len(visit_dimrecs))
print(' ')
print('Notice the order can change between query and results:')
for r, rec in enumerate(visit_dimrecs):
    print(r, rec.id)

del visits_list, visits_string, query

visit.id IN (2025042800283, 2025042800284, 2025042800285)
Number of visit dimension records:  3

Notice the order can change between query and results:
0 2025042800285
1 2025042800284
2 2025042800283

Display a single visit dimension record.

In [7]:

visit_dimrecs[0]

Out[7]:

visit:
  instrument: 'LSSTCam'
  id: 2025042800285
  day_obs: 20250428
  physical_filter: 'g_6'
  name: 'MC_O_20250428_000285'
  seq_num: 285
  exposure_time: 30.0
  target_name: 'Rubin_SV_212_-7'
  observation_reason: 'field_survey_science'
  science_program: 'BLOCK-365'
  azimuth: 327.9714778675945
  zenith_angle: 26.4855665598186
  region: ConvexPolygon([UnitVector3d(-0.8570240891474094, -0.4964478218917627, -0.13801909563527054), UnitVector3d(-0.8563861158843464, -0.49642453110839396, -0.14200530072618398), UnitVector3d(-0.8539505165805482, -0.4995056670532897, -0.1458170216863002), UnitVector3d(-0.8509515563234203, -0.5033818402425094, -0.1499605671662167), UnitVector3d(-0.8482610078742504, -0.5069244116009853, -0.15323479840804982), UnitVector3d(-0.846263025636131, -0.510339668539157, -0.152945461379292), UnitVector3d(-0.8335176013096868, -0.5314352911907763, -0.15107924934921058), UnitVector3d(-0.8314221484327073, -0.5347961220360454, -0.15076577513065878), UnitVector3d(-0.8313601827838318, -0.5348950632816482, -0.15075648496351796), UnitVector3d(-0.8292523210110355, -0.5382457263598531, -0.15043977583459908), UnitVector3d(-0.8276396507921289, -0.5417970395096628, -0.14652159026999306), UnitVector3d(-0.8259442055421382, -0.5456574432627616, -0.14168318158240362), UnitVector3d(-0.8246604107715179, -0.548705520366856, -0.13732246292973704), UnitVector3d(-0.8252923841438019, -0.5487386986289712, -0.1333353715306723), UnitVector3d(-0.8294167157876624, -0.5486939891751155, -0.10489431736327028), UnitVector3d(-0.8299433033997793, -0.548648251632541, -0.1008920666971285), UnitVector3d(-0.8324107280569694, -0.5455889256418881, -0.09708297498839172), UnitVector3d(-0.8354097765121759, -0.5417128071119741, -0.09293944221234465), UnitVector3d(-0.8380676329525523, -0.5381501287817626, -0.08964977127398868), UnitVector3d(-0.8401771347131286, -0.5347994479187236, -0.0899551711190951), UnitVector3d(-0.8550767036986998, -0.5102464072005147, -0.09215440700656866), UnitVector3d(-0.8570749971823157, -0.5068278409806087, -0.09245533414502596), UnitVector3d(-0.8587215938453885, -0.5032975531855148, -0.09637840640469889), UnitVector3d(-0.8604171272934579, -0.49943720409172204, -0.10121682779615772), UnitVector3d(-0.861669010850021, -0.4963672570529472, -0.10557491116017935), UnitVector3d(-0.8611484226108966, -0.4964031082955186, -0.1095780466576776)])
  timespan: Timespan(begin=astropy.time.Time(2460795.0, -0.27935152640565974, scale='tai', format='jd'), end=astropy.time.Time(2460795.0, -0.2789933796296296, scale='tai', format='jd'))

In [8]:

del visit_dimrecs

2.2.2. For all visits¶

Retrieve the visit dimension records for all visits from the Butler, for use in the rest of this tutorial.

In [9]:

visit_dimrecs = butler.query_dimension_records("visit", limit=None)
print('Number of visit dimension records: ', len(visit_dimrecs))

Number of visit dimension records:  29756

There are more dimension records (29756) than rows of the visit table (28698) because the visit table is limited to successfully acquired and processed visits.

Convert the visit dimension records to a pandas dataframe, keeping only the scheduler information and the visit identifier.

In [10]:

temp_df = pd.DataFrame()
temp0 = []
temp1 = []
temp2 = []
temp3 = []
for rec in visit_dimrecs:
    temp0.append(rec.id)
    temp1.append(rec.target_name)
    temp2.append(rec.observation_reason)
    temp3.append(rec.science_program)
temp_df['visit'] = np.asarray(temp0, dtype=int)
temp_df['target_name'] = np.asarray(temp1, dtype='str')
temp_df['observation_reason'] = np.asarray(temp2, dtype='str')
temp_df['science_program'] = np.asarray(temp3, dtype='str')
del temp0, temp1, temp2, temp3

Option to view the temporary dataframe.

In [11]:

# temp_df

2.3. Join Visit table and dimension records dataframes¶

Do an inner join on the two dataframes, which means keeping only the rows of the temp_df that are matched to rows of the tap_results_df. Store the result in df.

In [12]:

df = pd.merge(tap_results_df, temp_df, on='visit', how='left')

Assert that the result of the inner join has the same length as the TAP visit table.

In [13]:

assert len(df) == len(tap_results_df)
print('Number of joined dataframe rows: ', len(df))

Number of joined dataframe rows:  28698

Delete the data which is no longer needed, and continue with just the joined dataframe df.

In [14]:

del visit_dimrecs, temp_df, tap_results_df

2.4. Fix non-standard DDF names¶

As mentioned in Section 1, the scheduler metadata was not standardized during commissioning and multiple formats exist for the deep drilling field (DDF) names. Convert the target names for the DDFs to be all lowercase and use underscores instead of spaces.

In [15]:

df['target_name'] = df['target_name'].str.replace('DDF ECDFS', 'ddf_ecdfs', regex=False)
df['target_name'] = df['target_name'].str.replace('DDF EDFS_a', 'ddf_edfs_a', regex=False)
df['target_name'] = df['target_name'].str.replace('DDF EDFS_b', 'ddf_edfs_b', regex=False)
df['target_name'] = df['target_name'].str.replace('DDF ELAISS1', 'ddf_elaiss1', regex=False)
df['target_name'] = df['target_name'].str.replace('DDF XMM_LSS', 'ddf_xmm_lss', regex=False)

3. Explore the scheduler information¶

3.1. Target names¶

The target_name column is a comma-separated string list of named commissioning targets that overlap the field of view covered by the visit.

Caveat: A single visit can be associated with multiple target names.

Caveat: These target names are not necessarily: (1) standard names; (2) used consistently; (3) to be used in the future. New values for target names might be introduced as the LSST strategy evolves.

The wide-fast-deep regions covered during commissioning are:

• lowdust - low dust sky region
• dusty_plane - dusty regions of the Galactic plane
• bulgy - Galactic bulge and plane regions
• nes - north ecliptic spur
• LMC_SMC - Large and Small Magellanic Clouds
• scp - south celestial pole

Small field areas done as part of the science validation survey: Rubin_SV_212_-7, Rubin_SV_216_-17, Rubin_SV_225_-40, Rubin_SV_280_-48, Rubin_SV_300_-41, Rubin_SV_320_-15, Abell 2764, DESI SV3 R1, M49 and regions in the Virgo Cluster, Prawn nebula, Trifid and Lagoon nebulas, and regions related to observations with the New Horizons and Euclid space telescopes.

Deep drilling fields (DDFs): COSMOS, ELAIS S1, ECDFS (Extended Chandra Deep Field South), EDFS (Euclid Deep Field South), and XMM_LSS.

There were also a few target of opportunity (ToO) follow-ups of gravitational wave (GW) events.

Option to print all unique values of the target_name column.

In [16]:

# values, counts = np.unique(df['target_name'], return_counts=True)
# for value, count in zip(values, counts):
#     print('%-40s %5i' % (value, count))

Separate the comma-separated values and create a single list of unique targets as unique_target_names.

In [17]:

temp_list = []
for target_name in df['target_name']:
    names = target_name.split(',')
    for name in names:
        temp_list.append(name.replace(" ", ""))
unique_target_names = np.unique(temp_list)
unique_target_names

Out[17]:

array(['Abell_2764', 'COSMOS', 'DESI_SV3_R1', 'ELAIS_S1',
       'GW_case_large_1', 'LMC_SMC', 'M49', 'New_Horizons', 'Prawn',
       'Rubin_SV_212_-7', 'Rubin_SV_216_-17', 'Rubin_SV_225_-40',
       'Rubin_SV_280_-48', 'Rubin_SV_300_-41', 'Rubin_SV_320_-15',
       'ToO_3I_i0', 'ToO_BBH_i0', 'ToO_SSO_i0', 'Trifid-Lagoon', 'bulgy',
       'ddf_cosmos', 'ddf_ecdfs', 'ddf_edfs_a', 'ddf_edfs_b',
       'ddf_elaiss1', 'ddf_xmm_lss', 'dusty_plane', 'euclid_overlap',
       'lowdust', 'nes', 'scp'], dtype='<U16')

Option to list the number of visits associated with each unique target. The total of this list will be more than the total number of visits because visits can be associated with more than one unique target.

In [18]:

# for name in unique_target_names:
#     df_temp = df.query("target_name.str.contains(@name)")
#     print('%-20s %5i' % (name, len(df_temp)))
#     del df_temp

As an example, plot a density map of the sky location for visits of the low-dust, dusty plane, and Galactic bulge regions with target names lowdust, dusty_plane, and bulgy.

In [19]:

cmap = plt.get_cmap('Blues')
new_blues_cmap = truncate_colormap(cmap, 0.2, 0.8)
cmap = plt.get_cmap('Oranges')
new_oranges_cmap = truncate_colormap(cmap, 0.2, 0.8)
cmap = plt.get_cmap('Purples')
new_purples_cmap = truncate_colormap(cmap, 0.2, 0.8)

fig, ax = plt.subplots(figsize=(12, 8))
sp = skyproj.McBrydeSkyproj(ax=ax)
df_temp = df.query("target_name.str.contains('lowdust')")
sp.draw_hpxbin(df_temp['ra'], df_temp['dec'], nside=19, alpha=1, cmap=new_blues_cmap)
sp.draw_colorbar(label='Number of visits (lowdust)', location='right', shrink=0.3, pad=-0.07)
df_temp = df.query("target_name.str.contains('bulgy')")
sp.draw_hpxbin(df_temp['ra'], df_temp['dec'], nside=19, alpha=1, cmap=new_oranges_cmap)
sp.draw_colorbar(label='Number of visits (bulgy)', location='right', shrink=0.3, pad=-0.05)
df_temp = df.query("target_name.str.contains('dusty_plane')")
sp.draw_hpxbin(df_temp['ra'], df_temp['dec'], nside=19, alpha=1, cmap=new_purples_cmap)
sp.draw_colorbar(label='Number of visits (dusty_plane)', location='right', shrink=0.3, pad=0.01)
plt.plot(266.417, -29.008, '*', ms=20, color='yellow', mec='black')
sp.ax.set_xlabel("Right Ascension", fontsize=14)
sp.ax.set_ylabel("Declination", fontsize=14)
plt.tight_layout()
plt.show()

del new_blues_cmap, new_oranges_cmap, new_purples_cmap, cmap, df_temp, sp

Figure 1: The density of the number of visits, in any filter, with target name of dusty_plane (purples), bulgy (oranges), or lowdust (blues). Notice that bulgy does not strictly mean Galactic bulge but extends into regions of the Galactic plane; the Galactic center is marked with a yellow star for reference.

3.2. Observation reason¶

The observation_reason column records the source of the visit in the Feature Based Scheduler (FBS).

Caveat: These observation reasons are not necessarily: (1) standardized in name or meaning; (2) used consistently over commissioning; (3) to be used in the future. New values of the observation reason might be introduced as the LSST strategy evolves.

The feature-based scheduler (FBS) and Rubin operations were in the early stages when these visits were obtained, and the values of the observation_reason column exhibit diversity; explanations for every single value are not provided, but in general

• ddf - observing a Deep Drilling Field
• field_survey_science - observing a small field area
• pairs - taking observations in filter pairs
• template - building templates for difference image anaysis
• too - observing a Target of Opportunity
• twilight - observing fields at twilight

These are all part of the LSST baseline strategy.

Option to print all unique values of the observation_reason column.

In [20]:

# values, counts = np.unique(df['observation_reason'], return_counts=True)
# for value, count in zip(values, counts):
#     print('%-60s %5i' % (value, count))

Print the number of observations for ToO that were included in DP2.

In [21]:

df_temp = df.query("observation_reason.str.contains('too')")
print('Number of ToO-related visits: ', len(df_temp))

Number of ToO-related visits:  1003

3.3. Science program¶

For commissioning, the science program is a designation with no science application. It is accessible as column science_program, but it is just the word "BLOCK" and then a number. The observing block was simply the bundle of science-motivated commissioning observations used during a given era of commissioning. For example, the "BLOCK-365" was used initially for commissioning observations. Then the "BLOCK-400" programs were defined for the pre-LSST observations, and the different blocks were just configurations for different conditions (e.g., good and bad seeing). Caveat: This will not be how the feature-based scheduler works for the LSST, and the science program should not be used to constrain data for analyses. Use, e.g., the measured image quality parameters in the VisitDetector table instead.

Option to print the unique values of the science_program column.

In [22]:

# values, counts = np.unique(df['science_program'], return_counts=True)
# for value, count in zip(values, counts):
#     print('%-60s %5i' % (value, count))

To further demonstrate that this is not an informative column - just a name given to science observations on commissioning nights - show that different block numbers were adopted over time.

In [23]:

bins = np.arange((61070-60780)/10)*10 + 60780
lws = [1, 3, 1, 3, 2, 3]
als = [0.9, 0.5, 0.9, 0.5, 0.9, 0.5]
lss = ['solid', 'solid', 'dashed', 'dashed', 'dotted', 'dotted']
fig = plt.figure(figsize=(6, 4))
temp = []
values = np.unique(df['science_program'])
for v, value in enumerate(values):
    df_temp = df.query("science_program.str.contains(@value)")
    plt.hist(df_temp['expMidptMJD'], bins=bins, histtype='step',
             lw=lws[v], ls=lss[v], alpha=als[v], log=True, label=value)
plt.legend(loc='upper right', ncol=2)
plt.xlabel('MJD')
plt.ylabel('Number of visits')
plt.title('Science programs')
plt.show()

Figure 2: Distribution of visit dates by "science_program", illustrating how different "program" block names were used during the commissioning period. The point being made with this plot is that "science_program" is not a scientifically useful piece of scheduler info for EDP2.

In [ ]: