MLG updates to JB input

Author: MelissaGraham

04_Intro_to_Butler.ipynb

@@ -392,9 +392,13 @@
     "\n",
     "The following examples show how to query for data sets that include a desired coordinate and observation date.\n",
     "\n",
-    "Above, we can see that for visit 971990, the (RA,Dec) are (70.37770,-37.1757) and the observation date is 20251201\n",
+    "##### Temporal queries\n",
+    "\n",
+    "Above, we can see that for visit 971990, the (RA,Dec) are (70.37770,-37.1757) and the observation date is 20251201.\n",
     "But these are just human-readable summaries of the more precise spatial and temporal information stored in the registry, which are represented in Python by `Timespan` and `Region` objects, respectively.\n",
-    "`DimensionRecord` objects that represent spatial or temporal concepts (a `visit` is both) have these objects attached to them:"
+    "`DimensionRecord` objects that represent spatial or temporal concepts (a `visit` is both) have these objects attached to them.\n",
+    "\n",
+    "Retrieve the `DimensionRecord` for a visit and show its timespan and region."
    ]
   },
   {
@@ -404,15 +408,17 @@
    "outputs": [],
    "source": [
     "(record,) = registry.queryDimensionRecords('visit', visit=971990)\n",
+    "\n",
     "print(record.timespan)\n",
+    "print(' ')\n",
     "print(record.region)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If the timespan or spatial region that's being used as a query constraint is already associated with a data ID in the database, spatial and temporal overlap constraints are automatic.\n",
+    "If the timespan or spatial region that are being used as query constraints are already associated with a data ID in the database, the spatial and temporal overlap constraints are automatic.\n",
     "For example, if we query for `deepCoadd` datasets with a `visit`+`detector` data ID, we'll get just the ones that overlap that observation and have the same band (because a visit implies a band):"
    ]
   },
@@ -430,7 +436,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "To query for dimension records or datasets that overlap an arbitrary time range, we can use the `bind` argument to pass times through to `where`; we'll use this to look for visits within one minute of this one on either side:"
+    "To query for dimension records or datasets that overlap an arbitrary time range, we can use the `bind` argument to pass times through to `where`.\n",
+    "Use this, along with [astropy.time](https://docs.astropy.org/en/stable/time/index.html), to look for visits within one minute of this one on either side."
    ]
   },
   {
@@ -454,17 +461,20 @@
     "Using `bind` to define an alias for a variable saves us from having to string-format the times into the `where` expression.\n",
     "Unfortunately, there is a bug in `queryDatasets` that prevents `bind` from working there (fixed in `w_2021_25`).\n",
     "\n",
-    "A `Timespan` can have a `begin` or `end` of `None` if it is unbounded on that side."
+    "Note that a `dafButler.Timespan` will accept a `begin` or `end` value that is equal to `None` if it is unbounded on that side."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Arbitrary spatial queries are not supported, but we do have set of dimensions that correspond to different levels of the HTM (hierarchical triangular mesh) pixelization of the sky.\n",
+    "##### Spatial queries\n",
     "\n",
-    "So one can transform a region or point into one or more HTM IDs, and then seach using that as a spatial data ID.\n",
-    "The `lsst.sphgeom` library is what backs our region objects, and we can also use it to find the HTM ID for a point."
+    "Arbitrary spatial queries are not supported, but we do have set of dimensions that correspond to different levels of the HTM (hierarchical triangular mesh) pixelization of the sky ([HTM primer](http://www.skyserver.org/htm/)).\n",
+    "The process is to transform a region or point into one or more HTM identifiers (HTM IDs), and then create a query using the HTM ID as the spatial data ID.\n",
+    "The `lsst.sphgeom` library supports region objects and HTM pixelization in the LSST Science Pipelines.\n",
+    "\n",
+    "Import the `lsst.sphgeom` package, initialize a sky pixelization to level 10 (the level at which one sky pixel is about five arcmin across), and find the HTM ID for a desired sky coordinate."
    ]
   },
   {
@@ -475,7 +485,7 @@
    "source": [
     "import lsst.sphgeom\n",
     "\n",
-    "pixelization = lsst.sphgeom.HtmPixelization(7)"
+    "pixelization = lsst.sphgeom.HtmPixelization(10)"
    ]
   },
   {
@@ -486,18 +496,43 @@
    "source": [
     "htm_id = pixelization.index(\n",
     "    lsst.sphgeom.UnitVector3d(\n",
-    "        lsst.sphgeom.LonLat.fromDegrees(70.37699524983329, -37.17573628348882)\n",
+    "        lsst.sphgeom.LonLat.fromDegrees(70.376995, -37.175736)\n",
     "    )\n",
     ")\n",
+    "\n",
+    "# Obtain and print the scale to provide a sense of the size of the sky pixelization being used\n",
     "scale = pixelization.triangle(htm_id).getBoundingCircle().getOpeningAngle().asDegrees()*3600\n",
     "print(f'HTM ID={htm_id} at level={pixelization.getLevel()} is a ~{scale:0.2}\" triangle.')"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "visits = registry.queryDimensionRecords(\"visit\", htm20=htm_id,\n",
+    "                                        where=\"visit.timespan OVERLAPS my_timespan\",\n",
+    "                                        bind={\"my_timespan\": timespan})\n",
+    "exposures = registry.queryDimensionRecords(\"exposure\", htm20=htm_id,\n",
+    "                                           where=\"visit.timespan OVERLAPS my_timespan\",\n",
+    "                                           bind={\"my_timespan\": timespan})\n",
+    "detectors = registry.queryDimensionRecords(\"detector\", htm20=htm_id,\n",
+    "                                           where=\"visit.timespan OVERLAPS my_timespan\",\n",
+    "                                           bind={\"my_timespan\": timespan})\n",
+    "\n",
+    "for visit, exposure, detector in zip(visits, exposures, detectors):\n",
+    "    print(visit.id, visit.timespan, visit.physical_filter, exposure.tracking_ra, exposure.tracking_dec, detector.id)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And we can use that to query for (e.g.) the set of all `src` data products that overlap this point in i:"
+    "Thus, with the above query, we have uniquely identified the visit and detector for our desired temporal and spatial constraints.\n",
+    "Note that if a smaller HTM level is used (like 7), which is a larger sky pixel (~2200 arcseconds), the above query will return many more visits and detectors which overlap with that larger region. Try it and see!\n",
+    "\n",
+    "Note that queries using the HTM ID can also be used to, e.g., find the set of all `src` catalog data products that overlap this point in i."
    ]
   },
   {
@@ -516,15 +551,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "Why is does that search take tens of seconds?\n",
     "The butler's spatial reasoning is designed to work well for regions the size of full data products, like detector- or patch-level images and catalogs, and it's a poor choice for object-scale searches.\n",
-    "The query above is slow in large part because it actually searches for all `src` datasets that overlap the much larger htm7 pixel (about a degree on a side), and then filters the results down to the htm20 pixel in Python."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "That said, it's something we'll use frequently below, so it will be useful to wrap this in a function:"
+    "The above search is slow in part because `queryDatasets` searches for all `src` datasets that overlap a larger region and then filters the results down to the specified HTM ID pixel.\n",
+    "\n",
+    "Options for exploring and retrieving catalog data with the Butler is covered in more depth in Section 5."
    ]
   },
   {