Skip to content
This repository was archived by the owner on Apr 28, 2025. It is now read-only.

Commit 05dae9f

Browse files
MelissaGrahamMelissaGraham
authored
Merge pull request #12 from rubin-dp0/tickets/PREOPS-558
PREOPS-558: Updates to butler tutorial notebook.
2 parents ef9766c + f95456d commit 05dae9f

1 file changed

Lines changed: 178 additions & 68 deletions

File tree

‎04_Intro_to_Butler.ipynb‎

Lines changed: 178 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,13 @@
66
"source": [
77
"<img align=\"left\" src = https://project.lsst.org/sites/default/files/Rubin-O-Logo_0.png width=250> \n",
88
"<b>Introduction to the LSST data Butler</b> <br>\n",
9-
"Last verified to run on <b>TBD</b> with LSST Science Pipelines release <b>TBD</b> <br>\n",
9+
"Last verified to run on <b>Jun 17 2021</b> with LSST Science Pipelines release <b>w_2021_25</b> <br>\n",
1010
"Contact author: Alex Drlica-Wagner <br>\n",
11-
"Credit: Originally developed by Alex Drlica-Wagner in the context of the LSST Stack Club <br>\n",
11+
"Credit: Originally developed by Alex Drlica-Wagner in the context of the LSST Stack Club. <br>\n",
1212
"Target audience: All DP0 delegates. <br>\n",
1313
"Container Size: medium <br>\n",
14-
"Questions welcome at <a href=\"https://community.lsst.org/c/support/dp0\">community.lsst.org/c/support/dp0</a> <br>\n",
15-
"Find DP0 documentation and resources at <a href=\"https://dp0-1.lsst.io\">dp0-1.lsst.io</a> <br>"
14+
"Questions welcome at <a href=\"https://community.lsst.org/c/support/dp0\">community.lsst.org/c/support/dp0</a>. <br>\n",
15+
"Find DP0 documentation and resources at <a href=\"https://dp0-1.lsst.io\">dp0-1.lsst.io</a>. <br>"
1616
]
1717
},
1818
{
@@ -92,7 +92,7 @@
9292
"### 1. Create an instance of the Butler\n",
9393
"\n",
9494
"To create the Butler, we need to provide it with a path to the data set, which is called a \"data repository\".\n",
95-
"Butler repositories can be remote (i.e., pointing to an S3 bucket) or local (i.e., pointing to a directory on the local file system).\n",
95+
"Butler repositories have both a database component and a file-like storage component; the latter can can be remote (i.e., pointing to an S3 bucket) or local (i.e., pointing to a directory on the local file system), and it contains a configuration file (usually `butler.yaml`) that points to the right database\n",
9696
"\n",
9797
"S3 (Simple Storage Service) buckets are public cloud storage resources that are similar to file folders, store objects, and which consist of data and its descriptive metadata.\n",
9898
"\n",
@@ -122,15 +122,13 @@
122122
"source": [
123123
"#### 2.1 Butler registry and collections\n",
124124
"\n",
125-
"The registry is a database containing information about available data products.\n",
126-
"The registry helps the user to examine what collections of data products exist.\n",
125+
"The database side of a data repository is called a registry.\n",
126+
"The registry contains entries for all data products, and organizes them by _collection_, _dataset type_, and _data ID_.\n",
127127
"Use the registry to investigate a repository by listing all collections.\n",
128128
"\n",
129-
"Find more about the registry schema [here](https://dmtn-073.lsst.io/).\n",
130-
"\n",
131129
"Find more about collections [here](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/organizing.html#collections).\n",
132130
"\n",
133-
"Create a registry for the DP0.1 data set using the Butler."
131+
"A registry client is part of our butler object:"
134132
]
135133
},
136134
{
@@ -173,9 +171,9 @@
173171
"* `calib` - refers to calibration products that are used for instrument signature removal\n",
174172
"* `runs` - refers to processed data products\n",
175173
"* `refcats` - refers to the reference catalogs used for astrometric and photometric calibration\n",
176-
"* `skymaps` - are the geometric representations of the sky coverage\n",
174+
"* `skymaps` - definitions for the _tract_ and _patch_ grids that coadds are built on\n",
177175
"\n",
178-
"Collections are nested, and DP0 delegates can access all the data for DC2 Run 2.2i, which is the DP0.1 data set, by selecting the collection `2.2i/runs/DP0.1`.\n",
176+
"Some collections are nested, and DP0 delegates can access all the data for DC2 Run 2.2i, which is the DP0.1 data set, by selecting the collection `2.2i/runs/DP0.1`.\n",
179177
"\n",
180178
"Expand the pointer recursively to show the full contents of the selected collection."
181179
]
@@ -261,17 +259,21 @@
261259
"cell_type": "markdown",
262260
"metadata": {},
263261
"source": [
264-
"#### 2.3 Butler dataId\n",
262+
"#### 2.3 Butler data IDs\n",
265263
"\n",
266-
"The `dataId` (data identifier) is how specific data within a data set is accessed. Find more about the `dataId` [here](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/dimensions.html#data-ids).\n",
264+
"The data ID is a dictionary-like identifier for a data product.\n",
265+
"Find more about the data IDs [here](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/dimensions.html#data-ids).\n",
267266
"\n",
268-
"Each `DatasetType` uses a different set of keys as the `dataId`.\n",
269-
"For example, in the `DatasetType` list printed to screen (above), next to `calexp` in curly brackets is listed the band, instrument, detector, physical_filter, visit_system, and visit. These are the keys of the `dataId` for a `calexp`.\n",
267+
"Each `DatasetType` uses a different set of keys in its data ID.\n",
268+
"For example, in the `DatasetType` list printed to screen (above), next to `calexp` in curly brackets is listed the band, instrument, detector, physical_filter, visit_system, and visit.\n",
269+
"These are the keys of the data ID for a `calexp`, which are also called \"dimensions\".\n",
270270
"\n",
271-
"In the following cell, the `DatasetRef` is queried for `calexp` data in our collection of interest, and the full `dataId` are printed to screen (for just a few examples).\n",
271+
"In the following cell, the `DatasetRef` is queried for `calexp` data in our collection of interest, and the full data IDs are printed to screen (for just a few examples).\n",
272+
"Data IDs can be represented in code as regular Python `dict` objects, but when returned from the `Butler` the `DataCoordinate` class is used instead.\n",
272273
"\n",
273-
"The `dataId` contains both *implied* and *required* keys. For example, the value of *band* would be *implied* by the *visit*, because a single visit refers to a single exposure at a single pointing in a single band. \n",
274-
"In the following cell, printing the `dataId` without specifying `.full` shows only the required keys.\n",
274+
"The data ID contains both *implied* and *required* keys.\n",
275+
"For example, the value of *band* would be *implied* by the *visit*, because a single visit refers to a single exposure at a single pointing in a single band. \n",
276+
"In the following cell, printing the data ID without specifying `.full` shows only the required keys.\n",
275277
"The value of a single key, in this case *band*, can also be printed by specifying the key name.\n",
276278
"\n",
277279
"The following cell will fail and return an error if the query is requesting a `DatasetRef` for data that does not exist."
@@ -350,10 +352,13 @@
350352
"source": [
351353
"<br>\n",
352354
"\n",
353-
"The `dataId` can be retrieved directly by using `queryDataIds` instead of `queryDatasets`, as in the following two examples.\n",
354-
"Note the flexibility in the use of the query keys and the where statement.\n",
355-
"Also note that both the `calexp` and `src` data sets can be found by the registry, but this will not always necessarily be the case.\n",
356-
"Queries for non-existent data will cause an error to be returned."
355+
"Each data ID key-value pair is associated with a metadata row called a `DimensionRecord`.\n",
356+
"Like dataset types, these exist independent of any collection, but they are also identified by data IDs.\n",
357+
"\n",
358+
"The `queryDimensionsRecords` method provides a way to query for these records.\n",
359+
"Most of the arguments accepted by `queryDatasets` can be used here (including `where`).\n",
360+
"\n",
361+
"An example of this is provided below:"
357362
]
358363
},
359364
{
@@ -362,12 +367,38 @@
362367
"metadata": {},
363368
"outputs": [],
364369
"source": [
365-
"dataIds = registry.queryDataIds([\"visit\", \"detector\", \"band\"], datasets=[\"calexp\"],\n",
366-
" where='visit = 703697', collections=collection)\n",
367-
"for i, dataId in enumerate(dataIds):\n",
368-
" print(dataId.full)\n",
369-
" if i > 2:\n",
370-
" break"
370+
"for dim in ['exposure', 'visit', 'detector']:\n",
371+
" print(list(registry.queryDimensionRecords(dim, where='visit = 971990 and detector=0'))[0])\n",
372+
" print()"
373+
]
374+
},
375+
{
376+
"cell_type": "markdown",
377+
"metadata": {},
378+
"source": [
379+
"Another query method, `queryDataIds`, can be used to query for data IDs independent of any dataset, but it's less useful for general data exploration.\n",
380+
"\n",
381+
"It is also possible to pass `datasets` and `collections` to both `queryDataIds` and `queryDimensionRecords` in order to return records whose data IDs match those of existing datasets.\n",
382+
"But this is quite a bit more subtle than searching directly for a dataset, and rarely wanted when exploring a data repository.\n",
383+
"\n",
384+
"More information on all of the query methods can be found [here](https://pipelines.lsst.io/v/weekly/middleware/faq.html#when-should-i-use-each-of-the-query-methods-commands)."
385+
]
386+
},
387+
{
388+
"cell_type": "markdown",
389+
"metadata": {},
390+
"source": [
391+
"#### 2.5 Temporal and spatial queries\n",
392+
"\n",
393+
"The following examples show how to query for data sets that include a desired coordinate and observation date.\n",
394+
"\n",
395+
"##### Temporal queries\n",
396+
"\n",
397+
"Above, we can see that for visit 971990, the (RA,Dec) are (70.37770,-37.1757) and the observation date is 20251201.\n",
398+
"But these are just human-readable summaries of the more precise spatial and temporal information stored in the registry, which are represented in Python by `Timespan` and `Region` objects, respectively.\n",
399+
"`DimensionRecord` objects that represent spatial or temporal concepts (a `visit` is both) have these objects attached to them.\n",
400+
"\n",
401+
"Retrieve the `DimensionRecord` for a visit and show its timespan and region."
371402
]
372403
},
373404
{
@@ -376,24 +407,55 @@
376407
"metadata": {},
377408
"outputs": [],
378409
"source": [
379-
"dataIds = registry.queryDataIds([\"visit\", \"detector\"], datasets=[\"src\"],\n",
380-
" where=\"band='g' and detector=0 and visit > 700000\",\n",
381-
" collections=collection)\n",
382-
"for i, dataId in enumerate(dataIds):\n",
383-
" print(dataId.full)\n",
384-
" if i > 2:\n",
385-
" break"
410+
"(record,) = registry.queryDimensionRecords('visit', visit=971990)\n",
411+
"\n",
412+
"print(record.timespan)\n",
413+
"print(' ')\n",
414+
"print(record.region)"
386415
]
387416
},
388417
{
389418
"cell_type": "markdown",
390419
"metadata": {},
391420
"source": [
392-
"<br>\n",
421+
"If the timespan or spatial region that are being used as query constraints are already associated with a data ID in the database, the spatial and temporal overlap constraints are automatic.\n",
422+
"For example, if we query for `deepCoadd` datasets with a `visit`+`detector` data ID, we'll get just the ones that overlap that observation and have the same band (because a visit implies a band):"
423+
]
424+
},
425+
{
426+
"cell_type": "code",
427+
"execution_count": null,
428+
"metadata": {},
429+
"outputs": [],
430+
"source": [
431+
"for ref in registry.queryDatasets(\"deepCoadd\", visit=971990, detector=50):\n",
432+
" print(ref)"
433+
]
434+
},
435+
{
436+
"cell_type": "markdown",
437+
"metadata": {},
438+
"source": [
439+
"To query for dimension records or datasets that overlap an arbitrary time range, we can use the `bind` argument to pass times through to `where`.\n",
440+
"Using `bind` to define an alias for a variable saves us from having to string-format the times into the `where` expression.\n",
441+
"Note that a `dafButler.Timespan` will accept a `begin` or `end` value that is equal to `None` if it is unbounded on that side.\n",
393442
"\n",
394-
"The `queryDimensions` method provides a more flexible way to query for multiple datasets (requiring an instance of all datasets to be available for that `dataId`) or to ask for different `dataId` keys than what is used to identify the dataset (which invokes various built-in relationships).\n",
443+
"Use `bind` and `where`, along with [astropy.time](https://docs.astropy.org/en/stable/time/index.html), to look for visits within one minute of this one on either side."
444+
]
445+
},
446+
{
447+
"cell_type": "code",
448+
"execution_count": null,
449+
"metadata": {},
450+
"outputs": [],
451+
"source": [
452+
"# import astropy.time\n",
453+
"# minute = astropy.time.TimeDelta(60, format=\"sec\")\n",
454+
"# timespan = dafButler.Timespan(record.timespan.begin - minute, record.timespan.end + minute)\n",
395455
"\n",
396-
"An example of this is provided below:"
456+
"# for visit in registry.queryDimensionRecords(\"visit\", where=\"visit.timespan OVERLAPS my_timespan\", \n",
457+
"# bind={\"my_timespan\": timespan}):\n",
458+
"# print(visit.id, visit.timespan, visit.physical_filter)"
397459
]
398460
},
399461
{
@@ -402,22 +464,33 @@
402464
"metadata": {},
403465
"outputs": [],
404466
"source": [
405-
"for dim in ['exposure', 'visit', 'detector']:\n",
406-
" print(list(registry.queryDimensionRecords(dim, where='visit = 971990 and detector=0'))[0])\n",
407-
" print()"
467+
"import astropy.time\n",
468+
"minute = astropy.time.TimeDelta(60, format=\"sec\")\n",
469+
"timespan = dafButler.Timespan(record.timespan.begin - minute, record.timespan.end + minute)\n",
470+
"\n",
471+
"datasetRefs = registry.queryDatasets(\"calexp\", where=\"visit.timespan OVERLAPS my_timespan\",\n",
472+
" bind={\"my_timespan\": timespan})\n",
473+
"\n",
474+
"for i, ref in enumerate(datasetRefs):\n",
475+
" print(ref)\n",
476+
" if i > 6:\n",
477+
" break"
408478
]
409479
},
410480
{
411481
"cell_type": "markdown",
412482
"metadata": {},
413483
"source": [
414-
"<br>\n",
484+
"##### Spatial queries\n",
415485
"\n",
416-
"**NEED HELP HERE WITH THIS FINAL BIT!!**\n",
486+
"Arbitrary spatial queries are not supported at this time, such as the \"POINT() IN (REGION)\" example found in this [Butler queries](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/queries.html) documentation.\n",
487+
"In other words, at this time it is only possible to do queries involving regions that are already \"in\" the data repository, either because they are HTM pixel regions or because they are tract/patch/visit/visit+detector regions.\n",
417488
"\n",
418-
"The following examples show how to query for data sets that include a desired coordinate and observation date.\n",
489+
"Thus, for this example we use the set of dimensions that correspond to different levels of the HTM (hierarchical triangular mesh) pixelization of the sky ([HTM primer](http://www.skyserver.org/htm/)).\n",
490+
"The process is to transform a region or point into one or more HTM identifiers (HTM IDs), and then create a query using the HTM ID as the spatial data ID.\n",
491+
"The `lsst.sphgeom` library supports region objects and HTM pixelization in the LSST Science Pipelines.\n",
419492
"\n",
420-
"Above, we can see that for visit 971990, the (RA,Dec) are (70.37770,-37.1757) and the observation date is 20251201. The following example uses the RA,Dec and date to retrieve the visit."
493+
"Import the `lsst.sphgeom` package, initialize a sky pixelization to level 10 (the level at which one sky pixel is about five arcmin across), and find the HTM ID for a desired sky coordinate."
421494
]
422495
},
423496
{
@@ -426,39 +499,76 @@
426499
"metadata": {},
427500
"outputs": [],
428501
"source": [
429-
"ra = 70.37770\n",
430-
"dec = -37.1757\n",
431-
"s1 = \"exposure.day_obs = 20251201\"\n",
432-
"s2 = \"exposure.tracking_ra > \"+str(ra-1.0)\n",
433-
"s3 = \"exposure.tracking_ra < \"+str(ra+1.0)\n",
434-
"s4 = \"exposure.tracking_dec > \"+str(dec-1.0)\n",
435-
"s5 = \"exposure.tracking_dec < \"+str(dec+1.0)\n",
502+
"import lsst.sphgeom\n",
436503
"\n",
437-
"results = registry.queryDimensionRecords('visit',\n",
438-
" where=s1+\" AND \"+s2+\" AND \"+s3+\" AND \"+s4+\" AND \"+s5,\n",
439-
" collections=collection)\n",
504+
"pixelization = lsst.sphgeom.HtmPixelization(10)"
505+
]
506+
},
507+
{
508+
"cell_type": "code",
509+
"execution_count": null,
510+
"metadata": {},
511+
"outputs": [],
512+
"source": [
513+
"htm_id = pixelization.index(\n",
514+
" lsst.sphgeom.UnitVector3d(\n",
515+
" lsst.sphgeom.LonLat.fromDegrees(70.376995, -37.175736)\n",
516+
" )\n",
517+
")\n",
440518
"\n",
441-
"# Use expandDataId to fill in the implicit dataId keys with values\n",
442-
"for i, ref in enumerate(results):\n",
443-
" tempId = butler.registry.expandDataId(ref.dataId)\n",
444-
" print(tempId.full)\n",
445-
" if i > 10:\n",
519+
"# Obtain and print the scale to provide a sense of the size of the sky pixelization being used\n",
520+
"scale = pixelization.triangle(htm_id).getBoundingCircle().getOpeningAngle().asDegrees()*3600\n",
521+
"print(f'HTM ID={htm_id} at level={pixelization.getLevel()} is a ~{scale:0.2}\" triangle.')"
522+
]
523+
},
524+
{
525+
"cell_type": "code",
526+
"execution_count": null,
527+
"metadata": {},
528+
"outputs": [],
529+
"source": [
530+
"datasetRefs = registry.queryDatasets(\"calexp\", htm20=htm_id,\n",
531+
" where=\"visit.timespan OVERLAPS my_timespan\",\n",
532+
" bind={\"my_timespan\": timespan})\n",
533+
"\n",
534+
"for i, ref in enumerate(datasetRefs):\n",
535+
" print(ref)\n",
536+
" if i > 6:\n",
446537
" break"
447538
]
448539
},
449540
{
450541
"cell_type": "markdown",
451542
"metadata": {},
452543
"source": [
453-
"Above, our query terms were not sufficiently unqiue to return only visit 971990, because there were other images of that sky location obtained on that date. **IS IT WEIRD THEY ARE ALL Z BAND?**\n",
544+
"Thus, with the above query, we have uniquely identified the visit and detector for our desired temporal and spatial constraints.\n",
454545
"\n",
455-
"<br>\n",
546+
"Note that if a smaller HTM level is used (like 7), which is a larger sky pixel (~2200 arcseconds), the above query will return many more visits and detectors which overlap with that larger region. Try it and see!\n",
456547
"\n",
457-
"**TO BE ADDED:**\n",
548+
"Note that queries using the HTM ID can also be used to, e.g., find the set of all i-band `src` catalog data products that overlap this point."
549+
]
550+
},
551+
{
552+
"cell_type": "code",
553+
"execution_count": null,
554+
"metadata": {},
555+
"outputs": [],
556+
"source": [
557+
"for i, src_ref in enumerate(registry.queryDatasets(\"src\", htm20=htm_id, band=\"i\")):\n",
558+
" print(src_ref)\n",
559+
" if i > 2:\n",
560+
" break"
561+
]
562+
},
563+
{
564+
"cell_type": "markdown",
565+
"metadata": {},
566+
"source": [
567+
"Why is does that search take tens of seconds?\n",
568+
"The butler's spatial reasoning is designed to work well for regions the size of full data products, like detector- or patch-level images and catalogs, and it's a poor choice for object-scale searches.\n",
569+
"The above search is slow in part because `queryDatasets` searches for all `src` datasets that overlap a larger region and then filters the results down to the specified HTM ID pixel.\n",
458570
"\n",
459-
"* use of regions instead of the above kludge with RA and Dec within a degree\n",
460-
"* how to figure out which detector the coordinates are in, instead of matching to exposure center\n",
461-
"* how to use timespan, to be more specific about time instead of just date"
571+
"Options for exploring and retrieving catalog data with the Butler is covered in more depth in Section 5."
462572
]
463573
},
464574
{
@@ -588,7 +698,7 @@
588698
" Parameters\n",
589699
" ----------\n",
590700
" butler: lsst.daf.persistence.Butler\n",
591-
" Servant providing access to a data repository\n",
701+
" Client providing access to a data repository\n",
592702
" ra: float\n",
593703
" Right ascension of the center of the cutout, degrees\n",
594704
" dec: float\n",

0 commit comments

Comments
 (0)