nf_MIDAS.py User Manual

Version: 11.0 Contact: hsharma@anl.gov

Note

For multi-resolution NF-HEDM (iterative coarse-to-fine reconstruction), see the companion manual NF_MultiResolution_Analysis.md.

1. Introduction

nf_MIDAS.py is the primary driver for Near-Field High-Energy Diffraction Microscopy (NF-HEDM) analysis using the MIDAS software suite. It orchestrates a pipeline of high-performance C binaries to reconstruct a 3D microstructure voxel map from multi-distance detector images.

The script supports two modes:

Microstructure Reconstruction (-refineParameters 0) — the default. Solves for the crystal orientation at every point in a hexagonal grid.
Parameter Refinement (-refineParameters 1) — an advanced mode for refining experimental geometry (detector distance, tilts, beam center) using known grain locations.

Key Capabilities

End-to-end workflow: preprocessing → image processing → fitting → mic output
Parallel execution via Parsl (cluster) or multiprocessing (local)
Memory-mapped binary I/O (mmap) for high-throughput fitting directly from DataDirectory
Automatic retry with exponential backoff for Parsl tasks
Far-field seeding from Grains.csv
Auto-extracted seed orientations from master lookup tables when no explicit seeds are provided
Interactive or file-based parameter refinement
Consolidated HDF5 output via nf_consolidate.py
Robust cleanup of Parsl resources on exit

2. Prerequisites

Requirement	Details
MIDAS Installation	Script auto-detects install dir from its own location
Python Packages	`parsl`, `numpy`, `tqdm`
Input Data	Raw TIFF diffraction images in `DataDirectory`
Parameter File	Text file defining the experiment (see Section 4)
Seed Orientations	A `SeedOrientations` file listing candidate crystal orientations, or `-ffSeedOrientations 1` with a `GrainsFile`, or omit both to auto-extract seeds from master lookup tables based on the space group number
Far-Field Results (optional)	`Grains.csv` from FF-HEDM if using `-ffSeedOrientations 1`

3. Workflow Architecture

Standard Reconstruction (`-refineParameters 0`)

graph LR
    A[Preprocessing] --> B["Image Processing<br/>(skippable)"]
    B --> C[Memory Mapping]
    C --> D[FitOrientationOMP]
    D --> E[ParseMic]
    E --> F["nf_consolidate.py"]
    F --> G["MicFileText + HDF5 output"]

Preprocessing:

GetHKLListNF — compute theoretical Bragg reflections → hkls.csv
GenSeedOrientationsFF2NFHEDM — (if -ffSeedOrientations 1) convert FF orientations to NF seeds
Auto-extract seed orientations from master lookup tables — (if no SeedOrientations file is specified and -ffSeedOrientations is not set) generates seeds based on the space group number
Auto-update NrOrientations in parameter file from seed file line count
MakeHexGrid — create hexagonal reconstruction grid → grid.txt
Grid filtering — (optional) apply TomoImage or GridMask to reduce computation
MakeDiffrSpots — simulate diffraction spots for all seed orientations → DiffractionSpots.bin, Key.bin, OrientMat.bin

Image Processing (skipped if -doImageProcessing 0):

ProcessImagesCombined — single-pass median background computation and peak extraction per detector layer → SpotsInfo.bin (written directly via mmap)

Note

ProcessImagesCombined is the primary image processing path. It computes the median and processes images in one step, writing SpotsInfo.bin directly via mmap/SetBit. The legacy two-step approach (MedianImageLibTiff + ImageProcessingLibTiffOMP) is still available but deprecated.

Fitting & Postprocessing:

MMapImageInfo — prepare memory-mapped binary data (auto-skipped when direct binary files from ProcessImagesCombined and MakeDiffrSpots already exist)
FitOrientationOMP — parallel orientation fitting across all grid points (with progress bar for single-node runs)
ParseMic — consolidate results → {MicFileText}
nf_consolidate.py — generate consolidated HDF5 output containing voxel positions, Euler angles, confidences, and metadata

Parameter Refinement (`-refineParameters 1`)

graph LR
    A[Preprocessing] --> B["Image Processing<br/>(skippable)"]
    B --> C[Memory Mapping]
    C --> D{multiGridPoints?}
    D -->|0| E["Prompt for (x,y) →<br/>FitOrientationParameters"]
    D -->|1| F["FitOrientationParametersMultiPoint"]
    E --> G["Refined parameters<br/>printed to console"]
    F --> G

Single-Point Refinement (-multiGridPoints 0):

Script prompts interactively for (x,y) coordinates
Finds the closest grid point
Runs FitOrientationParameters on that single point
Displays refined geometry to console

Multi-Point Refinement (-multiGridPoints 1):

Uses grid points defined via GridPoints in the parameter file
Runs FitOrientationParametersMultiPoint using -nCPUs threads
Displays refined geometry to console

4. Parameter File Reference

The parameter file is a whitespace-delimited text file. Lines starting with # are comments. Each line has the format Key Value [Value ...].

Core Parameters (Required)

Key	Values	Description
`DataDirectory`	path	Base directory containing input data (raw TIFFs, `ReducedFileName` files). Also used as the output directory unless `OutputDirectory` is specified
`ReducedFileName`	string	Base name pattern for input TIFF files
`StartNr`	int	First frame number
`EndNr`	int	Last frame number
`nDistances`	int	Number of detector distances (layers)
`Lsd`	float	Sample-to-detector distance (μm). Repeated `nDistances` times on separate lines
`BC`	float float	Beam center (y, z) in pixels. Repeated `nDistances` times
`px`	float	Pixel size (μm)
`NrPixelsY`	int	Horizontal detector dimension in pixels (default: `2048`). Overrides `NrPixels`
`NrPixelsZ`	int	Vertical detector dimension in pixels (default: `2048`). Overrides `NrPixels`
`NrPixels`	int	(deprecated) Square detector shorthand — sets both `NrPixelsY` and `NrPixelsZ` to the same value. Ignored if `NrPixelsY`/`NrPixelsZ` are specified
`Wavelength`	float	X-ray wavelength (Å)
`SpaceGroup`	int	Space group number (e.g., 225 for FCC)
`LatticeParameter`	6 floats	a b c α β γ (Å and degrees)
`OmegaStart`	float	Starting rotation angle (degrees)
`OmegaStep`	float	Rotation step size (degrees)
`OmegaRange`	float float	Omega range [start, end]. Can appear multiple times
`BoxSize`	4 floats	Bounding box per omega range. Must match number of `OmegaRange` lines
`ExcludePoleAngle`	float	Exclude reflections within this angle of the pole (degrees)
`GridSize`	float	Reconstruction voxel size (μm)
`MaxRingRad`	float	Maximum ring radius on detector (pixels)
`OrientTol`	float	Orientation tolerance for fitting (degrees)
`MinFracAccept`	float	Minimum fractional overlap to accept a solution
`MicFileBinary`	string	Binary mic output filename
`MicFileText`	string	Text mic output basename
`SeedOrientations`	path	File of candidate crystal orientations
`tx`, `ty`, `tz`	float	Detector tilt angles (degrees)

Optional Parameters

Key	Values	Description
`OutputDirectory`	path	If specified, all generated output files (mic, grid, logs, intermediate binaries) are written here instead of `DataDirectory`. This avoids modifying the raw data directory and prevents data duplication when working with remote or read-only datasets. Falls back to `DataDirectory` if not set
`RingsToUse`	int	Restrict to specific ring number. Can appear multiple times. Useful when structure factors vary significantly between rings — include only rings with strong, reliable diffraction signal
`SaveNSolutions`	int	Number of top solutions to save per grid point (default: `1`)
`Wedge`	float	Wedge angle (degrees, default: `0`)
`Ice9Input`	(no value)	Flag to enable Ice9 mode
`NearestMisorientation`	int	Enable nearest-neighbor misorientation filtering
`TomoImage`	path	Tomography image for grid masking
`TomoPixelSize`	float	Pixel size of the tomography image
`GridMask`	4 floats	`Xmin Xmax Ymin Ymax` — rectangular mask applied to the hex grid
`GrainsFile`	path	Path to `Grains.csv` for FF seeding
`GridFileName`	string	Custom grid filename (default: `grid.txt`)
`GridPoints`	12 floats	Custom grid point specification (for multi-point refinement)
`BCTol`	2 floats	Beam center tolerance for refinement
`PrecomputedSpotsInfo`	int	`1` = read existing `SpotsInfo.bin` instead of generating from images (default: `0`)
`WriteLegacyBin`	int	`1` = re-enable legacy per-frame `.bin` and `.txtOld` output in `ProcessImagesCombined` (default: `0`)

Tip

Separating raw data from results: Set DataDirectory to the location of your raw images and OutputDirectory to a local working directory. The workflow reads input data from DataDirectory and writes all generated files (mic, grid, logs, binaries) to OutputDirectory. This is especially useful when raw data lives on a remote filesystem or read-only mount — you avoid copying the data locally.

5. Command-Line Arguments

python nf_MIDAS.py \
    -paramFN <param_file> \
    [-nCPUs <int>] [-machineName <str>] [-nNodes <int>] \
    [-ffSeedOrientations <0|1>] [-doImageProcessing <0|1>] \
    [-refineParameters <0|1>] [-multiGridPoints <0|1>] \
    [-gpuFit <0|1>]

Argument	Type	Default	Description
`-paramFN`	string	(required)	Path to the parameter file
`-nCPUs`	int	`10`	CPU cores per node for OpenMP tasks
`-machineName`	string	`local`	Execution target: `local`, `orthrosnew`, `orthrosall`, `umich`, `marquette`, `purdue`
`-nNodes`	int	`1`	Number of compute nodes (Parsl workers)
`-ffSeedOrientations`	int	`0`	`1` = generate seeds from FF `Grains.csv`; `0` = use existing `SeedOrientations` file (or auto-extract from master lookup tables if no seed file is specified)
`-doImageProcessing`	int	`1`	`1` = run median and image processing; `0` = skip (reuse existing processed images)
`-refineParameters`	int	`0`	`1` = run parameter refinement mode; `0` = run reconstruction mode
`-multiGridPoints`	int	`0`	(only if `-refineParameters 1`) `1` = use multiple grid points from param file; `0` = prompt for single (x,y)
`-gpuFit`	int	`0`	`1` = use GPU-accelerated screening and fitting via `FitOrientationGPU`; `0` = use CPU-only `FitOrientationOMP`
`-resume`	string	`''`	Path to a pipeline H5 to resume from. Auto-detects the last completed stage.
`-restartFrom`	string	`''`	Explicit stage to restart from. Valid stages: `preprocessing`, `image_processing`, `mmap`, `fitting`, `parse_mic`, `consolidation`.

6. Execution Examples

Example 1: Standard Reconstruction on a Cluster

python nf_MIDAS.py \
    -paramFN /path/to/nf_params.txt \
    -machineName purdue \
    -nNodes 4 \
    -nCPUs 128 \
    -ffSeedOrientations 1

Example 2: Local Reconstruction, Skip Image Processing

python nf_MIDAS.py \
    -paramFN nf_params.txt \
    -machineName local \
    -nCPUs 8 \
    -doImageProcessing 0

Example 3: Interactive Single-Point Parameter Refinement

python nf_MIDAS.py \
    -paramFN nf_params.txt \
    -machineName local \
    -nCPUs 8 \
    -refineParameters 1 \
    -multiGridPoints 0

# Script will prompt:
# "Enter the x,y coordinates to optimize (e.g., 1.2,3.4): "
# Type coordinates and press Enter

Example 4: Multi-Point Parameter Refinement

python nf_MIDAS.py \
    -paramFN nf_params.txt \
    -machineName local \
    -nCPUs 8 \
    -refineParameters 1 \
    -multiGridPoints 1

Example 5: Resume from a Pipeline Checkpoint

# Auto-detect last completed stage and resume:
python nf_MIDAS.py -paramFN nf_params.txt -nCPUs 8 \
    -resume /path/to/pipeline.h5

# Restart from fitting (re-runs fitting → parse_mic → consolidation):
python nf_MIDAS.py -paramFN nf_params.txt -nCPUs 8 \
    -restartFrom fitting

Example 6: GPU-Accelerated Reconstruction

python nf_MIDAS.py \
    -paramFN nf_params.txt \
    -nCPUs 8 \
    -gpuFit 1

7. Iterative Calibration and Optimization Workflow

A high-confidence NF-HEDM reconstruction requires iterating between parameter refinement and full reconstruction. This section describes the complete end-to-end procedure.

Important

Start with the calibration procedure in NF_Calibration.md to determine initial values for Lsd, BC, and detector tilts. Those values become the initial guess in your parameter file.

graph TD
    A["Initial Calibration<br>(NF_Calibration.md)"] --> B["Update Parameter File<br>with Lsd, BC values"]
    B --> C["Single-Point Optimization<br>(2-3 iterations)"]
    C --> D["Full Reconstruction"]
    D --> E["Inspect .mic in GUI<br>(Load Mic → .map file)"]
    E --> F["Select 5-10 GridPoints<br>from high-confidence grains"]
    F --> G["Multi-Point Optimization<br>(2-3 iterations)"]
    G --> H["Full Reconstruction"]
    H --> I{"Confidence<br>acceptable?"}
    I -->|No| G
    I -->|Yes| J["Done — Final .mic"]

7.1 Single-Point Parameter Refinement

After entering the initial Lsd and BC values from calibration, run single-point refinement a few times:

python nf_MIDAS.py -paramFN nf_params.txt -nCPUs 8 -refineParameters 1 -multiGridPoints 0

The script prompts for (x, y) coordinates — choose a point near the center of your sample. After each run, the refined Lsd, BC, and tilt values are printed to the console. Update them in the parameter file and re-run. Repeat 2–3 times until values stabilize.

7.2 Initial Full Reconstruction

python nf_MIDAS.py -paramFN nf_params.txt -nCPUs 8

This produces a .mic text file and a .map binary file in DataDirectory.

7.3 Grid Point Selection for Multi-Point Optimization

Open the reconstruction in the NF GUI (see NF_GUI.md):
```
cd <DataDirectory>
python ~/opt/MIDAS/gui/nf.py &
```
Click Load Mic and select the .map file (preferred over .mic — imshow rendering is significantly faster).
Set visualization to Confidence using the radio buttons.
Identify 5–10 high-quality grid points with the following criteria:
- Confidence less than 1 (values of exactly 1 are suspicious)
- Not on grain boundaries — at this early stage, grain boundary voxels do not have correctly determined orientations. Multi-point optimization cannot search over all possible orientations; it assumes the orientation assigned to the voxel is approximately correct. If a grain boundary voxel is chosen, its orientation guess may be wrong, leading the optimizer to refine geometry toward incorrect parameters.
- Close to grain boundaries — provides geometric diversity
- Distributed across all four quadrants of the sample (roughly balanced)

Open the .mic text file in a text editor. Each line is a grid point:

OrientRowNr  ID  Time  X  Y  Size  UD  Euler1  Euler2  Euler3  Confidence

Copy the corresponding lines to your parameter file, prefixing each with GridPoints:

GridPoints  0  0  0  -123.4  456.7  5.0  1  0.123  0.456  0.789  0.85
GridPoints  0  0  0   234.5 -345.6  5.0  1  1.234  0.567  0.890  0.82
...

7.4 Multi-Point Optimization Loop

Run multi-point refinement:

python nf_MIDAS.py -paramFN nf_params.txt -nCPUs 8 -refineParameters 1 -multiGridPoints 1

Update the parameter file with the refined values printed to the console.
Repeat multi-point refinement 2–3 times until values stabilize.

Run a full reconstruction again:

python nf_MIDAS.py -paramFN nf_params.txt -nCPUs 8

Inspect the new reconstruction in the GUI. If confidence is not yet satisfactory, re-run multi-point optimization (the GridPoints lines in the parameter file can remain from the previous iteration).
Iterate until you achieve a high-confidence map you are satisfied with.

8. Output Directory Structure

All output is generated within OutputDirectory if specified, otherwise within DataDirectory.

<OutputDirectory>/    (or <DataDirectory>/ if OutputDirectory is not set)
├── midas_log/                  # Logs for every binary
│   ├── midas_nf_workflow.log   # Master workflow log
│   ├── hkls_out/err.csv        # HKL generation
│   ├── seed_out/err.csv        # Seed orientation generation
│   ├── hex_out/err.csv         # Hex grid creation
│   ├── spots_out/err.csv       # Spot simulation
│   ├── tomo_out/err.csv        # Tomo grid filtering
│   ├── median{N}_out/err.csv   # Median image for distance N
│   ├── image{N}_out/err.csv    # Image processing for block N
│   ├── map_out/err.csv         # Memory mapping
│   ├── fit{N}_out/err.csv      # Orientation fitting for block N
│   ├── parse_out/err.csv       # Mic file parsing
│   ├── fit_singlepoint_out.csv # Single-point refinement output
│   └── fit_multipoint_out.csv  # Multi-point refinement output
├── {MicFileText}               # FINAL TEXT MIC OUTPUT (reconstruction mode)
├── {MicFileBinary}             # Binary mic output
├── nf_consolidated.h5          # Consolidated HDF5 output (auto-generated after ParseMic)
├── grid.txt                    # Reconstruction grid (filtered if applicable)
├── grid_unfilt.txt             # Pre-tomo-filter backup of grid
├── grid_old.txt                # Pre-mask-filter backup of grid
├── hkls.csv                    # Theoretical HKL reflections
├── DiffractionSpots.bin        # Simulated spots (binary)
├── SpotsInfo.bin               # Processed experimental spots (binary)
├── Key.bin / Key.txt           # Orientation-to-spot index mapping
└── OrientMat.bin / OrientMat.txt  # Orientation matrices

Mic File Format

The final text mic file has one line per reconstructed grid point. Lines starting with % are header/comments.

Column	Index	Description
OrientationRowNr	0	Row number from the orientation matrix file
ID	1	Orientation ID
Time	2	(unused, typically 0)
X	3	X position (μm)
Y	4	Y position (μm)
Size	5	Grid cell size (μm)
UD	6	Up/Down triangle indicator (+1 or -1)
Euler1	7	Euler angle φ₁ (radians)
Euler2	8	Euler angle Φ (radians)
Euler3	9	Euler angle φ₂ (radians)
Confidence	10	Fractional overlap (0–1), higher = better fit

Consolidated HDF5 Output

After ParseMic completes, the workflow automatically runs nf_consolidate.py to produce a consolidated HDF5 file (nf_consolidated.h5). This file packages the voxel positions, Euler angles, confidences, and experiment metadata into a single portable container, suitable for downstream analysis and archival.

9. Binary Executables Reference

Binary	Purpose	Key Arguments
`GetHKLListNF`	Generate HKL list	`<paramFN>`
`GenSeedOrientationsFF2NFHEDM`	Convert FF grains to NF seeds	`<GrainsFile> <SeedOrientations>`
`MakeHexGrid`	Create hexagonal reconstruction grid	`<paramFN>`
`filterGridfromTomo`	Filter grid using tomography image	`<TomoImage> <TomoPixelSize>`
`MakeDiffrSpots`	Simulate diffraction spots for all seeds	`<paramFN>`
`ProcessImagesCombined`	Single-pass median + image processing (primary path)	`<paramFN> <distanceNr> <nCPUs>`
`MedianImageLibTiff`	Compute median background image (legacy)	`<paramFN> <distanceNr>`
`ImageProcessingLibTiffOMP`	Process raw images — background subtraction (legacy)	`<paramFN> <nodeNr> <nNodes> <nCPUs>`
`MMapImageInfo`	Prepare memory-mapped binary data (auto-skipped when direct binaries exist)	`<paramFN>`
`FitOrientationOMP`	Fit crystal orientations at each grid point	`<paramFN> <blockNr> <nBlocks> <nCPUs>`
`FitOrientationGPU`	GPU-accelerated orientation fitting (selected via `-gpuFit 1`)	`<paramFN> <blockNr> <nBlocks>`
`ParseMic`	Consolidate fitting results into mic file	`<paramFN>`
`FitOrientationParameters`	Single-point geometry refinement	`<paramFN> <gridPointNr>`
`FitOrientationParametersMultiPoint`	Multi-point geometry refinement	`<paramFN> <nCPUs>`

10. Technical Implementation Details

10.1. High-Performance Data Structures (`MMapImageInfo`)

Bitmasking: To enable ultra-fast collision detection during fitting, the experimental diffraction images are pre-processed into a monolithic bitmask (ObsSpotsInfo.bin).
Index Calculation: A 4D coordinate (Layer, Rotation, Y, Z) is mapped to a linear index in the bitmask using NrPixelsY and NrPixelsZ. If a pixel contains a diffraction peak, the corresponding bit is set to 1.
Memory Mapping: The binary files are memory-mapped (mmap) into the process address space directly from DataDirectory. This allows the OS to efficiently manage memory paging and enables multiple processes (or threads) to access the massive dataset (often 10s of GBs) with near-RAM speeds via the OS page cache.

10.2. Orientation Fitting Algorithm (`FitOrientationOMP`)

Grid Parallelization: The reconstruction volume is discretized into a hexagonal grid. The fitting process for each grid point is independent, allowing trivial parallelization using OpenMP. Each CPU thread processes a subset of grid points.
Two-Stage Optimization:
1. Discrete Search: The algorithm first tests a pre-computed list of 'seed' orientations (from SeedOrientations or converted FF results). It calculates the fractional overlap for each seed and keeps the best candidates.
2. Continuous Refinement: The best candidate is refined using the Nelder-Mead Simplex algorithm (via nlopt). The objective function is maximizing the Fractional Overlap between simulated and experimental spots.
Overlap Calculation:
- Projection: Theoretical spots are projected onto the detector plane, accounting for sample position (X, Y), detector tilt, and wedge angle.
- Rasterization: The projected spot shape is rasterized into a set of pixels.
- Collision Check: Each rasterized pixel is checked against the ObsSpotsInfo bitmask. This is an $O(1)$ operation, making the loop extremely fast.

10.3. Parallel I/O

Writer Locks: While the computation is parallel, writing the results to the output file (MicFileBinary) uses pwrite (parallel write) with thread-safe file offsets to avoid race conditions and ensure data integrity without serialization bottlenecks.

11. Troubleshooting

Fitting Issues

Symptom	Likely Cause	Fix
All confidences near 0	Wrong geometry parameters	Check `Lsd`, `BC`, `tx/ty/tz`, `px`
Fitting hangs	Missing nlopt stopping criteria	Ensure your `FitOrientationOMP` binary has `nlopt_set_maxeval` and `nlopt_set_ftol_rel`
Very few good points	Seeds too sparse or wrong	Check `SeedOrientations` file contents; try FF seeding

General Debugging

Check midas_log/midas_nf_workflow.log for the master log
Check individual *_err.csv files for binary-level error output
For fitting issues specifically, examine fit{N}_out.csv — each line corresponds to a completed grid point
Verify grid.txt has the expected number of points before fitting

Common Errors

Error	Fix
`FileNotFoundError` on param file	Check `-paramFN` path
`DataDirectory not found`	Ensure `DataDirectory` is set in parameter file
Parsl configuration errors	Check machine config files and partition names
Automatic retries exhausting	Likely a systemic issue — check `*_err.csv` logs

12. GPU Acceleration

NF-HEDM supports GPU-accelerated orientation fitting via the -gpuFit 1 flag:

python nf_MIDAS.py -paramFN params.txt -nCPUs 8 -gpuFit 1

This accelerates both screening (Phase 1: discrete orientation search) and fitting (Phase 2: Nelder-Mead continuous refinement) using FitOrientationGPU.

For double-precision GPU computation (exact CPU/GPU parity):

export MIDAS_GPU_DOUBLE=1
python nf_MIDAS.py -paramFN params.txt -gpuFit 1

The -gpuFit flag also works with multi-resolution workflows:

python nf_MIDAS_Multiple_Resolutions.py -paramFN params.txt -gpuFit 1

See GPU_Acceleration.md for build instructions and full GPU documentation.

13. New Features Since v10

Streaming Histogram-Based Median

ProcessImagesCombined.c replaces the legacy all-in-memory median computation with a streaming histogram approach:

Memory usage reduced from ~11.5 GB to ~500 MB
Streams TIFFs one at a time, builds per-pixel uint16 histogram (65536 bins)
Writes SpotsInfo.bin directly via mmap/SetBit
MedianImageLibTiff + ImageProcessingLibTiffOMP are deprecated in favor of this single-pass approach

New Parameters

Parameter	Description
`LocalMaximaOnly`	New peak search mode for powdery/textured samples
`NrPixelsY` / `NrPixelsZ`	Dynamic detector size (replaces hardcoded 2048)
`PrecomputedSpotsInfo`	Use existing `SpotsInfo.bin` instead of regenerating
`WriteLegacyBin`	Re-enable legacy per-frame `.bin` output

I/O Improvements

FitOrientation executables now use SpotsInfo.bin mmap directly from DataDirectory instead of per-frame .bin files
Misorientation uniqueness filter for nSaves — filters redundant orientations
Binary mmap-based I/O replaces text-based I/O for FitOrientationParameters

Consolidated HDF5 Output

After reconstruction, nf_consolidate.py automatically generates a consolidated HDF5 file containing all voxel data and metadata
Each resolution level in multi-resolution workflows is saved to the HDF5 via add_resolution_to_h5()

Auto-Extracted Seed Orientations

When no SeedOrientations file is specified and -ffSeedOrientations is not set, the workflow automatically extracts seed orientations from master lookup tables based on the space group number
This eliminates the need to manually prepare seed files for common crystal structures

14. See Also

NF_MultiResolution_Analysis.md — Multi-resolution iterative NF-HEDM reconstruction
NF_Calibration.md — NF detector geometry calibration
NF_GUI.md — Interactive NF-HEDM analysis GUI
Forward_Simulation.md — Forward simulation (simulateNF)
README.md — High-level MIDAS overview and manual index

If you encounter any issues or have questions, please open an issue on this repository.

FilesExpand file tree

NF_Analysis.md

Latest commit

History

NF_Analysis.md

File metadata and controls

nf_MIDAS.py User Manual

1. Introduction

Key Capabilities

2. Prerequisites

3. Workflow Architecture

Standard Reconstruction (-refineParameters 0)

Parameter Refinement (-refineParameters 1)

4. Parameter File Reference

Core Parameters (Required)

Optional Parameters

5. Command-Line Arguments

6. Execution Examples

Example 1: Standard Reconstruction on a Cluster

Example 2: Local Reconstruction, Skip Image Processing

Example 3: Interactive Single-Point Parameter Refinement

Example 4: Multi-Point Parameter Refinement

Example 5: Resume from a Pipeline Checkpoint

Example 6: GPU-Accelerated Reconstruction

7. Iterative Calibration and Optimization Workflow

7.1 Single-Point Parameter Refinement

7.2 Initial Full Reconstruction

7.3 Grid Point Selection for Multi-Point Optimization

7.4 Multi-Point Optimization Loop

8. Output Directory Structure

Mic File Format

Consolidated HDF5 Output

9. Binary Executables Reference

10. Technical Implementation Details

10.1. High-Performance Data Structures (MMapImageInfo)

10.2. Orientation Fitting Algorithm (FitOrientationOMP)

10.3. Parallel I/O

11. Troubleshooting

Fitting Issues

General Debugging

Common Errors

12. GPU Acceleration

13. New Features Since v10

Streaming Histogram-Based Median

New Parameters

I/O Improvements

Consolidated HDF5 Output

Auto-Extracted Seed Orientations

14. See Also

Standard Reconstruction (`-refineParameters 0`)

Parameter Refinement (`-refineParameters 1`)

10.1. High-Performance Data Structures (`MMapImageInfo`)

10.2. Orientation Fitting Algorithm (`FitOrientationOMP`)