|
| 1 | +""" |
| 2 | + examples/README.md |
| 3 | + |
| 4 | +Worked examples and tutorials for StructureFunctions.jl |
| 5 | +""" |
| 6 | + |
| 7 | +# Examples: Getting Started with StructureFunctions.jl |
| 8 | + |
| 9 | +This directory contains runnable examples demonstrating different aspects of structure function computation in StructureFunctions.jl. |
| 10 | + |
| 11 | +## Quick Start |
| 12 | + |
| 13 | +### 1. Simple 2D Calculation (`simple_2d.jl`) |
| 14 | + |
| 15 | +**Best for**: First-time users, understanding basics |
| 16 | + |
| 17 | +```bash |
| 18 | +julia examples/simple_2d.jl |
| 19 | +``` |
| 20 | + |
| 21 | +**What it does**: |
| 22 | +- Generates synthetic 2D turbulent velocity data (256×256 grid) |
| 23 | +- Computes 2nd-order structure functions |
| 24 | +- Validates K41 scaling ($S_2(r) \sim r^{2/3}$) |
| 25 | +- Produces visualization (plot saved) |
| 26 | + |
| 27 | +**Key concepts**: |
| 28 | +- Regular grids and structure functions |
| 29 | +- Spectral scaling laws |
| 30 | +- Visualization of results |
| 31 | + |
| 32 | +**Learn next**: `docs/theory.md` for physics background |
| 33 | + |
| 34 | +--- |
| 35 | + |
| 36 | +### 2. Threaded Parallelization (`threaded_calculation.jl`) |
| 37 | + |
| 38 | +**Best for**: Medium datasets (10M–500M points), multi-core machines |
| 39 | + |
| 40 | +```bash |
| 41 | +# Set thread count |
| 42 | +JULIA_NUM_THREADS=4 julia examples/threaded_calculation.jl |
| 43 | + |
| 44 | +# Or let Julia auto-detect |
| 45 | +JULIA_NUM_THREADS=auto julia examples/threaded_calculation.jl |
| 46 | +``` |
| 47 | + |
| 48 | +**What it does**: |
| 49 | +- Generates 50M points (scalable based on CPU cores) |
| 50 | +- Compares serial vs threaded execution |
| 51 | +- Measures speedup and parallel efficiency |
| 52 | +- Demonstrates best practices |
| 53 | + |
| 54 | +**Key metrics**: |
| 55 | +- Speedup vs serial: typically 2–8x on 4–32 cores |
| 56 | +- Efficiency: depends on memory bandwidth |
| 57 | +- Optimal for cache-resident data |
| 58 | + |
| 59 | +**Advanced**: Try different dataset sizes; plot speedup curve vs thread count |
| 60 | + |
| 61 | +**Learn next**: `docs/backends.md` for backend comparison |
| 62 | + |
| 63 | +--- |
| 64 | + |
| 65 | +### 3. GPU Acceleration (`gpu_acceleration.jl`) |
| 66 | + |
| 67 | +**Best for**: Large datasets (>500M points), with GPU hardware |
| 68 | + |
| 69 | +```bash |
| 70 | +# Requires CUDA.jl + KernelAbstractions.jl |
| 71 | +julia --project -e 'using Pkg; Pkg.add(["CUDA", "KernelAbstractions"])' |
| 72 | + |
| 73 | +# Run example |
| 74 | +julia examples/gpu_acceleration.jl |
| 75 | +``` |
| 76 | + |
| 77 | +**What it does**: |
| 78 | +- Generates 1 billion points (or 10M if GPU unavailable) |
| 79 | +- Uses GPUBackend for computation |
| 80 | +- Measures memory efficiency of Float32 vs Float64 |
| 81 | +- Estimates speedup |
| 82 | + |
| 83 | +**Expected results**: |
| 84 | +- NVIDIA A100: 20–50x faster than single CPU |
| 85 | +- RTX 4090: 10–30x faster than CPU |
| 86 | +- AMD MI250X: 15–80x faster than CPU |
| 87 | + |
| 88 | +**Key insights**: |
| 89 | +- Float32 sufficient for turbulence (saves 50% memory) |
| 90 | +- GPU shines for >1B points |
| 91 | +- Kernel compilation amortized over many calls |
| 92 | + |
| 93 | +**Learn next**: `docs/backends.md#gpubackend`, `docs/real_data.md#performance-tips` |
| 94 | + |
| 95 | +--- |
| 96 | + |
| 97 | +### 4. Distributed Computing (`distributed_parallel.jl`) |
| 98 | + |
| 99 | +**Best for**: Massive datasets across clusters (>1B points, multi-node) |
| 100 | + |
| 101 | +```bash |
| 102 | +# Local cluster (spawn 4 processes) |
| 103 | +julia examples/distributed_parallel.jl |
| 104 | + |
| 105 | +# Custom process count |
| 106 | +julia examples/distributed_parallel.jl 8 |
| 107 | + |
| 108 | +# Or via Distributed.jl |
| 109 | +julia -p 8 examples/distributed_parallel.jl |
| 110 | +``` |
| 111 | + |
| 112 | +**What it does**: |
| 113 | +- Starts multiple Julia processes |
| 114 | +- Distributes data locally (no copy) |
| 115 | +- Computes structure functions in parallel |
| 116 | +- Demonstrates SLURM submission script |
| 117 | + |
| 118 | +**HPC Usage**: |
| 119 | +```bash |
| 120 | +# Save the SLURM template: |
| 121 | +# sbatch submit_distributed.sbatch |
| 122 | + |
| 123 | +# Scales to 100s of processes on HPC clusters |
| 124 | +``` |
| 125 | + |
| 126 | +**Scaling characteristics**: |
| 127 | +- Near-linear speedup with N nodes (for large problems) |
| 128 | +- Suitable for >500M points |
| 129 | +- Minimal communication overhead if data is local |
| 130 | + |
| 131 | +**Learn next**: `docs/backends.md#distributedbackend` |
| 132 | + |
| 133 | +--- |
| 134 | + |
| 135 | +### 5. Real Climate Data (`real_data_climate.jl`) |
| 136 | + |
| 137 | +**Best for**: Practical atmospheric/ocean applications |
| 138 | + |
| 139 | +```bash |
| 140 | +# Optional: install NetCDF support |
| 141 | +julia -e 'using Pkg; Pkg.add("NetCDF")' |
| 142 | + |
| 143 | +# Run with synthetic data (generates demo) |
| 144 | +julia examples/real_data_climate.jl |
| 145 | +``` |
| 146 | + |
| 147 | +**What it does**: |
| 148 | +- Simulates realistic atmospheric turbulence data |
| 149 | +- Handles missing data (NaNs) and outliers |
| 150 | +- Constructs 3D coordinate systems |
| 151 | +- Performs K41 scaling analysis |
| 152 | +- Multi-time-step statistics |
| 153 | + |
| 154 | +**Key workflows**: |
| 155 | +1. Data loading (NetCDF, HDF5, CSV) |
| 156 | +2. Preprocessing (unit conversion, detrending) |
| 157 | +3. Structure function calculation |
| 158 | +4. K41 validation |
| 159 | +5. Temporal statistics |
| 160 | + |
| 161 | +**Real data sources**: |
| 162 | +- SOCRATES campaign: https://data.eol.ucar.edu/ |
| 163 | +- ERA5 reanalysis: https://cds.climate.copernicus.eu/ |
| 164 | +- MERRA-2: https://gmao.gsfc.nasa.gov/reanalysis/MERRA-2/ |
| 165 | +- WRF simulations: Custom NetCDF output |
| 166 | + |
| 167 | +**Learn next**: `docs/real_data.md` |
| 168 | + |
| 169 | +--- |
| 170 | + |
| 171 | +## Advanced Workflows |
| 172 | + |
| 173 | +### Combining Backends |
| 174 | + |
| 175 | +**Scenario**: Process each file with GPU, aggregate across time |
| 176 | + |
| 177 | +```julia |
| 178 | +using StructureFunctions, DistributedArrays |
| 179 | + |
| 180 | +# Load time series in parallel |
| 181 | +for t in 1:n_time_steps |
| 182 | + # Each worker computes one time step |
| 183 | + result_t = @distributed (+) for i in workers() |
| 184 | + data_i = load_time_slice(t, i) |
| 185 | + x, u = construct_coordinates(data_i) |
| 186 | + calculate_structure_function(x, u, bins; backend=GPUBackend()) |
| 187 | + end |
| 188 | +end |
| 189 | +``` |
| 190 | + |
| 191 | +### Batch Processing Out-of-Core |
| 192 | + |
| 193 | +**Scenario**: Dataset too large for memory (terabytes) |
| 194 | + |
| 195 | +```julia |
| 196 | +# Process in streaming chunks |
| 197 | +all_sums = zeros(n_bins) |
| 198 | +all_counts = zeros(Int, n_bins) |
| 199 | + |
| 200 | +for chunk in read_chunks("huge_data.zarr") |
| 201 | + x, u = extract_data(chunk) |
| 202 | + result = calculate_structure_function(x, u, bins; backend=AutoBackend()) |
| 203 | + all_sums .+= result.sums |
| 204 | + all_counts .+= result.counts |
| 205 | +end |
| 206 | + |
| 207 | +# Normalize at end |
| 208 | +final_sf = all_sums ./ all_counts |
| 209 | +``` |
| 210 | + |
| 211 | +### Multi-Scale Nested Analysis |
| 212 | + |
| 213 | +**Scenario**: Analyze turbulence at multiple length scales |
| 214 | + |
| 215 | +```julia |
| 216 | +# Coarse grid: long-range structure |
| 217 | +bins_coarse = 100:500:100_000 |
| 218 | +result_coarse = calculate_structure_function(x, u, bins_coarse) |
| 219 | + |
| 220 | +# Fine grid: local correlation |
| 221 | +bins_fine = 1:0.5:100 |
| 222 | +result_fine = calculate_structure_function(x, u, bins_fine) |
| 223 | + |
| 224 | +# Compare scaling exponents |
| 225 | +alpha_coarse = estimate_exponent(result_coarse) |
| 226 | +alpha_fine = estimate_exponent(result_fine) |
| 227 | +``` |
| 228 | + |
| 229 | +--- |
| 230 | + |
| 231 | +## Running All Examples |
| 232 | + |
| 233 | +```bash |
| 234 | +# Prerequisites |
| 235 | +julia --project -e 'using Pkg; Pkg.instantiate()' |
| 236 | + |
| 237 | +# Run all in sequence |
| 238 | +for script in simple_2d threaded_calculation gpu_acceleration distributed_parallel real_data_climate; do |
| 239 | + echo "Running example: $script" |
| 240 | + julia examples/${script}.jl |
| 241 | +done |
| 242 | +``` |
| 243 | + |
| 244 | +--- |
| 245 | + |
| 246 | +## Performance Expectations |
| 247 | + |
| 248 | +| Example | Input Size | Time | Backend | |
| 249 | +|---------|-----------|------|---------| |
| 250 | +| simple_2d | 65K pts | ~1 sec | Serial | |
| 251 | +| threaded_calculation | 50M pts | ~1 sec | Threaded (4 cores) | |
| 252 | +| gpu_acceleration | 1B pts | ~20 sec | GPU (A100) | |
| 253 | +| distributed_parallel | 1B pts | ~30 sec | Multi-process | |
| 254 | +| real_data_climate | 50K pts | ~0.5 sec | Serial | |
| 255 | + |
| 256 | +*Actual times vary by hardware. Use `@time` to measure.* |
| 257 | + |
| 258 | +--- |
| 259 | + |
| 260 | +## Choosing an Example to Start With |
| 261 | + |
| 262 | +### I want to... |
| 263 | + |
| 264 | +- **Learn the basics** → `simple_2d.jl` |
| 265 | +- **Speed up my computation** → `threaded_calculation.jl` |
| 266 | +- **Use my GPU** → `gpu_acceleration.jl` |
| 267 | +- **Scale to a supercomputer** → `distributed_parallel.jl` |
| 268 | +- **Analyze real atmospheric data** → `real_data_climate.jl` |
| 269 | +- **Combine multiple approaches** → Mix the examples! |
| 270 | + |
| 271 | +--- |
| 272 | + |
| 273 | +## Common Issues & Solutions |
| 274 | + |
| 275 | +### "ThreadedBackend not found" |
| 276 | +```julia |
| 277 | +julia> ThreadedBackend() |
| 278 | +ERROR: UndefVarError: ThreadedBackend not defined |
| 279 | + |
| 280 | +# Solution: Load OhMyThreads |
| 281 | +using OhMyThreads # Triggers extension |
| 282 | +``` |
| 283 | + |
| 284 | +### "GPU out of memory" |
| 285 | +```julia |
| 286 | +# Solution: Reduce dataset size or use Float32 |
| 287 | +x = Float32.(x) # Half the memory |
| 288 | +u = Float32.(u) |
| 289 | + |
| 290 | +result = calculate_structure_function(x, u, bins; backend=GPUBackend()) |
| 291 | +``` |
| 292 | + |
| 293 | +### "Structure function is NaN" |
| 294 | +```julia |
| 295 | +# Solution: Check for NaN inputs or remove them |
| 296 | +sum(isnan.(u)) # Check NaNs |
| 297 | +u_clean = u[.!isnan.(u), :] # Remove |
| 298 | + |
| 299 | +# Or let calculate_structure_function handle it |
| 300 | +result = calculate_structure_function(x, u, bins) # Skips NaN pairs |
| 301 | +``` |
| 302 | + |
| 303 | +### "Serial computation is too slow" |
| 304 | +```julia |
| 305 | +# Solution: Use ThreadedBackend |
| 306 | +backend = ThreadedBackend() # Requires OhMyThreads |
| 307 | + |
| 308 | +# Or try GPU |
| 309 | +backend = GPUBackend() # Requires KernelAbstractions |
| 310 | +``` |
| 311 | + |
| 312 | +--- |
| 313 | + |
| 314 | +## Further Reading |
| 315 | + |
| 316 | +After the examples, dive into the comprehensive docs: |
| 317 | + |
| 318 | +- **Theory**: `docs/theory.md` — Physics and mathematics |
| 319 | +- **Architecture**: `docs/architecture.md` — Internal design |
| 320 | +- **Backends**: `docs/backends.md` — Detailed backend guide |
| 321 | +- **Extensions**: `docs/extensions.md` — Lazy loading system |
| 322 | +- **Real Data**: `docs/real_data.md` — File I/O and workflows |
| 323 | + |
| 324 | +--- |
| 325 | + |
| 326 | +## Contributing Examples |
| 327 | + |
| 328 | +To add a new example: |
| 329 | + |
| 330 | +1. Create `examples/new_example.jl` with: |
| 331 | + - Clear docstring explaining what it does |
| 332 | + - Section headers for organization |
| 333 | + - Comments explaining key concepts |
| 334 | + - Print statements summarizing results |
| 335 | + - Links to relevant docs |
| 336 | + |
| 337 | +2. Update this README with: |
| 338 | + - Brief description |
| 339 | + - Run instructions |
| 340 | + - Key concepts learned |
| 341 | + - Next steps |
| 342 | + |
| 343 | +3. Test it runs without errors: |
| 344 | + ```bash |
| 345 | + julia examples/new_example.jl |
| 346 | + ``` |
| 347 | + |
| 348 | +--- |
| 349 | + |
| 350 | +## Questions? |
| 351 | + |
| 352 | +- Check docstrings: `?calculate_structure_function` |
| 353 | +- Search examples/ for similar workflows |
| 354 | +- Read the relevant doc file (theory / backends / real_data) |
| 355 | +- Open an issue on GitHub |
| 356 | + |
| 357 | +--- |
| 358 | + |
| 359 | +**Happy computing! 🚀** |
| 360 | + |
| 361 | +Last updated: v0.3.0 |
0 commit comments