Skip to content

Commit 09793c0

Browse files
author
Jordan Benjamin
committed
docs(Phase 5): comprehensive worked examples for all major workflows
- examples/simple_2d.jl: Basic 2D turbulence demonstration * Generates 256×256 grid of synthetic CF data * Computes 2nd-order structure functions * Validates K41 scaling (S_2~r^2/3) * Produces visualization (PNG output) - examples/threaded_calculation.jl: Multi-core parallelization tutorial * 10M-100M points scalable by CPU count * Compares serial vs threaded speedup * Measures parallel efficiency * Best practices: thread count, memory bandwidth, profiling - examples/gpu_acceleration.jl: GPU-accelerated computation * 1 billion points (or 10M CPU fallback) * Uses GPUBackend with KernelAbstractions * Memory efficiency analysis (Float32 vs Float64) * Expected 10-100x speedup over single CPU - examples/distributed_parallel.jl: Multi-process cluster computing * Setup multiple Julia processes * Data locality optimization (generate local) * Demonstrates SLURM submission script template * Linear scaling with process count - examples/real_data_climate.jl: Practical atmospheric data analysis * Synthetic climate data (50k horizontal × 50 vertical × 100 time) * Realistic NaN handling (50% missing in cloud layer) * Outlier detection and removal * K41 scaling validation on real-like data * Multi-time-step ensemble analysis - examples/README.md: Index and tutorial guide * Quick start with run instructions for each example * Performance expectations and hardware requirements * Advanced workflows: combining backends, out-of-core, nested analysis * Common issues and solutions * Links to all documentation files * Guidance for choosing which example to start with All examples include: - Clear docstrings explaining purpose and learning outcomes - Run instructions (with environment variables where needed) - Detailed comments explaining key concepts - Print output summarizing results - Links to relevant documentation - 'Next steps' guidance after execution
1 parent 1473c88 commit 09793c0

6 files changed

Lines changed: 1603 additions & 0 deletions

File tree

examples/README.md

Lines changed: 361 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,361 @@
1+
"""
2+
examples/README.md
3+
4+
Worked examples and tutorials for StructureFunctions.jl
5+
"""
6+
7+
# Examples: Getting Started with StructureFunctions.jl
8+
9+
This directory contains runnable examples demonstrating different aspects of structure function computation in StructureFunctions.jl.
10+
11+
## Quick Start
12+
13+
### 1. Simple 2D Calculation (`simple_2d.jl`)
14+
15+
**Best for**: First-time users, understanding basics
16+
17+
```bash
18+
julia examples/simple_2d.jl
19+
```
20+
21+
**What it does**:
22+
- Generates synthetic 2D turbulent velocity data (256×256 grid)
23+
- Computes 2nd-order structure functions
24+
- Validates K41 scaling ($S_2(r) \sim r^{2/3}$)
25+
- Produces visualization (plot saved)
26+
27+
**Key concepts**:
28+
- Regular grids and structure functions
29+
- Spectral scaling laws
30+
- Visualization of results
31+
32+
**Learn next**: `docs/theory.md` for physics background
33+
34+
---
35+
36+
### 2. Threaded Parallelization (`threaded_calculation.jl`)
37+
38+
**Best for**: Medium datasets (10M–500M points), multi-core machines
39+
40+
```bash
41+
# Set thread count
42+
JULIA_NUM_THREADS=4 julia examples/threaded_calculation.jl
43+
44+
# Or let Julia auto-detect
45+
JULIA_NUM_THREADS=auto julia examples/threaded_calculation.jl
46+
```
47+
48+
**What it does**:
49+
- Generates 50M points (scalable based on CPU cores)
50+
- Compares serial vs threaded execution
51+
- Measures speedup and parallel efficiency
52+
- Demonstrates best practices
53+
54+
**Key metrics**:
55+
- Speedup vs serial: typically 2–8x on 4–32 cores
56+
- Efficiency: depends on memory bandwidth
57+
- Optimal for cache-resident data
58+
59+
**Advanced**: Try different dataset sizes; plot speedup curve vs thread count
60+
61+
**Learn next**: `docs/backends.md` for backend comparison
62+
63+
---
64+
65+
### 3. GPU Acceleration (`gpu_acceleration.jl`)
66+
67+
**Best for**: Large datasets (>500M points), with GPU hardware
68+
69+
```bash
70+
# Requires CUDA.jl + KernelAbstractions.jl
71+
julia --project -e 'using Pkg; Pkg.add(["CUDA", "KernelAbstractions"])'
72+
73+
# Run example
74+
julia examples/gpu_acceleration.jl
75+
```
76+
77+
**What it does**:
78+
- Generates 1 billion points (or 10M if GPU unavailable)
79+
- Uses GPUBackend for computation
80+
- Measures memory efficiency of Float32 vs Float64
81+
- Estimates speedup
82+
83+
**Expected results**:
84+
- NVIDIA A100: 20–50x faster than single CPU
85+
- RTX 4090: 10–30x faster than CPU
86+
- AMD MI250X: 15–80x faster than CPU
87+
88+
**Key insights**:
89+
- Float32 sufficient for turbulence (saves 50% memory)
90+
- GPU shines for >1B points
91+
- Kernel compilation amortized over many calls
92+
93+
**Learn next**: `docs/backends.md#gpubackend`, `docs/real_data.md#performance-tips`
94+
95+
---
96+
97+
### 4. Distributed Computing (`distributed_parallel.jl`)
98+
99+
**Best for**: Massive datasets across clusters (>1B points, multi-node)
100+
101+
```bash
102+
# Local cluster (spawn 4 processes)
103+
julia examples/distributed_parallel.jl
104+
105+
# Custom process count
106+
julia examples/distributed_parallel.jl 8
107+
108+
# Or via Distributed.jl
109+
julia -p 8 examples/distributed_parallel.jl
110+
```
111+
112+
**What it does**:
113+
- Starts multiple Julia processes
114+
- Distributes data locally (no copy)
115+
- Computes structure functions in parallel
116+
- Demonstrates SLURM submission script
117+
118+
**HPC Usage**:
119+
```bash
120+
# Save the SLURM template:
121+
# sbatch submit_distributed.sbatch
122+
123+
# Scales to 100s of processes on HPC clusters
124+
```
125+
126+
**Scaling characteristics**:
127+
- Near-linear speedup with N nodes (for large problems)
128+
- Suitable for >500M points
129+
- Minimal communication overhead if data is local
130+
131+
**Learn next**: `docs/backends.md#distributedbackend`
132+
133+
---
134+
135+
### 5. Real Climate Data (`real_data_climate.jl`)
136+
137+
**Best for**: Practical atmospheric/ocean applications
138+
139+
```bash
140+
# Optional: install NetCDF support
141+
julia -e 'using Pkg; Pkg.add("NetCDF")'
142+
143+
# Run with synthetic data (generates demo)
144+
julia examples/real_data_climate.jl
145+
```
146+
147+
**What it does**:
148+
- Simulates realistic atmospheric turbulence data
149+
- Handles missing data (NaNs) and outliers
150+
- Constructs 3D coordinate systems
151+
- Performs K41 scaling analysis
152+
- Multi-time-step statistics
153+
154+
**Key workflows**:
155+
1. Data loading (NetCDF, HDF5, CSV)
156+
2. Preprocessing (unit conversion, detrending)
157+
3. Structure function calculation
158+
4. K41 validation
159+
5. Temporal statistics
160+
161+
**Real data sources**:
162+
- SOCRATES campaign: https://data.eol.ucar.edu/
163+
- ERA5 reanalysis: https://cds.climate.copernicus.eu/
164+
- MERRA-2: https://gmao.gsfc.nasa.gov/reanalysis/MERRA-2/
165+
- WRF simulations: Custom NetCDF output
166+
167+
**Learn next**: `docs/real_data.md`
168+
169+
---
170+
171+
## Advanced Workflows
172+
173+
### Combining Backends
174+
175+
**Scenario**: Process each file with GPU, aggregate across time
176+
177+
```julia
178+
using StructureFunctions, DistributedArrays
179+
180+
# Load time series in parallel
181+
for t in 1:n_time_steps
182+
# Each worker computes one time step
183+
result_t = @distributed (+) for i in workers()
184+
data_i = load_time_slice(t, i)
185+
x, u = construct_coordinates(data_i)
186+
calculate_structure_function(x, u, bins; backend=GPUBackend())
187+
end
188+
end
189+
```
190+
191+
### Batch Processing Out-of-Core
192+
193+
**Scenario**: Dataset too large for memory (terabytes)
194+
195+
```julia
196+
# Process in streaming chunks
197+
all_sums = zeros(n_bins)
198+
all_counts = zeros(Int, n_bins)
199+
200+
for chunk in read_chunks("huge_data.zarr")
201+
x, u = extract_data(chunk)
202+
result = calculate_structure_function(x, u, bins; backend=AutoBackend())
203+
all_sums .+= result.sums
204+
all_counts .+= result.counts
205+
end
206+
207+
# Normalize at end
208+
final_sf = all_sums ./ all_counts
209+
```
210+
211+
### Multi-Scale Nested Analysis
212+
213+
**Scenario**: Analyze turbulence at multiple length scales
214+
215+
```julia
216+
# Coarse grid: long-range structure
217+
bins_coarse = 100:500:100_000
218+
result_coarse = calculate_structure_function(x, u, bins_coarse)
219+
220+
# Fine grid: local correlation
221+
bins_fine = 1:0.5:100
222+
result_fine = calculate_structure_function(x, u, bins_fine)
223+
224+
# Compare scaling exponents
225+
alpha_coarse = estimate_exponent(result_coarse)
226+
alpha_fine = estimate_exponent(result_fine)
227+
```
228+
229+
---
230+
231+
## Running All Examples
232+
233+
```bash
234+
# Prerequisites
235+
julia --project -e 'using Pkg; Pkg.instantiate()'
236+
237+
# Run all in sequence
238+
for script in simple_2d threaded_calculation gpu_acceleration distributed_parallel real_data_climate; do
239+
echo "Running example: $script"
240+
julia examples/${script}.jl
241+
done
242+
```
243+
244+
---
245+
246+
## Performance Expectations
247+
248+
| Example | Input Size | Time | Backend |
249+
|---------|-----------|------|---------|
250+
| simple_2d | 65K pts | ~1 sec | Serial |
251+
| threaded_calculation | 50M pts | ~1 sec | Threaded (4 cores) |
252+
| gpu_acceleration | 1B pts | ~20 sec | GPU (A100) |
253+
| distributed_parallel | 1B pts | ~30 sec | Multi-process |
254+
| real_data_climate | 50K pts | ~0.5 sec | Serial |
255+
256+
*Actual times vary by hardware. Use `@time` to measure.*
257+
258+
---
259+
260+
## Choosing an Example to Start With
261+
262+
### I want to...
263+
264+
- **Learn the basics**`simple_2d.jl`
265+
- **Speed up my computation**`threaded_calculation.jl`
266+
- **Use my GPU**`gpu_acceleration.jl`
267+
- **Scale to a supercomputer**`distributed_parallel.jl`
268+
- **Analyze real atmospheric data**`real_data_climate.jl`
269+
- **Combine multiple approaches** → Mix the examples!
270+
271+
---
272+
273+
## Common Issues & Solutions
274+
275+
### "ThreadedBackend not found"
276+
```julia
277+
julia> ThreadedBackend()
278+
ERROR: UndefVarError: ThreadedBackend not defined
279+
280+
# Solution: Load OhMyThreads
281+
using OhMyThreads # Triggers extension
282+
```
283+
284+
### "GPU out of memory"
285+
```julia
286+
# Solution: Reduce dataset size or use Float32
287+
x = Float32.(x) # Half the memory
288+
u = Float32.(u)
289+
290+
result = calculate_structure_function(x, u, bins; backend=GPUBackend())
291+
```
292+
293+
### "Structure function is NaN"
294+
```julia
295+
# Solution: Check for NaN inputs or remove them
296+
sum(isnan.(u)) # Check NaNs
297+
u_clean = u[.!isnan.(u), :] # Remove
298+
299+
# Or let calculate_structure_function handle it
300+
result = calculate_structure_function(x, u, bins) # Skips NaN pairs
301+
```
302+
303+
### "Serial computation is too slow"
304+
```julia
305+
# Solution: Use ThreadedBackend
306+
backend = ThreadedBackend() # Requires OhMyThreads
307+
308+
# Or try GPU
309+
backend = GPUBackend() # Requires KernelAbstractions
310+
```
311+
312+
---
313+
314+
## Further Reading
315+
316+
After the examples, dive into the comprehensive docs:
317+
318+
- **Theory**: `docs/theory.md` — Physics and mathematics
319+
- **Architecture**: `docs/architecture.md` — Internal design
320+
- **Backends**: `docs/backends.md` — Detailed backend guide
321+
- **Extensions**: `docs/extensions.md` — Lazy loading system
322+
- **Real Data**: `docs/real_data.md` — File I/O and workflows
323+
324+
---
325+
326+
## Contributing Examples
327+
328+
To add a new example:
329+
330+
1. Create `examples/new_example.jl` with:
331+
- Clear docstring explaining what it does
332+
- Section headers for organization
333+
- Comments explaining key concepts
334+
- Print statements summarizing results
335+
- Links to relevant docs
336+
337+
2. Update this README with:
338+
- Brief description
339+
- Run instructions
340+
- Key concepts learned
341+
- Next steps
342+
343+
3. Test it runs without errors:
344+
```bash
345+
julia examples/new_example.jl
346+
```
347+
348+
---
349+
350+
## Questions?
351+
352+
- Check docstrings: `?calculate_structure_function`
353+
- Search examples/ for similar workflows
354+
- Read the relevant doc file (theory / backends / real_data)
355+
- Open an issue on GitHub
356+
357+
---
358+
359+
**Happy computing! 🚀**
360+
361+
Last updated: v0.3.0

0 commit comments

Comments
 (0)