feat: updated structure setup script to keep development artifacts in strict isolation(prevents monolithic notebook workflows)

Author: iTrauco

README.md

@@ -7,6 +7,62 @@ Built from lessons learned through iterative experimentation to enable reproduci
 
 ---
 
+## Table of Contents
+
+- [Project Structure](#project-structure)
+- [Scope](#scope)
+- [Previous Work](#previous-work)
+- [Quick Start](#quick-start)
+- [Notebook Tools Installation](#notebook-tools-installation)
+- [Reproducibility Framework](#reproducibility-framework)
+  - [Environment Setup](#environment-setup)
+  - [Environment Details](#environment-details)
+  - [Environment Management](#environment-management)
+- [Scripts](#scripts)
+- [Workflows](#workflows)
+
+---
+
+## Project Structure
+
+```
+experiments-framework/
+├── notebooks/                                  # all jupyter notebooks
+│   ├── scripts/                               # notebook utility scripts
+│   ├── templates/                             # clean starting templates
+│   │   ├── 01_preprocessing.working.ipynb     # data prep template
+│   │   ├── 02_annotation.working.ipynb        # labeling template
+│   │   ├── 03_training.working.ipynb          # model training template
+│   │   └── systems.working.ipynb              # infrastructure template
+│   ├── machine_learning/                      # ml development
+│   │   ├── 01_preprocessing.dev.ipynb         # active preprocessing
+│   │   ├── 02_annotation.dev.ipynb            # active annotation
+│   │   ├── 03_training.dev.ipynb              # active training
+│   │   ├── preprocessing/                     # preprocessing experiments
+│   │   ├── annotation/                        # annotation experiments
+│   │   └── training/                          # training experiments
+│   └── systems/                               # infrastructure notebooks
+│       └── systems.dev.ipynb                  # systems development
+├── data/                                      # data organization
+│   ├── raw/                                   # original recordings
+│   ├── clips/                                 # extracted video clips
+│   ├── frames/                                # extracted images
+│   └── annotations/                           # labels and metadata
+├── configs/                                   # workflow configurations
+├── models/                                    # trained ml models
+├── scripts/                                   # project setup scripts
+│   ├── setup_experiments_structure.sh         # creates dirs/notebooks
+│   ├── setup_provenance.sh                    # tracks repo evolution
+│   └── setup_orcid.sh                         # citation setup
+├── lib/                                       # reusable code modules
+│   └── notebook_tools/                        # notebook utilities
+├── references/                                # citations and refs
+├── environment.yml                            # conda environment
+└── PROVENANCE.md                              # repo history tracking
+```
+
+---
+
 ## Scope
 
 * Modular workflow patterns for preprocessing, annotation, and training
@@ -23,6 +79,25 @@ Built from lessons learned through iterative experimentation to enable reproduci
 
 This repo rebuilds the workflow framework to be modular and scalable.
 
+## Quick Start
+
+1. Clone and setup:
+   ```bash
+   git clone https://github.com/iTrauco/experiments-framework.git
+   cd experiments-framework
+   chmod +x scripts/*.sh
+   ./scripts/setup_experiments_structure.sh
+   ```
+
+2. Track your work:
+   ```bash
+   ./scripts/setup_provenance.sh
+   ```
+
+3. Start developing in the `.dev` notebooks or copy templates to begin new experiments.
+
+---
+
 ## Notebook Tools Installation
 
 ```bash
@@ -107,6 +182,42 @@ The environment includes essential data science and computer vision packages:
 
 ---
 
+## Scripts
+
+### setup_experiments_structure.sh
+Creates the complete project directory structure with interactive menu. Features:
+- Default creates in parent directory (`../`)
+- Remembers last used location
+- Rollback option to undo
+- Installs GitHub Actions and pre-commit hooks
+
+### setup_provenance.sh
+Documents your experimental evolution across repository rebuilds:
+- Links to previous repos and branches
+- Records what you tested and learned
+- Builds a timeline in PROVENANCE.md
+
+### setup_orcid.sh
+Adds ORCID identifier and citation infrastructure to your project.
+
+---
+
+## Workflows
+
+### Development Flow
+1. Copy templates from `notebooks/templates/` to start new work
+2. Develop in `.dev` notebooks at the machine_learning level
+3. Create experimental variations in subdirectories
+4. Track successful patterns in LESSONS_LEARNED.md files
+
+### GitHub Actions
+Automatically converts notebooks to markdown on push for better documentation and diffs.
+
+### Pre-commit Hooks
+Validates notebook metadata and cleans outputs before commits.
+
+---
+
 ### Environment Management
 
 For collaborators who enhance the environment with additional packages:
@@ -119,16 +230,5 @@ conda env export > environment.yml
 
 This ensures full reproducibility across systems by preserving all dependencies and versions.
 
-
 ---
-**Author:** Christopher Trauco | [ORCID: 0009-0005-8113-6528](https://orcid.org/0009-0005-8113-6528)
-
----
-
-## References
-
-This repository includes citation tracking files located in the `references/` directory:
-
-* [`citations.bib`](references/citations.bib)
-* [`datasets.bib`](references/datasets.bib)
-* [`software.bib`](references/software.bib)
+**Author:** Christopher Trauco | [ORCID: 0009-0005-8113-6528](https://orcid.org/0009-0005-8113-6528)

scripts/README.md

@@ -35,22 +35,33 @@ chmod +x scripts/setup_experiments_structure.sh
 ```
 notebooks/
 ├── README.md                                    # main notebooks info
+├── scripts/
+│   └── README.md                               # notebook utilities
 ├── templates/
-│   └── README.md                               # reusable templates
-├── preprocessing/
-│   ├── README.md                               # preprocessing docs
-│   └── 01_preprocessing.working.ipynb          # video to frames
-├── annotation/
-│   ├── README.md                               # annotation docs
-│   └── 02_annotation.working.ipynb             # labeling workflow
-└── training/
-    ├── README.md                               # training docs
-    └── 03_training.working.ipynb              # model training
+│   ├── README.md                               # reusable templates
+│   ├── 01_preprocessing.working.ipynb          # preprocessing template
+│   ├── 02_annotation.working.ipynb             # annotation template
+│   ├── 03_training.working.ipynb               # training template
+│   └── systems.working.ipynb                   # systems template
+├── machine_learning/
+│   ├── README.md                               # ml workflows
+│   ├── 01_preprocessing.dev.ipynb              # preprocessing dev
+│   ├── 02_annotation.dev.ipynb                 # annotation dev
+│   ├── 03_training.dev.ipynb                   # training dev
+│   ├── preprocessing/                          # preprocessing experiments
+│   │   └── README.md                           
+│   ├── annotation/                             # annotation experiments
+│   │   └── README.md                           
+│   └── training/                               # training experiments
+│       └── README.md
+└── systems/
+    ├── README.md                               # systems docs
+    └── systems.dev.ipynb                       # dev notebook
 data/
 ├── raw/
 │   └── README.md                               # original recordings
 ├── clips/
-│   └── README.md                               # 60s extracts
+│   └── README.md                               # extracted clips
 ├── frames/
 │   ├── README.md                               # extracted images
 │   └── sample_camera_20250622/                 # sample folder