|
1 | 1 | # Random Video Clips Streaming Server |
2 | 2 |
|
3 | | -[](https://github.com/vineethvijay/Random-Video-Clips-Streaming-Server/actions/workflows/ci.yml) |
| 3 | +A containerized live streaming server that shuffles and streams short clips from your video collection seamlessly into a never-ending HLS live stream—complete with continuous background audio and a cross-platform Flutter TV/Web UI. |
4 | 4 |
|
5 | | -A containerized live streaming server that continuously shuffles and streams random short clips from your video collection and plays them as a never-ending HLS live stream - with optional continuous background audio that plays independently from the video shuffling. |
6 | | - |
7 | | -## Features |
8 | | - |
9 | | -- **Web Dashboard** — view and control video chunks (pagination, Now Playing, progress), see stream stats (hours played, chunks created), open a live HLS stream, check source videos, and watch live system info (CPU, memory, GPU). |
10 | | -- **Continuous live stream** — no playback gaps; clips are piped into a single persistent RTMP connection. |
11 | | -- **Continuous background audio** — mount an MP3 folder; the same track plays across chunk switches (position tracked and resumed). Dashboard shows audio “Now Playing” and progress within the track. |
12 | | -- **Strict LRU + segment tracking** — video files are rotated with a Least-Recently-Used queue. A JSON-based **segment tracker** records used time ranges per file so new clips prefer **unused** segments; a Python helper (`scripts/segment_tracker.py`) picks start times accordingly. |
13 | | -- **Chunk metadata** — each generated chunk has a `.meta.json` sidecar listing source video filenames (no database). |
14 | | -- **Persistent stats** — optional `STATS_DIR` (e.g. `./stats`) stores hours played and chunks-ever-created so they survive restarts and new deployments. |
15 | | -- **Normalized output** — all clips transcoded to a consistent resolution/fps so transitions are smooth. |
16 | | -- **Compatible** — works with VLC, Safari, Samsung TV IPTV apps, and any HLS player. |
17 | | -- **Production-grade** — Dockerized with Gunicorn, healthchecks, and log rotation. |
| 5 | +## Key Features |
| 6 | +- **Interactive Flutter Client** — A cross-platform web and Android UI providing complete control over chunks, live stream viewing, system metrics, and audio management. |
| 7 | +- **Continuous Live Stream** — Zero playback gaps; clips are piped into a single persistent RTMP connection. |
| 8 | +- **Continuous Audio Track** — Mount an MP3 folder; the same track plays across video chunk switches. |
| 9 | +- **Strict LRU + Segment Tracking** — Clips prefer unused segments, minimizing repetitive content. |
| 10 | +- **Normalized Output** — All clips transcoded to a consistent resolution/fps so transitions are butter-smooth. |
| 11 | +- **Production-grade** — Dockerized with Gunicorn, healthchecks, GPU-support, and log rotation. |
18 | 12 |
|
19 | 13 | <details> |
20 | | -<summary><strong style="color:#0366d6">📷 Dashboard Preview</strong></summary> |
21 | | - |
22 | | - |
23 | | - |
24 | | -</details> |
25 | | - |
26 | | -## Architecture |
| 14 | +<summary><strong style="color:#0366d6">🏗️ System Architecture Diagram</strong></summary> |
27 | 15 |
|
28 | 16 | ```mermaid |
29 | 17 | flowchart TD |
@@ -86,165 +74,56 @@ flowchart TD |
86 | 74 | style Host fill:transparent,stroke:#6a737d,stroke-dasharray: 5 5 |
87 | 75 | bash ~~~ API |
88 | 76 | ``` |
| 77 | +</details> |
89 | 78 |
|
90 | | -### Chunk Generation Randomization |
91 | | - |
92 | | -See [Chunk Generation Flow](docs/chunk-generation.md) for the full diagram and behavior summary. |
93 | | - |
94 | | -The system operates across three decoupled, robust containers: |
95 | | -- **`chunk-generator`** — runs in the background polling for manual UI triggers or automated schedules to build 5-minute `.mp4` chunks from your video library. Uses a per-video LRU queue and a segment tracker (`.used_segments.json`) so clip start times prefer unused ranges within each file. |
96 | | -- **`random-video-streamer`** — the brain and web interface. Mixes the chunks with continuous looping background audio, preventing gaps in playback and pushing a 24/7 RTMP feed to NGINX. |
97 | | -- **`nginx-rtmp`** — receives the feed, packages it efficiently into HLS segments on a temporary filesystem, and serves it seamlessly to devices over HTTP. |
98 | | - |
99 | | -## Quick Start |
| 79 | +## 🚀 Quick Start |
100 | 80 |
|
101 | | -**1. Configure `.env`:** |
| 81 | +**1. Configure Environment:** |
102 | 82 | ```bash |
103 | 83 | cp .env.example .env |
| 84 | +# Edit .env to set your VIDEO_FOLDER and optionally AUDIO_FOLDER |
104 | 85 | ``` |
105 | | -Edit `.env`: |
106 | | -```bash |
107 | | -VIDEO_FOLDER=/path/to/your/videos |
108 | 86 |
|
109 | | -# Optional: folder of MP3s to play as continuous background audio |
110 | | -# Leave empty for a silent stream |
111 | | -AUDIO_FOLDER=/path/to/your/music |
112 | | - |
113 | | -PORT=8081 # Flask API port |
114 | | -``` |
115 | | - |
116 | | -**2. Start:** |
| 87 | +**2. Start the Backend Servers:** |
117 | 88 | ```bash |
| 89 | +# Natively starts the API, Chunk Generator, and NGINX HLS packager |
118 | 90 | docker compose up -d |
119 | 91 | ``` |
120 | 92 |
|
121 | | -**3. Watch & Manage:** |
122 | | -| URL | Purpose | |
123 | | -|-----|---------| |
124 | | -| `http://server-ip:8081/` | **Web Dashboard** (chunks, audio, live HLS player, stream stats, system info with live CPU/mem/GPU, Generate Chunks, Play next) | |
125 | | -| `http://server-ip:8082/hls/stream.m3u8` | **Live HLS stream** (VLC, Safari, TV apps) | |
126 | | -| `http://server-ip:8081/iptv.m3u` | IPTV playlist (points to the HLS stream) | |
127 | | -| `http://server-ip:8081/api/status` | Server status | |
128 | | -| `http://server-ip:8081/api/stream-status` | Clip pusher status (current chunk/audio, hours played, etc.) | |
129 | | -| `http://server-ip:8081/api/system-usage` | Live CPU, memory, GPU usage (for dashboard) | |
130 | | - |
131 | | -## TV Setup |
132 | | - |
133 | | -1. Install an IPTV app (e.g. SS IPTV, TiviMate, Smart IPTV) |
134 | | -2. Add playlist URL: `http://server-ip:8081/iptv.m3u` |
135 | | - |
136 | | -The IPTV playlist points to the live HLS stream automatically. |
137 | | - |
138 | | -## Configuration |
139 | | - |
140 | | -The system is configured via environment variables in the `.env` file and service definitions in `docker-compose.yml`. |
141 | | - |
142 | | -### 1. Streamer & API Configuration (`random-video-streamer`) |
143 | | - |
144 | | -These variables control the Flask API and the RTMP clip pusher. |
145 | | - |
146 | | -| Variable | Default | Description | |
147 | | -|----------|---------|-------------| |
148 | | -| `PORT` | `8081` | Host port for the Flask API and IPTV playlist | |
149 | | -| `VIDEO_FOLDER` | `/videos` | Source video directory (mount in Compose) | |
150 | | -| `AUDIO_FOLDER` | `/audio` | Background MP3 directory. If empty, uses video audio. | |
151 | | -| `CHUNK_FOLDER` | `/chunks` | Directory where `.mp4` chunks are read from | |
152 | | -| `STATS_DIR` | *(none)* | **Persistent stats dir.** When set (e.g. `./stats`), hours played and chunks-ever-created are stored here so they survive new deployments. Compose mounts `${STATS_DIR:-./stats}` as `/app/stats`. Ensure this dir exists and is writable by both containers (e.g. `mkdir -p ./stats && chmod 777 ./stats` or match the streamer’s UID). | |
153 | | -| `RTMP_URL` | `rtmp://nginx-rtmp:1935/live/stream` | Internal target for the RTMP stream | |
154 | | - |
155 | | -### 2. Chunk Generator Configuration (`chunk-generator`) |
156 | | - |
157 | | -These variables control how new video chunks are created from your library. |
158 | | - |
159 | | -| Variable | Default | Description | |
160 | | -|----------|---------|-------------| |
161 | | -| `CHUNK_DURATION` | `300` | Target length of a single consolidated chunk (seconds) | |
162 | | -| `CLIP_MIN` | `6` | Minimum length of an individual clip within a chunk | |
163 | | -| `CLIP_MAX` | `6` | Maximum length of an individual clip within a chunk | |
164 | | -| `CHUNKS_PER_RUN` | `4` | How many chunks to generate per execution | |
165 | | -| `MAX_CHUNKS` | `56` | Max number of chunks to keep before pruning oldest | |
166 | | -| `CRON_SCHEDULE` | `0 2 * * *` | Cron schedule for automatic generation (2am daily). Set empty to disable. | |
167 | | -| `VIDEO_DIR` | `/videos` | Where the generator searches for `.mp4`, `.mkv`, `.avi` | |
168 | | -| `OUTPUT_DIR` | `/chunks` | Where the generator writes the final chunks | |
169 | | -| `TUBEARCHIVIST_URL` | *(none)* | TubeArchivist instance URL for metadata extraction (e.g. `https://ta.example.com`). Requires `TUBEARCHIVIST_TOKEN`. | |
170 | | -| `TUBEARCHIVIST_TOKEN` | *(none)* | TubeArchivist API token (from Settings → Application). | |
171 | | - |
172 | | -## Hardware Requirements |
| 93 | +**3. Build & Launch the Client UI:** |
| 94 | +```bash |
| 95 | +cd scripts/ |
| 96 | +./build-deploy-flutter.sh |
| 97 | +``` |
| 98 | +_Follow the setup prompt to build your preferred Web or Android APK target. The script will automatically serve the web app locally!_ |
173 | 99 |
|
174 | | -### GPU Acceleration (NVIDIA) vs CPU (Mac / Linux) |
175 | | -The **Chunk Generator** supports both CPU encoding (`libx264`) and NVIDIA hardware acceleration (`h264_nvenc`). |
| 100 | +## 📺 Raw Playback Links |
| 101 | +If you just want to consume the stream directly on your Smart TV or Phone (VLC, Safari, Apple TV): |
| 102 | +- **Live HLS Stream:** `http://<server-ip>:8082/hls/stream.m3u8` |
| 103 | +- **Smart TV IPTV Playlist:** `http://<server-ip>:8081/iptv.m3u` |
176 | 104 |
|
177 | | -**For CPU Encoding (Default & Mac-compatible):** |
178 | | -- Ensure `.env` has `HW_ACCEL=none`. |
179 | | -- Run completely natively with `docker compose up -d`. |
| 105 | +## ⚙️ Configuration & Hardware |
| 106 | +Environment parameters are centrally managed in `.env` and `scripts/flutter_config.env`. |
180 | 107 |
|
181 | | -**For NVIDIA GPU Acceleration:** |
182 | | -- **Drivers**: Host must have NVIDIA drivers and `nvidia-container-toolkit` installed. |
183 | | -- **Config**: Set `HW_ACCEL=nvidia` in `.env`. |
184 | | -- **Run**: Use the GPU override file to reserve hardware resources: |
185 | | - ```bash |
186 | | - docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d |
187 | | - ``` |
| 108 | +By default, standard CPU encoding (`libx264`) is enabled. If you have an NVIDIA GPU installed, set `HW_ACCEL=nvidia` in `.env` and run with the GPU architecture override: |
| 109 | +```bash |
| 110 | +docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d |
| 111 | +``` |
| 112 | +_For extended metadata extraction features via TubeArchivist, see `TUBEARCHIVIST_URL` inside `.env.example`._ |
188 | 113 |
|
189 | | -## API Endpoints |
| 114 | +## Core API Endpoints |
| 115 | +The Flutter GUI interfaces with the following Flask API (Running on `:8081`): |
190 | 116 |
|
191 | 117 | | Method | Endpoint | Description | |
192 | 118 | |--------|----------|-------------| |
193 | | -| GET | `/` | Web Dashboard HTML | |
194 | 119 | | GET | `/api/status` | Full server status & config overview | |
195 | | -| GET | `/api/stream-status` | RTMP pusher & current chunk/audio, hours played, chunks created | |
196 | | -| GET | `/api/system-usage` | Live CPU %, memory %, GPU % (for dashboard polling) | |
197 | | -| GET | `/chunks/<filename>` | Serve a chunk file (e.g. for “Play” in dashboard) | |
198 | | -| GET | `/audio/<path>` | Serve an audio file for playback in browser | |
199 | | -| GET | `/iptv.m3u` | IPTV playlist (M3U) for external players | |
200 | | -| POST | `/api/generate_chunk` | Triggers the generator to build new chunks | |
201 | | -| POST | `/api/skip_to_next` | Skip current chunk and advance to next (audio position preserved) | |
202 | | -| POST | `/api/skip_to_next_audio` | Skip to the next audio track | |
203 | | -| POST | `/api/play_chunk` | Play a specific chunk next in the live stream (body: `{"chunk_name": "xyz.mp4"}`) | |
204 | | - |
205 | | -## Troubleshooting |
206 | | - |
207 | | -**Stream not starting / HLS 404** |
208 | | -The stream takes ~10 seconds to appear after startup. The app waits for `nginx-rtmp` to be healthy before pushing. |
209 | | - |
210 | | -**Check logs:** |
211 | | -```bash |
212 | | -docker compose logs -f random-video-streamer # Clip pusher & API logs |
213 | | -docker compose logs -f chunk-generator # Encoding progress logs |
214 | | -docker compose logs -f nginx-rtmp # HLS server logs |
215 | | -``` |
| 120 | +| GET | `/api/stream-status` | RTMP pusher & current chunk/audio progress | |
| 121 | +| GET | `/api/system-usage` | Live CPU %, memory %, GPU % | |
| 122 | +| POST | `/api/generate_chunk` | Force triggers the generator to build new chunks | |
| 123 | +| POST | `/api/skip_to_next` | Skip current chunk and advance to next video smoothly | |
| 124 | +| POST | `/api/play_chunk` | Play a specific chunk next in the live stream | |
216 | 125 |
|
217 | | -## Deploy to Proxmox |
218 | | - |
219 | | -See `deploy/README.md`. Run `./deploy/deploy-to-proxmox.sh` to sync to the server. Config in `deploy/config`. |
220 | | - |
221 | | -## Scripts |
222 | | - |
223 | | -- `scripts/segment_tracker.py` — used by the chunk generator to pick unused time ranges and record used segments (JSON file). |
224 | | -- `scripts/tubearchivist_metadata.py` — fetches TubeArchivist video metadata and extracts model info from descriptions (when `TUBEARCHIVIST_URL` + token set). |
225 | | -- `deploy/deploy-to-proxmox.sh` — rsync deploy to Proxmox/Linux (excludes .env, chunks, videos, etc.) |
226 | | -- `scripts/setup.sh` — initial setup and environment check |
227 | | -- `scripts/start.sh` — start the streaming server |
228 | | - |
229 | | -## Segment tracking |
230 | | - |
231 | | -The chunk generator keeps a JSON file (`.used_segments.json`, under `STATS_DIR` or the chunk folder) that records which time ranges have been used in each source video. The Python helper `scripts/segment_tracker.py` (used by `generate_chunk.sh`) picks a start time in an **unused** range; if none fit or the tracker is unavailable, it falls back to a random start. This reduces repetition of the same segment within a video. |
232 | | - |
233 | | -## TubeArchivist metadata |
234 | | - |
235 | | -When your source videos come from [TubeArchivist](https://github.com/tubearchivist/tubearchivist), you can enable metadata extraction so the dashboard shows **model info** (e.g. `Model - https://www.instagram.com/...`) from video descriptions. |
236 | | - |
237 | | -**Setup:** |
238 | | -1. Add to `.env`: |
239 | | - ``` |
240 | | - TUBEARCHIVIST_URL=https://your-ta-instance.com |
241 | | - TUBEARCHIVIST_TOKEN=your_api_token |
242 | | - ``` |
243 | | -2. Get the token from TubeArchivist → Settings → Application → API Token. |
244 | | -3. Ensure the chunk-generator container can reach the TubeArchivist URL (same host, Docker network, or public URL). |
245 | | - |
246 | | -The generator extracts video IDs from paths (`channel_id/video_id.mp4`), fetches metadata via the REST API, and parses lines matching `Model - <url>` in the description. Model links appear in the dashboard’s **Model** column. |
| 126 | +_Refer to `backend/app.py` for all native REST/JSON routes._ |
247 | 127 |
|
248 | 128 | ## License |
249 | | - |
250 | 129 | MIT |
0 commit comments