pscheduler-result-archiver

The Result Archiver is a REST-based service for ingesting, storing, and visualizing pScheduler test results such as latency, throughput, RTT, MTU, and trace data. It performs idempotent upserts (by run_id and metric_name) into a TimescaleDB backend and exposes endpoints for archival and retrieval, along with built-in OpenAPI/Swagger documentation.

It also supports NMEA 0183 navigation data (GPS position, heading, roll/pitch/heave) for correlating network performance with vessel motion on research ships.

The stack also includes Grafana for visualization and NGINX for TLS termination and routing.

---

Architecture Overview

    ┌──────────────┐
    │  Test Nodes  │  (pScheduler JSON uploads)
    └──────┬───────┘
           │  HTTPS /ps/measurements/*
           ▼
    ┌──────────────────────────┐
    │   Archiver (Python)      │
    │ Connexion + Waitress API │
    │ - /ps/measurements/*    │
    │ - /ps/archives/{run_id} │
    │ - /ps/ui (Swagger UI)   │
    └──────────┬───────────────┘
               │ SQL
               ▼
    ┌──────────────────────────┐
    │   TimescaleDB (Postgres) │
    │   Database: perfsonar    │
    │   Tables: ps_test_results │
    │           ps_trace_hops   │
    │           nav_data        │
    └──────────┬───────────────┘
               │ Grafana datasource (readonly)
               ▼
    ┌──────────────────────────┐
    │   Grafana Dashboards     │
    │   Pre-provisioned views  │
    │   http(s)://<host>:3000  │
    └──────────┬───────────────┘
               │
               ▼
    ┌──────────────────────────┐
    │   NGINX Reverse Proxy    │
    │   TLS (port 8443)        │
    │   /ps → archiver:3500   │
    │   /    → grafana:3000    │
    └──────────────────────────┘

---

Quick Start

1. Clone the repository

git clone https://github.com/kthare10/pscheduler-result-archiver.git
cd pscheduler-result-archiver

2. Prepare directories

mkdir -p tsdb_data grafana_data provisioning/datasources provisioning/dashboards logs certs

3. Generate or copy certificates

Place your valid certificate and key files under:

certs/fullchain.pem
certs/privkey.pem

Note: The certs/ directory is git-ignored. Never commit TLS certificates to version control.

4. Configure environment

Copy the example environment file and fill in your secrets:

cp .env.example .env

Required variables in .env:

Variable	Description
ARCHIVER_DB_PASSWORD	PostgreSQL password for grafana_writer
ARCHIVER_BEARER_TOKEN	Bearer token for API authentication
GRAFANA_ADMIN_PASSWORD	Grafana admin UI password
GRAFANA_ADMIN_USER	Grafana admin username (default: admin)

Docker Compose will refuse to start if required variables are missing.

Optionally review archiver/config.yml for pool tuning, logging, and SSL settings. Environment variables always take precedence over config file values.

---

Docker Compose Deployment

Compose file overview

services:
  timescaledb:
    image: timescale/timescaledb:latest-pg16
    environment:
      - POSTGRES_DB=perfsonar
      - POSTGRES_USER=grafana_writer
      - POSTGRES_PASSWORD=${ARCHIVER_DB_PASSWORD}
    volumes:
      - ./tsdb_data:/var/lib/postgresql/data

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
    volumes:
      - ./grafana_data:/var/lib/grafana
      - ./provisioning/datasources:/etc/grafana/provisioning/datasources
      - ./provisioning/dashboards:/etc/grafana/provisioning/dashboards
    depends_on:
      timescaledb:
        condition: service_healthy

  archiver:
    build:
      context: .
      dockerfile: Dockerfile
    image: kthare10/archiver:1.0.0
    environment:
      - APP_CONFIG_PATH=/etc/archiver/config/config.yml
      - ARCHIVER_DB_PASSWORD=${ARCHIVER_DB_PASSWORD}
      - ARCHIVER_BEARER_TOKEN=${ARCHIVER_BEARER_TOKEN}
    volumes:
      - ./archiver/config.yml:/etc/archiver/config/config.yml
      - ./logs:/var/log/archiver
    depends_on:
      - grafana

  nginx:
    image: nginx:1
    ports: ["8443:443"]
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf
      - ./certs/fullchain.pem:/etc/ssl/public.pem
      - ./certs/privkey.pem:/etc/ssl/private.pem
    depends_on:
      - archiver

---

Start all services

docker-compose up -d

Check running containers

docker ps

Logs

docker-compose logs -f archiver
docker-compose logs -f archiver-nginx

---

Access Points

Service	URL (default)	Notes
Archiver API	https://localhost:8443/ps	Base path for ingestion endpoints
Swagger UI	https://localhost:8443/ps/ui	OpenAPI documentation
Grafana	https://localhost:8443/	Dashboards / visualization
TimescaleDB	timescaledb:5432	Internal DB connection

API Endpoints

Method	Path	Description
POST	/ps/measurements/{category}	Ingest pScheduler test results (latency, throughput, rtt, trace, mtu, clock)
GET	/ps/archives/{run_id}	Retrieve archived results by run ID
POST	/ps/measurements/nav	Ingest NMEA navigation data (batch of GPS/heading/motion points)
GET	/ps/nav	Retrieve navigation data by time range, vessel ID
GET	/ps/health	Health check

---

Authentication

• API supports Bearer and X-API-Key auth schemes.
• The bearer token is set via the ARCHIVER_BEARER_TOKEN environment variable.
• The service will refuse to start if the token is missing or set to a known insecure default.

---

Example Ingestion Request

curl -sk -X POST https://localhost:8443/ps/measurements/throughput \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <your-token>' \
  -d '{
        "run_id": "test-123",
        "src": {"ip": "192.168.1.10", "name": "ship-a"},
        "dst": {"ip": "23.134.232.50", "name": "shore"},
        "direction": "forward",
        "raw": {"tool": "iperf3", "result": {"bits_per_second": 50000000}}
      }'

Expected response:

{
  "status": 200,
  "type": "no_content"
}

Ingest Navigation Data

curl -sk -X POST https://localhost:8443/ps/measurements/nav \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <your-token>' \
  -d '{
        "points": [{
          "ts": "2025-06-15T18:30:00Z",
          "vessel_id": "rv-thompson",
          "latitude": 47.6062,
          "longitude": -122.3321,
          "heading_true": 315.0,
          "roll_deg": 3.5,
          "pitch_deg": 1.2,
          "heave_m": 0.4
        }]
      }'

---

Grafana Integration

Grafana is pre-provisioned with:

• a TimescaleDB datasource (user: grafana_writer)
• prebuilt dashboards under provisioning/dashboards:
  • pScheduler Result Archiver (Pairs) — throughput, latency, RTT, MTU, clock, trace per source/destination pair
  • Navigation Correlation — vessel position map, throughput vs. roll/pitch, RTT vs. heave, heading & GPS quality, motion detail (requires NMEA listener feeding nav_data table)
  • Time-Aligned Environmental Correlation — stacked time-aligned panels for throughput, RTT, latency, heading, roll, pitch, heave (modeled after Sikuliaq paper Figure 6)

Login credentials are controlled by GRAFANA_ADMIN_USER and GRAFANA_ADMIN_PASSWORD environment variables.

To reset password:

docker exec -it grafana grafana-cli admin reset-admin-password newpass
docker restart grafana

---

HTTPS and Reverse Proxy

nginx/default.conf routes:

• /ps/* → Archiver (http://archiver:3500)
• / → Grafana (http://grafana:3000)

TLS is enabled via /etc/ssl/public.pem and /etc/ssl/private.pem.

NGINX is configured with:

• Security headers (HSTS, X-Frame-Options, X-Content-Type-Options, etc.)
• Rate limiting (10 req/s per IP on /ps endpoints)
• Request body size limit (10 MB)

---

Development

Run the API standalone (no Docker):

pip install -r requirements.txt
python -m archiver

Default listens on port 3500. Swagger UI: http://localhost:3500/ps/ui

Run tests

pip install -r test-requirements.txt
pytest archiver/openapi_server/test/

---

Data Model

ps_test_results

Stores pScheduler network measurement results. Composite PK (run_id, metric_name, ts, src_ip, dst_ip, direction) enables idempotent re-ingestion. JSONB aux column stores raw tool output.

Metrics: throughput_mbps, retransmits, delay_ms, jitter_ms, loss_pct, rtt_ms (mean/min/max), mtu_bytes, hop_count, clock_diff_ms, clock_offset_s

ps_trace_hops

Stores traceroute hop-by-hop data. PK (run_id, hop_idx).

nav_data

Stores NMEA 0183 navigation data from research vessels. Composite PK (ts, vessel_id).

Columns: latitude, longitude, altitude_m, fix_quality, num_satellites, hdop, heading_true, motion_status, roll_deg, pitch_deg, heave_m, aux (JSONB)

All tables are TimescaleDB hypertables with 180-day retention and 7-day compression policies.

---

Maintenance

Backup TimescaleDB

docker exec -t timescaledb pg_dump -U grafana_writer perfsonar > backup.sql

Upgrade containers

docker-compose pull
docker-compose up -d --build

---

License

MIT License - 2025 Komal Thareja Part of the FABRIC Testbed Ship-to-Shore Monitoring Stack.

---

References

• pScheduler Documentation
• TimescaleDB
• Grafana Provisioning
• Connexion Framework