Skip to content

Commit e6d4183

Browse files
chrishagglund-ship-itchrishagglund-ship-it
committed
certification worker
1 parent 1d4cbdf commit e6d4183

11 files changed

Lines changed: 887 additions & 0 deletions

‎Dockerfile‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
FROM node:18-alpine AS build
2+
WORKDIR /package
3+
COPY package*.json ./
4+
RUN npm ci
5+
COPY . .
6+
RUN npm run build
7+
8+
FROM build AS harness-build
9+
RUN npx tsup harness/main.ts \
10+
--outDir /app \
11+
--format cjs \
12+
--target node18 \
13+
--no-splitting
14+
15+
FROM node:18-alpine AS harness-deps
16+
WORKDIR /package
17+
COPY package*.json ./
18+
RUN npm ci --omit=dev
19+
20+
FROM node:18-alpine AS harness
21+
RUN adduser -D -u 65532 nonroot
22+
USER nonroot
23+
WORKDIR /app
24+
COPY --from=harness-deps /package/node_modules /app/node_modules
25+
COPY --from=harness-deps /package/package.json /app/package.json
26+
COPY --from=harness-build /app/main.js /app/main.js
27+
ENTRYPOINT ["node", "main.js"]

‎harness/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# JS SDK Docker Harness
2+
3+
Two Docker targets built from the root `Dockerfile`: an **SDK build** and a **long-running worker harness**.
4+
5+
## Worker Harness
6+
7+
A self-feeding worker that runs indefinitely. On startup it registers five simulated tasks (`js_worker_0` through `js_worker_4`) and the `js_simulated_tasks_workflow`, then runs two background services:
8+
9+
- **WorkflowGovernor** -- starts a configurable number of `js_simulated_tasks_workflow` instances per second (default 2), indefinitely.
10+
- **SimulatedTaskWorkers** -- five task handlers, each with a codename and a default sleep duration. Each worker supports configurable delay types, failure simulation, and output generation via task input parameters. The workflow chains them in sequence: quickpulse (1s) → whisperlink (2s) → shadowfetch (3s) → ironforge (4s) → deepcrawl (5s).
11+
12+
### Building Locally
13+
14+
```bash
15+
docker build --target harness -t js-sdk-harness .
16+
```
17+
18+
### Multiplatform Build and Push
19+
20+
To build for both `linux/amd64` and `linux/arm64` and push to GHCR:
21+
22+
```bash
23+
# One-time: create a buildx builder if you don't have one
24+
docker buildx create --name multiarch --use --bootstrap
25+
26+
# Build and push
27+
docker buildx build \
28+
--platform linux/amd64,linux/arm64 \
29+
--target harness \
30+
-t ghcr.io/conductor-oss/javascript-sdk/harness-worker:latest \
31+
--push .
32+
```
33+
34+
> **Note:** Multi-platform builds require `docker buildx` and a builder that supports cross-compilation. On macOS this works out of the box with Docker Desktop. On Linux you may need to install QEMU user-space emulators:
35+
>
36+
> ```bash
37+
> docker run --privileged --rm tonistiigi/binfmt --install all
38+
> ```
39+
40+
### Running
41+
42+
```bash
43+
docker run -d \
44+
-e CONDUCTOR_SERVER_URL=https://your-cluster.example.com/api \
45+
-e CONDUCTOR_AUTH_KEY=$CONDUCTOR_AUTH_KEY \
46+
-e CONDUCTOR_AUTH_SECRET=$CONDUCTOR_AUTH_SECRET \
47+
-e HARNESS_WORKFLOWS_PER_SEC=4 \
48+
js-sdk-harness
49+
```
50+
51+
You can also run the harness locally without Docker:
52+
53+
```bash
54+
export CONDUCTOR_SERVER_URL=https://your-cluster.example.com/api
55+
export CONDUCTOR_AUTH_KEY=$CONDUCTOR_AUTH_KEY
56+
export CONDUCTOR_AUTH_SECRET=$CONDUCTOR_AUTH_SECRET
57+
58+
npx tsx harness/main.ts
59+
```
60+
61+
Override defaults with environment variables as needed:
62+
63+
```bash
64+
HARNESS_WORKFLOWS_PER_SEC=4 HARNESS_BATCH_SIZE=10 npx tsx harness/main.ts
65+
```
66+
67+
All resource names use a `js_` prefix so multiple SDK harnesses (C#, Python, Go, etc.) can coexist on the same cluster.
68+
69+
### Environment Variables
70+
71+
| Variable | Required | Default | Description |
72+
|---|---|---|---|
73+
| `CONDUCTOR_SERVER_URL` | yes | -- | Conductor API base URL |
74+
| `CONDUCTOR_AUTH_KEY` | no | -- | Orkes auth key |
75+
| `CONDUCTOR_AUTH_SECRET` | no | -- | Orkes auth secret |
76+
| `HARNESS_WORKFLOWS_PER_SEC` | no | 2 | Workflows to start per second |
77+
| `HARNESS_BATCH_SIZE` | no | 20 | Number of tasks each worker polls per batch |
78+
| `HARNESS_POLL_INTERVAL_MS` | no | 100 | Milliseconds between poll cycles |

‎harness/main.ts‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
import {
2+
OrkesClients,
3+
ConductorWorkflow,
4+
TaskHandler,
5+
simpleTask,
6+
} from "../src/sdk";
7+
import { MetadataResource } from "../src/open-api/generated";
8+
import type { ConductorWorker } from "../src/sdk/clients/worker/types";
9+
import { SimulatedTaskWorker } from "./simulatedTaskWorker";
10+
import { WorkflowGovernor } from "./workflowGovernor";
11+
12+
const WORKFLOW_NAME = "js_simulated_tasks_workflow";
13+
14+
const SIMULATED_WORKERS: {
15+
taskName: string;
16+
codename: string;
17+
sleepSeconds: number;
18+
}[] = [
19+
{ taskName: "js_worker_0", codename: "quickpulse", sleepSeconds: 1 },
20+
{ taskName: "js_worker_1", codename: "whisperlink", sleepSeconds: 2 },
21+
{ taskName: "js_worker_2", codename: "shadowfetch", sleepSeconds: 3 },
22+
{ taskName: "js_worker_3", codename: "ironforge", sleepSeconds: 4 },
23+
{ taskName: "js_worker_4", codename: "deepcrawl", sleepSeconds: 5 },
24+
];
25+
26+
function envIntOrDefault(key: string, defaultVal: number): number {
27+
const s = process.env[key];
28+
if (!s) return defaultVal;
29+
const v = parseInt(s, 10);
30+
return isNaN(v) ? defaultVal : v;
31+
}
32+
33+
async function registerMetadata(
34+
client: Awaited<ReturnType<typeof OrkesClients.prototype.getClient>>,
35+
workflowClient: ReturnType<typeof OrkesClients.prototype.getWorkflowClient>,
36+
): Promise<void> {
37+
const taskDefs = SIMULATED_WORKERS.map((def) => ({
38+
name: def.taskName,
39+
description: `JS SDK harness simulated task (${def.codename}, default delay ${def.sleepSeconds}s)`,
40+
retryCount: 1,
41+
timeoutSeconds: 300,
42+
responseTimeoutSeconds: 300,
43+
totalTimeoutSeconds: 0,
44+
}));
45+
46+
await MetadataResource.registerTaskDef({
47+
client,
48+
body: taskDefs,
49+
});
50+
51+
const wf = new ConductorWorkflow(workflowClient, WORKFLOW_NAME)
52+
.version(1)
53+
.description("JS SDK harness simulated task workflow")
54+
.ownerEmail("js-sdk-harness@conductor.io");
55+
56+
for (const def of SIMULATED_WORKERS) {
57+
wf.add(simpleTask(def.codename, def.taskName, {}));
58+
}
59+
60+
await wf.register(true);
61+
62+
console.log(
63+
`Registered workflow ${WORKFLOW_NAME} with ${SIMULATED_WORKERS.length} tasks`,
64+
);
65+
}
66+
67+
async function main(): Promise<void> {
68+
const clients = await OrkesClients.from();
69+
const workflowClient = clients.getWorkflowClient();
70+
const client = clients.getClient();
71+
72+
await registerMetadata(client, workflowClient);
73+
74+
const workflowsPerSec = envIntOrDefault("HARNESS_WORKFLOWS_PER_SEC", 2);
75+
const batchSize = envIntOrDefault("HARNESS_BATCH_SIZE", 20);
76+
const pollIntervalMs = envIntOrDefault("HARNESS_POLL_INTERVAL_MS", 100);
77+
78+
const workers: ConductorWorker[] = SIMULATED_WORKERS.map((def) => {
79+
const sim = new SimulatedTaskWorker(
80+
def.taskName,
81+
def.codename,
82+
def.sleepSeconds,
83+
batchSize,
84+
pollIntervalMs,
85+
);
86+
return {
87+
taskDefName: sim.taskName,
88+
execute: sim.execute.bind(sim),
89+
concurrency: sim.batchSize,
90+
pollInterval: sim.pollInterval,
91+
};
92+
});
93+
94+
const handler = new TaskHandler({
95+
client,
96+
workers,
97+
scanForDecorated: false,
98+
});
99+
await handler.startWorkers();
100+
101+
const governor = new WorkflowGovernor(
102+
workflowClient,
103+
WORKFLOW_NAME,
104+
workflowsPerSec,
105+
);
106+
governor.start();
107+
108+
const shutdown = async () => {
109+
console.log("Shutting down...");
110+
governor.stop();
111+
await handler.stopWorkers();
112+
process.exit(0);
113+
};
114+
115+
process.on("SIGINT", shutdown);
116+
process.on("SIGTERM", shutdown);
117+
}
118+
119+
main().catch((err) => {
120+
console.error("Fatal error:", err);
121+
process.exit(1);
122+
});

‎harness/manifests/README.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# Kubernetes Manifests
2+
3+
This directory contains Kubernetes manifests for deploying the JS SDK harness worker to the certification clusters.
4+
5+
## Prerequisites
6+
7+
**Set your namespace environment variable:**
8+
```bash
9+
export NS=your-namespace-here
10+
```
11+
12+
All kubectl commands below use `-n $NS` to specify the namespace. The manifests intentionally do not include hardcoded namespaces.
13+
14+
**Note:** The harness worker images are published as public packages on GHCR and do not require authentication to pull. No image pull secrets are needed.
15+
16+
## Files
17+
18+
| File | Description |
19+
|---|---|
20+
| `deployment.yaml` | Deployment (single file, works on all clusters) |
21+
| `configmap-aws.yaml` | Conductor URL + auth key for certification-aws |
22+
| `configmap-azure.yaml` | Conductor URL + auth key for certification-az |
23+
| `configmap-gcp.yaml` | Conductor URL + auth key for certification-gcp |
24+
| `secret-conductor.yaml` | Conductor auth secret (placeholder template) |
25+
26+
## Quick Start
27+
28+
### 1. Create the Conductor Auth Secret
29+
30+
The `CONDUCTOR_AUTH_SECRET` must be created as a Kubernetes secret before deploying.
31+
32+
```bash
33+
kubectl create secret generic conductor-credentials \
34+
--from-literal=auth-secret=YOUR_AUTH_SECRET \
35+
-n $NS
36+
```
37+
38+
If the `conductor-credentials` secret already exists in the namespace (e.g. from the e2e-testrunner-worker), it can be reused as-is.
39+
40+
See `secret-conductor.yaml` for more details.
41+
42+
### 2. Apply the ConfigMap for Your Cluster
43+
44+
```bash
45+
# AWS
46+
kubectl apply -f manifests/configmap-aws.yaml -n $NS
47+
48+
# Azure
49+
kubectl apply -f manifests/configmap-azure.yaml -n $NS
50+
51+
# GCP
52+
kubectl apply -f manifests/configmap-gcp.yaml -n $NS
53+
```
54+
55+
### 3. Deploy
56+
57+
```bash
58+
kubectl apply -f manifests/deployment.yaml -n $NS
59+
```
60+
61+
### 4. Verify
62+
63+
```bash
64+
# Check pod status
65+
kubectl get pods -n $NS -l app=js-sdk-harness-worker
66+
67+
# Watch logs
68+
kubectl logs -n $NS -l app=js-sdk-harness-worker -f
69+
```
70+
71+
## Building and Pushing the Image
72+
73+
From the repository root:
74+
75+
```bash
76+
# Build the harness target and push to GHCR
77+
docker buildx build \
78+
--platform linux/amd64,linux/arm64 \
79+
--target harness \
80+
-t ghcr.io/conductor-oss/javascript-sdk/harness-worker:latest \
81+
--push .
82+
```
83+
84+
After pushing a new image with the same tag, restart the deployment to pull it:
85+
86+
```bash
87+
kubectl rollout restart deployment/js-sdk-harness-worker -n $NS
88+
kubectl rollout status deployment/js-sdk-harness-worker -n $NS
89+
```
90+
91+
## Tuning
92+
93+
The harness worker accepts these optional environment variables (set in `deployment.yaml`):
94+
95+
| Variable | Default | Description |
96+
|---|---|---|
97+
| `HARNESS_WORKFLOWS_PER_SEC` | 2 | Workflows to start per second |
98+
| `HARNESS_BATCH_SIZE` | 20 | Tasks each worker polls per batch |
99+
| `HARNESS_POLL_INTERVAL_MS` | 100 | Milliseconds between poll cycles |
100+
101+
Edit `deployment.yaml` to change these, then re-apply:
102+
103+
```bash
104+
kubectl apply -f manifests/deployment.yaml -n $NS
105+
```
106+
107+
## Troubleshooting
108+
109+
### Pod not starting
110+
111+
```bash
112+
kubectl describe pod -n $NS -l app=js-sdk-harness-worker
113+
kubectl logs -n $NS -l app=js-sdk-harness-worker --tail=100
114+
```
115+
116+
### Secret not found
117+
118+
```bash
119+
kubectl get secret conductor-credentials -n $NS
120+
```
121+
122+
## Resource Limits
123+
124+
Default resource allocation:
125+
- **Memory**: 256Mi (request) / 512Mi (limit)
126+
- **CPU**: 100m (request) / 500m (limit)
127+
128+
Adjust in `deployment.yaml` based on workload. Higher `HARNESS_WORKFLOWS_PER_SEC` values may need more CPU/memory.
129+
130+
## Service
131+
132+
The harness worker does **not** need a Service or Ingress. It connects to Conductor via outbound HTTP polling. All communication is outbound.

‎harness/manifests/configmap-aws.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: js-sdk-harness-config
6+
labels:
7+
app: js-sdk-harness-worker
8+
data:
9+
CONDUCTOR_SERVER_URL: "https://certification-aws.orkesconductor.io/api"
10+
CONDUCTOR_AUTH_KEY: "7ba9d0ec-247b-11f1-8d42-ea3efeda41b2"

‎harness/manifests/configmap-azure.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: js-sdk-harness-config
6+
labels:
7+
app: js-sdk-harness-worker
8+
data:
9+
CONDUCTOR_SERVER_URL: "https://certification-az.orkesconductor.io/api"
10+
CONDUCTOR_AUTH_KEY: "bf170d61-2797-11f1-833e-4ae04d100a03"

‎harness/manifests/configmap-gcp.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: js-sdk-harness-config
6+
labels:
7+
app: js-sdk-harness-worker
8+
data:
9+
CONDUCTOR_SERVER_URL: "https://certification-gcp.orkesconductor.com/api"
10+
CONDUCTOR_AUTH_KEY: "e6c1ac61-286b-11f1-be01-c682b5750c3a"

0 commit comments

Comments
 (0)