docs: add freesearch stability implementation plan

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs/superpowers/plans/2026-03-28-freesearch-stability.md

@@ -0,0 +1,309 @@
+# FreeSearch Stability & Scheduler Healthcheck Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make the `freesearch-enrichment` container stay alive when FreeSearch is down, clean up stale running jobs on restart, and fix the scheduler's perpetually-failing Docker healthcheck.
+
+**Architecture:** Three targeted edits across two scripts and docker-compose. `enrich-with-freesearch.ts` gets a `waitForFreeSearch()` startup loop and a stale-job cleanup before job creation. `scheduler.ts` writes a heartbeat file on each hourly cron tick. `docker-compose.yml` swaps the `pgrep` healthcheck for a file-age check on that heartbeat file.
+
+**Tech Stack:** TypeScript/tsx, Prisma, Docker Compose, node-cron, bash (healthcheck command)
+
+---
+
+## Files
+
+- Modify: `scripts/enrich-with-freesearch.ts:872-880` — add `waitForFreeSearch()` function
+- Modify: `scripts/enrich-with-freesearch.ts:1272-1296` — replace startup exit with wait call + stale job cleanup
+- Modify: `scripts/scheduler.ts:747-758` — write heartbeat file in hourly cron
+- Modify: `docker-compose.yml:275-280` — replace scheduler healthcheck
+
+---
+
+### Task 1: Add `waitForFreeSearch()` to the enrichment script
+
+**Files:**
+- Modify: `scripts/enrich-with-freesearch.ts`
+
+The existing `healthCheck()` function (line 872) returns a boolean. We add `waitForFreeSearch()` directly below it — a loop that calls `healthCheck()` and sleeps with exponential backoff until it succeeds.
+
+- [ ] **Step 1: Add `waitForFreeSearch()` after `healthCheck()`**
+
+In `scripts/enrich-with-freesearch.ts`, find this block (around line 872):
+
+```typescript
+async function healthCheck(): Promise<boolean> {
+  try {
+    const resp = await axios.get(`${FREESEARCH_URL}/api/health`, { timeout: 5000 });
+    return resp.status === 200;
+  } catch {
+    return false;
+  }
+}
+```
+
+Add the following function immediately after it:
+
+```typescript
+async function waitForFreeSearch(): Promise<void> {
+  let backoffMs = 30_000;
+  const maxBackoffMs = 300_000; // 5 minutes
+  let attempt = 0;
+
+  while (!shuttingDown) {
+    attempt++;
+    const healthy = await healthCheck();
+    if (healthy) {
+      if (attempt > 1) log('FreeSearch is back. Continuing...');
+      return;
+    }
+    const waitSec = Math.round(backoffMs / 1000);
+    logError(`FreeSearch not reachable at ${FREESEARCH_URL} (attempt ${attempt}). Retrying in ${waitSec}s...`);
+    await sleep(backoffMs);
+    backoffMs = Math.min(backoffMs * 2, maxBackoffMs);
+  }
+}
+```
+
+- [ ] **Step 2: Replace the startup health check block in `main()`**
+
+Find this block in `main()` (around line 1272):
+
+```typescript
+  // Health check
+  log('Checking FreeSearch health...');
+  const healthy = await healthCheck();
+  if (!healthy) {
+    logError(`FreeSearch not reachable at ${FREESEARCH_URL}`);
+    logError('Make sure FreeSearch is running and accessible.');
+    process.exit(1);
+  }
+  log('FreeSearch health check: OK');
+```
+
+Replace with:
+
+```typescript
+  // Wait for FreeSearch to be reachable (indefinite retry with backoff)
+  log('Waiting for FreeSearch to be reachable...');
+  await waitForFreeSearch();
+  if (shuttingDown) return;
+  log('FreeSearch health check: OK');
+```
+
+- [ ] **Step 3: Add stale job cleanup before job creation**
+
+Find this block in `main()` (around line 1291):
+
+```typescript
+  // Job tracking
+  let jobId = await createOrResumeJob(args);
+  if (!jobId) {
+    jobId = await createNewJob({ countryCode, limit, continuous, dryRun, reSearch });
+  }
+  log(`Job ID: ${jobId}`);
+```
+
+Replace with:
+
+```typescript
+  // Job tracking — clean up any running jobs left by a previous container restart
+  await prisma.backgroundJob.updateMany({
+    where: { type: 'freesearch-enrichment', status: 'running' },
+    data: { status: 'failed', error: 'Container restarted', completedAt: new Date() },
+  });
+
+  let jobId = await createOrResumeJob(args);
+  if (!jobId) {
+    jobId = await createNewJob({ countryCode, limit, continuous, dryRun, reSearch });
+  }
+  log(`Job ID: ${jobId}`);
+```
+
+- [ ] **Step 4: Verify the script compiles**
+
+```bash
+cd /home/albert/Documents/ScraperControl
+npx tsc --noEmit
+```
+
+Expected: no errors (or only pre-existing errors unrelated to this change).
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add scripts/enrich-with-freesearch.ts
+git commit -m "fix: wait for FreeSearch on startup instead of exiting; clean stale jobs"
+```
+
+---
+
+### Task 2: Write heartbeat file in scheduler
+
+**Files:**
+- Modify: `scripts/scheduler.ts`
+
+The scheduler already has an hourly cron that logs a heartbeat message (lines 747-758). We add a single `fs.writeFileSync` call inside it to write the timestamp to `/app/logs/scheduler.heartbeat`. The `logs/` directory is already created by `ensureLogsDir()` at startup.
+
+- [ ] **Step 1: Add heartbeat file write inside the hourly cron**
+
+Find this block in `scripts/scheduler.ts` (around line 747):
+
+```typescript
+  // Heartbeat every hour — logs cycle state
+  cron.schedule('0 * * * *', () => {
+    const currentGroup = cycleState.currentGroupIndex < PIPELINE_GROUPS.length
+      ? PIPELINE_GROUPS[cycleState.currentGroupIndex].name
+      : 'none';
+    const jobs = runningJobs.size > 0
+      ? `Running: ${[...runningJobs.keys()].join(', ')}`
+      : 'No jobs running';
+    const state = cycleState.waitingForCooldown
+      ? 'cooldown'
+      : `group ${cycleState.currentGroupIndex + 1}/${PIPELINE_GROUPS.length} (${currentGroup})`;
+    log(`Heartbeat: Cycle ${cycleState.cycleNumber + 1}, ${state}. ${jobs}`);
+  }, { timezone: 'UTC' });
+  log('Registered cron job: heartbeat (hourly)');
+```
+
+Replace with:
+
+```typescript
+  // Heartbeat every hour — logs cycle state and writes heartbeat file for Docker healthcheck
+  cron.schedule('0 * * * *', () => {
+    const currentGroup = cycleState.currentGroupIndex < PIPELINE_GROUPS.length
+      ? PIPELINE_GROUPS[cycleState.currentGroupIndex].name
+      : 'none';
+    const jobs = runningJobs.size > 0
+      ? `Running: ${[...runningJobs.keys()].join(', ')}`
+      : 'No jobs running';
+    const state = cycleState.waitingForCooldown
+      ? 'cooldown'
+      : `group ${cycleState.currentGroupIndex + 1}/${PIPELINE_GROUPS.length} (${currentGroup})`;
+    log(`Heartbeat: Cycle ${cycleState.cycleNumber + 1}, ${state}. ${jobs}`);
+    fs.writeFileSync(path.join(LOGS_DIR, 'scheduler.heartbeat'), new Date().toISOString());
+  }, { timezone: 'UTC' });
+  log('Registered cron job: heartbeat (hourly)');
+```
+
+`fs` and `path` are already imported in `scheduler.ts`. `LOGS_DIR` is already defined as `'/app/logs'`.
+
+- [ ] **Step 2: Verify the script compiles**
+
+```bash
+cd /home/albert/Documents/ScraperControl
+npx tsc --noEmit
+```
+
+Expected: no errors.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add scripts/scheduler.ts
+git commit -m "fix: write heartbeat file for Docker healthcheck"
+```
+
+---
+
+### Task 3: Fix scheduler healthcheck in docker-compose.yml
+
+**Files:**
+- Modify: `docker-compose.yml`
+
+- [ ] **Step 1: Replace the scheduler healthcheck**
+
+Find this block in `docker-compose.yml` (around line 275):
+
+```yaml
+    healthcheck:
+      test: ["CMD-SHELL", "pgrep -f scheduler.ts || exit 1"]
+      interval: 60s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
+```
+
+Replace with:
+
+```yaml
+    healthcheck:
+      test: ["CMD-SHELL", "find /app/logs/scheduler.heartbeat -mmin -120 2>/dev/null | grep -q . || exit 1"]
+      interval: 90s
+      timeout: 10s
+      retries: 3
+      start_period: 90s
+```
+
+The `find ... -mmin -120` check passes if the file exists and was modified within the last 120 minutes (2 hours). The `start_period: 90s` gives the scheduler time to reach its first hourly cron tick before Docker starts evaluating health.
+
+- [ ] **Step 2: Commit**
+
+```bash
+git add docker-compose.yml
+git commit -m "fix: replace pgrep healthcheck with heartbeat file check"
+```
+
+---
+
+### Task 4: Deploy and verify
+
+- [ ] **Step 1: Sync dev directory to Docker deployment**
+
+```bash
+cd /home/albert/Documents/ScraperControl
+bash scripts/deploy-local.sh
+```
+
+Expected: rsync output showing the three changed files transferred to `/opt/docker/scraper-control/`.
+
+- [ ] **Step 2: Restart the two affected containers**
+
+```bash
+docker compose -f /opt/docker/scraper-control/docker-compose.yml restart freesearch-enrichment scheduler
+```
+
+- [ ] **Step 3: Verify freesearch-enrichment is stable**
+
+```bash
+docker logs scraper-control-freesearch-enrichment-1 --tail 30 -f
+```
+
+Expected: logs showing "Waiting for FreeSearch to be reachable..." with retry messages if FreeSearch is still down, OR "FreeSearch health check: OK" and normal enrichment if FreeSearch is up. Container should NOT exit. Wait 2 minutes to confirm no restart.
+
+- [ ] **Step 4: Confirm stale jobs were cleaned up**
+
+```bash
+docker exec scraper-control-db-1 psql -U postgres -d nearestmass \
+  -c "SELECT type, status, started_at, completed_at, error FROM background_jobs WHERE type = 'freesearch-enrichment' ORDER BY started_at DESC LIMIT 5;"
+```
+
+Expected: the two previously-stuck `running` jobs from Mar 22 and Mar 26 now show `status = 'failed'` with `error = 'Container restarted'`.
+
+- [ ] **Step 5: Verify scheduler heartbeat file is written**
+
+Check if the file already exists from before (it won't — it's new). Wait for next hourly cron tick, or check after 60 minutes:
+
+```bash
+docker exec scraper-control-scheduler-1 cat /app/logs/scheduler.heartbeat
+```
+
+Expected: an ISO timestamp, e.g. `2026-03-28T14:00:00.000Z`
+
+- [ ] **Step 6: Verify scheduler becomes healthy**
+
+```bash
+docker ps --format "table {{.Names}}\t{{.Status}}" | grep scheduler
+```
+
+Expected: `scraper-control-scheduler-1   Up X hours (healthy)` — but only after the first heartbeat fires AND Docker's `start_period` (90s) passes. If the next cron tick hasn't happened yet, `status` will remain `starting` or `unhealthy` until it does.
+
+To force an immediate test without waiting for the cron:
+
+```bash
+docker exec scraper-control-scheduler-1 bash -c \
+  "date -u +%Y-%m-%dT%H:%M:%S.000Z > /app/logs/scheduler.heartbeat && echo 'written'"
+docker exec scraper-control-scheduler-1 \
+  find /app/logs/scheduler.heartbeat -mmin -120 2>/dev/null | grep -q . && echo "PASS" || echo "FAIL"
+```
+
+Expected: `written` then `PASS`.