ScraperControl/docs/superpowers/plans/2026-03-28-freesearch-stability.md

# FreeSearch Stability & Scheduler Healthcheck Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make the `freesearch-enrichment` container stay alive when FreeSearch is down, clean up stale running jobs on restart, and fix the scheduler's perpetually-failing Docker healthcheck.

**Architecture:** Three targeted edits across two scripts and docker-compose. `enrich-with-freesearch.ts` gets a `waitForFreeSearch()` startup loop and a stale-job cleanup before job creation. `scheduler.ts` writes a heartbeat file on each hourly cron tick. `docker-compose.yml` swaps the `pgrep` healthcheck for a file-age check on that heartbeat file.

**Tech Stack:** TypeScript/tsx, Prisma, Docker Compose, node-cron, bash (healthcheck command)

---

## Files

- Modify: `scripts/enrich-with-freesearch.ts:872-880` — add `waitForFreeSearch()` function
- Modify: `scripts/enrich-with-freesearch.ts:1272-1296` — replace startup exit with wait call + stale job cleanup
- Modify: `scripts/scheduler.ts:747-758` — write heartbeat file in hourly cron
- Modify: `docker-compose.yml:275-280` — replace scheduler healthcheck

---

### Task 1: Add `waitForFreeSearch()` to the enrichment script

**Files:**
- Modify: `scripts/enrich-with-freesearch.ts`

The existing `healthCheck()` function (line 872) returns a boolean. We add `waitForFreeSearch()` directly below it — a loop that calls `healthCheck()` and sleeps with exponential backoff until it succeeds.

- [ ] **Step 1: Add `waitForFreeSearch()` after `healthCheck()`**

In `scripts/enrich-with-freesearch.ts`, find this block (around line 872):

```typescript
async function healthCheck(): Promise<boolean> {
  try {
    const resp = await axios.get(`${FREESEARCH_URL}/api/health`, { timeout: 5000 });
    return resp.status === 200;
  } catch {
    return false;
  }
}
```

Add the following function immediately after it:

```typescript
async function waitForFreeSearch(): Promise<void> {
  let backoffMs = 30_000;
  const maxBackoffMs = 300_000; // 5 minutes
  let attempt = 0;

  while (!shuttingDown) {
    attempt++;
    const healthy = await healthCheck();
    if (healthy) {
      if (attempt > 1) log('FreeSearch is back. Continuing...');
      return;
    }
    const waitSec = Math.round(backoffMs / 1000);
    logError(`FreeSearch not reachable at ${FREESEARCH_URL} (attempt ${attempt}). Retrying in ${waitSec}s...`);
    await sleep(backoffMs);
    backoffMs = Math.min(backoffMs * 2, maxBackoffMs);
  }
}
```

- [ ] **Step 2: Replace the startup health check block in `main()`**

Find this block in `main()` (around line 1272):

```typescript
  // Health check
  log('Checking FreeSearch health...');
  const healthy = await healthCheck();
  if (!healthy) {
    logError(`FreeSearch not reachable at ${FREESEARCH_URL}`);
    logError('Make sure FreeSearch is running and accessible.');
    process.exit(1);
  }
  log('FreeSearch health check: OK');
```

Replace with:

```typescript
  // Wait for FreeSearch to be reachable (indefinite retry with backoff)
  log('Waiting for FreeSearch to be reachable...');
  await waitForFreeSearch();
  if (shuttingDown) return;
  log('FreeSearch health check: OK');
```

- [ ] **Step 3: Add stale job cleanup before job creation**

Find this block in `main()` (around line 1291):

```typescript
  // Job tracking
  let jobId = await createOrResumeJob(args);
  if (!jobId) {
    jobId = await createNewJob({ countryCode, limit, continuous, dryRun, reSearch });
  }
  log(`Job ID: ${jobId}`);
```

Replace with:

```typescript
  // Job tracking — clean up any running jobs left by a previous container restart
  await prisma.backgroundJob.updateMany({
    where: { type: 'freesearch-enrichment', status: 'running' },
    data: { status: 'failed', error: 'Container restarted', completedAt: new Date() },
  });

  let jobId = await createOrResumeJob(args);
  if (!jobId) {
    jobId = await createNewJob({ countryCode, limit, continuous, dryRun, reSearch });
  }
  log(`Job ID: ${jobId}`);
```

- [ ] **Step 4: Verify the script compiles**

```bash
cd /home/albert/Documents/ScraperControl
npx tsc --noEmit
```

Expected: no errors (or only pre-existing errors unrelated to this change).

- [ ] **Step 5: Commit**

```bash
git add scripts/enrich-with-freesearch.ts
git commit -m "fix: wait for FreeSearch on startup instead of exiting; clean stale jobs"
```

---

### Task 2: Write heartbeat file in scheduler

**Files:**
- Modify: `scripts/scheduler.ts`

The scheduler already has an hourly cron that logs a heartbeat message (lines 747-758). We add a single `fs.writeFileSync` call inside it to write the timestamp to `/app/logs/scheduler.heartbeat`. The `logs/` directory is already created by `ensureLogsDir()` at startup.

- [ ] **Step 1: Add heartbeat file write inside the hourly cron**

Find this block in `scripts/scheduler.ts` (around line 747):

```typescript
  // Heartbeat every hour — logs cycle state
  cron.schedule('0 * * * *', () => {
    const currentGroup = cycleState.currentGroupIndex < PIPELINE_GROUPS.length
      ? PIPELINE_GROUPS[cycleState.currentGroupIndex].name
      : 'none';
    const jobs = runningJobs.size > 0
      ? `Running: ${[...runningJobs.keys()].join(', ')}`
      : 'No jobs running';
    const state = cycleState.waitingForCooldown
      ? 'cooldown'
      : `group ${cycleState.currentGroupIndex + 1}/${PIPELINE_GROUPS.length} (${currentGroup})`;
    log(`Heartbeat: Cycle ${cycleState.cycleNumber + 1}, ${state}. ${jobs}`);
  }, { timezone: 'UTC' });
  log('Registered cron job: heartbeat (hourly)');
```

Replace with:

```typescript
  // Heartbeat every hour — logs cycle state and writes heartbeat file for Docker healthcheck
  cron.schedule('0 * * * *', () => {
    const currentGroup = cycleState.currentGroupIndex < PIPELINE_GROUPS.length
      ? PIPELINE_GROUPS[cycleState.currentGroupIndex].name
      : 'none';
    const jobs = runningJobs.size > 0
      ? `Running: ${[...runningJobs.keys()].join(', ')}`
      : 'No jobs running';
    const state = cycleState.waitingForCooldown
      ? 'cooldown'
      : `group ${cycleState.currentGroupIndex + 1}/${PIPELINE_GROUPS.length} (${currentGroup})`;
    log(`Heartbeat: Cycle ${cycleState.cycleNumber + 1}, ${state}. ${jobs}`);
    fs.writeFileSync(path.join(LOGS_DIR, 'scheduler.heartbeat'), new Date().toISOString());
  }, { timezone: 'UTC' });
  log('Registered cron job: heartbeat (hourly)');
```

`fs` and `path` are already imported in `scheduler.ts`. `LOGS_DIR` is already defined as `'/app/logs'`.

- [ ] **Step 2: Verify the script compiles**

```bash
cd /home/albert/Documents/ScraperControl
npx tsc --noEmit
```

Expected: no errors.

- [ ] **Step 3: Commit**

```bash
git add scripts/scheduler.ts
git commit -m "fix: write heartbeat file for Docker healthcheck"
```

---

### Task 3: Fix scheduler healthcheck in docker-compose.yml

**Files:**
- Modify: `docker-compose.yml`

- [ ] **Step 1: Replace the scheduler healthcheck**

Find this block in `docker-compose.yml` (around line 275):

```yaml
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f scheduler.ts || exit 1"]
      interval: 60s
      timeout: 10s
      retries: 3
      start_period: 30s
```

Replace with:

```yaml
    healthcheck:
      test: ["CMD-SHELL", "find /app/logs/scheduler.heartbeat -mmin -120 2>/dev/null | grep -q . || exit 1"]
      interval: 90s
      timeout: 10s
      retries: 3
      start_period: 90s
```

The `find ... -mmin -120` check passes if the file exists and was modified within the last 120 minutes (2 hours). The `start_period: 90s` gives the scheduler time to reach its first hourly cron tick before Docker starts evaluating health.

- [ ] **Step 2: Commit**

```bash
git add docker-compose.yml
git commit -m "fix: replace pgrep healthcheck with heartbeat file check"
```

---

### Task 4: Deploy and verify

- [ ] **Step 1: Sync dev directory to Docker deployment**

```bash
cd /home/albert/Documents/ScraperControl
bash scripts/deploy-local.sh
```

Expected: rsync output showing the three changed files transferred to `/opt/docker/scraper-control/`.

- [ ] **Step 2: Restart the two affected containers**

```bash
docker compose -f /opt/docker/scraper-control/docker-compose.yml restart freesearch-enrichment scheduler
```

- [ ] **Step 3: Verify freesearch-enrichment is stable**

```bash
docker logs scraper-control-freesearch-enrichment-1 --tail 30 -f
```

Expected: logs showing "Waiting for FreeSearch to be reachable..." with retry messages if FreeSearch is still down, OR "FreeSearch health check: OK" and normal enrichment if FreeSearch is up. Container should NOT exit. Wait 2 minutes to confirm no restart.

- [ ] **Step 4: Confirm stale jobs were cleaned up**

```bash
docker exec scraper-control-db-1 psql -U postgres -d nearestmass \
  -c "SELECT type, status, started_at, completed_at, error FROM background_jobs WHERE type = 'freesearch-enrichment' ORDER BY started_at DESC LIMIT 5;"
```

Expected: the two previously-stuck `running` jobs from Mar 22 and Mar 26 now show `status = 'failed'` with `error = 'Container restarted'`.

- [ ] **Step 5: Verify scheduler heartbeat file is written**

Check if the file already exists from before (it won't — it's new). Wait for next hourly cron tick, or check after 60 minutes:

```bash
docker exec scraper-control-scheduler-1 cat /app/logs/scheduler.heartbeat
```

Expected: an ISO timestamp, e.g. `2026-03-28T14:00:00.000Z`

- [ ] **Step 6: Verify scheduler becomes healthy**

```bash
docker ps --format "table {{.Names}}\t{{.Status}}" | grep scheduler
```

Expected: `scraper-control-scheduler-1   Up X hours (healthy)` — but only after the first heartbeat fires AND Docker's `start_period` (90s) passes. If the next cron tick hasn't happened yet, `status` will remain `starting` or `unhealthy` until it does.

To force an immediate test without waiting for the cron:

```bash
docker exec scraper-control-scheduler-1 bash -c \
  "date -u +%Y-%m-%dT%H:%M:%S.000Z > /app/logs/scheduler.heartbeat && echo 'written'"
docker exec scraper-control-scheduler-1 \
  find /app/logs/scheduler.heartbeat -mmin -120 2>/dev/null | grep -q . && echo "PASS" || echo "FAIL"
```

Expected: `written` then `PASS`.