2026-03-10 21:49:51 -04:00
# DiscoverMass.com Importer Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Import 20,284 US Catholic churches with mass/confession/adoration schedules from discovermass.com into the NearestMass database.
**Architecture:** Enumerate 11 WordPress sitemaps → fetch each church page at 10s intervals (respecting Crawl-delay) → parse server-rendered HTML for name/address/coordinates/schedules → match against existing US churches via church-matcher → upsert with full schedule data.
**Tech Stack:** TypeScript/tsx, Prisma 7 + PrismaPg adapter, pg Pool, Node.js `fetch` , regex HTML parsing (no DOM library needed — HTML is server-rendered and predictable).
---
## Chunk 1: Schema + church-matcher
### Task 1: Add discovermassId to schema
**Files:**
- Modify: `prisma/schema.prisma`
The schema lives in this repo but migrations run in BethelGuide. After editing schema.prisma here, run `npx prisma generate` to regenerate the Prisma client. Do NOT run `prisma migrate` .
- [ ] **Step 1: Find the right place in schema.prisma **
Open `prisma/schema.prisma` . Find the block of source ID fields — they look like:
```prisma
gottesdienstzeitenId String? @unique @map ("gottesdienstzeiten_id")
```
This is inside the `model Church { ... }` block, after `kerknetId` and before `claimed` .
- [ ] **Step 2: Add discovermassId field **
After `gottesdienstzeitenId` :
```prisma
discovermassId String? @unique @map ("discovermass_id")
```
Also find the `@@index` block near the bottom of the Church model (it groups all the index definitions). Add:
```prisma
@@index ([discovermassId])
```
- [ ] **Step 3: Regenerate Prisma client **
```bash
cd /home/albert/Documents/ScraperControl
npx prisma generate
```
Expected output: `✔ Generated Prisma Client` (no errors). This does NOT touch the database — it only updates the TypeScript client.
- [ ] **Step 4: Apply migration to database **
The schema source of truth is BethelGuide. Run the migration there, then sync back. Since we're on the same dev server:
```bash
# Check if discovermass_id column already exists (it shouldn't yet)
psql postgresql://postgres:postgres@192 .168.0.145:5434/nearestmass -c "\d churches" | grep discovermass
```
If the column doesn't exist, apply it directly:
```bash
psql postgresql://postgres:postgres@192 .168.0.145:5434/nearestmass -c "
ALTER TABLE churches ADD COLUMN IF NOT EXISTS discovermass_id VARCHAR UNIQUE;
CREATE INDEX IF NOT EXISTS churches_discovermass_id_idx ON churches(discovermass_id);
"
```
Expected output: `ALTER TABLE` and `CREATE INDEX`
- [ ] **Step 5: Verify column exists **
```bash
psql postgresql://postgres:postgres@192 .168.0.145:5434/nearestmass -c "\d churches" | grep discovermass
```
Expected output: `discovermass_id | character varying | ...`
- [ ] **Step 6: Commit **
```bash
cd /home/albert/Documents/ScraperControl
git add prisma/schema.prisma
git commit -m "feat: add discovermassId field to Church schema"
```
---
### Task 2: Update church-matcher
**Files:**
- Modify: `src/lib/church-matcher.ts`
The `ExistingChurch` interface (line ~11) lists all source IDs. The `ChurchCandidate` type (line ~122) lists optional source IDs for the candidate. The `findDuplicateChurch` function has sequential passes checking each ID before falling back to proximity+name.
- [ ] **Step 1: Add discovermassId to ExistingChurch interface **
Find the `export interface ExistingChurch {` block. After the `gottesdienstzeitenId` line, add:
```typescript
discovermassId: string | null;
```
- [ ] **Step 2: Add discovermassId to ChurchCandidate type **
Find `export type ChurchCandidate = {` . After `gottesdienstzeitenId?: string;` , add:
```typescript
discovermassId?: string;
```
- [ ] **Step 3: Add discovermassId matching pass in findDuplicateChurch **
Find the `findDuplicateChurch` function. It has a series of passes like:
```typescript
if (candidate.gottesdienstzeitenId) {
const match = existingChurches.find(c => c.gottesdienstzeitenId === candidate.gottesdienstzeitenId);
if (match) return match;
}
// Proximity + name similarity
```
Add a new pass BEFORE the proximity+name pass (after gottesdienstzeitenId):
```typescript
if (candidate.discovermassId) {
const match = existingChurches.find(c => c.discovermassId === candidate.discovermassId);
if (match) return match;
}
```
- [ ] **Step 4: Update all callers that construct ExistingChurch objects **
Search for places that build ExistingChurch objects (the in-memory push after creating a new church). Each importer has a block like:
```typescript
existingChurches.push({
id: newChurch.id,
...
gottesdienstzeitenId: null,
...
});
```
Run:
```bash
grep -rn "gottesdienstzeitenId: null" scripts/
```
For each file found: add `discovermassId: null,` after `gottesdienstzeitenId: null,` . These are the in-memory dedup arrays — they need the new field or TypeScript will complain.
Also update the `loadExistingChurches` select queries if any importer has one (check with `grep -rn "gottesdienstzeitenId: true" scripts/` ).
- [ ] **Step 5: Verify TypeScript compiles **
```bash
cd /home/albert/Documents/ScraperControl
npx tsc --noEmit
```
Expected: no errors. Fix any type errors (they'll be missing `discovermassId` fields).
- [ ] **Step 6: Commit **
```bash
# Stage church-matcher AND all importer scripts that were updated in Step 4
git add src/lib/church-matcher.ts
git add scripts/
git commit -m "feat: add discovermassId to church-matcher ExistingChurch and ChurchCandidate"
```
---
## Chunk 2: import-discovermass.ts — utilities and parsing
### Task 3: Create file skeleton + utilities
**Files:**
- Create: `scripts/import-discovermass.ts`
- [ ] **Step 1: Create the file with header, imports, constants, types **
Create `scripts/import-discovermass.ts` with this content:
```typescript
#!/usr/bin/env tsx
/**
* Import Catholic churches and mass schedules from discovermass.com (USA)
*
* discovermass.com is a US Catholic church directory with 20,284 churches.
* Data includes name, address, phone, website, coordinates, mass times,
* confessions, and adoration schedules.
*
* robots.txt specifies Crawl-delay: 10 — this importer follows that rule.
*
* Usage:
* npx tsx scripts/import-discovermass.ts --all
* npx tsx scripts/import-discovermass.ts --all --dry-run
* npx tsx scripts/import-discovermass.ts --all --resume-from 5000
* npx tsx scripts/import-discovermass.ts --all --job-id {uuid}
*/
import dotenv from 'dotenv';
import path from 'path';
dotenv.config({ path: path.resolve(process.cwd(), '.env.local') });
dotenv.config({ path: path.resolve(process.cwd(), '.env') });
import { Pool } from 'pg';
import { PrismaPg } from '@prisma/adapter -pg';
import { PrismaClient } from '@prisma/client ';
const dbUrl = process.env.DATABASE_URL || 'postgresql://postgres:postgres@localhost:5432/nearestmass ';
console.log(`Connecting to database: ${dbUrl.replace(/:[^:@]+@/, ':***@')}` );
const pool = new Pool({
connectionString: dbUrl,
ssl: dbUrl.includes('neon') ? { rejectUnauthorized: false } : undefined,
});
const adapter = new PrismaPg(pool);
const prisma = new PrismaClient({ adapter });
import { findDuplicateChurch } from '../src/lib/church-matcher';
import type { ExistingChurch } from '../src/lib/church-matcher';
// ─── Constants ───────────────────────────────────────────────────────────────
const SITE_BASE = 'https://discovermass.com';
const SITEMAP_COUNT = 11;
const USER_AGENT = 'NearestMass-Importer/1.0 (parish data aggregator; contact: privacy@nearestmass .com)';
const REQUEST_DELAY_MS = 10_000; // Crawl-delay: 10 from robots.txt
// ─── Types ───────────────────────────────────────────────────────────────────
interface ParsedChurch {
name: string;
address: string | null;
city: string | null;
state: string | null;
zip: string | null;
phone: string | null;
website: string | null;
lat: number;
lng: number;
}
interface ParsedMass {
dayOfWeek: number; // 0=Sun, 1=Mon, ..., 6=Sat
time: string; // HH:MM 24-hour
language: string;
notes?: string;
}
interface ParsedConf {
dayOfWeek: number;
startTime: string; // HH:MM 24-hour
endTime: string; // HH:MM 24-hour
notes?: string;
}
interface ParsedAdoration {
dayOfWeek: number;
startTime: string; // HH:MM 24-hour
endTime: string; // HH:MM 24-hour
notes?: string;
}
interface ImportStats {
total: number;
created: number;
updated: number;
skipped: number;
errors: number;
massSchedulesCreated: number;
confessionSchedulesCreated: number;
adorationSchedulesCreated: number;
}
interface CLIArgs {
all: boolean;
dryRun: boolean;
resumeFrom?: number;
jobId?: string;
}
```
- [ ] **Step 2: Add day mappings and time utilities **
Append to the file:
```typescript
// ─── Day Mappings ─────────────────────────────────────────────────────────────
// Full day names used in mass schedule <li> labels
const FULL_DAY_NAMES: Record<string, number> = {
Sunday: 0, Monday: 1, Tuesday: 2, Wednesday: 3,
Thursday: 4, Friday: 5, Saturday: 6,
};
// Abbreviated day prefixes used in confession/adoration serviceTime text
const ABBREV_DAY_NAMES: Record<string, number[]> = {
Sun: [0], Mon: [1], Tue: [2], Wed: [3],
Thr: [4], Thu: [4], Fri: [5], Sat: [6],
Weekdays: [1, 2, 3, 4, 5],
Daily: [0, 1, 2, 3, 4, 5, 6],
};
// ─── Time Utilities ───────────────────────────────────────────────────────────
/**
* Convert "5:00pm", "11:00am", "12:00pm", "12:00am" to "HH:MM" 24-hour format.
* Returns the original string unchanged if it doesn't match expected format.
*/
function convertTo24h(timeStr: string): string {
const cleaned = timeStr.trim().toLowerCase();
const m = cleaned.match(/^(\d{1,2}):(\d{2})(am|pm)$/);
if (!m) return cleaned;
let hours = parseInt(m[1], 10);
const mins = m[2];
const meridiem = m[3];
if (meridiem === 'pm' && hours !== 12) hours += 12;
if (meridiem === 'am' && hours === 12) hours = 0;
return `${String(hours).padStart(2, '0')}:${mins}` ;
}
/**
* Parse "8:30am-9:00am" → ["08:30", "09:00"].
* Handles the case where both sides need to infer AM/PM from context.
* E.g. "8:30am-9:00am" → both explicit. "9:00am-6:00pm" → both explicit.
*/
function parseTimeRange(rangeStr: string): [string, string] {
// Split on '-' but careful: times may contain only one '-' between start and end
// Pattern: "8:30am-9:00am" or "3:30pm-4:30pm"
const hyphenIdx = rangeStr.indexOf('-', rangeStr.indexOf(':') + 1);
if (hyphenIdx === -1) {
const t = convertTo24h(rangeStr.trim());
return [t, t];
}
const start = convertTo24h(rangeStr.slice(0, hyphenIdx).trim());
const end = convertTo24h(rangeStr.slice(hyphenIdx + 1).trim());
return [start, end];
}
/**
* Expand abbreviated day prefix to array of dayOfWeek integers.
* Returns empty array if prefix is not recognized.
*/
function expandDayAbbrev(prefix: string): number[] {
return ABBREV_DAY_NAMES[prefix] ?? [];
}
// ─── Address Parsing ──────────────────────────────────────────────────────────
/**
* Parse "14085 Peyton Drive, Chino Hills, CA 91709" into components.
* Returns partial result on malformed input.
*/
function parseAddress(raw: string): { address: string | null; city: string | null; state: string | null; zip: string | null } {
const parts = raw.split(', ');
if (parts.length < 3) return { address: raw, city: null, state: null, zip: null };
const last = parts[parts.length - 1].trim();
const stateZipMatch = last.match(/^([A-Z]{2})\s+(\d{5}(?:-\d{4})?)$/);
if (!stateZipMatch) return { address: raw, city: null, state: null, zip: null };
return {
address: parts.slice(0, parts.length - 2).join(', ').trim(),
city: parts[parts.length - 2].trim(),
state: stateZipMatch[1],
zip: stateZipMatch[2],
};
}
```
- [ ] **Step 3: Verify utilities compile **
```bash
cd /home/albert/Documents/ScraperControl
npx tsc --noEmit
```
Expected: no errors related to import-discovermass.ts. Other files may have pre-existing errors — focus only on this file's errors.
---
### Task 4: Add HTML parsing functions
**Files:**
- Modify: `scripts/import-discovermass.ts`
The HTML is server-rendered. The page structure has:
- `<meta property="og:title">` for church name
- US address embedded as text in a known pattern
- `<div id="sidebar-info">` for phone/website/coordinates
- Two `<ul>` blocks: one containing `<h5>Mass Times</h5>` , another containing `<h5>Other Services</h5>`
- [ ] **Step 1: Add parseChurch function **
Append to the file:
```typescript
// ─── HTML Parsing ─────────────────────────────────────────────────────────────
/**
* Parse church metadata from page HTML.
* Returns null if the page doesn't look like a valid church listing.
*/
function parseChurch(html: string): ParsedChurch | null {
// Name from OpenGraph meta tag
const nameMatch = html.match(/<meta property="og:title" content="([^"]+)"/);
if (!nameMatch) return null;
const name = nameMatch[1].trim();
if (!name || name === 'Discover Mass') return null;
// Address: match US address pattern (number + street, city, STATE ZIP)
let address: string | null = null;
let city: string | null = null;
let state: string | null = null;
let zip: string | null = null;
const addrMatch = html.match(/(\d+[^<\n,]+),\s*([^<,\n]+),\s*([A-Z]{2})\s+(\d{5}(?:-\d{4})?)/);
if (addrMatch) {
const raw = `${addrMatch[1].trim()}, ${addrMatch[2].trim()}, ${addrMatch[3]} ${addrMatch[4]}` ;
const parsed = parseAddress(raw);
address = parsed.address;
city = parsed.city;
state = parsed.state;
zip = parsed.zip;
}
// Phone from sidebar
const phoneMatch = html.match(/<span class='side-phone attribute'>([^<]+)<\/span>/);
const phone = phoneMatch ? phoneMatch[1].trim() : null;
// Website from sidebar
const websiteMatch = html.match(/<span class='side-website attribute'><a href='([^']+)'/);
const website = websiteMatch ? websiteMatch[1].trim() : null;
// Coordinates from Google Maps daddr parameter
let lat = 0;
let lng = 0;
const coordMatch = html.match(/daddr=([-\d.]+),([-\d.]+)/);
if (coordMatch) {
2026-03-10 22:34:51 -04:00
const rawLat = parseFloat(coordMatch[1]);
const rawLng = parseFloat(coordMatch[2]);
// Validate: reject NaN, Infinity, and out-of-range values; fall back to 0 sentinel
if (isFinite(rawLat) && isFinite(rawLng) && Math.abs(rawLat) <= 90 && Math.abs(rawLng) <= 180) {
lat = rawLat;
lng = rawLng;
}
2026-03-10 21:49:51 -04:00
}
return { name, address, city, state, zip, phone, website, lat, lng };
}
```
- [ ] **Step 2: Add parseMassTimes function **
Append to the file:
```typescript
/**
* Parse the mass schedule from the "Mass Times" <ul> block.
*
* HTML structure:
* <ul><li><h5>Mass Times</h5></li>
* <li class=""><span class="label">Saturday</span>
* <span class='serviceTime'><span class='time'>5:00pm</span></span>,
* <span class='serviceTime'><span class='time'>7:00pm</span> <span class='language'>(Vietnamese)</span></span>
* </li>
* </ul>
*/
function parseMassTimes(html: string): ParsedMass[] {
2026-03-10 22:34:51 -04:00
// Cap HTML to first 100KB to prevent slow regex backtracking on malformed pages
const safeHtml = html.length > 100_000 ? html.slice(0, 100_000) : html;
const massUlMatch = safeHtml.match(/<ul>\s*<li>\s*<h5>Mass Times<\/h5>[\s\S]*?<\/ul>/);
2026-03-10 21:49:51 -04:00
if (!massUlMatch) return [];
const massUl = massUlMatch[0];
const results: ParsedMass[] = [];
// Each day is in a <li ...> (the first li has the h5 header, skip it).
// Use regex split to handle any class value on the li, not just empty class.
const liParts = massUl.split(/<li[^>]*>/);
for (let i = 1; i < liParts.length; i++) {
const li = liParts[i];
const labelMatch = li.match(/<span class="label">([^<]+)<\/span>/);
if (!labelMatch) continue;
const dayLabel = labelMatch[1].trim();
const dayOfWeek = FULL_DAY_NAMES[dayLabel];
if (dayOfWeek === undefined) continue;
// Each time entry is in a <span class='serviceTime'>
const serviceTimeParts = li.split("<span class='serviceTime'>");
for (let j = 1; j < serviceTimeParts.length; j++) {
const st = serviceTimeParts[j];
const timeMatch = st.match(/<span class='time'>([^<]+)<\/span>/);
if (!timeMatch) continue;
const time = convertTo24h(timeMatch[1].trim());
const langMatch = st.match(/<span class='language'>\(([^)]+)\)<\/span>/);
const language = langMatch ? langMatch[1].trim() : 'English';
const commentMatch = st.match(/<span class='comment'>([^<]+)<\/span>/);
const notes = commentMatch ? commentMatch[1].trim() : undefined;
results.push({ dayOfWeek, time, language, notes });
}
}
return results;
}
```
- [ ] **Step 3: Add parseOtherServices function **
Append to the file:
```typescript
/**
* Parse confessions and adoration from the "Other Services" <ul> block.
*
* HTML structure:
* <ul><li><h5>Other Services</h5></li>
* <li class="Confessions"><span class="label">Confessions</span>
* <span class='serviceTime'>Tue: <span class='time'>8:30am-9:00am</span></span>
* <span class='serviceTime'>Sat: <span class='time'>3:30pm-4:30pm</span> - <span class='comment'>In the Chapel</span></span>
* </li>
* <li class="Adoration"><span class="label">Adoration</span>
* <span class='serviceTime'>Weekdays: <span class='time'>9:00am-6:00pm</span> - <span class='comment'>In the Chapel</span></span>
* </li>
* </ul>
*/
function parseOtherServices(html: string): { confessions: ParsedConf[]; adorations: ParsedAdoration[] } {
2026-03-10 22:34:51 -04:00
// Cap HTML to first 100KB to prevent slow regex backtracking on malformed pages
const safeHtml = html.length > 100_000 ? html.slice(0, 100_000) : html;
const otherUlMatch = safeHtml.match(/<ul>\s*<li>\s*<h5>Other Services<\/h5>[\s\S]*?<\/ul>/);
2026-03-10 21:49:51 -04:00
if (!otherUlMatch) return { confessions: [], adorations: [] };
const otherUl = otherUlMatch[0];
function parseServiceItems(liHtml: string): Array<{ dayOfWeek: number; startTime: string; endTime: string; notes?: string }> {
const items: Array<{ dayOfWeek: number; startTime: string; endTime: string; notes?: string }> = [];
const stParts = liHtml.split("<span class='serviceTime'>");
for (let i = 1; i < stParts.length; i++) {
const st = stParts[i];
// Text before <span class='time'> contains the day abbreviation and colon
const dayTimeMatch = st.match(/^([A-Za-z]+):\s*<span class='time'>([^<]+)<\/span>/);
if (!dayTimeMatch) continue;
const days = expandDayAbbrev(dayTimeMatch[1].trim());
if (days.length === 0) continue;
const [startTime, endTime] = parseTimeRange(dayTimeMatch[2]);
const commentMatch = st.match(/<span class='comment'>([^<]+)<\/span>/);
const notes = commentMatch ? commentMatch[1].trim() : undefined;
for (const dayOfWeek of days) {
items.push({ dayOfWeek, startTime, endTime, notes });
}
}
return items;
}
const confessions: ParsedConf[] = [];
const adorations: ParsedAdoration[] = [];
const confMatch = otherUl.match(/<li class="Confessions">[\s\S]*?<\/li>/);
if (confMatch) confessions.push(...parseServiceItems(confMatch[0]));
const adorMatch = otherUl.match(/<li class="Adoration">[\s\S]*?<\/li>/);
if (adorMatch) adorations.push(...parseServiceItems(adorMatch[0]));
return { confessions, adorations };
}
```
- [ ] **Step 4: Smoke-test parsing on a real page **
Create a quick test at the end of the file temporarily:
```typescript
// TEMP: smoke test — remove before committing
if (process.argv[2] === '--test-parse') {
const testUrl = 'https://discovermass.com/church/st-paul-the-apostle-chino-hills-ca/';
const html = await (await fetch(testUrl, { headers: { 'User-Agent': USER_AGENT } })).text();
const church = parseChurch(html);
const masses = parseMassTimes(html);
const { confessions, adorations } = parseOtherServices(html);
console.log('Church:', church);
console.log('Masses:', masses.length, masses.slice(0, 3));
console.log('Confessions:', confessions.length, confessions);
console.log('Adorations:', adorations.length, adorations);
process.exit(0);
}
```
Run it:
```bash
cd /home/albert/Documents/ScraperControl
npx tsx scripts/import-discovermass.ts --test-parse
```
Expected output:
```
Church: { name: 'St. Paul the Apostle', address: '14085 Peyton Drive', city: 'Chino Hills', state: 'CA', zip: '91709', phone: '(909) 465-5503', website: 'http://www.sptacc.org', lat: 33.996887, lng: -117.732407 }
Masses: 14 [ { dayOfWeek: 6, time: '17:00', language: 'English', notes: undefined }, ... ]
Confessions: 4 [...]
Adorations: 6 [...]
```
If the counts don't match, debug the regex patterns against the raw HTML:
```bash
curl -s https://discovermass.com/church/st-paul-the-apostle-chino-hills-ca/ > /tmp/dm-test.html
# Then inspect the relevant sections manually
```
- [ ] **Step 5: Remove the temp test block and commit **
Remove the `if (process.argv[2] === '--test-parse')` block from the file.
```bash
git add scripts/import-discovermass.ts
git commit -m "feat: add discovermass parsing utilities (church, mass, confession, adoration)"
```
---
## Chunk 3: import-discovermass.ts — main import loop
### Task 5: Add HTTP helpers + sitemap enumeration
**Files:**
- Modify: `scripts/import-discovermass.ts`
- [ ] **Step 1: Add HTTP helpers and loadExistingChurches **
Append to the file:
```typescript
// ─── HTTP Helpers ─────────────────────────────────────────────────────────────
async function fetchHtml(url: string): Promise<string> {
const res = await fetch(url, {
headers: { 'User-Agent': USER_AGENT },
});
if (!res.ok) throw new Error(`HTTP ${res.status} for ${url}` );
return res.text();
}
function sleep(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}
// ─── Sitemap Enumeration ──────────────────────────────────────────────────────
/**
* Fetch all 11 WordPress item sitemaps and return every church URL.
* No rate limiting needed — only 11 sitemap requests.
*/
async function getAllChurchUrls(): Promise<string[]> {
const urls: string[] = [];
for (let i = 1; i <= SITEMAP_COUNT; i++) {
const sitemapUrl = `${SITE_BASE}/wp-sitemap-posts-item-${i}.xml` ;
console.log(`Fetching sitemap ${i}/${SITEMAP_COUNT}...` );
const xml = await fetchHtml(sitemapUrl);
const matches = xml.matchAll(/<loc>(https:\/\/discovermass\.com\/church\/[^<]+)<\/loc>/g);
for (const match of matches) {
urls.push(match[1]);
}
}
console.log(`Total church URLs: ${urls.length}` );
return urls;
}
// ─── DB Helpers ───────────────────────────────────────────────────────────────
/**
* Load all US churches from DB into memory for dedup matching.
* Only loads US churches to keep the array manageable.
*/
async function loadExistingChurches(): Promise<ExistingChurch[]> {
console.log('Loading existing US churches from DB...');
const churches = await prisma.church.findMany({
where: { country: 'US' },
select: {
id: true, name: true, latitude: true, longitude: true,
osmId: true, baiduId: true, masstimesId: true,
orarimesseId: true, massSchedulesPhId: true, philmassId: true,
horariosMisasId: true, mszeInfoId: true, weekdayMassesId: true,
messesInfoId: true, bohosluzbyId: true, miserendId: true,
kerknetId: true, gottesdienstzeitenId: true, discovermassId: true,
source: true, website: true, phone: true, address: true, country: true,
},
});
console.log(`Loaded ${churches.length} existing US churches` );
return churches as ExistingChurch[];
}
```
---
### Task 6: Add processChurch + main() + CLI parsing
**Files:**
- Modify: `scripts/import-discovermass.ts`
- [ ] **Step 1: Add processChurch function **
Append to the file:
```typescript
// ─── Church Processing ────────────────────────────────────────────────────────
async function processChurch(
url: string,
existingChurches: ExistingChurch[],
args: CLIArgs,
stats: ImportStats,
): Promise<void> {
const slug = url.replace('https://discovermass.com/church/', '').replace(/\/$/, '');
stats.total++;
try {
const html = await fetchHtml(url);
const parsed = parseChurch(html);
if (!parsed) {
console.log(` [skip] Could not parse: ${slug}` );
stats.skipped++;
return;
}
const masses = parseMassTimes(html);
const { confessions, adorations } = parseOtherServices(html);
if (args.dryRun) {
console.log(` [dry-run] ${parsed.name} — ${masses.length} masses, ${confessions.length} confessions, ${adorations.length} adorations` );
return;
}
const candidate = {
name: parsed.name,
lat: parsed.lat,
lng: parsed.lng,
discovermassId: slug,
};
const duplicate = findDuplicateChurch(candidate, existingChurches);
if (duplicate) {
// Update existing church — only fill blank fields, always replace schedules
const updateData: Record<string, unknown> = { discovermassId: slug };
if (!duplicate.phone && parsed.phone) updateData.phone = parsed.phone;
if (!duplicate.website && parsed.website) {
updateData.website = parsed.website;
updateData.hasWebsite = true;
}
if (parsed.lat !== 0 && duplicate.latitude === 0) {
updateData.latitude = parsed.lat;
updateData.longitude = parsed.lng;
}
try {
await prisma.$transaction(async (tx) => {
await tx.church.update({ where: { id: duplicate.id }, data: updateData });
if (masses.length > 0) {
await tx.massSchedule.deleteMany({ where: { churchId: duplicate.id } });
await tx.massSchedule.createMany({
data: masses.map(m => ({
churchId: duplicate.id,
dayOfWeek: m.dayOfWeek,
time: m.time,
language: m.language,
notes: m.notes ?? null,
})),
});
}
if (confessions.length > 0) {
await tx.confessionSchedule.deleteMany({ where: { churchId: duplicate.id } });
await tx.confessionSchedule.createMany({
data: confessions.map(c => ({
churchId: duplicate.id,
dayOfWeek: c.dayOfWeek,
startTime: c.startTime,
endTime: c.endTime,
notes: c.notes ?? null,
})),
});
}
if (adorations.length > 0) {
await tx.adorationSchedule.deleteMany({ where: { churchId: duplicate.id } });
await tx.adorationSchedule.createMany({
data: adorations.map(a => ({
churchId: duplicate.id,
dayOfWeek: a.dayOfWeek,
startTime: a.startTime,
endTime: a.endTime,
notes: a.notes ?? null,
})),
});
}
await tx.church.update({
where: { id: duplicate.id },
data: { lastScrapedAt: new Date() },
});
});
// Update in-memory entry for within-run dedup
duplicate.discovermassId = slug;
stats.updated++;
} catch (err) {
if (err instanceof Error && err.message.includes('Unique constraint')) {
stats.skipped++;
return;
}
throw err;
}
} else {
// Create new church
try {
const church = await prisma.church.create({
data: {
name: parsed.name,
address: parsed.address,
city: parsed.city,
state: parsed.state,
zip: parsed.zip,
country: 'US',
phone: parsed.phone,
website: parsed.website,
hasWebsite: !!parsed.website,
latitude: parsed.lat,
longitude: parsed.lng,
discovermassId: slug,
source: 'discovermass',
},
});
// Add to in-memory array for within-run dedup
existingChurches.push({
id: church.id,
name: parsed.name,
latitude: parsed.lat,
longitude: parsed.lng,
osmId: null,
baiduId: null,
masstimesId: null,
orarimesseId: null,
massSchedulesPhId: null,
philmassId: null,
horariosMisasId: null,
mszeInfoId: null,
weekdayMassesId: null,
messesInfoId: null,
bohosluzbyId: null,
miserendId: null,
kerknetId: null,
gottesdienstzeitenId: null,
discovermassId: slug,
source: 'discovermass',
website: parsed.website,
phone: parsed.phone,
address: parsed.address,
country: 'US',
});
if (masses.length > 0) {
await prisma.massSchedule.createMany({
data: masses.map(m => ({
churchId: church.id,
dayOfWeek: m.dayOfWeek,
time: m.time,
language: m.language,
notes: m.notes ?? null,
})),
});
}
if (confessions.length > 0) {
await prisma.confessionSchedule.createMany({
data: confessions.map(c => ({
churchId: church.id,
dayOfWeek: c.dayOfWeek,
startTime: c.startTime,
endTime: c.endTime,
notes: c.notes ?? null,
})),
});
}
if (adorations.length > 0) {
await prisma.adorationSchedule.createMany({
data: adorations.map(a => ({
churchId: church.id,
dayOfWeek: a.dayOfWeek,
startTime: a.startTime,
endTime: a.endTime,
notes: a.notes ?? null,
})),
});
}
await prisma.church.update({
where: { id: church.id },
data: { lastScrapedAt: new Date() },
});
stats.created++;
} catch (err) {
if (err instanceof Error && err.message.includes('Unique constraint')) {
stats.skipped++;
return;
}
throw err;
}
}
stats.massSchedulesCreated += masses.length;
stats.confessionSchedulesCreated += confessions.length;
stats.adorationSchedulesCreated += adorations.length;
console.log(
` [${duplicate ? 'update' : 'create'}] ${parsed.name} — ` +
`${masses.length}M ${confessions.length}C ${adorations.length}A — ` +
`${stats.total} total (${stats.created} new, ${stats.updated} upd, ${stats.errors} err)`
);
} catch (err) {
stats.errors++;
console.error(` [error] ${slug}: ${err instanceof Error ? err.message : err}` );
}
}
```
- [ ] **Step 2: Add parseCLIArgs + main() **
Append to the file:
```typescript
// ─── CLI Parsing ──────────────────────────────────────────────────────────────
function parseCLIArgs(): CLIArgs {
const args = process.argv.slice(2);
const result: CLIArgs = { all: false, dryRun: false };
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case '--all': result.all = true; break;
case '--dry-run': result.dryRun = true; break;
case '--resume-from': result.resumeFrom = parseInt(args[++i], 10); break;
case '--job-id': result.jobId = args[++i]; break;
}
}
return result;
}
// ─── Main ─────────────────────────────────────────────────────────────────────
async function main() {
const args = parseCLIArgs();
if (!args.all) {
console.error('Usage: npx tsx scripts/import-discovermass.ts --all [--dry-run] [--resume-from N] [--job-id UUID]');
process.exit(1);
}
// Update job status to 'running' if job-id provided
if (args.jobId) {
try {
await prisma.backgroundJob.update({
where: { id: args.jobId },
data: { status: 'running', startedAt: new Date() },
});
} catch { /* Job might not exist yet */ }
}
const stats: ImportStats = {
total: 0, created: 0, updated: 0, skipped: 0, errors: 0,
massSchedulesCreated: 0, confessionSchedulesCreated: 0, adorationSchedulesCreated: 0,
};
try {
// Step 1: Enumerate all church URLs from sitemaps
const urls = await getAllChurchUrls();
// Step 2: Load existing US churches for dedup
const existingChurches = await loadExistingChurches();
// Step 3: Apply resume-from offset
const startIdx = args.resumeFrom ?? 0;
const churchUrls = urls.slice(startIdx);
console.log(`\nProcessing ${churchUrls.length} churches (starting from index ${startIdx})...\n` );
// Step 4: Process each church with 10s delay between requests
for (let i = 0; i < churchUrls.length; i++) {
const url = churchUrls[i];
const overallIdx = startIdx + i;
console.log(`[${overallIdx + 1}/${urls.length}] ${url}` );
await processChurch(url, existingChurches, args, stats);
// Rate limit: 10s delay between church pages (robots.txt Crawl-delay: 10)
if (i < churchUrls.length - 1) {
await sleep(REQUEST_DELAY_MS);
}
}
} finally {
// Always print stats and update job status
console.log('\n─── Import Complete ───────────────────────────────────────');
console.log(`Total processed: ${stats.total}` );
console.log(`Created: ${stats.created}` );
console.log(`Updated: ${stats.updated}` );
console.log(`Skipped: ${stats.skipped}` );
console.log(`Errors: ${stats.errors}` );
console.log(`Mass schedules: ${stats.massSchedulesCreated}` );
console.log(`Confession sched: ${stats.confessionSchedulesCreated}` );
console.log(`Adoration sched: ${stats.adorationSchedulesCreated}` );
if (args.jobId) {
const status = stats.errors > stats.total * 0.1 ? 'failed' : 'completed';
try {
await prisma.backgroundJob.update({
where: { id: args.jobId },
data: {
status,
completedAt: new Date(),
processed: stats.total,
succeeded: stats.created + stats.updated,
failed: stats.errors,
itemsFound: stats.massSchedulesCreated,
},
});
} catch { /* Ignore */ }
}
await prisma.$disconnect();
await pool.end();
}
}
main().catch((err) => {
console.error('Fatal error:', err);
process.exit(1);
});
```
- [ ] **Step 3: Run dry-run on first 3 churches **
```bash
cd /home/albert/Documents/ScraperControl
# Get a few URLs from the first sitemap to test with
curl -s https://discovermass.com/wp-sitemap-posts-item-1.xml | grep -o '<loc>[^<]*</loc>' | head -3
```
Then test dry-run:
```bash
npx tsx scripts/import-discovermass.ts --all --dry-run --resume-from 0
```
Wait ~30 seconds (3 churches ×
```
Connecting to database: postgresql://...
Fetching sitemap 1/11...
...
Total church URLs: 20284
Loading existing US churches from DB...
Loaded XXXX existing US churches
Processing 20284 churches (starting from index 0)...
[1/20284] https://discovermass.com/church/some-church/
[dry-run] Some Church — 8 masses, 2 confessions, 3 adorations
[2/20284] ...
```
Stop with Ctrl+C after a few churches.
- [ ] **Step 4: Verify TypeScript compiles **
```bash
npx tsc --noEmit
```
Fix any type errors before committing.
- [ ] **Step 5: Commit **
```bash
git add scripts/import-discovermass.ts
git commit -m "feat: add import-discovermass.ts — USA church importer with 10s crawl delay"
```
---
## Chunk 4: Integration + Docker deployment
### Task 7: package.json + scheduler integration
**Files:**
- Modify: `package.json`
- Modify: `scripts/scheduler.ts`
- [ ] **Step 1: Add import:discovermass to package.json **
Open `package.json` . Find the `"scripts"` section. After `"import:masstimes-api"` (or whichever is last in the import group), add:
```json
"import:discovermass": "tsx scripts/import-discovermass.ts",
```
- [ ] **Step 2: Add discovermass-import case to getJobCommand in scheduler.ts **
Open `scripts/scheduler.ts` . Find the `getJobCommand` function. After the `masstimes-api-import` case block, add:
```typescript
case 'discovermass-import': {
const args = ['tsx', 'scripts/import-discovermass.ts', '--all'];
if (config?.resumeFrom) args.push('--resume-from', String(config.resumeFrom));
return { command: 'npx', args };
// Note: --job-id is appended by startJobProcess() in the scheduler, not here.
}
```
- [ ] **Step 3: Add discovermass-import to PIPELINE_GROUPS **
In `scripts/scheduler.ts` , find `PIPELINE_GROUPS` . In the first group's `phases` array, after the `masstimes-api-import` entry, add:
```typescript
{ name: 'discovermass-import', type: 'discovermass-import', config: {} },
```
- [ ] **Step 4: Verify TypeScript compiles **
```bash
cd /home/albert/Documents/ScraperControl
npx tsc --noEmit
```
- [ ] **Step 5: Commit **
```bash
git add package.json scripts/scheduler.ts
git commit -m "feat: add discovermass-import to scheduler pipeline and package.json"
```
---
### Task 8: Deploy to Docker and run
The importer will run for ~56 hours in the Docker scheduler container. The scheduler picks it up as part of the PIPELINE_GROUPS sequence.
- [ ] **Step 1: Deploy to Docker directory **
```bash
bash /home/albert/Documents/ScraperControl/scripts/deploy-local.sh
```
Or manually:
```bash
rsync -avz --exclude node_modules --exclude .next --exclude '.env*' \
--exclude .git --exclude .claude --exclude .playwright-mcp \
~/Documents/ScraperControl/ /opt/docker/scraper-control/
```
- [ ] **Step 2: Rebuild Docker images to pick up new script **
```bash
cd /opt/docker/scraper-control
docker compose build scheduler
```
Expected: build completes without errors.
- [ ] **Step 3: Create a manual job via the admin API to trigger the import immediately **
The scheduler can run imports as manual jobs (priority over pipeline):
```bash
curl -X POST http://localhost:3001/api/admin/jobs \
-H "Content-Type: application/json" \
-H "X-Admin-Key: $(grep ADMIN_API_KEY /opt/docker/scraper-control/.env | cut -d= -f2)" \
-d '{"type": "discovermass-import", "config": {}}'
```
Expected: `{"id": "...", "status": "pending", ...}`
- [ ] **Step 4: Restart the scheduler to pick up the new job **
```bash
cd /opt/docker/scraper-control
docker compose restart scheduler
```
- [ ] **Step 5: Monitor the job **
Check logs:
```bash
docker compose logs -f scheduler --since 1m
```
Expected output:
```
Fetching sitemap 1/11...
...
Total church URLs: 20284
Loading existing US churches from DB...
[1/20284] https://discovermass.com/church/...
[create] St. Paul the Apostle — 14M 4C 6A — 1 total (0 new, 1 upd, 0 err)
```
The St. Paul the Apostle church we seeded earlier should show as `[update]` (matched by name+proximity), linking the `discovermassId` to the existing record.
- [ ] **Step 6: Verify St. Paul the Apostle was matched (not duplicated) **
After the importer processes the Chino Hills area (~a few hours in since it's alphabetical by state/slug), run:
```bash
psql postgresql://postgres:postgres@192 .168.0.145:5434/nearestmass \
-c "SELECT name, city, state, discovermass_id, created_at FROM churches WHERE name ILIKE '%Paul%' AND city = 'Chino Hills';"
```
Expected: 1 row with `discovermass_id = 'st-paul-the-apostle-chino-hills-ca'` and `created_at` from the earlier seed (not a new timestamp — it's an update, not a create).
- [ ] **Step 7: Let the full run complete (~56 hours) **
The scheduler will log progress. You can check status anytime:
```bash
docker compose logs scheduler --since 10m | tail -20
```
Or via the admin dashboard at `http://192.168.0.241:3001` — the job will appear in the Jobs tab with status `running` and progress tracked in the `processed` /`succeeded` /`failed` fields.
Final expected stats after completion:
```
Total processed: 20284
Created: ~17000-19000 (new US churches)
Updated: ~1000-3000 (matched against OSM/MassTimes churches)
Errors: < 50 (network blips)
Mass schedules: ~70000-90000
Confession sched: ~30000-50000
Adoration sched: ~10000-20000
```
