136 lines
4.7 KiB
Markdown
136 lines
4.7 KiB
Markdown
|
# BuscarMisas Network Importer — Design Spec
|
|||
|
|
|
|||
|
|
**Date:** 2026-03-16
|
|||
|
|
**Status:** Approved
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Overview
|
|||
|
|
|
|||
|
|
Add a single importer `scripts/import-buscarmisas-network.ts` that scrapes church data and mass schedules from a network of 5 identical WordPress-based Catholic mass-time directories covering 5 Latin American countries (~15,294 churches total).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Network Sites
|
|||
|
|
|
|||
|
|
| Domain | Country | Churches | Language | Sitemap Type |
|
|||
|
|
|--------|---------|----------|----------|--------------|
|
|||
|
|
| `horariosmissa.com.br` | BR (Brazil) | ~4,732 | Portuguese | `page-sitemap*.xml` |
|
|||
|
|
| `buscarmisas.com.mx` | MX (Mexico) | ~3,950 | Spanish | `page-sitemap*.xml` |
|
|||
|
|
| `horariosmisa.com.ar` | AR (Argentina) | ~3,012 | Spanish | `page-sitemap*.xml` |
|
|||
|
|
| `buscarmisas.co` | CO (Colombia) | ~2,665 | Spanish | `page-sitemap*.xml` |
|
|||
|
|
| `horariomisa.cl` | CL (Chile) | ~935 | Spanish | `post-sitemap.xml` |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Architecture
|
|||
|
|
|
|||
|
|
### Config Map
|
|||
|
|
|
|||
|
|
```ts
|
|||
|
|
interface SiteConfig {
|
|||
|
|
country: string; // ISO 3166-1 alpha-2
|
|||
|
|
language: 'pt' | 'es';
|
|||
|
|
sitemapType: 'page' | 'post';
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
const NETWORK_SITES: Record<string, SiteConfig> = {
|
|||
|
|
'horariosmissa.com.br': { country: 'BR', language: 'pt', sitemapType: 'page' },
|
|||
|
|
'buscarmisas.com.mx': { country: 'MX', language: 'es', sitemapType: 'page' },
|
|||
|
|
'horariosmisa.com.ar': { country: 'AR', language: 'es', sitemapType: 'page' },
|
|||
|
|
'buscarmisas.co': { country: 'CO', language: 'es', sitemapType: 'page' },
|
|||
|
|
'horariomisa.cl': { country: 'CL', language: 'es', sitemapType: 'post' },
|
|||
|
|
};
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### CLI Interface
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Single domain
|
|||
|
|
npx tsx scripts/import-buscarmisas-network.ts --domain horariosmissa.com.br
|
|||
|
|
|
|||
|
|
# All domains sequentially
|
|||
|
|
npx tsx scripts/import-buscarmisas-network.ts --all
|
|||
|
|
|
|||
|
|
# Resume after interruption
|
|||
|
|
npx tsx scripts/import-buscarmisas-network.ts --domain horariosmissa.com.br --resume-from 1200
|
|||
|
|
|
|||
|
|
# Dry run (no DB writes)
|
|||
|
|
npx tsx scripts/import-buscarmisas-network.ts --domain horariosmissa.com.br --dry-run
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Source slug stored in DB: domain with dots replaced by dashes, e.g. `horariosmissa-com-br`.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Data Flow
|
|||
|
|
|
|||
|
|
### 1. Sitemap Discovery
|
|||
|
|
|
|||
|
|
- Fetch `https://{domain}/sitemap_index.xml`
|
|||
|
|
- Extract child sitemap URLs
|
|||
|
|
- For `sitemapType: 'page'`: collect all `page-sitemap*.xml` URLs
|
|||
|
|
- For `sitemapType: 'post'`: collect `post-sitemap.xml`
|
|||
|
|
- Fetch each child sitemap, filter to 3-segment church URLs (path depth = `/{region}/{city}/{church-slug}/`)
|
|||
|
|
- Collect deduplicated list of church page URLs
|
|||
|
|
|
|||
|
|
### 2. Church Page Parsing
|
|||
|
|
|
|||
|
|
For each church URL, fetch the HTML and extract:
|
|||
|
|
|
|||
|
|
| Field | Source |
|
|||
|
|
|-------|--------|
|
|||
|
|
| Name | Table cell after `<strong>Nome</strong>` (PT) or `<strong>Nombre</strong>` (ES) |
|
|||
|
|
| Address | Table cell after `<strong>Endereço</strong>` (PT) or `<strong>Dirección</strong>` (ES) |
|
|||
|
|
| Phone | `href="tel:..."` anchor |
|
|||
|
|
| Latitude/Longitude | Google Maps iframe `src` — `center={lat}%2C{lng}` parameter |
|
|||
|
|
| Country | From `SiteConfig.country` |
|
|||
|
|
| State/Region | 1st path segment of church URL |
|
|||
|
|
| City | 2nd path segment of church URL |
|
|||
|
|
| Mass schedule | Mon–Sun table rows: day name → time string (skip `-` entries) |
|
|||
|
|
|
|||
|
|
### 3. Schedule Parsing
|
|||
|
|
|
|||
|
|
- Day names resolved via existing `src/scrapers/i18n/day-names.ts` patterns
|
|||
|
|
- Portuguese: `Segunda-feira`, `Terça-feira`, `Quarta-feira`, `Quinta-feira`, `Sexta-feira`, `Sábado`, `Domingo`
|
|||
|
|
- Spanish: `Lunes`, `Martes`, `Miércoles`, `Jueves`, `Viernes`, `Sábado`, `Domingo`
|
|||
|
|
- Times are comma-separated within a single cell (e.g. `10:00, 18:00`) — split and create one schedule entry per time
|
|||
|
|
- `-` entries indicate no mass that day — skip
|
|||
|
|
|
|||
|
|
### 4. Upsert to Database
|
|||
|
|
|
|||
|
|
- Use `findDuplicateChurch` (existing church-matcher) to check for duplicates before insert
|
|||
|
|
- Upsert church with `source = {domain-slug}`
|
|||
|
|
- Upsert mass schedules linked to church
|
|||
|
|
- Track progress: log every 100 churches, support `--resume-from {index}`
|
|||
|
|
|
|||
|
|
### 5. Rate Limiting
|
|||
|
|
|
|||
|
|
- 2-second delay between church page fetches (same as `import-discovermass.ts`)
|
|||
|
|
- 5-second delay between domains when running `--all`
|
|||
|
|
- Respect HTTP 429 / 503 with exponential backoff (up to 3 retries)
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Error Handling
|
|||
|
|
|
|||
|
|
- Skip churches where lat/lng cannot be extracted (log warning, continue)
|
|||
|
|
- Skip churches where name is empty
|
|||
|
|
- On fetch error: log and continue to next URL (don't abort the run)
|
|||
|
|
- On DB error: log and continue
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Integration
|
|||
|
|
|
|||
|
|
- Add `import:buscarmisas-network` script to `package.json`
|
|||
|
|
- Add to scheduler pipeline alongside other importers
|
|||
|
|
- No new dependencies required
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Out of Scope
|
|||
|
|
|
|||
|
|
- The `horairemesses.ch` (Switzerland), `gottesdienstheute.de` (Germany), and `masstime.co.uk` (UK) network sites are excluded — those countries already have dedicated importers
|
|||
|
|
- Chile's `page-sitemap.xml` contains only city pages (not churches) — only `post-sitemap.xml` is used for Chile
|