83 lines
3.3 KiB
Markdown
83 lines
3.3 KiB
Markdown
# Vision Scanner Service
|
|
|
|
TypeScript/Express service that scans shelf and pantry photos to extract product information and prices using a local vision LLM running on an AMD NPU. Uses ChromaDB for vector embeddings storage and Ollama for embedding generation. Supports image tiling for high-resolution photos.
|
|
|
|
## How It Works
|
|
|
|
1. A photo of a store shelf or pantry is uploaded
|
|
2. The image is tiled into smaller sections for better accuracy on high-res photos
|
|
3. Each tile is sent to the vision LLM (qwen2.5vl-it:3b via FLM proxy) for product extraction
|
|
4. Extracted products are matched against existing entries using vector embeddings (ChromaDB + Ollama)
|
|
5. Optionally enriched via Gemini API as a fallback
|
|
|
|
## Endpoints
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| `POST` | `/scan/shelf` | Scan a store shelf photo (multipart: `image`, `store_name`) |
|
|
| `POST` | `/scan/pantry` | Scan a pantry photo (multipart: `image`) |
|
|
| `POST` | `/enrich/product` | Extract detailed product info from a single product image |
|
|
| `GET` | `/health` | Health check (reports status of vision model, Ollama, ChromaDB) |
|
|
|
|
## Configuration
|
|
|
|
All configuration is via environment variables (`.env` file):
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `PORT` | `8002` | Service port |
|
|
| `VISION_AI_URL` | `http://localhost:8000/v1/chat/completions` | Vision LLM endpoint |
|
|
| `VISION_AI_MODEL` | `qwen2.5vl-it:3b` | Vision model to use |
|
|
| `VISION_AI_TIMEOUT` | `120000` | Timeout for vision LLM calls (ms) |
|
|
| `OLLAMA_HOST` | `http://192.168.0.15:11434` | Ollama server for embeddings |
|
|
| `OLLAMA_EMBED_MODEL` | `nomic-embed-text` | Embedding model |
|
|
| `CHROMA_HOST` | `http://192.168.0.15:8000` | ChromaDB server |
|
|
| `GEMINI_API_KEY` | — | Optional Gemini API key for fallback |
|
|
| `GEMINI_MODEL` | `gemini-2.5-flash` | Gemini model for fallback |
|
|
| `MAX_CONCURRENT_TILES` | `4` | Max parallel tile processing |
|
|
| `UPLOAD_DIR` | `uploads` | Temporary upload directory |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
npm install # Install dependencies
|
|
npm run build # Compile TypeScript → dist/
|
|
npm start # Run the service
|
|
npm run dev # Development mode with hot-reload
|
|
|
|
# Windows service
|
|
node service-install.js
|
|
node service-uninstall.js
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
src/
|
|
server.ts — Express app, routes
|
|
config.ts — Configuration from environment
|
|
vision.ts — Vision LLM API calls
|
|
tiling.ts — Image tiling for high-res photos
|
|
shelf.ts — Shelf scanning logic
|
|
pantry.ts — Pantry scanning logic
|
|
enrich.ts — Product info enrichment
|
|
parsing.ts — LLM response parsing
|
|
embeddings.ts — Ollama embedding generation
|
|
chroma.ts — ChromaDB vector storage
|
|
matching.ts — Product matching via embeddings
|
|
gemini.ts — Gemini API fallback
|
|
```
|
|
|
|
## External Dependencies
|
|
|
|
- **FLM Proxy** (localhost:8000) — Vision LLM inference on AMD NPU
|
|
- **Ollama** (192.168.0.15:11434) — Embedding generation with `nomic-embed-text`
|
|
- **ChromaDB** (192.168.0.15:8000) — Vector database for product embeddings
|
|
- **Gemini API** (optional) — Fallback for product enrichment
|
|
|
|
## Environment
|
|
|
|
- **OS:** Windows 11, AMD NPU hardware
|
|
- **Runtime:** Node.js + TypeScript
|
|
- **Vision LLM:** qwen2.5vl-it:3b served by FLM proxy on localhost:8000
|