Siedzieje (Polish for roughly “what’s going on”) collects cultural events in one town, Nowy Sącz, from wherever they are announced: web pages, RSS feeds, Instagram and Facebook posts, ticketing sites, posters and even audio. Each event ends up as a structured record: what, when, where, price, organizer and category. The repository also includes a database of about 725 local venues and sources.
The pipeline#
The system is a set of independent agents that talk to each other only through NATS JetStream:
discovery → fetch → convert → normalize → classify → extract → dedup → validate → store
- convert turns everything into text: HTML to text, OCR for posters, speech recognition for audio;
- classify and extract use LLMs to decide whether a text is an event and to pull out its fields;
- dedup and validate catch the same event announced in several places and records with missing or impossible data;
- the result is stored in PostgreSQL. Binary files live in MinIO only temporarily, because text is the source of truth.
Agents are written in Go, Python and TypeScript, whichever suits the job.
Debugging one stage at a time#
A snapshot tool records the input events of any stage and replays them later. An agent can be debugged on real data without running the whole pipeline, and it doesn’t know it is being fed a replay.
Planned outputs include RSS, iCal, Telegram, Discord, ntfy push notifications, Bluesky and Mastodon.