About this project
This database aggregates Freedom of Information Act releases published by U.S. federal agencies. It is a search index over public documents — it does not file requests and it stores no personal data beyond what agencies themselves publish.
How it works
- A scheduler runs the ingestion job on the cron schedule below (default 03:00 daily).
- Each source is scraped from its public listing; outlet RSS feeds provide current events.
- New records are de-duplicated by canonical URL and stored in SQLite.
- A local AI model writes a one-sentence summary for each new record.
- Fuzzy matching links releases to recent news coverage.
Schedule
0 3 * * * (UTC)
Current contents
- 483 documents from 10 enabled sources
- 98 with local AI summaries
- 119 news items matched for context
Caveats
AI summaries are machine-generated and can be wrong. Always check the original release before relying on anything here. Scrapers track public pages that change without notice; a source may be temporarily unavailable.
API
Read-only JSON endpoints:
- /api/stats
-
/api/documents — supports
?q=,?limit=,?offset=,?agency= - /api/highlights
- /api/news
- /api/sources
- /api/ingest-runs