Agent Backend: One Postgres vs. Redis + Kafka + Pinecone

If you're building an agent backend in 2026, the default architecture in most blog posts still looks like this: Redis for the queue, Kafka for events, Pinecone for vectors, a document store for memory, a scheduler for cron. Five systems, five credentials, five failure modes, all before a single agent handles a single user.

Kevin Keller's Postgres Agent Orchestrator is a 200-line agent orchestrator in Python that argues you don't need any of it. One Postgres 18 instance, two cooperating agents, and a handful of Postgres capabilities doing the work of the polyglot stack: the pgmq and ltree extensions plus native JSONB and LISTEN/NOTIFY. It's the clearest recent statement of a thesis we agree with at Powabase: your database is the framework. It's also a demo, and where it stops is exactly where a production agent backend has to start. This post walks both halves: what the 200-line orchestrator gets right, and what has to sit on top of the same primitives before you put real tenants on it.

The Agent Backend Everyone Overbuilds: Redis + Kafka + Pinecone vs. One Postgres

The standard agent stack accumulated one dependency at a time. Postgres holds rows, Redis runs the cache and job queue, Kafka carries events, Pinecone holds vectors, a scheduler fires cron. Each addition brings its own credentials, failure modes, and on-call runbook, a surface disproportionate to the actual load of a small-to-medium agent fleet.

AI agents change the math against that sprawl, not for it. Handing an agent seven data systems means teaching it seven schemas, seven query languages, and seven failure modes, and the odds the agent reaches for the wrong one on any given tool call rise with every additional system. A single Postgres gives the agent one mental model: everything is SQL, everything shares a transaction boundary. Tiger Data makes the same point about test environments: spinning up a forked copy of production for an agent to try a fix is one command on one database and a coordination nightmare across seven.

That's the case for consolidation. The 200-line orchestrator is what it looks like when you actually build it.

The 200-Line Agent Orchestrator: What Kevin Keller's Postgres Build Actually Does

The demo coordinates two agents (a fetcher and a summarizer) around a pgmq task queue of research topics. Three tasks (artificial intelligence, PostgreSQL internals, data sovereignty in Europe) sit in the queue. The fetcher pulls one, hits the Hacker News Algolia API for headlines, calls a local Ollama model for a short summary, and writes the result back. The summarizer wakes on a NOTIFY, reads the memory row, and produces a one-sentence executive briefing. Parent-child relationships use ltree for agent lineage tracking; memory and retrieval live in JSONB columns on the agent_memory table. That's the whole system.

The point isn't the topic list. It's that every piece of infrastructure a "real" agent stack usually pulls in is replaced by a Postgres feature that already ships.

pgmq for Task Queuing Instead of Kafka or Redis Streams

pgmq is a queue extension that lives inside your database. It gives you exactly-once delivery, visibility timeouts, and dead-letter queues, transactionally, inside your existing database. "Transactionally" is the word that matters. When the fetcher writes its summary to a memory table and marks the task done, both happen in one transaction. If the process dies mid-write, the task reappears on the queue when the visibility timeout expires. No orphan job, no half-written memory row.

For most agent workloads, that's the whole queue story. A SKIP LOCKED job queue on a well-tuned Postgres cluster handles tens of thousands of jobs per second on a single box, well above the traffic profile of nearly any agent fleet short of ad-serving.

LISTEN/NOTIFY for Event-Driven Agent Coordination Instead of a Message Broker

The fetcher and summarizer don't poll. When the fetcher commits a memory row, a trigger fires NOTIFY agent_events, '{...}', and the summarizer, sitting on LISTEN agent_events, wakes immediately. LISTEN/NOTIFY event-driven coordination is Postgres's built-in publish-subscribe: one connection notifies, every listening connection receives.

Two limits are worth naming up front. NOTIFY has a payload size limit, and messages are not durable. A listener that isn't connected when the event fires misses it. In the demo, that's fine: the queue row is the source of truth, and the notify is just a wake-up. That distinction (durable state in the queue, ephemeral wake-ups over NOTIFY) is the pattern that keeps this design honest at scale.

JSONB for Agent Memory Instead of a Document Store

JSONB agent memory sounds crude until you look at what JSONB actually gives you: it's stored as decomposed binary at insert time, with a GIN index mapping keys directly to row IDs, so nested-field queries and joins to relational tables happen in one ACID transaction. Full SQL queryability over the same rows an agent reads and writes, with no separate document DB and no dual-write consistency problem. In the orchestrator, JSONB replaces a vector database for memory: retrieval reads and writes structured JSON, not embeddings.

ltree for Agent Lineage Tracking (Parent-Child Task Trees)

When one agent spawns a subtask, you need to know which agent spawned which, for cancellation, cost accounting, debugging, and audit. The demo uses ltree for agent lineage tracking: a task at path root.fetcher.summarizer is queryable as "everything under root.fetcher" with a single indexed operator. No graph database, no recursive CTE gymnastics for the common cases.

pgvector vs Pinecone for RAG Retrieval

When you do want embeddings (the orchestrator itself doesn't), vectors can live in the same database as everything else, indexed with pgvector. The pgvector vs Pinecone question mostly comes down to volume and existing investment. These aren't approximations of the specialists' methods; they're the same HNSW and DiskANN algorithms for vectors, and BM25 for text. Unless you're serving Google-scale query volume, the marginal quality gap doesn't justify a second data plane.

For a deeper walk through why agent memory belongs in Postgres rather than a bolt-on vector store, we wrote the pillar to this piece: agent memory in one Postgres, no vector store.

Why 'Your Database Is the Framework' Is the Right Thesis for an Agent Backend

Most AI infrastructure is accidental complexity, dependencies added reflexively, not because the workload required them.

One Connection String, One Backup, One Monitoring Dashboard

A polyglot stack multiplies operational surface without multiplying capability. One connection string means one credential to rotate. One backup means one point-in-time restore command instead of a coordination protocol across Redis snapshots, Kafka topic offsets, and Pinecone index dumps. One monitoring dashboard means the on-call engineer runs one query instead of correlating a Redis dashboard, a Kafka lag metric, and a Pinecone latency graph to figure out whether the queue is backed up or the vector index is slow.

Transactional Agent State: No Orphaned Redis Keys or Stale Kafka Topics

When the queue, memory, lineage, and vectors all live in one database, an agent step is a transaction. Either the task is marked done, the memory row is written, the lineage edge is inserted, and the embedding is upserted, or none of it happens. Split those across Redis, Postgres, Kafka, and Pinecone, and you're back to hand-rolling saga patterns and reconciliation jobs to clean up orphans when any one system flakes.

The Compounding Cost of a Polyglot Stack

The pre-adoption checklist dev.to publishes is honest about when a broker or dedicated search cluster earns its keep: streaming to external systems, replayable event logs, fan-out beyond one app's scope. Absent those triggers, the additional systems aren't paying rent; they're just something else to page you when a certificate expires.

Where the Demo Breaks: What a 200-Line Orchestrator Is Missing for Production

The 200-line demo proves the primitives work. It does not, and doesn't try to, prove that 200 lines is enough to run a real product on. Four gaps stand out.

No Authentication or Multi-Tenant Isolation

The demo has one user: whoever runs the Python script. Real agent backends have tenants, and tenants must not see each other's tasks, memory, or embeddings. Multi-tenant isolation via Postgres RLS is the known solution, but it's not something a queue-plus-a-notify script gives you. Without RLS policies on every table (agent_runs, memory rows, vector rows, lineage edges), a single application bug leaks tenant A's conversations into tenant B's retrieval results.

The LISTEN/NOTIFY Global Lock and Connection Pooling Bottleneck

LISTEN/NOTIFY is elegant and has sharp edges at scale. Notifications acquire a global commit lock inside Postgres, so a firehose of tiny notifies serializes on that lock. Our own docs are blunt about the pooler side: use LISTEN/NOTIFY for service-to-service eventing, but not over the PgBouncer pooler. Production designs need a session-mode pool for listeners, or an outbox pattern where notifies are wake-ups only and the queue row remains the source of truth.

Single-Instance Blast Radius: Queue, Vectors, and Scheduling Share One Node

Consolidation is the win, and it's also the risk. When the queue, memory, lineage, and vectors all share one node, a runaway HNSW build or a bloated JSONB column affects job dispatch. This is manageable (read replicas for retrieval, partitioning for hot tables, careful autovacuum tuning), but none of it is in a 200-line demo, and none of it happens by accident.

Also worth naming: pgmq is a great queue, but the demo has no cron. Our Postgres image enables pg_cron via shared_preload_libraries, so scheduled agent work can run in the database when that fits, or through a workflow with a cron trigger when it doesn't. Either way, the scheduler is a decision the demo doesn't have to make.

No Governed Schema, Migrations, or Access Policies

The demo owns its schema because it's the only writer. A production agent backend has to survive schema evolution (new columns on the runs table, new indexes on memory, new embedding dimensions) without downtime and without losing in-flight tasks. That means versioned migrations, forward-compatible message shapes, and a policy layer over who can call which endpoint. None of that is Postgres's fault; it's just work the demo doesn't do.

From Demo to Production: What an AI-Native BaaS Adds on Top of the Same Primitives

Our bet at Powabase is that the demo's thesis is right and the demo's gaps are exactly the surface an AI-native BaaS should cover. Same primitives, hardened.

Auth and Row-Level Security for Per-Tenant Agent Data

Every Powabase project ships with a full auth system and Postgres RLS, so every agent run, memory row, and embedding is scoped to a tenant by policy, not by convention. Our typed /api/* surface for agents and RAG is backed by PostgREST over the AI schema, which means the same RLS policies you write for your own tables apply to the AI state (agent runs, workflow executions, retrieval rows) with no separate policy engine to keep in sync.

A Governed, Versioned Schema Over pgmq, ltree, and pgvector

Underneath the API sits the same Postgres primitives the 200-line demo uses. The difference is that our schema is versioned, migrated, and documented: agent runs, workflow executions, per-block logs are stable tables with a known contract. When we add a column, your existing agents don't break. When you want a supervisor pattern where a coordinator delegates to entity agents, that's a first-class strategy in the platform, not something you thread through pgmq by hand. Streaming falls out for free: our streaming patterns push agent state to your UI without you managing a NOTIFY channel yourself.

Connection Pooling, Scaling, and Isolation Without Rewriting the Stack

Every Powabase project runs with row-level security and network isolation over its own Postgres, with retrieval, rerank, and the agent runtime co-located so RAG stays hot and agent loops stay short. Connection pooling is configured so LISTEN/NOTIFY works where it's supposed to and the pooler is used where it's supposed to be. The queue-and-notify pattern the demo pioneered still runs underneath; you just don't have to hand-tune autovacuum or figure out where to draw the read-replica line.

When You Should Still Reach for Redis, Kafka, or Pinecone

Consolidation isn't universal. A few honest exceptions:

• Streaming at Spotify scale. If you're doing petabytes of event streaming with replayable logs consumed by many external systems, Kafka earns its keep. Most agent backends aren't in this regime.
• Vector query volume at Google scale. If you're serving billions of vector queries a day and the marginal latency of a specialist matters commercially, Pinecone or Qdrant Cloud may pencil out. If your team has already standardized on a managed vector DB, the switching cost may outweigh the consolidation win.
• Cache hot paths where microseconds count. Redis's in-memory latency is real. An unlogged Postgres table with pooling covers most caching needs, but sub-millisecond hot paths at high QPS are still Redis's territory.
• Fan-out to external systems. When events need to reach consumers outside your database, third-party webhooks with replay or cross-region streaming, a broker's semantics are worth the operational tax.

For the small-to-medium agent fleet that most product teams are actually building, none of these apply. The pgmq + JSONB + LISTEN/NOTIFY + pgvector stack is enough, and consolidating on it removes more risk than it adds.

Frequently Asked Questions

Is pgmq really a replacement for Kafka?

For durable task queuing with visibility timeouts and dead-letter queues, yes. For replayable event logs consumed by many independent downstreams over long retention windows, no; that's Kafka's domain. Most agent workloads are the first, not the second.

Can LISTEN/NOTIFY handle production event volume?

For wake-ups on top of a durable queue, yes, if you use a session-mode connection for listeners and keep payloads small. Don't treat NOTIFY as durable; it isn't persisted, and a listener that's disconnected misses the event. The queue row is the source of truth.

Is pgvector as good as Pinecone?

For most workloads, functionally yes. pgvector (with pgvectorscale) uses HNSW and DiskANN, the same algorithms the specialists use. The reasons to pick a specialist are volume, latency SLAs at very high QPS, or existing team investment, not algorithmic superiority.

Do I still need Redis for caching?

Sometimes. An UNLOGGED Postgres table with pooling covers a lot of caching needs. Sub-millisecond hot paths at very high QPS are still Redis's territory.

What does Powabase add over running my own pgmq + pgvector setup?

Auth, per-tenant RLS, a versioned schema over the primitives, a typed API for agents and RAG, streaming, and an isolated Postgres per project, so you get the "one database is the framework" architecture without hand-building the production layer around it.