Designing Multi-Tenant Vector Storage on Cloudflare Vectorize: Per-Org Indexes vs. Shared Namespaces

While building a hobby RAG platform on Cloudflare Workers, I had to make a decision early on: how do you isolate vector data between different users in Cloudflare Vectorize?

The obvious first answer is one index per user (or organization). It’s clean and simple. But as I added features, I realized it doesn’t hold up, and I redesigned the architecture before it became a real problem. Here’s what changed and why.

The Initial Design: One Index Per Tenant

The first approach was straightforward: when a user sets up their first AI agent, create a dedicated Vectorize index for them.

user_a -> index: user-a-1755321380084  (384 dims, cosine)
user_b -> index: user-b-1755400000000  (384 dims, cosine)

This maps cleanly to the product model. No cross-tenant data exposure, easy cleanup on deletion, simple mental model.

What Made It Awkward

The platform supported configurable embedding models. Some agents used all-MiniLM-L6-v2 (384 dimensions), others used text-embedding-3-small (1536 dimensions). Vectorize indexes have a fixed dimension count set at creation, so you can’t mix sizes in the same index. An agent with two different embedding configurations needed two physical indexes.

So one user with a couple of differently configured agents could already need several indexes. At that point, the per-user model produces more indexes than users, and the naming and selection logic starts to scatter across the codebase.

I also modeled the concept of “partitions” at the application layer, where each agent could have one or more partitions with different dimension and metric settings. Every partition mapping to a physical Vectorize index was the wrong level of abstraction.

The Redesign: Shared Indexes with Namespace Isolation

Cloudflare Vectorize supports namespaces: a logical partition within an index. When you insert vectors, you attach a namespace string. When you query, you filter by namespace. Vectors from different namespaces never appear in each other’s results. On paid plans, each index supports up to 50,000 namespaces.

The revised approach: one physical index per dimension-and-metric combination, with namespaces handling the actual tenant isolation.

d-384-m-cosine-shard-0   (shared, 384 dims, cosine)
  namespace: partition-1  -> user A, agent 1
  namespace: partition-2  -> user B, agent 1
  namespace: partition-3  -> user A, agent 2

d-1536-m-cosine-shard-0  (shared, 1536 dims, cosine)
  namespace: partition-1  -> user B, agent 2

The naming convention encodes the configuration. d-384 is 384 dimensions, m-cosine is the distance metric, shard-0 is the first physical index for that configuration. If a shard hits the application-layer partition cap (I set it at 10), a new shard is created automatically.

The Database Structure

This requires separating global state from per-tenant state.

Global database tracks physical indexes and their partitions:

CREATE TABLE vector_stores (
  id TEXT PRIMARY KEY,
  index_name TEXT NOT NULL UNIQUE,  -- e.g., d-384-m-cosine-shard-0
  dimensions INTEGER NOT NULL,
  metric TEXT NOT NULL,
  partition_count INTEGER DEFAULT 0,
  created_at TEXT DEFAULT (CURRENT_TIMESTAMP)
);

CREATE TABLE vector_store_partitions (
  id TEXT PRIMARY KEY,
  vector_store_id TEXT REFERENCES vector_stores(id),
  namespace TEXT NOT NULL,
  created_at TEXT DEFAULT (CURRENT_TIMESTAMP)
);

Tenant database maps agents to their partitions:

CREATE TABLE agent_partitions (
  id TEXT PRIMARY KEY,
  agent_id TEXT REFERENCES agents(id),
  partition_id TEXT NOT NULL,  -- references global vector_store_partitions
  dimensions INTEGER NOT NULL,
  metric TEXT NOT NULL
);

Cross-database references are soft (no hard foreign keys across databases). Resolution happens at the application layer.

Real Insertion Numbers

To test it, I processed a 16MB PDF (491 text chunks after splitting) into the shared index:

Attempting to insert into index: d-384-m-cosine-shard-0, namespace: partition-3
Processing batch 1/1 with 491 vectors
Successfully inserted 491/491 vectors
Queue processing time: 6,883ms

After insertion, the Wrangler CLI showed the shared index state:

Index: d-384-m-cosine-shard-0
Dimensions:  384
Vector Count: 1,022
Processed at: 2025-08-16T19:47:57.906Z

1,022 vectors from two different agents in one physical index, isolated by namespace. A query with namespace: partition-3 returns only those 491 vectors.

The Tradeoff

Shared indexes are a softer isolation boundary than separate indexes. The namespace filter handles data isolation correctly, but a misconfigured query without a namespace would touch all vectors in the index. I enforced this at the data access layer: namespace is always required, never optional, with no default fallback.

If you need absolute physical isolation (e.g., enterprise compliance requirements), separate indexes per tenant is the right call. But for most RAG applications where you control the query path, namespace isolation is sufficient and far easier to manage.

The Takeaway

If you’re building a multi-tenant RAG system on Vectorize, think carefully before creating one index per tenant. The moment you add configurable embedding models, multi-source agents, or any feature that multiplies logical stores per user, you’re creating more indexes than you want to manage.

Namespaces let Vectorize handle the isolation while you keep the physical index count proportional to your embedding configurations, not your user count.