RAG Infrastructure Integration Documentation

Feature ID: PF-60
Version: 1.0
Last Updated: 2026-01-30
Status: 📋 Specification

Overview

This document describes how PF-60 (RAG Infrastructure) integrates with other platform components and cores. RAG (Retrieval-Augmented Generation) provides semantic search capabilities that enable AI features to reference organization-specific documents and knowledge.

Integration Pattern

Pattern Type: Event-Based Integration + Platform Layer PF-60 acts as both:

Event Consumer: Subscribes to document lifecycle events from PF-11
Platform Layer: Provides semantic search API consumed by all AI features

Publisher Dependencies

PF-11: Document Management

PF-60 subscribes to document lifecycle events to maintain embedding synchronization. Events are emitted with the DomainEvent envelope (see EVENT_CONTRACTS.md): top-level event_type, payload, and metadata. The payload contains only identifiers and timestamps; consumers must fetch document content from pf_documents under RLS.

Event	Action	Payload (wire — payload only)
`document_published`	Generate embeddings	`{organization_id, document_id, timestamp, user_id}` — fetch content from `pf_documents`
`document_updated`	Regenerate embeddings	`{organization_id, document_id, timestamp, user_id}` — fetch content from `pf_documents`
`document_deleted`	Delete embeddings	`{organization_id, document_id, timestamp, user_id}`

Payload-only interfaces (align with EVENT_CONTRACTS and migration 20260226140000_pf_rag_document_knowledge_events.sql):

/** Payload only — event_type is at envelope level. */
interface DocumentPublishedPayload {
  organization_id: string;
  document_id: string;
  timestamp: string;
  user_id: string;
}

interface DocumentUpdatedPayload {
  organization_id: string;
  document_id: string;
  timestamp: string;
  user_id: string;
}

interface DocumentDeletedPayload {
  organization_id: string;
  document_id: string;
  timestamp: string;
  user_id: string;
}

Consumers use these types to type the payload field of the envelope. For embedding generation, the generate-embeddings edge function fetches title, extracted_content, category, tags from pf_documents by document_id; the event does not carry full content on the wire.

Consumer Cores

PF-60 provides semantic search capabilities to all modules with AI features.

Core Consumers

Core	Usage	Source Type Preferences
GR (Governance)	Policy search for compliance questions	`source_type = 'policy'`
HR (Workforce)	Handbook search for HR queries	`source_type = 'document', category = 'handbook'`
FA (Finance)	Procedure search for accounting queries	`source_type = 'document', category = 'procedure'`
All Modules	General AI enhancement	All source types

Integration Flow

User Query → ai-assistant Edge Function
    ↓
Generate Query Embedding (OpenAI API)
    ↓
pf_search_embeddings(query_embedding, org_id, ...)
    ↓
Inject matched chunks into AI prompt
    ↓
AI Response with Source Citations

API Contracts

Database Functions

pf_search_embeddings Performs semantic similarity search against document embeddings.

FUNCTION pf_search_embeddings(
  query_embedding extensions.halfvec(1536),  -- halfvec for 50% storage savings
  org_id UUID,
  match_count INT DEFAULT 5,
  match_threshold FLOAT DEFAULT 0.7,
  source_types TEXT[] DEFAULT NULL
) RETURNS TABLE (
  id UUID,
  source_type TEXT,
  source_id UUID,
  source_table TEXT,
  content TEXT,
  metadata JSONB,
  similarity FLOAT
)

Security: SECURITY DEFINER with SET search_path = public

Edge Functions

generate-embeddings Generates embeddings for a document and stores in database.

// Request
interface GenerateEmbeddingsRequest {
  sourceType: 'document' | 'policy' | 'knowledge_article';
  sourceId: string;
  content: string;
  organizationId: string;
  metadata?: Record<string, unknown>;
}

// Response
interface GenerateEmbeddingsResponse {
  success: boolean;
  chunksProcessed: number;
  embeddingsCreated: number;
}

ai-assistant (Updated) Enhanced to support RAG when useRAG: true.

// Request (extended)
interface AIAssistantRequest {
  messages: AIMessage[];
  moduleContext?: AIModuleContext;
  useRAG?: boolean; // Default: true
  ragConfig?: {
    sourceTypes?: string[];
    matchCount?: number;
    matchThreshold?: number;
  };
}

// Response (extended)
interface AIAssistantResponse {
  content: string;
  sources?: RAGSource[]; // Present when useRAG=true
  usage?: TokenUsage;
}

interface RAGSource {
  sourceType: string;
  sourceId: string;
  title?: string;
  excerpt: string;
  similarity: number;
}

Data Flow Diagrams

Document Indexing Flow

┌─────────────────┐         ┌────────────────────┐
│  PF-11 Document │───────▶ │ document_published │
│  Management     │  Event  │     Event          │
└─────────────────┘         └─────────┬──────────┘
                                      │
                                      ▼
                            ┌─────────────────────┐
                            │ generate-embeddings │
                            │   Edge Function     │
                            └─────────┬───────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
           ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
           │ Text Chunking│  │ OpenAI API   │  │   Database   │
           │ (6000 chars) │  │ Embeddings   │  │    Insert    │
           └──────────────┘  └──────────────┘  └──────────────┘

RAG Query Flow

┌─────────┐      ┌──────────────────┐      ┌─────────────────┐
│  User   │─────▶│  ai-assistant    │─────▶│  OpenAI API     │
│  Query  │      │  Edge Function   │      │  (Query Embed)  │
└─────────┘      └────────┬─────────┘      └────────┬────────┘
                          │                         │
                          ▼                         │
                 ┌──────────────────┐               │
                 │ pf_search_       │◀──────────────┘
                 │ embeddings       │
                 └────────┬─────────┘
                          │
                          ▼
                 ┌──────────────────┐      ┌─────────────────┐
                 │ Inject Context   │─────▶│  AI Response    │
                 │ into Prompt      │      │  with Citations │
                 └──────────────────┘      └─────────────────┘

Security Considerations

Multi-Tenancy

All embeddings include organization_id
RLS policies enforce tenant isolation on all operations
pf_search_embeddings uses SECURITY DEFINER but validates org_id
UPDATE policy includes WITH CHECK to prevent org_id modification

Access Control

Embeddings inherit access control from source documents:

Users can only search embeddings from their organization
Same permissions as source documents apply
No cross-organization embedding access

PHI/PII Handling

Embedding content field contains document text (may include PHI)
RLS provides the same protection as source documents
Embeddings are cascade deleted when source is deleted
No PHI stored in metadata field

Performance Considerations

Indexing Performance

Metric	Target	Notes
Embedding generation	<5s per document	OpenAI API latency
Bulk indexing	100 docs/batch	Rate limit handling
Retry delays	Exponential backoff	1s, 2s, 4s

Search Performance

Metric	Target	Notes
Semantic search	<500ms	HNSW index
Top 5 results	<500ms	Cosine similarity
10,000+ embeddings	Supported	Per organization

Index Configuration

-- HNSW index for fast approximate nearest neighbor search (better than IVFFlat)
CREATE INDEX idx_pf_document_embeddings_vector_hnsw
ON pf_document_embeddings 
USING hnsw (embedding halfvec_cosine_ops);

Migration Guide

For Existing Documents

Use the bulk indexing job to index existing documents:

// Trigger via edge function
POST /functions/v1/bulk-index-documents
{
  "organizationId": "uuid",
  "sourceType": "document",
  "batchSize": 100
}

For New Integrations

Emit document_published event when content is created
Emit event with identifiers only (organization_id, document_id, timestamp, user_id). The consumer (generate-embeddings) fetches content from pf_documents by document_id under RLS.
PF-60 handles embedding generation automatically

Troubleshooting

Common Issues

Issue	Cause	Solution
No search results	Embeddings not generated	Check `pf_document_embeddings` for source_id
Low similarity scores	Query mismatch	Adjust `matchThreshold` parameter
Slow generation	Large documents	Check chunking, reduce chunk size if needed
429 errors	Rate limiting	Bulk indexing handles this automatically

Monitoring

Check embedding status:

SELECT source_type, COUNT(*) as count
FROM pf_document_embeddings
WHERE organization_id = 'your-org-id'
GROUP BY source_type;

Dependencies

Dependency	Type	Required For
PF-59 (AI Provider Migration)	Required	OpenAI/OpenRouter API access
PF-11 (Document Management)	Required	Source documents and events
PF-27 (Platform AI)	Required	AI infrastructure base
pgvector extension	Required	Vector similarity search

Spec: specs/pf/specs/PF-60-rag-infrastructure.md
Tasks: specs/pf/tasks/PF-60-TASKS.md
AI Strategy: docs/architecture/analysis/AI_INTEGRATION_STRATEGY_2026.md
Event Contracts: docs/architecture/integrations/EVENT_CONTRACTS.md

Last Updated: 2026-01-30

Architecture

Development

Database

Testing

Migration

Operations

Governance & Security

Platform Internals

RAG Infrastructure Integration Documentation

Overview

Integration Pattern

Publisher Dependencies

PF-11: Document Management

Consumer Cores

Core Consumers

Integration Flow

API Contracts

Database Functions

Edge Functions

Data Flow Diagrams

Document Indexing Flow

RAG Query Flow

Security Considerations

Multi-Tenancy

Access Control

PHI/PII Handling

Performance Considerations

Indexing Performance

Search Performance

Index Configuration

Migration Guide

For Existing Documents

For New Integrations

Troubleshooting

Common Issues

Monitoring

Dependencies

Architecture

Development

Database

Testing

Migration

Operations

Governance & Security

Platform Internals

Documentation Index

​Overview

​Integration Pattern

​Publisher Dependencies

​PF-11: Document Management

​Consumer Cores

​Core Consumers

​Integration Flow

​API Contracts

​Database Functions

​Edge Functions

​Data Flow Diagrams

​Document Indexing Flow

​RAG Query Flow

​Security Considerations

​Multi-Tenancy

​Access Control

​PHI/PII Handling

​Performance Considerations

​Indexing Performance

​Search Performance

​Index Configuration

​Migration Guide

​For Existing Documents

​For New Integrations

​Troubleshooting

​Common Issues

​Monitoring

​Dependencies

​Related Documentation

Overview

Integration Pattern

Publisher Dependencies

PF-11: Document Management

Consumer Cores

Core Consumers

Integration Flow

API Contracts

Database Functions

Edge Functions

Data Flow Diagrams

Document Indexing Flow

RAG Query Flow

Security Considerations

Multi-Tenancy

Access Control

PHI/PII Handling

Performance Considerations

Indexing Performance

Search Performance

Index Configuration

Migration Guide

For Existing Documents

For New Integrations

Troubleshooting

Common Issues

Monitoring

Dependencies

Related Documentation