> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encoreos.io/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG Infrastructure Integration Documentation

> Feature ID: PF-60 Version: 1.0 Last Updated: 2026-01-30 Status: 📋 Specification

**Feature ID:** PF-60\
**Version:** 1.0\
**Last Updated:** 2026-01-30\
**Status:** 📋 Specification

***

## Overview

This document describes how PF-60 (RAG Infrastructure) integrates with other platform components and cores. RAG (Retrieval-Augmented Generation) provides semantic search capabilities that enable AI features to reference organization-specific documents and knowledge.

***

## Integration Pattern

**Pattern Type:** Event-Based Integration + Platform Layer

PF-60 acts as both:

1. **Event Consumer:** Subscribes to document lifecycle events from PF-11
2. **Platform Layer:** Provides semantic search API consumed by all AI features

***

## Publisher Dependencies

### PF-11: Document Management

PF-60 subscribes to document lifecycle events to maintain embedding synchronization. Events are emitted with the **DomainEvent envelope** (see EVENT\_CONTRACTS.md): top-level `event_type`, `payload`, and `metadata`. The payload contains only identifiers and timestamps; consumers must fetch document content from `pf_documents` under RLS.

| Event                | Action                | Payload (wire — payload only)                                                            |
| -------------------- | --------------------- | ---------------------------------------------------------------------------------------- |
| `document_published` | Generate embeddings   | `{organization_id, document_id, timestamp, user_id}` — fetch content from `pf_documents` |
| `document_updated`   | Regenerate embeddings | `{organization_id, document_id, timestamp, user_id}` — fetch content from `pf_documents` |
| `document_deleted`   | Delete embeddings     | `{organization_id, document_id, timestamp, user_id}`                                     |

**Payload-only interfaces (align with EVENT\_CONTRACTS and migration `20260226140000_pf_rag_document_knowledge_events.sql`):**

```typescript theme={null}
/** Payload only — event_type is at envelope level. */
interface DocumentPublishedPayload {
  organization_id: string;
  document_id: string;
  timestamp: string;
  user_id: string;
}

interface DocumentUpdatedPayload {
  organization_id: string;
  document_id: string;
  timestamp: string;
  user_id: string;
}

interface DocumentDeletedPayload {
  organization_id: string;
  document_id: string;
  timestamp: string;
  user_id: string;
}
```

Consumers use these types to type the `payload` field of the envelope. For embedding generation, the generate-embeddings edge function fetches `title`, `extracted_content`, `category`, `tags` from `pf_documents` by `document_id`; the event does not carry full content on the wire.

***

## Consumer Cores

PF-60 provides semantic search capabilities to all modules with AI features.

### Core Consumers

| Core                | Usage                                   | Source Type Preferences                            |
| ------------------- | --------------------------------------- | -------------------------------------------------- |
| **GR (Governance)** | Policy search for compliance questions  | `source_type = 'policy'`                           |
| **HR (Workforce)**  | Handbook search for HR queries          | `source_type = 'document', category = 'handbook'`  |
| **FA (Finance)**    | Procedure search for accounting queries | `source_type = 'document', category = 'procedure'` |
| **All Modules**     | General AI enhancement                  | All source types                                   |

### Integration Flow

```
User Query → ai-assistant Edge Function
    ↓
Generate Query Embedding (OpenAI API)
    ↓
pf_search_embeddings(query_embedding, org_id, ...)
    ↓
Inject matched chunks into AI prompt
    ↓
AI Response with Source Citations
```

***

## API Contracts

### Database Functions

**pf\_search\_embeddings**

Performs semantic similarity search against document embeddings.

```sql theme={null}
FUNCTION pf_search_embeddings(
  query_embedding extensions.halfvec(1536),  -- halfvec for 50% storage savings
  org_id UUID,
  match_count INT DEFAULT 5,
  match_threshold FLOAT DEFAULT 0.7,
  source_types TEXT[] DEFAULT NULL
) RETURNS TABLE (
  id UUID,
  source_type TEXT,
  source_id UUID,
  source_table TEXT,
  content TEXT,
  metadata JSONB,
  similarity FLOAT
)
```

**Security:** SECURITY DEFINER with `SET search_path = public`

***

### Edge Functions

**generate-embeddings**

Generates embeddings for a document and stores in database.

```typescript theme={null}
// Request
interface GenerateEmbeddingsRequest {
  sourceType: 'document' | 'policy' | 'knowledge_article';
  sourceId: string;
  content: string;
  organizationId: string;
  metadata?: Record<string, unknown>;
}

// Response
interface GenerateEmbeddingsResponse {
  success: boolean;
  chunksProcessed: number;
  embeddingsCreated: number;
}
```

**ai-assistant (Updated)**

Enhanced to support RAG when `useRAG: true`.

```typescript theme={null}
// Request (extended)
interface AIAssistantRequest {
  messages: AIMessage[];
  moduleContext?: AIModuleContext;
  useRAG?: boolean; // Default: true
  ragConfig?: {
    sourceTypes?: string[];
    matchCount?: number;
    matchThreshold?: number;
  };
}

// Response (extended)
interface AIAssistantResponse {
  content: string;
  sources?: RAGSource[]; // Present when useRAG=true
  usage?: TokenUsage;
}

interface RAGSource {
  sourceType: string;
  sourceId: string;
  title?: string;
  excerpt: string;
  similarity: number;
}
```

***

## Data Flow Diagrams

### Document Indexing Flow

```
┌─────────────────┐         ┌────────────────────┐
│  PF-11 Document │───────▶ │ document_published │
│  Management     │  Event  │     Event          │
└─────────────────┘         └─────────┬──────────┘
                                      │
                                      ▼
                            ┌─────────────────────┐
                            │ generate-embeddings │
                            │   Edge Function     │
                            └─────────┬───────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
           ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
           │ Text Chunking│  │ OpenAI API   │  │   Database   │
           │ (6000 chars) │  │ Embeddings   │  │    Insert    │
           └──────────────┘  └──────────────┘  └──────────────┘
```

### RAG Query Flow

```
┌─────────┐      ┌──────────────────┐      ┌─────────────────┐
│  User   │─────▶│  ai-assistant    │─────▶│  OpenAI API     │
│  Query  │      │  Edge Function   │      │  (Query Embed)  │
└─────────┘      └────────┬─────────┘      └────────┬────────┘
                          │                         │
                          ▼                         │
                 ┌──────────────────┐               │
                 │ pf_search_       │◀──────────────┘
                 │ embeddings       │
                 └────────┬─────────┘
                          │
                          ▼
                 ┌──────────────────┐      ┌─────────────────┐
                 │ Inject Context   │─────▶│  AI Response    │
                 │ into Prompt      │      │  with Citations │
                 └──────────────────┘      └─────────────────┘
```

***

## Security Considerations

### Multi-Tenancy

* All embeddings include `organization_id`
* RLS policies enforce tenant isolation on all operations
* `pf_search_embeddings` uses SECURITY DEFINER but validates org\_id
* UPDATE policy includes WITH CHECK to prevent org\_id modification

### Access Control

Embeddings inherit access control from source documents:

* Users can only search embeddings from their organization
* Same permissions as source documents apply
* No cross-organization embedding access

### PHI/PII Handling

* Embedding `content` field contains document text (may include PHI)
* RLS provides the same protection as source documents
* Embeddings are cascade deleted when source is deleted
* No PHI stored in `metadata` field

***

## Performance Considerations

### Indexing Performance

| Metric               | Target              | Notes               |
| -------------------- | ------------------- | ------------------- |
| Embedding generation | \<5s per document   | OpenAI API latency  |
| Bulk indexing        | 100 docs/batch      | Rate limit handling |
| Retry delays         | Exponential backoff | 1s, 2s, 4s          |

### Search Performance

| Metric             | Target    | Notes             |
| ------------------ | --------- | ----------------- |
| Semantic search    | \<500ms   | HNSW index        |
| Top 5 results      | \<500ms   | Cosine similarity |
| 10,000+ embeddings | Supported | Per organization  |

### Index Configuration

```sql theme={null}
-- HNSW index for fast approximate nearest neighbor search (better than IVFFlat)
CREATE INDEX idx_pf_document_embeddings_vector_hnsw
ON pf_document_embeddings 
USING hnsw (embedding halfvec_cosine_ops);
```

***

## Migration Guide

### For Existing Documents

Use the bulk indexing job to index existing documents:

```typescript theme={null}
// Trigger via edge function
POST /functions/v1/bulk-index-documents
{
  "organizationId": "uuid",
  "sourceType": "document",
  "batchSize": 100
}
```

### For New Integrations

1. Emit `document_published` event when content is created
2. Emit event with identifiers only (`organization_id`, `document_id`, `timestamp`, `user_id`). The consumer (generate-embeddings) fetches content from `pf_documents` by `document_id` under RLS.
3. PF-60 handles embedding generation automatically

***

## Troubleshooting

### Common Issues

| Issue                 | Cause                    | Solution                                      |
| --------------------- | ------------------------ | --------------------------------------------- |
| No search results     | Embeddings not generated | Check `pf_document_embeddings` for source\_id |
| Low similarity scores | Query mismatch           | Adjust `matchThreshold` parameter             |
| Slow generation       | Large documents          | Check chunking, reduce chunk size if needed   |
| 429 errors            | Rate limiting            | Bulk indexing handles this automatically      |

### Monitoring

Check embedding status:

```sql theme={null}
SELECT source_type, COUNT(*) as count
FROM pf_document_embeddings
WHERE organization_id = 'your-org-id'
GROUP BY source_type;
```

***

## Dependencies

| Dependency                    | Type     | Required For                 |
| ----------------------------- | -------- | ---------------------------- |
| PF-59 (AI Provider Migration) | Required | OpenAI/OpenRouter API access |
| PF-11 (Document Management)   | Required | Source documents and events  |
| PF-27 (Platform AI)           | Required | AI infrastructure base       |
| pgvector extension            | Required | Vector similarity search     |

***

## Related Documentation

* **Spec:** `specs/pf/specs/PF-60-rag-infrastructure.md`
* **Tasks:** `specs/pf/tasks/PF-60-TASKS.md`
* **AI Strategy:** `docs/architecture/analysis/AI_INTEGRATION_STRATEGY_2026.md`
* **Event Contracts:** `docs/architecture/integrations/EVENT_CONTRACTS.md`

***

**Last Updated:** 2026-01-30