> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encoreos.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Platform Workflows

> Step-by-step guides for common data management workflows using (Picklists), (Custom Fields), (Field Configuration), (Object Browser), (Custom Objects), (Raw Da…

> **Purpose:** Step-by-step guides for common data management workflows using PF-15 (Picklists), PF-16 (Custom Fields), PF-17 (Field Configuration), PF-23 (Object Browser), PF-24 (Custom Objects), PF-25 (Raw Data Editor), and PF-26 (Object Permissions).

***

## Workflow 1: Adding Custom Data to Core Entities

**Scenario:** Add "Badge Number" field to HR Employees with validation

**Duration:** 10-15 minutes\
**Prerequisites:** Org admin access

### Step 1: Create Picklist (Optional)

If the field needs dropdown options, create a picklist first:

<Steps>
  1. Navigate to **Settings → Picklists**
  2. Click **+ New Picklist**
  3. Fill in:
     * **Name:** `employee_badge_prefix` (snake\_case)
     * **Label:** "Employee Badge Prefix"
     * **Category:** HR
     * **Description:** "Prefix codes for employee badges"
  4. Click **Create**
  5. Add items:
     * `ADM` - Administrative
     * `CLN` - Clinical
     * `OPS` - Operations
     * `SUP` - Support
  6. Click **Save**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { usePicklistMutation } from '@/platform/picklists';

const { createPicklist, createItem } = usePicklistMutation();

// Create picklist
const picklist = await createPicklist({
  name: 'employee_badge_prefix',
  label: 'Employee Badge Prefix',
  category: 'hr',
  description: 'Prefix codes for employee badges',
});

// Add items
await createItem({
  picklist_id: picklist.id,
  value: 'ADM',
  label: 'Administrative',
  display_order: 1,
});
```

### Step 2: Define Custom Field

<Steps>
  1. Navigate to **Settings → Custom Fields**
  2. Click **+ New Field**
  3. Fill in:
     * **Entity Type:** `hr_employees`
     * **Field Key:** `badge_number` (snake\_case, unique per entity)
     * **Field Label:** "Badge Number"
     * **Field Type:** `text`
     * **Required:** Yes
     * **Validation Rules:**
       ```json theme={null}
       {
         "pattern": "^[A-Z]{3}-\\d{4}$",
         "minLength": 8,
         "maxLength": 8
       }
       ```
     * **Help Text:** "Format: XXX-#### (e.g., ADM-1234)"
     * **Placeholder:** "ADM-1234"
  4. Click **Create**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useCustomFieldMutation } from '@/platform/custom-fields';

const { createDefinition } = useCustomFieldMutation();

await createDefinition({
  entity_type: 'hr_employees',
  field_key: 'badge_number',
  field_label: 'Badge Number',
  field_type: 'text',
  is_required: true,
  validation_rules: {
    pattern: '^[A-Z]{3}-\\d{4}$',
    minLength: 8,
    maxLength: 8,
  },
  help_text: 'Format: XXX-#### (e.g., ADM-1234)',
  placeholder: 'ADM-1234',
  display_order: 100,
  field_group: 'Employment Details',
});
```

### Step 3: Configure Field Layout (Optional)

<Steps>
  1. Navigate to **Settings → Field Configuration**
  2. Select entity: **HR Employees**
  3. Select view context: **Create Form** or **Edit Form**
  4. Find `badge_number` field
  5. Configure:
     * **Display Order:** 15 (after employee\_id)
     * **Field Section:** "Employment Details"
     * **Column Span:** 1
     * **Visible to Roles:** All (or specific roles)
     * **Editable by Roles:** HR Admin, Org Admin
  6. Click **Save**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useFieldConfigMutation } from '@/platform/field-config';

const { createConfig } = useFieldConfigMutation();

await createConfig({
  entity_type: 'hr_employees',
  view_context: 'create_form',
  field_source: 'custom',
  field_key: 'badge_number',
  custom_field_definition_id: definitionId,
  display_order: 15,
  field_section: 'Employment Details',
  column_span: 1,
  is_visible: true,
  is_editable: true,
  editable_by_roles: ['hr_admin', 'org_admin'],
});
```

### Step 4: Use in Employee Forms

Update HR employee form to include custom fields:

```typescript theme={null}
import { CustomFieldsSection } from '@/platform/custom-fields';

function EmployeeForm({ employee, onSubmit }) {
  const [formData, setFormData] = useState(employee);
  
  const handleCustomFieldChange = (fieldKey, value) => {
    setFormData(prev => ({
      ...prev,
      custom_fields: {
        ...prev.custom_fields,
        [fieldKey]: value,
      },
    }));
  };
  
  return (
    <Form onSubmit={handleSubmit}>
      {/* Standard fields */}
      <FormField name="first_name" />
      <FormField name="last_name" />
      
      {/* Custom fields */}
      <CustomFieldsSection
        entityType="hr_employees"
        values={formData.custom_fields}
        onChange={handleCustomFieldChange}
      />
      
      <Button type="submit">Save</Button>
    </Form>
  );
}
```

### Step 5: View in Data Manager

<Steps>
  1. Navigate to **Data Manager**
  2. Find **Employees** object
  3. Click to view details
  4. Go to **Fields** tab
  5. See `badge_number` in "Custom Fields" section
  6. Go to **Raw Data** tab to see/edit badge numbers
</Steps>

**Common Pitfalls:**

* ❌ Forgetting to set `display_order` → Field appears at bottom
* ❌ Missing validation rules → Invalid data accepted
* ❌ Not grouping fields → Cluttered form
* ✅ Always test validation before deploying

***

## Workflow 2: Creating a Custom Entity

**Scenario:** Create "Clinical Licenses" custom object to track employee certifications

**Duration:** 20-30 minutes\
**Prerequisites:** Org admin access

### Step 1: Create Custom Object

<Steps>
  1. Navigate to **Data Manager**
  2. Click **+ New Object**
  3. Fill in wizard:
     * **Step 1: Basic Information**
       * Name: "Clinical Licenses"
       * API Name: `clinical_licenses` (auto-generated, can edit)
       * Description: "Track clinical licenses and certifications for staff"
       * Category: "HR & Workforce"
       * Icon: "certificate" (or custom)
       * Color: "blue"
     * Click **Next**
  4. **Step 2: Review** → Click **Create**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useCustomObjectMutation } from '@/platform/data-manager';

const { createCustomObject } = useCustomObjectMutation();

const customObject = await createCustomObject({
  name: 'Clinical Licenses',
  api_name: 'clinical_licenses',
  description: 'Track clinical licenses and certifications for staff',
  category_id: categoryId, // From pf_object_categories
  icon: 'certificate',
  color: 'blue',
});
```

### Step 2: Add Fields

<Steps>
  1. Navigate to **Data Manager → Clinical Licenses**

  2. Go to **Fields** tab

  3. Click **+ Add Field**

  4. For each field:

     **Field 1: Employee**

     * Select existing field definition: "Employee Lookup" (or create new)
     * **Required:** Yes
     * **Show in List:** Yes
     * **List Order:** 1
     * Click **Add**

     **Field 2: License Type**

     * Create new field definition:
       * Field Key: `license_type`
       * Field Type: `select`
       * Picklist: "License Types" (create if needed)
     * **Required:** Yes
     * **Show in List:** Yes
     * **List Order:** 2
     * Click **Add**

     **Field 3: License Number**

     * Create new field definition:
       * Field Key: `license_number`
       * Field Type: `text`
       * **Required:** Yes
       * Validation: Pattern `^[A-Z]{2}-\\d{6}$`
     * **Show in List:** Yes
     * **List Order:** 3
     * Click **Add**

     **Field 4: Expiration Date**

     * Create new field definition:
       * Field Key: `expiration_date`
       * Field Type: `date`
       * **Required:** Yes
     * **Show in List:** Yes
     * **List Order:** 4
     * Click **Add**

  5. Drag fields to reorder if needed
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useCustomObjectFields } from '@/platform/data-manager';

const { addField } = useCustomObjectFields(customObjectId);

// Add employee lookup field
await addField({
  field_definition_id: employeeLookupDefId,
  is_required: true,
  show_in_list: true,
  list_order: 1,
});

// Add license type field (create definition first)
const licenseTypeDef = await createDefinition({
  entity_type: 'clinical_licenses',
  field_key: 'license_type',
  field_label: 'License Type',
  field_type: 'select',
  picklist_id: licenseTypesPicklistId,
  is_required: true,
});

await addField({
  field_definition_id: licenseTypeDef.id,
  is_required: true,
  show_in_list: true,
  list_order: 2,
});
```

### Step 3: Create Records

<Steps>
  1. Go to **Records** tab
  2. Click **+ New Record**
  3. Fill in form:
     * Employee: Select from dropdown
     * License Type: Select from picklist
     * License Number: Enter (validated)
     * Expiration Date: Select date
  4. Click **Save**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useCustomObjectRecords } from '@/platform/data-manager';

const { createRecord } = useCustomObjectRecords('clinical_licenses');

await createRecord({
  data: {
    employee_id: employeeId,
    license_type: 'RN',
    license_number: 'RN-123456',
    expiration_date: '2026-12-31',
  },
});
```

### Step 4: Import Existing Data (Optional)

If you have existing license data in CSV:

<Steps>
  1. Go to **Records** tab
  2. Click **Import**
  3. **Step 1:** Upload CSV file
  4. **Step 2:** Map columns:
     * CSV "Employee Name" → "Employee" (lookup)
     * CSV "Type" → "License Type"
     * CSV "Number" → "License Number"
     * CSV "Expires" → "Expiration Date"
  5. **Step 3:** Preview validation
     * Fix any errors highlighted in red
  6. **Step 4:** Confirm import
</Steps>

* Review success/failure count
* Click **Done**

**CSV Format:**

```csv theme={null}
Employee Name,Type,Number,Expires
John Smith,RN,RN-123456,2026-12-31
Jane Doe,LCSW,LC-789012,2025-06-30
```

**Code Example (Programmatic):**

```typescript theme={null}
import { useCSVImport } from '@/platform/data-manager';

const { importRecords } = useCSVImport('clinical_licenses');

const result = await importRecords(csvFile, {
  columnMapping: {
    'Employee Name': 'employee_id', // Lookup mapping
    'Type': 'license_type',
    'Number': 'license_number',
    'Expires': 'expiration_date',
  },
});

console.log(`Imported: ${result.imported}, Errors: ${result.errors.length}`);
```

### Step 5: Configure Permissions

<Steps>
  1. Go to **Permissions** tab (admin only)
  2. Configure object-level permissions:
     * **HR Admin:** View, Create, Edit, Delete
     * **HR Manager:** View, Create, Edit
     * **Staff:** View only
  3. Configure field-level permissions:
     * **License Number:** Hide from Staff role
     * **Expiration Date:** View only for HR Manager
  4. Click **Save**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useObjectPermissions } from '@/platform/data-manager';

const { updateObjectPermissions } = useObjectPermissions('clinical_licenses');

await updateObjectPermissions({
  object_permissions: [
    {
      app_role: 'hr_admin',
      can_view: true,
      can_create: true,
      can_edit: true,
      can_delete: true,
    },
    {
      app_role: 'hr_manager',
      can_view: true,
      can_create: true,
      can_edit: true,
      can_delete: false,
    },
  ],
  field_permissions: [
    {
      field_key: 'license_number',
      app_role: 'staff',
      permission_level: 'hide',
    },
  ],
});
```

**Common Pitfalls:**

* ❌ Forgetting to set `show_in_list` → Fields don't appear in table
* ❌ Missing required fields → Records can't be created
* ❌ Invalid CSV format → Import fails
* ✅ Always test with sample data first

***

## Workflow 3: Managing Organization Data Structure

**Scenario:** Organize all data objects by category and set up favorites

**Duration:** 15-20 minutes\
**Prerequisites:** Org admin or power user access

### Step 1: Browse All Objects

1. Navigate to **Data Manager**
2. View objects table:
   * **Grouped by category** (default)
   * **Search** by name or API name
   * **Filter** by type (Core vs Custom)
   * **Filter** by favorites

### Step 2: Manage Categories

<Steps>
  1. Click **Categories** in Data Manager sidebar
  2. **Create Category:**
     * Click **+ New Category**
     * Name: "Clinical Operations"
     * Description: "Clinical and operational data"
     * Icon: "stethoscope"
     * Color: "green"
     * Click **Create**
  3. **Edit Category:**
     * Click category row
     * Update name, description, icon, color
     * Click **Save**
  4. **Delete Category:**
</Steps>

* Click category row → Delete
* Confirm (only if no objects assigned)

**Code Example (Programmatic):**

```typescript theme={null}
import { useCategoryMutation } from '@/platform/data-manager';

const { createCategory } = useCategoryMutation();

await createCategory({
  name: 'Clinical Operations',
  description: 'Clinical and operational data',
  icon: 'stethoscope',
  color: 'green',
  display_order: 5,
});
```

### Step 3: Assign Objects to Categories

<Steps>
  1. In Data Manager, click an object row
  2. Go to **Settings** tab
  3. Select **Category:** "Clinical Operations"
  4. Click **Save**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useObjectMetadataMutation } from '@/platform/data-manager';

const { updateMetadata } = useObjectMetadataMutation();

await updateMetadata('hr_employees', {
  category_id: clinicalOpsCategoryId,
});
```

### Step 4: Set Favorites

<Steps>
  1. In Data Manager objects table
  2. Click **star icon** on frequently-used objects
  3. Filter by **★ Favorites** to see only favorited objects
  4. Favorites persist across sessions
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useObjectFavorites } from '@/platform/data-manager';

const { toggleFavorite } = useObjectFavorites();

await toggleFavorite('hr_employees', true);
```

### Step 5: Edit Object Metadata

<Steps>
  1. Click object row → **Settings** tab
  2. Update:
     * **Display Name:** Custom name for your org
     * **Description:** What this object is used for
     * **Icon:** Visual identifier
     * **Category:** Organize with related objects
  3. Click **Save**
</Steps>

**Common Pitfalls:**

* ❌ Creating too many categories → Hard to navigate
* ❌ Not setting favorites → Slow to find objects
* ❌ Missing descriptions → Users don't understand purpose
* ✅ Use consistent naming conventions

***

## Workflow 4: Bulk Data Operations

**Scenario:** Export employee data, edit offline, import updates

**Duration:** 30-45 minutes\
**Prerequisites:** Org member access (view), org admin (import)

### Step 1: Export Data

<Steps>
  1. Navigate to **Data Manager → Employees**
  2. Go to **Raw Data** tab
  3. Apply filters if needed (e.g., specific department)
  4. Click **Export** button
  5. CSV downloads with:
</Steps>

* All standard fields
* All custom fields (from PF-16 definitions)
* Current filter applied

**Code Example (Programmatic):**

```typescript theme={null}
import { useRawDataExport } from '@/platform/data-manager';

const { exportToCSV } = useRawDataExport('hr_employees');

// Export with filters
await exportToCSV({
  filters: {
    department_id: 'dept-123',
    employment_status: 'active',
  },
  includeCustomFields: true,
});
```

### Step 2: Edit Offline

<Steps>
  1. Open CSV in Excel, Google Sheets, or text editor
  2. Make bulk edits:
     * Update multiple records
     * Add new records (follow CSV format)
     * Fix data errors
  3. **Important:** Don't modify:
     * `id` column (primary key)
     * `organization_id` (will be set automatically)
     * Column headers
  4. Save CSV file
</Steps>

**CSV Format:**

```csv theme={null}
id,first_name,last_name,email,department_id,custom_fields
uuid-1,John,Smith,john@example.com,dept-123,"{""badge_number"":""ADM-1234""}"
uuid-2,Jane,Doe,jane@example.com,dept-456,"{""badge_number"":""CLN-5678""}"
```

> **Important:** Custom fields are exported as a single JSON column named `custom_fields`. The value is a JSON object stored as a string within the CSV cell (with quotes escaped as `""`). When editing custom fields in Excel or Google Sheets:
>
> * Keep the JSON structure valid: `{"field_key":"value","another_field":"value2"}`
> * Escape internal quotes by doubling them: `"{""badge_number"":""ADM-1234""}"`
> * Do not add spaces inside the JSON unless they are part of the value
> * To add a new custom field, include it in the JSON: `"{""badge_number"":""ADM-1234"",""employee_code"":""E001""}"`
>
> The system validates JSON syntax on import and will flag rows with malformed custom\_fields.

### Step 3: Import Updates

<Steps>
  1. Go back to **Raw Data** tab
  2. Click **Import** button
  3. **Step 1: Upload**
     * Select CSV file
     * Click **Upload**
  4. **Step 2: Map Columns**
     * System auto-detects column mappings
     * Verify mappings are correct
     * Fix any mismatches
  5. **Step 3: Preview**
     * Review validation results
     * **Green:** Valid rows
     * **Red:** Errors (fix before importing)
     * **Yellow:** Warnings (review but can import)
  6. **Step 4: Confirm**
     * Review summary:
       * Total rows: 150
       * Valid: 148
       * Errors: 2
     * Click **Import** to proceed
  7. **Step 5: Results**
</Steps>

* Success: 148 rows imported
* Errors: 2 rows failed (download error report)
* Click **Done**

**Code Example (Programmatic):**

```typescript theme={null}
import { useRawDataImport } from '@/platform/data-manager';

const { importFromCSV } = useRawDataImport('hr_employees');

const result = await importFromCSV(csvFile, {
  columnMapping: {
    'first_name': 'first_name',
    'last_name': 'last_name',
    'email': 'email',
    'Badge Number': 'custom_fields.badge_number', // Custom field mapping
  },
  updateExisting: true, // Update records with matching id
  createNew: false, // Don't create new records
});

if (result.errors.length > 0) {
  console.error('Import errors:', result.errors);
  // Download error report
  downloadErrorReport(result.errors);
}
```

### Step 4: Verify Import

<Steps>
  1. Go to **Raw Data** tab
  2. Verify updated records show new values
  3. Check error report if any failures
  4. Re-import fixed rows if needed
</Steps>

**Common Pitfalls:**

* ❌ Modifying `id` column → Creates duplicates
* ❌ Wrong date format → Validation fails
* ❌ Missing required fields → Rows rejected
* ❌ Custom field JSON format wrong → Data not saved
* ✅ Always export first to see format
* ✅ Test with 5-10 rows before bulk import

***

## Workflow 5: Configuring Field-Level Permissions

**Scenario:** Hide salary field from non-HR users

**Duration:** 10 minutes\
**Prerequisites:** Org admin access

### Step 1: Navigate to Object Permissions

<Steps>
  1. Go to **Data Manager → Employees**
  2. Click **Permissions** tab (admin only)
  3. View permission matrix
</Steps>

### Step 2: Configure Object-Level Permissions

1. In **Object Permissions** section:
   * **HR Admin:** ✅ View, ✅ Create, ✅ Edit, ✅ Delete
   * **HR Manager:** ✅ View, ✅ Create, ✅ Edit, ❌ Delete
   * **Staff:** ✅ View, ❌ Create, ❌ Edit, ❌ Delete
2. Click **Save Object Permissions**

### Step 3: Configure Field-Level Permissions

<Steps>
  1. Scroll to **Field Permissions** section
  2. Find `salary` field (or custom field)
  3. Set permissions per role:
     * **HR Admin:** Edit
     * **HR Manager:** View
     * **Staff:** Hide
  4. Click **Save Field Permissions**
</Steps>

**Code Example (Programmatic):**

```typescript theme={null}
import { useFieldPermissions } from '@/platform/data-manager';

const { updateFieldPermissions } = useFieldPermissions('hr_employees');

await updateFieldPermissions({
  field_key: 'salary',
  permissions: [
    {
      app_role: 'hr_admin',
      permission_level: 'edit',
    },
    {
      app_role: 'hr_manager',
      permission_level: 'view',
    },
    {
      app_role: 'staff',
      permission_level: 'hide',
    },
  ],
});
```

### Step 4: Verify Permissions

<Steps>
  1. Log in as different role (Staff)
  2. Navigate to **HR → Employees → \[Employee]**
  3. Verify `salary` field is hidden
  4. Log in as HR Manager
  5. Verify `salary` field is visible but read-only
</Steps>

**Common Pitfalls:**

* ❌ Forgetting to save → Changes not applied
* ❌ Hiding required fields → Forms break
* ❌ Too restrictive → Users can't do their job
* ✅ Test with different roles after configuring

***

## Troubleshooting

### Custom Fields Not Appearing

**Symptoms:** Field defined but not showing in form

**Solutions:**

<Steps>
  1. Check field is active (`is_active = true`)
  2. Verify field configuration (PF-17) allows visibility
  3. Check user role has permission to view field
  4. Verify `CustomFieldsSection` component is in form
  5. Check browser console for errors
</Steps>

### Picklist Items Not Loading

**Symptoms:** Dropdown empty or loading forever

**Solutions:**

<Steps>
  1. Verify picklist exists and is active
  2. Check picklist has items (`is_active = true`)
  3. Verify `usePicklistItems` hook is called correctly
  4. Check QueryClient staleTime/gcTime configuration
  5. Verify organization context is set
</Steps>

### Import Validation Errors

**Symptoms:** CSV import fails with validation errors

**Solutions:**

<Steps>
  1. Check CSV format matches export format
  2. Verify required fields are present
  3. Check data types match field definitions
  4. Validate custom field JSON format
  5. Review error report for specific issues
</Steps>

### Permissions Not Working

**Symptoms:** Users can see/edit fields they shouldn't

**Solutions:**

<Steps>
  1. Verify RLS policies are enabled
  2. Check object-level permissions are set
  3. Verify field-level permissions are configured
  4. Check user role assignment
  5. Clear browser cache and re-login
</Steps>

***

## Best Practices

### Custom Fields

* ✅ Use descriptive field keys (snake\_case)
* ✅ Add help text for complex fields
* ✅ Set validation rules to prevent bad data
* ✅ Group related fields in sections
* ✅ Test validation before deploying

### Custom Objects

* ✅ Use clear, descriptive names
* ✅ Follow API naming conventions (snake\_case)
* ✅ Set appropriate categories
* ✅ Add helpful descriptions
* ✅ Test with sample data first

### Data Management

* ✅ Organize objects into logical categories
* ✅ Set favorites for frequently-used objects
* ✅ Keep object metadata up-to-date
* ✅ Use consistent naming conventions
* ✅ Document custom objects for team

### Bulk Operations

* ✅ Always export first to see format
* ✅ Test import with small dataset
* ✅ Validate data before importing
* ✅ Keep backups before bulk changes
* ✅ Review error reports carefully

***

## Related Documentation

* [Data Management Architecture](/architecture/DATA_MANAGEMENT_ARCHITECTURE)
* [PF-15: Picklist System](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-15-picklist-system.md)
* [PF-16: Custom Field Definitions](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-16-custom-field-definitions.md)
* [PF-17: Entity Field Configuration](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-17-entity-field-configuration.md)
* [PF-23: Data Manager - Object Browser](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-23-data-manager-object-browser.md)
* [PF-24: Custom Objects](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-24-custom-objects.md)
* [PF-25: Raw Data Editor](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-25-raw-data-editor.md)
* [PF-26: Object Permissions](https://github.com/Encore-OS/encoreos/blob/development/specs/pf/specs/PF-26-object-permissions.md)

***

**Last Updated:** 2026-01-08\
**Next Review:** When new workflows are added or existing workflows change