1. Rippling-Style Reference (What We Like)
From the Rippling “Import HR history” flow:
Rippling strengths we could adopt:
- Central Data import area – One place (e.g. Settings → Data import) that lists all import types and past batches (with delete).
- Max rows per file – e.g. 10,000 with clear message so users know limits.
- Optional manual entry – For small data sets, type rows instead of CSV.
- Choose unique identifier – For history/update flows: email vs profile ID vs national ID (we’re mostly create-only today).
- Batch list + delete batch – Store import batches and allow “delete this import” to undo.
- Clear disclaimer – “This import only adds history / never updates active data” (when we add history-style imports).
2. Current Encore OS Bulk Import Inventory
2.1 Summary Table
2.2 Shared CSV (Already in Place)
@/platform/csvprovidesparseCsvLine,parseCsv,formatCsvValue,BULK_IMPORT_CHUNK_SIZE(50), andBULK_IMPORT_MAX_ROWS(10,000). All bulk import wizards must enforce max rows in the UI after parse.- HR employee parser uses
parseCsvLinefrom@/platform/csv/parse. - Other modules (CE, credential, oversight, data-manager, FA template, BulkGenerationDialog) should use
@/platform/csvper Constitution §5.10 and BULK_IMPORT_MIGRATION_RECOMMENDATIONS.md; some still have local or duplicate parsing.
2.3 Standard Steps (Aligned)
We already follow:- Upload – Dropzone + “Download template” where applicable.
- Mapping – CE and PF-24; HR employee has fixed schema (no mapping).
- Preview / Validation – Row-level errors, stats, “rows with errors skipped.”
- Import – Edge (HR employee) or client-side chunked (CE, HR credential, HR oversight, PF-24).
- Complete – Summary (created/skipped/failed); optional “Done” / close.
2.4 Gaps vs Rippling / Best Practice
3. Recommendations and Enhancement Plan
3.1 Quick Wins (No Schema Change)
- Document max rows – In bulk-import-patterns and each wizard: “Recommended max 10,000 rows per file; larger files may be slow or fail.”
- Enforce max rows in UI – After parse, if
rows.length > 10000, show error and block “Next” until user reduces or splits. - CE contact template – Add a “Download template” button and a small CSV template (e.g. first_name, last_name, email, phone, contact_type) using
@/platform/csvand the same download pattern as HR. - In-app tips – Add QuickTip or help snippet on first step of each import: “Each file can have up to 10,000 rows. Use the template for correct column headers.”
3.2 Medium-Term (Schema + UI)
-
Import batch table (e.g.
pf_import_batchesor per-core)- Columns:
id,organization_id,import_type(e.g.hr_employee,ce_contact),filename,row_count,created_count,skipped_count,failed_count,created_at,created_by. - Optional:
metadataJSONB for skipped/failed details or list of created IDs. - Enables “list past imports” and “delete this batch” (with care: delete batch might mean “mark as rolled back” and then run a rollback flow, or only hide from list and keep data—product decision).
- Columns:
-
Settings → Data import
- New section under Platform (or Company) Settings: “Data import” with links to:
- Import employee roster (→ HR Employee Directory or open BulkEmployeeImportDialog)
- Import contacts (→ CE Contacts or open ContactImportDialog)
- Import credentials (→ HR Credentialing)
- Import custom object records (→ PF-24, from custom object detail)
- Once batches exist: “Recent imports” list with date, type, filename, counts, and “Delete” where supported.
- New section under Platform (or Company) Settings: “Data import” with links to:
3.3 Longer-Term (Rippling-Style Parity)
- History-only import – If we add “HR history” (promotions, compensation history) that does not create employees or update active records, add disclaimer and separate flow; consider “choose unique identifier” in that flow.
- Manual entry – Optional table/form to add rows without CSV for small datasets.
- Field value mapping – For flexible CSVs, “match CSV values to system values” (e.g. “Full-Time” → “full_time”) in addition to column mapping; CE and PF-24 are the best candidates.
4. Implementation Status (Bulk Import Recommendations)
5. Module Audit (All Cores)
6. Consistency Checklist (Per Module)
When adding or refactoring a bulk import, use this and the existing bulk-import-migration checklist:- Use
@/platform/csvfor all parsing (no localparseCsvLine/parseCsv). - Choose edge vs client per bulk-import-migration decision tree.
- Steps: Upload → [Mapping] → Preview/Validation → Import → Complete.
- “Download template” (or sample CSV) when format is fixed or suggested.
- Progress: determinate for client chunked, indeterminate (or future progress) for edge.
- Enforce or document max rows (e.g. 10,000) and show clear error if exceeded.
- Row-level validation errors;
sanitizeErrorMessagefor user-facing errors. - Invalidate all affected query keys on success.
- Use
BULK_IMPORT_CHUNK_SIZE(50) for client-side chunked inserts. - Record import batch in
pf_import_batcheson success (for Data import hub “Recent imports” and “Delete batch” ).
7. References
- Bulk import migration guide – Module inventory, decision tree, checklist
- BULK_IMPORT_MIGRATION_RECOMMENDATIONS.md – Shared CSV, edge vs client, progress
- .cursor/rules/bulk-import-patterns.md – Shared CSV, batch size, steps, validation
- Constitution §5.10 – Bulk Import & Data Migration Standards
- AGENTS.md – Bulk Import Pattern, Pre-Flight Checklist