Ingest
Receive data from events or scheduled retrieval.
D2 Automation Systems
Case study công khai 03
Data IntegrationIntegration architecture
D2 tách riêng kiến trúc, implementation evidence và evidence boundary để người đọc phân biệt rõ system design đã chứng minh với production outcome chưa được xác minh.
Xem methodology về evidencen8n Automation PortfolioRecruiter view · private link
Architecture Prototype / Integration System DesignRole: Automation Engineer / Workflow Architect
Selected project 03
Multi-Platform Data Integration Hub
Webhook-Driven Integration, Normalization & Data Reliability Architecture
A reusable integration layer for moving business data reliably across APIs, operational systems and analytics infrastructure — with normalization, validation, deduplication, governance and routing as first-class concerns.
n8nWebhooksPollingSalesforceQuickBooksShopifyHubSpotPostgreSQLSnowflakeSupabase
01 · System problem
Disconnected systems create a data-contract problem, not just an API problem.
CRM, accounting, commerce, marketing databases and warehouses expose different schemas, identifiers and transport patterns. A durable integration layer must convert those differences into a controlled internal model before downstream systems can trust the data.
Normalize
Map heterogeneous fields into a canonical business shape.
Govern
Validate, protect, deduplicate and preserve lineage.
Distribute
Route trustworthy records to operational and analytical destinations.
02 · Source → canonical model → destinations
One integration layer between platform-specific APIs and downstream business systems.
The workflow routes five source types into a shared processing pipeline, then fans validated records out to operational, analytical and governance destinations.
Source systems
Salesforce
QuickBooks
Shopify
HubSpot
Generic API
Canonical integration layer
Source routing
Metadata
Schema validation
Canonical transform
Deduplication
PII hashing
Quality scoring
Destination routing
Destinations
Operational systems
QuickBooks · Salesforce · HubSpot
Analytics & storage
PostgreSQL Analytics DB · Snowflake Data Warehouse
Audit & lineage
Supabase Audit Log · Google Sheets Audit Backup · Supabase Data Lineage
03 · Ingestion
Event-driven when immediacy matters. Polling when the source requires it.
The source includes both a POST webhook entry point and a scheduled polling trigger. Enterprise integration rarely has one universal ingestion model: some systems push events, while others must be queried on a schedule.
Webhook ingestion
A POST webhook provides an event-driven entry point for source events that can be pushed into the integration layer.
Scheduled polling
A schedule trigger supports pull-based collection where the upstream platform does not emit the required event or where periodic reconciliation is preferred.
Webhook / Poll
Workflow config
Route by source
Platform fetch
Merge
Metadata enrichment
04 · Canonical data pipeline
Normalize business meaning before broad distribution.
The workflow enriches source data with ingestion metadata, validates required identity fields, then maps heterogeneous platform fields into a shared record shape before deduplication and downstream routing.
Source
Source routing
Fetch
Metadata
Validation
Transform
Dedupe
PII protection
Quality check
Metadata
Adds ingestion timestamp, workflow execution ID, source and record identifier.
Schema validation
Checks recordId and dataSource and records validation errors with a validation timestamp.
Canonical transform
Maps id, source, email, name, createdAt, updatedAt, status and source metadata into one shape.
Deduplication
Uses a source + id compound key inside the workflow to remove duplicate records in the current processing set.
PII protection
The current crypto nodes apply SHA-256 hashing to email fields. This is hashing, not reversible encryption.
Quality control
Runs anomaly analysis and quality scoring separately from deterministic schema validation.
05 · Destination layer
Operational sync and analytical storage are treated as different destination classes.
Validated records can route back into business applications or into analytics/storage systems. Audit and lineage sinks sit alongside those destinations rather than being mixed into the business payload itself.
Operational systems
QuickBooks
Salesforce
HubSpot
Analytics & storage
PostgreSQL Analytics DB
Snowflake Data Warehouse
Audit & lineage
Supabase Audit Log
Google Sheets Audit Backup
Supabase Data Lineage
06 · Reliability
Reliability controls are explicit — and prototype boundaries are explicit too.
Validation alerts
Invalid records branch to a Slack validation-failure alert instead of being silently accepted.
Retry policy
Error handling calculates up to 3 retries with a 1,000 ms base exponential-backoff policy.
Error notification
The error path sends execution context and retry state to Slack.
Audit logging
Destination activity is prepared for Supabase audit logging, Google Sheets backup and lineage tracking.
Prototype boundary: the source computes retry count, delay, shouldRetry and nextRetryAt, but does not wire those values back into a wait/re-execution loop. The page therefore treats exponential backoff as a retry-policy prototype, not a proven automatic retry mechanism.
07 · Governance
Governance is part of the data path, not an afterthought.
The source includes data-lineage records, audit sinks, PII hashing controls, a scheduled key-rotation hook and a GDPR deletion-request workflow. Where the source stops short of full implementation, the boundary is stated explicitly.
PII protection
Email fields pass through SHA-256 crypto nodes. Despite their node labels, the implemented operation is hashing rather than reversible encryption.
Data lineage
A lineage record tracks source system, destination system, transformation names, quality score, record count, execution ID and validation status into Supabase.
Key rotation hook
A 30-day schedule calls a rotate_keys HTTP endpoint. The endpoint itself is a placeholder, so this demonstrates rotation orchestration rather than a completed key-management implementation.
GDPR deletion request
A header-authenticated webhook prepares deletion scope across PostgreSQL, Snowflake, Supabase and Google Sheets. The source comments explicitly state that actual deletion would be handled by subsequent nodes, which are not present in this workflow.
08 · Data quality
A successful API response only proves transport succeeded. It does not prove the data is trustworthy.
The workflow separates deterministic schema validation from anomaly-oriented quality analysis. That distinction matters: structural validity and business-data quality are different controls.
Deterministic validation
Required record identity and source fields are checked in code. Invalid records retain validation errors and can branch to a Slack alert.
Anomaly & quality scoring
A GPT-4o anomaly-analysis branch produces structured anomaly fields. A separate scoring step derives data-quality, completeness and consistency signals, with a quality threshold of 70 used for routing.
These are workflow heuristics and model-assisted checks from the prototype — not benchmarked quality guarantees.
09 · Engineering mindset
Integration architecture should reduce coupling as the system grows.
Normalize before downstream use
Validate before persistence
Make integrations source-agnostic where possible
Separate transport from business meaning
Design retries explicitly
Maintain data lineage
Protect PII before broad distribution
Build one integration layer instead of point-to-point spaghetti
10 · Business applications
Where this architecture pattern can apply.
These are example use cases for the architecture, not claims of customer deployments.
CRM ↔ ERP synchronization
Commerce data pipelines
Marketing attribution pipelines
Finance integrations
Central analytics
Operational data warehouse ingestion
11 · Production roadmap
Possible next steps beyond the prototype.
Schema registry and versioned contracts
Dedicated event queues
Change Data Capture (CDC)
Contract testing between integrations
Tenant isolation
End-to-end observability
Data freshness SLAs
Final takeaway
“The workflow is designed around a reusable integration layer rather than tightly coupling every application to every other application.”
The architecture demonstrates how I think about integration as a governed data system: transport is only the beginning; normalization, validation, quality, privacy, lineage and recovery determine whether downstream systems can trust the result.
Back to automation portfolioD2 Automation Systems
Cần một hệ thống được thiết kế từ bottleneck vận hành thực tế?
D2 map quy trình, source of truth, deterministic rule, failure path và evidence boundary trước khi đề xuất scope Automation.