Enterprise Integration & Data

Enterprise integration is the work of connecting systems you did not build, cannot change, and cannot turn off.

Point-to-point does not scale. Ten systems each talking directly to the others is up to forty-five integrations, each with its own format and failure mode. The traditional answer was an enterprise service bus — a central component doing routing, transformation, and protocol mediation. It genuinely reduces the connection count and it becomes a bottleneck, a single point of failure, and a place where business logic accumulates until nobody dares change it. The modern equivalent is an event backbone: systems publish events, consumers subscribe, and the broker stays deliberately simple.

A canonical data model defines one shared representation of core entities so each system translates once, to and from the canonical form, rather than to every other system. The tradeoff is real: the canonical model becomes a committee artefact that fits nobody perfectly and changes slowly.

Master data management decides which system is authoritative for each core entity — customer, product, employee. Most integration pain traces back to two systems both believing they own the customer record.

ETL versus ELT. Traditionally you extracted, transformed, then loaded, because storage and compute in the warehouse were expensive. Modern warehouses invert this: load raw, transform inside the warehouse with SQL. ELT keeps the raw data so you can re-derive when the logic turns out to be wrong, which it does.

Warehouse, lake, lakehouse. A warehouse is structured and schema-on-write — you decide the shape before loading, and queries are fast and governed. A lake stores raw files, schema-on-read, cheap and flexible, and degrades into a swamp without discipline. A lakehouse aims for lake economics with warehouse guarantees.

OLTP is not OLAP. Transactional systems handle many small reads and writes; analytical systems scan enormous ranges of few columns. Running heavy analytics on your production database is the most common cause of mysterious latency in an otherwise healthy application.