Background
Organizations implementing Data Vault 2.0 often face challenges involving complex SQL development, metadata management, and the orchestration of Hubs, Links, and Satellites. The client requested a modular, visual, and no‑codesolution that allows data analysts to build and manage a full Data Vault 2.0 environment—without writing code.
Solution
Our team designed and engineered the Data Vault Framework—a secure, scalable, automated platform for building Data Vault 2.0 environments through a fully visual interface.
Architecture Overview
- Client (React): Visual UI to select source tables, define business keys and hashdiff fields.
- Node.js (Express): Secure middle tier; manages sessions and JWT flow.
- Spring Boot Application Server:
- Orchestration logic, task generation, and metadata management.
- Authenticates users via LDAPand issues/verifies JWT.
- Persists metadata via Spring Datato the repository database.
- Transformation Queue: Database‑backed queue with sync service; enables horizontal scaling.
- Spring Boot Workers: Stateless agents using Spring Schedule + Spring Data; execute DDL creation, Data Lake loads, and transformation pipelines.
- Repository Database & State Machine: Metadata, configuration, process states, and execution logs.
Execution
- Architecture planning & documentation (HLD/LLD).
- Cross‑functional delivery: 2 Spring Boot backend developers, 1 React frontend developer, 1 UI/UX designer (also QA).
- Dynamic runtime layer generating transformation pipelines with no manual SQL or field mapping.
Outcome
- Zero‑code experience for creating and managing Data Vault 2.0 pipelines.
- End‑to‑end automation for modeling, execution, and transformations.
- Horizontal scalability with task queues and worker replication.
- Enterprise‑grade security via LDAP + JWT.
- User‑friendly interface that accelerates Data Vault adoption.
High‑level architecture (JWT + LDAP auth; REST between React/Express and Application Server, and between Workers and Application Server).
