Data-API Project
Project Overview
This project focuses on building a data-API system that serves different types of business data to various internal teams within the organization. The system is designed to handle large volumes of data efficiently and ensures that data is always available in real-time to the teams that need it.
Core Engineering Workflow
The data engineering pipeline is structured around a multi-source data flow:
-
Data Sources:
- CPS (Core Processing System): The initial source of data.
- DC & DR (Data Center & Disaster Recovery): Data flows through these systems to ensure redundancy and availability across various data centers.
-
Data Syncing:
- Using Oracle Golden Gate (OGG), the data is synchronized between these systems and transferred to an Oracle Database for central storage and management.
-
Data Serving:
- The application then exposes this synchronized data through REST API and GraphQL, allowing internal teams to fetch data efficiently based on their specific requirements.
High-Level Design (HLD)
System Architecture
Technologies Used
- CPS, DC, DR Systems: For sourcing and routing data to different data centers.
- Oracle Golden Gate (OGG): Ensures real-time data synchronization between systems.
- Oracle Database: Serves as the centralized data storage, providing robust data management.
- REST API & GraphQL: APIs are exposed for flexible, high-performance data access across teams.
Impact and Benefits
By leveraging real-time data synchronization and ensuring seamless data accessibility through REST and GraphQL APIs, this project has contributed to significant cost savings annually. The enhanced data availability and optimized infrastructure have improved business operations and decision-making.
This project highlights my role in:
- System integration: Ensuring smooth data flow from the CPS to the Oracle database.
- API development: Building REST and GraphQL APIs for internal data consumption.
- Performance optimization: Enhancing data accessibility and infrastructure efficiency, resulting in considerable revenue savings.
Conclusion
This project demonstrates my ability to work on complex, high-impact data systems, using cutting-edge data sync technologies like OGG and API frameworks like GraphQL. It has proven to be a valuable asset to the business by driving operational efficiency and cost reduction.