Data Migration & Consolidation - Weptex Technology
Case Study
Data Migration & Consolidation
Domain: Service Industry
Service Line: Data Analytics
Key Challenges
Client required data migration form 300+ databases across 16+ data sources, both on premise, cloud and file based to be consolidated, and pipeline created for moving data across to a BI dashboard.
The existing source comprised 50+ TB of data, which had to be based initially and then parallel streams enabled for near real time updates to BI dashboard
Our Approach & Solution
Business
Multi process data pull pipelines were created to consolidate data from different sources, using API pulls, JDBC DB drivers, FTP transfers, data pipeline plugins.
Backup processes were integrated into pipeline development to create failsafe, for unaccounted failures.
As data transfer was required to be encrypted, data was anonymized during transfer and storage. Also de-anonymized output was provided in final data lake. This was further transformed and business concepts applied, to provide the KPI level inputs to dashboarding software
Technical
The process of orchestration for moving data across multiple sources, was implemented with Cloud Composer.
DataFusion provided the ETL tool of choice, as it provides data anonymization using keys for encryption and decryption
Stagingackup layer was implemented using cloud storage
Final data load and transformations were effected in BigQuery, while the dashboards were created in Qlik.
Benefit
Client is a multi-geo telecom provider with established networks in over 25+ countries, providing mobile, broadband and digital TV platforms all across the globe.