Comprehensive Data Migration Solutions for a complex industry
Overview
Our fully-digital Banking client aimed to accommodate its rapid growth and gain insights for better member service through revamped data systems. The client came to evolv for assistance in migrating data for their Credit Card, Investment, and Banking products to Snowflake. The data migration project had two objectives: migrate all historical and future data to Snowflake, and recreate reports and queries for legal, operational, and strategic purposes.
The data migration was proven a challenge to scope, particularly in identifying undocumented complexities within the large data warehouse. These complexities ranged from outdated business rules, conflicting logic, unused tables, and more. evolv proactively employed detailed status reports to unveil the hidden complexities and refine the migration strategy accordingly. This iterative process allowed for shifted project priorities as necessary, such as building new and improved reporting dashboards, deprecating unused tables, and fixing known issues in the data.
tech stack used
action plan
To kick off the project, the Data Scientists and Data Engineers compiled a comprehensive list of tables and reports required for migration. This cross-team collaboration was instrumental in ensuring a holistic enterprise perspective and optimizing the overall impact on the business.
As evolv progressed through the list, they encountered numerous unexpected challenges. Following agile principles, the team diligently documented and categorized each issue, classifying them as ‘business critical’, ‘fast follow’, or ‘nice to haves’. All the identified issues were transparently displayed on a dashboard and shared with stakeholders to monitor progress. This practice fostered trust with the client and empowered the entire team to forge ahead.
To ensure continuous progress, evolv held weekly working sessions with stakeholders, addressing their queries and unblocking the development process. They established a dedicated communications channel to seamlessly transition end-users to the new data platform, fostering transparency and connectivity across teams. By leveraging project trackers, daily updates, and transparent communication, evolv successfully delivered on their commitment to efficiently and comprehensively migrating all data and reports.
Steps
1
Developed a list of tables and reports that were needed to start the project and that were essential to migration.
2
Document and place all encountered issues into three categories and display for stakeholders to evaluate progress on the project.
3
Hosted weekly working sessions between the team and stakeholders to answer questions and demystify the project to migrate all data reports.
Results
Optimized project efficiency and risk management:
- Systematically identified and addressed process gaps to maximize collaborative efforts.
- Ensured alignment with industry standards to mitigate penalties and safeguard operations.
Streamlined data management processes:
- Centralized intake form created to manage data and report requests from multiple product teams.
- Reduced costs and increased efficiency in data ingestion, optimizing resource utilization.
Enhanced system resilience and reliability:
- Mitigated downtime and large-scale failures with Micro Metrics architecture.
- Ensured system resilience and reliability.
For one of the DataMarts, evolv was able to:
- Reduce daily runtime of a DataMart from 2 hours to 23 minutes
- Reduce backfill runtime of a DataMart from 72 hours (about 3 days) to 54 minutes
- Reduce yearly backfill consumption in Snowflake from 144 to 1.8 credits
