Comprehensive Data Migration Solutions for a complex industry
challenge
Our fully-digital Banking client aimed to accommodate its rapid growth and gain insights for better member service through revamped data systems. The client came to evolv for assistance in migrating data for their Credit Card, Investment, and Banking products to Snowflake. The data migration project had two objectives: migrate all historical and future data to Snowflake, and recreate reports and queries for legal, operational, and strategic purposes.
The data migration was proven a challenge to scope, particularly in identifying undocumented complexities within the large data warehouse. These complexities ranged from outdated business rules, conflicting logic, unused tables, and more. evolv proactively employed detailed status reports to unveil the hidden complexities and refine the migration strategy accordingly. This iterative process allowed for shifted project priorities as necessary, such as building new and improved reporting dashboards, deprecating unused tables, and fixing known issues in the data.
approach
To kick off the project, the Data Scientists and Data Engineers compiled a comprehensive list of tables and reports required for migration. This cross-team collaboration was instrumental in ensuring a holistic enterprise perspective and optimizing the overall impact on the business.
As evolv progressed through the list, they encountered numerous unexpected challenges. Following agile principles, the team diligently documented and categorized each issue, classifying them as ‘business critical’, ‘fast follow’, or ‘nice to haves’. All the identified issues were transparently displayed on a dashboard and shared with stakeholders to monitor progress. This practice fostered trust with the client and empowered the entire team to forge ahead.
To ensure continuous progress, evolv held weekly working sessions with stakeholders, addressing their queries and unblocking the development process. They established a dedicated communications channel to seamlessly transition end-users to the new data platform, fostering transparency and connectivity across teams. By leveraging project trackers, daily updates, and transparent communication, evolv successfully delivered on their commitment to efficiently and comprehensively migrating all data and reports.
tools used
results
- Systematically identified and addressed all process gaps using a prioritization strategy, maximizing all collaborative efforts for the entirety of the migration project.
- Ensured alignment and compliance with industry-required standards, mitigating potential penalties, and safeguarding the company’s reputation and operations.
- Streamlined the Data Engineer backlog by creating a centralized intake form to manage future data and report requests from multiple product teams
- Minimized downtime & risk of large-scale failures due to the Micro Metrics architecture, ensuring system resilience & reliability
- Reduced costs & increased efficiency in data ingestion, optimizing resource utilization & overall data management processes
For just one of the DataMarts, evolv was able to:
- Reduce daily runtime of a DataMart from 2 hours to 23 minutes
- Reduce backfill runtime of a DataMart from 72 hours (about 3 days) to 54 minutes
- Reduce yearly backfill consumption in Snowflake from 144 to 1.8 credits
