by Gabriel Grynsztein
Data fuels decision-making in every modern industry. As businesses grow, legacy systems often become cluttered with outdated tables, inconsistent metrics, and redundant calculations. Migrating to a cloud platform like Snowflake delivers scalability, improves performance, and saves costs.
But moving legacy data is far more complex than a simple copy-paste job.
evolv recently partnered with one of the fastest-growing financial technology banks in the United States to migrate their legacy data. The bank’s legacy data warehouse was a fragmented mix of on-premise Postgres databases, isolated departmental systems, and scattered reports in Excel and Tableau. Our goal was to unify these into Snowflake, creating a single, reliable source for critical reporting needs in Tableau. The migration focused on ensuring data accuracy, security, and accessibility to support the bank’s rapid growth.
The Challenge
Migrations like these aren’t always as simple as moving a bunch of data from one location to another. Before we could begin, we had to problem-solve for a number of challenges, including:
- No Single Source of Truth: The Postgres system was filled with redundant or unused tables. Departments created their own datasets and reports, leading to conflicting metrics. For example, the marketing team’s churn rate differed from that of the credit card team, causing leaders to report out inconsistent KPIs.
- Historical Metric Shifts: Correcting calculations during migration altered historical data, threatening reporting consistency. Teams were reluctant to adopt new figures that differed from familiar, yet inaccurate, legacy reports.
- Poor Data Understanding: Many legacy tables lacked documentation, and only a few employees understood their details. Calculations were scattered across data tables, Tableau, or Excel, making it hard to pinpoint accurate formulas.
The Solution
To tackle these challenges, we implemented a clear, structured migration plan:
- Establish a Single Source of Truth: We started by mapping all existing tables, their uses, and connections. Working with data architects, we designed clean, cost-effective, and intuitive Snowflake schemas. We collaborated with department leaders to standardize calculations and fields. For instance, the credit card team became the sole owner of credit card KPIs, and other teams could access these metrics via Snowflake’s secure data-sharing feature but couldn’t modify core formulas. If adjustments were needed, teams could build documented, derived tables.
- Handling Historical Metric Changes: Before migrating, we gained stakeholder approval for updated calculations. To build trust, we demonstrated why new metrics were accurate, such as showing how old fraud rate formulas missed key data points. For teams wary of changes, we maintained parallel metrics in Snowflake, clearly documenting the differences until approval was given.
- Automated Checks and Documentation: We developed Python scripts to automatically validate tables and reports after each migration batch or schema update. These scripts flagged issues for review by table owners and SMEs. Regular updates kept everyone aligned, fostering collaboration. We also used a data dictionary tool to certify approved metrics, making them easy to reference.
Our Tool Kit
As a Snowflake Elite Partner, evolv leveraged powerful tools to ensure a seamless migration. Below are the key tools and how they supported the process:
- Snowflake: Snowflake served as the central platform for data storage, processing, and sharing, enabling efficient handling of large datasets. Its zero-copy cloning allowed us to test migrations without duplicating data, saving time and resources. Features like Time Travel were invaluable for accessing historical data to resolve discrepancies or validate legacy metrics during the migration.
- Python: Python scripts automated critical tasks like data validation, profiling, and ETL processes, streamlining the migration workflow. By scripting checks for data consistency and quality, we reduced manual errors and ensured reliable data transfers.
- dbt (Data Build Tool): dbt enabled modular data transformations, ensuring consistent metric calculations across all tables. By centralizing logic in dbt, we eliminated the scattered calculations previously found in Excel or Tableau, creating a unified data pipeline.
- Data Dictionary Tools: Tools like Alation and Snowflake’s native metadata catalog provided a centralized hub for documenting and certifying metrics. By making metadata accessible and searchable, these tools improved collaboration and ensured long-term maintainability of data.
Future Trends to Watch
As fintechs embrace Snowflake, three trends will shape future migrations:
- AI-Powered Data Mapping: Banks often deal with decades-old, undocumented systems, making migrations slow and risky. AI-driven tools, like those in Snowflake’s ecosystem, use machine learning to map table relationships, lineages, and usage patterns. They identify redundant tables, propose efficient Snowflake schemas, and catch data quality issues, speeding up migrations, cutting costs, and ensuring compliance with regulations like GDPR.
- Real-Time Data with Snowpipe Streaming: Fintech banks rely on instant data for fraud detection, risk analysis, and personalized customer experiences. Snowflake’s Snowpipe Streaming pulls data continuously from sources like Kafka or AWS Kinesis, replacing slow batch systems. Its serverless design scales effortlessly during high-traffic periods, enabling faster insights and cost efficiency.
- Zero-Trust Security and Dynamic Masking: Strict regulations demand robust security. Snowflake’s zero-trust features—row-level security, column-level encryption, and dynamic data masking—protect sensitive data like customer PII. These tools enforce compliance during migrations and support secure data sharing for initiatives like open banking, preparing banks for rising cyber risks.
Overall, the migration project was a success. evolv’s expertise improved the bank’s data infrastructure by reducing refresh load times by 92% and decreasing time of detection for anomalies by 99%.
By adopting these trends, banks can turn Snowflake migrations into opportunities for innovation, compliance, and efficiency, solidifying their leadership in fintech. evolv’s expertise is here to guide you. Ready to transform your data? Contact us today!
Gabriel Grynsztein is a seasoned Senior Consultant at evolv Consulting and one of the company’s longest-serving team members. Drawing on his extensive background in financial services, Gabriel excels at using cutting-edge tools like Snowflake to create smart, data-powered solutions that make a real impact. He’s backed by key certifications such as CSM, PSPO, and PMP, which highlight his strong foundation in agile practices and project leadership. From his home-base in Miami, Gabriel brings a proven record of success making him a go-to expert for driving meaningful change in the industry.
