How Kroger is Leveraging Data Mesh and Data Fabric to Unlock Value
Published on July 18, 2024
Key Insights
Kroger, the largest retail grocery chain in the US by revenue, built a data mesh architecture to support unique domains, with data fabric "connective tissue" to enable interoperability and data sharing of data products between them.
After several in-house data cataloging initiatives that focused solely on a technological solution, the company partnered with Alation and focused on people, process, and technology to make data easier to find, understand, and use across the organization.
Using Alation together with Databricks, Kroger has created a common language around data, with standardized governance and SLAs for products, as well as automated data profiling, classification, and access control. Data users now benefit from a unified view of data assets across the entire enterprise.
Unity Catalog is used to govern data access and security, making data available in different technologies. Alation brings metadata to the table for self-service data access through features like the Business Glossary, wiki pages, and additional context about the data beyond just the structure and location. This enables more efficient decision-making across the organization.
Introduction
In today's fiercely competitive business landscape, data has emerged as a key driver of competitive advantage. Companies that can effectively harness their data to gain insights, optimize operations, and personalize customer experiences are positioned to outperform their rivals.
Kroger, one of the largest grocery retailers in the United States, knows this challenge well. Its subsidiary, 84.51°, is dedicated to addressing it. This retail data science, insights, and media company is focused on creating more personalized and rewarding experiences for shoppers.
"Data is foundational to everything we do, from creating products and making decisions to enhancing customer experiences,” says Nate Sylvester, VP of Architecture at 84.51°. However, leveraging data is often easier said than done, especially for large, complex organizations.
As a highly federated organization with multiple business units, banners, and subsidiaries, Kroger has vast amounts of data spread across different silos. This fragmented data landscape makes it difficult to gain a unified view of the business, customers, and operations.
"It was difficult to understand what was available across the organization and who were the owners," Sylvester shares. Without a cohesive data strategy and architecture, Kroger risked missing opportunities to drive efficiencies, innovate, and create value from its data assets.
Recognizing the imperative for change, Kroger embarked on an ambitious initiative to transform its data culture and capabilities.
We set out to understand what our hurdles were. We set out to change our perspective. It wasn't just about bringing technology. It was really about how can we change the way that our culture organizes and thinks about the data.
The company sought to break down silos, improve data accessibility and quality, and enable more data-driven decision-making across the enterprise.
A key component of this transformation was adopting a data mesh architecture, supported by a data fabric and other technologies. By organizing around data domains, treating data as a product, and implementing federated computational governance, Kroger aims to unleash the full potential of its data to drive competitive advantage.
Starting with Culture, Not Technology
Kroger's data transformation journey began with a critical realization – its past data strategies had focused too heavily on technology rollouts without addressing the underlying organizational culture.
They recognized that simply implementing new software packages and technologies was not enough to solve their data challenges. Time and again, as they rolled out new tech, they found that without corresponding changes to processes, culture, and expected behaviors, these initiatives failed to deliver the desired results.
This insight led Kroger to approach its latest data strategy differently. Instead of starting with technology, they focused on defining the target data culture and behaviors they wanted to drive within the organization. They understood that for data to be leveraged effectively, the right organizational mindset, processes, and ways of working needed to be in place.
As Sylvester noted, "Data is foundational to everything we do, from creating products to making decisions to enhancing customer experiences." Technology would act as an accelerator, but the foundation had to be a data-driven culture.
To shape this culture, Kroger conducted a series of listening sessions and "discovery sessions" across the business. These sessions aimed to assess the current state of data literacy and understand how different parts of the organization viewed and used data.
The sessions revealed a wide spectrum of perspectives, from those who had a deep understanding of data pipelines and processes to others who viewed data more narrowly through the lens of their specific job functions.
Armed with these insights, Sylvester and his team were able to identify targeted areas to focus on to uplift data literacy, align on common terminology, and drive the desired data behaviors across the enterprise. This focus on culture and organizational alignment set the stage for the subsequent technology and architectural changes to enable Kroger's data mesh implementation.
Defining Guiding Principles for the Data Culture
As Kroger embarked on their data transformation journey, they recognized the importance of establishing clear guiding principles to shape the desired data culture. These principles would serve as the foundation for driving the right behaviors and mindset across the organization.
"It wasn't just about bringing technology,” Sylvester emphasizes. “It was really about: How can we change the way that our culture organizes and thinks about the data?"
After careful consideration, Kroger identified three key principles: ownership, communication, and collaboration.
The first principle, ownership, emphasized the need for clear accountability and responsibility for data within each domain. Teams were expected to take ownership of their data assets, ensuring data quality, governance, and lifecycle management.
However, Kroger also recognized that ownership alone was not enough – it had to be coupled with the right skills and resources for teams to be successful. "New talent must be injected into the areas of data engineering and data science while existing talent is upskilled on data literacy," Sylvester shares. This meant investing in talent management, upskilling existing staff, and injecting new talent in areas like data engineering and data science.
The second principle, communication, focuses on making information widely available across the organization. Sylvester understood that for data to be leveraged effectively, there needed to be a shared understanding of data importance, uses, key metrics, and quality concerns.
"A shared understanding of data importance and uses, key metrics, and quality concerns will drive better outcomes," Sylvester explained. This required establishing clear communication channels, both within and across teams, to align on terminology, share knowledge, and drive better outcomes.
The third principle, collaboration, aimed to foster a community mindset within the federated organization. Kroger acknowledged the tendency for teams to become inwardly focused on their own objectives and priorities. To counteract this, they encouraged a balance between achieving team-level goals and contributing to the broader organizational good – what Sylvester calls "a high level of collaboration within the organization."
By carving out intentional time for cross-functional collaboration, teams could build solutions that benefited areas beyond their own, even if the near-term impact was not immediately apparent. This outward mindset was critical for breaking down silos, driving standardization where needed, and enabling data to flow seamlessly across the enterprise.
Implementing the Data Mesh
Kroger's implementation of the data mesh architecture reorganized teams and data around domains aligned to business capabilities. These domains, such as merchandising or supply chain, encapsulate related data assets and the associated workflows and processes.
The goal was to decentralize data ownership and decision-making while maintaining consistency and interoperability across domains.
Data mesh is really about how we organize, how we create decentralized teams within the business units.
Within each domain, data is treated as a product with clear ownership, interfaces, and service level agreements (SLAs). The domain teams take end-to-end ownership for the data products they provide, managing the entire lifecycle from data curation to consumption. They define data contracts by specifying the structure, quality, and intended use of the data.
Importantly, the teams focus on crafting customer-centric experiences and interactions around the data, rather than just dumping data into a lake. "Our goal is not to create one-off bespoke data products for every use case that exists out there,” Sylvester emphasizes. “We want to do that in the product world. We have thousands and thousands of products that we would have to maintain." This product thinking ensures the data is usable, discoverable, and valuable to consumers.
Data mesh is really about how we organize and create decentralized teams within the business units. Data fabric is the connective tissue that allows us to interoperate.
However, the domains are not islands unto themselves. A key aspect of Kroger's data mesh is the data fabric that serves as the “connective tissue” as Sylvester calls it, between domains.
The data fabric provides standards and consistency for how domains interact and exchange data, such as through well-defined APIs, events, and bulk interfaces. This allows the domains to maintain autonomy in their internal implementations while still enabling seamless interoperability and data sharing across the mesh. The data fabric, supported by technologies like a unified data catalog, makes it possible to democratize data across the organization at scale.
We're not creating swamps of data. We are curating interactions with our consumers. Because if we treat that consumer as a customer and we focus on curating these interactions around the data, then we can have better outcomes.
Driving Adoption through Discoverability and Governance
Kroger recognized that to drive widespread adoption of their data mesh, they needed to make data easily discoverable and governed. A key enabler was implementing a robust data catalog to support self-service discovery.
"We create data all day,” Sylvester says. “Whether it's data, code, metrics, or definitions, it's all over the place. I can't tell you the number of quote-unquote knowledge repositories that we have within the organization. But if a new team is spinning [that] up in order to solve a problem, how do we find what they need?"
By cataloging data assets and capturing rich metadata, Kroger aimed to create a "shop for data" experience where users could easily find, understand, and access the data they needed without relying on tribal knowledge. The catalog would include information on data lineage, quality, intended purpose, and ownership to provide the necessary context for effective use.
"We want teams to be able to move at their pace without slowing them down,” says Sylvester. “So how do we leverage these catalogs? Not just to tell you the attribute names and the data types, but how do we actually leverage that to tell you the purpose, the intended purpose?"
To track their progress on this journey, Kroger defined balanced scorecards. These scorecards measured key metrics around data cataloging, ownership assignment, quality checks, and more.
"We set out to do all these things,” Sylvester recalls. “We want to change behavior. We're asking people to do things differently. How are we doing? How are we measuring ourselves?"
The scorecards provided visibility into areas that were doing well and those that needed more focus and investment. Importantly, the scorecards were not meant to be punitive, but rather to highlight opportunities for improvement and celebrate successes along the way.
"The scorecards are not meant to be like a stick,” Sylvester says. “We don't want to go to teams and whack them over the head with it. But to showcase the opportunities that we have. How are we doing on cataloging our data? How are we doing with assigning ownership and those responsibilities? How are we doing with our QA checks and communicating those different things?"
Underpinning Kroger's data mesh was a hybrid governance model. They realized that in a large, federated organization, a one-size-fits-all approach to governance would not work. Instead, they aimed to strike a balance between consistency and flexibility.
Centrally, they defined standards and processes around key areas like data cataloging, domain interfaces, and quality expectations. This ensured a level of interoperability and trust across domains. However, within each domain, teams had the autonomy to make decisions suited to their specific needs and use cases.
We believe in hybrid governance. There are things at the center that we need to ensure. When we talk about things like cataloging data with the community catalog, maybe our data formats, those domain interfaces, those are defined centrally. But there's a level of governance that happens within each domain that they have to self-govern.
This “hub-and-spoke” approach allowed the domains to be agile and innovative while still adhering to the guardrails set by the central governance team.
Leveraging Technology as an Accelerator
While Kroger's data transformation journey began with a focus on culture and behaviors, they recognized that technology would play a critical role in accelerating their progress.
"We need technology in order to accelerate us,” Sylvester points out. “We need it] to get the efficiencies to automate a lot of these things in a scalable organization." As they defined their target data culture and implemented the data mesh architecture, the team sought out technologies that could help them scale their efforts and drive automation. Three key technologies that have been foundational in this regard are Databricks' Lakehouse platform, Unity Catalog, and Alation.
The Databricks Lakehouse platform provides a unified environment for data engineering, data science, and analytics workloads. It combines the best elements of data lakes and data warehouses, enabling Kroger to store and process vast amounts of structured and unstructured data at scale.
With the Lakehouse, Kroger can break down silos and bring together data from across the enterprise into a single platform for analysis and insights. As Sylvester explains, the Lakehouse provides "a holistic user experience for data and science user communities." Importantly, the Lakehouse also provides a common platform for the data mesh domain owners to build and share their data products.
Unity Catalog, a key component of the Lakehouse platform, has been instrumental in enabling Kroger to automate and productize their data products at scale. Unity Catalog provides a unified governance layer across the data mesh, enabling consistent security, governance, and discoverability of data products.
With Unity Catalog, the data mesh domain owners can define and share data contracts, specifying the structure, quality, and SLAs of their data products. This allows the domains to build data products once and share them consistently across the mesh, without consumers needing to worry about the underlying infrastructure.
With Unity Catalog, "We start to see this vision, not just the behavior, but the realization of these things started to come to life," Sylvester says. By automating many of the governance and sharing processes, Unity Catalog helps Kroger scale their data mesh implementation and accelerate the delivery of valuable data products to the business.
Alation has also played a crucial role in Kroger's data transformation journey, serving as the enterprise data catalog to enable data discovery and governance at scale. With Alation, Kroger has been able to create a unified view of their data assets across the organization, making it easy for users to find, understand, and trust the data they need.
"Within Alation, we leverage the business glossary quite a bit to try to create a common language for the way we talk about things," says Sylvester. Alation's collaborative features, such as data dictionaries, user annotations, and data lineage, have helped foster a shared understanding of data across business and technical teams.
This has been particularly valuable in driving adoption of the data mesh, as users can easily discover and access data products created by different domains. Alation's integration with Unity Catalog has further streamlined data governance, enabling automated data profiling, classification, and access control.
By providing a user-friendly interface for data discovery and governance, Alation has accelerated Kroger's ability to democratize data across the enterprise while ensuring data security and compliance.
Conclusion
Kroger's data transformation journey has yielded valuable lessons that can benefit other organizations embarking on similar initiatives. One key takeaway is the importance of starting with culture and behaviors, rather than just focusing on technology rollouts.
We set out to understand what our hurdles were. We set out to change our perspective. It wasn't just about bringing technology. It was really about how can we change the way that our culture organizes and thinks about the data.
By defining the target data culture and driving the right behaviors first, Kroger laid a solid foundation for the subsequent technology and architectural changes.
Another crucial lesson is the need for clear data ownership and accountability within a federated organizational structure. Kroger's approach of organizing around data domains, with each domain taking end-to-end ownership of their data products, has been instrumental in driving data quality, trust, and accessibility. "Data mesh is really about how we organize, how we create decentralized teams within the business units," Sylvester reveals.
As Kroger looks ahead, the next horizon lies in enabling true data democratization across the enterprise. With the foundational elements of the data mesh architecture and supporting technologies like Databricks' Lakehouse platform, Unity Catalog, and the Alation Data Intelligence platform in place, Kroger is well-positioned to scale data as a product.
"Tech has set the foundation to view 'Data as a Product'," Sylvester states. Going forward, the focus will be on empowering more teams and individuals to easily discover, access, and leverage data for insights and decision-making. This will require continued efforts in data literacy, self-service capabilities, and fostering a community mindset where teams balance their own priorities with the greater good of the organization.
Sylvester emphasizes the importance of deepening relationships and partnerships across the enterprise when he says, "[We want to] Deepen our relationships and partnerships across the Enterprise. Identify critical business partners to closely work with Technology."
By staying committed to the core principles of ownership, communication, and collaboration, Kroger can unlock the full potential of its data assets and drive competitive advantage in the dynamic retail landscape.
Are you curious about your organization's data culture maturity?
Take the first step in your data transformation journey with our free 15-minute quiz. This assessment will provide valuable insights into your current data culture and offer personalized recommendations on progressing from one maturity level to the next.
Don't miss this opportunity to learn more about enhancing your data governance practices and empowering your teams with the knowledge they need to drive meaningful change.
- Key Insights
- Introduction
- Starting with Culture, Not Technology
- Defining Guiding Principles for the Data Culture
- Implementing the Data Mesh
- Driving Adoption through Discoverability and Governance
- Leveraging Technology as an Accelerator
- Conclusion
