Unifying the Data Model and Business Glossary

The growing adoption of data democratization is creating new frameworks and technology for sharing data across data silos. These strategies are reducing the friction of data sharing between business domains, and access to data is becoming effortless. One of the central challenges with integrating data is working with disparate data models that describe diverse databases and data sets in unique ways.

The traditional approach to merging data sets was to extract one data set from its database, transform it, and load it into another database to match the data structure of that database. For performing the ETL process, data engineers are required to understand the technical aspects of moving and transforming data along with organizing and labeling each data set. Their understanding of how the two data sets are modeled is so important to ensure they can be mapped together into one.

Modern data virtualization technology provides greater access to disparate data sources by abstracting away data from its underlying data structure, simplifying the process, and eliminating the need for ETL. While this technology is powerful, it does not provide a uniform way to access data.

Data virtualization provides a single interface or connectivity layer that enables access to distributed data from one place. But to understand what the data means, analysts must still rely on each separate data model for each database to gain context. For effective analysis, we need to understand what the data in each system represents and how they relate to one another. These insights require an effective data federation strategy that standardizes how we access different data stores. A unified data model that maps data and relationships across data silos is a crucial component. For even easier access, a business glossary that maps these relationships to business terms can make this data model even more valuable by increasing its accessibility to business leaders and decision-makers.

The Federated Data Model

A federated data model is based on metadata extracted from the connected source systems and merged into a uniform logical data structure. When data is organized around a single data model, data platforms can interact with all your heterogeneous databases as if they were one. Using this approach, you can pull data from multiple systems with one federated query. This capability saves a substantial amount of time for data engineers and skilled analysts when integrating data and creating data assets and data products.

Abstracting the logic from the physical layer also makes self-serve data analytics easier as tools are less complex and don’t need to interact with multiple underlying database structures.

Federated Data Catalog

In a federated data strategy, metadata is used to create a global or federated data catalog to access the data. This data catalog leverages the central metadata repository to create a searchable inventory of data assets for analysts to build the federated data queries.

A federated data catalog enables searches across all your data assets. It can also consolidate lineage so users and data stewards can understand how data was changed in the past.

A federated data strategy can also manage who has access to what data. Instead of managing access at each database individually or applying uniform rules to all databases, a federated data catalog can act as a security gateway to manage identity in one place. Also, it supports authorized access to all the data assets.

With a standardized data catalog, creating self-serve capabilities is much less complex. Self-serve platforms can automate the process of accessing data but with more uniform terminology. As business users know what data they are looking for, making them much more self-sufficient. A simpler model also helps AI better understand how to access data. A consolidated and standardized set of data semantics that uniformly defines data elements makes it easier for an LLM model to translate data requests into an SQL query, using business terminology.

Unified Business Glossary

While a federated data model is great for creating single data queries across data sources, these models are not typically geared to business users. Business glossaries are particularly important when federating data across domains and regions as business terms are sometimes defined differently in each business domain. Terminology also differs across regions.

For example, “Turnover” in the UK vs. “revenues” in the US. Both terms mean the same thing in the data model, but each region uses a different lexicon. A detailed business glossary that precisely defines business terms and their synonyms makes it easier to find the data and understand its meaning, especially for business-oriented decision-makers.

In the past, business glossaries have existed in standalone documents that define each term. Today, business glossaries are connected to data dictionaries and data catalogs making it automatically accessible for users to get the data by just using business terms. This improvement enables business users to access data freely in the organization with just an understanding of the business terms that describe the data they seek.

Modern business glossaries include:

Centralized repository

This capability creates a single source of truth for business terms, definitions, and associated metadata.

Structured taxonomy

This functionality organizes business terms into structured taxonomies or hierarchies. Hierarchical categorization allows users to explore related terms and concepts, promoting a deeper understanding of the organization's domain.

Auto-Assignment

Sometimes, business terms from glossaries can be auto-assigned to data assets, linking technical metadata with relevant business context. This auto-assignment process helps normalize technical metadata by adding business essence to each data asset, enhancing its relevance and usability.

Normalization of technical metadata

This capability connects business terms with technical metadata. The business glossary helps to standardize terminology across data sets. Normalizing technical metadata ensures consistency in data descriptions, making it easier for users to interpret and analyze information.

Standardizing a Business Glossary

The business glossary should be built from the top down, aligning with the requirements of the business. An excellent way to create your business glossary is by using the existing standard industry terminology. This approach will give you a solid foundation and facilitate improved data sharing with third parties. You can also utilize a hierarchical taxonomy structure to build your business glossary which will help to organize and classify the data more effectively.

Data Interpretation Conflict Management

With each domain having its own business glossaries and logical models, conflicts can arise around how different business groups interpret terms and data, when merged. Having a resource to manage these disagreements is an integral part of a well-functioning universal data glossary.

Data stewards can also be helpful in tagging data assets to designate their value or flag data quality issues. While data stewards can take the lead in data classification, correctly classifying data to be more accessible and discoverable is everyone's responsibility when interacting with data assets. AI can help support this process across the organization. AI can learn from existing data models and suggest classification designations if conflict or uncertainty does arise.

A unified data model and business glossary can be a massive asset in aligning the business data and the business itself. As different domains think about data more uniformly and communicate more consistently, decision-making can be more collaborative and efficient as business terminology and metrics are standardized.

Application of AI for Intelligent Data Catalog

AI will be increasingly important in facilitating efficient data catalogs and business glossaries. As AI models become more effective, they will gain a better understanding of the data assets across your organization. With AI’s assistance, analysts will have a copilot to help them find the exact data set that enables them to get the answers they need.

Unifying data access and abstracting metadata from the actual data enables greater agility in data utilization. A unified data catalog makes finding and accessing data much faster and more efficient. Business questions can be answered quicker and more effectively with this capability. The faster organizations can make quality decisions, the more competitive they will be in the market.

The increasing demand for data creates an environment where replicating data wherever it is needed through ETL pipelines is unsustainable. A model that consolidates information on where data is stored and how to access it is much more scalable. Federated data strategies that manage metadata and the context around data provide the flexibility and agility needed for the future.