Unstructured data with the modern data stack

Most of the world's data is unstructured, and humans are much more adept at processing this type of information than machines, but we can't do it at scale. The advent of the AI era is changing this dichotomy as machines are getting much better at learning how to process unstructured data. Since the dawn of the digital age, machines have been more capable of managing structured data, but with ML, revolutionary LLM models, and Generative AI, unstructured data will have a much more significant role in how humans and machines work together to understand the world.

Enterprises have become very good at capturing and storing unstructured data. According to Gartner, 80%-90% of enterprise data is unstructured. The amount of unstructured data is also growing considerably quicker than structured data. Generating business value from this data is an emerging opportunity.

Structured vs Unstructured Data

Structured data is data that is well organized and defined. Typically, it is organized in columns and rows with schema that defines the meaning of each. It is also usually very qualitative and straightforward to analyze.

Unstructured data is more like the data that we engage with every day. It is unorganized, much more qualitative and usually stored in its native format. Examples of unstructured data include,

• Text messages
• Post on Social Media
• Images
• PDF Documents

Semi-structured data is unstructured data with some structure or tags added to it, making it easier to organize and analyze. This data has some structure but does not follow the same structure as a traditional relational database. Flat CSV files, files created using markup languages such as XML or HTML, and JSON files are common examples of semi-structured data.

Unstructured Data Challenges

Unstructured data cannot be searched, filtered, sorted, or otherwise manipulated. It is also hard to find and access unstructured data. This makes it hard to use for valuable decision-making at scale.

Connected digital devices operating worldwide are creating a never-ending flow of unstructured data, which is growing exponentially. Data such as text messages, social media posts, sensor data, and log files contribute to the 328 million terabytes of data created every day. Ritcher unstructured data such as PDFs, audio, and video files are also adding to the deluge of unstructured data that could be analyzed to support better decision-making and better-performing models.

Enterprises are saving more and more of their unstructured data due to dropping storage costs, resulting in a much larger pool of available data. Still, the sheer volume of this data makes finding value much harder. These challenges leave valuable data unused, and opportunities to improve business performance are missed.

Value of Unstructured Data

The ways leaders can generate value from unstructured data to improve operations are infinite and unquantifiable. Unstructured data can provide valuable insights into customer behavior and market trends, for example. Analyzing social media posts created by specific customer segments can provide marketers insights into how they see their brand or what topics customers are interested in. This type of analysis can help product managers spot trends early and identify opportunities for new products.

Sophisticated analysis of external communications can measure how customers are feeling. Sentiment analysis can measure whether a customer is having a positive or negative experience with your company by analyzing emails or engagement with customer service agents.

These techniques can also track sentiment in internal emails and communications to understand employees' mindsets. This information can help prevent burnout and drops in morale and productivity. Managers can give their teams breaks when sentiment analysis detects a negative trend. When employees feel their employers care about them and understand when they need a break stronger corporate culture will emerge that drives growth.

Quickly analyzing a variety of communications can also help to identify fraud. By analyzing social media posts, emails, and customer service call transcripts, sophisticated models can identify fraudulent data. AI analysis of this data can spot inconsistencies across communications that can flag fabrications.

Computers' ability to analyze documents can provide significant productivity gains. By analyzing a database of legal documents, organizations can efficiently measure their exposure to litigation. Storing, retrieving, and analyzing financial data from regulatory filing can also help save financial analysts many hours of work.

Processing business documents from legacy systems can also be streamlined using unstructured data processing. While technology is always moving forward, not all companies keep up, but more advanced firms still need to work with them. Systems that can process and store document-based maintenance records, invoices, or other important paperwork can increase productivity and analyze trends.

Solutions

The key to managing and processing unstructured data is to build structures around it to transform it into semi-structured data. Tagging strategies are evolving to make unstructured data more discoverable and manageable. Efficiently searching the vast amounts of unstructured data in the world in its raw form is still evolving, but searching metadata or data about the data is much more established.

With a strong metadata strategy and management platform, you can find and access unstructured data using SQL queries. SQL scripts can access data by referencing basic metadata such as Document ID, Time Stamp, Authors, and document category. This is helpful, but it does not tell you much about the content of unstructured data or what it means. To extract more insights from the content of your unstructured data, you need to enrich your metadata. Data tagging is one way to do this.

Data can be tagged either manually, or automated processes can be created to label data. Pure manual approaches are much more error-prone, slower, and do not scale well. Typically, a data steward will head up a manual tagging process to establish and maintain a set of data tagging standards, putting a tremendous burden on an already challenging position.

Manual tagging limitations are creating opportunities to streamline the process with AI-assisted tagging. Tags are approved manually with this approach, but an AI assistant will suggest how data should be tagged or classified, making the job much less time-consuming. An example would be an AI bot recognizing a social security number or address while a data steward classifies data, and the bot suggests that this data should be classified as sensitive information.

Automating data tagging

Automating more of your data tagging processes requires more sophisticated ML techniques. Multiple approaches have emerged in the marketplace as more advanced AI technology has evolved. These techniques help machines understand the content of unstructured data so it can be accessed and analyzed. These approaches are based on foundational technology such as optical character recognition (OCR), natural language processing (NLP), and supervised and unsupervised learning.

Optical Character Recognition

OCR technology recognizes characters within a document or image, enabling machines to identify letters or words in typed documents, PDFs, images, or handwritten documents. This technology is mature but provides the foundation for machines' ability to understand human language. Once machines can identify characters, they can turn this text into meaning so content can be tagged correctly. Natural Language Processing techniques can then be used to extract meaning from unstructured data.

Natural Language Processing

NLP models are based on AI technology that can process human language. Machine learning and computational linguistics enable machines to comprehend our communications so documents, audio files, and other communications can be tagged and organized. Over the years, natural language processing has evolved, incorporating increasingly more sophisticated ML and AI techniques. Simple frameworks have evolved into deep learning unsupervised AI models that are capable of understanding the meaning of unstructured data.

Computational linguistics is at the heart of NLP technology because it provides the framework for computers to understand human language. Syntactic analysis, which helps machines understand meaning based on how words are arranged, is one example. Sentiment analysis, which helps computers understand the tone of human language, is another. These technologies are relatively mature and provide the foundation for more sophisticated deep-learning models that can capture more meaning from unstructured data.

Supervised learning

Named Entity Recognition (NER) is a central task in training NLP models. The process involves identifying predefined entities in text and classifying them into a specific category. Medical terms, names, organizations, or locations are common categories. To train the model, humans will create particular categories and rules around classifying different entities.

Text Classification is where text is assigned a particular predefined category. Certain words could be categorized as positive or negative, for example. In a support ticket use case, words in a customer communication could be classified as either feedback, complaint, or question, providing more information about the nature of the interaction. Content can be categorized using machine learning models, human-defined rules, or a combination of both. With a rules-based approach, rules define how text is classified. For example, logic that defines the frequency of keywords used in a document will dictate how it is classified. An ML-based approach uses machine learning models to recognize patterns in the text and automatically classify the content. Combining both techniques can lead to even more precise tagging, and AI can eventually learn to label text without help.

Unsupervised learning techniques and Vectors

AI learning techniques have emerged that can understand the meaning of text without the help of a human. Technology is also coming to market that can turn this meaning into numbers so it can be searched by traditional data query tools used to analyze structured data.

Topic modeling is another NLP technique where an unsupervised AI model can identify a group or cluster of words in a body of text. The model can learn that certain words are common in particular types of documents. One example of topic modeling is identifying words that are common to a contract or invoice and labeling them accordingly.

Dependency graphs will identify relationships between words that enable AI models to better understand the meaning of text. This includes grammatical relationships between words in a sentence how a verb relates to a noun, for example. These types of associations in language provide the foundation for vector analysis, where relationships between words can be expressed as vectors.

Vectors make it all work.

Vector embedding is a technique that converts words, sentences, and other unstructured data into numbers that can be understood by machine learning models and query engines. This allows ML to analyze text and classify content appropriately.

Embedding vectors in databases also allows analysts to create complex SQL queries to pull documents, text, or data based on their meaning and context. This can enable powerful, complex queries that pull data from both structured and unstructured sources. It also enables semantic searching.

Searching your vector data across all your unstructured data stores can be cumbersome and inefficient. Well-organized metadata can support semantic searching by narrowing down the volume of data it needs to search. Metadata can filter data to reduce the resources required to search for assets.

A robust metadata management strategy can optimize the process of finding meaning in unstructured data. Centralizing metadata management allows unstructured and structured data to be accessed from the same place. This metadata can also support central data catalogs where analysts can more easily find structured and unstructured data.

Data Products

Once unstructured data is labeled or embedded vectors are created, data can be accessed using SQL queries, and datasets can be merged and enriched to add more business value. The data product is an excellent way to package structured and unstructured data to make it more beneficial to business leaders and analysts.

Data products can be created to merge rich structured data with more contextual unstructured data to provide deeper insights. For example, structured financial market data and portfolio data can be merged with unstructured content like news, financial statements, and social media sentiment. This data can then be fed into a model that can analyze the drivers behind portfolio value fluctuations.

Structure and unstructured data can also be utilized to predict human behavior. Data products can be built that combine sales data with sentiment analysis across social media platforms to understand how chatter on social platforms focused on your brand may be affecting sales.

In healthcare settings, structured test data can be combined with doctors' notes to provide greater context. This type of solution also enables a much larger number of cases to be analyzed to identify connections, correlations, and trends.

Insurance adjusters work with substantial amounts of valuable unstructured data that is hard to access and analyze at scale. Data products can be developed to combine unstructured data and structured data to support more accurate predictions leading to better risk assessments. For example, combining adjusters field reports and notes with structure data such as claim amounts, accident locations, and type of vehicle can be used to help identify trends and patterns the can support better risk assessment.

Working with unstructured data and unsupervised AI is tricky and can result in hallucinations or bad results. Data products incorporate data governance and human supervision to provide greater oversight. Data product producers can evaluate data lineage to better understand the underlying NLP models and data product consumers can provide feedback on the quality of the outcomes from analysis based on these sophisticated data models.

Machines will continue to get better at understanding unstructured data, leading to new use cases and business opportunities. Monitoring unsupervised learning models will be required to reduce the risk that AI will make costly mistakes.