Your AI Agent Does Not Know What Your Variables Mean

Jul 28

AI is making analytics reporting and analysis much faster and more efficient.

Instead of manually building reports, users can increasingly ask questions in natural language, request visualizations, compare segments, identify trends, and receive summarized answers. Connections such as Model Context Protocol may also allow your own AI tools to access the data within your analytics platform.

However, in order for the reporting and analysis by an AI agent to be accurate and meaningful, your analytics implementation needs to be ready.

The Practical Takeaways

The big one: Garbage In, Garbage Out applies here - if the semantic descriptions of your analytics data aren’t complete and accurate, AI won’t be able to give you good answers to your questions, and may in fact give you wrong answers
A semantic layer must include some level of business context.
Ongoing upkeep and governance of the semantic layer is critical.
The objective of a good semantic layer is to reduce ambiguity so that people and AI systems are more likely to interpret the same data in the same way.
Use of AI exposes weaknesses in the digital measurement foundation.

First, what does a semantic layer consist of?

A semantic layer is not a single document or platform feature. It is the combined set of definitions and metadata that explains what analytics data means and how it should be used.

That can include variable descriptions in a solution design document; approved KPI definitions; expected and acceptable values for variables; CJA Data Dictionary entries, component names, descriptions, and context labels; attribution, persistence, and calculation logic; and an indication of which metrics, dimensions, segments, and other components are authoritative. Together, these give both people and AI the context needed to interpret the data correctly.

The AI may be able to access your data without understanding what the data means to your business

An analytics system can identify a variable, metric, schema field, segment, or calculated component. It can inspect its values and use it in an analysis. But unless the relevant business meaning has been clearly defined and documented, the system may not know whether it selected the right component - or whether the resulting answer should be trusted.

Technical access is not business understanding

Consider a simple request:

How many qualified leads did we generate last quarter?

The analytics environment may contain several plausible measures of a qualified lead, including completed forms, marketing- or sales-qualified leads, unique submitters, or submissions excluding existing customers.

A person familiar with the organization may know which definition is authoritative. An AI tool may see several plausible choices.

The problem becomes harder when the available components have similar names, incomplete descriptions, or undocumented differences.

The AI can still produce an answer. It may even explain the result clearly and confidently.

But confidence is not evidence that it selected the correct definition.

Most analytics environments contain hidden context

Analytics implementations accumulate years of business and technical decisions.

A variable may have been created for one campaign and later reused for another purpose. A metric may have changed after a redesign. A schema field may technically contain data but not be considered reliable for reporting. Two calculated metrics may appear nearly identical while handling exclusions differently.

Much of the context needed to distinguish among them may exist only in an analyst’s memory, outdated documentation, implementation tickets, tag-manager comments, or informal conversations.

People who have worked with the implementation for years may navigate these ambiguities without consciously thinking about them.

AI does not inherit that institutional knowledge automatically.

A variable name is not a semantic layer

Technical metadata is useful, but it is not always enough.

A variable name rarely explains its business purpose, collection method, processing rules, valid values, limitations, current status, or appropriate use.

Even a more descriptive name can be misleading. “Revenue” may represent gross revenue, net revenue, booked revenue, recognized revenue, or revenue excluding returns. “Customer” may mean an account, an authenticated user, a purchaser, or a person with an active subscription.

A useful semantic layer connects technical structures to approved business meaning.

Solution design becomes AI infrastructure

Solution design and documentation have traditionally supported implementation quality, maintenance, troubleshooting, onboarding, and governance.

They now serve another purpose: providing the context required for reliable AI-assisted analysis.

Strong documentation should connect:

1. The business question

What decision or objective does the organization need to support?

2. The business definition

How are the relevant terms, metrics, and outcomes defined?

3. The collection design

Which variables, events, schema fields, and data elements capture the necessary information?

4. The processing logic

What attribution, persistence, transformation, filtering, or exclusion rules apply?

5. The reporting components

Which metrics, segments, derived fields, and data views are approved for use?

6. The limitations

Where is the data incomplete, inconsistent, delayed, or inappropriate for certain analyses?

Without that chain, an AI tool may see the final component without understanding the decisions behind it.

Governance matters as much as documentation

Semantic readiness is not achieved simply by creating a larger data dictionary.

Organizations also need to identify which definitions and components are authoritative.

An analytics environment may contain hundreds or thousands of metrics, segments, filters, audiences, and calculated components. Some may be current and governed. Others may be duplicates, experiments, personal versions, or remnants of earlier implementations.

Before AI is given broad access, organizations should be able to answer:

Which metrics represent approved business definitions?
Who owns those definitions?
Which components are current?
Which components should be deprecated?
How are changes reviewed and communicated?
What may an AI tool create or modify?
Which actions require human approval?
How will generated answers be validated?

AI can make analytics more accessible. It can also make inconsistency more scalable.

What semantic readiness looks like

An AI-ready analytics environment does not need perfect documentation of every object ever created.

It does need reliable meaning around the data that matters most.

Semantic readiness depends on clear business definitions, current solution design, understandable metadata, documented collection and processing logic, traceability from requirements to reporting, governed components, defined ownership, and clear distinctions between authoritative and legacy assets

The objective is not documentation for its own sake.

The objective is to reduce ambiguity so that people and AI systems are more likely to interpret the same data in the same way.

AI will expose weak analytics foundations

Organizations have been able to tolerate documentation and governance gaps because experienced analysts often compensate for them.

They know which metric to use. They recognize the misleading segment. They remember that a field changed meaning two years ago. They understand which dashboard Finance accepts and which one Marketing prefers.

As AI expands access to analytics, more users will be able to ask questions without relying on those experienced intermediaries.

That increases the value of self-service analytics, but it also removes some of the informal safeguards that knowledgeable analysts provide.

The better the interface becomes, the easier it may be to generate a polished answer from the wrong inputs.

Prepare the meaning before scaling the access

AI can accelerate analysis. It cannot independently resolve every ambiguity created by years of inconsistent definitions, incomplete documentation, and unmanaged components.

Before connecting AI to Adobe Analytics, Customer Journey Analytics, Adobe Experience Platform, or another analytics environment, organizations should assess whether the data includes enough reliable business context to support trustworthy answers.

That requires more than data availability. It requires clear definitions, current solution design and metadata, documented lineage, governed components, defined ownership and permissions, and reliable validation processes.

AI does not make analytics fundamentals obsolete.

It makes weaknesses in those fundamentals more consequential.

Build a foundation AI can understand

Heavey Digital Consulting’s Semantic Readiness Assessment evaluates whether your analytics environment contains the definitions, documentation, metadata, and governance needed to support reliable AI-assisted analysis.

The assessment identifies gaps, conflicting definitions, high-risk components, and documentation weaknesses, then provides a prioritized roadmap for strengthening the foundation.

Before asking AI to interpret your analytics, make sure your analytics environment gives it enough meaning to produce an answer you can trust.

Patrick Heavey