I have spent 11 years in the trenches of technical SEO and analytics. In that time, I’ve worked with household names—think brands the size of Coca-Cola—and agile startups. If there is one thing I have learned, it is that most marketing reports are designed to keep you from asking the real questions. Agencies love vanity KPIs because they are safe, predictable, and—most importantly—easy to manipulate.
If your monthly report is filled with "Year-over-Year Impressions" or "Keyword Ranking Fluctuations," stop reading this, open your agency’s dashboard, and look at the actual business impact. If you can’t find it, you’re paying for a performance that doesn’t exist. Before you believe a single slide, ask yourself: Does this metric help the business pay its bills, or does it just keep the account manager from getting fired?
The Anatomy of a Vanity KPI
Vanity KPIs are the "fluff" metrics of the marketing world. They give the illusion of success without the substance of revenue. Agencies push them because they provide a constant, upward-trending narrative that hides stagnant performance.
Common vanity KPIs I consistently see on my "do not trust" list:
- Keyword Rankings: In an era of personalized search and AI-generated answers, a "position" is a relic of the past. Ranking #1 for a term that converts 0% of visitors is a failure, not a success. Impressions: If your ad or organic result is shown to 10,000 people who have zero intent to buy, those impressions are just digital noise. Traffic Volume (without segmentation): Traffic is a cost center if it isn’t qualified.
When an agency presents these, they are usually trying to distract you from the fact that your revenue metrics—the only ones that actually matter—are flatlining. If you want to know if your agency is performing, don’t look at their slide deck; ask for their raw data source or the dashboard link. If they refuse, cut the cord.
Search is Shifting: From Blue Links to AI Answers
The industry has been slow to pivot, but the game has fundamentally changed. We are no longer operating in the era of "ten blue links." We are operating in the era of Answer Engine Optimization (AEO). Users are asking LLMs and AI-integrated search engines for solutions, and if your brand isn’t being cited as the authority within those models, you don't exist.
Agencies that focus purely on traditional SEO are chasing algorithms from 2018. True modern visibility requires managing your entity signals so that when a user asks an AI, "What is the best [product category]?", your brand is the entity the model trusts. This is where organizations like AEO FD are leading the conversation—moving away from the guesswork of "SEO" and toward the precision of "AEO."
The Comparison: Vanity vs. Revenue Metrics
Vanity Metric Revenue/Value Metric Keyword Ranking Position Share of Voice in AI/LLM Citations Total Pageviews Qualified Leads/Conversion Rate Domain Authority Score Entity Sentiment & Trust Signals Average Session Duration Pipeline Contribution (MQL to SQL conversion)AEO as Measurement-First, Not Guesswork
If your agency claims they can’t track AI visibility, they are lying. They just aren’t using the right tools. We are currently witnessing a transition where brands are moving to Four Dots-style approaches to entity management—where every content piece is audited not for "keyword density," but for its ability to feed factual, verifiable data to AI models.
AEO is not a "black-box" mystery. It is a technical discipline. When we talk about measuring AI visibility, we are talking about tracking how often your entity appears in conversational search outputs and verifying the accuracy of those answers. This requires a pipeline that can handle massive amounts of unstructured data.
The Technical Stack: FAII-node and FAII.ai
To move beyond vanity KPIs, you need a data stack that can handle the complexity of the aeo.is new search landscape. I’ve spent years building pipelines, and the current gold standard involves leveraging tools like FAII-node and FAII.ai.
Why do these tools matter? Because they provide multi-model verification. You cannot rely on one AI model to tell you how "visible" you are. If you only check ChatGPT, you’re missing the signal from Perplexity, Gemini, and Claude. FAII.ai allows you to verify your entity signals across these models, ensuring that the information being fed to users is accurate and consistent.
This is "measurement-first" marketing. You aren't guessing if your content is working; you are looking at the verification logs to see where your entity appears in the answer stream. If the models are hallucinating about your brand, you fix the source data. That is real work. That is not an agency "ranking" a keyword for you.
What You Should Ask for Instead
The next time you are in a quarterly business review, stop the presentation and ask these four questions. Watch how they react.
"Can you show me our AI visibility score across three different LLM providers?" If they say "we don't track that yet," they aren't equipped for the current market. "How is our entity signal consistency affecting our conversion rate?" If they can't link content strategy to entity trust, they are just churning out blog posts for the sake of it. "What is the multi-model verification status for our top-tier keywords?" If they aren't checking for hallucinations in the search results where your brand appears, they are ignoring your reputation. "Can I see the daily change in our AI share-of-voice?" If they only provide monthly snapshots, they are hiding volatility.Daily Visibility Tracking: The New Standard
Vanity KPIs like "Monthly Rankings" allow agencies to smooth over bad weeks with "average" data. That is unacceptable. In the world of AI, information changes in real-time. You need daily monitoring of your visibility. If an AI model suddenly decides your competitor is the "best" in your category, you need to know within 24 hours, not 30 days.
This is what I tell my clients: If you aren't monitoring daily, you're just auditing a graveyard.
Conclusion: Demand Accountability
The digital marketing industry is littered with agencies that sell "packages." These packages are designed to lock you into a contract where the work is generic, the metrics are vanity-based, and the results are non-existent. They rely on you not knowing enough about the technical side of AEO or entity management to challenge them.
It’s time to stop the algorithm-chasing talk. It’s time to move away from black-box reporting that makes you feel good about "impressions" while your competitors capture the AI answer share.
Ask for the data. Ask for the verification methodology. Ask for the revenue impact. And if they can't give it to you, tell them you're taking your budget to a partner who actually understands that AI visibility is a technical requirement, not a marketing bonus.
In this industry, there are two types of people: those who chase rankings, and those who build the infrastructure to own the answers. Which one do you want to be?