Five Smart Questions Every Fund COO Should Ask Service Providers About AI

Chief Strategic Solutions Officer Ravi Gupta shared his thoughts on AI integration and fund administration in the AIMA Journal Q3 Edition. Check out the abridged feature article in the AIMA Journal or, for a deeper dive, read the full unedited article below.

By Ravi Gupta, Chief Strategic Solutions Officer, NAV Fund Services

There's a strong likelihood that everyone you run into these days is 'AI-enabled' — your fund administrator, your prime broker, and three vendors at the last conference you went to. At this point, the phrase tells you almost nothing about how AI is actually implemented, or how you might benefit.

'AI-enabled' is becoming code for 'human-disabled.' At a recent tradeshow, a fund administrator approached me to tell me they'd automated their entire NAV calculation process via AI. But as soon as I started asking more pointed questions, like, "How does your team you handle edge cases?" I realized they had no idea how the underlying logic worked. My guess is that they just brought in AI to populate their marketing messages with the latest tech buzzword.

Unvalidated 'AI-enabled' claims introduce invisible risks — cognitive debt, oversight effects, compounding errors — which you may unknowingly absorb as a client. Here are five questions that I recommend you ask a vendor to figure out if they have adopted AI thoughtfully or if they're just jumping on the AI bandwagon.

1. Can you explain your validation process?

It's easy for a vendor to say "our models are highly accurate," but compared to what?

A strong validation protocol runs AI systems parallel to existing human workflows before deployment. You add the AI, keep the humans, compare the outputs and investigate the mismatch. Over multiple iterations, human team members essentially teach the model where to look for specific data, creating a continuous improvement loop versus a one-time correction. This process should take months, not weeks.

Providers also need a validation process specifically designed to catch hallucinations. A University of California, San Diego, study found that LLMs hallucinate 60% of the time when summarizing product reviews, but users were still overwhelmingly likely to choose an AI-generated summary over a human generated one. We are psychologically predisposed to trust AI, which makes catching hallucinations a design problem.

Your vendors should be able to walk you through exactly how their models are tested, including their validation process, what specific mechanisms exist to catch hallucinations, and what their confidence intervals look like.

2. When something goes wrong, who is accountable?

Humans can explain their reasoning, own their mistakes, and course correct. AI cannot. In a fiduciary context, there's got to be human accountability for mistakes.

Each AI function should have a clear owner and an escalation process. But it's also important to ask how the humans are holding up. Harvard Business Review studied the impact of AI usage on mental fatigue and burnout. It found that if AI requires a lot of oversight, then total mental effort is increased, not reduced. The more AI output demands review and supervision, the more stress it causes — leading to higher decision fatigue, poorer decision making, and increased employee turnover.

Ironically, the study also suggests that humans reviewing AI output start to "hallucinate" themselves — accepting errors they'd normally catch because the volume and velocity of AI output overwhelms their ability to evaluate it. The oversight layer you think is protecting you might itself be degrading.

What you don't want is a vendor who points to the model itself for accountability or describes an extensive oversight structure that sounds like a burnout factory. AI should take over repetitive tasks to support staff, not create a new category of exhausting work.

3. What triggers human review?

Journalist Cory Doctorow introduced the idea of the "reverse centaur" in a recent Medium column. For people who choose when and how to use it, AI can become a valuable tool — strong and tireless, directed by human judgement. Doctrow calls these users 'centaurs.' Reverse centaurs are the opposite: humans trying to manage heads of endless AI output with fallible human attention.

Service providers should be building centaurs, not reverse centaurs. They should be able to clearly explain when human review is necessary and deploy AI in scenarios that don’t require human review for every output. Asking your provider how a human review is triggered ensures the system doesn’t collapse under its own weight.

4. Where specifically is AI not being used, and how is our data protected?

Where AI is not being used is just as important as where it is. Modern generative AI, at its core, is a turbocharged auto-complete machine. It is extraordinary at pattern recognition but not at reasoning. When something falls outside the training data, rather than reason its way to an answer, it often hallucinates one.

Certain calculations in fund administration are deterministic. The formulas for fee computations, allocations, and P&L work, for example, have always worked and are easy to write, understand, and debug. If your vendor is automating these with AI, ask why. They are probably introducing risks into a process that previously had none.

AI shouldn't have a role in your vendor's decision-making. It should be focused on repetitive and routine tasks, not creative work or anything that requires judgement. MIT's Media Lab ran brain scans on students using ChatGPT to write essays and found that participants who used an LLM to write had significantly poorer recall of what they had just produced. If we rely too much on AI for critical outputs, we might lose the confidence to even understand what was produced. Critical thinking remains a key competitive edge — it shouldn't be outsourced.

Data privacy deserves equal scrutiny. Can AI access your investor data, especially in third-party systems? Many third-party models retain data for training purposes. A vendor should conduct rigorous data privacy and security audits and obtain enterprise-grade licenses when available. Make sure client data isn't used as part of model training that can be accessed by other third parties. If your service provider is operating AI systems but cannot explain how the underlying architecture actually handles your data, you might be handing sensitive information to a black box.

5. How do you measure success, and what's your baseline?

MIT NANDA recently published a study that rattled Wall Street, showing that of the 80% of organizations that have implemented AI programs, 95% have achieved zero return. Despite the billions invested into GenAI, a meaningful return remains elusive.

Take AI productivity gains with a pinch of salt. A recent METR study found that developers using AI actually built new features and resolved bug fixes at a slower pace with AI, but they thought they were working faster than before. If your vendor is measuring success by how users "feel" about the tools rather than what the data shows, you may have a problem.

Individual productivity doesn't always translate to organizational productivity either. I may be able to write an email in two minutes instead of four with AI, but because it's verbose, the recipient needs twice as long to read it. AI can also create unnecessary work, especially if you don't understand the topic. When I asked AI the best way to heat and cool my house, it suggested building a geothermal plant! By the time I invalidated all of its crazy suggestions, I spent more time than I would have if I had just done the research myself.

Don't accept arbitrary metrics like lines of code generated. Larger amounts of generated code can be a warning sign of AI debt — those small AI errors that build up over time. When GenAI is used to code, it produces far more lines of code than a human would. That much code is harder to review, so mistakes slip through the cracks and compound. Recently, Amazon's retail website suffered a six-hour outage due to AI errors in coding workflows. In response, the company implemented senior engineer reviews and introduced additional human oversight. Imagine if this happened with your fund administrator, and compounding errors impacted multiple NAV reporting cycles.

Ask the vendor how AI saves time, in end-to-end processes versus at the individual level. A vendor's outcome metrics should be tied to client-facing results and measured against pre-AI benchmarks.

Final Thoughts

None of this is to say that AI is useless. It's an impressive technology that works quite well in specific and bounded applications: reading documents, extracting data, managing operations, and reducing repetitive tasks. The companies that are getting it right are the ones that have matched the tool to the problem with clear acceptance criteria and measurable outcomes.

Those that don't quite get it are the ones chasing mandates and measuring success by perception and hoping no one asks the hard questions.

AI gives you the output, but it's your people who give you accountability. Before you sign off on any vendor's AI pitch, make sure that they can tell you who is standing behind that work — not which model is running it.

- - - - - - - - - - - - - - -

Ravi Gupta is Chief Strategic Solutions Officer at NAV Fund Services, where he leads product development strategy and manages the AI implementation, software development, and product teams. He's also instrumental in analyzing, road mapping, and prioritizing development of new services and features for the firm's innovative technology platform. He joined the firm in 2016 as an account manager. He holds a Bachelor of Science in Mathematics, Economics, and Philosophy from the University of Chicago and is a Chartered Financial Analyst (CFA).