Skeptical about how AI-derived recommendations align with corporate strategy?

As a senior executive or CIO, how can you assure yourself that Artificial intelligence (AI) or Machine Learning (ML)-derived recommendations are reasonable and flow logically from the project work that has been performed?

While you want to be supportive and encouraging of your team’s work, you don’t want to be misled inadvertently, and you want to confirm that the data science team hasn’t misled itself.

“The link, or links, between the AI/ML project goal and the corporate strategy, must be clear for the project team to be successful and for senior executives to evaluate the AI/ML-derived recommendations as useful,” says Ron Murch, retired Senior Instructor at the Haskayne School of Business at The University of Calgary. “If the corporate strategy doesn’t exist, is confusing or not clear, then it will be difficult to have confidence in the usefulness of the AI/ML-derived recommendations.”

Here are some high-level questions that you can ask the team about corporate strategy. They’re designed to raise everyone’s assurance that the AI/ML recommendations are sound and can be confidently implemented even though you and everyone else know you’re not an expert. Start by selecting one question that you’re most concerned about, and you’re most comfortable asking.

Corporate strategy

The goal of the AI/ML project must advance an aspect of the corporate strategy. The importance or relevance of the AI/ML-derived recommendations is highly dependent on the alignment between the project goal and the corporate strategy. Here are some related questions that will illuminate the alignment:

1. Please describe how you ensured that the model’s goal is aligned with our business goals or pressing issues.
2. Please describe how you ensured that the project goal advances an element of the published corporate strategy.
3. Did the project team find itself debating what the published corporate strategy actually meant and if there’s a link to this project?
4. Do the AI/ML-derived recommendations suggest that some revisions to the corporate strategy should be considered?
5. Could you elaborate on the risks, particularly public risk, that the organization will face if we implement your recommendations?
6. What is a straightforward way to communicate your recommendations and associated risks to other organizational leaders?

Evaluating answers

Here’s how to evaluate the answers that you’ll receive to these questions from your data science team:

1. If you receive blank stares, that means your question’s topic has not been addressed and needs more attention before the recommendations should be accepted. It will be necessary to add missing skills to the team or even replace the entire team.
2. If you receive a lengthy answer filled with a lot of data science jargon or techno-babble, the topic has not been sufficiently addressed, or worse, your team may be missing critical skills required to deliver confident recommendations. Your confidence in the recommendations should decrease or even evaporate.
3. If you receive a thoughtful answer that references uncertainties and risks associated with the recommendations, your confidence in the work should increase.
4. If you receive a response that describes potential unanticipated consequences, your confidence in the recommendations should increase.
5. If the answers you receive are supported by additional slides containing relevant numbers and charts, your confidence in the team should increase significantly.
6. If the project team acknowledges that your question’s topic should receive more attention, your confidence in the team should increase. It will likely be necessary to allocate more resources, such as external data science consultants, to address the deficiency.

For a summary discussion of the topics you should consider as you seek to assure yourself that AI/ML recommendations are sound, please read this article: Skeptical about AI-derived recommendations? Here are some tips to get you started.

What ideas can you contribute to help senior executives assure themselves that the AI/ML-derived recommendations are reasonable and flow logically from the project work performed? Let us know in the comments below.