Skeptical about the team competencies that delivered the AI-derived recommendations?

As a senior executive or CIO, how can you assure yourself that artificial intelligence (AI) or machine learning (ML)-derived recommendations are reasonable and flow logically from the project work that has been performed?

You want to support and encourage your team’s work, but you don’t want to be unwittingly misled, and you want to confirm that the data science team has not misled itself.

“To confidently act on AI-derived recommendations, management must ensure that the project team includes sufficient expertise in data science and business experience. Otherwise, the model may be technically correct but based on flawed business assumptions,” said says Jack Wells, a data scientist at Invoice2go in San Francisco, California,

Here are some high-level questions that you can ask the team about team competencies. They’re designed to raise everyone’s assurance that the AI/ML recommendations are sound and can be confidently implemented even though you and everyone else know that you’re not an expert. Start by selecting one question that you’re most concerned about and you’re most comfortable asking.

Team competencies

No project team ever has all the talent they’d like to have on board. Nevertheless, the confidence you can have in AI/ML-derived recommendations depends on the adequacy of the team competencies. Here are some related questions that will illuminate team competencies:

1. Given the project characteristics, which technical competencies are adequately or inadequately represented on the project team?
2. Which subject-matter expertise is adequately or inadequately represented on the project team given the project characteristics?
3. How much project team turnover occurred during the project?
4. What technologies did you introduce during the project that required the addition of new competencies?
5. Please describe the training you provided the team to boost competencies during the project.
6. To what extent has the project rely on external consultants to round out team competencies?

Evaluate answers

Here’s how to evaluate the answers that you’ll receive to these questions from your data science team:

1. If you receive blank stares, this means the topic of your question has not been addressed and requires more attention before the recommendations should be adopted. It will be necessary to add missing skills to the team or even replace the entire team.
2. If you receive a lengthy answer filled with a lot of data science jargon or techno-chatter, the topic has not been addressed sufficiently, or worse, your team may lack the critical skills required to deliver confident recommendations. Your confidence in the recommendations should decrease or even disappear.
3. If you receive a thoughtful response that points to uncertainties and risks associated with the recommendations, your confidence in the work should increase.
4. If you receive a response that describes potential unanticipated consequences, your confidence in the recommendations should increase.
5. If the answers you receive are supported by additional slides with relevant figures and charts, your confidence in the team should increase significantly.
6. If the project team acknowledges that the topic of your question should receive more attention, your confidence in the team should increase. It will probably be necessary to allocate more resources, such as external data science consultants, to remedy the shortfall.

For a summary discussion of the topics you should consider as you seek to assure yourself that AI/ML recommendations are reasonable, please read this article: Skeptical about AI-derived recommendations? Here are some tips to get you started.

What ideas can you contribute to help senior executives assure themselves that the AI/ML-derived recommendations are reasonable and flow logically from the project work performed? Let us know in the comments below.