As data science has grown in importance, it has encountered the IT department with increasing frequency. Observing these encounters in some organizations is like watching two vastly different galaxies collide. There are many loud explosions, furious disagreements, blinding thunderbolt flashes and a lot of misunderstanding.
What is the source of these vastly different viewpoints? Nearly everything. The culture, education, work practices, organization expectations, attitudes, reward systems, and reputation of the two communities couldn’t be more different.
Below are some examples of the profound differences between the sometimes inflexible IT department and the often swashbuckling data scientists. Narrowing these differences and building mutual understanding will deliver more value from data science for organizations.
Project kickoff
The IT department starts projects with a comprehensive project charter that has been thoroughly reviewed with the project sponsor and key project stakeholders.
The data scientists start projects with a barely-legible hypothesis that’s scrawled on a whiteboard accompanied by large PLO letters in a messy workroom.
Requirements
The IT department expects business analysts to develop a formal Business Requirements Document (BRD) that includes lots of descriptive text and diagrams. The goal is to elaborate the high-level requirements statements in the project charter and confirm the detail with the project sponsor. In addition to functional requirements, the non-functional requirements receive lots of attention.
The data scientists collaborate informally to develop the requirements using an online collaboration environment, lots of whiteboard space and a myriad of emails. The goal is to bring leading research and data science experience to bear on testing the hypothesis. Non-functional requirements are an unimportant distraction.
Design
The IT department wants project teams to elaborate the many requirements statements into comprehensive end-user stories and then prioritize the stories for development. The goal is to fill in the inevitable gaps in the initial requirements and to ensure the most important functionality is developed first.
The data scientists add to the design of their model iteratively as each execution reveals problems and new opportunities to converge on the hypothesis. The goal is to apply the latest data science thinking to irrefutably proving or rejecting the hypothesis.
Software development methodology
The IT department applies its accepted and sometimes overly bureaucratic software development methodology to the characteristics of the project to ensure quality in the planned solution.
The data scientists view development methodologies contemptuously as straitjackets that constrain creativity at best or as a crutch that helps dullards produce useful work at worst.
Data requirements
The IT department expects project teams to include data structures and data quality in the design to ensure that the planned application contributes to enterprise data integrity.
The data scientists scour the organization for data with the ferociousness that V’Ger consumed stars and planets in Star Trek: The Motion Picture. They create new columns and tables as their usefulness arises from the detailed analysis of existing data with no thought to data modeling best practices.
Software development
The IT department develops application software with rigorous peer reviews and comprehensive testing. The goal is to deliver software with as few defects as possible. The data scientists like to keep coding until they see something they like. The goal is to fail fast and often on the road to a viable and defensible solution.
Success criteria
The IT department views success as delivering functional, stable applications for routine, no-drama use by the planned end-user community. If the project is reasonably on schedule and on budget that’s a bonus.
The data scientists view success as discovering breakthrough, actionable insights in the organization’s data. Error checking, disaster recovery, and data integrity are not part of the success criteria.
Operational methodology
The IS department is proud of their detailed ITIL procedures. These implement the operational methodology to achieve a high-availability computing environment that ensures superior customer satisfaction levels.
The data scientists never think about operational methodology. They’re looking for hidden insights that can outsmart the competition and advance the company’s business plan by leaping over tall buildings. They’re only building code for themselves; never others.
Deployment
The IT department plans the deployment of the new application with careful attention to people change management issues and the cutover from the previous application.
The data scientists don’t expect to deploy their software to anyone else, so they don’t think about deployment. Awareness of the idea that the software should be designed to run again next week or next month without a lot of babysitting or further development is generally absent.
Project elapsed time
The IT department tries to minimize the project elapsed time but achieving scope and quality typically take precedence. The data scientists understand how critical first-to-market is and focus on the shortest possible project elapsed time. Considerations of software stability, ease-of-use, data quality, and deployability take a backseat.
What strategies would you pursue to bridge the chasm between data science and the IT department? Let us know in the comments below.