Rick Devenuti has one of the most high-profile IT jobs on the planet.
In addition to being a first-round tester for unreleased Microsoft Corp. products, he faces a user base made up of some of the most computer-savvy people in the world. He also knows that if anything goes wrong, the details may end up as front-page news within hours–witness Microsoft’s very public tussle with the Slammer virus early this year.
And IT isn’t Devenuti’s only concern. As corporate vice president and CIO for Microsoft’s Operations and Technology Group (OTG), he’s also responsible for Microsoft’s day-to-day manufacturing and distribution operations. Devenuti recently sat down with CIO editors and discussed how his job isn’t much different from everyone else’s–despite what some may think.
CIO: You use new Microsoft products before anybody else. Do you have to, by default, deploy all new Microsoft products?
RICK DEVENUTI: We set as our number-one priority to be Microsoft’s first and best customer. We define scenarios for why we would use the product and how we could use it. If we can’t come up with a scenario that says the product makes sense, we won’t implement it. But when we’re talking about enterprise products, there are very few that wouldn’t make sense. For example, we’re rolling out Exchange 2003 Beta 2, formerly known by the code name Titanium. For Exchange 2003, we’ll ship 15,000 copies to users in corporate. Today we’ve got about 6,000 users–2,500 users in “dog food” [prerelease Microsoft product] and 3,500 users in deployment.
But we don’t just throw it on servers and hope that we’re still in business tomorrow. We’ve got a plan for 15,000 users on Exchange 2003 in our environment, at a certain level of availability, for two weeks before we can ship that product to customers. We will complete 72,000 mailboxes on Exchange 2003 before we release the product.
Lots of vendors are now using their own products before rolling them out to customers.
We’ve always done it, but the way we do it changed two years ago. We’ve always “eaten our own dog food” is the term. But rather than be a test lab, I want to be an IT organization that adds value to the feedback. Instead of just taking a product and rolling it out, we say, What are the scenarios we want to see for it? For an IT pro, that means, What do you have that’s of value to me? Then I go up against, How are you going to market this thing? What is it that you’re going to say to customers? If you’re going to say, “server consolidation,” I’d better have a scenario that does server consolidation. As opposed to saying, “It’s running in Microsoft,” which we’ve been doing for years and years.
What we wanted to change was not that it runs here, but that it runs under these scenarios. And so we get the product group to align under shared goals that say, It runs under these scenarios, and if you’re going to market server consolidation and we can’t prove it, [Microsoft CEO] Steve [Ballmer] will stop that marketing. I just have to go to Steve and say, “It doesn’t work.”
We haven’t had one of those meetings in a while. OTG has developed its credibility with the product groups, so instead of requiring escalations [to upper management] to address key issues, the product groups are invested in a strong partnership with us. The internal deployment through OTG has become an integral part of the development process.
Given the importance of successful product rollouts at Microsoft, you’ve obviously developed a process for doing that. Do your customers ask for that?
We have a lot of discussions with customers about our deployment strategy. If we look back and say, “If we knew then what we know today, we probably would have done things differently,” we share that with our customers.
When we first rolled out Windows 2000, there was a lot of group discussion about how did you decide the number of forests [or groups of users] and why did you create these domains, and how did you lay out the Active Directory. Since the Active Directory was brand new, being the first ones to use it was very helpful to customers. We had a lot of deep technical discussions with our deployment team. Since we’re early adopters, we spent a lot of time as we went through the process communicating what we were thinking and, frankly, what doesn’t work well.
The best example revolves around Windows 2000 and the Active Directory. There were two scenarios that needed extra work. One was getting the Active Directory and server applications such as Exchange to work together to utilize the Active Directory effectively so that the end-to-end solution was solid. The second was the multiforest scenario. The problems here revolved around ensuring that all the components that modified data in the directory (account creation and management systems, applications, synchronization tools) worked together to keep the data in the directory consistent while reducing the complexity of managing the whole system. OTG’s impact really becomes important with those systemwide issues that span multiple product groups.
How do you handle enforcing IT policy?
As a company, we really try to avoid any type of bureaucracy. We do not manage desktops. It’s a decision we made a long time ago. We don’t want to tell people that they can only use a particular version of Office. We don’t think that’s going to grow a company that will have people thinking outside of what they see today. We have to support the desktops, but we don’t manage them. The only policies that we put in place are around security. What they can do and what we will force them to do if they don’t do it voluntarily. Outside of that, we let them do virtually whatever they want on the desktop.
If people go to a server and pull something down, and it crashes their machines, we as a team will have to go fix them. Does that have a productivity hit? Yeah. It has a cost and a productivity hit for them. On the other hand, if they pull something down that they haven’t seen before, and it excites them and results in a better product, that’s a pretty good price to pay.
Has the recent Slammer incident caused you to change any of your policies?
The security of the network relies on three things being in balance: people, process and technology. In the case of the Slammer virus, we had a process in place for forcing patches in the data center, we had a process of notifying end users of the patch, and we had the technology that enabled the patching process. Where we fell short was with our people. The Slammer hit our developer labs that sit outside of our managed network environment. We were relying on these lab managers to update the machines to the latest patches, and they didn’t do it in a timely manner. As a result of the Slammer, we are considering whether a forced patching policy or further network segmentation to isolate our labs is preferable on a go-forward basis. Our recommendation will be finalized during the next several weeks.
Speaking of security, what is your role in relation to all the people who think about security within Microsoft?
I like to think I’m the person most concerned about security in the enterprise. If there’s an incident that puts our information or our customers’ information at risk, it’s happening on my watch, so I’m involved at a fairly granular level.
To be secure you need great technology, but just as important, you need great people and great processes. We’ve got a team that focuses on the technology; I focus on people and processes. The biggest weakness is people.
As in any environment, there’s a myriad of ways that users can significantly reduce the security of an enterprise. One simple example is password management. All too often, employees unconsciously put our network security at risk by any number of things such as posting their passwords on their monitors, always using the same passwords, sharing their passwords with others and the like. Another issue is not implementing patches in a timely manner. The people part is always the hardest to manage.
It seems to be human nature not to patch daily or not to update antivirus definitions until it’s too late. Is it the same at Microsoft?
We’ve done a lot of [internal] marketing under the Safe Secure Smart banner around security. We decided IT needed to market security to the user base. A lot of that started after 9/11. We took that opportunity while people were thinking about physical security to also start a marketing campaign around information security. But people are people, and we’re hoping that they’re pretty busy with their jobs. So there are some things that we’ve had to do. We’ve just rolled out on the remote access side; 27,000 people in Microsoft need to have a smart card to connect remotely. We’ve introduced another new policy where you have to use Connection Manager. With Connection Manager we’re able to make sure that you have the current version of the antivirus or you can’t connect. The feature pack of Systems Management Server that will be introduced shortly is something that we’ve worked very closely with the product folks on, which allows us to force patches to the desktop. We’ll advertise them, and depending on the patch, give our clients an opportunity to decide when they want to patch. But when we decide it’s time, if they haven’t patched, we can force them.
The reason why we like being the “first and best customer” is the ability to sit down with the product group during a series of meetings and say, “This is what I need to run the enterprise securely. I need this to be in your product, or I have to go find a different product to do it. Windows Update is not it.”
Does the nature of the in-house Microsoft user play into this? Because he’s more technically sophisticated than the average user, does that present challenges for you or make things easier?
I think it plays both ways. Microsoft has more than 50,000 employees now, with another 10,000 to 15,000 interns and consultants. We’ve got users who are sophisticated technically, but we’ve also got accountants, we’ve got salespeople–we’ve got the full gamut that any enterprise would. We’re a little tech heavy, but we’ve got to think about our technology meeting the needs of the entire company.
Some people put every bit that they find on any server anywhere in the company on their machines, and we applaud that creativity and support it, but it does make supporting desktops a little difficult. We’re not talking beta versions of the next code; we’re talking about alpha versions, or “Something I just threw on the server that I wrote yesterday, and it’s cool, and you know, let me try it.” So we do run into some interesting experiences trying to maintain the systems.
Do you think there’s a higher standard of performance for your IT department than for most because you’re so high profile?
We had an incident in October of 2000 [when a hacker penetrated the corporate network] on the security side that alerted me to the impact on the company of us not doing our job well. So we take that standard pretty high as an organization. There are very few things that we do here that don’t become public. It takes about four hours for Steve to send an e-mail before it shows up somewhere. So if we have a large outage because of something we do, we can say it’s [a prerelease product], but it’s going to look like a large outage. And somebody’s going to write, “Exchange 2003 isn’t ready and that probably means a delay in the product.” We realize that comes with the territory. Holding yourself to that standard is pretty fun.
