Transparency, a 500-mile Journey

There’s an entertaining story that has been going around for years in the sysadmin forums about how a University of North Carolina sysadmin gets a call from a user who’s growing desperate over a problem with the department’s email server: nobody in his department is able to send emails to destinations that are over 500 miles in distance. The sysadmin argues that’s just impossible, email is not bound by physical distances. However, the user insists he has researched the problem thoroughly.

The sysadmin gives in and decides to test the user’s hypothesis. To his upmost surprise, an email to Providence fails while one to New York works. He repeats the test many times over to similar results. To his amazement, the claim was true although it defied common sense.

The user also had a hunch. He pinpointed a system patch that occurred around those days. But the consultant that made the changes attested he “didn’t touch the email system.”

Digging deeper, these were some of the user assertions checked:

• Emails won’t go past a 500 mile radius (confirmed)
• System was patched recently (confirmed)
• Consultant who patched says email is not involved (claim)

The sysadmin realizes that the problem is a misconfiguration related to changes applied by the patch: sendmail timeout values were reset to defaults. That meant emails would timeout trying to reach servers beyond 500 miles, and thus never be delivered.

The user was right

Here’s another story: Database guy stumbles with some hot issues reported in the company’s service management system: several users are complaining about sluggish response times. Those issues are not related directly to his area, database administration, they have not been assigned to him and the database dashboard is reporting normal response times. Then, as the morning progresses, he happens to notice some broker log cleanup is still running from the night before in one of the backend machines he has access to. Looks like somebody else’s faulty script had gone berserk, but kept below the monitoring thresholds. He then comments on the issue, alerting the assigned engineers his take on a probable cause. Swiftly, the problem is tackled and system is back to normal.

The lesson here is that our hero had access to all company issue queues. In fact, at this particular organization everyone has access to any issue, request or bug queue. It’s company policy to give everyone read access to even the most unrelated systems.

Why? Well, how many times does a user have a hunch of what’s going on behind a system problem? By “a hunch” I don’t mean right or wrong. Many times information relayed by the user in a support ticket is, well, just a wild guess. These wild guesses may be wildly unrelated to the underlying problem. So, how do we increase the value of everyone’s contribution to delivering better services?

Transparency is the answer. Giving teams access to information increases the chances of detecting patterns only the human brain can recognize, connecting apparently unrelated dots. Patterns that no report, Excel spreadsheet or hypercube can reveal.

There are certainly many ways to implement transparency in an organization. They go from opening communication channels, to setting up cross-team meetings and CCing emails out. The key here is to always relay information up and down the chain. More uncommonly, but just as critical, are how corporate systems inform and report to users in every role or department. That’s where things like version repository commits, test plans and issue queues come in.

Jack Welsh, the mythic GE CEO and corporate, once said:

“Hierarchy is dead. The organization of the future will be virtually layerless and increasingly boundaryless, a series of information networks in which more electrons and fewer people will manage processes. Information will become transparent. No leader will be able to hoard the facts that once made the corner office so powerful.”

Information is so powerful, people tend to nurture it privately as gold nuggets. Information is the raw material used to build silos. The surprising, or not so surprising, fact is that silos are built from the inside. And by people who feel confortable not being accountable for anything beyond their boundaries.

It’s Empowerment, Stupid

We can give teams a better way to share across silos. Think of the way user access policies are implemented in many enterprise tools. Usually large organizations only give broad read-only access to external auditors but not to employees. That’s a true corporate-world paradox: external auditors and consultants have greater freedom and a wealth of information than employees.

Sharing information is not for teams to audit each other. It’s about feeling part of a larger collective and really being able to collaborate on attaining organizational goals. Isn’t it odd that, while a company goal for a given quarter may be to pursue customer acquisition or an increase in sales revenue, no one has up-to-the-minute access to the sales system or at least some sales dashboards. Most organizations only relay that information back on quarterly calls and, if you’re lucky, in weekly or monthly town hall meetings.

Business suits may doubt that having “Webmaster Mary” see a daily update of bar chart P&L. But the reason is just one: empowerment. People actually feel responsible for delivering a great service if they see how their work cascades down to many departments. The feeling of empowerment generates trust and empathy, two key qualities for happy workers building solid services that are valuable from the ground up.

Yes, building transparent IT requires lots of work, either enabling special accesses or creating department dashboards and mashups. It also may require negotiating with software vendors special deals on read-only or watered down licenses (we at Clarive give those for free). There’s sensitive data in there sometimes. So make sure your tool can hide those sensitive fields from prying eyes, like Clarive’s field-level permissions. And too much transparency may lead to anxiety and a false understanding on the meaning of data. This can be avoided with good information sharing and a culture of trust. These are minor hitches, but before you panic, think: how much real damage can really be done when compared to the benefits of collective business intelligence?