InformationWeek > Opinion > Ask The Secret CIO

Welcome Guest. | Log In| Register | Membership Benefits

June 6, 2001

letter image Secret CIO image Your letters to my print column and E-mail forum raise some serious issues about managing information technology in today's world. Since today's world is essentially absurd, my serious responses may sometimes sound a little whimsical, and my occasional whimsical ones, serious. In any case, if you want to participate, or comment, write to me at lovelace@home.com. I reserve the right to edit for size and content. Just sign your E-mail the way you want it to appear online.

Testing A Candidate's Ability
Question

Dear Mr. Lovelace,
I've noticed a disturbing trend in IT recruiting. Recruiters and employers are giving poorly written tests to potential new hires and consultants. Because of market conditions, and because so many people embellish their qualifications, employers have turned to testing to screen talent.

I've found that the tests I've taken are not created equal. As a result, qualified applicants are not getting jobs because of exams that may be poorly written, or worse yet, wrong.

I'll use myself as an example. I am a Sun certified programmer and developer. Depending on which test supplier wrote the exam, I receive significantly different scores.

Because this situation cost me personally in lost assignments, I did more research. I discovered that some test vendors spend a ton of money developing questions, beta testing, and statistically normalizing questions. However, many tests that recruiters use have none of this work put into them--but, of course, they are a lot cheaper to buy. Try convincing hiring managers that their tests are flawed!

I am sure I am not the only one experiencing this problem. It hurts good candidates and hiring managers to make decisions based on faulty information.

Thank You,
Fumbeya

Dear Fumbeya:
While testing of candidates has some value, I have never been a great believer in its use, to the exclusion of other tools, when considering potential hires. I have found, over the years, that it is more useful to have a good discussion with people. I'd ask them how they would approach solving problems, and which steps they would take first in resolving situations that arise in supporting our systems.

Of course, testing has some real advantages. It is cheaper than an in-depth interview. It is consistent (even if inaccurate) and it avoids the potential of personalities becoming more important than ability. All of that said, however, I think it is wise to take test results with a grain of salt.

To be useful, test questions have to be validated. They also have to be relevant to the situations for which you are hiring a person. It doesn't make a whole lot of sense to test someone's knowledge of a programming language if his or her main job will be to interact with users and diagnose the problems they are experiencing. Other skills will be far more important.

Recruiting agencies, or companies doing the hiring themselves, are foolish if they use tests that are not germane to the position or that are inaccurate. In addition to the fact that they will probably not hire the right people for the job, they also run the very real risk of being discriminatory in their hiring--which, the last time I looked, was against all sorts of state and federal laws.

Since you don't have a great deal to lose once you have been turned down for a job, it might make sense to share your scores in validated tests with the no-longer-potential employers. Calmly explain what you have communicated to me. You may not do yourself any good, but you may make it easier for someone else in the future.

Internet Site Metrics
Question

Dear Mr. Lovelace:
My team is rolling out Web sites that provide information to end users. What sort of success metrics can I utilize to tell whether these sites are actually useful to them?

Amy

Dear Amy:
There are all sorts of metrics that experts will tell you are important: time spent on your site, the number of repeat visits, the number of hits daily, and so forth. These can all be measured through the use of cookies and other monitoring tools. However, I have a basic dislike of such mechanisms, probably because some of them don't seem to be very indicative of how useful the site actually is, and because others strike me as too intrusive on an individual's privacy.

There is a time-honored way to find out how useful these sites are to the people who visit them: Ask them. Put a question on the site's home page inviting the visitor to tell you how useful the information was. Provide a satisfaction scale or a simple query, perhaps something like, "Did you find the information you were seeking?"

You probably won't get a lot of feedback; most people will not take the time to respond. But the information that you do get will be useful to you, and a lot more accurate than trying to figure out what to measure to learn how people feel about the value of your Web site.

Rearranging Deck Chairs On The Titanic
Question

Dear Herbert:
Is there any benefit to changing the organization chart every time a new senior manager comes in? My direct report to the CIO came in, and created a new organization. My new boss came in, and wants to reorganize. My spouse, as a VP of IT, is reorganizing. One person I know has had 15 reorganizations in four years--including three months without a boss. His work and his responsibilities never changed in all that time.

Reorganizing does not change the work. Does it somehow give the new kid something to accomplish, and nothing else? I am thinking, of course, of providing reorganization consulting to new CIOs as a career.

By the way, I look for you first in my InformationWeek.

A deck chair fool,
Matt

Dear Matt:
There is an old story about the CIO who decided to retire. She called in her second in command, telling him the news and informing the assistant that he was going to be the new CIO. The man was noticeably nervous about the added responsibility since a great deal of turmoil was underway in the company. "Don't worry, said the retiring CIO, "I've made three envelopes to help you, in case you get in a bind."

The new CIO was gratified and all went well for about nine months before trouble started to happen. The CIO didn't know what to do, so he opened the first envelope. Inside, written on a piece of paper was the phrase, "Appoint a task force."

He did as suggested, and three months later received a report with some suggested changes, which he implemented. All was well for a year when grumbling was once again heard, so it was time for the second envelope. Inside, this time was a single word, "Reorganize."

"Brilliant," thought the CIO. So he called in a consulting firm, spent a bunch of money for a study and reorganized the shop. Peace ensued for more than two years as people struggled to understand their new roles, but, alas, things again began to get bad--very bad. Not being able to think of anything else to do, the CIO opened the third envelope and read the final piece of advice, "Make three envelopes, and call in your second in command."

All too often, Matt, reorganizing is done in a misguided attempt to try to solve problems, or simply to show that the new leader is in charge. Instead of changing the processes that are creating the difficulties for the group, the assumption is made that, somehow, a new face will figure out how to make things run more smoothly. In fact, sometimes this strategy works, but not as often as it fails--as you've seen from your own experience.

The knee-jerk reaction of reorganizing whenever there are problems in information technology is not a good idea. Certainly, there are times when it makes sense to change the organizational structure--I've done it myself, many times. But reorganizations should be considered because of specific reasons and for targeted results. If the business units were decentralized, for example, and now become centralized, it probably makes sense to change the IT structure.

I don't know if you can make your fortune giving advice on reorganizing to new CIOs. I do know that you will have lots of competition from some big firms. Have you thought of starting by sharing your philosophy with your spouse who is busy reorganizing her IT shop? Any man or woman who can get away with telling his or her spouse how to do that job is a sure bet to survive in the consulting world.

Dedicated Network Or Virtual Private Network?

Dear Herbert:
This is a somewhat philosophical question about a decision that I believe IT professionals will be facing more and more as digital-subscriber-line and broadband services proliferate and companies upgrade from older midrange technologies to client/server applications.

Given a company with multiple remote locations, a need to access an applications services provider application and basic E-mail/file sharing, would you advise them to build a private intranet (leased/frame) with a single Internet gateway or simply use an Internet service provider or VPN connections (DSL) at each location and have their entire network remotely managed via the ASP?

Jim

Dear Jim:
In keeping with your statement, that yours is a somewhat philosophical question, let me give you an answer in that same vein: It depends.

As with any choice of IT architecture, you should define criteria on which to base the decision. The key points that come to mind, when determining whether your organization is better off with building a dedicated network or using a virtual one, are as follows: cost, reliability, security, and flexibility.

If you do not have the telecommunications expertise in-house to handle the design of such a network and its management, you may be better off outsourcing the whole thing. On the other hand, any problems with your network will be immediately apparent to the people in your company. Since you are already using an ASP, you have some experience with the build vs. buy decision. The bottom line is that the selection is an important one and no one answer is suitable for everyone.

Herbert W. Lovelace shares his experiences (changing most names, including his own, to protect the guilty) as CIO of a multibillion-dollar international company. Send him E-mail at lovelace@home.com.

NOTE TO READERS: As I've mentioned, I am planning to put my InformationWeek columns together into a book with a little bit of additional commentary around the events and people about whom I write. If any reader would like to be notified of such an event, please drop me an E-mail. Just use the word BOOK as the subject line.