Why Government IT Sucks
Why the federal government is so bad at information technology and getting worse by the day.
Sean Gallagher, Ars Technica‘s IT editor and a former Navy officer, explains, “Why US government IT fails so hard, so often.”
Despite efforts to make government IT systems more modern and efficient, many agencies are stuck in a technology time warp that affects how projects like the healthcare exchange portal are built. Long procurement cycles for even minor government technology projects, the slow speed of approval to operate new technologies, and the vast installed base of systems that government IT managers have to deal with all contribute to the glacial adoption of new technology. With the faces at the top of agency IT organizations changing every few years, each bringing some marquee project to burnish their résumés, it can take a decade to effect changes that last.
That inertia shows on agency networks. The government lags far behind current technology outside the islands of modernization created by high-profile projects. In 2012, according to documents obtained by MuckRock, the Drug Enforcement Agency’s standard server platform was still Windows Server 2003.
Magnifying the problem is the government’s decades-long increase in dependency on contractors to provide even the most basic technical capabilities. While the Obama administration has talked of insourcing more IT work, it has been mostly talk, and agencies’ internal IT management and procurement workforce has continued to get older and smaller.
Over 50 percent of the federal workforce is over 48 years old—and nearly a quarter is within five years of retirement age. And the move to reliance on contractors for much of IT has drained the government of a younger generation of internal IT talent that might have a fresher eye toward what works in IT.
But even the most fresh and creative minds might go numb at the scale, scope, and structure forced on government IT projects by the way the government buys and builds things in accordance with “the FAR”—Federal Acquisition Regulations. If it isn’t a “program of record,” government culture dictates, it seems it’s not worth doing.
So, how bad is government IT, anyway? Well, the abysmal launch of the Obamacare website is a pretty good indicator. But that’s a brand new system being operated by a swarm of untrained, external users for the first time during a mad deluge of interest. Internally, it’s much worse.
The government has kept using Windows XP and Server 2003 despite warnings from the National Security Agency that “Windows XP and Windows Server 2003 lack critical security features and are near the end of their extended support lifecycle.” Still, the federal government (much like most of the business world) largely took a pass on Windows Vista. And even though the National Institute of Standards and Technology added Windows 7 to the US Government Configuration Baseline a year after its release, most agencies didn’t start migrating off Windows XP until 2011 or later. While the Army and Air Force had adopted Windows Vista, XP was still fairly widespread when the Army began migrating to Windows 7 in July of 2012.
It’s not as if the Powers That Be don’t realize there’s a problem.
That’s not to say that the DOD hasn’t tried to make its IT more modern and efficient—the DOD has long been among the government’s IT “haves,” at least budget-wise, and the military is counting on IT efficiency to help reduce costs. But the scale of the changes it has to make in those efforts is staggering, and the process takes years to implement.
The US Army’s Enterprise Email (EE) program is a case in point. EE, which the DOD’s CIO calls “Army’s #1 information technology efficiency initiative,” is a unified e-mail system for the DOD. The Army, its first customer, had over 440 separate networks—each of them with their own e-mail systems and directories on a variety of different software platforms—when it started looking for an Army-wide solution in 2009. The Army gave up on trying to find its own answer and turned to the Defense Information Systems Agency. The agency, which is in essence the DOD’s internal IT and telecommunications provider, started rolling out its internally built system based on Microsoft Exchange to the Army in February of 2011.
The migration was finally (mostly) completed in July, two-and-a-half years after it started, after a few fits and starts (and a Congressionally mandated cooling-off period). It now reaches over 1.43 million users on the Army’s unclassified network, and 115,000 on the secret SIPRNET network. That’s a tiny fraction of the over 425 million users that Google serves with Gmail, but it’s the biggest Exchange deployment outside of Microsoft’s Office 365 cloud—and it all has to run within the DOD’s security specifications on DOD hardware.
So, what gives?
Part of the reason is the metrics that the government uses for “success” in its IT programs and part is how they get bought in the first place.
The DOD CIO’s IT Dashboard entry for EE rates it as a top program with a metrics score of 5 out of 5. But the Army hasn’t begun to measure one of its key metrics—the speed of mail delivery. And service availability, rated at 99.998 percent, is measured based on “a collective average using the Exchange servers in all the pods”—not on the user experience.
There’s a similar disconnect in metrics over at the Department of Health and Human Services. Take the HealthCare.gov Plan Finder program, part of the HealthCare.gov site that, as described by HHS’ project dashboard, is “a portal that allows consumers to search for both public and private health coverage options through an easy to use health insurance finder tool. Based on answers to a series of questions, the coverage finder produces a menu of potential coverage choices personalized for the user.”
[…]
The bottom line is that federal IT programs’ success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to.
I’ve been in my own little IT circle of hell since re-entering federal service on August 26–seven weeks ago. That morning, I posted here that
I’ll likely be off the grid for a day or three owing to one of the Catch-22-s of working for the Defense Department: You can’t start working until you have a Common Access Card and you can’t get a CAC until you start working. Presumably, I’ll fill out the paperwork and get my photo taken today but it’ll likely be a bit before the card gets issued and, if this is typical of my previous DoD experience, I won’t be able to get the process rolling to get set up with a work station with Internet access until then.
Alas, it was not typical of my previous DoD experience. When I last worked for the federal government, it was as a contractor for DISA back in 2004. The private firm that actually paid my salary was able to arrange for my to get my security badge on my first day in the building but it took some time to get my CAC issued and then a couple more days to get access to a personal workstation. It took nearly a week before I was really able to do my job, which struck me as absurdly inefficient.
Things are much, much worse now.
On the day I reported in, they told us that we wouldn’t be able to get our CACs issued until we were “in the system,” which would likely be several working days. Why didn’t they already have us “in the system,” given that they’d known for quite some time that we were going to be onboarding? Well, sometimes people don’t show up for work and then the time spend getting them “in the system” would have been wasted. Far better to waste the time of new employees, who are in many cases being paid far more for not working during this interval than those responsible for putting them “in the system” are for working than risk having some time wasted in the rare event someone who went through all the hassles of jumping a hundred hoops to get a government job didn’t report for work.
Roughly ten days later, I was “in the system.” That allowed me to set up an appointment with the people who issue a CAC; they won’t do it until then. So, I called over and was told that the first appointment was several weeks later. Most new employees dutifully sign up for an appointment but, anxious to actually be able to get work done at the office, I instead showed up early the next morning and endured the long wait for those without an appointment. Two hours later, I was next in line. Forty minutes later, I still hadn’t been called. Finally, they announced that “the system” (a different system, I gather than the previous system) was down. Sometimes, they explained, it would be back up in a little while and sometimes it would be down for a couple days in a row. I wasted another hour or so to see if it was the former and then gave up.
I came back the next morning, a Friday, and was thankfully able to get my CAC issued. At that point, they asked what my DoD email address was, so they could put it on the card. I told them that my agency had told me that they wouldn’t issue me an email until I got my CAC. Which was true and, indeed, the policy. So the CAC-issuing people told me that, once I got an email address, I’d have to come back over and have it added to my CAC. Great.
So, CAC in hand, I went to my IT office. The woman in charge of getting me started on the process of getting an email address wasn’t in. The system (different than the two aforementioned systems) had just changed, so no one else was able to help.
The next day, the woman who understood how to apply for an email address told me that I would need to sign up for an online system that allowed me to take two classes, which I would have to retake on an annual basis, that would enable me to apply for an email address.
It took nearly two weeks to get approved to take said classes.
Access in hand, I dutifully went over to the library to take them. The system didn’t work, so I finally gave up in frustration after a wasted hour. I tried again at home that afternoon, figuring my superior broadband connection would do the trick. It did not. I tried the library again the next morning. Same deal. Finally, I went back to the IT department and they let me take the classes there. Essentially, they told me stuff that we either obvious to people who had ever used email or which were absurdly silly policies that were so impractical that few seem to actually follow them. Regardless, half a day later, I had passed the classes and printed out the certificates to prove it.
That allowed me to apply for an email address.
A week and a half later, the person in the agency responsible for sending up the request to whoever it is that processes said requests in fact sent up the request. Three days later, those people input it into their system. I was told that an email address would be forthcoming the next Tuesday. Or, certainly, the next Wednesday.
The next Tuesday, the government shut down.
The Monday after that, I reported back to work, as my part of the government was back in business. Still no email address. Ditto Tuesday. And Wednesday. And Thursday. So, finally, they called over to the email issuing place and were told that my stuff, which they told us had been entered into the system two Fridays earlier, was in fact not in the system. So, the IT people sent my stuff back over via email.
That’s where I stand now. I’ll check back in when the place opens back up on Tuesday.
Note that this is just to get an email and access to the ordinary Internet on an outdated PC with Windows XP installed. It’s much, much more cumbersome to get access to systems which can access classified materials.
Aside from all the issues Gallagher points to, a lot of the nonsense is overreaction to various legitimate concerns like cyber attacks, the spread of viruses, and protection of privileged personal information. And, of course, the recent whole-scale theft of classified information by Edward Snowden and the person formerly known as PFC Bradley Manning. But, of course, Snowden and Manning made it through all these hoops and then some.
But this is the very system which prominent Republicans thing will spring into action, to instantly create a cash based accounting system.
Sorry to hear this James – very frustrating. Your manager has to take the lion’s share of the blame here though. A great deal of this stuff could have been avoided if administrative tasks were done in the weeks leading up to your start date, appointments made ahead of time, etc. — ready for you when you first show up.
The manager presumably knows where the seams are in the system, and is the person ultimately able to make the right things happen.
More generally, large organization IT works, but for each increment in “large” you must be a bit more forgiving in “works.”
I mean, Windows Server 2003 was bad because MS, a large organization could not make it better. Then other large organizations chose it, over say Solaris.
To put things into perspective, a significant proportion of all software development projects are failures, whether public or private. So the federal government isn’t unique in its problems.
In terms of software development the federal government operates under some handicaps that are distinctive. For example, the way budgeting is done, clearly within the province of the Congress, hampers effective development.
Another problem is that very large projects have certain inherent difficulties which are particularly troublesome in any field, like IT, that evolves very quickly. If a project is to take five years, in IT its architecture is very likely to be obsolete before it’s completed. I’ve argued in favor of a different approach to project management, projects with more completely usable components during the process, but that’s typically met with tremendous resistance.
My personal experience as the CEO of an organization that executed federal contracts is that as with all large organizations there are an awful lot of oars in the water. The personal and counter-productive agendas within the government bureaucracy dwarfed anything I’ve ever seen in the private sector. I frequently encountered project managers whose primary objective was setting up a lucrative consulting contract after they’d left public service.
My experience, too, is that governments tend to be very standards-oriented and standards are inherently backward-looking. Combine that with the frequently-countered problem that organizational “experts” are self-taught and you have standards can be decades out of date.
It sucks because right wing extremists and terrorists in the GOP and TP may have hacked into the Obamacare website to make sure folks could not get info or enroll. Time for the FBI to launch a nationwide investagation, specifically in southern states, where the problems seem to predominate. Right-wing extremists don’t care about middle class or poor people, only rich corporations who make billions in profits, pay no taxes and are subsidized by poor and middle class folks. If the rich and corporations were paying their fair share of taxes our economy would be very very strong.
James,
I guess I’m lucky working for the Air Force since it only took me a week to get a CAC and our standard desktop is Windows 7.
I’m curious – how long did it take to fill your position from the time it was advertised to the day you started working? For me it was almost exactly a year and I was a “by name request” which is supposed to shorten the process. My hiring process was a bit unusual though, 6-8 months is more typical….
Also, I hate to be the one to tell you this but you are just entering the rabbit hole and it goes pretty deep. I hope you’re prepared for more of the same. One big change is that comm is no long a support element – your information security people will receive directives from US Cyber Command which will flow down to you with a short suspense. Your problems as a user are completely secondary.
And the really sad thing is that none of the problems with government IT or the civil service are likely to change. Neither political party is interested in reform and the senior executives are too busy protecting their rice bowls….
@Dave Schuler: Precisely Dave, Government IT failures get publicity – corporate ones do not. As I said in an earlier post I worked for a large multinational corporation and was part of the team to introduce a new Enterprise software system. It was a disaster. In part this was because we were dealing with code sweat shops in India but even more because the software chosen by upper management didn’t really fit our business model. This received no publicity.
@Dave Schuler:
I’ve seen that happen a few times. In a particularly bad case about 15 years ago the large intelligence organization I worked for contracted to change their unix destop systems and backend with PC’s and Microsoft products. We had a warehouse with about 5000 Dell PC’s siting in boxes waiting for deployment. Two years went by, lots of delays and then it was determined that all those PC’s were now obsolete so they were all carted off to DRMO and a “new” ones were purchased….
Excellent point, that is my experience as well.
and we still wonder why so many people don’t want the gov’t. running healthcare and charity?
@bill:
Sadly, bill, a market-based system just isn’t an option. Markets promise to optimize supply and demand. They don’t promise to ensure that poor people don’t die because they don’t receive basic medical care.
Besides we haven’t had anything resembling a market in healthcare for more than a century. Nobody wants a really free market healthcare system although lots of people define their preferred oligopoly or oligopsony as “free market”.
We’re going to have a healthcare system with lots and lots of government involvement. That is a given regardless of whether Democrats’ or Republicans’ views prevail. The challenge is a good healthcare system with prudent government involvement and for that we’ve got a long way to go.
@Dave Schuler:
There is so much truth in these four sentences that they need to be repeated.
Heck, after about 5 years of planning with Accenture and IBM, the large, multinational company I work for managed to roll out a heavily modified (and therefore barely working) version of Vista after Windows 7 came out. Thanks to to many ridiculous mistakes, we experienced 45-minute reboots, shared drive outages daily, and inaccessible program servers for years after the roll out. It brought R&D to its knees. Even today when we want to transfer gigabyte-sized files, it can be faster to overnight ship an encrypted flash drive than try to SFTP them past our firewall.
The problem wasn’t just that most of the work came from contracted companies, but also because by that time, we had fired most of the in-house people who had the necessary technical knowledge to properly manage the projects. So while government has a set of in-built problems with IT projects, they’re not alone in having institutional inefficiencies.
@mattbernius:
That’s the gospel I’ve been preaching about healthcare reform on my blog for the last eight or nine years.
Keep your grubby hands off my Medicare!
@Tony W: Alas, my manager (a retired 0-5 who’s been at CSC for more than a decade) was powerless. The system, as I’ve described it, is the system and there are no shortcuts. You must cross phase line Black to commence operations towards phase line White.
@Dave Schuler: Yes, big is nearly as much of the problem as government in these cases.
@Andy: Quantico is its own special little hell because there are so many senior folks on base and so little support staff. The Marine Corps prides itself on doing with less support than its sister services, and happily suffers the consequences.
It took a little over a year between the job announcement going out and my coming on board. Sequestration probably cost five weeks. The rest was internal processes.
I went off to read the paper. Three stories on adjacent pages were: problems in the multi-college application software, a computer error blocking food benefits, and problems with the ACA lainch.
Its interesting because the kids are all heading in a different direction, with nimble startups leveraging many parts loosely connected. That is, AirBnB did’t have to invent much, or even write that many lines of code.
Large organizations seldom constrain themselves to such incremental solutions, and they don’t often understand how that affects their chance of failure.
It’s not just the government that has this silliness. One of my friends was CIO of a Japanese firm. Said firm decided suddenly to consolidate all of its credit card services down in Australia. Company hired outside stable of programmers to write customized code and document all of it. Now, given that there’s one person in charge of all of this, he’s under an insane deadline (which means nothing is getting documented as written), and is extremely protective of his stable of Indian programmers, what do you do? You appoint as go-between someone with the history of being a sleazy bastard and a bigot, then act surprised when he insults the programmers, causing the project manager to up and quit, taking all the programmers with him.
The whole thing was a fiasco that cost them several millions of dollars and resulted in zip.
@Tony W:
AAAAAAHAHAHAHAHAHAHAHAHAHAHAHA!!!!!
Okay, I’m sorry, Tony. See, your problem is, you’re thinking logically, rationally and efficiently. Things that have absolutely NOTHING to do with what’s going on in government administration. I can count on both hands and feet — and yours, too! — how many times my colleagues and I have provided suggestions to streamline processes and make things more efficient. FUHGGETABOUTIT!!! “We have met the enemy . . . and he is us.” LOL
The Republicans did their damnedest to screw up the Obamacare roll-out, with Republican governors refusing to set up their exchanges (what happened to state rights?) and Republican Congressmen refusing to vote legislative fixes and money for project setup. Not surprisingly, the Obamacare roll-out didn’t go smoothly.
Now could the Administration have done better? Sure, but fighting off saboteurs didn’t help. Note that California, the most populous state in the Union, had a relatively smooth roll-out. Amazing how things can go well when you actually want something to succeed.
When a IT system sucks, who suffers negative consequences? Do any of those people have the power to make the IT system not suck?
@john personna: small, agile startups succeed because the market will happily absorb a 99% failure rate. Government systems could be designed and implemented in a similar fashion if such a failure rate was acceptable. It would be interesting to see if it would actually be cheaper to develop government IT systems, I suspect it would be, but the bureaucracy would never allow it.
@Michael Hall:
That’s pretty much what I was thinking (and that “failed for technology” is less common with incremental projects).
I generally regard the majority of the federal government’s failure to operate at a minimally acceptable level of effectiveness as a failure of policy (-ies) and a complete abdication of leadership. Bad decision upon bad decision, and no one bothers to think about the effects of their decisions, much less actually gather evidence.
My favorite recent example from personal experience: as a military retiree, my pension is managed by Defense Finance and Accounting Services. Any online interactions with them requires that I set up an account, and the password for that account must:
* be 15 to 30 characters in length
* contain at least two lower case letters
* contain at least two upper case letters
* contain at least two numbers
* contain at least two of the following special characters: # @ $ % ^ ! * + = _
* not contain any other special characters
* not contain a space
* differ from your immediately previous password by a minimum of four characters
* differ from any of your previous 10 passwords
* be changed every 60 days
And of course, this in the context of “NEVER WRITE DOWN YOUR PASSWORD!! EVER!!!”
Imagine some poor ol’ 75-year-old retired sailor and Luddite trying to figure out how to manage his military pension online, while under that monstrosity of a password policy.
This is just one more example of why I have refused to work for the government, directly or indirectly, after I retired from the Navy.
@Boyd: Yes, that’s all try stupid, as XKCD noted some time back:
Still, it’s not just a government problem. Lots of firms do that sort of nonsense.
Yes, James, even my current employer has silly periodicity requirements, but sometimes I think only a governmental body could come up with such a conglomeration of requirements.
And for the record, I’m a huge proponent of Diceware-generated passphrases.
How much time and money went into developing the train wreck that was Windows Vista?
Remember the marketing for Windows 7? “Hey, we are the biggest software company in the world, and our new OS actually works! Don’t you want to run out and buy it?
Yep, private enterprise is so much more effective than the government…
Software fails are about a rare as pretty leaves in autumn.
@Dave Schuler: You need a much bigger platform.
Also, just to re-iterate and expand on one of your points above, the management of software development has been known to be difficult since basically the advent of software development.
@Dave Schuler: oh god, the dreaded “poor people” analogy again- like obamacare is going to help “the poor” take care of themselves…..don’t bet the ranch on it. and speaking of lame IT- the food stamp program crashed this weekend, it was probably Bush’s fault though.
Government IT lags industry because Congress requires that it be cheap and that it works. With those stipulations you aren’t going to be able to hire the gray matter you need for real innovation and even if you could….no one is going to tolerate the failures that leads up to innovative new techniques/applications. That leaves you with using stable legacy equipment that is proven and has out-of-the-box functionality you can hire cheap technicians to maintain.
No one wants to go above the radar and explain some outsized IT budget to a congressional staffer. Now Congressman X is going to looking deeper into everything and making a case that you’re wasting taxpayer money and funding should be reduced or diverted. Then clowns like Bill come along saying: Government can’t do anything and/or wastes money. Its a destructive cycle but hey….that’s what happens when a bunch of lawyers and political aspirants are at the controls.
The real headline should be how Government IT works in spite of massive strategic roadblocks. Comparisons to private sector IT are apples and tomatoes. Gov’t IT on the intel side however, is quite comparable and efficient because Congress is in full support of all things Nat’l Security right now…so intel organizations have money to pay experts and get really good equipment. For now….
@Pharoah Naim,
I have to laugh at that one. The IC isn’t immune to the problems in this thread. As a matter of fact, they are also stuck with obsolete equipment, bloated bad software, and stuff that don’t work. They pay through the nose for stuff like Palantir that are questionable in terms of cost.
@anjin-san:
That is certainly where I was going with my second comment. How much money was spent on Windows Vista? Too much, and not just in a judgmental sense, in a causal sense as well.
Compare to something like Android which was very incremental and successful. It is a Linux, SQL, Java stack. All well understood, all invented elsewhere. And then features wete layered in with frequent releases.
Android is a good example of a big project run in small steps.
Okay recognizing the irony that I fight Android 2.2 text input to write these messages.
I should use voice more often, as I do in this post
@DC Loser: Palantiir actually works and the company provides good support for the product although at a bloated price combined with Wal-Mart like competition practices. The company has support on the Hill which allows it a competitive advantage. I worked for clients that balked at Palantiir because of price only to find that their phones subsequently rang off the hook from Hill power players asking, “Why aren’t you buying Palantiir?!?!” At least it’s an effective tool.
My Intel comments weren’t really directed at regular Intel assets but the “non-regular” ones. They have smart people, good processes, and are flexible…..mainly because Congress leaves them alone with wide latitude to achieve an objective. That trend has been shrinking however as Congress looks to examine every dollar to ensure its going to their pet donors and the wars are ramping down.