Over the past twenty years I’ve been around enough web site launches, redesigns, platform swaps, RFPs, RFQs, consultants, vendors, third-party developers, project managers, stakeholder/steering committee meetings and total crashes and site failures to feel eminently qualified to say the smoking crater in the ground known as the Obamacare website was guaranteed to happen the day it was conceived.
And the fix is simple. Seriously. Pick up the phone. Call Werner Vogel at Amazon or Larry Page at Google — arguably the two best web infrastructure guys on the planet — and ask them to fix it. Privatize it. Make it a marketing triumph for IBM or Oracle. Screw the GSA bidding process, screw the consultants, the parasitic systems integrators looking for $10 million engagements, and the whole federal procurement process that is designed to deliver broken, grey blobs that never work. There’s only one solution and that’s to go to the private sector and have a single company accustomed to zero-defects and a lot of uptime “nine’s” build it.
Some people are saying the government should have gone open source and used WordPress or Drupal to build this pig. Wrong. Those are wonderful content management systems, but the problem isn’t the content and it’s not whether or not this thing was built with MySQL or Oracle or a left-handed monkey wrench — the problem lies deep beneath the timed-out sessions, 404 pages and error messages — it’s the underlying transaction processing engine that lies under the surface like the giant fungus on the Olympic Peninsula that is the world largest living organism. This isn’t about code, this isn’t about servers or bandwidth, Hadoop, PHP or whatever the technical problem is. It’s about a bureaucracy building the ultimate bureaucratic system.
Here’s how it works: healthcare.gov is just the visible manifestation of the visible portion of the whole platform, one that is supposed to steer consumers to an informed selection of a health plan offered by a distributed network of commercial health insurance companies. The platform needs to talk to each insurer’s own web infrastructure and accommodate their rules and standards. That’s complex enough, but let’s assume some middle-ware solution to define a common data-interchange was put in place so the government site could handle all the private insurers systems. Check. But before the consumer can get to the insurers they need to register in order to browse.
And we have failure point number one. Any ecommerce operator knows registration is a total buzz killer to a shopper. Get the crap in the cart, and then, when the trigger is ready to be pulled, you roll out registration. Registration itself is database intensive. It means creating a record via a form of fields — last name, DOB, SSN, etc. — and then some form of confirmation via an email that is generated by the system and sent to the user to click on a validation link, etc.. We all know the drill. This is like asking all the fans of tonight’s World Series game at Fenway to stand outside on Landsdowne Street and Yawkey Way until 7:55 pm with first pitch at 8 pm. The gates open, but before each fan can pass through the turnstile they need to fill out a form, submit it, wait for a confirmation to appear on their phone, click that, and then the turnstile can turn. People start to panic. They hit submit a few times. They hear the Star Spangled Banner and they start to push, next thing you know people are being trampled like door busters at a Long Island Walmart at 12:01 am on Black Friday.
Some government dimwit(s) decided to make registration mandatory before browsing. Dumb. Register at the point of the transaction, not the window shopping. Most of the traffic wanted to see what the big deal was. Sure, some entered expecting to leave with a transaction successfully completed, but some just wanted to browse. So let them browse.
The reason it was decided to make users register first was determine their eligibility and show them the plans that fit their circumstances. This is where the website is more than a bunch of pretty screens and a Drupal CMS — this is the Octopus underneath it all that has to take the registration information, (probably the social security number) and run it through a ton of government databases. I have no idea which ones are being pinged, but let’s assume they range from the Social Security Administration to the IRS to Health and Human Services, maybe state social services, Immigration, Veterans ….the list goes on. Who cares? It had to happen and it wasn’t going to be easy. When the requirements were put together for the exchange, the main challenge was determining eligibility and segmenting the set of plans any consumer could see based on certain factors — e.g., oh, you live in Massachusetts, you’re already in an exchange set up by Mitt Romney, so you can go away. Goodbye. One can assume — the government loving complexity because complexity fuels bureaucracies and bureaucracies exists to project the bureaucracy — that the Obamacare website is being asked to route and receive a ton of distributed requests because one after another, as the requirements were gathered, a lot of hands went up and said, “Well, we need to include this of course.” And so the system started to sprout a lot of hair.
This country put a man on the moon. Now it can’t build an online exchange?
Anyway, so Failure Point number two: too much bureaucracy and trying to hit too many external systems to qualify the consumer. Got it. That one was unavoidable but manageable and is by no means a trivial thing to solve.
Gauging from the squirming testimony before Congress and the finger pointing among the contractors that is going on, the biggest failure here was not Cool Hand Luke’s “failure to communicate” but a lack of leadership and a strong project management office. One can imagine what the set of different stakeholders looked like on this project. You’ve got elected officials who can’t find their ass with both hands when it comes to voting on ordinary legislation, let alone technology development; professional government bureaucrats who guard their silos; a bunch of external contractors trying to salvage a 15% profit from their work, all screaming that the requirements are changing too much and the deadline is too aggressive while they offshore the coding to a “low cost operating center”….. This is why communists are shitty capitalists. You can’t get stuff done in a Soviet tractor factory.
Failure point number three: lack of governance. There was no “one throat to choke” on this. Otherwise we’d be seeing a single hapless victim being pilloried as the Obama administration throws a sacrifice to the angry gods. This project needed to be owned at the top, managed like an army on a campaign with a strong project management office, and that project management office should have been throwing warning flags all last summer. Rather than launch it when it was ready, it was launched under a political maelstrom of tin-foil hat Congressmen trying to defund the entire program, a President determined to hit deadlines instead of usability levels and the result was a classic case of too many cooks, not enough accountability, and political forces trumping logical best practices in project management.
Final point of failure: complexity. Too many contractors, too many stakeholders, too many systems …. The President likes to dine with the CEOs of Silicon Valley. He rubs elbows with Gates and Schmidt and the rest of the gang. Hell, Bezos just bought the Post and is officially a heavy hitter in D.C. — pick up the phone, call one of them, say, “Name your price, just make it work.” And tah dah. Let Congress piss and moan that the RFP didn’t follow procurement guidelines. At least America would have a website that works instead of a national embarrassment alongside its Congress, it’s infrastructure, its banks, it’s college loans …. This thing was built by committee and nothing good ever was built by committee.
The fact that Congress is holding hearings to find the smoking gun in the smoking crater is risible. They need some serious infrastructure geeks to get up there and pull a Richard Feynman drolly demonstrating why space shuttles blow up when rubber O-rings are dipped in a glass of ice water. Not the good Congressman from North Dakota who can’t operate anything more complicated than his Blackberry.