Crammed into conference rooms with pizza for dinner, some programmers building the Obama administration's showcase health insurance website were growing increasingly stressed. Some worked past 10 p.m., energy drinks in hand. Others rewrote computer code over and over to meet what they considered last-minute requests for changes from the government or other contractors.
As questions mount over the website's failure, insider interviews and a review of technical specifications by The Associated Press found a mind-numbingly complex system put together by harried programmers who pushed out a final product that congressional investigators said was tested by the government and not private developers with more expertise.
The government spent at least $394 million in contracts to build the federal health care exchange and the data hub. Those contracts included major awards to Virginia-based CGI Federal Inc., Maryland-based Quality Software Services Inc. and Booz Allen Hamilton Inc.
Project developers who spoke to the AP on condition of anonymity — because they feared they would otherwise be fired — said they raised doubts among themselves whether the website could be ready in time. They complained openly to each other about what they considered tight and unrealistic deadlines. One was nearly brought to tears over the stress of finishing on time, one developer said. Website builders saw red flags for months.
A review of internal architectural diagrams obtained by the AP revealed the system's complexity. Insurance applicants have a host of personal information verified, including income and immigration status. The system connects to other federal computer networks, including ones at the Social Security Administration, IRS, Veterans Administration, Office of Personnel Management and the Peace Corps.
President Barack Obama on Monday acknowledged technical problems that he described as "kinks in the system." He also promised a "tech surge" by leading technology talent to repair the painfully slow and often unresponsive website that has frustrated Americans trying to enroll online for insurance plans at the center of Obama's health care law.
But in remarks at a Rose Garden event, Obama offered no explanation for the failure except to note that high traffic to the website caused some of the slowdowns. He said it had been visited nearly 20 million times — fewer monthly visits so far than many commercial websites, such as PayPal, AOL, Wikipedia or Pinterest.
"The problem has been that the website that's supposed to make it easy to apply for and purchase the insurance is not working the way it should for everybody," Obama said. "There's no sugarcoating it. The website has been too slow. People have been getting stuck during the application process. And I think it's fair to say that nobody is more frustrated by that than I am."
The online system was envisioned as a simple way for people without health insurance to comparison-shop among competing plans offered in their state, pick their preferred level of coverage and cost and sign up. For many, it's not worked out that way so far.
Just weeks before the launch of HealthCare.gov on Oct. 1, one programmer said, colleagues huddled in conference rooms trying to patch "bugs," or deficiencies in computer code. Unresolved problems led to visitors experiencing cryptic error messages or enduring long waits trying to sign up.
Congressional investigators have concluded that the government's Centers for Medicare and Medicaid Services, not private software developers, tested the exchange's computer systems during the final weeks. That task, known as integration testing, is usually handled by software companies because it ferrets out problems before the public sees the final product.
CGI Federal said in a statement Monday it was working with the government and other contractors "around the clock" to improve the system, which it called "complex, ambitious and unprecedented."
The schematics from late 2012 show how officials designated a "data services hub" — a traffic cop for managing information — in lieu of a design that would have allowed state exchanges to connect directly to government servers when verifying an applicant's information. On Sunday, the Health and Human Services Department said the data hub was working but not meeting public expectations: "We are committed to doing better."
Administration officials so far have refused to say how many people actually have managed to enroll in insurance during the three weeks since the new marketplaces became available. Without enrollment numbers, it's impossible to know whether the program is on track to reach projections from the Congressional Budget Office that 7 million people would gain coverage during the first year the exchanges were available.
Instead, officials have selectively cited figures that put the insurance exchanges in a positive light. They say more than 19 million people have logged on to the federal website and nearly 500,000 have filled out applications for insurance through both the federal and state-run sites.
The flood of computer problems since the website went online has been deeply embarrassing for the White House. The snags have called into question whether the administration is capable of implementing the complex policy and why senior administration officials — including the president — appear to have been unaware of the scope of the problems when the exchange sites opened.
Even as the president spoke at the Rose Garden, more problems were coming to light. The administration acknowledged that a planned upgrade to the website had been postponed indefinitely and that online Spanish-language signups would remain unavailable, despite a promise to Hispanic groups that the capability would start this week. And the government tweaked the website's home page so visitors can now view phone numbers to apply the old-fashioned way or window-shop for insurance rates without registering first.
The House Energy and Commerce Committee was expected to conduct an oversight hearing Thursday, probably without Health and Human Services Secretary Kathleen Sebelius testifying. She could testify on Capitol Hill on the subject as early as next week.
Uninsured Americans have until about mid-February to sign up for coverage if they are to meet the law's requirement that they be insured by the end of March. If they don't, they will face a penalty. The administration says it's working to address the timing issue to provide more flexibility.
Sen. Marco Rubio, R-Fla., plans to introduce legislation to delay that requirement because: "It's not fair to punish people for not buying something that's not available," Rubio told "CBS This Morning" on Tuesday.
On Monday, the White House advised people frustrated by the online tangle that they can enroll by calling 1-800-318-2596 in a process that should take 25 minutes for an individual or 45 minutes for a family. Many reporters who called the number after the announcement received a busy signal.
Assistance is also supposed to be available in communities from helpers who can be found at LocalHelp.HealthCare.gov.