Robert McLaws, who hosts the Longhorn Blogs site, fed the nearly 28,000 bug reports from the Connect site's database -- open only to Microsoft-chosen beta testers -- into Excel. Of the 27,479 bugs (as of July 3), 5,743 remained classified as "open." The remainder had either been tagged as "closed" or "resolved."
"They have a lot of them fixed," said McLaws, "and of the ones which haven't been fixed, most are [from the] Beta 2 timeframe." According to his accounting, only about 250 of the unfixed bugs are over two months old. (For reference, Microsoft released Vista Beta 2 in a public preview a month ago.)
"Based on the numbers [of bugs] closed, it really does look like they're pushing for quality," McLaws added. "It proves that they're doing a good job."
But McLaws admitted that his analysis was based on incomplete data. Bug status information from the database is limited, he said, and wouldn't let him determine the number of duplicates or those not reproducible.
"It's apparent that Microsoft is not accurately differentiating between closed bugs and resolved bugs. If Closed = 'Can't Reproduce' and Resolved = Fixed,' that means that only a little over 1,000 customer issues were actually addressed. That can't be right," he wrote on the Longhorn site.
McLaws also graphed the number of posted bugs against the various Vista build release dates, and concluded that testers are quick to log problems: within 24 hours of build's release, an average of 200 bugs are submitted. The number of bugs has also been increasing, particularly since the debut of Beta 2 and the associated CPP (Consumer Preview Program) in early June.
"One might look at all this data and think that the builds are getting buggier," said McLaws. "But I don't think that is a correct assumption. I don't think there are more bugs, just more people with access to the builds." An analyst had a different take.
"Twenty-thousand-some, that's a lot of bugs," said Joe Wilcox of JupiterResearch.
He also wondered if the Connect database was accounting for compatibility issues, either those generated by changes to Vista's underpinnings -- not unusual in any Windows update -- or those caused by problems by its security system dubbed User Account Controls (UAC). Unlike earlier versions of the OS, Vista will limit rights to typical users through UAC, which may break applications that expect users to be running in the more permissive administrator mode.
"There are a couple of fronts where Microsoft could run into serious problems," said Wilcox, and ticked off changes at the kernel level, the enhanced security of the Internet Explorer 7+ browser, and a paucity of third-party drivers.
The number of bugs still open also is a concern, Wilcox said, because of Microsoft's past history. Several times, the company has said an unexpected surplus of feedback from testers was behind delays or changes in Vista or Office 2007. The large number of Vista bugs may augur another delay for the operating system, or may be used by Microsoft as the rationale for pushing it back beyond January 2007.
"We heard that in November [2005 when Microsoft dropped the monthly CTP [Community Technology Preview builds] and we heard it last week with Office.