Microsoft went through this with Windows Mobile starting in about 2005, and it created serious problems as the platform aged. Palm also had fragmentation in the later years of PalmOS, and it caused enough issues that some developers quit developing for the platform. The economics of developing apps has changed radically in the last two years, though, so fragmentation is no longer something that can be avoided. Instead, it must be dealt with.
As Charlie Kindel's article explains, fragmentation causes problems--but it won't be the death knell for Android.
Let's start with the UI. Users moving from one phone to another may have a very different experience because each manufacturer has customized the interface. Consumers are generally used to a consistent experience across different machines if the platform is the same. Windows computers are built by many manufactures, for example, but the desktop is the same overall. If you have an Android phone and move to another one, it can be a jarring experience to see that new screen for the first time and figure out how to do things with the new phone that were second nature with the old one.
Google is at least trying to alleviate some of the pain here. Manufacturers can put their custom UI on a device, but they are now required to also support the stock Holo theme so the user can revert to that if desired.
Another big issue is that carriers and manufacturers too often fail to provide updates for phones, even though the vast majority of phones are capable of running one, two, or more OS revisions beyond the one it shipped with. Today there are eleven major versions of Android in use on phones and tablets, and probably dozens of sub-versions. Fully 30% run Froyo, which is nearly two years old.
You cannot argue with the success Android has had in the market. But you have to wonder if Android's satisfaction rate, which is around 50%, would be higher if its fragmentation was more tightly controlled.