InformationWeek Stories by George Crump

InformationWeek Stories by George Crumphttp://www.informationweek.comInformationWeeken-usCopyright 2012, UBM LLC.2013-05-22T09:06:00ZSoftware-Defined Storage Vs. Traditional Storage SystemsHere's why software-defined storage won't drive established storage companies out of business.http://www.informationweek.com/storage/systems/software-defined-storage-vs-traditional/240155241?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/storage/data-protection/8-great-cloud-storage-services/240151180"><img src="http://twimgs.com/informationweek/galleries/automated/967/Cloud_Storage_Services_01_tn.jpg" alt="8 Great Cloud Storage Services" title="8 Great Cloud Storage Services" class="img175" /></a> <div class="storyImageTitle">8 Great Cloud Storage Services</div>(click image for larger view and for slideshow)</div>In my last <a href="http://www.informationweek.com/storage/reviews/is-software-defined-storage-right-for-yo/240154877">column</a>, we discussed software-defined storage (SDS) and what type of data center might best benefit from that type of storage architecture. This time we'll compare software-defined storage to more-traditional storage hardware to see if SDS eventually will eliminate the high-value storage-system market. Most storage systems that you buy from major manufacturers are really storage software running on servers designed specifically to house a lot of hard disk or flash solid state drives (SSD). They have a lot in common with software-defined storage and there is almost nothing that would prevent these vendors from virtualizing their storage software and also being software defined. Storage systems can differ in a few ways from SDS. First, they can provide newer and more innovative services. To some extent, though, this is just out-developing SDS vendors with better software. Storage system vendors do have a lead in the basic core services they provide today and that might give them an advantage in developing the next generation of storage services. They might also have an advantage in knowing what lies beneath their software. [ Want more on software-defined storage? Read <a href="http://www.informationweek.com/storage/systems/software-defined-storage-a-buzzword-wort/240146615?itc=edit_in_body_cross">Software-Defined Storage: A Buzzword Worth Examining</a>. ] For example, as I discussed in <a href="http://www.storage-switzerland.com/Articles/Entries/2011/8/8_Primary_Storage_Deduplication_and_SSD.html">"Primary Storage Deduplication and SSD,"</a> a flash vendor might decide it's worth investing in deduplication technology because they have IOPS to spare and expense to eliminate. A software-defined storage manufacturer would not have this advantage and might not invest in primary storage deduplication. The key difference between SDS and storage systems is the level of assembly required. As I mentioned in my earlier column, software-defined storage has a "kit" nature to it and the IT designer has to do the integration work between the storage software and the physical hardware. A storage system essentially has that work done ahead of time. While SDS wears its "kit" nature as a badge of honor, most storage systems pride themselves on their turnkey nature. With a storage system, you know in advance that everything within that system is designed or at least has been tested to work together. Also, support is greatly simplified -- if one vendor provides most of the storage infrastructure, it is liable for making it all work. For the under-staffed IT department these are welcomed advantages over software-defined storage. For these reasons I think many data centers will continue to invest in turnkey, value-added storage systems, not go the roll-your-own route. But, as we will discuss in our upcoming webinar, <a href="https://www.brighttalk.com/webcast/5583/75361">"Designing a Cost Effective Solid State Hyperscale Data Center,"</a> there will be a measurable number of data centers that will take this do-it-yourself approach to storage. The appeal of reduced costs and potentially greater scale might be too much to turn down. If this transition occurs on a larger scale than what I predict, then all is still not lost for the lead storage-system vendors -- they could easily virtualize their own offering and lay some claim to being software defined. Next time I'll talk about how this would be especially easy for storage system companies that are also large server companies. (Three guesses who I'm talking about.) These companies all have struggled lately and I think part of their challenge is they are not leveraging some of their greatest asset: the servers they build.2013-05-16T09:06:00ZIs Software-Defined Storage Right For You?Data centers have new storage options, but "new" isn't always "better."http://www.informationweek.com/storage/reviews/is-software-defined-storage-right-for-yo/240154877?cid=SBX_iwk_related_commentary_Data_protection_storageSoftware-defined storage was on center stage at EMC World last week, as <a href="http://www.informationweek.com/quickview/emc-vipr-bets-big-on-sdn/3263?wc=4">EMC announced ViPR</a>. As Storage Switzerland said in our <a href="http://www.storage-switzerland.com/Blog/Entries/2013/5/7_Analyzing_EMCs_Software_Defined_Storage.html">briefing note</a>, this is an important step for EMC and provides it with an interesting way to integrate its disparate storage offerings. But, will you and your data center embrace a software-defined storage offering? Right now the answer seems to be a resounding no. What Is Software Defined Storage? While the industry has not (and probably never will) settle on a single definition for software-defined storage, my working definition is that the technology abstracts storage controller functions away from storage hardware and moves them into software. This software is run on an appliance or as a virtual machine in a hypervisor. The concept behind software-defined storage seems very appealing: a single point of data services, like volume management and snapshots, form a single console while using practically anyone's disk hardware. [ Learn more about EMC's ViPR. Read <a href="http://www.informationweek.com/storage/systems/at-emc-scale-out-storage-grows-up/240154455?itc=edit_in_body_cross">At EMC, Scale Out Storage Grows Up</a>. ] As I discussed in a <a href="http://www.storage-switzerland.com/Blog/Entries/2013/5/7_Does_Viper_Eliminate_Vendor_Lock_IN.html">recent article</a>, while software-defined storage does provide new freedoms in hardware acquisition, it does not totally eliminate vendor lock-in. In fact, you could make the case that it locks you even tighter to a single console for storage services and delivery of those services. Unless vendors create a way to transparently move your storage from one software-defined storage platform to another, some vendor lock-in is reality. Does Software Defined Storage Make Sense For You? While it seems that everyone would want to use software-defined storage, at least right now, few do. Most data centers continue to buy integrated, purpose-built storage systems, a trend that we expect will and should continue. While the software has improved, the software-defined storage architecture has a "kit" element to it since drives and storage servers have to be sourced separately. For most data centers that are stretched too thin and have bigger priorities than creating storage servers, the integrated, purpose-built approach has a lot of appeal. So who should consider software-defined storage? As we will discuss in an our upcoming webinar, "<a href="http://www.storage-switzerland.com/HyperCDC.html">Designing Cost Effective, All Flash HyperScale Data Centers</a>," ideal candidates are organizations where the data center is the core of the operation and where every dollar saved in the delivering of capacity and performance makes a big difference. Software-defined storage is especially appealing when those solutions can run at the compute hypervisor level. This allows them to consolidate server compute and storage on a single tier. In essence what we are looking for is a data center with lots of compute, the ability to take advantage of local, high-speed storage, and the competitive incentive to invest in building the architecture.2013-05-09T09:06:00ZActive Data Vs. Active ArchiveWe need better metrics to help us decide what data should be on primary storage and what should be on archive storage.http://www.informationweek.com/storage/data-protection/active-data-vs-active-archive/240154236?cid=SBX_iwk_related_commentary_Data_protection_storageIn my <a href="http://www.informationweek.com/storage/systems/active-data-demands-storage-rethink/240154117">last column</a> I discussed how what we used to consider active data is changing. We now have to look at the potential working set instead of the actual working set. Thanks to initiatives like real-time analytics, some data that we used to classify as archivable now needs to be at the ready. If this is the case, what is the role of archive? How do disk and tape archives participate in an increasingly active world? The key to a balanced storage strategy, even with all this active data, is to change how we decide to archive a certain set of data. Under the current archive methodology the most common decision point was last modification date. In other words, data that is X days/years old can be archived, everything else has to stay on primary storage. The problem with this methodology is it is not compatible with real-time analytics and not even really compatible with the way users use data. We need better metrics to help us decide what data should be on primary storage and what should be on archive storage. A key criteria is going to be what data, if it needs to be accessed, will need to be delivered instantly -- in other words, something that may need to be analyzed in the future. This data should probably not go to an archive no matter how old it gets since it could have a statistical probability of value. [ Learn more about virtual desktop infrastructure. Read <a href="http://www.informationweek.com/storage/virtualization/vdi-performance-and-cost-a-deeper-dive/240153398?itc=edit_in_body_cross">VDI Performance And Cost: A Deeper Dive</a>. ] However, if we know for sure that a certain data set will not be part of a real-time processing application or be needed for analytics then lets archive it as soon as possible and not even wait for it to age. Maybe some of this data could even spend all of its data lifecycle on archive storage because the performance of the archive is "good enough" for the use case. There is also the need to understand relationships between files. As a simple example, I am writing a couple of books right now. Each of those books have multiple iterations on the file name but large chunks of the content within those files are the same. Each draft gets a different file name. When I get to the end of any of these books, I really don't think I will need all of these drafts but, because all data has become a "you never know" situation, I will want to keep all of them around but I doubt I will ever access them again. The question is how many of these drafts will I require instant access to and how many could I wait 10 minutes before I view them? For my purposes, all I really will need is the final copy and maybe a couple of the iterations. It would be nice to have software analyze this data and keep versions of the files with the most significant internal changes and then archive the rest. Interestingly, one of the things we are learning from our <a href="http://www.storage-switzerland.com/Blog/Entries/2013/4/16_Primary_Storage_Deduplication_Test_Drive.html">primary storage deduplication test</a> is how big of a role this technology can play in these circumstances. Essentially, I can keep all of the files with minimal impact on space utilization. And since they can be disk-based, retrieval time is excellent. Another classification point is how is that data acted on when recovered? From beginning to end, or at some random point in the file? Basically, can the data be utilized sequentially? If this is the case, then just the front section of that data needs to be stored on primary storage, enough so that it can start being accessed while the back end catches up and the users see no delay in response time. This capability will require a file system intelligent enough to deliver data from two different sources at the same time. When these attributes of the data are known and understood, then it can be properly placed in the proper types of storage in the data center. Data that whose recovery need is random and unpredictable will need to go on fast storage if analytics are being used. Data that is very similar to other data can be archived or deduplicated. This archive, depending on what the known recovery need is, can easily be tape based because for a large chunk of the data set how quickly it is recovered is less import than how cost effectively can it be stored.2013-05-06T09:06:00ZActive Data Demands Storage RethinkWhen data must be accessible, often instantly, you need to consider a dedicated flash array.http://www.informationweek.com/storage/systems/active-data-demands-storage-rethink/240154117?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/big-data/slideshows/big-data-analytics/5-big-wishes-for-big-data-deployments/240153214"><img src="http://twimgs.com/informationweek/galleries/automated/981/Big_Hadoop_01_tn.jpg" alt="5 Big Wishes For Big Data Deployments" title="5 Big Wishes For Big Data Deployments" class="img175" /></a> <div class="storyImageTitle">5 Big Wishes For Big Data Deployments</div>(click image for larger view and for slideshow)</div> Active data used to mean the data you were working on at that moment in time. Data in this category was typically active for a short period of time. Active data was created, collaborated on and distributed, but then within a few days or weeks would become permanently dormant. The capacity requirement of this working set would be relatively static. But active data is changing, and we need to re-think how that data is stored. Active data is changing in two important ways. The first is probably no surprise to any IT professional: There is more of it than ever. The working set of data is now substantially larger than what was dealt with in the past. Part of this change is that users simply create more data per device and everything is created on a device. And they have more devices on which they want to create that data. A great example can be found in simple note taking -- fewer and fewer people take notes on paper any more. It starts out and remains digital. [ Looking for more advice on flash storage? Read <a href="http://www.informationweek.com/storage/systems/why-flash-storage-excels-in-virtual-envi/240149952?itc=edit_in_body_cross">Why Flash Storage Excels In Virtual Environments</a>. ] The second area where active data is changing -- and the important one for our discussion -- is recall. The old profile of data going dormant and staying dormant has changed. More data than ever needs to be accessible and many times that accessibility needs to be instant. This means that it can't be off-line, on a tape drive -- or in the case of real-time analytics -- even on a hard disk. Also, the predictability of just what data will be needed at a given moment in time is difficult. Many vendors are suggesting the use of a solid-state drive (SSD) tier for this working set but as I discussed in my recent article <a href="http://www.storage-switzerland.com/Articles/Entries/2013/4/30_Is_SSD_Enough_To_Stop_Active_Data_Onslaught.html">"Is SSD Enough To Stop Active Data Onslaught?"</a>, the size of this potential working set, because of the unpredictable nature, might be too large for a typical SSD cache. Many organizations are finding that much of their working set needs to be on a dedicated flash array to be able to respond to real-time data requests. Real-time environments have a need to assemble an answer, often from disparate data sources, in a moment's notice, even if the data being assembled or analyzed is relatively old. If that older set of data is not in the SSD cache, the real-time part of the user experience is lost. For example, I saw an application recently where this need for real-time data is obvious. You hold your smartphone or tablet's camera so it is aimed at a restaurant you're passing by. Without taking a picture, it provides real-time information about that restaurant -- phone number, menus, ratings from various sites, health department reports, available slots in their reservation system, and even entertainment such as movies or sporting events happening nearby -- all simply by being pointed at the restaurant. All of the above information points needed to be accessed and processed instantly but they were all being accessed from different repositories. This is an excellent example of how a large flash system can create a customized experience in real time for a user. There Is A Line The need for active data and real-time processing is increasing but there is a line. Not all businesses need that level of real-time processing. Even if you do need that type of active working set, there is a line where moving data to another form of storage is appropriate. The key going forward is not to ask ourselves what data might become active but to assume that all data eventually will become active again. Then we need to ask which data, when it becomes active again, will be needed randomly and which will be needed sequentially. Randomly needed data should be on a SSD; it might make more sense to keep sequential data on tape. Hard disks might be the odd man out here. Data that has even the slightest potential to become randomly active because of an analytics need will have to go on disk. The remaining data set, and still the largest set, can be well served sequentially by tape. In my next column I will discuss the surprising new importance of tape in an increasingly active data world.2013-04-30T10:46:00ZThe Primary Storage Deduplication ProblemVendors struggle to develop a deduplication engine that won't impact performance and still maintain data integrity.http://www.informationweek.com/storage/systems/the-primary-storage-deduplication-proble/240153890?cid=SBX_iwk_related_commentary_Data_protection_storageDeduplication was successful in backup for two reasons. First, it dramatically improved the efficiency of storing backup data and, second, the vendors needed that efficiency so that they could be price competitive against the prevalent tape technology. Primary storage deduplication has not yet been as successful as backup dedupe technology for a number of reasons, creating a lack of enthusiasm from the vendors and potential users. The Primary Dedupe Payoff Problem One of the key challenges that primary storage deduplication faces and one of the reasons for lack of enthusiasm is that dedupe's level of efficiency won't be as high, in most cases, as it was in backup since there is not the level of data redundancy. That said, the results of primary storage deduplication are still impressive and there should be enough efficiency gain to stir the interest of any data center manager. For example, in our <a href="http://www.storage-switzerland.com/Blog/Entries/2013/4/16_Primary_Storage_Deduplication_Test_Drive.html">primary storage deduplication lab test</a> we are getting an 80% reduction on a production NFS share that stores our home directory. Like most home directories it contains mostly office productivity files. A future step in our testing is on a VMware data store, where we expect even more impressive results. Primary Dedupe Is Hard Almost every vendor that I speak with has or had some form of deduplication technology in house or under development. Larger vendors have bought their way in, while smaller vendors have leveraged some of the open source code that is available. But the reality is that most of these products have not made it to primary storage. It is hard to develop a deduplication engine that won't impact performance and maintain data integrity, especially as the storage system scales to meet capacity requirements. As a result, some vendors have either shelved or downplayed their dedupe capabilities. Only a few have managed to strike the performance and data integrity balance. Lack of Cheaper Competitor As I mentioned in the opening, another motivator for backup deduplication success was that it was competing against tape for the hearts and minds of the backup administrator. The price delta between standard disk and tape made the investment in deduplication necessary so that gap could be closed. Until recently this was not the case in primary storage. There was not as large a price gap between fast enterprise disks and slower hard drives. The price difference between a 15K RPM hard disk and a 10K RPM hard disk was not great enough to make an investment in deduplication worth the effort. The appearance of flash-based storage however is re-creating the tape vs disk dynamic in the form of hard disk vs. solid state disk (SSD). There is a significant price delta between SSD and HDD. As we discussed in our article <a href="http://www.storage-switzerland.com/Articles/Entries/2012/3/28_Overcoming_The_All_Flash_Storage_Challenge...Cost.html">"Overcoming The All Flash Storage Challenge ... Cost"</a> deduplication is a go-to option when trying to drive down the cost of flash-based storage systems. Similar to how deduplication drove disk into the backup process, deduplication could be the key to driving flash-based storage into the mainstream of primary storage. My opinion is that deduplication will eventually be integrated into all primary storage systems, operating systems and hypervisors. But most of the innovation will come from vendors that don't have a vested interest in continuing to sell more hard disks. Look for All-Flash and Hybrid Storage vendors to take the lead on primary storage deduplication, either by developing their own dedupelication technology or leveraging an existing API set.2013-04-23T10:15:00ZVDI Performance and Cost: A Deeper DiveLab tests aren't perfect simulations, but they do help businesses decide whether a virtual desktop infrastructure is for them.http://www.informationweek.com/storage/virtualization/vdi-performance-and-cost-a-deeper-dive/240153398?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/storage/data-protection/8-great-cloud-storage-services/240151180"><img src="http://twimgs.com/informationweek/galleries/automated/967/Cloud_Storage_Services_01_tn.jpg" alt="8 Great Cloud Storage Services" title="8 Great Cloud Storage Services" class="img175" /></a> <div class="storyImageTitle">8 Great Cloud Storage Services</div>(click image for larger view and for slideshow)</div> Recently in his blog, Art Wittmann <a href="http://www.networkcomputing.com/next-generation-data-center/commentary/storage/calculating-the-true-cost-of-vdi/240152954">took exception</a> to a column in which I stated that capital expenditure (CapEx) could and should become a key factor in the evaluation of a virtual desktop infrastructure (VDI) project. Although the goal of saving money on hardware and software acquisitions should not replace the goal of reducing operational expenditures (OPEx), a VDI project that achieves both would be that much more appealing. In my experience, the main component that drives VDI costs up is the supporting storage infrastructure. If storage can be made to perform significantly better and still remain cost effective, the CapEx costs of a VDI solution would come down. As I covered in my <a href="https://www.informationweek.com/storage/systems/what-should-a-vdi-desktop-cost/240152792">original column</a>, intelligent caching is one way to accomplish this. My analyst firm, Storage Switzerland, recently was hired by caching software manufacturer VeloBit to run a lab test and potentially verify their product claims. My original column was not about any particular vendor, but more about the importance of reducing CapEx and the effectiveness of caching as a method to drive down the cost of the storage components of a VDI project. I did, however, link to that report as an example of our research. [ Want more on VDI's increased acceptance? Read <a href="http://www.informationweek.com/storage/virtualization/is-storage-saving-virtual-desktop-infras/240150414?itc=edit_in_body_cross">Is Storage Saving Virtual Desktop Infrastructure?</a> ] The first of Art's concerns was how "real world" our tests were. That's a fair concern of any lab test as, obviously, no test can perfectly recreate a production environment. All you can do is accurately document the test methodology and the configurations that you tested. Users have to translate that into their reality. We learn something from every test we do and attempt to apply that to our next test suite. The second concern raised was that we used non-persistent desktops. We acknowledged this and stated that we will specifically test both persistent and non-persistent desktops in the future. That said, based on what we saw, I don't think the use of non-persistent desktops would significantly change the overall efficiency, but we will need to verify that through separate testing. Clearly the desktop type is something that the reader should factor into his assessment, but the level of improvement from technologies such as what we tested can't be ignored no matter what type of desktop is eventually chosen. We wanted to test both desktop types because that would have been an interesting comparison and because it's a choice that VDI project planners routinely have to make. But this time we simply didn't have the time to get both tests done and compiled for this particular report.The third concern, also legitimate, is how we simulated a 200-plus desktop workload. You don't want to create an environment that purposely makes the storage solution look good, which means the simulation software is critical. It has to be able to recreate a real-world, highly randomized work day. The testing tool we used for this report is called Login VSI. The tool seems to be an increasingly popular choice for testing VDI storage, based on what we've seen at Storage Switzerland. Although I haven't researched it exhaustively, I've seen several published tests that used Login VSI, know of several others in the works, and am not aware of any complaints about its ability to simulate a real-world VDI environment. This simulation software, made by a company called Login VSI B.V., puts a simultaneous number of virtual desktop instances through the steps of doing things a typical user would do, including receiving, creating and sending emails; manipulating Word, Powerpoint and Excel documents; and creating Zip files. These are all done randomly at different times by each desktop, creating what would seem to be a worst-case scenario for the storage system. Another concern that comes up in tests like ours is disclosure. As regular readers of my column and the Storage Switzerland website know, we are often paid by storage vendors to create various forms of content. Velobit, the sponsor of the VDI lab report, and many other companies in the storage industry, are clients. LoginVSI B.V. is not a client of Storage Switzerland but it might be in the future. Without exception, all of those relationships have been accurately and repeatedly disclosed. Further, we accept no compensation from vendors based on the sale of products or on the success of their companies. As do most analysts, we accept compensation for the papers and lab reports we write. This was the case with VeloBit and it was properly disclosed. I believe that this allows us to provide an educational service to our readers in a balanced, reliable way. We've never received a complaint from a user on the reliability or accuracy of our reports or our level of disclosure. The VeloBit caching software we tested is one of a new breed of caching software products that use DRAM combined with software intelligence to oversubscribe memory for the purpose of delivering better performance than the classic 4-GB-per-desktop model. Our tests were an attempt to verify Velobit's particular product claims. In addition, we've spent countless hours researching the VDI storage market and will continue to do so. It's up to the reader to determine if this technology is worth consideration in their own VDI project. We believe our lab report has value in two areas. First, it points out an important new tool that both storage manufacturers and customers can use to test VDI scalability. But its primary value is that it introduces a new concept to solving the storage I/O problem. This concept should significantly reduce the storage cost of a VDI project -- making CapEx savings a potential factor along with operational savings.2013-04-16T13:00:00ZThe Role Of DRAM In StorageDon't discount DRAM. Thanks to increased server capacities and sophisticated caching technologies, DRAM can serve as a tier in the storage infrastructure.http://www.informationweek.com/storage/systems/the-role-of-dram-in-storage/240153010?cid=SBX_iwk_related_commentary_Data_protection_storageDRAM is faster and more durable than flash-based storage. But DRAM also has a tendency to lose data when it loses power, and it is more expensive than flash. This has made flash the "go-to" high-performance storage option, with DRAM storage options often shelved in favor of flash. But does DRAM still have a role in the storage infrastructure? The fact is, data center professionals still have a keen interest in DRAM, and their interest is on the rise. The 2011 Storage Switzerland article <a href=http://www.storage-switzerland.com/Articles/Entries/2011/3/29_The_Advantages_of_DRAM_SSD.html>The Advantages of DRAM-Based SSD</a> has more daily readers now than it did when it first appeared. What's driving the increased interest in DRAM as a storage solution? First, an increasing number of servers can support much higher RAM capacities than they did a few years ago. Even mid-range server hardware is typically able to hold more than 1TB of DRAM, and while that 1TB might cost 3 to 5 times more than flash storage, its performance capabilities are very attractive. Also, this storage area is directly accessible via the CPU slot, so there's no storage protocol interconnect to worry about -- in other words, the lowest possible latency. Second, there are several solutions to volatility and cost challenges that a DRAM-based storage solution presents. [ Are you paying too much for your VDI desktop devices? Read <a href="http://www.informationweek.com/storage/systems/what-should-a-vdi-desktop-cost/240152792?itc=edit_in_body_cross">What Should A VDI Desktop Cost?</a> ] Volatility can be addressed by making sure your servers don't lose power. Battery backup is a common solution for most mission-critical servers. If that's not enough, volatility can be overcome by implementing a capacitor in conjunction with DRAM, as discussed in <a href=http://www.storage-switzerland.com/Articles/Entries/2012/8/1_How_To_Design_A_Better_SSD_Storage_Appliance.html>this article</a>. In these cases, if power fails the capacitor charges the DRAM long enough for it to dump its contents to flash memory on the same DRAM circuit. Of course, neither of these solutions will protect you from a hardware lockup. The power-saving circuitry is not likely to kick in because there is still power to the server -- you just can't use it. If DRAM is being used to store writes, data could be lost. Thanks to virtualization, the risk is likely isolated to a single VM -- but some risk remains. The issue of cost can be overcome by using DRAM space efficiently for a cache or a tier within a caching architecture. A number of companies are already leveraging flash in this way. As discussed in <a href="http://www.storage-switzerland.com/Articles/Entries/2013/4/9_Making_Storage_the_VDI_Solution,_Not_the_Problem.html">this recent study</a>, companies that add duplication or compression technologies can minimize the cost disadvantage of DRAM. Additionally, deduplication should work even better in DRAM than on flash since the redundancy check occurs at memory speeds. It should also have greater value since the cost of DRAM is higher. These caching methods also help minimize data exposure caused by DRAM's volatility. If designed correctly, they should be able to flush the cache even if an individual VM locks up. Therefore, DRAM's role going forward is probably as a tier within a caching technology, where writes are sent to the DRAM area and coalesced prior to being written to the flash area. This will increase application response time and increase flash life expectancy. In fact, most SSDs use a small amount of DRAM as a buffer area anyway. As discussed <a href="http://www.storage-switzerland.com/Articles/Entries/2013/4/15_Improving_SSD_Performance_Through_Better_Flash_Management.html">in this article</a>, vendors are going to great lengths to ensure that data is correctly written and protected. DRAM is not dead. Thanks to the memory capacity of modern servers and the intelligence of next-generation caching solutions, DRAM has a significant role to play in the storage infrastructure. It is a role we expect to see expanding over the next few years. Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. <a href="http://www.interop.com/lasvegas/?_mc=MP_BTMEDIWKAXE">Register today</a>! 2013-04-12T10:25:00ZWhat Should A VDI Desktop Cost?A small business should be able to configure a viable 250 desktop VDI installation for less than $150 per desktop.http://www.informationweek.com/storage/systems/what-should-a-vdi-desktop-cost/240152792?cid=SBX_iwk_related_commentary_Data_protection_storageRecently I met with an organization's IT team after they had completed the initial rollout of their virtual desktop infrastructure project. The VDI install went quite well and desktop performance was by their description "acceptable." Although I am not convinced that the users think "acceptable" is, well, acceptable, but that is a story for another entry. What jumped out at me in our meeting was the cost per desktop. Based on the number of desktop instances they felt that their "pod" of host and storage could support, the average cost per desktop was about $500 to $600, prior to the actual device that sits on the desktop. Obviously the device has some bearing on the cost per desktop, but this company was counting on the Bring Your Own Device (BYOD) trend to alleviate all of those costs. However, the company reimbursed for the device brought in by employees, so it really should have been factored into the overall cost. But for now, let's leave the cost per desktop at $500 to $600. I think that cost is still way too high for broad adoption of VDI to make sense. What Is Driving The VDI Cost Per Desktop? The key factor that drives up the cost per desktop is simple. How many desktops can you support per host? Obviously 2,000 desktops per host is going to be far less expensive per desktop than 1,000. What is the big limiter to desktop density? Storage -- primarily storage performance, not storage capacity. Capacity issues, assuming your storage has the performance capabilities to handle the dynamic write nature of thin provisioning, golden masters and linked clones, have largely been solved thanks to the efficiency of these technologies. [ Learn more about this issue. Read <a href="http://www.informationweek.com/storage/systems/how-server-side-storage-memory-impacts-v/240150807?itc=edit_in_body_cross">How Server-Side Storage Memory Impacts VDI Costs</a>. ] It is important to note what is not the problem -- processing power. In almost every VDI environment we have studied there is more than enough processing power to increase desktop density. The bottleneck almost always is somewhere in the storage infrastructure. Storage performance is a problem because the higher the density of desktops the more random the I/O becomes. Also, as mentioned above, in a persistent desktop VDI strategy, the need to handle the dynamic writes caused by thin provisioning, golden masters and linked clones also puts stress on the storage infrastructure. This puts IT in an awkward position. They can either solve the storage performance problem with an expensive, high-end storage system or stand up more hosts as the bottleneck is reached. The most common is for IT to create "pods" for their VDI rollout. Each pod is a dedicated number of hosts connected to a dedicated storage device. The aggregate use of the CPUs in that pod is rarely above 25% but the storage system is running at full tilt. Adding more "pods" to increase the number of supported desktops becomes an expensive and inefficient solution that increases the cost per desktop. Ending The VDI Vs. Storage Battle The answer from the storage community has been to throw flash at the problem. And basically throw flash everywhere: in the server, in the network and on the storage system itself. We don't buy into this strategy of haphazardly applying flash technology, but each technique has an advantage and applying the right technique for your environment can make a significant difference and most importantly bring down the per desktop cost of a VDI infrastructure. As we saw in our recent lab report <a href="http://www.storage-switzerland.com/Articles/Entries/2013/4/9_Making_Storage_the_VDI_Solution%2C_Not_the_Problem.html">"Making Storage the VDI Solution, Not the Problem"</a>, where DRAM in the server was leveraged we projected a significant increase in the number of desktop instances per host, a reduction in the shared storage expense and a huge decrease in the cost per virtualized desktop. Based on that study and a few more that are still in the works we suggest that the new target per virtualized desktop should be less than $250 while delivering better performance than what is available to a standalone desktop. This type of technology makes VDI justifiable from a CapEx perspective in addition to its traditional strength as an operational saver. It also helps VDI make more sense for organizations that traditionally considered themselves too small for a VDI project. We project that a small business may be able to configure a viable 250 desktop VDI installation for less than $150 per desktop. In my next entry "Are You Delivering The Right VDI IOPs" we will examine the new reality in IOPs per desktop and see if "acceptable" really is acceptable.2013-04-01T09:06:00ZMeeting The MSP/CSP Storage ChallengeAs customers push for more storage and hosting solutions, MSPs and CSPs are turning to a software-defined approach to purchasing resources. Here's how it works.http://www.informationweek.com/storage/systems/meeting-the-mspcsp-storage-challenge/240151925?cid=SBX_iwk_related_commentary_Data_protection_storageManaged service providers (MSPs) and cloud service providers (CSPs) face a formidable task: to provide storage and backup services that are more efficient and more affordable than organizations can accomplish internally. That challenge puts MSPs on the leading edge when it comes to storage infrastructure and backup services design. The Online Backup Storage Challenge For most companies, online backup is the initial entry path to the cloud. It's also an excellent way for a legacy storage reseller to become a MSP. The challenge: traditional, off-the-shelf backup software and hardware is not well suited for a business model that's based on adding customers incrementally. The good news, from a software perspective, is that there are plenty of MSP-focused backup applications to consider -- as the number of MSPs grows, so does the number of backup solutions. The problem, however, is that the online backup provider subscriber base is evolving from consumers to businesses, and MSP-specific backup products don't have the robust features those businesses require. [ Data backup is just the beginning. Here are some more ways the cloud can benefit virtual infrastructures: <a href=" http://www.informationweek.com/storage/systems/3-ways-the-cloud-can-complement-virtuali/240146931?itc=edit_in_body_cross">3 Ways The Cloud Can Complement Virtualization</a>. ] As discussed in <a href="http://www.storage-switzerland.com/BTMSP5.html">this webinar</a>, many MSPs are looking to improve and replace their current offerings. Customers are pushing MSPs to meet new business demands caused by virtualization and increased use of databases in the SMB market. The Primary Storage Challenge Some data centers are willing to move beyond the cloud for secondary data and are leveraging CSPs to host their applications. Plenty of software-as-a-service (SaaS) solutions are available as well. These applications are hosted by MSPs/CSPs. The primary storage challenge is a big one for CSPs. They don't have the latency of the Internet to let them get away with "cheap and deep" storage, as backup providers might -- all the processing and related storage I/O happens locally. They need high-performance storage, but they need to buy it in a way that matches their business model -- incrementally, adding storage capacity and performance as they add customers. As discussed <a href="http://www.storage-switzerland.com/Articles/Entries/2013/3/19_Cloud_Defined_Storage.html">in this article</a>, CSPs can't afford to buy big storage up front and hope they can add customers fast enough to justify the purchase, so this incremental approach is ideal. Software-defined storage allows CSPs to converge their compute and storage infrastructure for a more natural upgrade. As they add customers, they add compute -- if they can place storage and data services inside the same nodes, their environment will scale automatically. Why Does This Matter To Users? There are two key reasons this software-defined storage approach important for businesses that might leverage an MSP/CSP. First, customers should understand how their MSP/CSP supports its service. Can it actually execute on its service level agreements (SLAs)? After all, if something goes wrong, an SLA might give you grounds for a lawsuit, but it won't get your data back or your application running. Second, MSP/CSPs are going to be the storage proving ground going forward. They are the companies that will stretch computing and storage to its limits. We will learn a lot of solid best practices from these organizations. Knowing how to push the envelope will help your organization meet its internal compute and storage challenges. Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. <a href="http://www.interop.com/lasvegas/?_mc=MP_BTMEDIWKAXE">Register today</a>! 2013-03-26T13:10:00ZDrive Form-Factor SSDs Make A ComebackLooking to improve server-side performance in your enterprise? Recent advances suggest it might be time to give drive form-factor SSDs another look.http://www.informationweek.com/storage/systems/drive-form-factor-ssds-make-a-comeback/240151686?cid=SBX_iwk_related_commentary_Data_protection_storageIn the enterprise, solid-state storage has become the go-to option for eliminating storage performance problems. One of the most obvious and common places to install solid-state storage is in the server itself. This puts the performance boost as close to the problem as possible and eliminates potential bottlenecks such as the storage network. The early choice for server-side SSD, as for shared storage, was to use drive form-factor SSDs. These SSDs look like hard disk drives (HDDs) but contain solid-state memory instead of rotating media. They installed into a conventional HDD expansion bay, so support and installation was easy. Drive form-factor SSDs gave way in large part to PCIe SSDs, which are PCIe boards with flash modules installed. PCIe SSD provided a performance advantage over drive form-factor SSD thanks to their direct connectivity to the PCIe bus and avoidance of the storage protocol stack. But PCIe SSDs have their own challenges. First, because they avoid the storage protocol stack, they require a customized driver and you can't boot from them. This means not all platforms are supported by every vendor, requiring at least two types of drives in the server -- typically HDD to boot and PCIe SSD for performance. Second, PCIe SSDs have a redundancy problem. Most servers don't have an abundance of free PCIe slots. Even if it is supported, installing multiple cards can be a challenge. Without multiple cards it's difficult to develop mirroring or RAID capability -- and for most servers, hot-swapping PCIe cards is out of the question. [ Server-side storage memory can offer a cost-effective solution to VDI projects. Here's how: <a href="http://www.informationweek.com/storage/systems/how-server-side-storage-memory-impacts-v/240150807?itc=edit_in_body_cross">How Server-Side Storage Memory Impacts VDI Costs</a>. ] Drive form-factor SSDs don't have these challenges. A good number of drive bays are usually available, and these drives can leverage the redundancy options built into the RAID controller they connect to. As discussed in <a href=http://www.storage-switzerland.com/BTPCIeSSD.html>this webinar</a>, the comeback of drive form-factor SSD is being driven by improvements in performance and smaller form factors, which leads to greater density. 6-GB (and soon 12-GB) SAS and SATA protocols can overcome much of the performance bottleneck. As <a href=http://www.storage-switzerland.com/Blog/Entries/2013/3/22_Video_Review_of_Sonnet_Tempo_And_Samsung_840_Pro_SSD.html>discussed here</a>, even 6-GB SATA is achieving 600+ MB performance per drive. With the appropriate RAID controller capable of driving a server full of these drives, 200k+ IOPS is very achievable. In addition, SSD manufacturers have been making great strides in performance and durability. Improvements in storage protocol performance, RAID controller processing power and in the devices themselves should encourage enterprises looking for server-side SSD to give drive form-factor technology a fresh look. Add in their easy availability, and these devices make a compelling case for companies looking to improve server-side performance. Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. <a href="http://www.interop.com/lasvegas/?_mc=MP_BTMEDIWKAXE">Register today</a>! 2013-03-20T13:22:00ZVirtual Machine Recovery Keeps ImprovingThanks to technologies like changed block recovery, downtime is becoming a thing of the past.http://www.informationweek.com/storage/disaster-recovery/virtual-machine-recovery-keeps-improving/240151269?cid=SBX_iwk_related_commentary_Data_protection_storageVirtualization and the cloud have brought significant change to the data center. Server resources are now better utilized and shared storage is better justified. But one of the most dramatic impacts of virtualization is how it has changed the recovery process -- what used to involve many hours and prayers is now nearly instantaneous and 100% accurate. Unlike rapid-recovery solutions of the past, this recovery is available and affordable for almost any environment. Assuming that your backup is good and you have a viable copy of data, the first challenge when recovering a failed application is the time required to transfer data across the network to the destination. Part of this time is eliminated by virtualization itself. You no longer need to find another server and configure it -- simply move the copy of the virtual machine (VM) back into place. In my opinion, this is all the justification needed to virtualize even the smallest data center. While virtualization makes it easier to know what to transfer, transferring that VM can still take some time, depending on its size and the speed of the network. Software developers have stepped in with in-place recovery solutions that allow the VM to run from the backup storage area (typically a disk backup appliance). [ Improved technology and falling costs are resulting in more VDI projects. Read more at <a href="http://www.informationweek.com/storage/virtualization/is-storage-saving-virtual-desktop-infras/240150414?itc=edit_in_body_cross">Is Storage Saving Virtual Desktop Infrastructure?</a> ] That means that within moments of a failure, a VM can be returned to operation without the need to wait for data to be transferred across the network. Executing backups from the backup appliance, however, could benefit from a second look at how the disk backup appliance is designed. In <a href="http://www.storage-switzerland.com/Articles/Entries/2013/2/28_Designing_Disk_For_VMware_and_Hyper-V_Backups.html">a recent article</a> I discussed how disk backup systems need to be able to respond to the changing state of backup. Thanks to the widespread use of incremental or changed block backups in VM data protection applications, backups happen more frequently and with much smaller files than in the past. Also, these backup appliances now may actually host a VM for a period of time. A second development is the concept of changed block recovery. Similar to changed block backups, changed block recovery restores only the data needed to make a VM look like it did prior to the failure. This type of recovery applies if, for example, the VM has failed due to data corruption, where the physical hardware (storage and server) are operational. Since most studies on the subject of downtime point to software rather than hardware as the primary cause, this is a common occurrence. With a changed block recovery capability, the backup administrator can select from within the backup application a version of the VM that exists prior to the time of the corruption. The changed block recovery technology in the backup software would then recover only the blocks of data that had changed since that point in time, cutting the amount of data to be transferred significantly. Changed block recovery is particularly important for backup products that leverage the cloud as a potential backup destination, as discussed <a href="http://www.storage-switzerland.com/Articles/Entries/2013/3/7_Cloud_Backup_Requires_Intelligent_Recovery.html">in this article</a>. While capabilities like compression and deduplication help tremendously with cloud-based backup, they have limited value in cloud-based restores. Changed block recovery can leverage what is already in place at your site -- i.e., data that does not need to be updated -- and recover only the data impacted by changes. Recovery has changed, and so have users' expectations. Most users these days don't expect to ever be down -- and if there is an outage, they don't expect it to last for long. Thanks to modern hardware and software, their expectations can be met, and in some cases exceeded. Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. <a href="http://www.interop.com/lasvegas/?_mc=MP_BTMEDIWKAXE">Register today</a>! 2013-03-14T11:00:00ZHow Server-Side Storage Memory Impacts VDI CostsServer-side storage memory can provide a cost-effective solution to VDI projects if you overcome challenges in mobility and data protection. Here's how.http://www.informationweek.com/storage/systems/how-server-side-storage-memory-impacts-v/240150807?cid=SBX_iwk_related_commentary_Data_protection_storageIn <a href="http://www.informationweek.com/storage/virtualization/is-storage-saving-virtual-desktop-infras/240150414">my last column</a> I posited that advances in storage technology -- mostly innovative use of memory-based storage -- is making virtual desktop infrastructure (VDI) projects more likely to generate a return on investment beyond just an operational one. Moving beyond operational VDI project justification is critical for the large-scale deployment of VDI projects. It is simply easier to justify to non-IT decision makers something that will save the organization dollars than it is to rationalize something that will save IT department time or increase security. I see three key areas where flash and DRAM (as storage) are being used to significantly increase virtual desktop density (which saves money and improves user acceptance by increasing performance): server-side storage memory, network caching and shared SSD appliances/arrays. In this column I'll discuss server-side storage memory, and I'll cover the other methods later. I'm avoiding the use of server-side flash intentionally. Much of the innovation we are seeing involves the use of DRAM as the first tier of caching for virtual desktop images. VDI typically has a very mixed read/write workload, and because DRAM is ideal for writes it is a perfect complement to VDI. [ For more on VDI and storage solutions, read <a href="http://www.informationweek.com/storage/virtualization/is-storage-saving-virtual-desktop-infras/240150414?itc=edit_in_body_cross">Is Storage Saving Virtual Desktop Infrastructure?</a> ] Thanks to the capacity-savings capabilities of the hypervisors, thousands of persistent desktop images can be stored in a very small storage space, which overcomes DRAM's cost challenges. But these capacity-saving techniques typically have a high level of latency caused by their need to dynamically allocate writes. DRAM's aforementioned write performance capability overcomes the write performance penalty of the capacity-saving techniques. Further, some products perform compression and/or deduplication in the RAM cache space itself, making RAM utilization even more efficient. The challenge with DRAM is its volatility. To maintain performance, these products must cache both reads and writes, which risks data loss until the write is flushed to permanent storage. This may be a generally acceptable risk since this is desktop data, but some users will likely push back. One solution is a non-volatile DRAM solution, as discussed in <a href=" http://www.storage-switzerland.com/Articles/Entries/2012/9/18_How_to_Solve_the_SSD_Endurance_Problem.html">this article</a>. A more common approach for users who don't want to take that risk is server-side flash, in either in PCIe or SSD form. As we'll discuss in the upcoming webinar <a href="http://www.storage-switzerland.com/BTPCIeSSD.html">Is PCIe Dead?</a>, while PCIe is considered the performance leader, drive form factor SSDs are gaining ground and certainly have a cost advantage. Challenges to Server-Side Storage Memory There are several challenges to server-side storage memory. First, it inherits all the challenges of any directly attached storage device. Data protection like RAID is not typically built in as it is on a shared storage system, and the capacity of the SSD or DRAM is isolated to the server in which it is installed. It might be too big or too small, and it can't be easily allocated to other servers. As mentioned above, however, the actual capacity needs per server should be relatively small, so this may not be a significant issue. The greater challenge is that VDI mobility is hampered. More software caching solutions are now integrated with the hypervisor so that they know to evict the cache prior to a VM migration. But this means that the Virtual Desktop may see decreased performance until its unique data is re-cached on the new server. How big a problem this is depends largely on how often you migrate virtual desktops between hosts. In my next column I'll discuss network-based caches and shared flash arrays, both of which overcome the challenges discussed here, but with the added cost of an appliance and/or storage system. They also, of course, add the potential latency of the network. I'll also provide some guidance on how to choose between the three options. Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. <a href="http://www.interop.com/lasvegas/?_mc=MP_BTMEDIWKAXE">Register today</a>! 2013-03-11T09:06:00ZIs Storage Saving Virtual Desktop Infrastructure?We're seeing a significant climb in VDI implementation as old storage performance and cost roadblocks fall.http://www.informationweek.com/storage/virtualization/is-storage-saving-virtual-desktop-infras/240150414?cid=SBX_iwk_related_commentary_Data_protection_storageWhile virtual desktop infrastructure (VDI) projects have been steadily increasing over the last few years, 2013 is seeing a significant spike upward in terms of both interest and actual implementation. There are two reasons for this. First, the cost of the client has come down while its capabilities have gone up. This is thanks to thin clients, tablets and thin laptops. The other reason for the increase in adoption is that storage roadblocks to successful adoption, like performance and cost, are quickly being minimized -- if not eliminated all together. In fact, a case could be made that the removal of these storage roadblocks is the top reason behind the increased adoption of VDI. One of the keys to VDI adoption, outside of the classic call center use case, has been the use of persistent desktops. This type of virtual desktop allows for users to customize their virtual instance like they used to be able to customize their physical system. It also allows them to add their own applications and utilities. [ Looking to update storage? See <a href="http://www.informationweek.com/storage/systems/why-flash-storage-excels-in-virtual-envi/240149952?itc=edit_in_body_cross">Why Flash Storage Excels In Virtual Environments</a>. ] The problem with persistent desktops is that, as a default, they require hard storage allocation, which of course is expensive and ruins much of the VDI return on investment (ROI). Most hypervisors have a method of dealing with this cost challenge by leveraging a common VDI image and then cloning it for each user. Each clone stores only the unique changes that each user makes and leverages the golden master for areas that desktops have in common, like operating system, core applications, etc. If this technique is used to its fullest extent, the cost challenge of storage in VDI is largely solved. The problem with the golden master and linked clone approach is that each write has to be dynamically allocated as the user is making changes to the persistent desktop instance. If you think about your use of a desktop, once you've booted the system, much of your work is creating or editing data. Also the user storage performance expectation is well beyond what the conventional wisdom suggests we budget for. While as a percentage there may not be more write traffic than read, the write traffic is quite high and each write has to be dynamically allocated to storage. Essentially writes are harder in a virtual world and, as we discussed in our article "<a href="http://www.storage-switzerland.com/Articles/Entries/2013/2/26_Using_Network_Caching_To_Solve_VDI_Storage_Problems.html">Using Network Caching To Solve VDI Storage Problems</a>," most storage systems give priority to write requests over read requests, so even if writes are lower as a percentage, their complexity impacts read performance. The initial response to this storage-performance challenge was to add solid-state storage to the system and move much (if not all) of the VDI environment to it. This is really a sledgehammer approach to the problem, since the entire VDI environment does not need to be on SSD all the time. Lately we have seen technologies that lead to a more balanced approach that can keep costs in line while delivering the performance that users have become accustomed to. These solutions range from using high amounts of DRAM in the VDI host, to using cost-effective deduplicated SSD appliances and arrays, to leveraging caching technologies in the server, network or storage system. My next entry will take a look at each of these to help you determine which might be the best for your environment. Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. <a href="http://www.interop.com/lasvegas/?_mc=MP_BTMEDIWKAXE">Register today</a>! 2013-03-05T09:06:00ZWhy Flash Storage Excels In Virtual EnvironmentsAll-flash systems do more than replace a lot of hard drives. They let you create extremely dense virtual server environments that eliminate physical server hosts.http://www.informationweek.com/storage/systems/why-flash-storage-excels-in-virtual-envi/240149952?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/7-cheap-cloud-storage-options/240134947"><img src="http://twimgs.com/informationweek/galleries/automated/905/01_Cloud_tn.jpg" alt="7 Cheap Cloud Storage Options" title="7 Cheap Cloud Storage Options" class="img175" /></a> <div class="storyImageTitle">7 Cheap Cloud Storage Options</div>(click image for larger view and for slideshow)</div>When we suggest that a company consider an all-flash storage system to solve a performance problem, there is almost always resistance to the cost. That is a fair concern -- it's common knowledge that flash storage is more expensive than hard disk storage. But this is only true when compared device to device. When they compare the cost against the full benefit of the system, we have seen many data centers become believers and actually save money by going with a flash appliance. The first thing to realize is that few if any flash array or flash appliance vendors really expect you to replace all of your storage with flash. And as we will discuss in our upcoming webinar, "<a href="http://www.storage-switzerland.com/BTSSDBudget.html">SSD on a Budget</a>," when used to augment existing storage, an all-flash system can be had for well under $100,000. Many all-flash vendors will try to steer the conversation away from cost per GB to cost per input/output operations per second (IOP) -- a good strategy for vendors given the tremendous advantage that flash systems have in IOPS. If you have an application, such as a database, that is specifically IOPS constrained then this might catch your attention. [ Want more on how flash storage has advanced? Read <a href="http://www.informationweek.com/storage/systems/flash-storage-ready-for-writes/240146606?itc=edit_in_body_cross">Flash Storage: Ready For Writes</a>. ] The traditional way of dealing with a storage performance problem is to create very high-drive-count arrays. The challenge with this approach, of course, is the sheer cost of deploying 100-plus hard drives to generate the right amount of performance. In addition, many data centers will only format the outer edges of those drives to make sure that data is landing on the fastest portion of the drive. This technique obviously wastes even more capacity and consumes even more power and budget dollars. All-flash arrays and appliances eliminate the high drive count array problem by moving the applications data to a high-performance, low-latency storage area that actually ends up being less expensive, requires less power and provides better performance. But this scenario is the low hanging fruit for an all-flash return on investment (ROI). Where all-flash systems can make a real difference to the broader set of data center applications is in virtual desktop or virtual server environments to increase virtual machine density. High VM-count physical servers create a lot of random storage input/output traffic, a traffic pattern that memory-based storage is well able to handle. As we discussed in our article "<a href=http://www.storage-switzerland.com/Blog/Entries/2012/4/13_Optimize_The_Storage_Infrastructure_For_Increased_VM_Density.html>Optimize The Storage Infrastructure For Increased VM Density</a>," flash storage systems are ideally suited to solve the performance challenges caused by dense environments; RAM, storage network I/O and storage device latency. The real value is that flash solves these problems to enable a dense VM-to-host ratio. As a result, the ROI becomes more than just how many hard drives you can replace. Instead, all-flash systems enable the creation of extremely dense virtual server environments that eliminate many of the physical server hosts that would be required. Because these hosts can start at $16,000 and cost well over $50,000, the elimination of as few as two or three can easily pay for the all-flash investment. Server and desktop virtualization, though, are not the only places that an all-flash investment can easily pay for itself. Databases, as mentioned, also can benefit from the implementation of this type of device. By putting the entire database, not just hot files, on flash the number of users can be increased significantly. This again eliminates some of the additional database host servers that would be required to keep application response time acceptable to users. Finally, a database environment that leverages all-flash can also respond more quickly to large batch processing requests such as analytical reports. The end result would mean greater and more rapid decision-support capabilities derived from the goldmine housed within databases. The ROI of flash is essentially all about density. How many more virtual machines can be supported, and how many more users can be supported on a per-host basis? Although the elimination of unneeded hard drives is important and certainly can save power, the elimination of unneeded hosts can save even more dollars and potentially even more power. Attend Interop Las Vegas, May 6-10, and attend the most thorough training on Apple Deployment at the NEW Mac & iOS IT Conference. Use Priority Code DIPR03 by March 9 to save up to $500 off the price of Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 350+ exhibiting companies, and the latest technology. Register for <a href="http://www.interop.com/lasvegas/?CID=MP_ILV_IWK_Article_TL&_mc=DIPR03">Interop</a> today!2013-02-21T08:36:00Z4 Ways $50K Of Flash Can Boost PerformanceYou want to use solid state storage to boost your network's performance. Here's advice on adding SSDs without breaking the company.http://www.informationweek.com/storage/systems/4-ways-50k-of-flash-can-boost-performanc/240148993?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/7-cheap-cloud-storage-options/240134947"><img src="http://twimgs.com/informationweek/galleries/automated/905/01_Cloud_tn.jpg" alt="7 Cheap Cloud Storage Options" title="7 Cheap Cloud Storage Options" class="img175" /></a> <div class="storyImageTitle">7 Cheap Cloud Storage Options</div>(click image for larger view and for slideshow)</div> Almost every IT professional I talk to wants to use solid state drives (SSDs) in some way to solve performance problems. The most common culprit is the random I/O caused by server or desktop virtualization. At the same time, this same group has to stick to a budget, so they are looking for the most cost-effective way to deploy SSDs. Budget constraints typically rule out replacing existing storage with a flash-enabled storage system or an all-flash array -- but not always. Next time we'll look at how an ROI can be developed to justify that investment. In the meantime, let's say you have $50,000 to augment your existing storage with SSD. Here are four possible money-saving implementations that we'll also discuss in my upcoming webinar <a href=http://www.storage-switzerland.com/BTSSDBudget.html>"SSD on a Budget?"</a> 1. SSD Appliances. A solid-state appliance typically is an all-flash shared storage device that has limited data services. Instead it's usually focused on delivering performance. This lowers costs, but at the expense of a feature-rich array and the increased complexity of managing another silo of storage. [ Afraid of losing data during cache writes? Read <a href="http://www.informationweek.com/storage/data-protection/3-ways-to-protect-cache-writes/240148369?itc=edit_in_body_cross">3 Ways To Protect Cache Writes</a>. ] Virtualized environments such as VMware help resolve these shortcomings. As I wrote in <a href="http://www.storage-switzerland.com/Articles/Entries/2013/2/4_Preparing_Storage_For_Increased_VM_Density.html">"Preparing Storage For Increased VM Density,"</a> with the right tools you can easily identify sluggish VMs and migrate them to a high-performance SSD. Hypervisors like VMware can also provide much of the data services -- thin provisioning, snapshots and replication -- that these appliances lack. 2. Network Caching Appliances. Another option similar to SSD appliances is the network caching appliance. As I wrote in <a href=http://www.storage-switzerland.com/Blog/Entries/2013/1/18_NAS_Acceleration_Is_More_Than_Just_SSD.html>"NAS Acceleration Is More Than Just SSD"</a>, these devices sit in front of existing SAN or NAS systems and optimize performance by caching hot data to an SSD area on the appliance. In many cases they also optimize network I/O and other operations as well. These devices have the advantage of not requiring manual movement of VMs, but they have the disadvantage of potential cache misses, which SSD appliances do not because the entire VM's image is statically placed on them. 3. Server-Side PCIe Solid-State Devices. Yet another option is to use PCIe solid-state devices either for virtual memory swap space or for local caching. These devices are ideal if you have a few problematic hosts that can use a performance boost. They're also great if you can't upgrade or optimize the storage network overall because of budget concerns. In these situations PCIe SSDs offer the surgical quick strike that many environments need. There are problems, though, with PCIe SSDs. First, if you have a dozen hosts with performance problems, this can be an expensive solution. Second, if you're using the device for caching reads or read/writes, VM migration can cause challenges that many caching software solutions are not prepared to manage. Finally, it's more expensive and sometimes more complex to provide PCIe SSD high availability (HA). You'll have to work around all of these issues. 4. Server-Side SSD. Finally, you have the option of the drive form factor of solid state devices. SSDs provide the same network elimination benefits that PCIe SSD do without the HA complexity. Building a RAID from what are essentially drives is relatively easy. There are no special drivers to load because the operating system recognizes the drives as storage devices. Also, solid state drives are far less expensive than PCIe SSDs. The apparent shortcoming of SSD when compared to PCIe SSD is the latency of the storage protocol stack and its raw performance. Modern SSDs are closing the gap, though, by increasing their performance and using the latest SAS connectivity. Which One? Which setup is best for you? It depends on your environment and IT capabilities. A shared scenario, such as an SSD appliance or network caching appliance, boosts speed across multiple storage devices and connected hosts. The PCIe or solid state drive scenarios are ideal for helping a few hosts struggling with a performance problem. Attend Interop Las Vegas, May 6-10, and attend the most thorough training on Apple Deployment at the NEW Mac & iOS IT Conference. Use Priority Code DIPR02 by March 2 to save up to $500 off the price of Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 350+ exhibiting companies, and the latest technology. Register for <a href="http://www.interop.com/lasvegas/?CID=MP_ILV_IWK_Article_TL&_mc=DIPR02">Interop</a> today!2013-02-13T09:06:00Z3 Ways To Protect Cache WritesHere's how to protect your storage from three situations that could cause it to lose data.http://www.informationweek.com/storage/data-protection/3-ways-to-protect-cache-writes/240148369?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/7-cheap-cloud-storage-options/240134947"><img src="http://twimgs.com/informationweek/galleries/automated/905/01_Cloud_tn.jpg" alt="7 Cheap Cloud Storage Options" title="7 Cheap Cloud Storage Options" class="img175" /></a> <div class="storyImageTitle">7 Cheap Cloud Storage Options</div>(click image for larger view and for slideshow)</div> Recently I discussed the importance of <a href=http://www.informationweek.com/the-vmware-write-cache-challenge-solved/240147594>write caching</a> in a VMware environment. A key issue when addressing write caching is protecting the cache until data is written to the hard disk. Although protecting the data in the cache is always important, cache safety is especially important when caching writes. A write cache acknowledges the write before it is safely committed to hard disk. In write I/O-intensive environments, the cache always has data that has been acknowledged to the application but is not safely written to hard disk. If the cache storage area fails, this data could be lost and corruption might ensue. The key three circumstances from which you should protect a write cache are: power failure, cache device failure and server failure. [ Why run two backups, one for virtualized servers and one for non-virtualized, when one backup will do? Read <a href="http://www.informationweek.com/storage/systems/the-virtualized-backup-gap/240147628?itc=edit_in_body_cross">The Virtualized Backup Gap</a>. ] 1. Power Failure. Power failure was a bigger concern when caches were mostly made out of server DRAM, but as I discussed in my article <a href=http://www.storage-switzerland.com/Articles/Entries/2011/10/31_The_need_for_Server_Grade_SSDs.html >"The Need For Server Grade SSDs,"</a> most flash devices also use a small amount of DRAM to organize inbound data. If the device loses power, data in that DRAM area could be lost. The DRAM area in flash devices should leverage capacitors to charge DRAM long enough so that the data it stores can be flushed to the flash area of the device prior to the drive shutting down. There also has to be some intelligence in the drive to sense a power loss and take this corrective action. 2. Cache Device Failure. Failure of the entire flash device can be a larger problem because many caches often are built using a single drive to keep down costs. A write-cached environment should provide greater redundancy. For server-side write caching, consider drive mirroring or PCIe card mirroring. Caches built from shared storage systems will leverage the mirroring or RAID built into the storage system. 3. Server Failure. Server failure normally is not a caching problem, because the server hardware would typically be returned to operation and caching should pick up where it left off. In a virtual server environment, though, it is entirely possible to restart a virtual machine on another host if the primary host failed. If the caching was server side, the write data in the cache would need to be flushed somehow prior to the VM being restarted elsewhere. To protect against uncommitted writes held captive in a failed server, you must get those writes outside of the host. This can be done by either mirroring the cache externally to a shared solid state device or by mirroring one server's cache to another server. Although both of these methods do introduce some latency because a network of some sort has to be traversed, they should still provide better write performance than hard disks, and reads would still be serviced from within the local physical host. Flash storage has made it to a point of reliability that the technology is able to sustain the higher write traffic of write-back caching, and the surrounding technology has improved to make sure that the cache is protected in case of a device or server failure. With proper design, writes can be safely cached and both sides of the I/O equation can benefit from memory-based storage. Our four business scenarios show how to improve disaster recovery, boost disk utilization and speed performance. Also in the new, all-digital <a href="http://www.informationweek.com/gogreen/120312smb/?k=axxe&cid=article_axxt_os">Storage Virtualization Gets Real</a> issue of InformationWeek SMB: While Intel remains the biggest manufacturer of chips in the world, the next few years will prove vexing for the company. (Free registration required.)2013-02-05T09:06:00ZThe Virtualized Backup GapMost enterprises run two backup systems -- one for virtualized servers and one for non-virtualized servers. New products in the works should resolve that problem.http://www.informationweek.com/storage/systems/the-virtualized-backup-gap/240147628?cid=SBX_iwk_related_commentary_Data_protection_storageIn many ways, server virtualization has helped data protection. Servers are now encapsulated into a single file instead of thousands of files, capabilities like changed-block tracking have reduced the amount of data that needs to be moved to the backup target and new features like recovery-in-place or changed-block recovery promise to dramatically reduce the amount of time it takes to recover a server instance. But there is a gap in protection when you deal with the reality that most data centers are not anywhere close to 100% virtualized. Depending on which study you read, most data centers are less that 50% virtualized. That means that more than half of the server population is still basically a single application running on a single physical server. In my experience, many of these servers are often not virtualized because they run some mission-critical application that is resource intensive and there is concern about placing that application in the highly shared virtual environment. While legacy backup systems are gaining some of the features that virtualization-specific backup products have, they still lag in this area. At the same time, most of the virtualization-specific applications do not provide important legacy capabilities like complete application protection, robust data retention and tape support. The result is customers end up running at least two backup applications in their environment. In fact, even in 100% virtualized environments, we have seen many data centers running two applications -- virtualization-specific backup for day to day protection and a legacy application to be able to create a robust archive of the virtual environment as well as to get a copy of it on tape. The Solution To The Backup Gap Running two or more separate backup applications to meet the needs of the enterprise is not looked on as ideal. As we discussed in a recent webinar <a href="http://www.storage-switzerland.com/BT4EntBackup.html ">"What's Breaking Enterprise Backup and How to Fix it"</a>, there are three viable ways to address the gap in capabilities of virtualized and non-virtualized products. First, you can choose to wait for the virtualization backup software to add legacy features like protecting non-virtual machines and providing support for tape libraries. Second, you can choose to wait for legacy backup applications to add the features that have made virtualization-specific backup applications so popular. Finally, you can choose to wait for legacy backup applications to develop an API-like capability that would allow VM specific backup applications to plug into their enterprise features. As you can see each of these options require some form of waiting -- but several vendors are closing the gap. We have seen several virtualization-specific backup applications add support for non-virtualized systems and a few have announced plans for tape support. We have also seen many of the enterprise backup applications add features that were once only available from virtualization specific backup applications. You can talk to these vendors to see what is available now and what their roadmaps look like and get a sense of who can get you to a single backup strategy the soonest. The third option, the API or modularization approach, makes the most sense for the long term. As we discussed in our recent article <a href="http://www.storage-switzerland.com/Articles/Entries/2012/5/10_Enterprise_Backup_is_Broken.html">"Enterprise Backup is Broken"</a>, backup software vendors have to come to the realization that no one vendor can do it all. The data center would be much better off if legacy enterprise backup systems allowed smaller point products to integrate into their legacy capabilities. We would have unified and best of breed data protection at the same time. In my next column I'll discuss how -- if backup software vendors are not careful -- the disk backup appliance may end up being the central point of consolidation. Already today multiple backup applications can send data to a single disk backup appliance and disk backup appliance vendors are very focused on providing tighter integration with a broad range of backup software solutions.2013-01-31T15:38:00ZThe VMware Write Cache Challenge, SolvedServer-side write caching can help improve storage network performance. Here's what you need to know.http://www.informationweek.com/the-vmware-write-cache-challenge-solved/240147594?cid=SBX_iwk_related_commentary_Data_protection_storageImplementing solid state disk (SSD) in a VMware host server has become a popular option to maximize the return on solid state investment. Server-side SSD eliminates potential bottleneck of the storage network and places high-performance storage directly in the hosts that need it most. Caching is the most common choice to add to the server-side SSD to automate the use of this high-performance tier. Caching automatically moves the most active portion of data from the SAN or local DAS storage and into the SSD in the server. This means that subsequent reads actually come from the SSD in the server, not from mechanical hard drives on the other side of the storage network. Most caching solutions that operate in the VMware environment are read only, meaning that the write portion of I/O drops through the cache and onto the intended storage medium -- in most cases, a shared hard disk. The good news is that most virtualized server environments are slightly more read-heavy; 60% reads are common. Also, when those writes do occur, they essentially get unfettered access to the storage network, so, in theory, write performance should improve even with read-only caching. [ Flash storage is ready for both sides of the I/O stream. Read more at <a href="http://www.informationweek.com/storage/systems/flash-storage-ready-for-writes/240146606?itc=edit_in_body_cross">Flash Storage: Ready For Writes</a>. ] Still, given those numbers, 40% of I/O is still a significant amount of writes, and writes are generally slower. As discussed in "<a href="http://www.storage-switzerland.com/Articles/Entries/2012/12/11_Can_You_Trust_VDI_Storage_Benchmarks.html">Can You Trust VDI Benchmarks?</a>" after the initial boot storm is complete, most virtual desktop infrastructures (VDI) become more write-heavy. In other words, write performance is important to both virtual server and virtual desktop environments. How Caches Write To understand the challenge with write caching, it's helpful to compare the processes. Without caching, when an application needs to write data, that application sends the write to the locally or SAN-attached storage. The application then waits for an acknowledgement from the storage device that the data has been safely written before moving on to the next operation. A read cache, similar to no caching, passes the write directly to the hard disk. Alternatively, a read cache can capture the write while simultaneously writing it to the hard drive. This cache alternative is known as write-through caching, and has the advantage of pre-caching popular data. The theory being what was most recently written is most likely to be read next. This technique helps get the right data in the cache sooner for more accurate read results, but it does not help with write performance. In both cases, though, the data is safe on hard disk and the cache is always in sync with it. The Write Risk In both of these read-cache implementation methods, there is little risk of data loss if the cache fails. VMware does change how read caches should be repopulated, but no data should be at risk. Write caching, known as write-back caching, stores the application write request in the cache and acknowledges a successful write prior to that data being stored on the hard disk. There is no latency waiting for the hard disk to receive data. While there can be tremendous performance advantages to server-side write caching, there's risk as well. For example, if there is a cache or server failure, then data will be lost. In a VMware environment, there is also the issue of how to handle the virtual machine migration. In my next column I'll discuss some of the ways that vendors are working around the write risk to provide high performance in a reliable fashion. Our four business scenarios show how to improve disaster recovery, boost disk utilization and speed performance. Also in the new, all-digital <a href="http://www.informationweek.com/gogreen/120312smb/?k=axxe&cid=article_axxt_os">Storage Virtualization Gets Real</a> issue of InformationWeek SMB: While Intel remains the biggest manufacturer of chips in the world, the next few years will prove vexing for the company. (Free registration required.)2013-01-24T13:27:00Z3 Ways The Cloud Can Complement VirtualizationVirtual infrastructures -- what vendors like to call internal clouds -- can benefit from the public cloud in ways beyond data backup.http://www.informationweek.com/storage/systems/3-ways-the-cloud-can-complement-virtuali/240146931?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/10-cloud-computing-pioneers/240142397 "><img src="http://twimgs.com/informationweek/galleries/automated/909/01_cloud_gurus_tn.jpg" alt="10 Cloud Computing Pioneers" title="10 Cloud Computing Pioneers" class="img175" /></a> <div class="storyImageTitle">10 Cloud Computing Pioneers </div> (click image for larger view and for slideshow)</div>Most IT professionals correctly refer to the internal cloud as their virtualization project, and often roll their eyes when a vendor refers to it as a cloud. Nevertheless, virtual infrastructures can benefit from the actual public cloud to augment what they are doing. The public cloud can help provide smaller businesses a place to send data for off-site backup storage, a place to recover in the event of a disaster, and even a place to move applications when peak loads have exceeded the capabilities of the internal data center. Let's look at the possibilities. 1. Public Cloud As Off-Site Backup. One of the earliest and most frequent uses of cloud storage is as a backup target. First, it was used to back up laptops and desktops, but now it is being used to protect physical and virtual servers. Using cloud storage creates an ideal separate location from the data center, and most providers have multiple geographically dispersed facilities as well. The farther your data is away from you the better protected it is from a regional disaster. [ Here's how one organization is moving every operation possible to the cloud. <a href="http://www.informationweek.com/smb/hardware-software/how-the-uso-makes-most-of-cloud/240144222?itc=edit_in_body_cross">How The USO Makes Most Of Cloud</a>. ] The challenge, though, for many virtual backup processes is that the use of cloud storage is not integrated directly into the virtual backup software. It has to be added on as an afterthought via some sort of gateway appliance. This separation reduces efficiency and increases complexity. As we will discuss in our upcoming webinar, <a href="http://www.storage-switzerland.com/BTCldSMBVM.html">"Using The Cloud To Solve the SMB VMware Backup Problem,"</a> integration with the cloud is critical for virtualized backup applications going forward. It should be just another checkbox within the backup software. 2. Public Cloud As Disaster Recovery Site. Several data protection suppliers and cloud providers are beginning to leverage the fact that they have your virtual machine images in the cloud to provide disaster recovery in their facility. As I discussed in the article <a href="http://www.storage-switzerland.com/Articles/Entries/2012/10/31_Is_Hybrid_Cloud_DR_The_Next_Step_in_Small_to_Mid-sized_Company_Data_Protection.html">"Hybrid Cloud DR"</a>, this capability is ideal for the small- to medium-size business that does not have a fully staffed second site that can be used for disaster recovery. It allows cloud providers to add value by becoming more than just a storage tank at the end of an Internet connection -- they are providing full-service recovery. 3. Public Cloud As Bursting Option. Finally, you can use the cloud for more than just backup and disaster recovery. You can use it as a destination point for applications when peak loads hit the data center. The concept, called cloud bursting, allows for additional instances of an application that can be started in the cloud or less active applications can be moved to the cloud temporarily to accommodate a peak load. The challenge has been how to integrate the move to the cloud with the existing virtualization investment. Bridging the gap between a local hypervisor and a cloud compute environment can be complex and expensive. But help is on the way. Companies such as <a href="http://hotlink.com/index.html ">HotLink</a> are providing the capability to integrate multi-hypervisor management, including cloud compute, into the vCenter infrastructure. The cloud can offer many benefits to data centers of all sizes, but its benefits can be especially useful to the medium-size business. These data centers need to start considering the cloud for more than off-site data storage. Cloud-hosted disaster recovery and bursting are now within the reach of these companies. Our four business scenarios show how to improve disaster recovery, boost disk utilization and speed performance. Also in the new, all-digital <a href="http://www.informationweek.com/gogreen/120312smb/?k=axxe&cid=article_axxt_os">Storage Virtualization Gets Real</a> issue of InformationWeek SMB: While Intel remains the biggest manufacturer of chips in the world, the next few years will prove vexing for the company. (Free registration required.)2013-01-19T09:06:00ZFlash Storage: Ready For WritesAdvanced over-provisioning and write modification technology have made flash storage solutions ready for both sides of the I/O stream.http://www.informationweek.com/storage/systems/flash-storage-ready-for-writes/240146606?cid=SBX_iwk_related_commentary_Data_protection_storageMost data centers have embraced flash-based solid state storage to address storage performance problems, but this adoption has primarily focused on read performance issues. Solid state disks (SSD) and solid state systems (SSS) are now ready for the other half of the I/O problem: writes. Write avoidance remains a goal for most solid state storage solutions because flash memory's life expectancy is based on the number of writes -- the more a solid state device is written to, the faster it wears out. To read data from a flash device, you must write some data to it at least once. SSDs and SSS have, at a minimum, some technology to spread out writes evenly across the available flash cells. This technique, called wear leveling, is commonplace today. Most solutions also over-provision flash memory to increase overall lifespan. For example, a drive might add 25% or more flash storage to allow the flash controller to spread writes across more flash cells. The problem with this is that vendors don't give away this unused flash for free, so the more they provision, the more you pay for a flash solution. [ High-availability appliances combined with software-defined storage could prolong the life of your storage system. Read more at <a href="http://www.informationweek.com/storage/systems/the-impact-of-high-availability-flash-ap/240146032?itc=edit_in_body_cross">The Impact Of High-Availability Flash Appliances</a>. ] Wear leveling and over-provisioning are basic table stakes in the flash market today. As I discussed in <a href=http://www.storage-switzerland.com/Articles/Entries/2012/11/27_One_Server_SSD_For_The_Enterprise.html>this recent article</a>, companies have begun to extend these basic capabilities to include the ability to customize the over-provisioning setting. For example, if you are installing an SSD into a very high-write environment, you could sacrifice some useable capacity to get higher life expectancy. SSD vendors are also adding the ability to adjust the energy charge used to write data to the flash. The softer the charge, the easier it is on the flash cell and the longer it should last. When there is not a lot of write queue depth, they can essentially slow the write down and extend the overall flash life without impacting performance. As I discussed in <a href=http://www.storage-switzerland.com/Articles/Entries/2012/12/5_Leverage_DRAM_to_Fix_Flash_Endurance.html>this article</a>, vendors can also integrate DRAM, which does not have a write penalty, to help extend the life of flash memory. With this technique, flash memory is front-ended by capacitor-charged DRAM. Writes are sent to the DRAM buffer first, which speeds performance and allows the write to flash to be organized for maximum efficiency, resulting in fewer wasted flash writes. The combination of advanced over-provisioning and write modification has led to flash-based devices that claim 5 to 20 drive fills per day for five years. For the data center, that is certainly a practical life span. All this investment in making flash more write-worthy is working: flash is ready for write traffic. As I covered in <a href="http://www.storage-switzerland.com/BT4RedSSD.html">this recent webinar</a>, many virtual workloads, especially virtual desktops, have a significant write I/O demand. Caching or storing these writes in flash can significantly improve performance as well as virtual machine density. Now is the time for a shift in how and why SSS is used. Advancements in technology have improved the write-worthiness of the solutions, and they are ready for both sides of the I/O stream. Our four business scenarios show how to improve disaster recovery, boost disk utilization and speed performance. Also in the new, all-digital <a href="http://www.informationweek.com/gogreen/120312smb/?k=axxe&cid=article_axxt_os">Storage Virtualization Gets Real</a> issue of InformationWeek SMB: While Intel remains the biggest manufacturer of chips in the world, the next few years will prove vexing for the company. (Free registration required.) 2013-01-15T12:31:00ZTrue Cost Of Open Source Storage SoftwareSome open source storage software is free, but what will support cost you?http://www.informationweek.com/storage/systems/true-cost-of-open-source-storage-softwar/240146328?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/7-cheap-cloud-storage-options/240134947"><img src="http://twimgs.com/informationweek/galleries/automated/905/01_Cloud_tn.jpg" alt="7 Cheap Cloud Storage Options" title="7 Cheap Cloud Storage Options" class="img175" /></a> <div class="storyImageTitle">7 Cheap Cloud Storage Options</div>(click image for larger view and for slideshow)</div> Open-source storage software is software that is available for download, typically at no cost, that can provide valuable data services to traditional storage hardware. These services include features we have grown accustomed to, such as thin provisioning, snapshots and cloning. Prior to open-storage software, these services typically came with the storage array that you purchased and were specific to that vendor's products. Open-source storage software offers the advantage of letting you use commodity storage hardware. As I discussed in my article <a href="http://www.storage-switzerland.com/Articles/Entries/2012/3/21_The_Value_Of_Open_Source_Storage_Software_In_The_Enterprise.html">"The Value Of Open Source Storage Software In The Enterprise,"</a> open-source storage software is moving out of the hobbyist stage and into large production data centers. An open- source software solution is available for a variety of storage infrastructure use cases. There are open-source filers, open-source block storage solutions, open-source object storage solutions, and in many cases there are several options to choose from. [ Need more tips on choosing storage products? Check out <a href="http://www.informationweek.com/storage/systems/storage-software-vs-hardware-whats-more/240006946?itc=edit_in_body_cross">Storage Software Vs. Hardware: What's More Important? </a> ] The key benefits to open-source software are its affordability and flexibility. It is essentially available for free plus support costs and can use almost any storage available. There are a variety of purchase options. You can go pure open source and seek help from the community when needed. You can go with a company that provides the open-source storage software and support for a fee. Or, you can go with the growing number of vendors that provide the software, enhance it beyond what is available in open source, and provide support. Total Cost Of Free Storage Software Open source has some hidden costs that you need to be aware of. It might be free or nearly free, but the support costs are not. Although support fees on open-source storage software are in line with turnkey storage vendor support, they do need to be factored into the overall cost. The second cost factor is the time it takes to resolve an issue. Some of the organizations that provide after-market support on these solutions have little advantage over you in accessing advanced technical help. They are mostly counting on their own internal storage skill set. Also keep in mind that the supporting storage can be more challenging because it has to cover a wider range of variables. A few organizations do have source-code-level understanding of the product they support, and they can actually go in and fix or work around a bug. This is unique in open source. In my years of running support organizations, the ability to access the source code could have made life much easier. Also, there are several vendors who add value to open source by enhancing the software. This can include integration to operating systems, hypervisors or applications. It also can come in the form of better storage components that outperform what the open-source software can do. A good example is ZFS, which I discuss in the article <a href=http://www.storage-switzerland.com/Articles/Entries/2012/11/13_VM_Aware_Vs._ZFS_Storage.html>"VM Aware Vs. ZFS Storage"</a>. Many vendors use ZFS as the foundation of their storage system, but then add or enhance its core capabilities. Vendors have enhanced the SSD integration, deduplication and even the metadata management of ZFS. These enhancers all are excellent additions to the community but they do come at a cost, and in many cases that cost is buying the hardware from that vendor. There is also the cost of the physical storage hardware. Depending on the software product, this often means still purchasing a shareable storage array. And you want a device that is built well and will perform well. That costs money. Even if you are looking at a storage software solution that can cluster and share direct attached storage across standard compute hosts, the quality of the hard drive does matter. In the end, open storage software is not free, but it is affordable. You need to calculate what these savings will be vs. other competitive, turnkey storage systems on the market. A key factor will be the skill set of the staff. If there is a deep technical storage bench, that can go a long way to keeping the total cost of open source storage software to a minimum. Our four business scenarios show how to improve disaster recovery, boost disk utilization and speed performance. Also in the new, all-digital <a href="http://www.informationweek.com/gogreen/120312smb/?k=axxe&cid=article_axxt_os">Storage Virtualization Gets Real</a> issue of InformationWeek SMB: While Intel remains the biggest manufacturer of chips in the world, the next few years will prove vexing for the company. (Free registration required.)2013-01-10T12:58:00ZThe Impact Of High-Availability Flash AppliancesHigh-availability flash appliances used with software-defined storage can prolong the life of your current storage system while meeting performance demands.http://www.informationweek.com/storage/systems/the-impact-of-high-availability-flash-ap/240146032?cid=SBX_iwk_related_commentary_Data_protection_storageSolid state disk (SSD) storage comes in many forms. It can be placed in the server, in the storage system and on the network. 2012 saw the maturation of one of the oldest categories in solid state: the flash memory storage appliance. Thanks to high-availability hardware and the maturation of storage software, these appliances may now move from niche use to broad adoption. As I discussed in <a href="http://www.storage-switzerland.com/Blog/Entries/2012/12/31_Flash_Arrays_vs_Flash_Appliances.html">a recent article</a>, the memory-based storage appliance has evolved into a system that is designed to accept flash in its most optimal form: a memory module instead of a disk drive form factor SSD. These appliances allow flash memory to achieve its maximum potential without supporting legacy hard drive technology. While they often sacrifice data service features, the devices focus on high-performance data movement into and through the appliance. They often also deliver very dense form factors that consume less overall space and power, therefore reducing costs. Last year's key addition to these hardware designs was high availability, meaning that the systems no longer have a single point of failure. Previously, flash appliances had to be bought in pairs and mirrored or used only for temporal data. [ For more on software-defined storage, see <a href="http://www.informationweek.com/what-is-software-defined-storage/240145344?itc=edit_in_body_cross">What Is Software-Defined Storage?</a> ] Mirroring a flash appliance has its own issues. First, flash appliances are priced at a premium -- buying two can be painful. Second, many operating systems, hypervisors and databases don't have built-in mirroring capability. Enter high availability (HA). Companies like Astute Networks, Violin Memory Systems and Texas Memory Systems (now owned by IBM) created devices or configurations designed with no single points of failure, just like their enterprise storage brethren. For only a slightly higher cost, you can buy one unit and get all the availability you need. HA positions these devices to be more than just solutions to niche problems -- they can become the location for the active primary storage dataset. However, they lack the data services provided by most enterprise storage systems, which flash array vendors claim as the differentiation point. Enter software-defined storage. The fact that most flash appliance vendors do not offer data services like thin provisioning, snapshots and cloning has become less of a problem as these services increasingly become available through software. In fact, a hypervisor like VMware can even help facilitate the movement of active data to these appliances by way of Storage vMotion. The combination of HA flash appliances and the maturation of the software-defined storage market has opened up new opportunities for these vendors, making them a viable alternative for users considering a system upgrade. Now users can install these systems as a complement to their current storage system, changing its role to delivering capacity and letting the flash appliance handle performance. The result is a longer life for legacy storage hardware that still keeps up with performance demands. Our 2012 State Of Servers report takes a look at three major technology trends emerging from our latest survey. Also in the new issue of <a href="http://www.informationweek.com/gogreen/121712it/?k=axxe&cid=article_axxt_os">IT Trends</a>: Performance and endurance gains plus lower cost give multilevel cell flash the edge over expensive single-level cell. (Free registration required.)2013-01-08T09:06:00ZPlot An Effective Data Archive StrategyGood archiving systems should do three things: Save money, let you find data fairly quickly and last a long, long time. http://www.informationweek.com/storage/systems/plot-an-effective-data-archive-strategy/240145714?cid=SBX_iwk_related_commentary_Data_protection_storage<div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/7-cheap-cloud-storage-options/240134947"><img src="http://twimgs.com/informationweek/galleries/automated/905/01_Cloud_tn.jpg" alt="7 Cheap Cloud Storage Options" title="7 Cheap Cloud Storage Options" class="img175" /></a> <div class="storyImageTitle">7 Cheap Cloud Storage Options</div>(click image for larger view and for slideshow)</div> An effective archiving or data retention solution should do three things. It should let you store less-frequently-accessed data at a lower cost than if you kept it on primary storage. It should allow you find and access data relatively quickly -- but it does not have to be instantaneous like it is on primary storage. Finally, stored data should be durable. Data you archive today should be readable 10 years from now and beyond. We used to say that all data has a decaying value; the further away from its creation date it gets, the less valuable that data becomes. Compliance and regulatory requirements as well as big data analytics and archive have changed that. We now have to assume that all data will become valuable again -- we just don't know which data or when. If decades from now your grandchildren check into a hospital, the doctors might want to access your medical records. They need them quickly and they better be readable. In theory, these archiving needs strengthen the position of many disk-based object-storage vendors. Their systems can provide data durability as well as quick access and cost effectiveness when compared to <a href="http://www.informationweek.com/storage/systems/measuring-the-state-of-primary-storage-d/240142331">primary storage</a>. The problem is that object storage is not as inexpensive as tape storage nor is it as power efficient. [ Learn more about archiving schemes. Read <a href="http://www.informationweek.com/storage/data-protection/find-the-right-data-archive-method/240144127">Find The Right Data Archive Method</a>. ] Because we are talking about potentially storing all data for decades, we need to do everything we can, without putting data at risk, to reduce the overall storage cost of the system. After all, those records won't do you any good if the hospital can't afford to keep the system that stores them powered on and up-to-date. However, before we turn over all archive data to the object storage vendors, there is a part of that "all data has a decaying value" theory that is still applicable. It's this: All data has a decaying speed at which it needs to be accessed. Using our medical example above, the doctors might need to access your medical records 50 years from now, but they probably don't need to have them in seconds. They can probably wait a minute or two. As I noted in my article <a href="http://www.storage-switzerland.com/Articles/Entries/2012/3/1_Comparing_LTO-6_to_Scale-Out_Storage_for_Long-Term_Retention.html ">"Comparing LTO-6 to Scale-Out Storage for Long-Term Retention,"</a> in these situations tape is an ideal storage type. Data on tape can still be automatically scanned for durability and it certainly meets the cost-effectiveness requirements. What surprises most people that are either new to tape or have forgotten about it is how quickly a modern tape library can deliver data. In most cases access takes less than a minute; in the worst case it is two to three minutes. Understanding The Data Access Decay Rate The speed at which you need to have data returned to primary storage will depend on the needs of the business. Because the predictable response to, "How long can you wait?" is, "I need it now," it is important to make sure that business line managers understand the value of waiting. If they understand that waiting two minutes could save the organization $2 million a year in storage expenses, waiting sounds much more attractive. In almost every case the durability of the data is far more important than the speed at which it can be recovered. I typically suggest a blended strategy: As little primary storage as possible, a reasonable amount of object/archive storage, and a hefty amount of tape. The amount of object/archive disk storage will be driven by your data access decay rate. For many organizations that might mean keeping all data on object storage for three to five years. For almost all organizations, longer-term retention should be on tape. This blended strategy gives the right balance between access, affordability and durability.2013-01-02T09:07:00ZWhat Is Software-Defined Storage?Is it really a new product or simply a reframing of existing storage technology? Actually, it's a little bit of both.http://www.informationweek.com/what-is-software-defined-storage/240145344?cid=SBX_iwk_related_commentary_Data_protection_storageWith increasing frequency, I'm asked for my thoughts on the emerging software-defined storage category. Whenever I'm presented with a new tech term, I ask whether it truly defines a new product category, or if it's simply an attempt to make an existing technology seem more glamorous. Software-defined storage is a little bit of both. If we really have to have a separate term for this group of products, here is what I think that definition should be. We have been using software to define storage for as long as there has been storage. One could take the stance that a volume-manager application is essentially software defining storage. But those promoting the current term clearly have more in mind. You could also easily lump anything that does storage virtualization into this category, and we are seeing many of the storage virtualization vendors do just that. For me, though, there is a difference. Both storage virtualization and software-defined storage abstract the storage services from the storage system, allowing them to provide those services across a variety of disk and solid-state storage systems. Storage virtualization, however, should be isolated to products that must run on a dedicated piece of hardware. For many vendors, this is a purpose-built appliance; for others, it is software that you load on a dedicated server. I don't think there is anything controversial about this separation thus far. However, I would refine the above to also include products that require their software to be run as a dedicated virtual machine. The fact that your appliance is virtual does not mean it does not require an appliance; it simply means that it does not require hardware. It is essentially virtualized storage virtualization. That said, storage appliances running virtually can be seen as an improvement over dedicated external devices, as they bring storage performance and costs in lockstep with the scaling of the virtual infrastructure. This means then that software-defined storage is storage software that is an extension of the existing operating system or hypervisor and does not require a specific virtual machine to run its software in. As we discuss in "<a href="http://www.storage-switzerland.com/Articles/Entries/2011/9/26_The_Storage_Hypervisor.html">What is the Storage Hypervisor?</a>" this means that either the operating system / hypervisor provider or (via extension) a third party has added features like thin provisioning, snapshots, cloning and replication. At that point all that is needed from the physical storage hardware is a reliable design and potentially high availability. For the IT professional this is more than just a discussion of semantics. Each has its place and can bring significant value to the enterprise. As the data center becomes increasingly virtualized, software-defined storage and virtualized storage virtualization becomes an ideal method for scaling storage capacity and performance as the virtual environment scales. Until that time, storage virtualization running on dedicated hardware provides the benefits of software-defined storage across both virtualized and non-virtualized platforms.2012-12-27T09:06:00ZHow To Choose A Data Archiving PlatformNo matter what your motivation for archiving data, the storage system needs to provide data integrity, scalability and power management.http://www.informationweek.com/storage/systems/how-to-choose-a-data-archiving-platform/240145234?cid=SBX_iwk_related_commentary_Data_protection_storage <div class="inlineStoryImage inlineStoryImageRight"><a href="http://www.informationweek.com/cloud-computing/infrastructure/7-cheap-cloud-storage-options/240134947"><img src="http://twimgs.com/informationweek/galleries/automated/905/01_Cloud_tn.jpg" alt="7 Cheap Cloud Storage Options" title="7 Cheap Cloud Storage Options" class="img175" /></a> <div class="storyImageTitle">7 Cheap Cloud Storage Options</div>(click image for larger view and for slideshow)</div> At the end of the year, the topic of data archiving heats up. My <a href="http://www.informationweek.com/storage/data-protection/find-the-right-data-archive-method/240144127">last column</a> covered different methods for moving data to that archive. This column we will take a look at storage systems that want to be your repository for storing this information. In general, no matter what your motivation for archiving data, the archive storage system needs to provide data integrity, scalability and power management -- and, of course, do so at competitive pricing. There are several types of devices that you can archive to. The first and one that might be overlooked is a big disk array. Although these often don't have the capabilities to do continuous data verification and might not have the large scaling capabilities that other, more archive-specific systems do, they do have one big advantage: Price. These systems tend to be very cost effective if your archive requirements won't reach the limits of a single array. A few of these systems also have very mature power-saving capabilities such as spin-down drives. Another option outside of traditional archive storage systems is cloud storage services. Cloud has the advantage of not taking up any of your data center footprint and never running out of capacity. Some cloud providers via third-party archive solutions also can provide complete data integrity checking. They also, of course, have the advantage of a pay-as-you-go license, so the upfront investment is minimal. The downside to these systems is that they are pay-as-you-grow as well. You keep paying and paying. Storing terabytes and terabytes of information in the cloud for decades could be very expensive over time. [ Wondering how cloud storage differs from online? Read <a href="http://www.informationweek.com/byte/personal-tech/consumer-services/online-backup-vs-cloud-storage/240142299">Online Backup Vs. Cloud Storage</a>. ] There is the option to build your own cloud storage system in house; in other words, a private cloud. As I recently described in my article <a href="http://www.storage-switzerland.com/Blog/Entries/2012/11/30_What_Is_Object_Storage.html">"What is Object Storage,"</a> most of these systems tend to use an object file layout. This gives them tremendous scalability and consistent performance even as the amount of archive data increases. Leveraging an object layout also provides the foundation for doing continuous data verification. These systems also tend to scale one node at a time, providing a similar pay-as-you-grow capability. Unlike the cloud, though, you own it. This has its pros and cons. There is also the challenge that you have to store all your data on disk. That means these systems need to be powered and running in order to operate. Few scale-out object storage systems have developed the capability to "spin-down" nodes. Finally, there is tape. Tape wins hands down for price competitiveness and for power efficiency. The above technologies all provide near-instant retrieval. Tape does not. But you have to ask yourself, if a request comes in for data that is 10 years old do you really need to recover it in seconds? Or can it wait a few minutes for the tape to be loaded into a tape drive, found and then recovered? If that is the case then tape might be for you. Another concern about tape is data integrity. As we discussed in our webinar <a href="http://www.storage-switzerland.com/BT4ReaTape.html">The Four Reasons The Data Center is Returning To Tape</a>, tape cartridges have actually been proven to be more reliable than a disk drive but they don't have the built-in data integrity checks that some of the above methods do. However, some archiving solutions that support tape provide the ability to perform scheduled scans of tape drives so that integrity can be assured. So, which one to pick? Most vendors mistakenly look at the archive target as a zero sum game. It all must be on their hardware. We find that most data centers are better served by a mixed approach that leverages two or more of the above solutions: Use disk for the medium-term archive of data, and tape for the long-term deep archive. In fact, in an upcoming column I'll discuss how to leverage tape with either a private or public cloud.