High Speed PCIe SSDs

Those of you who’ve been reading this blog since it’s inception will remember that I used to post a lot about solid state drives, because we spent a lot of time trying to handle the 1 GB/sec bandwidth of sCMOS cameras back in 2013. We standardized on RAID 0 arrays of four Samsung Pro SSDs, and I stopped thinking about it, because that was good enough for our purposes.

Since then, however, quite a bit has changed. You can now get an 512 GB SSD card that can write at 1.5 GB/sec and read at 3 GB/sec (the Samsung Pro 950) for $350. Newer Samsung products promise slightly faster read / write speeds at disk sizes up to 1 TB. These use the M.2 interface, designed for SSDs, but PCIe to M.2 adapters are readily available if your motherboard doesn’t have an M.2 slot. To get full speeds you’ll need a motherboard with a PCIe 3.0  x4 slot available.

Another thing that’s changed is that sCMOS cameras have made it really easy to capture large data sets. In the last year people really seen to have taken advantage of this and we’re seeing a lot of people acquiring 100GB+ data sets, and often up to 1 TB. A big consequence of this is that data processing is increasingly becoming I/O bound. A mechanical hard drive tops out at around 120 MB/sec sequential read/write speed. At those speeds, just reading a 20 GB file takes around 3 min. We’ve seen this become a major issue where exporting a 500 GB data set from Nikon ND2 to TIFF takes hours.

If you’re doing any processing of large data sets, it’s very advantageous to have a fast drive like this to speed up I/O. There’s still a challenge in getting data on and off of these drives, since gigabit ethernet tops out at around 100 MB/sec, and most USB3 drives max out around 250 MB/sec (as an aside, the Samsung T3 looks pretty promising, with 450 MB/sec transfer rates). Finally, most people (at least here) still want their data on a mechanical drive for long term storage. But at least once you have data on the fast drive the I/O bottleneck gets better by about 10 – 20-fold.

All of this would be well worth thinking about if you’re building the compute environment for a microscopy core from scratch. I imagine you’d want fast local storage for data processing, 10 Gbit networking to move data around, and some kind of slow archiving to long term storage. You’d also want to educate people about data handling so that they think about these issues when designing their experiments.

2 thoughts on “High Speed PCIe SSDs

  1. Have you seen any good references on programming for efficient IO on these devices? We also put together a 4x 850 Pro RAID 0 array a while back, and while it benchmarked fantastically, I found that getting even a third the benchmark speed was quite hard from c++.

    • Unfortunately, that’s well beyond my programming expertise. Back in 2013 we did work pretty closely with the Micro-manager team so that they could optimize the performance of Micro-manager on the SSD array. I do remember a lot of code profiling, looking for bottlenecks, and moving operations into separate threads so that they could execute in parallel. It took several weeks of optimization to get data off of a Zyla 5.5 and written to the SSD array at full speed (1 GB/sec). I am sure there is some information out there about best practices, but I don’t know it myself.

Leave a Reply

Your email address will not be published. Required fields are marked *