Those of you who’ve been reading this blog since it’s inception will remember that I used to post a lot about solid state drives, because we spent a lot of time trying to handle the 1 GB/sec bandwidth of sCMOS cameras back in 2013. We standardized on RAID 0 arrays of four Samsung Pro SSDs, and I stopped thinking about it, because that was good enough for our purposes.
Since then, however, quite a bit has changed. You can now get an 512 GB SSD card that can write at 1.5 GB/sec and read at 3 GB/sec (the Samsung Pro 950) for $350. Newer Samsung products promise slightly faster read / write speeds at disk sizes up to 1 TB. These use the M.2 interface, designed for SSDs, but PCIe to M.2 adapters are readily available if your motherboard doesn’t have an M.2 slot. To get full speeds you’ll need a motherboard with a PCIe 3.0 x4 slot available.
Another thing that’s changed is that sCMOS cameras have made it really easy to capture large data sets. In the last year people really seen to have taken advantage of this and we’re seeing a lot of people acquiring 100GB+ data sets, and often up to 1 TB. A big consequence of this is that data processing is increasingly becoming I/O bound. A mechanical hard drive tops out at around 120 MB/sec sequential read/write speed. At those speeds, just reading a 20 GB file takes around 3 min. We’ve seen this become a major issue where exporting a 500 GB data set from Nikon ND2 to TIFF takes hours.
If you’re doing any processing of large data sets, it’s very advantageous to have a fast drive like this to speed up I/O. There’s still a challenge in getting data on and off of these drives, since gigabit ethernet tops out at around 100 MB/sec, and most USB3 drives max out around 250 MB/sec (as an aside, the Samsung T3 looks pretty promising, with 450 MB/sec transfer rates). Finally, most people (at least here) still want their data on a mechanical drive for long term storage. But at least once you have data on the fast drive the I/O bottleneck gets better by about 10 – 20-fold.
All of this would be well worth thinking about if you’re building the compute environment for a microscopy core from scratch. I imagine you’d want fast local storage for data processing, 10 Gbit networking to move data around, and some kind of slow archiving to long term storage. You’d also want to educate people about data handling so that they think about these issues when designing their experiments.