USB3 Flash drives for high speed data transfer

In the past few years, we’ve switched the NIC almost exclusively to sCMOS cameras. We still have a few EMCCDs, but we have no interline CCDs in the NIC anymore. This change has greatly increased data acquisition rates – we’ve gone from 1.4 megapixel images to 4+ megapixel images. It’s now very common for someone to sit down at a microscope for a few hours and end up with 20 GB of data, and time lapse experiments often produce 1 TB.

This increase in the amount of data has made data transfer a bottleneck. USB2.0 flash drives, in our experience, top out around 20-30 MB/sec transfer rates. At that speed, a 20 GB data set takes 10-15 minutes to transfer. To reduce this time, we’ve begun to upgrade all of our PCs to add USB3 ports. We’ve been using Startech cards and have had good luck with them. With a good USB3 thumb drive – we use a SanDisk Extreme for testing – we can get transfer speeds of ~180 MB/sec, a 6-8x improvement in speed. In the process of doing this I’ve learned a few useful things. First, USB2 cables do not support USB3 transfer speeds – you need to have a USB3 cable. Second, not all USB3 hubs are equal – we have one that doesn’t manage USB3 transfer speeds. We’ve had good luck with this Anker USB3 hub. Finally, some devices (in particular the Nikon Ti) do not like to run over USB3 ports, so you still need some USB2.0 ports for controlling hardware.

We also have a network server for data transfer, but with gigabit ethernet, it maxes out at about 100 MB/sec transfer speeds, so the USB3 drives end up being somewhat faster. We’re hoping that the USB drives will be reliable enough to allow direct acquisition of data to them (rather than saving to the local hard drive and then copying), although we haven’t tried that yet. We have had problems with transfer glitches causing experiments to be interrupted when we save over the network or to USB2.0 drives, so we don’t recommend that.

There is now a USB3.1 Gen 2 specification that promises a 2x speed improvement over USB3.0, but very few drives support it, so we haven’t started looking at that.

2 thoughts on “USB3 Flash drives for high speed data transfer

  1. We’ve had intermittent issues with USB3 performance plus a few external USB3 drives fail in the past year. As a result, we decided to invest in a Gigabit ethernet Network Attached Storage 5-RAID. Ethernet just works, there’s no special cables limited by length or power issues. The performance over LAN is good enough to support most acquisitions while allowing for analysis by multiple users on different computers. If you wanted to approach USB3 speeds, you could upgrade to 10Gb ethernet, but like USB3 you’ll still be slowed by disk speeds unless using SSD for everything.

  2. We were having problems with spinning disk server arrays so moved to a 9TB SSD array (made up of 250 GB SSDs set up in a 24 bay server, I picked up off eBay). Works great as an intermediate storage box for processing – data is moved from there to spinning disk storage nightly.

Leave a Reply

Your email address will not be published. Required fields are marked *