Sunday 30 March 2014

Raspberry Pi USB disk throughput test

Update (2015-02): There will be little to no change in throughput performance from the new Raspberry Pi Generation 2 Model B (900MHz quad-core ARM Cortex-A7 BCM2836) over the older Raspberry Pi Model B+ (700MHz ARM11 BCM2835), since the USB performance bottleneck is still the LAN9514 USB/Ethernet controller and will not be improved by additional CPU or RAM.

Update (2014-07): There is a new RPi model B+ with 4 USB ports (well 5 in fact when you include ethernet using a LAN9514 chip instead of 3 ports using the LAN9512 in the model B). Just to be clear the tests below were done on the model B. If and when I repeat the the tests with a model B+ I do not expect to see better performance.


Hardware Configuration:

Raspberry Pi model B - 512MB model, 128MB of RAM allocated to GPU, no overclocking
Inbuilt 10/100 NIC is connected.
Top USB port is a 2TB USB hard disk drive.
Bottom USB port is empty.

So no keyboard, no mouse. In an ideal world we would have the inbuilt USB NIC disabled (after setting the date and time) as well, but it appears that the RPi has the network module compiled directly into the kernel.
pi@raspberrypi ~ $ zgrep -i smsc95xx /proc/config.gz
CONFIG_USB_NET_SMSC95XX=y
pi@raspberrypi ~ $ sudo modprobe -r smsc95xx
FATAL: Module smsc95xx is builtin.

So to just have a hard disk  and RPi (and internal 3 port USB hub) would require me to recompile the kernel, which would make the results slightly off standard

The HDD that I'm using has an Internal write rate of about 100MiB/sec, so USB throughput should be the cause of any performance bottleneck.

 

Prerequisites:

sudo apt-get install at expect expect-dev



ext2, ext3, ext4, vfat file system write tests 


I split a 2 TB disk into four equal sized partitions.

pi@raspberrypi ~ $ lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/3p, 480M
        |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
        |__ Port 2: Dev 4, If 0, Class=stor., Driver=usb-storage, 480M

pi@raspberrypi ~ $ sudo fdisk -l /dev/sda

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x1ba9000c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048   976758827   488378390   83  Linux
/dev/sda2       976758828  1953515607   488378390   83  Linux
/dev/sda3      1953515608  2930272387   488378390   83  Linux
/dev/sda4      2930272388  3907029167   488378390    c  W95 FAT32 (LBA)


pi@raspberrypi ~ $ sudo mkfs.ext2 -L ext2 /dev/sda1

pi@raspberrypi ~ $ sudo mkfs.ext3 -L ext3 /dev/sda2
pi@raspberrypi ~ $ sudo mkfs.ext4 -L ext4 /dev/sda3
pi@raspberrypi ~ $ sudo mkfs.vfat -n vfat /dev/sda4
pi@raspberrypi ~ $ cd /media 
pi@raspberrypi /media $ sudo mkdir ext2 ext3 ext4 vfat




pi@raspberrypi ~ $ at midnight

warning: commands will be executed using /bin/sh
at> /home/pi/write-throughputtest.exp 2>&1 >/home/pi/at.output 
at> ^D




pi@raspberrypi ~ $

 


pi@raspberrypi ~ $ cat write-throughputtest.exp
#!/usr/bin/expect -f
#
# This Expect script was generated by autoexpect on Sun Mar 30 16:43:25 2014
set force_conservative 0  ;# set to 1 to force conservative mode even if
                          ;# script wasn't run conservatively originally
if {$force_conservative} {
        set send_slow {1 .1}
        proc send {ignore arg} {
                sleep .1
                exp_send -s -- $arg
        }
}



set timeout -1
spawn $env(SHELL)
match_max 100000

expect "$"
send -- "/bin/bash"
expect "pi@raspberrypi"
send -- "sudo /bin//bash\r"
expect "root@raspberrypi"
send -- "script /home/pi/output-root.txt\r"
expect "Script started, file is /home/pi/output-root.txt\r"
send -- "mount /dev/sda1 /media/ext2\r"
expect "root@raspberrypi"
send -- "mount /dev/sda2 /media/ext3\r"
expect "root@raspberry
pi"
send -- "mount /dev/sda3 /media/ext4\r"
expect "root@raspberry
pi"
send -- "mount /dev/sda4 /media/vfat\r"
expect "root@raspberry
pi"
send -- "/home/pi/write-throughputtest.sh /media/ext2\r"
expect "root@raspberry
pi"
send -- "/home/pi/
write-throughputtest.sh /media/ext3\r"
expect "root@raspberry
pi"
send -- "/home/pi/
write-throughputtest.sh /media/ext4\r"
expect "root@raspberry
pi"
send -- "/home/pi/
write-throughputtest.sh /media/vfat\r"
expect "root@raspberry
pi"
send -- "exit\r"
expect "Script done, file is /home/pi/output-root.txt\r"
send -- "exit\r"
expect "pi@raspberry
pi"
send -- "exit\r"

expect "$"
send -- "exit\r"
expect eof
 


pi@raspberrypi ~ $ cat write-throughputtest.sh
#!/bin/sh
DEST=$1

# 18 x 1GiB = 18GiB
# assuming maximum write 40MiB/sec this should take 8 minutes
# assuming poor write speed of 5 MiB/sec this should take 2 hours
# either way one pass should happen within 12 hours
echo touch $DEST/testbegin-write-starttime
touch $DEST/testbegin-write-starttime
date
echo sync
sync
date

for BS in 8 16 32 64 128 256 \
        512 1024 2048 4096 8192 16384 \
        32768 65536 131072 262144 524288 1048576

do
 COUNT=`expr 1073741824 / $BS`
echo touch $DEST/${BS}b-write-starttime
 touch $DEST/${BS}b-write-starttime
date
echo dd if=/dev/zero of=$DEST/${BS}b-data bs=${BS} count=${COUNT}
 dd if=/dev/zero of=$DEST/${BS}b-data bs=${BS} count=${COUNT}
date
echo touch $DEST/${BS}b-write-endtime
 touch $DEST/${BS}b-write-endtime
date
echo sync
 sync
date
# any delay caused by sync will show up on next starttime
done

echo touch $DEST/testover-write-endtime
touch $DEST/testover-write-endtime
date


pi@raspberrypi ~ $ cat read-throughputtest.sh
#!/bin/sh
SRCE=$1

# 18 x 1GiB = 18GiB

# assuming maximum read 40MiB/sec this should take 8 minutes
# assuming poor read speed of 5 MiB/sec this should take 2 hours
# This script assumes that write-throughputtest.sh was run
# previously to generate the data files.

date
echo sync
sync
date

for BS in 8 16 32 64 128 256 \

        512 1024 2048 4096 8192 16384 \
        32768 65536 131072 262144 524288 1048576

do

 COUNT=`expr 1073741824 / $BS`
date
echo dd of=/dev/null if=$SRCE/${BS}b-data bs=${BS} count=${COUNT}
 dd of=/dev/null if=$SRCE/${BS}b-data bs=${BS} count=${COUNT}
date
echo sync
 sync
date
done

date


pi@raspberrypi ~ $



 Results

USB disk performance check Raspberry Pi normal USB (internal 3 port hub)


Raspberry Pi Model B USB disk write results
-----------------------------------------------------------
block             filesystem type
size   -------------------------------------------            
(bytes)ext2       ext3       ext4       vfat
8       0.2 MiB/s  0.3 MiB/s  0.3 MiB/s  0.8 MiB/s
16      0.4 MiB/s  0.5 MiB/s  0.6 MiB/s  1.5 MiB/s
32      0.9 MiB/s  1.0 MiB/s  1.1 MiB/s  2.9 MiB/s
64      2.5 MiB/s  1.8 MiB/s  1.9 MiB/s  4.4 MiB/s
128     4.7 MiB/s  3.2 MiB/s  3.5 MiB/s  6.3 MiB/s
256     7.4 MiB/s  5.5 MiB/s  6.1 MiB/s  9.4 MiB/s
512    10.6 MiB/s  8.5 MiB/s  9.5 MiB/s 12.2 MiB/s
1k     14.9 MiB/s 11.3 MiB/s 13.8 MiB/s 13.1 MiB/s
2k     18.4 MiB/s 14.2 MiB/s 18.6 MiB/s 15.1 MiB/s
4k     20.8 MiB/s 17.7 MiB/s 22.5 MiB/s 17.0 MiB/s
8k     21.3 MiB/s 18.6 MiB/s 22.8 MiB/s 16.7 MiB/s
16k    21.3 MiB/s 19.0 MiB/s 22.4 MiB/s 17.0 MiB/s
32k    21.3 MiB/s 19.0 MiB/s 22.5 MiB/s 16.9 MiB/s
64k    20.7 MiB/s 18.7 MiB/s 22.9 MiB/s 16.9 MiB/s
128k   21.3 MiB/s 18.1 MiB/s 22.7 MiB/s 16.4 MiB/s
256k   21.3 MiB/s 18.4 MiB/s 22.4 MiB/s 16.5 MiB/s
512k   21.6 MiB/s 18.9 MiB/s 22.3 MiB/s 16.8 MiB/s

1M     20.9
MiB/s 19.2 MiB/s 22.4 MiB/s 16.9 MiB/s

For the small block sizes there is almost no disk activity, long pauses with no blinking RED LED's.
I'm begining to wonder if some options in Raspbian have been enabled to extend the life of the RPi SSD.

Raspberry Pi Model B USB disk read results
-----------------------------------------------------------
block             filesystem type
size   -------------------------------------------           
(bytes)ext2       ext3       ext4       vfat
8       1.6 MiB/s  2.0 MiB/s  1.7 MiB/s  1.9 MiB/s
16      3.9 MiB/s  3.5 MiB/s  3.6 MiB/s  3.8 MiB/s
32      6.9 MiB/s  6.7 MiB/s  7.2 MiB/s  6.8 MiB/s
64     
10.5 MiB/s 10.4 MiB/s 10.2 MiB/s 10.4 MiB/s
128    18.0 MiB/s 16.5 MiB/s 15.2 MiB/s 17.0 MiB/s
256    23.3 MiB/s 22.6 MiB/s 22.1 MiB/s 22.2 MiB/s
512    23.2 MiB/s 23.2 MiB/s 23.4 MiB/s 21.7 MiB/s
1k     23.6 MiB/s 23.2 MiB/s 23.6 MiB/s 22.0 MiB/s
2k     23.7 MiB/s 23.2 MiB/s 23.6 MiB/s 22.3 MiB/s
4k     23.6 MiB/s 22.9 MiB/s 23.6 MiB/s 23.3 MiB/s
8k     22.6 MiB/s 22.3 MiB/s 23.7 MiB/s 23.7 MiB/s
16k    23.6 MiB/s 23.6 MiB/s 23.7 MiB/s 23.7 MiB/s
32k    22.4 MiB/s 23.6 MiB/s 22.6 MiB/s 23.7 MiB/s
64k    22.5 MiB/s 23.6 MiB/s 22.7 MiB/s 23.7 MiB/s
128k   23.7 MiB/s 22.4 MiB/s 22.8 MiB/s 23.7 MiB/s
256k   23.7 MiB/s 22.5 MiB/s 23.8 MiB/s 23.7 MiB/s
512k   23.7 MiB/s 22.6 MiB/s 23.7 MiB/s 23.7 MiB/s
1M     23.7 MiB/s 22.4 MiB/s 23.7 MiB/s 23.7 MiB/s




USB disk performance check (midrange desktop PC) - dedicated USB port

For comparison with the above Raspberry Pi results , here are desktop PC results:

midrange desktop PC disk write results (dedicated HS USB 2.0 port)
------------------------------------------------------------------
block             filesystem type
size   -------------------------------------------             
(bytes)ext2       ext3       ext4       vfat
8       6.3 MiB/s  3.8 MiB/s  3.9 MiB/s  5.4 MiB/s
16     12.9 MiB/s  7.6 MiB/s  7.8 MiB/s 11.1 MiB/s
32     26.1 MiB/s 15.0 MiB/s 15.9 MiB/s 19.4 MiB/s
64     34.9 MiB/s 21.1 MiB/s 31.7 MiB/s 23.3 MiB/s
128    34.9 MiB/s 21.8 MiB/s 37.7 MiB/s 24.0 MiB/s
256    34.2 MiB/s 22.2 MiB/s 37.5 MiB/s 23.4 MiB/s
512    34.1 MiB/s 22.7 MiB/s 37.3 MiB/s 23.5 MiB/s
1k     34.1 MiB/s 22.8 MiB/s 37.1 MiB/s 24.7 MiB/s
2k     33.8 MiB/s 22.7 MiB/s 37.5 MiB/s 24.5 MiB/s
4k     34.0 MiB/s 22.3 MiB/s 37.2 MiB/s 24.4 MiB/s
8k     34.1 MiB/s 22.7 MiB/s 37.4 MiB/s 24.9 MiB/s
16k    35.1 MiB/s 22.9 MiB/s 37.5 MiB/s 23.7 MiB/s
32k    36.0 MiB/s 22.7 MiB/s 37.5 MiB/s 25.4 MiB/s
64k    35.4 MiB/s 23.5 MiB/s 37.6 MiB/s 24.8 MiB/s
128k   35.3 MiB/s 23.1 MiB/s 37.2 MiB/s 24.4 MiB/s
256k   35.6 MiB/s 23.0 MiB/s 37.6 MiB/s 25.5 MiB/s
512k   35.9 MiB/s 23.3 MiB/s 37.6 MiB/s 25.2 MiB/s
1M     35.9 MiB/s 22.7 MiB/s 37.6 MiB/s 25.2 MiB/s

For High Speed USB 2.0 the maximum data rate is about 40MB/sec (38.1MiB/sec) None of the tested filesystems hit this but ext4 did come closest, but not on the Raspberry Pi unfortunately.

('lsusb -t' does shows 480M, which in theory would correspond to 57.2MiB/sec, but there protocol overheads within USB. The maximum should be 53.248 MB/s, but this appears to be deliberately limited to around 30-42MB/sec to maintain interoperability with USB stacks in OS'es over throughput speed).

midrange desktop PC disk read results (dedicated HS USB 2.0 port)
-----------------------------------------------------------------
block             filesystem type
size   -------------------------------------------            
(bytes)ext2       ext3       ext4       vfat
8      18.4 MiB/s 16.8 MiB/s 18.4 MiB/s 18.6 MiB/s
16     32.5 MiB/s 29.3 MiB/s 36.6 MiB/s 35.0 MiB/s
32     32.5 MiB/s 29.0 MiB/s 35.9 MiB/s 34.5 MiB/s
64     33.6 MiB/s 30.4 MiB/s 36.2 MiB/s 35.6 MiB/s
128    33.3 MiB/s 29.9 MiB/s 36.0 MiB/s 35.0 MiB/s
256    32.7 MiB/s 30.3 MiB/s 36.1 MiB/s 34.8 MiB/s
512    33.4 MiB/s 29.9 MiB/s 36.0 MiB/s 35.0 MiB/s
1k     33.5 MiB/s 30.2 MiB/s 36.2 MiB/s 35.9 MiB/s
2k     33.7 MiB/s 30.7 MiB/s 36.4 MiB/s 34.7 MiB/s
4k     34.0 MiB/s 30.5 MiB/s 36.1 MiB/s 35.9 MiB/s
8k     34.0 MiB/s 30.8 MiB/s 36.4 MiB/s 35.4 MiB/s
16k    34.0 MiB/s 30.9 MiB/s 36.4 MiB/s 35.5 MiB/s
32k    34.0 MiB/s 30.7 MiB/s 36.5 MiB/s 35.2 MiB/s
64k    33.7 MiB/s 30.4 MiB/s 36.4 MiB/s 35.2 MiB/s
128k   33.5 MiB/s 30.7 MiB/s 36.4 MiB/s 36.0 MiB/s
256k   33.4 MiB/s 30.5 MiB/s 36.4 MiB/s 34.8 MiB/s
512k   33.5 MiB/s 30.5 MiB/s 36.1 MiB/s 35.8 MiB/s
1M     33.2 MiB/s 30.5 MiB/s 36.5 MiB/s 35.7 MiB/s



I really wonder what the USB performance is for a Model A, which does not have an internal 3 port USB hub inside with a USB NIC connected to one of the ports. For all of the tests above there was basically no network traffic to/from the RPi, and even with that the performance of the USB  is not all that great. But even so it does appear that USB is the fastest way to get data into and out of the Raspberry Pi.


Internal disk performance check (midrange desktop PC)

I reconnected the same 2TB disk to a dedicated 1.5Gbit/sec eSATA port on the back of the desktop PC just to confirm that USB was in fact the bottleneck for data transfers. And to verify that there were no problems with the hard disk that I used.

midrange desktop PC disk write results (1.5 Gbit/sec eSATA port)
------------------------------------------------------------------
block             filesystem type
size   -------------------------------------------            
(bytes)ext2       ext3       ext4       vfat
8       7.2 MiB/s  4.2 MiB/s  4.2 MiB/s  6.4 MiB/s
16     14.6 MiB/s  8.4 MiB/s  8.6 MiB/s 12.3 MiB/s
32     28.5 MiB/s 16.2 MiB/s 16.9 MiB/s 22.0 MiB/s
64     55.9 MiB/s 29.2 MiB/s 34.3 MiB/s 35.5 MiB/s
128    72.1 MiB/s 53.7 MiB/s 61.8 MiB/s 47.8 MiB/s
256    69.4 MiB/s 64.6 MiB/s 59.4 MiB/s 52.4 MiB/s
512    68.3 MiB/s 63.2 MiB/s 58.7 MiB/s 52.7 MiB/s
1k     66.9 MiB/s 62.1 MiB/s 58.4 MiB/s 54.1 MiB/s
2k     66.2 MiB/s 61.4 MiB/s 58.7 MiB/s 54.1 MiB/s
4k     64.8 MiB/s 60.7 MiB/s 58.1 MiB/s 54.1 MiB/s
8k     64.3 MiB/s 60.2 MiB/s 57.3 MiB/s 53.5 MiB/s
16k    64.5 MiB/s 60.6 MiB/s 57.2 MiB/s 52.5 MiB/s
32k    63.2 MiB/s 59.7 MiB/s 56.9 MiB/s 55.4 MiB/s
64k    63.4 MiB/s 60.7 MiB/s 56.8 MiB/s 53.3 MiB/s
128k   62.8 MiB/s 59.2 MiB/s 56.6 MiB/s 53.0 MiB/s
256k   62.6 MiB/s 59.6 MiB/s 56.4 MiB/s 53.1 MiB/s
512k   62.1 MiB/s 58.7 MiB/s 55.3 MiB/s 52.2 MiB/s
1M     61.6 MiB/s 58.3 MiB/s 53.9 MiB/s 52.3 MiB/s



midrange desktop PC disk read results (1.5 Gbit/sec eSATA port)
-----------------------------------------------------------------
block             filesystem type
size   -------------------------------------------           
(bytes)ext2       ext3       ext4       vfat
8      18.7MiB/s  18.6MiB/s  18.5MiB/s  18.5MiB/s
16     36.4MiB/s  36.4MiB/s  36.8MiB/s  36.6MiB/s
32     71.2MiB/s  70.1MiB/s  72.5MiB/s  70.0MiB/s
64    117.3MiB/s 110.6MiB/s  98.2MiB/s  82.3MiB/s
128   116.3MiB/s 111.6MiB/s  98.2MiB/s  82.8MiB/s
256   117.3Mib/s 110.6MiB/s  98.2MiB/s  81.9MiB/s
512   117.3Mib/s 110.6MiB/s  99.2MiB/s  81.3MiB/s
1k    116.3MiB/s 109.7MiB/s  98.2MiB/s  81.3MiB/s
2k    118.3MiB/s 108.7MiB/s  97.3MiB/s  81.3MiB/s
4k    117.3Mib/s 108.7MiB/s  96.3MiB/s  81.3MiB/s
8k    117.3Mib/s 108.7MiB/s  95.2MiB/s  81.3MiB/s
16k   118.3MiB/s 108.7MiB/s  95.4MiB/s  81.3MiB/s
32k   118.3MiB/s 108.7MiB/s  95.4MiB/s  81.3MiB/s
64k   118.3MiB/s 108.7MiB/s  95.3MiB/s  81.3MiB/s
128k  118.3MiB/s 108.7MiB/s  95.3MiB/s  81.0MiB/s
256k  118.3MiB/s 108.7MiB/s  95.2MiB/s  81.4MiB/s
512k  118.3MiB/s 108.7MiB/s  94.7MiB/s  81.3MiB/s
1M    118.3MiB/s 107.8MiB/s  94.2MiB/s  81.3MiB/s


All the results say that the HDD was functioning within expected parameters. So all transfer limits (64+ bytes) were due to USB protocol limitation or the 3 port USB hub internal to the Raspberry Pi.

3 comments:

  1. I was thinking about something similar, but thishow at least this time was saved. Thank you for making these tests!

    ReplyDelete
    Replies
    1. I was doing the tests for my own knowledge and decided to share them in case it saved someone else from having to do the same. Good to know I didn't waste my time in posting my tests and results. (Results alone, I find, are not much good unless you know how they were actually generated).

      Delete
  2. really good to know, thanks

    ReplyDelete