Created Sun, 19 Jan 2014 17:38:11 +0000 by majenko
Sun, 19 Jan 2014 17:38:11 +0000
I have been doing some work on the SD and DSPI libraries in UECIDE with the aim of making them work somewhat more efficiently.
The default SD library with MPIDE uses a purely bit-banging SPI interface, even when it's connected to a hardware SPI port. As a result it is really really really slow. I mean dog slow. Dead dog slow.
A crude little benchmark program I knocked up to write then read 10MB of data to an SD card told me:
Initializing SD card...initialization done. Deleting existing file Writing 10MB to test.txt...done. Time taken: 114712ms Reading data from test.txt: done. Time taken: 111641ms
That's run in MPIDE. It equates to around 89KB/s write speed, or 92KB/s read speed.
Now that really is slow.
So, step one was to switch to using hardware SPI when available. That meant a slight change to how the SD library works and how you configure it. You can now pass it a DSPI object to use a specific SPI bus (defaults to DSPI0), or you can give it a bunch of IO pins to use software SPI, which has also been replaced with shiftIn() and shiftOut() instead of custom code (after all, why duplicate code?).
Already, at 10MHz SPI clock speed it's considerably quicker:
Initializing SD card...initialization done. Deleting existing file Writing 10MB to test.txt...done. Time taken: 56602ms Reading data from test.txt: done. Time taken: 51964ms
Now we're flying along at 197KB/s read speed and 180.9KB/s write speed. That's quite an increase in speed. But just how fast can we push it? Up the SPI clock to 20MHz, and the speed increases a little to 220KB/s read and 200KB/s write.
But there's one more little trick we can use that is available on some boards, and it's what we use on RetroBSD, and that is Enhanced Buffering. Enable that and you have a small FIFO instead of a simple shift register for the SPI data. It just smooths things out a bit - you can be loading (and emptying) the FIFO while it's doing the transferring of data. It just makes the whole thing more efficient.
Initializing SD card...initialization done. Deleting existing file Writing 10MB to test.txt...done. Time taken: 45681ms Reading data from test.txt: done. Time taken: 39569ms
And yes, the results are even faster. Here's a little league table so you can see all the differences:
Setup Read Write MPIDE 91.7KB/s 89.3KB/s HW 10MHz 197KB/s 180.9KB/s HW 20MHz 220.8KB/s 200KB/s ENHBUF 258.8KB/s 224.2KB/s
And yes, those changes are now available in the latest chipKIT core in UECIDE.
Sun, 19 Jan 2014 22:19:01 +0000
That's pretty sweet. Any chance you can issue a pull request to MPIDE so these changes can be used there too?
Sun, 19 Jan 2014 22:21:43 +0000
I could, but the changes to the SD code do have a bit of a knock-on effect as far as Arduino compatability goes. The Sd2Card constructor, if you want to use anything other than DSPI0 (which for some you will need to) is very different, so it doesn't (yet) work out of the box for some boards, like the WF32.
I'd like to expand the board definitions so they include the default SD settings as well really. The original SD code had the definitions within the SD code itself, which was a bit silly really. Now it completely ignores those definitions and just uses DSPI0...
Sun, 19 Jan 2014 23:17:41 +0000
I agree- putting this in the board definitions so that the SD library can always 'do the right thing' makes a ton of sense.
Sun, 19 Jan 2014 23:38:01 +0000
Also the SD library requires the DSPI library to be included in the sketch. This is different to "normal". Not a problem for UECIDE, which deals with recursive library requirements, but could confuse new users in MPIDE.
Mon, 20 Jan 2014 00:04:33 +0000
Ahh, crap. The old "can't include a library within a library" problem. I can't wait until somebody figures out how to fix that in MPIDE.
Or maybe I should just join the dark side . . .
Mon, 20 Jan 2014 01:00:39 +0000
Do not underestimate the power of the dark side...
Mon, 20 Jan 2014 02:11:29 +0000
I just downloaded it, and will be trying it out shortly . . .
Mon, 20 Jan 2014 14:19:58 +0000
Great to see majenko finally took it in his hands and tried the EB. First time mentioned Sep2011 in this forum (Time to Market is an important aspect here as well). Also involving Bill (fat16lib) with his newest SdFat library would be a good move, I would say (if he would be interested to help). His SdFat gives 10-15x higher speeds with Due (with large buffers).. The SPI on pic32 is limited to something around 20Mhz, so the speeds around 500-1000kB/sec are to be expected (with cheap cards).
Sun, 26 Apr 2015 09:54:57 +0000
I made a post in the Libraries subforum about my progress porting SDFat lib to allow use of hardware DSPI. (I am apparently on "post probation", so it isn't up yet, and I can't link it.) Can someone look at my modifications and make them more "official"?
File size 5 MB Buffer size 4096 bytes Starting write test, please wait. write speed and latency speed,max,min,avg KB/Sec,usec,usec,usec 957.67,12269,4088,4272 964.70,11783,4151,4242 960.25,11811,4085,4261 962.09,11789,4085,4253 962.47,11806,4087,4251 960.25,11788,4085,4260 964.51,11784,4151,4242 959.88,11804,4085,4262 960.06,11786,4085,4262 964.51,11787,4088,4242 Starting read test, please wait. read speed and latency speed,max,min,avg KB/Sec,usec,usec,usec 1176.07,4079,3388,3482 1175.79,4087,3375,3482 1175.79,4132,3372,3482 1175.79,4126,3372,3482 1175.79,4085,3372,3482