speeding up SD card writing???

Created Wed, 02 Jan 2013 19:09:17 +0000 by spencoid

spencoid

Wed, 02 Jan 2013 19:09:17 +0000

I think I have the latest test IDE with the new SD library and do have it working but it is not fast enough. I would like to know what options might be available. Here are some possibilities. Any feedback on anything to check out further would be appreciated.

There was a mention of how to increase the SPI clock speed but it was too technical for me to try to work into my code. Also a mention that the original SD library had the ability to change the speed but that it was not kept in the Chipkit version? Any chance of this changing soon?

There is an open source shield for Arduino that has a separate controller to write to the SD card. I did not read all the details but would hope that it has some buffer memory and can write to the card faster. Could not find anything ready made and do not want to make PC boards etc and find out it doesn't work? Anything like this ready made? I could not find it in my searching.

Any ideas on using EEPROM to buffer to the SD card. Not sure how I can fit the writing into my program loop in any way that would work with the various interrupts etc. Anyone have ideas on how to do background buffering and writing to the SD card?

Any other ideas on speeding up SD card writing would be helpful. Is there overhead in each write from the SD card library that is significant? Is it worth keeping a small buffer and writing fewer times?

pito

Thu, 03 Jan 2013 00:31:27 +0000

The max SPI clock speed you can set (with a pic32@80MHz) is 20MHz. So the max theoretical rd/wr speed to an sdcard is ~2.2MBytes/second. Usually the rd/wr speeds are about 800/350kBytes/sec. You may use DMA or FIFO to speed up the block transfer. The highest speeds I've seen with an sdcard and pic32@80MHz are 680kBytes/sec write and 1740kBytes/sec read. Pls mind an sdcard may have a significant write latency (up to 250msecs, typically ~80-100msecs) so if you going to stream data to the sdcard you need a buffer which compensates for the latency. In order to speed up the existing SD driver you may consider the one we are using within the retrobsd.org project (FIFO-based 32bit SPI transfer). Just try to replace the rd/wr block transfer routines - it might help ;) . P.

spencoid

Thu, 03 Jan 2013 01:56:56 +0000

thanks for the suggestions. i would like to write a FIFO that sends data to the SD card when the main loop is running. my input is written to an array from an interrupt routine when a step pulse is seen at the interrupt pin. there is most likely plenty of time to do the writing as long as the FIFO is large enough. unfortunately i have no idea how to do this although i could figure it out eventually. if anyone has the code to do this, it would save me a lot of figuring time :)

i downloaded your code and might be able to work my way through it to find the rd/wr block transfer routines and possibly even patch them into the right place but it would sure help if you could copy them to me and tell me where to stick them (politely if possible) i did look at the download and have no idea what to do with the .img file. i am not a unix guy. also i am using a chipkit uno, not a max, in case that changes anything

majenko

Fri, 04 Jan 2013 13:36:22 +0000

The code you are looking for is in sys/pic32/rd_sd.c and sys/pic32/spi_bus.c

void spi_bulk_write_32_be(int dno, unsigned int len, char *data)

is the function that does the block transfer of data over SPI at high speed / low latency. It makes use of the enhanced SPI buffer and 32 bit data transfers.

spencoid

Fri, 04 Jan 2013 18:03:42 +0000

the only thing i got in your download is filesys.img i have no idea how to expand this into the code files. is this the wrong download? once i find the required code, where in existing .h files do i put it ? what should it replace? is it necessary to call a different function or does the read and write code exactly replace something else?

majenko

Sat, 05 Jan 2013 00:02:23 +0000

You need to check out the latest version from the SVN repository - [url]http://code.google.com/p/retrobsd/source/checkout[/url]

From there you examine the source code in the files I mentioned, then also examine the source code for the SD library. You take the essence of how the RetroBSD system works, and adjust the SD library accordingly. It will not be a simple copy and paste, but it will involve you understanding both how the PIC32 works, and how the SD library works.

It will make you a better programmer.

spencoid

Sat, 05 Jan 2013 06:40:17 +0000

why do you need to make this such a chore? i am not interested in figuring out which file contains the source code. it is not obvious to someone who does not know how this is all arranged. i have checked out the link you gave me and it is not at all obvious where the files you refer to are. it might make sense to you because you already know. i can figure out code but i can not waste all this time just trying to find the stuff.

pito

Sat, 05 Jan 2013 23:30:29 +0000

The best way to proceed is to involve Bill (aka fat16lib, the author of the SDFat library) - he has recently ported it to Teensy 3.0 and Due. The original arduino's SD library is based on an very old version of the SDFat, the newest SDFat is by far better.. http://arduino.cc/forum/index.php/topic,137847.0.html http://arduino.cc/forum/index.php/topic,128335.0.html P.

spencoid

Sat, 05 Jan 2013 23:48:40 +0000

i got some faster cards and changed the speed to full in sd.cpp. my speed is almost fast enough but not quite. the old cards would not initialize at full speed so obviously they were a big part of the problem.

i thought i would need to make some sort of buffer for the incoming data but when the processor is writing the card, it blocks everything else even interrupts. i figured i would need a separate controller to buffer the input or the output to the card but this looks like a really complicated hardware solution. there are some cpld based systems that threaten to do it but i am so close all i probably need is a better sdfat as you suggest.

for me, the problem is how to replace the old sdfat and have it still work on the chipkit32. there were serious problems SD cards writing on the chipkit 32it until the most recent test IDE. i forget exactly what bug was fixed but i need to make sure that i am not replacing something with a version that does not work.

should it be possible to just replace sdfat.h and .cpp with newer ones and have it work on the chipkit32 or does anything need to be edited or anything else changed to maintain compatibility?

spencoid

Sun, 06 Jan 2013 02:53:35 +0000

i did a bit of file comparison and i am somewhat convinced that the version of Sdfat that the chipkit MPIDE test version uses is the new one. can others confirm this?

any other opportunities to improve the SD card write speed that i might not have thought of and that would not be a huge effort? i just need to buffer some incoming data while the card is being written. i can run some benchmarks but i can't imagine that i need to buffer more than 1k bytes.

if i use a separate microcontroller like a teensy++2 to buffer and write the SD card data, what would be the best way to interface it to the chipkit uno?

spencoid

Sun, 06 Jan 2013 17:55:25 +0000

i found the following in an article on SD cards. does anyone know if the SDFat library as adapted to the Chipkit waits for the card to finish writing or if it releases the CS line to the card and only gets back to it on the next write? it is well beyond my abilities to examine the code and figure this out although i will try.

if the microcontroller is sitting waiting for the card to finish writing and if this can be changed to only wait for a response from the card on the next write, this could speed it up considerably. if anyone knows this and or can make the change it would be much appreciated. here is the quote

In principle of the SPI mode, the CS signal must be asserted during a transaction, however there is an exception to this rule. When the card is busy, the host controller can deassert CS to release SPI bus for any other SPI devices. The card will drive DO signal low again when reselect it during internal process is in progress. Therefore a preceding busy check (wait ready immediataly before command and data packet) instead of post wait can eliminate waste wait time. In addition the internal process is initiated a byte after the data response, this means eight clocks are required to initiate internal write operation. The state of CS signal during the eight clocks is negligible so that it can done by bus release process described below.

spencoid

Sun, 06 Jan 2013 20:41:29 +0000

there is a port of FatFS to the chipkit32. does anyone have a sample sketch showing the use of this for basic read and write? just need to know how to initialize and write to and read from SD card file. the information i need seems to be spread through a bunch of files and i am having a difficult time figuring it out.

pito

Mon, 07 Jan 2013 11:07:21 +0000

http://elm-chan.org/fsw/ff/00index_e.html

In Resources - FatFs sample projects - take ie. PIC24 demo - there is a small shell showing how to apply various fatfs functions..

spencoid

Mon, 07 Jan 2013 17:50:29 +0000

i have downloaded this file three times and have looked at everything in it. all i can find is C code, not a sketch simple or otherwise. i can not find a sketch that shows Arduino style coding. all i want to do is create open write to and read files. it is not set up in the "library" style that i am accustomed to so i need to what to include in my sketch to get fatFS to work. i have tried to read all the c code and figure out what the public methods are, what they do and how to call them but i have not succeeded yet. this is why i want any sort of sample sketch that i can get working so i know i have all the right stuff in the right places.