chipKIT® Development Platform

Inspired by Arduino™

SD card worst case write time

Created Mon, 01 Sep 2014 11:27:03 +0000 by tcsaba101

tcsaba101

Mon, 01 Sep 2014 11:27:03 +0000

I am preparing the datalogger to store a car battery current.

Yestreday I done some measurements to make the final specs. for logging requirements. I used a 34970A DAQ and a 34901A multiplexer card.

The enclosed pictures shows two cars starter current samplings on a 300A shunt at 42ms rate. Officially this unit should go close to 20ms sampling, but I have made some mistakes as I found out today.

The most important things are turned out: I expected around 150A peaks, but as you can see on the pictures the 3.0 liter engine's starter draws 400+A peak, and 350A the 1,4 liter engine.

And the sampling rate during starting cycle have to be less then 40ms, because the bigger engine current steps within 40ms to the peak, shorter sampling needed to track the change. In the other sampling the 40ms would be ok, there is a sample during the falling edge.

I am just curious how fast rate can on an SD card written? What is the max continous samling rate that can be logged to an SD? What is the max time period while the SD can't accept further records?

Please let me know your practical experience.

Csaba

GrahamM242

Tue, 02 Sep 2014 13:27:31 +0000

I suspect you'll find that there's a combination of factors that affect this! Factors I can think of on a lunch break include:

  • Used flash blocks take longer to write, as an entire page needs an erase cycle first. Remember that simply deleting a file from a file system does not erase the blocks that held the data!
  • The size of pages on the flash - erase page size can affect the speed of the erase cycle.
  • How many blocks you try to write at a time. The write multiple block command (CMD25) lets you send multiple blocks. It is likely the card will go busy when the write buffer is full.
  • The size of the write buffer on the SD card.
  • The SD Card class. This [i]should[i/] set the minimum performance.
  • How fast you clock data to the card!

I'm sure others will have a better idea of this than I.

tcsaba101

Wed, 03 Sep 2014 20:01:46 +0000

Good points. Thanks.

I am breeding the idea to make an application specific log about the peak writing delays to the card.

The application is collecting 128 sample records, by 10ms or 1 sec sampling rate depending on the current value changes. When the buffer is full, it writes to the SD the whole buffer what will be 10 bytes/ record alltogether 1280 bytes. The average writing cycletime for this amount should be 250usec on a Class 4 card (4Mbyte/sec). I have class 4 and 10 cards.

The SPI clock in theory should be 25MHz in standard chipkit setup. That clock writes the 1 kB in 400usec to the card.

Both above duration would be very fine, I have close to 10ms between samplings for this task.

The erase cycle can puch long delays in the write cycle.

It will turn out in the testing.

GrahamM242

Thu, 04 Sep 2014 09:26:38 +0000

It'll be interesting to see what happens! ;)

pito

Mon, 08 Sep 2014 10:00:45 +0000

The biggest issue with an sdcard is the "write latency". It could be up to 250ms, typically 10-50ms at random times (quite often, btw). So in order not to loose your data to be written onto the sdcard, you need a fifo, which will buffer the data when the sdcard is not accepting the data.. An example:

  1. the data rate is 512bytes each 10ms, that is 50kB/sec
  2. the worst case write latency is 250ms
  3. the fifo buffer shall be 250ms * 50kB/sec = 12.5kB large.

tcsaba101

Mon, 08 Sep 2014 17:55:30 +0000

The biggest issue with an sdcard is the "write latency". It could be up to 250ms, typically 10-50ms at random times (quite often, btw). So in order not to loose your data to be written onto the sdcard, you need a fifo, which will buffer the data when the sdcard is not accepting the data.. An example:

  1. the data rate is 512bytes each 10ms, that is 50kB/sec
  2. the worst case write latency is 250ms
  3. the fifo buffer shall be 250ms * 50kB/sec = 12.5kB large.

That is really a serious issue! Till now I had made only 1sec logging.

This case the 10ms record size is 14 bytes so I need 350 bytes to buffer, that is not a problem.

Can you explain where the write latency will be experienced in practice? My code part is for writing the card:

String dataString = String(ms10log_cnt) + ", " +  String(ms10_buffer[i].timestamp) + ", " + String(ms10_buffer[i].uBat_samp_mV) 
		  		+ ", " + String(ms10_buffer[i].iBat_samp_mA);
		  
		  if (logFile)
		  {
		    logFile.println(dataString);
		   }
			else
		  {
		    printError("Ms10 write failed");
		    return;
		  }
		  
		  //Increment ID number
		 	ms10log_cnt ++;

Will be delayed during writing the data (logFile.println()), or the write cycle will be rejected and "if(logFile)" will be false? Maybe it depends on the length of the string?

What is the correct solution to handle the write latency?

Thanks!

pito

Mon, 08 Sep 2014 18:56:31 +0000

Write latency occurs anytime randomly for a random time. The vendors guarantee 250ms max afaik (at least Sandisk does).

The writing latency is related to the internal housekeeping the sdcard does, most notably with wear leveling (it shuffles the data internally) - so the card lives its own life, no way how to predict what happens next few milliseconds.

As you may see from my graphs below, the pattern is just a "noise" with 2.5ms - 40ms write latencies 99.99% of the time. When you run the logging for longer time, ie. 10 minutes, you will catch latencies with ~100ms durations as well.

The only way how to mitigate the writing latency is to use FIFO buffer: 1.The "producer process" writes the data (ie. from sensors) full speed_1 into the FIFO (instead directly to the sdcard) when FIFO not full 2.The "consumer process" reads the data from the FIFO (when FIFO not empty) and writes them onto the sdcard, with a speed_2 the sdcard allows. 3.You have "watermarks" with the FIFO record pointers and you can see how the FIFO fills in with data and how it empties.

So even the sdcard will not accept data for let say 100ms from sensors during a WL hit, the FIFO fills in with those data, and when the sdcard is willing to accept the data the FIFO is emptied. The prerequisite here is the FIFO is large enough in order it can absorb enough data records before it becomes full (then you will get data overruns).

I did it with Nilrtos, you may use any rtos for that (you have to create producer and consumer processes). Maybe it is doable without an rtos as well.. P.

PS: http://forum.arduino.cc/index.php/topic,144715.0.html http://forum.arduino.cc/index.php?topic=144715.msg1092515#msg1092515 and search the arduino forum for a lot of other info on Nilrtos.. NilRtos is great for this kind of problems, not sure it works with chipkit, but I would recommend it..

tcsaba101

Tue, 09 Sep 2014 11:53:29 +0000

pito, thank you very much!

I started this post to collect exactly the same info what you posted!

I will search for a fifo buffer implementation in C++ or I create my own. Your tutorial helps a lot.

On the SD write side there should be a timing to the maximum waiting time within to write the data. Do you recommend 250ms delay as you posted? Over 250ms latency there should be an SD writing error justification.

Thanks once more.

pito

Tue, 09 Sep 2014 16:36:29 +0000

There is a FIFO implementation in the NilRtos which works, for example (for arduino): https://github.com/greiman/NilRTOS-Arduino/tree/master/libraries/NilRTOS The NilRtos comes from chibios as the "almost nil version of chibios": http://forum.chibios.org/phpbb/index.php There is a chibios port for pic32mx btw, maybe applicable (I did some benchmarks long time back).

The FIFO tries to write the data records to the sdcard as fast as possible (you do not care on delays). When writing latency hits, the sdcard write call will last for XX msecs (the latency duration) and the FIFO starts to fill itself with incoming records. When the sdcard's WL drops the FIFO empties. As the WL hit always, the FIFO "pumps" up and down..

The indication your system does not perform is visible on the FIFO watermark indicating the data overruns (the FIFO does not empty itself fast enough and remains full, not accepting data). So the FIFO size (for a specific data rate) is what determines the ability to cover even the largest WL.

If your FIFO is 10kB large, and you write 50kB/sec data, an FIFO overrun means there are 200ms latencies (plus/minus).

Mind the filling in the FIFO with records and writing FIFO records to the sdcard are two independent asynchronous processes synchronized by semaphores (in case of NilRtos).

tcsaba101

Tue, 09 Sep 2014 19:07:50 +0000

Ok I understood.

Usually I keep the program flow in my hands as much as possible and comfortable.

From the attached forum links I feel this NilRTOS is not really for PIC32 environment.

I plan to implement the fifo in my own coding.

Thanks for the help, it saved me many dead ends.

pito

Tue, 09 Sep 2014 20:13:27 +0000

Btw to catch the battery current peaks you need a "peak detector" (ie 2 opamps, diode, cap, mosfet for reset), otherwise you will miss the peaks. The 10ms resolution is far too slow to see those current peaks, I think. So I would read the peak detector with the adc, then reset the peak detector to zero at the end of a measurement, and so on (10ms apart). So you will get much better information on the current as it is now. Simplified: [attachment=0]pd.jpg[/attachment]

tcsaba101

Wed, 10 Sep 2014 14:59:34 +0000

Thanks the advice.

I will see the 10ms sampling as soon as I will be ready of the logger, then think over your suggestion.

From the point of my application the 10ms sampling should be far enough.

The INA226 is continously integrating the current and voltage in 9,54ms cycles and I read the result in every 10ms.

I will use this data to program an artificial load what simulates the car load/charge on the battery. My goal is to simulate the car as a charge (current * time period) push/pull device. For this purpose the present measuring and logging method seems to be fine.

The peak detection would be an interesting information what happening in a car during start.

To make a 10ms long 400A pulse will be another interesting project.

tcsaba101

Sun, 14 Sep 2014 15:31:14 +0000

Contimued: http://chipkit.net/forum/viewtopic.php?f=24&t=3046&p=12574#p12574