Compression of log data to extend storage

sonalideo

Since the memory on the board is limited to around 10K entries, is it possible with on board data processing to compress/decrease the resolution of the raw data from for example high frequency accelerometer sampling.

Example, if raw sensor data is 4 bytes per axis?, is it possible to compress down to 1 byte per axis, something like:

acceleration = ((int) x/10000)*10000 + ((int) y/10000)*100+ ((int) z/10000)

or take data from sampling 1-10 and store it as one log entry instead of 10?

Matt

Hi Sonalideo.  Presently the accelerometer and gyro output are 6 bytes in total -- 2 bytes (16 bits) per axis.

Using the data processing module, you could apply a root mean square (RMS) or root sum square (RSS).  RSS is equivalent to the vector magnitude when used with the accelerometer or gyro.

RSS = sqrt(x^2 + y^2 + z^2)

If you are willing to lose some precision, you could then use the simple math filter to apply a bit shift or divide to reduce the data to a single byte.

We do not yet have a programmable way to pack four of those bytes into a single log entry, but it is something that we have considered and will add in a release sometime soon.  It would take an input data source with length, and pack bytes until its output length is filled.  This way it could pack four of your bytes into one event.  It has some practical use in increasing BTLE throughput as well.

Depending on how you want to use the data there are other techniques you could use.  In our fitness example app, we aggregate motion data by accumulating and then reporting periodically.  For example, obtain the vector magnitude, sum it over time, and then log the total count every minute.  This is all possible with existing filters in the MetaWear data processor.

sonalideo

Hi Matt.

Thanks for the info! I'm getting to understand a bit more now the underlying mechanics of the logging.

We need the acceleration in each axis so a mean or sum are not feasible but would it be possible to take the 2 bytes axis data and compress it down to 1 byte and then bit shift it so it would fit into one log entry of 3 bytes?

Matt

Hi Sonalideo, there is no way to do this at present with the filters we have, but I can certainly see the value.  

We have a firmware release coming up very soon, and I will see how we can enable this -- likely in a way a way that is generally useful for other purposes as well.

Matt

@sonalideo

In a soon to be released firmware, the math operation filter will be enhanced to provide multiple input functionality.  For the use case you described, it will allow you to:

• Take 3 channels of 16 bits as input
• Right shift each channel by 8 bits
• Produce an output of 1 byte per input channel

This will do exactly what you describe in a single data processor stage.

sonalideo

That is great news Matt! Looking forward to testing it.

sonalideo

Until the boards with extended memory is released. Is there some other way to be able to squeeze more data into a log entry, for example having one entry with data for the last minutes of measurements? 

I guess now that some memory for a log entry is used for meta data.

Eric

Part of the memory allocated for a log entry is for a timestamp and an identifier.  I suppose a timestamp is unnecessary if the user is collecting data at a fixed rate and knows either the start or end time.  It's something worth considering.

sonalideo

Yes, Would be great if possible to aggregate say 10Hz measurements the last minute as one log entry instead of 600.

Eric

A log entry is not that big.  There is no way to compact 600 data samples into a few bytes  unless you use the data processor.

sonalideo

Now with the new firmware is it possible to take data from multiple samples and make them into one log entry to avoid the scale and the timestamp taking up space. I.e. something like this:

mwBoard.getModule(Accelerometer.class).routeData().fromAxes().process( wait for 100 samples then make one log entry).log("acceleration")

And then retrieve as:

RouteManager.MessageHandler() { public void process(Message message) { List<Float> entries = message.getData(FloatList);}   //{expexted x1,y1,z1,x2,y2,z2....

Can Sample or Buffer class be used in some way?

The data in a Buffer, is it processed on the board or on the android device?

In addition can you provide sample code on how to do the bitshift that has been added to the firmware?

Matt

@sonalideo : The logger works with 4 byte words, variable length entries are not supported at this point.  The technical complexity is very high to make variable length entries work reliably with power cycling, readout disconnections, etc -- and the maximum overhead reduction is only 50% in the best case, which is one entry filling the entire log.

The best gains are to take advantage of the asynchronous nature of the log.  For example, if you are developing a drop detector, the data processor can be used to only log entries around that drop event.  The finite number of drops means you can get 100 or even 1000x capacity improvement by pruning the data.

Data can be streamed at close to 200Hz to gather enough data to come up with a compression scheme like the one suggested above.

The primary use of the buffer class is to convert an asynchronous data stream into a synchronous one.  For example, imagine that you have a step detector that sends an event whenever there is a step, but you want to precisely log the steps per 10 minutes.  The steps can be accumulated and then stored in a buffer.  A timer can read that buffer precisely  every 10 minutes.  This turns the step input which may occur unpredictably into a consistent source.

The buffer exists in the firmware.

I'll see if the mobile guys can comment on using the multi-input bit shift.  Do you prefer iOS or Android examples?

Eric

For the new math features, something like this:

accelModule.routeData().fromAxes().stream("accelSub")

        .process(new Maths(Maths.Operation.RIGHT_SHIFT, 8)).stream("shift_sub")

.commit().onComplete(new CompletionHandler<RouteManager>() {

    @Override

    public void success(RouteManager result) {

        result.subscribe("accelSub", new RouteManager.MessageHandler() {

            @Override

            public void process(Message msg) {

                Log.i("test", "Original: " + msg.getData(CartesianFloat.class).toString());

            }

        });

        result.subscribe("shift_sub", new RouteManager.MessageHandler() {

            @Override

            public void process(Message msg) {

                Bmi160ThreeAxisMessage bmiMsg= (Bmi160ThreeAxisMessage) msg;

                float x= (bmiMsg.getData()[0] << 8) / bmiMsg.getScale();

                float y= (bmiMsg.getData()[1] << 8) / bmiMsg.getScale();

                float z= (bmiMsg.getData()[2] << 8) / bmiMsg.getScale();

                Log.i("test", String.format("Shifted: (%.3f, %.3f, %.3f)", x, y, z));

            }

        });

    }

});

Since you are right shifting data, the default computations from the Message object will not be valid.  You can reconstruct the data by left shifting each byte and dividing by the scale..