Downloading logs

stu68

I'm trying to use the new Route Manager, but am having difficulty figuring out exactly how to download the log. I am trying to just get the data from the X axis on the Accelerometer for now. I looked at the new API and Sample app, but it only seems to have Streaming functionality.

Is there any kind of quick sample code available for properly setting up and downloading offline logging with the new methods?

I've got this so far for starting the process:

AsyncOperation<RouteManager> routeManagerResult= accelModule.routeData().fromXAxis().log("accel_log").commit();
routeManagerResult.onComplete(new AsyncOperation.CompletionHandler<RouteManager>() {
    @Override
    public void success(RouteManager result) {
        accelModule.enableAxisSampling();
        accelModule.start();
        logModule.startLogging(true); // Starts logging data overwrite is true
    }
});

Eric

Logging Documentation

https://mbientlab.com/androiddocs/index.html#logging

You can also check out the freefall detector app we built as part of a video tutorial.

https://github.com/mbientlab-projects/FreeFallDetector/blob/master/app/src/main/java/com/mbientlab/freefalldetector/MainActivity.java

stu68

That freefall sample looks like what I need. I'll give it a try - thank you! Is the video tutorial released yet or a wip?

Eric

We just finished the Android portion of the tutorial videos which are on the MbientLab website, link here.  Scroll down to the "MetaWear Advanced Programming" section.

stu68

1.) When I run the FreeFall demo I get NullPointerException on

loggingModule.startLogging(true);

It happens when clicking on Start.

2.) I got around this in my own app, but I can't seem to get the data after downloading the log. 
After I download the log should the downloadhandler invoke the 

public void success(RouteManager result) 

in
 
public void onServiceConnected(ComponentName name, IBinder service)

or do I need to call it somehow after I download the log? I just want to be able to get 
the G values of X Axis Accelerometer data into an ArrayList. I don't see how to do this
in any of the documentation or anything.

Thank you for your help!

Eric

We should back up a bit.  First, do you have working code to stream accelerometer data to your device?  If you don't, then you should be focusing on getting a live stream working first before you try logging.  You can use the code snippets from the Android docs page to get a stream going:

https://mbientlab.com/androiddocs/index.html#dr-stream-log

When you have streaming working, then follow the instructions in the last sentence of the "Route Manager" to setup the log handlers (https://mbientlab.com/androiddocs/index.html#dr-route-manager).  Then, retrieve a reference to the Logging class and start logging / download data (https://mbientlab.com/androiddocs/index.html#logging).

The DownloadHandler class simply gives you some progress updates on the log download.  It does not need to interact with anything else.

stu68

I did get stream working. Here's what I got so far with the attempt at logging. I guess I don't understand what "triggers' the success(RouteManager result). I have everything else working, I am just not clear on this new approach how to get the data into an ArrayList. I hope it's ok to post my code here on where I'm at so far.

private MetaWearBleService.LocalBinder mwService = null;
private String deviceAddress;
private MetaWearBoard mwBoard;
private Logging logModule;
private Accelerometer accelModule;
private float axisData;

connectButton.setOnClickListener(new View.OnClickListener() {
    accelerometerMe(); //Didn't include all code here but it calls this to start

@Override

public void onServiceConnected(ComponentName name, IBinder service) {
    ///< Get a reference to the MetaWear service from the binder
    mwService = (MetaWearBleService.LocalBinder) service;

    final BluetoothManager btManager =
            (BluetoothManager) getActivity().getSystemService(Context.BLUETOOTH_SERVICE);
    if (deviceAddress != null) {
        final BluetoothDevice btDevice = btManager.getAdapter().getRemoteDevice(deviceAddress);
        this.mwBoard = mwService.getMetaWearBoard(btDevice);
        this.mwBoard.setConnectionStateHandler(new MetaWearBoard.ConnectionStateHandler() {
            @Override
            public void connected() {
                try {
                    logModule = mwBoard.getModule(Logging.class);
                    accelModule = mwBoard.getModule(Accelerometer.class);
                    setupAccelerometer();    //Sets range & data rate
                    //getReadings();
                } catch (UnsupportedModuleException e) {
                    e.printStackTrace();
                }
            }
            @Override
            public void disconnected() {
                Log.i(TAG, "Disconnected");
            }
        });
        mwBoard.connect();
    }
}

stu68

public void accelerometerMe() {
    accelModule.routeData().fromAxes().log("accelLogger").commit()
            .onComplete(new AsyncOperation.CompletionHandler<RouteManager>() {
                @Override
                public void success(RouteManager result) {  //How do you invoke or trigger this??
                    result.setLogMessageHandler("accelLogger", new RouteManager.MessageHandler() {
                        @Override
                        public void process(Message msg) {
                            axisData = msg.getData(float.class);
                            Log.i("test", String.format("Log: %.1f", axisData));
                        }
                    });

                }

                @Override
                public void failure(Throwable error) {
                    Log.e("test", "Error committing route", error);
                    //accelSetup = false;
                }
            });
    if (isFirstRun) {
        getActivity().runOnUiThread(new Runnable() {
            @Override
            public void run() {
                setButtonStateConnected();
            }
        });
    }
    logModule.startLogging();
    accelModule.enableAxisSampling();
    accelModule.start();
    downloadHandler.postDelayed(downloadLogs, pauseTime);
}

stu68

private final Runnable downloadLogs= new Runnable() {
    @Override
    public void run() {
        Log.i("AccList1 Size", String.valueOf(accList1.size()));
        // Stop Logging
        logModule.stopLogging();
        logModule.downloadLog(0.1f, new Logging.DownloadHandler() {
            @Override
            public void onProgressUpdate(int nEntriesLeft, int totalEntries) {
                if (nEntriesLeft == 0) {
                    Log.i(TAG, "Log download complete");
                }
            }
        });
        accList1.add(axisData);  //THIS IS WHERE I WANT TO FILL THE ARRAYLIST
        isFirstRun = false;
        counter = 0;
        
        accelerometerMe();  //INTENTION IS TO LOOP AROUND AND GET MORE DATA EVERY FEW SECONDS
    }
};

JpEncausse

When you call the DownloadHandler it will call the LogHandlers you set before. So you have to fill your ArrayList in your LogHandler:

    @Override

    public void process(Message message) {

        messageList.add(message);

        valueList.add(this.name);

    }

Then later the download code is called and you can iterate on the arraylist to format what you want.

stu68

I tried it there, but the problem seems to be invoking the LogHandler in the Route Manger. It skips over that every time.

Eric

Your use case seems strange to me.  You are logging data, then periodically downloading it at every few seconds.  This defeats the purpose of logging and it seems like you should just stream data to your device as that is effectively what you are doing with the logger but in a more roundabout manner.

As for your code snippets, you only need to add routes once.  Second, your message handler is typecasting the message data to the wrong type.  Routing data from all axes requires you to cast to CartersianFloat (or CartesianShort).  You should be setting plenty of error messages in Logcat saying float is not supported for the Message.  Third, the only thing your downloadLogs runnable should be doing is calling downloadLog.

You are trying to do to much in your switch to logging.  You need to take this step by step:

1. Start with your streaming code, convert it to use the logger as outlined in the docs (https://mbientlab.com/androiddocs/index.html#dr-route-manager).
2. Add calls to startLogging / stopLogging / downloadLog and get a simple start -> stop -> download loop going that you control manually.  The process will look something along the line of this commit and this video.  
   
3. Test the changes to make sure you can download the logged data without error and convince yourself the code is correct
   
4. Setup the periodic calls to downloadLog.

stu68

This is for a high frequency application/ project and I need to have the accelerometer at 200Hz. We were having problems getting the data to stream over ble at that rate. I had to develop an algorithm to get periodic samples for a vibration sine wave measurement (around 500 rpm as high as 800 rpm). I'm taking those samples and stitching them together to get a complete sine wave measurement. It would be way easier to stream, but we were told it wasn't possible over 50 or 100 Hz. The project requires high accuracy values for displacement calculations and 200Hz was the lowest we could go on the accelerometer to get good data.

I had all this working with the previous API. I am trying to convert my working code over to the new API.

I was going off this for the message type for single axis:

Three axis data can be interpreted as a CartesianFloat (G's) or a CartesianShort (milli G's) whereas single axis data can be interpreted as a float (G) or short/integer/long (milli G).

1.) I did do that and was able to get the streaming to start and stop. I then converted it to logging and I can get the logging to start and stop and download, but I can't get the downloadLog call to invoke public void success(RouteManager result). Stop, Start and Downloading the log all work.

2.) I had all that working with streaming and then logging and then tried to put it in the loop. The loop works to start, stop and download the log but never processes data. I do get a log message that the download is complete.

Again, thank you guys for your patience and helping me with this. My client is looking at 200 units/yr on this project.

Eric

Your most recent code snippet showed "fromAxes()" as the source and casting the data as a float but I guess that was just a typo.

Try overriding the receivedUnknownLogEntry method in the DownloadHandler and see if it is called with any unknown log data.  

What code is in your loop now, if anything has changed from the previous code snippets?  The Runnable should only download the log, something like the follow:

private final Runnable downloadLog= new Runnable() {
    @Override
    public void run() {
        logModule.downloadLog(0.1f, new Logging.DownloadHandler() {
            @Override
            public void onProgressUpdate(int nEntriesLeft, int totalEntries) {
                Log.i("test", String.format("Progress: %d/%d/", nEntriesLeft, totalEntries));

                if (nEntriesLeft == 0) {
                    Log.i("test", "Log download completed");
                    downloadHandler.post(downloadLog);
                }
            }

            @Override
            public void receivedUnknownLogEntry(byte logId, Calendar timestamp, byte[] data) {
                Log.i("test", String.format("Unknown log entry: {id: %d, data: %s}", logId, Arrays.toString(data)));
            }
        });
    }
};

Setting up the data route should only be done once in a setup function.

stu68

The fromAxes was a typo, but I changed it back for testing.

One question with the new API - do I still need to stop the logger and/or acc modules to download the log? In the previous API we had to stop both, download the logs, then start them back up again.

I add the receivedUnknownLogEntry method and here are the last few entries of the results:
09-04 03:36:42.634  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 3, data: [-28, -1, 0, 0]}
09-04 03:36:42.634  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 4, data: [27, -4, -8, -1]}
09-04 03:36:42.634  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 5, data: [-28, -1, 0, 0]}
09-04 03:36:42.634  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 6, data: [27, -4, -8, -1]}
09-04 03:36:42.644  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 7, data: [-28, -1, 0, 0]}
09-04 03:36:42.644  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 0, data: [27, -4, 0, 0]}
09-04 03:36:42.644  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 1, data: [-36, -1, 0, 0]}
09-04 03:36:42.644  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 2, data: [27, -4, 0, 0]}
09-04 03:36:42.644  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 3, data: [-36, -1, 0, 0]}
09-04 03:36:42.654  31722-31869/com.project.name I/test﹕ Unknown log entry: {id: 4, data: [27, -4, 0, 0]}
09-04 03:36:42.654  31722-31869/com.project.name I/test﹕ Progress: 0/13056/
09-04 03:36:42.654  31722-31869/com.project.name I/test﹕ Log download completed

I only get this on the first run through the loop. Then I only get this:
Skipped 4719 frames!  The application may be doing too much work on its main thread.
on subsequent loops.

It still never goes through the route manager to process. I also tried changing the delay time from 3 seconds to 100 ms and I still got 13056 records from receivedUnknownLogEntry.

stu68

I moved the route manager stuff into onServiceConnected:

@Override
public void onServiceConnected(ComponentName name, IBinder service) {
    ///< Get a reference to the MetaWear service from the binder
    mwService = (MetaWearBleService.LocalBinder) service;

    final BluetoothManager btManager =
            (BluetoothManager) getActivity().getSystemService(Context.BLUETOOTH_SERVICE);
    if (deviceAddress != null) {
        final BluetoothDevice btDevice = btManager.getAdapter().getRemoteDevice(deviceAddress);
        this.mwBoard = mwService.getMetaWearBoard(btDevice);
        this.mwBoard.setConnectionStateHandler(new MetaWearBoard.ConnectionStateHandler() {
            @Override
            public void connected() {
                try {
                    connectingDialog.cancel();
                    logModule = mwBoard.getModule(Logging.class);
                    accelModule = mwBoard.getModule(Accelerometer.class);
                    setupAccelerometer();
                    accelModule.enableAxisSampling();
                    accelModule.start();
                    accelModule.routeData().fromAxes().log("accelLogger").commit()
                            .onComplete(new AsyncOperation.CompletionHandler<RouteManager>() {
                        @Override
                        public void success(RouteManager result) {
                            result.setLogMessageHandler("accelLogger", new RouteManager.MessageHandler() {
                                @Override
                                public void process(Message msg) {
                                    axisData = msg.getData(CartesianFloat.class);
                                    Log.i("test", String.format("Log: %.1f", axisData));
                                    //accList1.add(axisData);
                                }
                            });

                        }

                        @Override
                        public void failure(Throwable error) {
                            Log.e("test", "Error committing route", error);
                            //accelSetup = false;
                        }
                    });
                } catch (UnsupportedModuleException e) {
                    e.printStackTrace();
                }
            }

            @Override
            public void disconnected() {
                Log.i(TAG, "Disconnected");
            }
        });
        mwBoard.connect();
    }
}

stu68

Here is my loop code:

private void accelerometerMe() {
    if (!isStoppedPress){
        //accelModule.start();   //Not sure if stopping and start is still needed
        logModule.startLogging(); //Not sure if stopping and start is still needed   
        downloadHandler.postDelayed(downloadLog, pauseTime);
    }
    else{
        logModule.stopLogging();
        cleanUp();
        setButtonStateDisconnected();
    }
}

private final Runnable downloadLog= new Runnable() {
    @Override
    public void run() {
        logModule.stopLogging();       
        //accelModule.stop();
        logModule.downloadLog(0.1f, new Logging.DownloadHandler() {
            @Override
            public void onProgressUpdate(int nEntriesLeft, int totalEntries) {
                Log.i("test", String.format("Progress: %d/%d/", nEntriesLeft, totalEntries));

                if (nEntriesLeft == 0) {
                    Log.i("test", "Log download completed");
                    downloadHandler.post(downloadLog);
                }
            }

            @Override
            public void receivedUnknownLogEntry(byte logId, Calendar timestamp, byte[] data) {
                Log.i("test", String.format("Unknown log entry: {id: %d, data: %s}", logId, Arrays.toString(data)));
            }
        });
        accelerometerMe();
    }
};

private final Handler downloadHandler= new Handler();

Eric

With all these unknown log entries, it looks like you are not resetting the board in between your debugging sessions, and simply tacking on more routes.  Try resetting the board and rerun the code.

Since you're calling accelerometerMe, remove the additional call to downloadHandler in the onProgressUpdate function.

stu68

Do I need to stop logging before downloading; as in previous API?

Eric

You can download while logging.

stu68

I have the logging mostly working now, but when downloading it seems to hang while downloading and never reports nEntries left == 0. Here is sample log entries and code:

09-08 12:10:19.418  13043-13285/com.project.name I/test﹕ Progress: 751/5103/
09-08 12:10:19.588  13043-13285/com.project.name I/test﹕ Progress: 495/5103/
09-08 12:10:19.748  13043-13285/com.project.name I/test﹕ Progress: 239/5103/    (It just stops here)

private final Runnable dlLog= new Runnable() {
    @Override
    public void run() {
        logModule.downloadLog(0.05f, new Logging.DownloadHandler() {
            @Override
            public void onProgressUpdate(int nEntriesLeft, int totalEntries) {
                Log.i("test", String.format("Progress: %d/%d/", nEntriesLeft, totalEntries));

                if (nEntriesLeft == 0) {
                    Log.i("dlLog", "Log download completed");
                    downloadHandler.post(dlLog);
                    
                }
            }

            @Override
            public void receivedUnknownLogEntry(byte logId, Calendar timestamp, byte[] data) {
                Log.i("test", String.format("Unknown log entry: {id: %d, data: %s}", logId, Arrays.toString(data)));
            }
        });
    }
};

stu68

Follow-up - it seems to hang (see previous post) when downloading data over 1000 records or so. I lowered the loop time to only download 200 or so records at a time and now I get this after a few downloads from the logger:
...several good downloads then...
09-08 14:28:56.788    6635-6887/com.project.name I/dlLog? Log download completed
09-08 14:28:56.788    6635-6887/com.project.name I/VibrationMeterFragment? accList1 size: 536
09-08 14:28:59.648    6635-6887/com.project.name I/test? Progress: 132/146/
09-08 14:28:59.688    6635-6887/com.project.name I/test? Progress: 118/146/
09-08 14:28:59.748    6635-6887/com.project.name I/test? Progress: 103/146/
09-08 14:28:59.818    6635-6887/com.project.name I/test? Progress: 89/146/
09-08 14:28:59.878    6635-6887/com.project.name I/test? Progress: 75/146/
09-08 14:28:59.908    6635-6887/com.project.name I/test? Progress: 61/146/
09-08 14:28:59.928    6635-6887/com.project.name I/test? Progress: 47/146/
09-08 14:28:59.948    6635-6887/com.project.name I/test? Progress: 33/146/
09-08 14:28:59.968    6635-6887/com.project.name I/test? Progress: 19/146/
09-08 14:28:59.988    6635-6887/com.project.name I/test? Progress: 5/146/
09-08 14:28:59.988    6635-6887/com.project.name I/test? Progress: 0/146/
09-08 14:28:59.988    6635-6887/com.project.name I/dlLog? Log download completed
09-08 14:28:59.998    6635-6887/com.project.name I/VibrationMeterFragment? accList1 size: 608
09-08 14:29:01.708    6635-6887/com.project.name I/test? Progress: 0/4/
09-08 14:29:01.708    6635-6887/com.project.name I/dlLog? Log download completed
09-08 14:29:01.708    6635-6887/com.project.name I/VibrationMeterFragment? accList1 size: 608
09-08 14:29:05.548    6635-6887/com.project.name I/test? Progress: 0/4/
09-08 14:29:05.548    6635-6887/com.project.name I/dlLog? Log download completed
09-08 14:29:05.548    6635-6887/com.project.name I/VibrationMeterFragment? accList1 size: 608
09-08 14:29:11.108    6635-6887/com.project.name I/test? Progress: 0/4/
09-08 14:29:11.108    6635-6887/com.project.name I/dlLog? Log download completed
09-08 14:29:11.108    6635-6887/com.project.name I/VibrationMeterFragment? accList1 size: 608
... it just keeps looping and never downloads more data

Eric

The logger cannot free memory while the connection is active.  It appears that the log has filled up so no more data is being recorded until you drop the connection.

Storing raw accelerometer data is generally not a good idea due to the sheer bulk of data.  What kind of post processing are you doing with the data?  If they are not too complex, then you will want to look into the on board data processing.