Logging and downloading data stopping abruptly
The sensors still seem to be connected but the data seems to stops abruptly. I'm not sure what I can do to resolve this behavior. is this a problem with the firmware or is it a bluetooth problem i'm unaware of? I've tried quite a few things but nothing seems to work. I might end up having to add a timeout mechanism but how would this behavior work in terms of the sensors. do I start from the beginning or where i left off?
This discussion has been closed.
Comments
Post the board model, hardware revision, and firmware revision.
What you're describing suggests that connection is being lost during the download so confirm that the device is in fact still maintaining connection. Test the boards with MetaBase to see if the same issues occur.
I was able to download small amounts of data correctly but anything longer then 500 seconds at 25hz only collecting accelerometer data gave me issue.
MetaMotion R
model number: 5
firmware: 1.4.2
Hardware 0.3
Hrm..okay, using the latest hardware and firmware.
I would like to confirm whether or not you are simply losing connection during a download and if you run into the samee issue with MetaBase.
I think it's because I was registering multiple loggers. I found running teardown and then configuring seems to have resolved the problem. I'm not sure but I also found the connection got really wonky without calling teardown. I think it was because I was creating multiple loggers. I was planning on ordering bluetooth dev board so that I could collect ble packets, but I think i've resolved my problem through trial and error.
I'm finding that downloading 300k samples takes quite some time, but I don't think that can be helped.
Reduce the max connection interal to 7.5ms before starting the download. See the 2nd paragraph in the documentation:
https://mbientlab.com/cppdocs/0/advanced_features.html#high-frequency-streaming
I've already done that. I'm just happy that I can download all the data.
We've been having a similar issue intermittently with the Python SDK, where data stops being transferred in the middle of a download from MetaMotion R sensors. We're on the latest version of the API (0.6.0), hardware 0.3 / model 5, and all sensors updated to firmware 1.4.2. As far as I can tell from our debug logs, it looks like the bluetooth connection remains open but that the data simply stops being transferred. For now, our workaround is to reset the connection and restart the download, but this is not ideal. We haven't seen this issue happening with MetaBase.
Any idea what might be causing this issue and are there any steps we can take to prevent it?
We are starting 3 loggers (accelerometer, altimeter and switch), storing their pointers on the host (a Raspberry Pi), and then when downloading the logged data, passing the pointers to
logger_subscribeto fetch the associated data. In our callbacks, we are just writing the data out to a file. After the download is complete, the sensor is reset withlogging_clear_entriesanddebug_reset_after_gcto prepare it for the next session.We have also modified the connection interval to 7.5ms as suggested in the documentation.
@donald_cascom
Post one of your debug logs that captures this event as well as the code you are running to setup the log download. If MetaBase works fine, then it could be an issue with your code or you should use an external USB bt adapter instead of the on-board chip.
Also, monitor the bt adapter during the download. Does it actually cease receiving notifications or does it in fact log a disconnect event?
Thanks for your quick response. Attached are a series of files from a session, which includes configuring two devices, starting them logging, stopping them after around 30 secs and then downloading data from them, one device at a time. Our debug logs are not as verbose as they could be, but paired with the hcidump packets you should be able to see roughly what is happening.
Have attached the following files:
- hcidump.cap - output from hcidump of the packets during the session, in Wireshark format
- debug.log.txt - our logs
- config.py.txt - snippet containing our configuration and download setup code
You can see from the packet capture that the connection remains open. It looks as if the
READOUT_PAGE_COMPLETEDmessage is not being sent by the API for some reason.Worth noting that we are not able to consistently reproduce this issue across all of the Pis we are testing on.
entry 13035 shows that
READOUT_PAGE_COMPLETEDwas received and in 13038, the progress update notification is received with 0 entries left. Your log also shows that the controller initiated a disconnect. As far as the hcidump is concerned, everything is working as expected.Regarding your script, it looks fine to me. See if you can distill your script down to just the
libmetawearfunctions that can reproduce this issue and put that into a Python script that can be run as is.Please look more closely - to me it does not look as though it is not working as expected. (Entry 13035 was after the download had been restarted several times - it did eventually complete successfully at this point).
You can see the issue happening at entry 5222, and again at 9252, 10699 and 12630 - note the large gaps between these entries and the following controller-initiated disconnect.
I will post again later with a runnable Python script that attempts to reproduce the issue.
@Eric - were you able to take a second look at the dump and verify that something is amiss here?
Yes, there is something odd going on. Entries 522, 9252, and 10699 all show that
READOUT_PAGE_COMPLETEDis being received by the host device, however, the command to continue the download is never written. Either the underlying BLE code is not sending the continue command or the Python SDK is not forwarding the command to the BLE code.Entry 12630 is the oddball case here, where, as you suspected,
READOUT_PAGE_COMPLETEDisn't being received. However, it cannot be determined if the firmware never sent it or the host device simply never received it.Thanks for taking another look. Responding to your queries:
Today I investigated the issue further and added some debugging code to the BLE notify handler in the Python SDK. This confirmed that the
READOUT_PAGE_COMPLETEDnotifications are going missing at some point before entering the Python notify callback (gatt_char.on_notification_received), rather than that the C++ API was failing to send aREADOUT_PAGE_CONFIRMresponse. This suggests to me that the issue might exist in the BLEPP state machine or in the way thatlibwarbleis using it.I have attached a complete version of the snippet I posted earlier, using only the metawear APIs (along with a small callback to write the logged data to CSVs.) I have been able to consistently reproduce the issue in this standalone script, but as before, only on one of our Pis.
What versions of the kernel and bluez are you running on your Pi? Which ones are exhibiting this behavior?
They should be effectively identical - all of our Pis are on:
We are using the Pi's built-in bluetooth module to communicate with the MetaMotion sensors.
I'll try your script out on our Pi when I get my hands on it. I assume you are using a RPi 3?
Yes - standard RPi3.
@Eric - did you get a chance to look into this further?
No, I have not been able to debug this further.
@Eric I have started to look into some possible workarounds as well as attempt to further narrow down where the packets are getting lost. Is there a way to get the sensors to resend the most recent page without breaking the connection and restarting the download? I have tried following the same procedure as in
mbl_mw_logging_download(sendingREADOUT_NOTIFY, requesting the length, then sending aREADOUTcommand using the retrieved length), but the device doesn't seem to respond.Your script hasn't caused any issues so far in my Linux VM. I haven't been able to test on our Pi yet but will do that soon.
Refer to this unit test
https://github.com/mbientlab/MetaWear-SDK-Android/blob/3.6.2/library/src/test/java/com/mbientlab/metawear/TestLoggingDownload.java#L124#L128