›

Buffer overflow

guilhermesilveiraguilhermesilveira
in Python

When invoking the device.connect method of a MetaTracker instance, sometimes I get a

*** buffer overflow detected ***: python terminated

Reading some older posts, here is some extra information:

  • its an Ubuntu machine, 6gb ram
  • ram is mostly free
  • a lot of swap available, not being used by any program

Here is btmon output for a simple run with this error. Its a straightforward call to MetaTracker(mac_address).connect:

https://gist.github.com/guilhermesilveira/07df7861741c93e2b55b91ee1bd5bdd7

It happens intermitently with many MetaTracker devices.

One interesting fact is that a guaranteed way to get the buffer overflow error is to run a program and try to connect to two devices. The first connect invocation to a MetaTracker instance works fine, the second one fails with the buffer overflow every time. Perhaps those are two different buffer overflows.

Any suggestions?

Regards

Comments

  • LauraLaura

    If you do the same thing with our standard MetaBase app, does an error occur?
    Does your custom code work fine with a single device?

  • guilhermesilveiraguilhermesilveira

    @Laura , I did the following:

    • removed batteries from all trackers (7 in total)
    • inserted battery in tracker E9, it showed up on metabase as the only one on
    • connected, retrieved firmware=1.2.5, asked for firmware update, choose LATER
    • "run diagnostic" does not execute anything, so i go back
    • it shows the e9 off for a while then back on, select it again, 15 times, worked fine
    • closed the app
    • turned off my cell phone bluetooth, just in case

    Open the computer, run the provided scan_connect.py from examples: https://github.com/mbientlab/MetaWear-SDK-Python/blob/master/examples/scan_connect.py

    • it shows E9 as the only metawear around
    • selects, works fine
    • disconnects
    • wait a few seconds, try again, repeat 3 times, all fine

    Which means device.connect works fine for the same device a few times if disconnection was succesful. That already helps isolating the problem.

    I noticed that I would get connection errors if disconnection was abrupt (i.e. due to an error if closes the program, the next connection typically fails).

    Next step, now I try the same process with two metatrackers on.

    • inserted battery in tracker FB (E9 is still on)
    • turn on the phone bluetooth, it showed up on metabase as the both on
    • connected to FB, same 1.2.5, firmware, back thing. did the same thing to E9
    • repeated 2 times for each
    • after the second time both show as OFF on Metabase, but when I tried to connect it worked, perhaps just a UI async glitch from the app itself
    • closed the app
    • turned off my cell phone bluetooth, just in case

    Now, on to the scan_connect.py script from github:

    • run the script twice, in two different terminal windows
    • both list both devices
    • choose FB on one, worked fine, quit
    • choose E9 on the second one, got a "Timed out while trying to connect to remote device". tried again, same error
    • just in case, on the second one tried FB, same error
    • tried FB on the first one again, same error
    • tried E9 on the first one, worked
    1. So far I can see I am getting random connection time outs. Are these random connection time outs expected? I saw in some post they were. If so, why do they happen and how to mitigate it besides a connection loop?

    Now that I was able to connect to both I go for both terminals and try again, first one on FB

    • first one connects successfuly, while reading the info I select E9 on the second:
    • second one dies with a *** buffer overflow detected ***

    1. The conclusion is that the script (from mbientlab itself) works most of the time with individual pieces, but seems to fail every time I try to connect to two devices (either from the same python process, from my original program, or in two different python processes running from the same directory, such as this example).

    2. I can not replicate connecting to two devices from the App since the app connects to one device at a time.

    Any suggestions on how to tackle the buffer overflow error? Is there any flag I can pass to the C library compilation or runtime that enables more debug information so we can better detect where the buffer overflow occurs and who ?

  • guilhermesilveiraguilhermesilveira

    If it helps, when trying to connect to two devices at the same time, the json file in the cache folder is created for the first one (connected successfuly) but the buffer overflow occurs before creating the cache folder json file for the second device. It definitely occurs during the device.connect() invocation in the script provided (debugged it to get there).

  • guilhermesilveiraguilhermesilveira

    More specifically, the error occurs on self.warble.connect_async(completed)

  • EricEric

    @guilhermesilveira said:
    Any suggestions on how to tackle the buffer overflow error? Is there any flag I can pass to the C library compilation or runtime that enables more debug information so we can better detect where the buffer overflow occurs and who ?

    There's no direct way to install a debug version of libwarble. You can checkout the warble C code, build the debug version, and replace the libwarble.so in the pywarble pyhton package.

    git clone https://github.com/mbientlab/Warble.git --recurse-submodules
make CONFIG=debug -C Warble

    The debug .so will be in Warble/dist/debug/lib/{arch}/libwarble.so

  • EricEric

    @guilhermesilveira said:
    Now, on to the scan_connect.py script from github:

    • run the script twice, in two different terminal windows
    • both list both devices
    • choose FB on one, worked fine, quit
    • choose E9 on the second one, got a "Timed out while trying to connect to remote device". tried again, same error
    • just in case, on the second one tried FB, same error
    • tried FB on the first one again, same error
    • tried E9 on the first one, worked
    1. So far I can see I am getting random connection time outs. Are these random connection time outs expected? I saw in some post they were. If so, why do they happen and how to mitigate it besides a connection loop?

    Could be a variety of things. Connections won't always be successful on the first attempt; your app needs to accommodate for that.

    Now that I was able to connect to both I go for both terminals and try again, first one on FB

    • first one connects successfuly, while reading the info I select E9 on the second:
    • second one dies with a *** buffer overflow detected ***

    1. The conclusion is that the script (from mbientlab itself) works most of the time with individual pieces, but seems to fail every time I try to connect to two devices (either from the same python process, from my original program, or in two different python processes running from the same directory, such as this example).

    2. I can not replicate connecting to two devices from the App since the app connects to one device at a time.

    Are you trying to connect to two devices simultaneously? Connection attempts should be done serially not in parallel.

  • guilhermesilveiraguilhermesilveira
    edited April 2020

    Thanks Eric, I will compile and run with the debug settings.

    Answering your question, the connections are done serialized. You can think of the following codes if it was just one script:

    m1 = MetaWear(mac1)
m1.connect()
m2 = MetaWear(mac2)
m2.connect()

    I suspect it might be the usage of a poor USB dongle on Ubuntu. I will receive a new device from another Branch tomorrow and try again.

    I will even try with both devices specifying the HCI Mac address for each one to see if the problem is with the devices or with some shared memory codes inside the Bluetooth library itselfI will even try with both devices specifying the HCI Mac address for each one to see if the problem is with the devices or with some shared memory buffer inside the Bluetooth library itself. Those libraries are usually stable enough, so I believe it will be just a poor quality dongle issue. I'll get back to it this week

    Thanks

  • guilhermesilveiraguilhermesilveira

    @minousoso can you tell me what's is the firmware update?

  • guilhermesilveiraguilhermesilveira

    @Eric @Laura my suspicion was correct, first there is an issue with the. bluetooth dongle. Some bluetooth dongles do not support more than one paired connection at a time. So after connecting with one device, the second one always fail. I got a few different models and most of them are quite stable, if they connect up to 5 devices, they always connect up to 5 devices.

    The problem with the buffer overflow error is that many situations result in a buffer overflow and since its a C level error (not a python error), it kills the program. I was able to isolate a few buffer overflow situations:

    First if the bluetooth dongle is removed while trying to make a connection, the buffer overflow occurs and kills the program.

    Second, when the bluetooth pairing limit is reached, it fails with the buffer overflow message. But we can't know beforehand - by code - if it reached the limit or not, since this is not something the bluetooth hardware provides us via code.

    Third, some random connection issues also seem to end up with a buffer overflow, I have not isolated it yet.

    I don't believe its an issue on mbientlab's code, it is most probably an issue with the linux bluetooth support. Unfortunately it does make it unsafe to create a product supporting linux to connect to the devices if your product allows the end user to choose their own bluetooth dongles. The program might crash with no chance of recovery.

    When required I will run some experiments with two different dongles and mac addresses and see what happens.

    regards

  • LauraLaura

    @guilhermesilveira,
    Amazing work! Please keep it up and do let us know the make and model of the dongles that performed better. We will be happy to let other users know and this is extremely useful for our community.

  • AKRAKR

    Is there a way to fix the *** buffer overflow detected ***: python terminated?
    Have issues while connecting to multiple sensors metawear sensors?

    example of the code used:
    from future import print_function
    import sys
    from mbientlab.metawear import MetaWear, libmetawear
    from mbientlab.metawear.cbindings import *
    from time import sleep
    from threading import Event

    def reset(MAC):

    device = MetaWear(MAC)
device.connect()
print("Connected")

libmetawear.mbl_mw_logging_stop(device.board)
libmetawear.mbl_mw_logging_clear_entries(device.board)
libmetawear.mbl_mw_macro_erase_all(device.board)
libmetawear.mbl_mw_debug_reset_after_gc(device.board)
print("Erase logger and clear all entries")
sleep(1.0)

libmetawear.mbl_mw_debug_disconnect(device.board)
sleep(1.0)

device.disconnect()
print("Disconnect")
sleep(1.0)

    if name == 'main':
    reset("F4:XX:XX:XX:XX:23")
    sleep(1.0)
    reset("D5:XX:XX:XX:XX:34")
    sleep(1.0)
    reset("D8:XX:XX:XX:XX:B4")
    sleep(1.0)
    reset("DF:XX:XX:XX:XX:AA")

    error 1592506866.952153: Error on line: 296 (src/blestatemachine.cc): Operation now in progress
    *** buffer overflow detected ***: python terminated

  • guilhermesilveiraguilhermesilveira

    In my case I:

    • first isolate the problem, does your entire program work with only one device?
    • if yes, try with two, if yes three
    • find out when it hangs

    If it hangs only with more than one, you might be having the same problem that I do: the chip your usb dongle uses might not support more than N connections at once. In that case you can do as I did, buy a better chip :( Mine currently supports 5 connections for my needs, but it does not handle 6.

    It is said that other apis are stabler than the python due to the underlying libraries, I did not test any other library.

    regards

  • LauraLaura

    The Javascript APIs are better for apps where you have many sensors because the bluetooth libraries are more reliable.

    As @guilhermesilveira mentioned, the Python libraries we use from third party vendors are very rudimentary and don't support multiple sensors and dongles as well.

    Please do make sure that your code is handling the multiple connections correctly. It should also handle failures in case one of your sensors isn't reset properly, your code should automatically retry.

  • AKRAKR
    edited July 2020

    @guilhermesilveira said:
    In my case I:

    • first isolate the problem, does your entire program work with only one device?
    • if yes, try with two, if yes three
    • find out when it hangs

    If it hangs only with more than one, you might be having the same problem that I do: the chip your usb dongle uses might not support more than N connections at once. In that case you can do as I did, buy a better chip :( Mine currently supports 5 connections for my needs, but it does not handle 6.

    It is said that other apis are stabler than the python due to the underlying libraries, I did not test any other library.

    regards

    Thanks @guilhermesilveira. What usb dongle are you using?

  • LauraLaura

    Thanks for the update @AKR

  • XiaoruiXiaorui

    @guilhermesilveira said:
    In my case I:

    • first isolate the problem, does your entire program work with only one device?
    • if yes, try with two, if yes three
    • find out when it hangs

    If it hangs only with more than one, you might be having the same problem that I do: the chip your usb dongle uses might not support more than N connections at once. In that case you can do as I did, buy a better chip :( Mine currently supports 5 connections for my needs, but it does not handle 6.

    It is said that other apis are stabler than the python due to the underlying libraries, I did not test any other library.

    regards

    Hello,

    I got a solution to make more than 5 sensors(10 sensors) to work at the same time.
    You can try to use multiple thread programming to reach this goal.

  • guilhermesilveiraguilhermesilveira

    @Xiaorui Nice! I actually use one thread to connect to all 5 sensors, then 1 thread per sensor to listen to their events. Do you create one thread per sensor before connecting, and inside each thread connect to them? And do you wait for the first one to be connected before launching the second thread or do you fire them all at once and request for connections simultaneously? (I don't know how lost the bluetooth device might get with 10 simultaneous requests)

  • XiaoruiXiaorui
    edited November 2020
  • XiaoruiXiaorui

    @guilhermesilveira said:
    @Xiaorui Nice! I actually use one thread to connect to all 5 sensors, then 1 thread per sensor to listen to their events. Do you create one thread per sensor before connecting, and inside each thread connect to them? And do you wait for the first one to be connected before launching the second thread or do you fire them all at once and request for connections simultaneously? (I don't know how lost the bluetooth device might get with 10 simultaneous requests)

    Hey, guilhermesilveira.

    I use 10 threads for ten sensors. It works. Then I tested more sensors(20 sensors) with 5 dongles on RaspberryPi / PC (Ubuntu1804) last night. Eventually, 16 sensors were connected successfully(another 4 were failed).

    ** Do you create one thread per sensor before connecting, and inside each thread connect to them?**
    Yes, one thread for one sensor. You can see the details in my code snippet.

    **And do you wait for the first one to be connected before launching the second thread or do you fire them all at once and request for connections simultaneously? **

    Yes, I use sleep function to make sure the sensors will connect the dongle one by one. Fire them all at once can't work.

    Here is my hardware(5 dongles)

    Here is my code snippet, feel free to try it (you need to set your own dongle addresses~~~~ and sensor addresses).

    Hey, I don't know why the code format is so strange.....if you want I can send you the .py file....

    usage: python stream_acc.py [mac1] [mac2] ... [mac(n)]

    from future import print_function
    from mbientlab.metawear import MetaWear, libmetawear, parse_value
    from mbientlab.warble import WarbleException
    from mbientlab.metawear.cbindings import *
    from time import sleep
    from threading import Event

    import platform

    import threading
    import time
    import pickle

    if sys.version_info[0] == 2:
    range = xrange

    class State:
    def init(self, device):
    self.device = device
    self.samples = 0
    self.callback = FnVoid_VoidP_DataP(self.data_handler)
    self.data = []

    def data_handler(self, ctx, data):
    if self.samples < 100:
    self.data.append([self.device.address, parse_value(data)])

    if self.samples % 100 == 99:
    print("%s -> %s" % (self.device.address, parse_value(data)))

self.samples+= 1

    exitFlag = 0

    class myThread (threading.Thread):
    def init(self, metawarelist, hci_mac):
    threading.Thread.init(self)
    self.metawarelist = metawarelist
    self.hci_mac = hci_mac
    self.states = []

    def run(self):
    for i in range(len(self.metawarelist)):
    d = MetaWear(self.metawarelist[i], hci_mac=self.hci_mac)
    connected = False
    while not connected:
    try:
    d.connect()
    connected = True
    except WarbleException:
    connected = False
    print("\33[31mTrying again to connect to \33[0m" + d.address)
    sleep(5.0)

        print("Connected to " + d.address)
    self.states.append(State(d))

for s in self.states:
    print("Configuring device")
    libmetawear.mbl_mw_settings_set_connection_parameters(s.device.board, 7.5, 7.5, 0, 6000)
    sleep(1.5)

    libmetawear.mbl_mw_acc_set_odr(s.device.board, 100.0)
    libmetawear.mbl_mw_acc_set_range(s.device.board, 16.0)
    libmetawear.mbl_mw_acc_write_acceleration_config(s.device.board)

    signal = libmetawear.mbl_mw_acc_get_acceleration_data_signal(s.device.board)
    libmetawear.mbl_mw_datasignal_subscribe(signal, None, s.callback)

    libmetawear.mbl_mw_acc_enable_acceleration_sampling(s.device.board)
    libmetawear.mbl_mw_acc_start(s.device.board)

sleep(300.0) # 300 seconds

for s in self.states:
    libmetawear.mbl_mw_acc_stop(s.device.board)
    libmetawear.mbl_mw_acc_disable_acceleration_sampling(s.device.board)

    signal = libmetawear.mbl_mw_acc_get_acceleration_data_signal(s.device.board)
    libmetawear.mbl_mw_datasignal_unsubscribe(signal)
    libmetawear.mbl_mw_debug_disconnect(s.device.board)

print("Total Samples Received")
for s in self.states:
    print("%s -> %d" % (s.device.address, s.samples))

    sensor_0 = "DB:"
    sensor_1 = "FE:"
    sensor_2 = "E6"
    sensor_3 = "D5"
    sensor_4 = "D3:"
    sensor_5 = "F5:"
    sensor_6 = "DC:"
    sensor_7 = "F4:C"
    sensor_8 = "EB:"
    sensor_9 = "E"
    sensor_10 = "DC:"
    sensor_11 = "DD:7"
    sensor_12 = "D8:"
    sensor_13 = "CF"
    sensor_14 = "EC"
    sensor_15 = "DF:"
    sensor_16 = "CAF2"
    sensor_17 = "E6:4"
    sensor_18 = "C7A:AC"
    sensor_19 = "E1:D55"

    sensors = [sensor_0, sensor_1, sensor_2, sensor_3, sensor_4, sensor_5, sensor_6, sensor_7, sensor_8, sensor_9,\
    sensor_10, sensor_11, sensor_12, sensor_13, sensor_14, sensor_15, sensor_16, sensor_17, sensor_18, sensor_19]

    dongle_0 = "008"
    dongle_1 = "00:15:2"
    dongle_2 = "00::BE"
    dongle_3 = "00:15:86"
    dongle_4 = "00:1588"

    dongles = [dongle_0, dongle_1, dongle_2, dongle_3, dongle_4]

    thread_0 = myThread([sensor_0], dongle_0)

    thread_1 = myThread([sensor_1], dongle_0)
    thread_2 = myThread([sensor_2], dongle_0)
    thread_3 = myThread([sensor_3], dongle_0)

    thread_4 = myThread([sensor_4], dongle_1)
    thread_5 = myThread([sensor_5], dongle_1)
    thread_6 = myThread([sensor_6], dongle_1)
    thread_7 = myThread([sensor_7], dongle_1)

    thread_8 = myThread([sensor_8], dongle_2)
    thread_9 = myThread([sensor_9], dongle_2)
    thread_10 = myThread([sensor_10], dongle_2)
    thread_11 = myThread([sensor_11], dongle_2)

    thread_12 = myThread([sensor_12], dongle_3)
    thread_13 = myThread([sensor_13], dongle_3)
    thread_14 = myThread([sensor_14], dongle_3)
    thread_15 = myThread([sensor_15], dongle_3)

    thread_16 = myThread([sensor_16], dongle_4)
    thread_17 = myThread([sensor_17], dongle_4)
    thread_18 = myThread([sensor_18], dongle_4)
    thread_19 = myThread([sensor_19], dongle_4)

    sleep_delay = 5.0
    thread_0.start()
    sleep(sleep_delay)

    thread_1.start()
    sleep(sleep_delay)
    thread_2.start()
    sleep(sleep_delay)

    thread_3.start()
    sleep(sleep_delay)

    sleep_delay = 5.0
    thread_4.start()
    sleep(sleep_delay)

    thread_5.start()
    sleep(sleep_delay)

    thread_6.start()
    sleep(sleep_delay)

    thread_7.start()
    sleep(sleep_delay)

    sleep_delay = 5.0
    thread_8.start()
    sleep(sleep_delay)

    thread_9.start()
    sleep(sleep_delay)

    thread_10.start()
    sleep(sleep_delay)

    thread_11.start()
    sleep(sleep_delay)

    thread_12.start()
    sleep(sleep_delay)

    thread_13.start()
    sleep(sleep_delay)

    thread_14.start()
    sleep(sleep_delay)

    thread_15.start()
    sleep(sleep_delay)

    thread_16.start()
    sleep(sleep_delay)

    thread_17.start()
    sleep(sleep_delay)

    thread_18.start()
    sleep(sleep_delay)

    thread_19.start()
    sleep(sleep_delay)

    thread_0.join()
    thread_1.join()
    thread_2.join()
    thread_3.join()
    thread_4.join()
    thread_5.join()
    thread_6.join()
    thread_7.join()
    thread_8.join()
    thread_9.join()
    thread_10.join()
    thread_11.join()
    thread_12.join()
    thread_13.join()
    thread_14.join()
    thread_15.join()
    thread_16.join()
    thread_17.join()
    thread_18.join()
    thread_19.join()

  • LauraLaura

    Can you fix the formating or send code as an attachment?

Sign In or Register to comment.