No connection timeout using javascript sdk

aribes

I have switched to the node.js bindings to connect and stream data from MetaMotionR (before I was using python but had a lot of problems). Now performance is better (lower cpu usage, more stable data rates and usually faster connection).

The connection is basically like this:

function connect() {

    console.log(format('Discovering {}', sensor_state.address));

    MetaWear.discoverByAddress(sensor_state.address.toLowerCase(), function(device) {

      console.log(format('Connecting to {}', sensor_state.address));

      // you can be notified of disconnects

      device.on('disconnect', function () {

        console.log(format('Disconnected from {}', sensor_state.address));

        process.exit(0);

      });

      // you'll need to call connect and set up

      device.connectAndSetUp(function (error) {

        console.log(format('Connected to {} ({})', sensor_state.address, error));

        

        // do stuff

      }

   }

}

Sometimes, the process is stuck at the "discovering" part, some others at "connecting" part, and I have to kill the process and restart. Isn't there a timeout?

Eric

Noble doesn't have any timeouts.  You can add them to the discover and connect functions with the NodeJS APIs.  

aribes

Do you mean add it manually using Noble api? MetaWear is using it on background, so I don't know where to start from. Any idea about how to implement that? 

Eric

No, i mean add it to your app.   If discoverByAddress doesn't complete within X seconds, throw an error, same with connectAndSetUp.

aribes

I added a process.exit(-1) when a disconnect is received or a maximum timeout is exceeded (using setTimeout and checking a global variable that indicates if it is connected).

However, sometimes I encounter that the device takes too long to connect or even does not connect after lots of retries (I have another process which monitors if the node.js process exited and simply relaunches it again).
Also, sometimes the sensor becomes undiscoverable (don't know if frozen or what). So I have to hard reset it, which is difficult because I have to take it out from a custom case we have.

How come that from MetaWear app (android) it is so fast and easy and using linux is so painful?

Eric

Android uses a different BT stack than desktop Linux (Bluedroid vs. BlueZ respectively).  It used to use BlueZ in older builds but switched starting from Android 4.2 and on.

aribes

What about the devices becoming frozen?

Eric

Just because the device isn't discoverable doesn't mean it's frozen.  It could be that Linux did not properly close the connection upon killing the app.  See if resetting the BT adapter gets the device to advertise again.

aribes

I would like to know if, inside the nodejs app I can reconnect in some way. That is, when I call discoverByAddress, after a timeout I was doing a process.exit(-1), but now I would like that the same process cancels in some way the discoverByAddress call and calls it again (same as if I was pressing ctrl-c and reruning).

Is it possible?

Eric

Use discoverAll and stopDiscoverAll
https://github.com/noble/noble-device#discovery-api

aribes

In fact, the device is discovered, but it is the connectAndSetUp function that "hangs" without any disconnect, error, timeout.

aribes

@Eric said:
Use discoverAll and stopDiscoverAll
https://github.com/noble/noble-device#discovery-api

The problem is not only with discover, but with connect. Sometimes the device disconnects after trying to connect using connectAndSetup, but when I try to run the connectAndSetup again, it does not connect (ends up in an endless try-to-connect/timeout/retry loop).

Basically, I need a nodejs process that is able to connect to multiple devices and can reconnect to them if they are: not discovered after X seconds/not connecting after X seconds/suddenly disconnecting.
Before, I solved it by having a process monitor and killing the nodejs process, but with Noble, I cannot connect to multiple devices using multiple processes that share the same hci interface.

Eric

The disconnect issue was patched in a C++ SDK update; the JavaScript SDK must still be referencing an earlier release.

You can work around this for now by initializing a new MblMwMetaWearBoard object

aribes

If I try to update the javascript sdk using npm, it still uses (I think) the 0.11.4 version.
If I try to install Javascript SDK from git+https, I get an error when it tries to download the submodule.
I downloaded from git the latest JavaScript SDK, compiled it (npm install, locally in that folder) and then copied the MetaWear-Cpp folder and the lib folder to my project node_modules/metawear folder.
Still, I see that the SDK is still compiling libmetawear.0.12.8 instead of 0.12.10.

After the dodgy update I managed to do, still getting the disconnected with error code = 19.

I don't understand what you mean by initializing a MblMwMetaWearBoard object.

aribes

I will ask the question in a simpler way:

"How can I implement reconnection using the JavaScript SDK?"

I modified the led.js example code to make a simple reconnect:

var MetaWear = require('metawear');

MetaWear.discover(function (device) {
    console.log("Discovered", device.address);
    connect(device);
});

function connect (device) {
    console.log("connecting", device.address);

    device.on('disconnect', function (error) {
        console.log("Disconnected", error);
        connect(device);
    });

    device.connectAndSetUp(function (error) {
        console.log("Connected", device.address);
        var pattern = new MetaWear.LedPattern({ repeat_count: 5 });
        MetaWear.mbl_mw_led_load_preset_pattern(pattern.ref(), MetaWear.LedPreset.BLINK);
        MetaWear.mbl_mw_led_write_pattern(device.board, pattern.ref(), MetaWear.LedColor.GREEN);
        MetaWear.mbl_mw_led_play(device.board);

        setTimeout(function () {
            MetaWear.mbl_mw_debug_disconnect(device.board);
        }, 5000);
    });
}

And while I get 2 or 3 connect-disconnect-reconnect loops, it gets stuck after a few successful tries. See the console output:

Discovered e2:3e:dd:94:99:5f
connecting e2:3e:dd:94:99:5f
Connected e2:3e:dd:94:99:5f
Disconnected 19
connecting e2:3e:dd:94:99:5f
Connected e2:3e:dd:94:99:5f
Disconnected 19
connecting e2:3e:dd:94:99:5f
Disconnected 19
connecting e2:3e:dd:94:99:5f

Eric

Your code repeatedly adds disconnect handlers, to the point where node will throw warnings about EventEmitter memory leaks. Modify your code such that the disconnect handler is only added once.

(node:3741) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 disconnect listeners added. Use emitter.setMaxListeners() to increase limit

A non-zero disconnect status is expected when using mbl_mw_debug_disconnect.

aribes

Now it seems to work better. However, sometimes I encounter this error:

Connecting to E2:3E:DD:94:99:5F 
[E2:3E:DD:94:99:5F] Disconnected (error code = 19)
Device already discovered, reconnecting
Timeout discovering
Stopping discover
Device already discovered, reconnecting
[E2:3E:DD:94:99:5F] Connection error: Error: Peripheral already connected
Disconnecting
[E2:3E:DD:94:99:5F] Disconnected (error code = 19)
Segmentation fault (core dumped)

The "Disconnected (error code = xx)" and "Connection error: XXX" lines receive the error code from MetaWear callbacks, executed by the disconnect handler and the connectAndSetUp handler.

aribes

@Eric said:
The disconnect issue was patched in a C++ SDK update; the JavaScript SDK must still be referencing an earlier release.

Maybe you will reference this message regarding my previous post. I manually updated the node_modules/metawear folder with the C++ SDK. I can see that inside it has the libmetawear.so.0.12.8 and that it is referenced in the nodejs code, so it should be updated even though "npm install metawear" command did not install the newest version.

You can work around this for now by initializing a new MblMwMetaWearBoard object

How can it be initialized? I see in the source code in metawear.js that it is done in the MetaWear library constructor.

aribes

@aribes said:
Now it seems to work better. However, sometimes I encounter this error:

Connecting to E2:3E:DD:94:99:5F 
[E2:3E:DD:94:99:5F] Disconnected (error code = 19)
Device already discovered, reconnecting
Timeout discovering
Stopping discover
Device already discovered, reconnecting
[E2:3E:DD:94:99:5F] Connection error: Error: Peripheral already connected
Disconnecting
[E2:3E:DD:94:99:5F] Disconnected (error code = 19)
Segmentation fault (core dumped)

The "Disconnected (error code = xx)" and "Connection error: XXX" lines receive the error code from MetaWear callbacks, executed by the disconnect handler and the connectAndSetUp handler.

Sorry, this was the output of my other code (not the test code I posted). It happens because I have a setTimeout that tries to execute again the connectAndSetup function, so it throws the "Peripheral already conected" error.

In the test code that I put before, the output is:

MetaWear initializing
Discovered e2:3e:dd:94:99:5f
connecting e2:3e:dd:94:99:5f
Disconnected 19
connecting e2:3e:dd:94:99:5f

Which means that the last "connecting..." doesn't finish: either connecting or giving an error. Resetting the board or rebooting is not an option, as in the production system (a raspberry pi) there can be potentially lots of sensors connected (and working).

Having a process manager that monitors (and kills) this process is also not a good option, given that there should be one process per HCI interface, so it can also be the case that there are 3-4 sensors connected and one is frozen connecting, so killing the process would disconnect those 3-4 sensors.

BTW, is this way of handling Ctrl-C correct? Particularly asking about usage of mbl_mw_debug_disconnect

process.on('SIGINT', function() {
    if (current_device != null) {
        console.log('Sending reset', current_device.address);
        MetaWear.mbl_mw_debug_disconnect(current_device.board);
    }

    process.exit(0);
});

Eric

@aribes said:

@Eric said:
The disconnect issue was patched in a C++ SDK update; the JavaScript SDK must still be referencing an earlier release.

Maybe you will reference this message regarding my previous post. I manually updated the node_modules/metawear folder with the C++ SDK. I can see that inside it has the libmetawear.so.0.12.8 and that it is referenced in the nodejs code, so it should be updated even though "npm install metawear" command did not install the newest version.

You can work around this for now by initializing a new MblMwMetaWearBoard object

How can it be initialized? I see in the source code in metawear.js that it is done in the MetaWear library constructor.

If you've manually updated the underlying C++ SDK to the latest release, then this is a moot point.

Eric

@aribes said:

The "Disconnected (error code = xx)" and "Connection error: XXX" lines receive the error code from MetaWear callbacks, executed by the disconnect handler and the connectAndSetUp handler.

Sorry, this was the output of my other code (not the test code I posted). It happens because I have a setTimeout that tries to execute again the connectAndSetup function, so it throws the "Peripheral already conected" error.

The error message states that a connection is already established to the peripheral so disconnect from the remote device before retrying connectAndSetUp.

BTW, is this way of handling Ctrl-C correct? Particularly asking about usage of mbl_mw_debug_disconnect

process.on('SIGINT', function() {
    if (current_device != null) {
        console.log('Sending reset', current_device.address);
        MetaWear.mbl_mw_debug_disconnect(current_device.board);
    }

    process.exit(0);
});

There's no way to guarantee the disconnect command was sent before the app closes. The BLE link should be severed anyways when the app terminates so there isn't really a need for a SIGINT handler.

aribes

Looks like it is very important to wait for the disconnect before finishing the process. So I have added code to capture the stdin and using it to issue a mbl_mw_debug_disconnect command.
Then in the disconnect handler, I trigger the process.exit(0). In this way, the connection is cleanly finished.

  var MetaWear = require('metawear');

  var current_device = null;
  var quitting = false;

  // Process stdin to capture exit event instead of SIGINT
  var stdin = process.stdin;
  stdin.setRawMode(true);
  stdin.resume();
  stdin.setEncoding('utf8');
  stdin.on('data', function(key) {
    // ctrl-c ( end of text )
    if ( key === '\u0003' ) {
      process.exit(0);
    }

    if ( key === 'q' ) {
        console.log('Sending reset', current_device.address);
        quitting = true;
        MetaWear.mbl_mw_debug_disconnect(current_device.board);
    }

    process.stdout.write( key );
  });


  var address = "E1:D7:26:30:36:1A";

  // Discover a custom address and setup disconnect handler to reconnect
  MetaWear.discoverByAddress(address.toLowerCase(), function (device) {
      console.log("Discovered", device.address);

      current_device = device;

      current_device.on('disconnect', function (error) {
          console.log("Disconnected", error, device._peripheral.state);

          if (quitting) process.exit(0);

          if (device._peripheral.state != 'disconnected'){
            device.disconnect(function() {
                console.log("peripheral disconneted");
                connect(device);
            });
          } else {
            connect(device);
          }
      });

      connect(current_device);
  });

  // Connect to an already discovered device and blink led
  function connect (device) {
      console.log("connecting", device.address);

      device.connectAndSetUp(function (error) {
          console.log("Connected", device.address);
          var pattern = new MetaWear.LedPattern({ repeat_count: 5 });
          MetaWear.mbl_mw_led_load_preset_pattern(pattern.ref(), MetaWear.LedPreset.BLINK);
          MetaWear.mbl_mw_led_write_pattern(device.board, pattern.ref(), MetaWear.LedColor.GREEN);
          MetaWear.mbl_mw_led_play(device.board);

          setTimeout(function () {
              MetaWear.mbl_mw_debug_disconnect(device.board);
          }, 5000);
      });
  }


      connect(current_device);
  });

  function connect (device) {
      console.log("connecting", device.address);

      device.connectAndSetUp(function (error) {
          console.log("Connected", device.address);
          var pattern = new MetaWear.LedPattern({ repeat_count: 5 });
          MetaWear.mbl_mw_led_load_preset_pattern(pattern.ref(), MetaWear.LedPreset.BLINK);
          MetaWear.mbl_mw_led_write_pattern(device.board, pattern.ref(), MetaWear.LedColor.GREEN);
          MetaWear.mbl_mw_led_play(device.board);

          setTimeout(function () {
              MetaWear.mbl_mw_debug_disconnect(device.board);
          }, 5000);
      });
  }

However, the disconnect part is still not working well. If the sensor is in a "good state", it will connect/reconnect each 5 seconds, as shown in the code. This is the normal output:

      MetaWear initializing
      Discovered e2:3e:dd:94:99:5f
      connecting e2:3e:dd:94:99:5f
      Connected e2:3e:dd:94:99:5f
      Disconnected 19 disconnected
      connecting e2:3e:dd:94:99:5f
      Connected e2:3e:dd:94:99:5f
      Disconnected 19 disconnected
      connecting e2:3e:dd:94:99:5f
      Connected e2:3e:dd:94:99:5f
      Disconnected 19 disconnected
      connecting e2:3e:dd:94:99:5f
      Connected e2:3e:dd:94:99:5f
      Sending reset e2:3e:dd:94:99:5f
      qDisconnected 19 disconnected

That 'q' is done by me in order to trigger the "clean disconnect" parsing stdin as I said above. But when the sensor is not in a "good state", as for example if the nodejs code unexpectedly quits (e.g. due to some bug), the output is:

  MetaWear initializing
  Discovered e1:d7:26:30:36:1a
  connecting e1:d7:26:30:36:1a
  Disconnected 19 disconnected
  connecting e1:d7:26:30:36:1a

Notice that the "disconnect" is emitted during the "connecting" process (connectAndSetUp call), and then the 'disconnect' handler tries to connect again, but in this case, the subsequent connectAndSetUp calls never go through, stay there in an endless loop.
I have tried to reset the bluetooth service/devices using the following script, even rebooting the machine, with no luck (sometimes it works for killing some connections and making some device discoverable again):

  sudo service bluetooth restart
  sudo rfkill unblock bluetooth
  sudo hciconfig hci0 reset
  sudo hciconfig hci1 reset
  hcitool -i hci0 lescan

Using Android MetaWear app, I can connect to the sensor, but disconnecting from Android and trying again in linux also does not change anything.
Here's the weird thing: if in Android MetaWear app I go to Sensor Fusion (in the other code that I use the sensors I am using SF) and start streaming, after 3-5 seconds the MetaWear app blocks and I have to restart it. The second time that I connect (Android) to the sensor and go to Sensor Fusion, streaming works well. Then I disconnect and now I am able to use the sensor again in linux.

aribes

@Eric any update in this topic?

Sometimes, in normal app functioning, I am getting a sudden disconnect with error code 8, then it tries to reconnect, gets an error code 19 and from then it cannot reconnect anymore.
In this case, if I close the nodejs (multi-)sensor manager, and open again, it can connect to that sensor.

Is there any way of knowing what the error codes are?

Eric

• Can you provide reproduction steps that reliably put the board in this limbo state?
• While in this state, is connectAndSetUp able to connect to the board or is it unable to even do that?
• You'll have to ask the Noble developers what those error codes mean.

aribes

The following code adds the sensor fusion configuration to my previously sent code. If you ctrl-c right after the "Configuring sensor fusion" message, it should enter in this "limbo state", that is, if you then re-run the code, it should not connect, giving you the error 19.

  var MetaWear = require('metawear');
  var ref = require('ref');

  var current_device = null;
  var quat_signal = null;
  var quitting = false;

  // Process stdin to capture exit event instead of SIGINT
  var stdin = process.stdin;
  stdin.setRawMode(true);
  stdin.resume();
  stdin.setEncoding('utf8');
  stdin.on('data', function(key) {
    // ctrl-c ( end of text )
    if ( key === '\u0003' ) {
      process.exit(0);
    }

    if ( key === 'q' ) {
        console.log('Sending reset', current_device.address);
        quitting = true;
        on_exit();
    }

    process.stdout.write( key );
  });


  var address = "E1:D7:26:30:36:1A";
  // var address = "E2:3E:DD:94:99:5F";

  // Discover a custom address and setup disconnect handler to reconnect
  MetaWear.discoverByAddress(address.toLowerCase(), function (device) {
      console.log("Discovered", device.address);

      current_device = device;

      current_device.on('disconnect', function (error) {
          console.log("Disconnected", error, device._peripheral.state);

          if (quitting) process.exit(0);

          if (device._peripheral.state != 'disconnected'){
            device.disconnect(function() {
                console.log("peripheral disconneted");
                connect(device);
            });
          } else {
            connect(device);
          }
      });

      connect(current_device);
  });

  // Connect to an already discovered device and blink led
  function connect (device) {
      console.log("connecting", device.address);

      device.connectAndSetUp(function (error) {
          console.log("Connected", device.address);
          var pattern = new MetaWear.LedPattern({ repeat_count: 5 });
          MetaWear.mbl_mw_led_load_preset_pattern(pattern.ref(), MetaWear.LedPreset.BLINK);
          MetaWear.mbl_mw_led_write_pattern(device.board, pattern.ref(), MetaWear.LedColor.GREEN);
          MetaWear.mbl_mw_led_play(device.board);

          console.log('Configuring sensor fusion');
          MetaWear.mbl_mw_sensor_fusion_set_mode(device.board, MetaWear.SensorFusionMode.IMU_PLUS);
          MetaWear.mbl_mw_sensor_fusion_set_acc_range(device.board, MetaWear.SensorFusionAccRange._4G);
          MetaWear.mbl_mw_sensor_fusion_set_gyro_range(device.board, MetaWear.SensorFusionGyroRange._500DPS);
          MetaWear.mbl_mw_sensor_fusion_write_config(device.board);

          quat_signal = MetaWear.mbl_mw_sensor_fusion_get_data_signal(device.board, MetaWear.SensorFusionData.QUATERION);
          MetaWear.mbl_mw_datasignal_subscribe(quat_signal, ref.NULL, MetaWear.FnVoid_VoidP_DataP.toPointer(function (context, dataPtr) {

          }));

          MetaWear.mbl_mw_sensor_fusion_enable_data(device.board, MetaWear.SensorFusionData.QUATERION);
          MetaWear.mbl_mw_sensor_fusion_start(device.board);

          setTimeout(on_exit, 5000);
      });
  }

  function on_exit() {
      console.log("Disconnecting");

      MetaWear.mbl_mw_sensor_fusion_stop(current_device.board);
      MetaWear.mbl_mw_sensor_fusion_clear_enabled_mask(current_device.board);
      MetaWear.mbl_mw_datasignal_unsubscribe(quat_signal);

      MetaWear.mbl_mw_debug_disconnect(current_device.board);
  }

The connectAndSetUp does not connect after the board is in this limbo state.

aribes

I found a list of HCI error codes in https://mynewt.apache.org/network/ble/ble_hs/ble_hs_return_codes/#return-codes-hci

According to this, errors that I am getting so far are:
19 - Remote User Terminated Connection --> after a normal disconnect (triggered by me) and also when the device is in "limbo state" and does not connect
8 - Connection Timeout --> this happens as i said two posts ago, a "sudden disconnect that happens sometimes"

Eric

Ok, I'll see what I can do with that script.

On a side note, if the NodeJS app needs to crash / unexpectedly exit in order for this board to be put in the aforementioned limbo state, perhaps your main focus should be on addressing these crashes.

aribes

@Eric said:
Ok, I'll see what I can do with that script.

Thank you

On a side note, if the NodeJS app needs to crash / unexpectedly exit in order for this board to be put in the aforementioned limbo state, perhaps your main focus should be on addressing these crashes.

Yes, and right now it does not (usually) crash, as now my nodejs program only deals with connection and sending data out. However, bad things happen, like the "sudden disconnect that happens sometimes" or that the user shuts down the raspberry, or it freezes, etc... and the system should be able to recover from that.

Eric

After trying out your reproduction steps, it appears that the issue is that the board is still streaming data at a high frequency (100Hz) when the script terminates. This results in the BLE link being immediately flooded with sensor data as soon as connection is restored and the setup part does not complete. You can replicate this behavior by removing the sensor fusion stop code then terminate the script with your q input.

function on_exit() {
    console.log("Disconnecting");
/*
    MetaWear.mbl_mw_sensor_fusion_stop(current_device.board);
    MetaWear.mbl_mw_sensor_fusion_clear_enabled_mask(current_device.board);
    MetaWear.mbl_mw_datasignal_unsubscribe(quat_signal);
*/
    MetaWear.mbl_mw_debug_disconnect(current_device.board);
}

You have two options here:

1. Serialize and cache the board state after the first successful call to connectAndSetup, then pass the cached state into subsequent calls to connectAndSetup
2. Program the board to pause the stream upon disconnect

aribes

I think the best is to serialize the board right after a successful connection, but before setting any sensorfusion or whatever handler. Am I right?

How can i use the mbl_mw_metawearboard_serialize call?

I tried with the following code, but I get an error:

var b = Buffer.alloc(4);
MetaWear.mbl_mw_metawearboard_serialize(device, b.ref());

TypeError: error setting argument 0 - writePointer: Buffer instance expected as third argument

I get the same error by passing "b" instead of "b.ref()".

Eric

Call it wherever it is convenient in your code after connection.

Check the bindings file to see what types are expected for the ffi interface. That function requires a reference to a MetaWearBoard object and a reference to a ref.uint32 object.