›

Unity works perfectly with MetaWear sensor on first run, hangs on 2nd run

jimjamjimjam
edited July 2018 in C#

Video recording of the issue: http://recordit.co/Pb0Y5B8xOx

Unity support has come up a few times in this forum with no clear instructions on how to put it all together. I've managed to do so using the C# SDK, the Windows10 plugin, Warble and the .NET bindings. I'm happy to share the unity project somewhere online for others to use, as well as keep working on it to add iOS and Android support, which I will need.

There is one problem with it, which I'm hoping someone here or at mbientlab can help me solve. Everything works perfectly the first time (connects, reading sensor data, etc) but if you stop the engine and run it a second time, unity hangs.

The issue happens whether I clean up the Metawearboard or not (see onApplicationQuit in code below). It happens even if I don't read anything from the sensor. I've been told Unity hangs this way sometimes when threads are not disposed of correctly when the game is stopped. I'm not sure how else to work through this problem.

Any hints would be tremendously appreciated!

Thank you!

public class ConnectToSensor : MonoBehaviour
{
    private IMetaWearBoard metawear = null;

    async void Start()
    {
        try
        {
            Debug.Log("hello world");

            metawear = MbientLab.MetaWear.NetStandard.Application.GetMetaWearBoard("C2:BD:D8:B2:EF:35");

            Debug.Log("connecting..." + metawear.IsConnected);
            await metawear.InitializeAsync();

            Debug.Log("connected!");
            Debug.Log(metawear.IsConnected);
            Debug.Log(metawear.ModelString);

        }
        catch (Exception e)
        {
       // No exceptions are ever thrown
            Debug.Log(e);
        }
    }

    // Unity hangs on 2nd run whether this block is executed or not!
    private async void OnApplicationQuit()
    {
        Debug.Log("App quitting!");
        if (metawear != null && !metawear.InMetaBootMode)
        {
            Debug.Log("Tearing down...");
            metawear.TearDown();
            Debug.Log("Disconnecting...");
            await metawear.GetModule<IDebug>().DisconnectAsync();
            Debug.Log("Disconnected!");
        }

    }
}
«1

Comments

  • EricEric

    There's nothing fancy about the C# SDK. It uses standard C# 7.0 features and classes, all of which work fine when used in a .NET Core, .NET Framework, or UWP app.

    Without knowing which line (or lines) of code is responsible for causing Unity to hang, I can't really provide any suggestions.

    Maybe more experienced Unity devs / users on the Unity forums can provide more insight.

  • jimjamjimjam

    That's the thing, it's not a line of code that causes the freeze. It's re-loading the assembly for a second run. The unity logs just stop at "Reloading Assembly". No code gets executed.

    From my research, this type of behaviour apparently can happen if the assembly leaves a running thread or socket connection open after the first run. Is it possible that Warble or the SDK is not cleaning up all resources properly?

    I don't think I've seen confirmation by anyone on this forum about getting the sensor working with Unity fully, and I'm sure it's a common use-case for these Mbientlab sensors. Also it's so close to working fully! Would be happy to work through it with you if you have any ideas on where we can investigate.

    Thanks

  • EricEric

    What I mean is, some line of code puts the Unity Editor into a state where it freezes upon the second play, unless you're saying that it always freezes regardless of whether you run any MetaWear code or not.

    Assuming it's the former, you'll need to start stripping away all the irrelevant code to pinpoint the exact cause. Since your example is pretty simple, the likely function is InitializeAsync. As previously stated, there's nothing fancy about the C# SDK. Everything there uses .NET Standard 2.0 libraries and C# 7.0 syntax.
    https://github.com/mbientlab/MetaWear-SDK-CSharp/blob/1.0.15/MetaWear/Impl/MetaWearBoard.cs#L756

    Regarding Warble, again, nothing fancy there. Everything there is standard functions / classes provided by the Windows 10 SDK.
    https://github.com/mbientlab/Warble/blob/master/src/warble/cpp/win10_api.cpp

  • jimjamjimjam

    Hi Eric

    The editor freezes before any of my code executes. Not even the first line of debug is executed. It freezes when it tries to reload the metawear DLLs. That's why I can't really debug it myself. As I mentioned, I've read that this type of freeze (on the 2nd run) can happen if the DLL does not clean up all of its resources fully, such as, for example, a running thread.

    I'm not saying that you are doing anything fancy, or wrong or outside the .NET Standard library, but there could be something in the deallocation/teardown code that could be done differently to make it work with Unity. Of course I can look through your code and try to solve it, but it will be faster with your help as I don't know the code.

    ...or I can look for another sensor I guess, but the Metawear one is really good.

    I'll see if I spot anything in your code later today.

  • EricEric

    You don't need to run a debugger to isolate the line(s) of code causing the issue. All you need to do is attempt to successfully play the scene in the editor multiple times. Start with a blank app, then progressively add MetaWear calls until it freezes. I've already suggested that the InitializeAsync function is probably the line that does it.

    I'm not asking you to solve the problem, i'm asking you to isolate the problematic lines of code.

  • jimjamjimjam

    Hey

    OK. Just to clarify. None of the MetaWear calls freeze Unity the first time around. The second time around, Unity won't even load the DLL. Now... the MetaWear call that needs to be run the first time, to cause the issue the second time is... InitializeAsync as you suspected.

    Here is a discussion about someone else who's encountered the exact same freeze behaviour whilst writing with a c++ plugin that has multithreading in it:

    https://answers.unity.com/questions/202558/can-custom-c-plugin-do-multiprocessing-threading-f.html

    It does seem to be that all of this has to do with thread management / de-allocation, although I don't know what the solution is.

  • jimjamjimjam

    Just as a word of warning, I tried threading a 'while true' loop using both a dedicated thread and a threadpool. Although the game would play once, it would crash when I tried to play again, or when i tried to exit Unity. I think this had something to do with unity failing to abort the thread when the game stops but I'm not sure.

  • EricEric

    Based on what was posted in that Unity forum thread, the Warble C++ code is most likely the cause of freeze then. InitializeAsync does make BLE calls so to confirm this, remove the MetaWear functions and call Warble directly. All you really need to test is ConnectAsync and Disconnect:

    https://github.com/mbientlab/Warble.NET/blob/master/Example.Connect/Connect.cs

  • jimjamjimjam

    Hi Eric. I will test both Warble calls directly tomorrow and get back to you. Thanks for your help working through this.

  • abcdabcd
    edited July 2018

    Hi Guys,
    I following this thread for couple of days cause we have the same problem with Unity so :)
    I just tried this and confirm that when i call directly the Warble theUnity is Freezes!
    Eric - can you please help on this as we are also struggling with Unity that blocking our progress..
    Looking forward for some solution on this

    Thanks!

  • jimjamjimjam

    I can confirm that calling Warble directly causes Unity to go into this weird state where either of the following actions will cause Unity to hang:

    • running play a second time
    • trying to quit

    Any ideas on what Warble is doing that can be done differently Eric?

    Thx

    PS: I'm also getting a WarbleException: Attribute cannot be written... since upgrading to firmware 1.4.1, but I'm pretty sure that's just an issue with the new firmware and will go away if I downgrade again, which I'm going to look into doing shortly.

  • EricEric

    Attached is a VS2017 solution that builds a native dll that creates and completes an async task using the same Windows PPL classes Warble uses.

    // C# bindings
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
internal delegate void Action_IntPtr_Int_Int(IntPtr context, int arg0, int arg1);

[DllImport("NativeFunctions", CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr async_add(int x, int y, IntPtr context, Action_IntPtr_Int_Int handler);

    // C# Usage
private Action_IntPtr_Int_Int handler;
void Start () {
    handler = new Action_IntPtr_Int_Int((ctx, result, status) => {
        if (status != 0) {
            Debug.Log("Task failed, status = " + status);
        } else {
            Debug.Log("result = " + result);
        }
    });
    async_add(31, 14, IntPtr.Zero, handler);
}

    If the same freezing behavior appears, then its highly likely a compatibility issue between Unity and Window's thread management for its async task classes. In that case, the issue is out of our hands and Unity and Microsoft need to find a way for PPL functions to properly work in Unity.

    Hrm, I'm not seeing any issues with firmware v1.4.1 on my Win10 machine. For me anyways, doing a lescan

  • jimjamjimjam

    Thanks for the test solution Eric. I can confirm that this recreates the same freezing behavior as Warble calls.

    The task result code is 45 the first time around, and then unity freezes on 2nd play. Now that we have a minimum reproducible test, I'll see if there is a potential work-around. I doubt we'll ever get MS or Unity to help us on such of an issue because it is of tiny importance to them. So it will have to be something that the community here, or yourself will have to solve most likely.

    Let me play around with it a bit.

  • jimjamjimjam

    Here is another discussion on how to get around this issue:

    https://forum.unity.com/threads/problem-with-callbacks.87513/

    Some stuff to check out:

    Example code on how to cleanup threads:

    make sure your C++ Plugin cleans up all its threads when (or before) Unity's OnApplicationQuit() Monobehavior method is called. Here's a rather lengthy code example showing how to handle this in the C++, C# binding, and C# Monobehavior respectively.

    Explanation on what Unity does when it exits

    When the application exits - UnityEditor or WindowsStandalone player, mono scans all the threads in the application and tries to close them, it also catches threads created by you, even if they were created from C++ not from C#, because mono didn't expect this, at some point it goes into infinite loop. Mono uses OpenThread (THREAD_ALL_ACCESS, TRUE, threadID) to access the designated thread. So to workaround this problem, you have to create a thread using DELETE restriction by passing special SECURITY_ATTRIBUTES, this way mono will ignore this thread.

    Example code on how to fix the freeze without cleaning up your threads

    Also for anyone that wants to know how to fix this w/o properly cleaning up your threads... This is the code I wrote for Windows that works. I use boost threads, so I got the native handle first.

  • EricEric

    I guess that is the crux of the problem. Developers aren't supposed to manage the threadpool when using an async task library and as far as I am aware, there is no way to access the underlying Windows ThreadPool object that schedules the tasks.

    I did find a similar topic on the MSDN forum but it is 7yrs old and doesn't offer a solution.
    https://social.msdn.microsoft.com/Forums/vstudio/en-US/fe0761c3-70ad-4740-95a6-52cdf58f9403/how-to-tell-concurrency-runtime-to-clean-after-itself?forum=parallelcppnative

  • jimjamjimjam
    edited July 2018

    Yes, exactly Eric.

    I tried option 3 from my previous post. I'm calling GetCurrentThread in your PPL async callback, and setting the DELETE restriction on the thread so that Unity ignores the thread when it shuts down the DLL. But that does not seem to be preventing the freeze.

    HANDLE handle = GetCurrentThread();

... mode code ...

  result = SetSecurityInfo(
        handle,
        SE_KERNEL_OBJECT,
        DACL_SECURITY_INFORMATION,
        pSidOwner,
        pSidGroup,
        pDaclNew,
        pSacl
    );

... more code ...

    I've also submitted a bug report with Unity (https://fogbugz.unity3d.com/default.asp?1065590_p5vr9ftohapf1apf), but I don't expect any movement there for weeks/months.

    How much of the code in Warble relies on PPL async tasks? I'm just curious how much work it is to swap it out with manual threading. Whether it's done by Mbientlab or the community, or myself.

  • EricEric

    All of it.

    You could remove PPL tasks and directly use the returned IAsyncOperation type but there is no guarantee that that would fix the freezing issue and you have then put yourself in callback hell for no reason.

  • abcdabcd
    edited August 2018

    Hello Eric,
    Correct me if i wrong, but i can see

    • include <pplawait.h>
    • include <ppltasks.h>

    Only in win10_api.cpp file
    Are there other libraries for PPL tasks in the warble C++ project?

  • jimjamjimjam
    edited August 2018

    Yes it does look like PPL is only used in that one file?

    Perhaps we can just build a proof of concept, (non-PPL version) of the NativeFunctions.zip test solution you shared in this thread? We might need two calls. One that starts the thread, and one that terminates it, that we can use in Unity's OnApplicationQuit().

  • EricEric

    What I meant is all the Windows BLE code uses PPL tasks, which is the relevant portion of this thread.

    Again, there is no thread control when using the Windows BLE API. All you have to work with are the provided wrapper classes that encapsulate async tasks.

  • jimjamjimjam
    edited August 2018

    Eric. Just to be clear... Are you (and by you I mean Mbientlab in general) working towards making the Mbientlab sensor work in Unity?

    I'm asking because it clearly says Windows Unity support on your product page, and you have two very engaged customers (myself and abcd) here willing to do whatever it takes to help.

    If you don't think you can get Unity to work, then make it an official stance, tells us and remove it from your marketing, so that we can move on and be productive elsewhere.

    If you do think you can get there, then lets start working on some options to explore. Can we can look at replacing PPL with another way of dealing with the IAsyncOperation? Maybe we can even do it all in C# directly using standard await, as explained here: https://stackoverflow.com/a/44102996/855551 ?

  • abcdabcd

    Eric,
    To continue JImJam question and double check -
    When you said "All of it." , and "all the Windows BLE code uses PPL tasks, "
    Do you mean that PPL exists in "Windows.Devices.Bluetooth.h" , or it is ONLY in "win10_api.cpp" ?

  • EricEric

    @jimjam said:
    If you do think you can get there, then lets start working on some options to explore. Can we can look at replacing PPL with another way of dealing with the IAsyncOperation? Maybe we can even do it all in C# directly using standard await, as explained here: https://stackoverflow.com/a/44102996/855551 ?

    That method works in Visual Studio when .NET Framework apps that need to use WinRT classes; the Win10 plugin does the same thing if you install it in a .NET Framework project:
    https://github.com/mbientlab/MetaWear-SDK-CSharp-Plugin-Win10/blob/master/nuget/build/net461/MetaWear.Win10.targets

    I have no idea if it works in Unity but if it does, then great, you can just use the .NET Framework plugin and avoid the native dll entirely. The same Win10 plugin package also provides the same code compiled as a UWP class library so you could also build a UWP app if you wanted.

    In the meantime, I can rework the code to not use tasks.

  • jimjamjimjam

    Good to hear you're going to try and help it getting it working in Unity Eric.

    Let's try going down both paths simultaneously.

    One of us (either abcd or myself) can try skipping the native dll entirely, whilst you rework the code to not use tasks. Will report findings as soon as possible, although I will be away for a couple of days.

  • EricEric
    edited August 2018

    Devolved code pushed to the noppl branch:
    https://github.com/mbientlab/Warble/commit/b0289b8835afc04fa8d93e28e607c79cd3db2e35

  • jimjamjimjam

    @abcd
    Any chance you can test the noppl branch? I’m away for a couple more days.

    Thank you Eric.

  • abcdabcd
    edited August 2018


    Hi
    1. I tested the noppl branch, and Unity still freezes, and now also crashing after second time. @Eric - Is there something else to do ? Is the PPL is also inside the "Windows.Devices.Bluetooth.h" ?
    2. I started to check the integration of MetaWear.Win10.DotNet DLL with Unity , but it seems that Unity doesn't recognize it . I share the code. As i am not Unity expert, @jimjam, it will be great if you can help on this when you back

  • EricEric

    Played around with the noppl version a bit and it turns out the service discovery part of the connect code causes the same hangup. To fix this, I changed the connect code to only retrieve the gatt services rather than discover everything at once; gatt characteristics are instead only retrieve upon request.

    This did fix the hangup, however, it comes back again when you enable characteristic notifications. Documentation for event handling suggests that handlers are executed in their own thread so we're back to square one with the Windows SDK doing its own thread management.

    You can check this out with commits 18894d5b (Warble) and 11c19bc8 (Warble.NET)

    gatt = new Gatt(args[1]);
await gatt.ConnectAsync();

var notify = await gatt.FindCharacteristicAsync("326A9000-85CB-9195-D9DD-464CFBBAE75A", "326A9006-85CB-9195-D9DD-464CFBBAE75A");
await notify.EnableNotificationsAsync();
notify.OnNotificationReceived = response => {
    //Debug.Log(string.Format("Response : [0x{0}]", BitConverter.ToString(response).ToLower().Replace("-", ", 0x")));
};

var cmd = await gatt.FindCharacteristicAsync("326A9000-85CB-9195-D9DD-464CFBBAE75A", "326A9001-85CB-9195-D9DD-464CFBBAE75A");            
await cmd.WriteAsync(new byte[] { 0x1, 0x1, 0x1 });

// press the button to trigger the usual lockup

    I also took some time to look into directly including System.Runtime.WindowsRuntime.dll directly into Unity, however, Unity repeatedly stated that that assembly would cause errors and unloaded it. This appears to be intentional according to this bug report:
    https://issuetracker.unity3d.com/issues/dot-net-4-dot-6-system-dot-runtime-dot-windowsruntime-dot-dll-and-system-dot-runtime-dot-interopservices-dot-windowsruntime-dot-dll-fail-to-load

    On the UWP side, there is an issue filed with loading the DataContractSerializer class, which is problematic as the C# API uses that class. A flag can be added to skip serialization so this isn't that big of an issue but still annoying nonetheless.
    https://issuetracker.unity3d.com/issues/system-dot-configuration-dot-configurationerrorsexception-failed-to-load-configuration-section-for-datacontractserializer

    This is pretty much going nowhere fast, so, as a last resort, the best option seems to be spawning a process that handles all of the MetaWear function calls and streams the sensor data back to the main app.

  • abcdabcd
    edited August 2018

    @Eric said:

    This is pretty much going nowhere fast, so, as a last resort, the best option seems to be spawning a process that handles all of the MetaWear function calls and streams the sensor data back to the main app.

    So just to be clear on this - as SDK is not supporting Unity on Windows, you suggest that the only option is doing some "Bypass" such as streaming data into Unity from zeromq or rabbitmq ?

  • EricEric

    You're confusing the MetaWear C# SDK with the BLE plugins. The C# SDK works in Unity 2018, barring that serialization bug on UWP which they need to fix (this can be worked around with some minor changes until then).

    The BLE code, external to the SDK, doesn't work in Unity, for all the reasons discussed in this thread. So, until they fix that threading issue, BLE code is best off handled separate from Unity, or you deal with the hassles of building a UWP app with the Unity editor.

Sign In or Register to comment.