Using TorchSharp in Unity

[ahwiggers]

Hi,

I was wondering if anyone has had success using TorchSharp in Unity. I've seen some discussion elsewhere of people having mixed results trying to use TorchSharp in Unity, but none were very recent. Have others had success? How did you go about installing TorchSharp in Unity?

I'm looking to run a neural network that I've pretrained using Pytorch. I'm not using Unity's built in Barracuda because my model uses a 3D Transpose Convolution, which has not been implemented in Barracuda.

[dayo05]

@ahwiggers
Unity supportes .NET standard 2.0 so TorchSharp is capable with Unity now. But major difference of using with nuget package is

1. I suggest Nuget for Unity to download managed plugins in Nuget. DO NOT DOWNLOAD TorchSharp-cpu or TorchSharp-cuda. Just download TorchSharp(without native binaries).
2. You should manually load Native Library due Nuget for Unity doesn't supports native library (I checked the information on Q2 so It might be changed. If Nuget for Unity supports loading native libraries, just ignore step 1 and just use TorchSharp-cpu or etc.).
   Use NativeLibrary::Load or Native Plugins with custom build of LibTorch and LibTorchSharp. You can see Method of compile LibTorchSharp on ARM based Mac OS #837 (just ignore step 2 if you targets non-ARM devices) for detailed information. You should load libtorch.(dll|dylib|so) and libTorchSharp.(dll|dylib|so) all before any call of TorchSharp and if you use NativeLibrary.Load, REMEMBER TO CALL THEM ONCE BY LIBRARY.

[jumpingbit]

I use your method and it works .

1. Get TorchSharp(without native binaries) from Nuget for Unity.
2. Copy all native binaries(get TorchSharp-cuda-windows from Nuget) to Assets\Plugins\Windows\x86_64
   Great thanks. @dayo05

[dayo05]

Extra information:
You should remember the library is NOT for moblie device.

[jacob-jacob-jacob]

Hello, could you elaborate on that? Will TorchSharp not work on any mobile devices?

[dayo05]

I just write that comment pretty long time ago so I forgot the main purpose of it. But, TorchSharp is highly uses LibTorch which the primary-native library of PyTorch. The library was not consider to used by mobile devices and as far as i know, there are no precompiled binary. If you manually create them, it might work fine. You can check Unity Docs.

I dont recommend you to use not only TorchSharp but also PyTorch based libraries on mobile device. Because even if you manually compile this library, feature are limited to only use CPU, not GPU or NPU. This can cause critical affect to mobile devices. Also size of base native library are huge, if you can then consider not to use it on mobile device.

Alternatively, you can pass data onto server and process data on server, then make response to device. This action can simply achieved by HTTP request.

[dayo05]

In case of iOS device, Im not sure but you have to change target library of P/Invoke or it can be better to directly use C++ backend of pytorch.

[TotallyReal]

I would also appreciate some help with installing and working with TorchSharp.
I tried installing only TorchSharp though NuGet (without the cuda version), and while I was in the visual studio window, it worked fine (it recognized the library, its classes, functions, etc). But the moment I go back to the Unity window it throws an error that:
The type or namespace name "TorchSharp" could not be found
The visual studio is still fine with it, but if I double click the error in Unity, it jumps back to visual studio and then suddenly it doesn't recognize it as well.

[dayo05]

Can you check your method works with other libraries? Due Unity uses its own assembly loading method, there might some difference on how Unity does and VS does. You must be sure that Unity is highly different with .NET core. (They are moving into modern .NET but it looks be a long time required...)
Before double-click was performed, Visual Studio may recognize the assembly with their own method. Once you do that, because Unity and Visual Studio sync each other, this can lead the managed library not found.
You can check this Unity document: Managed Plug-ins
The issue might not related to native library. The errors are entirely caused by managed part so there are no difference with using CUDA enabled or not.

[GeorgeS2019]

@NiklasGustafsson

There are tensorflow.NET and ML.NET used in Unity repos.

I doubt there is any TouchSharp used in public Unity repos.

TorchSharp is now routinely used in Godot, as Godot provides the closest latest .NET (e.g. pretty much native .NET8 with VS2022 Test adaptors beyond e.g. NUnit and XUnit for test-deiven (TDD) game frame simulation developemt ) environment.

Microsoft officially supports Unity through VS2022.

Perhaps Microsoft needs to think the same for providing TorchSharp integration to the Unity Game Engine development using VS2022.

[dayo05]

I saw the blog about using TF.NET on Unity. (Written with Chinese and I saw with translator)
The blog also does manual complication which is similar with mentioned method at top this discussion.

I think the Nuget is not capable with Unity is the real problem. And they are now working on but it might takes long time: Unity Blog

[GeorgeS2019]

China reported way to modernize Unity to .net8

[dayo05]

Is there any link?

[GeorgeS2019]

Saw it on twitter

[GeorgeS2019]

I focus on Godot and TorchSharp

[GeorgeS2019]

I do test driven development using VS2022..driving godot scenes

[dayo05]

Visual Studio supports IPC and maybe godot uses that.
Also, godot itself mentions only .NET development feature are needed for VS.
link

[dayo05]

One of the major difference with Unity and Godot is:

Godot uses .NET Core runtime as well but Unity uses their own-fully modified runtime.

[dayo05]

To support godot is as well as using with other .NET projects(ASP.NET or Console Application or etc.) but Unity is not.

[dayo05]

This can not done with modification of Visual Studio. The problem are shared with other managed or native libraries.

[dayo05]

If I have a time, Ill create some template of using TorchSharp with Unity.

[GeorgeS2019]

@dayo05
@NiklasGustafsson

Test driven Game (GODOT) development using VS2022

https://youtu.be/JLI0cIYMgr0

Next time I try sharing example using TorchSharp
godotengine/godot-proposals#432 (comment)

[GeorgeS2019]

Update May2024

What works
#874 (reply in thread)

[imadyTech]

This doesn't work, but I hope it can give you least clue that eventually solve the problem.

I used the method of #874 and it worked.
Keep in mind there're still some lib missing after using NugetForUnity to install TorchSharp and SkiaSharp, and you need remove some conflicting sub-libraries (added by NugetForUnity as dependencies) and add some manually which are not available from NugetForUnity. Here's the config file (~\Assets\Packages.config), pay attention to several references which were removed:

Then you need retrieve the following libraies (build a visual studio project, and add the reference through Nuget tool):
Microsoft.Bcl.AsyncInterfaces.dll
Microsoft.Bcl.HashCode.dll
Microsoft.DotNet.Interactive.dll
Microsoft.DotNet.Interactive.Documents.dll
Microsoft.DotNet.Interactive.Formatting.dll
Microsoft.Extensions.Logging.Abstractions.dll
SQLitePCLRaw.core.dll
System.Buffers.dll
System.CommandLine.dll
System.CommandLine.NamingConventionBinder.dll
System.Diagnostics.DiagnosticSource.dll
System.Memory.dll
System.Net.Http.WinHttpHandler.dll
System.Numerics.Vectors.dll
System.Runtime.CompilerServices.Unsafe.dll
System.Text.Json.dll
System.Threading.Tasks.Extensions.dll

Finally, you can add the TorchSharp examples libararies (from the TorchSharpExamples project, remember shift the framework to netstandard2.0):
Examples.Utils.dll
Models.dll

Anyway, you can retrieve the missing libraries and resolve conflicts, eventually you will be able to run when Unity prompts no exception.

[imadyTech]

Today I am trying to run a Unity scene with CIFAR10 only, but not successful.
It seems libtorch-cuda package is needed. I am trying to download the libtorch-cuda-11.3-win-x64 nuget package but totally there are 12 parts to install. Will update the result.

[imadyTech]

I found Unity won't load the native libraries automatically (someone suggested to copy the plugins into Assets/Plugins/Windows/x86_64 but it didn't work in my case. I am using Unity 2022, things might changed in new version).

with the following funciton I manually loaded the native plugins:
`

    [DllImport("kernel32.dll", SetLastError = true)]
    private static extern IntPtr LoadLibrary(string dllToLoad);
    public static IntPtr LoadNativeLibrary(string libraryPath)
    {
        IntPtr handle = LoadLibrary(libraryPath);

        if (handle == IntPtr.Zero)
        {
            UnityEngine.Debug.Log("Failed to load native library: " + libraryPath);
            return IntPtr.Zero;
        }
        return handle;
    }`

Now that torch.cuda is working (Yet there's an exception to be solved):

There was an issue that loading the native dlls takes quite a while, and there's likely some exceptions. You need stop Unity debugging, and wait for several minutes, and restart debugging, you will find the exceptions gone.

[imadyTech]

The CUDA mode running but stuck with some exception that I can't solve:

and CPU mode worked, but very slow. It took 365 sec for only 1 epoch. For comparison, it took only 168 sec for the same task (1 epoch) with CPU mode running in Visual Studio, and 0.3 sec with CUDA mode, on my 12700h/3070ti laptop.

Hope this can be somehow helpful for somebody who's familiar with native libraries, and find a way out for the CUDA mode.