MEF on .NET Core 2.1 fails to load plugins from folder , but works well with .NET Framework 4.6.1

[sdg002]

Issue Title

I have 2 plugin class libraries which reference different version of the Newtonsoft.Json package. Console EXE on .NET Fwk has no problems in loading both of the plugins and invoking the 2 different versions of Newtonsoft.Json simultaneously.

However, console EXE on .NET Core 2.1 will fail on the second plugin. It is almost as though .NET Core is unable to load 2 different versions of the same assembly.

My initial view is that this is less to do with Managed Extensibility Framework . The root cause most probably lies with how .NET Core is designed to load multiple versions of referenced assemblies. In this case the referenced assembly is NewtonSoft.

• Is this a problem with Newtonsoft? Unlikely, I am able to reproduce the same problem with the package Polly. Update - see further below. I could repro the same problem using Polly .
• Are there some additional steps to be taken care of when using .NET Core, e.g. Manually forcing the assembly resolution using Resolving event of AssemblyLoadContext .
• Or something else? Update - I ditched MEF and tried the old fashioned Assembly.LoadFrom. I could repro the same behaviour with .NET Core. See further below. Possible indication that .NET Core has a different behavior.

General

I have two .NET Standard 20 class libraries (PluginNewtonsoftv12) and (PluginNewtonsoftv9) which implement an interface defined in the class library project Contracts.

I have two EXE projects - ConsoleAppNetCore and ConsoleAppNETFramework which use the DirectoryCatalog of Managed Extensibility Framework to discover and load the plugins.

The output of the projects PluginNewtonsoftv9 and PluginNewtonsoftv12 are copied to a plugins folder using a post build step. This is the out folder in the project PluginsDeliveryFolder. Both the plugin projects have the attribute CopyLocalLockFileAssemblies to ensure that all dependent assemblies get copied over to their respective folders.

What is important that EXE should have no compile time knowledge of the plugin library except for the contractual interfaces.

Solution structure

• ConsoleAppNETFramework (EXE, fwk 461)
• ConsoleAppNetCore (EXE, Core 2.1)
• PluginsDeliveryFolder (Dummy class library)
• PluginNewtonsoftv12 (.NET Standard 20)
• PluginNewtonsoftv9 (.NET Standard 20)
• Contracts (.NET Standard 20)

Steps to reproduce - Framework

1. Do a build all.
2. Ensure that the plugins DLLs are copied over to PluginsDeliveryFolder
3. Run the project ConsoleAppNETFramework.
4. Specify the path to the folder where all the plugins are copied over
5. Specify the MEF meta-data of the plugin to load. newtonv9 and then newtonv12
6. Try the reverse sequence. newtonv12 and then newtonv9
7. Both should work

---

Steps to reproduce - NET Core

1. Do a build all.
2. Ensure that the plugins DLLs are copied over to PluginsDeliveryFolder
3. Run the project ConsoleAppNetCore.
4. Specify the path to the folder where all the plugins are copied over
5. Select newtonv9 and then newtonv12. The plugin newtonv9 works fine, but newtonv12 fails because it is unable to load version 12 of Newtonsoft
6. Select newtonv12 and then newtonv9. Both the plugins work fine.

---

Screen capture of failure when using .NET Core

Exception message when using .NET Core

System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
File name: 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' ---> System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.
at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
at System.AppDomain.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
at PluginNewtonsoftV12.Class1.DoWork()
at ConsoleAppNETCore.Program.Main(String[] args) in

---

Screen capture of success when using .NET Framework

Source code
https://github.com/sdg002/MEFFwkCoreComparison

Summary
,NET Framework works as expected. It is able to load the 2 versions of Newtonsoft side by side. However, .NET Core is unable to cope with the 2 versions side by side. Is this a limitation of .NET Core?

[sdg002]

Added detailed exception message

[sdg002]

Update-Experimenting with side by side loading of different versions of the NUGET package Polly

To rule out any issues with Newtonsoft, I tried the using the NUGET package Polly. I wrote 2 class library plugins. One which is compiled with package reference 6.0 of Polly and the other references version 7.0 of Polly.

.NET Framework executable had no problems in loading both the plugins and also loading both the versions of Polly

.NET Core executable however failed when it attempted to execute the plugin with Polly 7 , if the plugin with Polly 6 was loaded first

[sdg002]

Update-To rule out issues with MEF, I used Assembly.LoadFrom and I was able to repro the above problem on .NET Core

I added a second Program.cs to the projects ConsoleAppNetCore and ConsoleAppNETFramework .

Code

        static void Main(string[] args)
        {

            try
            {
                string folderNewtonsoftV9 = @"C:\Users\saurabhd\MyTrials\OtherStuff\MEFFwkCoreComparison\PluginsDeliveryFolder\out\PluginNewtonsoftv9\PluginNewtonsoftv9.dll"; //you will need to replace this
                string folderNewtonsoftV12 = @"C:\Users\saurabhd\MyTrials\OtherStuff\MEFFwkCoreComparison\PluginsDeliveryFolder\out\PluginNewtonsoftV12\PluginNewtonsoftv12.dll"; //you will need to replace this
                string[] pathsAssemblies = new string[] { folderNewtonsoftV9, folderNewtonsoftV12 };
                foreach (string pathAssembly in pathsAssemblies)
                {
                    var assembly = System.Reflection.Assembly.LoadFrom(pathAssembly);
                    Type tTarget = assembly.GetTypes().FirstOrDefault(t => t.Name.ToLower() == "class1");
                    var plugin = Activator.CreateInstance(tTarget) as Contracts.ISomePlugin;
                    plugin.DoWork();

                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }
}

Observations

.NET Framework has no problems in loading the plugin NewtonSoft v9 followed by loading the plugin which references NewtonSoft v12.

.NET Core succeeds loading the plugin which references NewtonSoft v9, but fails when it also tries to load the plugin with NewtonSoft v12.

Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)

As per my experience with .NET Framework, side by side loading was never so much of an issue. Have things changed with .NET Core? Where am I going wrong?

[jkotas]

Duplicate of https://github.com/dotnet/coreclr/issues/26861

[sdg002]

Hi @jkotas ,
I noticed that this issue has been Closed because of the duplicate issue https://github.com/dotnet/coreclr/issues/26861

Just wanted to verify with you that the problem in this issue concerns loading of plugin assemblies via lazy loading of Managed Extensibility Framework and that you concur that these two issues are indeed connected to the same underlying problem. (I think they are, but I am not the expert here).

Just to let you know MEF has been working wonderfully for us .NET 4.6.1. We have had no issues in loading plugins and their side by side dependencies from various folders. MEF has indeed been a boon in modularizing our back end document parsing application which was otherwise becoming a monolith.

Thank you,
Sau

[jkotas]

Your are right. The two issues are related, but not exactly same.

[jkotas]

This problem is caused by Assembly.LoadFrom use by MEF here: https://github.com/dotnet/corefx/blob/0cc22ccba707d7552159363d586262fa9c4e8a2a/src/System.ComponentModel.Composition/src/System/ComponentModel/Composition/Hosting/AssemblyCatalog.cs#L573

[danmoseley]

@jkotas do you already have insight into the fix here?

[jkotas]

If we want to fix it, the fix would be about implementing proper loading context isolation using AssemblyLoadContext in MEF.

Note that .NET Framework did not provided a proper plugin isolation either in this case. A more complex plugins scenario would fail with the same problem in .NET Framework as well.

[DavidOwenBailey]

Note that .NET Framework did not provided a proper plugin isolation either in this case. A more complex plugins scenario would fail with the same problem in .NET Framework as well.

From experience, using MEF in .NET Framework appears to provide exactly this level op plugin isolation. I recently created a test where I load different versions of the same plugin, each plugin version, in turn, using different versions of a core dependency shared between the plugin and the host application without any issues.

Host App -> Plugin V1 -> Core Dependency V1
Host App -> Plugin V2 -> Core Dependency V2
Host App -> Core Dependency V1 or V2 or V3

Duplicating this using MEF in .NET core has proven impossible thus far.

I am now experimenting using AssemblyLoadContext directly, but find the API to be poorly documented.

[sdg002]

Experimenting with AssemblyDependencyResolver and Resolving event

Approach

I was exploring if the issue of loading the correct dependency can be sorted out in the Resolving event of the AssemblyLoadContext class.

We have the PluginV9 and PluginV12 and their dependencies NewtonSoft.Json(9) and Newtonsoft.Json(12) in their respective folders.

• At the very start, create two instances of AssemblyDependencyResolver class (one with the absolute path of PluginV9.deps.json and the other with the absolute path of PluginV12.deps.json
• Add both of these instances to a central List (lstAllDepsJsonResolvers)
• Set up an event handler for the Resolving event of the AssemblyLoadContext.Default instance
• In the event handler , when you get a request for Newtonsfot.Json.dll, iterate over the lstAllDepsJsonResolvers and find out which gives the best match by calling the method ResolveAssembly of AssemblyDependencyResolver.
• Do an Assembly.LoadFrom on the path returned by the above

Challenges

• Assembly.LoadFrom does not allow side by side loading. This fails on the second call. Refer issue 41859

• Assembly.ReflectionOnlyLoadFrom is not supported on .NET Core. This prevents us from examining the AssemblyVersion and only loading the highest version. [41857]. (https://github.com/dotnet/corefx/issues/41857). We could have used this to bypass the limitations of LoadFrom method.

• AssemblyLoadContext.Default.LoadFromAssemblyPath also failed on the second attempt to load a different version of Newtonsoft.

Is there an alternative?

I would be happy if the designers of .NET Core can propose an alternative. I think the scenario is important.

[jkotas]

The AssemblyDependencyResolver needs to be paired with AssemblyLoadContext instance to allow SxS loading. An example on how to do it is here: https://github.com/dotnet/samples/tree/master/core/extensions/AppWithPlugin/AppWithPlugin

[jkotas]

cc @vitek-karas

[sdg002]

Thank you @jkotas for pointing me to the sample. I have made significant progress.

I have derived inspiration from this sample and finally able to have the Newtonsoft v9 and Newtonsoft v12 loaded on demand.

Background

We have the PluginV9 and PluginV12 and their dependencies NewtonSoft.Json(9) and Newtonsoft.Json(12) in their respective folders.

Initial steps

• At the very start, create two instances of AssemblyDependencyResolver class (one with the absolute path of PluginV9.deps.json and the other with the absolute path of PluginV12.deps.json
• Add both of these instances to a central List (lstAllDepsJsonResolvers)
• Set up a single event handler for the Resolving event of the AssemblyLoadContext.Default instance
• In the event handler , when you get a request for Newtonsfot.Json.dll, iterate over the lstAllDepsJsonResolvers and find out which gives the best match by calling the method ResolveAssembly of AssemblyDependencyResolver.
• Do an Assembly.LoadFile on the path returned by ResolveAssembly and collect the Assembly objects in a local List.
• Find the assembly in the local List whose Version is nearest to the requested assembly in the event handler Resolving.

What is the difference - Assembly.LoadFile vs Assembly.LoadFrom

I was using Assembly.LoadFrom in my first approach, but unlike .NET Framework , this refuses to load the second version of the same assembly (Newtonsoft.json.dll) if the first version is already loaded.

Therefore , I used Assembly.LoadFile. Thanks to https://github.com/dotnet/coreclr/issues/27269

Code snippet - Resolving event

        private System.Reflection.Assembly Default_Resolving(AssemblyLoadContext arg1, System.Reflection.AssemblyName arg2)
        {
            Trace.WriteLine($"Default_Resolving event, {arg2}");
            var lstMatchingAssemblies = new List<System.Reflection.Assembly>();
            foreach (var resolver in _lstResolvers)
            {
                var pathAssembly = resolver.ResolveAssemblyToPath(arg2);
                if (pathAssembly == null) continue;

                var assem = System.Reflection.Assembly.LoadFile(pathAssembly);
                lstMatchingAssemblies.Add(assem);
            }
            var assembliesWithClosestMatchingVersion = 
                        lstMatchingAssemblies.
                        OrderBy(a => Math.Abs( arg2.Version.CompareTo(a.GetName().Version)) ).
                        ToArray();
            var bestAssembly= assembliesWithClosestMatchingVersion.FirstOrDefault();
            Trace.WriteLine($"The Resolving event found {bestAssembly.GetName()} for the requested assembly:'{arg2}'");
            return bestAssembly;
        }

Challenges and further thoughts

• The combination of Resolving event handling and multiple AssemblyDependencyResolver appears to work.
• How reliable is the method ResolveAssembly ? It appears not to honor the public key token. Think of 2 different dependency assemblys like BlahJson.dll with entirely different version and public key token.
• Guidance from the think tank of Microsoft is extremely crucial. I would like Microsoft to approve/improve/reject this approach.

Sample code

https://github.com/sdg002/MEFFwkCoreComparison
Project - ConsoleAppNetDepsJson.csproj
Class - FindAndLoadDependencies.cs - This handles Resolving event and finds the best match in all the .deps.json files.

Thanks,
Sau

[sdg002]

I am hoping that the above approach will work with the DirectoryCatalog of Managed Extensibilty Framework. I am yet to try that out.

The property LoadedFiles of DirectoryCatalog should lead us to the .depa.json file for each of the assembly in LoadedFiles.

Add handler for AssemblyLoadContxt.Resolving and search through with all the .deps.json files using a collection of AssemblyDependencyResolver instances.

[jkotas]

It appears not to honor the public key token

.NET Core assembly loader ignores strong names. This was done to enable migration of the ecosystem away from strong naming that is endless source of pain.

2 different dependency assemblys like BlahJson.dll with entirely different version and public key token.

This situation does not happen in practice. The situations where strong names cause problems are multiple orders of magnitude more common.

assembliesWithClosestMatchingVersion

Does the .NET Framework equivalent of what you are trying to have the same closest matching version policy? This policy will lead to unpredictable behaviors in more complex systems (e.g. a new plugin is added to the system, and unrelated plugin breaks). My recommendation would be to avoid this version guessing at runtime.

[sdg002]

Understood. .NET Core assembly loader ignores strong names completely. I was not aware of this change. Thank you.

Coming back to the original problem, please help me understand, if the aforementioned behaviour is the reason for Assembly.LoadFrom is not searching for Newtonsoft(v12).dll when Newtonsoft(v9).dll has been loaded already?

I am guessing that the loader stops further discover when it finds that there is an already loaded assembly whose file name matches Newtonsoft.json.dll .

[jkotas]

I am guessing that the loader stops further discover when it finds that there is an already loaded assembly whose file name matches Newtonsoft.json.dll

Right. The loader does lookup based on the simple names only.

[dotnet-policy-service]

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.