Exported CoreML models are broken

[joebreaux]

Search before asking

• I have searched the HUB issues and found no similar bug report.

HUB Component

Models

Bug

After training a model, the exported CoreML version (with NMS) selected does not work correctly when used inside a swift project. Older models I trained and exported to the .mlmodel format worked correctly. However, the newer .mlpackage models do not work correctly.

I have tried exporting as CoreML and exporting as pytorch and then running yolo export model=model51.pt format=coreml imgsz=640 nms=True

The resulting model in both cases fails to perform correctly. The "preview" option within the console does show correct output. Here is an example of the preview output:

Here is an example when the CoreML version is run in a swift project. As you can see its detecting all sorts of things that are not in the image and location of the bounding boxes is all messed up.

If I use a model trained a few months ago that was exported as .mlmodel in the swift app, everything is fine. A good example of this is model27. I have a .mlmodel version I downloaded a few months ago - works fine. I just went and exported that exact model, this time it produced a .mlpackage and the resulting .mlpackage shows the same issue as above.

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

[UltralyticsAssistant]

👋 Hello @joebreaux, thank you for raising an issue about the Ultralytics HUB 🚀! We appreciate you taking the time to describe the problem and share example screenshots—it helps us a lot. An Ultralytics engineer will review this soon and provide further assistance.

In the meantime, please check out the following HUB Docs for helpful resources that may provide context or solutions:

• Models: Training and Exporting. Learn the options available for exporting YOLO models, including CoreML formats.
• Integrations: CoreML and iOS. Understand CoreML integration and tips for deploying models in your Swift project.
• Inference API. Explore how to test models cloud-side to cross-check their behavior before local deployment.

If this is indeed a 🐛 Bug Report, we kindly request that you provide a Minimum Reproducible Example (MRE) to help us effectively diagnose and resolve the issue. You can refer to our guidelines here for creating an MRE. In this case, it would be helpful to:

1. Specify the exact HUB workflow and steps you used to export the CoreML model (e.g., model version, parameters set during export, command-line commands used if applicable).
2. Share any additional screenshots or logs from the Swift environment that highlight the differences in behavior between .mlmodel and .mlpackage.
3. Test the exported CoreML model using another script or device (if possible) to confirm if the issue is specific to Swift integration.

We understand that debugging deployment issues can be tricky, but providing these details will allow us to better assist you! 🛠️

Thank you again for your patience and for using Ultralytics HUB. We’ll be in touch! 🔍

[pderrenger]

@joebreaux thank you for providing detailed information and screenshots regarding the issue you're facing with CoreML model exports. Let's address this step-by-step:

1. CoreML Format Changes:
   It seems the issue is related to the transition from .mlmodel to .mlpackage. Starting with newer versions, CoreML exports now default to .mlpackage for better compatibility and functionality across Apple's ecosystem. However, there may be differences in how .mlpackage models are handled in Swift projects compared to .mlmodel.

2. Verify Export Settings:
   To ensure the model is exported correctly, please make sure you're using the latest version of the ultralytics package. You can update it using:

   pip install ultralytics --upgrade

   Additionally, when exporting, verify that you are explicitly enabling Non-Maximum Suppression (NMS) using the nms=True argument, as you mentioned:

   yolo export model=model.pt format=coreml imgsz=640 nms=True

3. Validate CoreML Model:
   Since the "Preview" in the Ultralytics HUB console shows correct outputs, the model itself appears to be functional. The issue may lie in the integration or configuration within your Swift project. Ensure you are using the latest CoreML framework in your project and that the .mlpackage is being loaded correctly. Refer to Apple's guide on integrating CoreML models: Integrating a Core ML Model into Your App.

4. Debugging in Swift:

   • Confirm that the input image size matches the imgsz used during export (e.g., 640x640). Mismatched input dimensions can lead to incorrect predictions.
   • Check the preprocessing steps in your Swift code. Ensure that input normalization, resizing, and channel ordering match the model's requirements. For instance:
     • Resize the image to the correct dimensions.
     • Normalize pixel values to the range [0, 1] or [-1, 1], depending on the model.
     • Use the correct RGB or BGR channel order.

5. Testing Older .mlmodel Exports:
   If older .mlmodel exports work fine, you can try explicitly exporting the model in the .mlmodel format instead of .mlpackage. This can be done using:

   yolo export model=model.pt format=coreml nms=True dynamic=False

   By setting dynamic=False, the export may revert to the .mlmodel format. Test this in your Swift project to see if it resolves the issue.

6. Provide an MRE (Minimum Reproducible Example):
   If the issue persists, it would be helpful to provide a small reproducible example of your Swift integration code. This can help in isolating whether the problem is with the model export or the Swift integration.

7. Follow-Up:
   If none of the above resolves the issue, consider testing the .mlpackage model using Apple's CoreML tools, such as coremltools. You can validate the model's structure and outputs:

   import coremltools as ct
   
   model = ct.models.MLModel("model.mlpackage")
   print(model)

If you continue to face issues despite these steps, please let us know with additional details, and we'll assist further. The Ultralytics community is constantly improving workflows, and your feedback is invaluable in resolving such challenges! 😊

[joebreaux]

@pderrenger To eliminate any potential issues from my application itself, I used the preview function within Xcode. This first screenshot is of model27-back.mlmodel which is what i downloaded several months ago. You will notice that its output matches the preview output from within the console.

Here is the preview output within the console for model27. You can find this model within my account.

Those above screenshots match as you would expect. I then used the console to generate the export (to eliminate any potential issues with my computer). When I export using the console I got model27.mlpackage. Here is the screenshot when its run through preview on Xcode:

You should be able to use these same steps to reproduce. Simply export the model27 from my account and then run it using Xcode preview. Here is the image I ran through the model so that you can run the same image:

Hopefully this allows you to reproduce. Let me know if you need anything else.

[glenn-jocher]

@joebreaux thank you for the update, I'm attempting to reproduce now.

[glenn-jocher]

@joebreaux CoreML export is working correctly for me on newly exported models (if your model is out of date it may need to be trained again with the latest version of ultralytics):

For full NMS functionality make sure you select NMS in the advanced options tab when exporting to CoreML:

[joebreaux]

@glenn-jocher I am using Add NMS when exporting. One thing to note, this does not affect every model and export. You will need to export with my specific models to see this behavior.

Check model51 on my account. This was trained within the past few days. When exported, it has the same issue as model27. It works in console preview, but fails in Xcode preview - using the image from my last post.

However, I have a model trained on a different data set called "seatbelt15". This was also trained within the past week. The export on this model works perfectly fine.

That indicates its not something universally affecting all model exports. Can you take a look at my model51 to reproduce and determine why any model trained on that data set has this issue?

[joebreaux]

@glenn-jocher We did some additional testing to try and better understand why some trained models have the issue and others dont. The first thing we did was remove a lot of the classes, which worked. These class labels worked without any issue in Xcode preview:

We then added the rest of the "speed" labels back and trained. The only different between this one and the one above is that this one had 17 classes and the one above only had 7. This model failed in Xcode preview:

We then remapped all of the speed classes to just be a single class called "speed". This model again worked in Xcode preview:

I'm not quite sure what to make of this. Perhaps its when we have too many classes? Hopefully this helps you track down the issue.

[joebreaux]

Using format=mlmodel resolves the issue. The model is then exported as .mlmodel and works correctly.

[pderrenger]

Thank you for following up and confirming that exporting with format=mlmodel resolves the issue! This indicates that the .mlpackage format might be encountering issues specific to certain configurations, such as the number or naming of classes, as you previously observed.

For now, using format=mlmodel is a reliable workaround, and it ensures compatibility with your Swift project and Xcode preview. However, to improve the usability of .mlpackage exports and investigate this further, I will pass this information along to the development team. The transition to .mlpackage is intended to provide better long-term support, so resolving this discrepancy is important.

If you have any additional findings or patterns that might help isolate the root cause (e.g., specific class counts, label naming conventions, etc.), feel free to share them here. Your detailed testing has already been incredibly helpful in diagnosing this issue! 😊

In the meantime, if you need further assistance or have other questions, don’t hesitate to ask. Thank you again for your patience and for working with us to identify this behavior!