face detection returns [] #44

vidurajith-darshana · 2023-10-19T12:46:46Z

Hello, I'm facing an issue with my IOS device. When I send an image URL captured through a react-native-vision-camera it returns an empty array. Do you know how I can solve this?

const photo = await camera.current.takePhoto({
enableShutterSound: false,
});

try {
const faceResults = await FaceDetection.detect('file://' + photo.path, {
contourMode: 'all',
});
console.log(faceResults);
} catch (e) {
console.log(e);
}

EX: I didn't use any firebase package.

a7medev · 2023-10-19T17:42:30Z

Hi @vidurajith-darshana! Thanks for reporting this issue. I'll check it and get back to you once I have something to share 🙏🏼.

vidurajith-darshana · 2023-10-19T17:45:12Z

@a7medev thanks for the reply. Hope some quick solution as soon as possible. we are stuck with this issue. Please note I've never used firebase in my app and if there are any additional installation guides will be really helpful 🙏🏼.

zheoreh · 2023-10-24T21:19:58Z

Hi everyone! I have same issue but only with ios devices and selfie photos made by them. If I download selfie photo from internet it works). I have same issue in react-native-face-detection module too. I use react-native-image-picker module and its methods: launchImageLibrary and launchCamera to get a selfie. If I share problem selfie somewhere and then download it then this selfie works) magic! It looks like ios protects reading images made by its device.

Update:
Same issue is reproduced in your example too

iOS 16.5.1 (20F75)
iPad (5th generation) (Model A1822)

flexbox · 2023-10-26T17:47:58Z

@vidurajith-darshana

I've never used firebase in my app and if there are any additional installation guides will be really helpful 🙏🏼.

you can setup firebase with https://rnfirebase.io/
if you are using expo here is how https://docs.expo.dev/guides/using-firebase/#using-react-native-firebase
Do not use the Firebase JS SDK

senturatechnologies · 2023-10-29T13:19:46Z

@flexbox Actually he asked does he needs to setup Firebase for react-native-ml-kit ? he asked if there were any other installation instructions related to react-native-ml-kit based on Firebase, it's helpful. not about the Firebase setup

nth-zik · 2023-10-31T09:04:33Z

Hi everyone! I have same issue but only with ios devices and selfie photos made by them. If I download selfie photo from internet it works). I have same issue in react-native-face-detection module too. I use react-native-image-picker module and its methods: launchImageLibrary and launchCamera to get a selfie. If I share problem selfie somewhere and then download it then this selfie works) magic! It looks like ios protects reading images made by its device.

Update: Same issue is reproduced in your example too

iOS 16.5.1 (20F75) iPad (5th generation) (Model A1822)

Same on me

vidurajith-darshana · 2023-10-31T12:39:58Z

@a7medev is there any update, please? we have been stuck with this issue since last week.

zheoreh · 2023-11-13T17:10:05Z

@vidurajith-darshana have you found some solution?

vidurajith-darshana · 2023-11-14T01:55:42Z

no luck, still the same

codemem · 2023-12-07T16:52:16Z

after debugging, i think the problem is with the temp file path from vision-camera output. As the photo file is stored in a temporary location, that is not accessible directly by file path. It cannot work with that file path atm.

codemem · 2023-12-07T16:54:56Z

it's throwing error like “ABA4B2D6-A241-4DF6-A1BA-FD6F984A2EE3-1864-0000024DF9AE4C71.jpg” couldn’t be opened because there is no such file."

vidurajith-darshana · 2023-12-07T19:57:15Z

seems we don't have an answer here. I've been struggling for weeks here using this library. any PR is welcome for the issues. This is going to be an inactive, not-maintained repo.

codemem · 2023-12-07T21:17:55Z

I will try to see if i can open pr. Take a rest :)

vidurajith-darshana · 2023-12-08T14:25:40Z

great! but did it give your results as you expected? when I try to get the face detection results, it always returns []. did you have your results correctly?

codemem · 2023-12-08T14:51:54Z

hmm sorry, i messed up the lib. i supposed to comment on other issue from vision-camera-face-detection not here.
No it's still not working

codemem · 2023-12-13T22:00:38Z

it does not work with vision-camera. I think there is smt with temp file. It works fine with local photo from library but not with temp file name file:///private/blah
when working with local file it returns correct result. file path from photo library file:///var/mobile

[{"frame": {"height": 1206, "left": 530, "top": 968, "width": 1206}, "rotationX": -5.255302906036377, "rotationY": -3.0993707180023193, "rotationZ": -1.612923502922058}] faces

p/s: on IOS

write2sv · 2024-06-14T15:19:07Z

I had similar issue with another library teslamotors/react-native-camera-kit#568

For me, its not to do with file location as suggested but to do with image orientation. Here is how I patched mine so that I can make this work for the react-native-camera-kit library I am using. I could be wrong but I don't think this has to do with the mlkit itself.

Create a patch file (lookup react native patch if you are not familiar with this)
patches/@react-native-ml-kit+face-detection+1.3.1.patch
Paste these file contents

index 87fd4ff..f237b71 100644
--- a/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
+++ b/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
@@ -234,6 +234,15 @@ - (NSDictionary*)faceToDict: (MLKFace*)face
     return dict;
 }
 
+- (UIImage*)fixImageOrientation: (UIImage*)image
+  {
+    UIGraphicsBeginImageContext(image.size);
+    [image drawAtPoint:CGPointZero];
+    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
+    UIGraphicsEndImageContext();
+    return newImage ?: image;
+}
+
 RCT_EXPORT_METHOD(detect: (nonnull NSString*)url
                   withOptions: (NSDictionary*)optionsDict
                   resolver:(RCTPromiseResolveBlock)resolve
@@ -241,7 +250,9 @@ - (NSDictionary*)faceToDict: (MLKFace*)face
 {
     NSURL *_url = [NSURL URLWithString:url];
     NSData *imageData = [NSData dataWithContentsOfURL:_url];
-    UIImage *image = [UIImage imageWithData:imageData];
+    UIImage *imageraw = [UIImage imageWithData:imageData];
+    UIImage *image = [self fixImageOrientation:imageraw];
+
     MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image];
     MLKFaceDetectorOptions *options = [self getOptions:optionsDict];
     MLKFaceDetector *faceDetector = [MLKFaceDetector faceDetectorWithOptions:options];

SlZeroth · 2024-07-29T16:51:19Z

`
diff --git a/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec b/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec
index 0db7d5f..366aef5 100644
--- a/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec
+++ b/node_modules/@react-native-ml-kit/face-detection/RNMLKitFaceDetection.podspec
@@ -21,6 +21,6 @@ Pod::Spec.new do |s|
s.requires_arc = true

s.dependency "React"

s.dependency "GoogleMLKit/FaceDetection", "4.0.0"

s.dependency "GoogleMLKit/FaceDetection", "5.0.0"
end

diff --git a/node_modules/@react-native-ml-kit/face-detection/android/.gitignore b/node_modules/@react-native-ml-kit/face-detection/android/.gitignore
deleted file mode 100644
index 6125af2..0000000
--- a/node_modules/@react-native-ml-kit/face-detection/android/.gitignore
+++ /dev/null
@@ -1,4 +0,0 @@
-.iml
-gradle/
-gradlew
-gradlew.bat
diff --git a/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m b/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
index 87fd4ff..909e178 100644
--- a/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
+++ b/node_modules/@react-native-ml-kit/face-detection/ios/FaceDetection.m
@@ -234,15 +234,23 @@ - (NSDictionary)faceToDict: (MLKFace*)face
return dict;
}

-RCT_EXPORT_METHOD(detect: (nonnull NSString*)url

             withOptions: (NSDictionary*)optionsDict

+RCT_EXPORT_METHOD(detect:(NSString * _Nonnull)url

             withOptions:(NSDictionary *)optionsDict
             resolver:(RCTPromiseResolveBlock)resolve
             rejecter:(RCTPromiseRejectBlock)reject)

{
NSURL *_url = [NSURL URLWithString:url];
NSData *imageData = [NSData dataWithContentsOfURL:_url];

UIImage *image = [UIImage imageWithData:imageData];
MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:image];

UIImage *originalImage = [UIImage imageWithData:imageData];
// EXIF 정보를 고려하여 이미지 회전
UIImage *fixedImage = [self fixImageOrientation:originalImage];
// EXIF 없는 새 이미지 데이터 생성
NSData *fixedImageData = UIImageJPEGRepresentation(fixedImage, 1.0);
UIImage *imageForDetection = [UIImage imageWithData:fixedImageData];
MLKVisionImage *visionImage = [[MLKVisionImage alloc] initWithImage:imageForDetection];
MLKFaceDetectorOptions options = [self getOptions:optionsDict];
MLKFaceDetector faceDetector = [MLKFaceDetector faceDetectorWithOptions:options];
[faceDetector processImage:visionImage
@@ -260,4 +268,77 @@ - (NSDictionary)faceToDict: (MLKFace)face
}];
}

+// EXIF 정보를 고려하여 이미지 회전
+- (UIImage *)fixImageOrientation:(UIImage *)image {

if (image.imageOrientation == UIImageOrientationUp) return image;
CGAffineTransform transform = CGAffineTransformIdentity;
switch (image.imageOrientation) {
```
   case UIImageOrientationDown:
```

   case UIImageOrientationDownMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.width, image.size.height);

       transform = CGAffineTransformRotate(transform, M_PI);

```
       break;
```
```
   case UIImageOrientationLeft:
```

   case UIImageOrientationLeftMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.width, 0);

       transform = CGAffineTransformRotate(transform, M_PI_2);

```
       break;
```
```
   case UIImageOrientationRight:
```

   case UIImageOrientationRightMirrored:

       transform = CGAffineTransformTranslate(transform, 0, image.size.height);

       transform = CGAffineTransformRotate(transform, -M_PI_2);

```
       break;
```
```
   case UIImageOrientationUp:
```
```
   case UIImageOrientationUpMirrored:
```
```
       break;
```
}
switch (image.imageOrientation) {
```
   case UIImageOrientationUpMirrored:
```

   case UIImageOrientationDownMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.width, 0);

       transform = CGAffineTransformScale(transform, -1, 1);

```
       break;
```

   case UIImageOrientationLeftMirrored:

   case UIImageOrientationRightMirrored:

       transform = CGAffineTransformTranslate(transform, image.size.height, 0);

       transform = CGAffineTransformScale(transform, -1, 1);

```
       break;
```
```
   case UIImageOrientationUp:
```
```
   case UIImageOrientationDown:
```
```
   case UIImageOrientationLeft:
```
```
   case UIImageOrientationRight:
```
```
       break;
```
}
CGContextRef ctx = CGBitmapContextCreate(NULL, image.size.width, image.size.height,

                                        CGImageGetBitsPerComponent(image.CGImage), 0,

                                        CGImageGetColorSpace(image.CGImage),

                                        CGImageGetBitmapInfo(image.CGImage));

CGContextConcatCTM(ctx, transform);
switch (image.imageOrientation) {
```
   case UIImageOrientationLeft:
```

   case UIImageOrientationLeftMirrored:

```
   case UIImageOrientationRight:
```

   case UIImageOrientationRightMirrored:

       CGContextDrawImage(ctx, CGRectMake(0,0,image.size.height,image.size.width), image.CGImage);

```
       break;
```
```
   default:
```

       CGContextDrawImage(ctx, CGRectMake(0,0,image.size.width,image.size.height), image.CGImage);

```
       break;
```
}
CGImageRef cgimg = CGBitmapContextCreateImage(ctx);
UIImage *img = [UIImage imageWithCGImage:cgimg];
CGContextRelease(ctx);
CGImageRelease(cgimg);
return img;
+}

@EnD

`

It is working

just do remove the exif information ...

Fardeen-N · 2024-12-17T09:29:28Z

Ios device issue any one find solution if so please update

a7medev self-assigned this Oct 19, 2023

a7medev added bug Something isn't working face detection labels Oct 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

face detection returns [] #44

face detection returns [] #44

vidurajith-darshana commented Oct 19, 2023 •

edited

Loading

a7medev commented Oct 19, 2023

vidurajith-darshana commented Oct 19, 2023

zheoreh commented Oct 24, 2023 •

edited

Loading

flexbox commented Oct 26, 2023 •

edited

Loading

senturatechnologies commented Oct 29, 2023 •

edited

Loading

nth-zik commented Oct 31, 2023

vidurajith-darshana commented Oct 31, 2023

zheoreh commented Nov 13, 2023

vidurajith-darshana commented Nov 14, 2023

codemem commented Dec 7, 2023 •

edited

Loading

codemem commented Dec 7, 2023

vidurajith-darshana commented Dec 7, 2023

codemem commented Dec 7, 2023

vidurajith-darshana commented Dec 8, 2023

codemem commented Dec 8, 2023

codemem commented Dec 13, 2023

write2sv commented Jun 14, 2024

SlZeroth commented Jul 29, 2024 •

edited

Loading

Fardeen-N commented Dec 17, 2024

face detection returns [] #44

face detection returns [] #44

Comments

vidurajith-darshana commented Oct 19, 2023 • edited Loading

a7medev commented Oct 19, 2023

vidurajith-darshana commented Oct 19, 2023

zheoreh commented Oct 24, 2023 • edited Loading

flexbox commented Oct 26, 2023 • edited Loading

senturatechnologies commented Oct 29, 2023 • edited Loading

nth-zik commented Oct 31, 2023

vidurajith-darshana commented Oct 31, 2023

zheoreh commented Nov 13, 2023

vidurajith-darshana commented Nov 14, 2023

codemem commented Dec 7, 2023 • edited Loading

codemem commented Dec 7, 2023

vidurajith-darshana commented Dec 7, 2023

codemem commented Dec 7, 2023

vidurajith-darshana commented Dec 8, 2023

codemem commented Dec 8, 2023

codemem commented Dec 13, 2023

write2sv commented Jun 14, 2024

SlZeroth commented Jul 29, 2024 • edited Loading

Fardeen-N commented Dec 17, 2024

vidurajith-darshana commented Oct 19, 2023 •

edited

Loading

zheoreh commented Oct 24, 2023 •

edited

Loading

flexbox commented Oct 26, 2023 •

edited

Loading

senturatechnologies commented Oct 29, 2023 •

edited

Loading

codemem commented Dec 7, 2023 •

edited

Loading

SlZeroth commented Jul 29, 2024 •

edited

Loading