Features not detected while using data collected through iphone. Error: Not enough features or parallax; Move device around

[mansisarawata]

Basically while using VINS-Mono and VINS-Fusion for pinhole mono-imu setup it throws out error message: “Not enough features or parallax; Move device around” even when there is enough motion and different features in the environment. The camera data and imu data are collected from iphone12 and published via two separate nodes and the timestamps are matched by rospy.Time.now() and the frequency of imu is 100hz and that of camera is 30fps. Camera data is retrieved from a video file and imu from csv. Parameters are similar to euroc dataset’s mono-inertial config and VINS-Mono/Fusion is run by setting 2 in config ie we have no prior idea of the extrinsics. The rest of the parameters are adjusted to camera specifications(ie frame height,width etc). Below is the config file I am using:

%YAML:1.0

#common parameters
imu_topic: "/imu0"
image_topic: "/cam0/image_raw"
output_path: "/home/mansi/output_vins/"

#camera calibration
model_type: PINHOLE
camera_name: camera

image_width: 752 #og

image_height: 480 #og

image_width: 1920
image_height: 1080

distortion_parameters:
k1: -2.917e-01
k2: 8.228e-02
p1: 5.333e-05
p2: -1.578e-04
projection_parameters:
fx: 4.616e+02
fy: 4.603e+02
cx: 3.630e+02
cy: 2.481e+02

Extrinsic parameter between IMU and Camera.

estimate_extrinsic: 2 # 0 Have an accurate extrinsic parameters. We will trust the following imu^R_cam, imu^T_cam, don't change it.
# 1 Have an initial guess about extrinsic parameters. We will optimize around your initial guess.
# 2 Don't know anything about extrinsic parameters. You don't need to give R,T. We will try to calibrate it. Do some rotation movement at beginning.
#If you choose 0 or 1, you should write down the following matrix.
#Rotation from camera frame to imu frame, imu^R_cam
extrinsicRotation: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [0.0148655429818, -0.999880929698, 0.00414029679422,
0.999557249008, 0.0149672133247, 0.025715529948,
-0.0257744366974, 0.00375618835797, 0.999660727178]
#Translation from camera frame to imu frame, imu^T_cam
extrinsicTranslation: !!opencv-matrix
rows: 3
cols: 1
dt: d
data: [-0.0216401454975,-0.064676986768, 0.00981073058949]

#feature traker paprameters
max_cnt: 150 #150 # max feature number in feature tracking
min_dist: 30 #30 # min distance between two features
freq: 10 #10 # frequence (Hz) of publish tracking result. At least 10Hz for good estimation. If set 0, the frequence will be same as raw image
F_threshold: 1.0 # ransac threshold (pixel)
show_track: 1 # publish tracking image as topic
equalize: 1 # if image is too dark or light, trun on equalize to find enough features
fisheye: 0 # if using fisheye, trun on it. A circle mask will be loaded to remove edge noisy points

#optimization parameters
max_solver_time: 0.04 # max solver itration time (ms), to guarantee real time
max_num_iterations: 8 # max solver itrations, to guarantee real time
keyframe_parallax: 2.0 #10.0 # keyframe selection threshold (pixel)

#imu parameters The more accurate parameters you provide, the better performance
acc_n: 0.08 # accelerometer measurement noise standard deviation. #0.2 0.04
gyr_n: 0.004 # gyroscope measurement noise standard deviation. #0.05 0.004
acc_w: 0.00004 # accelerometer bias random work noise standard deviation. #0.02
gyr_w: 2.0e-6 # gyroscope bias random work noise standard deviation. #4.0e-5
g_norm: 9.81007 # gravity magnitude

#loop closure parameters
loop_closure: 1 # start loop closure
load_previous_pose_graph: 0 # load and reuse previous pose graph; load from 'pose_graph_save_path'
fast_relocalization: 0 # useful in real-time and large project
pose_graph_save_path: "/home/mansi/output_vins/pose_graph/" # save and load path

#unsynchronization parameters
estimate_td: 0 # online estimate time offset between camera and imu
td: 0.0 # initial value of time offset. unit: s. readed image clock + td = real image clock (IMU clock)

#rolling shutter parameters
rolling_shutter: 0 # 0: global shutter camera, 1: rolling shutter camera
rolling_shutter_tr: 0 # unit: s. rolling shutter read out time per frame (from data sheet).

#visualization parameters
save_image: 1 # save image in pose graph for visualization prupose; you can close this function by setting 0
visualize_imu_forward: 0 # output imu forward propogation to achieve low latency and high frequence results
visualize_camera_size: 0.4 # size of camera marker in RVIZ

Can anyone help me with this?

[tianyilim]

hi, if you are using an iphone, perhaps you need to modify the IMU data. You should flip the sign of ax ay and az and multiply them by 9.81 (g to ms)

see this for details: microsoft/lamar-benchmark#74 (comment)

Furthermore, I also needed to downsample the image before processing, else VINS would crash. You can use something like this in the launchfile:

    <!-- Downsample and crop image -->
    <node pkg="nodelet" type="nodelet" name="image_crop_decimate"
        args="standalone image_proc/crop_decimate standalone_nodelet">

        <remap from="camera/image_raw" to="/cam_phone/image_rect" />
        <remap from="camera/camera_info" to="/cam_phone/camera_info" />

        <remap from="camera_out/image_raw" to="/cam_phone_down/image_rect" />
        <remap from="camera_out/camera_info" to="/cam_phone_down/camera_info" />

        <param name="decimation_x" type="int" value="3" />
        <param name="decimation_y" type="int" value="3" />
        <param name="x_offset" type="int" value="0" />
        <param name="y_offset" type="int" value="0" />
        <param name="width" type="int" value="1440" />
        <param name="height" type="int" value="1920" />
    </node>

[mansisarawata]

thanks @tianyilim! It worked!