Struggling to get expected results

[carsoncramer]

Hi there,

I have been running several visual odometry systems and measuring their trajectory errors. I have been attempting to use COLMAP as a "baseline" to compare the results of the real-time VO systems to. Then I discovered GLOMAP, and have been using it due to its faster runtime, but I am struggling to get results that beat the real time systems with both colmap and glomap.

For example when running them on EuRoC's MH_01 sequence, I get the following results:

COLMAP trajectory RMSE: 0.00195m after running for 35823.9 seconds
GLOMAP trajectory RMSE: 0.00196m after running for 22299.0 seconds

What has been confusing me is when I compare this to real-time systems like DSO.

DSO trajectory RMSE: 0.00163m when running in real time

I would think colmap and glomap would blow any real-time system out of the water because they do not have the real time constraint, which leads me to think that I must be running colmap and glomap incorrectly.

I have been running colmap and glomap from a python script, here is the important snippet. I am getting the camera model and camera parameters from the datasets I have been running it on.

    print("Running feature extraction...")
    common_start_time = time.time()
    feature_extractor_cmd = [
        "colmap", "feature_extractor",
        "--database_path", database_path,
        "--image_path", image_dir,
        "--ImageReader.camera_model", camera_model,
        "--ImageReader.camera_params", camera_params,
        "--ImageReader.single_camera", "1",
        "--SiftExtraction.use_gpu", "1"
    ]
    subprocess.run(feature_extractor_cmd, check=True)

    print("Running sequential feature matching...")
    sequential_matcher_cmd = [
        "colmap", "sequential_matcher",
        "--database_path", database_path,
        "--SiftMatching.max_num_trials", "1000000",
        "--SiftMatching.use_gpu", "1"
    ]
    subprocess.run(sequential_matcher_cmd, check=True)
    common_end_time = time.time() 
    colmap_mapper_ba_start_time = time.time()
    print("Running mapper...")
    mapper_cmd = [
        "colmap", "mapper",
        "--database_path", database_path,
        "--image_path", image_dir,
        "--output_path", colmap_output_dir,
        "--Mapper.ba_refine_focal_length", "0",
        "--Mapper.ba_refine_principal_point", "0",
        "--Mapper.ba_refine_extra_params", "0",
        "--Mapper.extract_colors", "0"
    ]
    subprocess.run(mapper_cmd, check=True)
    colmap_output_dir = os.path.join(colmap_output_dir, "0") # Colmap forces output into this '0' directory
    print("Running bundle adjustment...")
    bundle_adjust_cmd = [
        "colmap", "bundle_adjuster",
        "--input_path", colmap_output_dir,
        "--output_path", colmap_output_dir,
        "--BundleAdjustment.refine_focal_length", "0",
        "--BundleAdjustment.refine_principal_point", "0",
        "--BundleAdjustment.refine_extra_params", "0",
        "--BundleAdjustment.refine_extrinsics", "1"
    ]
    subprocess.run(bundle_adjust_cmd, check=True) 
    colmap_mapper_ba_end_time = time.time()
    
    print("Running glomap bundle adjustment...")
    glomap_mapper_ba_start_time = time.time()
    glomap_mapper_cmd = [
        "glomap", "mapper",
        "--database_path", database_path,
        "--image_path", image_dir,
        "--output_path", glomap_output_dir, 
        "--BundleAdjustment.optimize_intrinsics", "0",
    ]
    subprocess.run(glomap_mapper_cmd, check=True)
    glomap_mapper_ba_end_time = time.time()

Do you have any advice on how I can improve the colmap and glomap trajectory errors?
Thank you so much for your time and for developing these awesome systems!!

[lpanaf]

Hi @carsoncramer, thanks for your question. I am a bit surprise that GLOMAP takes such a long time to finish, but apart from that, I think the result is reasonable. For this particular scene, the error is already quite small (<0.2cm?), so I don't think COLMAP or GLOMAP can do much better there. However, for more difficult cases, the difference might be larger. One thing I notice from your code is that you do not have loop_closure when do image matching (you can do so by changing --SequentialMatching.loop_detection=1 and provide --SequentialMatching.vocab_tree_path), and this might provide you with some extra accuracy. If you still have the logging information, do you mind sharing the detailed runtime information on GLOMAP so we can understand better why it is slow?

[carsoncramer]

You are right, on more difficult sequences we observe much better performances from COLMAP and GLOMAP than real-time methods. We also found an issue with the third-party evaluation framework we were using which was leading to some confusing results. Now that we fixed it, we are getting results that make sense.

Really the only concern was the accuracy problems, the runtime will not be a problem. Thank you so much for your reply!

[carsoncramer]

Ah apologies for reopening this!! We decided we want to log the runtimes for the systems, so I want to make sure that I accurately represent the runtime for GLOMAP. What runtime information you looking for? I have the shell output from the system, the files "cameras.bin cameras.txt images.bin images.txt points3D.bin points3D.txt" that are the output from the mapper, and then also the database for the run.

[lpanaf]

Hi, the raw shell output would tell how long each part of the system takes to finish, and the overall running time is logged at the end. For us, the detailed runtime information would be interesting (as we want to understand the performance of every part), but if you only want to put a single entry for GLOMAP, then the overall runtime would be more relevant.

[carsoncramer]

Here is the beginning of the glomap output for EuRoC_MH03_Medium:

Running glomap bundle adjustment...
 Loading Images 2700 / 2700
 Loading Image Pair 36580 / 36580
I1021 15:55:56.724975 357716 colmap_converter.cc:303] Pairs read done. 2132 / 36580 are invalid
I1021 15:55:56.728158 357716 global_mapper.cc:80] Loaded database
-------------------------------------
Running preprocessing ...
-------------------------------------
I1021 15:55:56.733162 357716 view_graph_manipulation.cc:249] Decompose relative pose for 34448 pairs
I1021 15:55:56.778054 357716 view_graph_manipulation.cc:303] Decompose relative pose done. 14142 pairs are pure rotation
I1021 15:55:56.778398 357716 timer.cc:87] Elapsed time: 0.05023 [seconds]
-------------------------------------
Running view graph calibration ...
-------------------------------------
I1021 15:55:56.778426 357716 view_graph_calibration.cc:16] Start ViewGraphCalibrator
I1021 15:55:56.784759 357716 view_graph_calibration.cc:33] No cameras to optimize
-------------------------------------
Running relative pose estimation ...
-------------------------------------
I1021 15:55:56.788542 357716 image_undistorter.cc:20] Undistorting images..
I1021 15:55:56.830986 357716 image_undistorter.cc:41] Image undistortion done
I1021 15:55:56.835500 357716 relpose_estimation.cc:26] Estimating relative pose for 34448 pairs
 Estimating relative pose: 100%
I1021 15:57:17.446753 357716 relpose_estimation.cc:86] Estimating relative pose done
I1021 15:57:18.066210 357716 relpose_filter.cc:46] Filtered 1464 relative poses with inlier number < 30
I1021 15:57:18.067431 357716 relpose_filter.cc:63] Filtered 62 relative poses with inlier ratio < 0.25
I1021 15:57:18.074391 357716 timer.cc:87] Elapsed time: 81.28679 [seconds]
-------------------------------------
Running rotation averaging ...
-------------------------------------
I1021 15:57:18.721763 357716 relpose_filter.cc:31] Filtered 35 relative rotation with angle > 10 degrees
I1021 15:57:19.355590 357716 relpose_filter.cc:31] Filtered 0 relative rotation with angle > 10 degrees
I1021 15:57:19.365832 357716 global_mapper.cc:105] 2700 / 2700 images are within the connected component.
I1021 15:57:19.365842 357716 timer.cc:87] Elapsed time: 1.29143 [seconds]
-------------------------------------
Running track establishment ...
-------------------------------------
 Initializing pairs 36580 / 36580
 Establishing pairs 36580 / 36580
 Establishing tracks 512457 / 512457
I1021 15:57:31.042546 357716 track_establishment.cc:149] Discarded 2734 tracks due to inconsistency
I1021 15:57:32.614913 357716 global_mapper.cc:125] Before filtering: 512457, after filtering: 327598
I1021 15:57:32.614934 357716 timer.cc:87] Elapsed time: 13.24841 [seconds]
-------------------------------------
Running global positioning ...
-------------------------------------
I1021 15:57:32.867395 357716 image_undistorter.cc:20] Undistorting images..
I1021 15:57:32.867404 357716 image_undistorter.cc:41] Image undistortion done
I1021 15:57:32.867741 357716 global_positioning.cc:43] Setting up the global positioner problem
I1021 15:57:37.152731 357716 global_positioning.cc:68] Solving the global positioner problem
I1021 16:10:17.387715 357716 global_positioning.cc:77] Ceres Solver Report: Iterations: 68, Initial cost: 5.356735e+07, Final cost: 2.601072e+01, Termination: CONVERGENCE
I1021 16:10:17.574951 357716 track_filter.cc:86] Filtered 2026 / 327598 tracks by angle error
I1021 16:10:17.574962 357716 timer.cc:87] Elapsed time: 764.70844 [seconds]
-------------------------------------
Running bundle adjustment ...
-------------------------------------
I1021 16:10:18.891165 357716 global_mapper.cc:177] Bundle adjustment start
I1021 16:12:59.077392 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 33, Initial cost: 1.483930e+06, Final cost: 1.266535e+06, Termination: CONVERGENCE
I1021 16:12:59.077435 357716 global_mapper.cc:193] Global bundle adjustment iteration 1 / 3, stage 1 finished (position only)
I1021 16:12:59.077437 357716 timer.cc:87] Elapsed time: 160.18626 [seconds]
I1021 16:20:21.513185 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 16, Initial cost: 1.266535e+06, Final cost: 1.186973e+06, Termination: CONVERGENCE
I1021 16:20:21.513233 357716 global_mapper.cc:205] Global bundle adjustment iteration 1 / 3, stage 2 finished
I1021 16:20:21.513235 357716 timer.cc:87] Elapsed time: 602.62206 [seconds]
I1021 16:20:21.514170 357716 image_undistorter.cc:20] Undistorting images..
I1021 16:20:21.553661 357716 image_undistorter.cc:41] Image undistortion done
I1021 16:20:21.554220 357716 global_mapper.cc:216] Filtering tracks by reprojection ...
I1021 16:20:21.725302 357716 track_filter.cc:48] Filtered 706 / 327598 tracks by reprojection error
I1021 16:21:10.932077 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 8, Initial cost: 1.179295e+06, Final cost: 1.178338e+06, Termination: CONVERGENCE
I1021 16:21:10.932123 357716 global_mapper.cc:193] Global bundle adjustment iteration 2 / 3, stage 1 finished (position only)
I1021 16:21:10.932125 357716 timer.cc:87] Elapsed time: 652.04095 [seconds]
I1021 16:22:25.426796 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 2, Initial cost: 1.178338e+06, Final cost: 1.178242e+06, Termination: CONVERGENCE
I1021 16:22:25.426843 357716 global_mapper.cc:205] Global bundle adjustment iteration 2 / 3, stage 2 finished
I1021 16:22:25.426846 357716 timer.cc:87] Elapsed time: 726.53567 [seconds]
I1021 16:22:25.427783 357716 image_undistorter.cc:20] Undistorting images..
I1021 16:22:25.468055 357716 image_undistorter.cc:41] Image undistortion done
I1021 16:22:25.468926 357716 global_mapper.cc:216] Filtering tracks by reprojection ...
I1021 16:22:25.639928 357716 track_filter.cc:48] Filtered 2485 / 327598 tracks by reprojection error
I1021 16:23:05.647286 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 6, Initial cost: 1.143247e+06, Final cost: 1.141178e+06, Termination: CONVERGENCE
I1021 16:23:05.647328 357716 global_mapper.cc:193] Global bundle adjustment iteration 3 / 3, stage 1 finished (position only)
I1021 16:23:05.647330 357716 timer.cc:87] Elapsed time: 766.75616 [seconds]
I1021 16:24:19.418749 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 2, Initial cost: 1.141178e+06, Final cost: 1.141150e+06, Termination: CONVERGENCE
I1021 16:24:19.418782 357716 global_mapper.cc:205] Global bundle adjustment iteration 3 / 3, stage 2 finished
I1021 16:24:19.419760 357716 image_undistorter.cc:20] Undistorting images..
I1021 16:24:19.460049 357716 image_undistorter.cc:41] Image undistortion done
I1021 16:24:19.460683 357716 global_mapper.cc:216] Filtering tracks by reprojection ...
I1021 16:24:19.633450 357716 track_filter.cc:48] Filtered 15623 / 327598 tracks by reprojection error
I1021 16:24:20.423748 357716 image_undistorter.cc:20] Undistorting images..
I1021 16:24:20.463219 357716 image_undistorter.cc:41] Image undistortion done
I1021 16:24:20.466074 357716 global_mapper.cc:242] Filtering tracks by reprojection ...
I1021 16:24:20.638327 357716 track_filter.cc:48] Filtered 0 / 327598 tracks by reprojection error
I1021 16:24:20.741349 357716 track_filter.cc:123] Filtered 56397 / 327598 tracks by too small triangulation angle
I1021 16:24:20.741358 357716 timer.cc:87] Elapsed time: 841.85019 [seconds]
-------------------------------------
Running retriangulation ...
-------------------------------------
I1021 16:24:20.741371 357716 database_cache.cc:65] Loading cameras...
I1021 16:24:20.741396 357716 database_cache.cc:75]  1 in 0.000s
I1021 16:24:20.741400 357716 database_cache.cc:83] Loading matches...
I1021 16:24:20.837239 357716 database_cache.cc:89]  34484 in 0.096s
I1021 16:24:20.837253 357716 database_cache.cc:105] Loading images...
I1021 16:24:20.933269 357716 database_cache.cc:155]  2700 in 0.096s (connected 2700)
I1021 16:24:20.933501 357716 database_cache.cc:166] Building correspondence graph...
I1021 16:24:23.050115 357716 database_cache.cc:195]  in 2.117s (ignored 0)
 Triangulating image 2700 / 2700
 Global bundle adjustment iteration 5 / 5
I1021 16:30:30.869962 357716 timer.cc:87] Elapsed time: 370.12860 [seconds]
-------------------------------------
Running bundle adjustment ...
-------------------------------------
I1021 16:30:30.870013 357716 global_mapper.cc:273] Bundle adjustment start
I1021 16:41:45.673218 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 10, Initial cost: 2.027058e+06, Final cost: 1.941254e+06, Termination: CONVERGENCE
I1021 16:41:45.674377 357716 image_undistorter.cc:20] Undistorting images..
I1021 16:41:45.716841 357716 image_undistorter.cc:41] Image undistortion done
I1021 16:41:45.717613 357716 global_mapper.cc:281] Filtering tracks by reprojection ...
I1021 16:41:45.979436 357716 track_filter.cc:48] Filtered 22092 / 258265 tracks by reprojection error
I1021 16:49:59.572402 357716 bundle_adjustment.cc:48] Ceres Solver Report: Iterations: 7, Initial cost: 1.737611e+06, Final cost: 1.719190e+06, Termination: CONVERGENCE
I1021 16:49:59.572441 357716 timer.cc:87] Elapsed time: 1538.83107 [seconds]
I1021 16:50:01.087427 357716 image_undistorter.cc:20] Undistorting images..
I1021 16:50:01.129920 357716 image_undistorter.cc:41] Image undistortion done
I1021 16:50:01.132157 357716 global_mapper.cc:296] Filtering tracks by reprojection ...
I1021 16:50:01.430837 357716 track_filter.cc:48] Filtered 7599 / 258265 tracks by reprojection error
I1021 16:50:01.632903 357716 track_filter.cc:123] Filtered 4193 / 258265 tracks by too small triangulation angle
I1021 16:50:01.632915 357716 global_mapper.cc:86] Reconstruction done in 3244.9 seconds
I1021 16:50:02.271281 357716 colmap_io.cc:27] Extracting colors ...
I1021 16:50:07.692243 357716 global_mapper.cc:91] Export to COLMAP reconstruction done

I don't have the end of the bundle adjustment yet, I will update this comment with the end time when I get it. Is there anything in this that hints to why the runtime is higher than expected?

EDIT: Added the rest of the bundle adjustment output

[lpanaf]

Thanks for sharing your log. For the steps so far, the run time is about ~(80 + 10 + 760 + 840 + 370 < 2100s). This number is normal for a scene with ~2000 images. In this case, it is strange that MH_01 takes ~20k s. Maybe the number of constructed points is very large, which can be slow. You can potentially speed it up by capping the allowed number of tracks.

[carsoncramer]

Will capping the allowed number of tracks reduce the accuracy?

[lpanaf]

This would unfortunately be the case. However, when the cap is high enough (for example, 1000 x num_img), the difference would not be large and should still be accurate. You would need to decide whether you want to make this trade-off.

[carsoncramer]

I added the rest of the BA output above. For the last two runs that I have data on for colmap and glomap, the times look like this.

MH_01_easy:

• Colmap: 28507s
• Glomap: 27908s

MH_03_medium:

• Colmap: 22988s
• Glomap: 3431s

So for MH03, it seems like the glomap is performing as expected. I am not sure what's happening with MH01. I'll post again once I get some more results on other datasets, maybe MH01 is just an outlier.