SfM on-the-fly: A robust near real-time SfM for spatiotemporally disordered high-resolution imagery from multiple agents

Abstract

Over the last decades, Structure from Motion (SfM) has been a constant research hotspot in the fields of photogrammetry, computer vision, robotics etc., whereas, real-time performance is widely concerned, recently. In this project, we present a new online SfM, namely SfM on-the-fly, based on our previous on-the-fly SfMv1 which runs online SfM while image capturing. The new updated version (SfM on-the-fly) is now with three significant improvements to get better from what you capture:

First, online image matching is further boosted by employing the Hierarchical Navigable Small World (HNSW) graphs, which can result in more true positive overlapping image candidates are faster identified;

Second, a reasonable self-adaptive weighting strategy is employed for hierarchical local bundle adjustment, which is expected to improve the SfM results;

Finally, but most notably, we incorporate with the capability to deal with images from multiple agents, and multiple reconstructions are seamlessly merged into a complete model when commonly registered images appear.

The comprehensive experiments demonstrate that our on-the-fly SfMv2 is able to generate better results from what is captured, in particular, a more complete reconstruction is yielded in a more time efficient way, yet state-of-the-art SfM results are obtained.

Video

Experiments

All experiments are run on the machine with i9-12900K CPU and RTX3080 GPU.

1.Dataset

A total of 9 datasets of various scenarios are tested to evaluate our on-the-fly SfMv2.

Name	Image Num	Source	Category	Num of agents	Origin
XZL	226	Self-captured	Close-range	2	Download link(Wait upload)
YD	291	Self-captured	UAV	1
YX	349	Self-captured	Close-range	1
JYYL	356	Self-captured	Close-range	3
fr1_xyz	798	Public	Close-range	\	Tum
UniKirche	1455	Public	UAV & Close-range	\	UniKirche (Michelini and Mayer, 2020)
Alamo	2915	Public	Crowdsource	\	Alamo (Wilson and Snavely, 2014)
Caffe	287	Public	Close-range	3	Poiesi et al., 2017
Saranta	1035	Public	Close-range	3	Poiesi et al., 2017

2.Running Parameters

In our experiments, there are some free parameters needed to be selected:

Parameters	Value	Introduce
max_elements	10000	corresponding HNSW graph can store up to 10,000 data points
ef_construction	200	each element’s dynamic candidate list contains up to 200 elements during graph construction in HNSW
M	16	each data point in the graph is connected to at most 16 other points in HNSW
Top-30	30	For online sub-reconstruction, each newly fly-in image selects the Top-30 most similar images for subsequent image matching and geometric verification.
L_h	4	the maximum number of levels in the hierarchical association tree
Top-8	8	Top-8 most similar images in each level to participate in local bundle adjustment

3.Fast image retrieval based on HNSW

For fast image retrieval, we compare 3 methods: exhaustive retrieval (compares all already registered images and the newly fly-in image) vocabulary tree and HNSW used in on-the-fly SfMv2. The results are as follows:

Here are the results for the time efficiency of retrieval as the number of images changes, based on the Alamo dataset.

4.Adaptive weighting local bundle adjustment

Here is a comparison of the performance of bundle adjustment between the current system (v2), the previous system (v1), and Colmap.

5.Sub-reconstructions

Capability for multiple agents and sub-reconstructions of on-the-fly SfMv2(compared with on-the-fly SfMv1) are shown as follow:

Complete reconstruction by v2	Sub-reconstructions by v1

Reconstruction of XZL

Reconstruction of UniKirche

Reconstruction of YD

Reconstruction of JYYL

6.Evaluation on 3D Object Space

To evaluate the precision of the results in real 3D object space, here is a comparison between the results of SfM OntheFly and referenced 3D point cloud from terrestrial laser scanner: