Learning From Simulation

Sim4CV – A Photo-Realistic Simulator for Computer Vision Applications

M. Mueller, V. Casser, J. Lahoud, N. Smith and B. Ghanem: “Sim4CV: A Photo-Realistic Simulator for Computer Vision.” International Journal of Computer Vision (IJCV), 2018. (full text)

Related publications:

G. Li and M. Mueller, V. Casser, N. Smith, D. Michels, B. Ghanem: “OIL: Observational Imitation Learning.” Robotics: Science and Systems (RSS’19). (full text)

V. Casser and M. Mueller, N. Smith, D. Michels and B. Ghanem: “Teaching UAVs to Race: End-to-End Regression of Agile Controls in Simulation.” 2nd International Workshop on Computer Vision for UAVs, ECCV’18, 2018. Best paper award. (full text)

M. Mueller, G. Li, V. Casser, N. Smith, D. Michels, B. Ghanem: “Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-based UAV Racing.” 3rd International Workshop on Computer Vision for UAVs, CVPR’19. (full text)

We present a photo-realistic training and evaluation simulator (Sim4CV) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a generic deep neural network (DNN) interface for real-time evaluation. It generates synthetic photo-realistic datasets with automatic groundtruth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool.


Example applications

Supplementary material

Data required for a variety of applications, all of which Sim4CV can provide.

Video showcasing autonomous driving and UAV tracking within Sim4CV.

Sim4CV is open-source and can be downloaded below. We invite anyone interested to contribute to its future development!

Download Sim4CV