Machine Learning: Image Segmentation

Protostar Labs - Oct 5, 2020

Complete solution for neural network dataset generation, automatic and random camera and object movement, segmented images and python scripts for export to COCO dataset format.

  • Supported Platforms
  • Supported Engine Versions
  • Download Type
    Asset Pack
    This product contains an assortment of Unreal Engine assets which can be imported into a pre-existing project of your choice.

This project contains several blueprints and python scripts designed to streamline capturing of images in UE4 with instance segmented images, exporting data from the scene and processing it into COCO dataset format in JSON file, to be ready for artificial neural network training. 

Everything is built in Blueprints, no C++ necessary. There are several blueprints that allow you to spawn desired static meshes, add tags to them and have the camera and objects move randomly and automatically while capturing the image of the scene and the segmented image pair. Included python scripts use OpenCV 2 python library to detect polygons (and bounding boxes) in the masked segmented images which are then exported to COCO dataset format (JSON file). 

Overview on YouTube

Technical Details


  •  Export of images and instance segmented images of the scene
  •  Automatic camera and actor movement
  •  Automatic static mesh segmentation by adding an actor tag to them
  • Export of scene data, colors of the objects in the masked segmented images linked with their actor tag in a CSV file format
  • Python scripts to extract the polygons, bounding boxes and actor tags to export them as a COCO dataset format JSON file, ready for NN training

Number of Blueprints: 8

Number of Levels : 1 showcase

Input: Keyboard and Mouse for preview, otherwise none necessary (automatic execution)

Network Replicated: No

Supported Development Platforms:

Windows: Yes

Mac: No, untested

Documentation: Documentation on GitHub

Important/Additional Notes:

  • Python scripts require OpenCV 2 and numpy python libraries, tested with Python version 3.7.7