Modern military and humanitarian missions require autonomous robots to intelligently reason and act in complex environments. For example, robots in disaster response scenarios must navigate cluttered spaces, locate occluded survivors, and generate a detailed map of the scene for first responders. This is a challenge for today’s robots in part because state-of-the-art robot map representations (e.g., grid maps and dense meshes) do not scale well to large, unstructured environments. Additionally, these maps often only encode structural information required for collision-free navigation, omitting the conceptual information contained in images that could be used for more advanced reasoning or human interpretation. 

The Autonomy al Fresco program -- a collaboration between the MIT Lincoln Laboratory and MIT Professors Sertac Karaman and Luca Carlone -- is addressing this conceptual reasoning capability gap by developing autonomy algorithms that leverage 3D scene graph mapping technology. A scene graph is a hierarchical graph where nodes represent concepts in the scene and edges represent the spatial and semantic links between concepts. This talk will overview our real-time scene graph mapping and planning methods designed for robots with resource-constrained compute. We will show recent experiments with a Boston Dynamics Spot quadruped robot to perform open language tasks such as object search.