• Evaluation tool: BOPtoolkit
• For 2D object detection: precision, recall, IoU, mAP
• For 6D pose estimation: ADD, ADD, COU, VSD

The first evaluation step consists of computing 2D bounding box detection metrics with the mean Average Precision (mAP) based on the Intersection over Union (IOU) scores, with a threshold of 0.5. In 6D pose estimation the most widely used metric is Average Distance to the model point (ADD) error, (eADD). If the model M has indistinguishable views, the error is calculated as the Average Distance to the Closest model point (ADI). Each estimated pose is considered correct if e < θ = k * d where k is a constant and d is the object diameter. We considered different costant k equal to 0.1, 0.2 and 0.3. In addition, we also computed Complement over Union (CoU) error recalls. This is a metric based on masks: it creates masks of the given rendered object (given the cad model and the estimated pose). Then it compares predicted and ground truth masks. CoU was computed with three threshold: 0.3, 0.5, 0.7.

• YOLOv4 + AAE
• EfficientPose
• Yolo6D

Despite our dataset being mostly composed of symmetrical objects, we achieved better results when comparing with similar state-of-the-art methodologies. The following Table for the hotstab and the jug compared three different methods in terms of ADD, ADI and mAP. In particular, we chose for comparison YOLO-6D [1], which is a feature-based approach that uses Perspective’n’Point (PnP) algorithm, and EfficientPose [2], a full-frame, single-shot method which achieves one of the higher results on LineMod Dataset. YOLO-6D and EfficientPose were trained with different input and batch sizes, different learning rates and Adam momentum as optimizer. On the other hand, for EfficientPose the best performance is achieved with a batch size equal to 1, a learning rate of 0.0001 and 500 epochs. To ensure comparability with the other two methods, we opted for lighter versions by selecting φ equal to 0 as scaling hyperparameter. Despite the higher results on LineMod Dataset, EfficientPose achieves very low performance scores. On the other hand, YOLO-6D achieves higher performances with respect to EfficientPose, but it does not reach our pipeline percentage scores.