mAP Drop
#1
by
mhyatt000
- opened
I tried to reproduce the results mentioned on this model card. Seems like my results do not match the claimed mAP in the model card. I cannot figure out how to get the correct numbers, can you help me find my mistake?
- Claimed mAP: 37.6
- Recieved mAP: 33.2
Here are the details for my validation:
- I instantiate pre-trained model with
transformers.pipeline()and use COCO API to calculate AP from detection bboxes. - Evaluation was performed on macOS CPU.
- Dataset was downloaded from cocodataset.org
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.332
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.530
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.340
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.115
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.352
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.538
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.282
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.411
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.423
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.161
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.454
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.661