AWS Machine Learning Blog· Khaled Thabit·· 6 h ago
Fix evaluation bias to unlock better models on Amazon Bedrock
Fix evaluation bias to unlock better models on Amazon Bedrock
AI summary
Property Finder found that 96.8% of their vision-AI evaluation labels were inherited from the incumbent model, hiding its true accuracy. See how they built unbiased ground truth, applied evolutionary prompt optimization, and selected a more accurate, cost-effective model on Amazon Bedrock, cutting cost by 86% across 11 image-quality attributes.
Selection record
Threshold 60Official, first-handFirst 42Second 42
Not admittedSum of both 84 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：详述Bedrock视觉模型评测偏差与提示优化
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: AWS Machine Learning Blog · aws.amazon.com