The results of the battery of tests were announced on Tuesday.
The Institute said it "doesn't typically crash-test sports cars, as they make up a small share of the consumer market."
But based on consumer requests, the muscle cars with optional V8 engines were evaluated because "they are big sellers in their class."
"The Mustang comes closest to earning Top Safety Pick, while the Camaro falls short in one category and lacks an available front-crash prevention system," the Institute said in a statement. "The Challenger is most in need of improvement."
The Camaro, Challenger and Mustang earn "good" ratings for occupant protection in a moderate overlap front crash, as well as a side impact.
In the Institute's small overlap front test, the Camaro earns a "good" rating, the Mustang earns "acceptable" and the Challenger is rated "marginal."
The test replicates what happens when a vehicle hits a tree or a pole or clips another vehicle that has crossed the center line.
"Given that sports cars have high crash rates, it's especially important that they offer the best occupant protection possible in a crash," said Adrian Lund, Institute president.
The National Highway Traffic Safety Administration said the 2016 Challenger and Mustang earned the highest five-star overall safety rating. The 2016 Camaro has not been rated by NHTSA.
Edmunds says: Safety-minded muscle-car shoppers will want to consider these new tests from the Insurance Institute for Highway Safety as they go through the purchase process.