How We Test Audio Systems
Anyone with even the least bit of interest in car audio knows that automakers have recently stepped up their audio game. If you're in the market, you have more car stereo options than ever before, with the brands available as factory options reading like a who's-who of high-end audio marques, including Mark Levinson, Bang & Olufsen, Bowers & Wilkins and Lexicon.
But a well-known name on a speaker grille doesn't guarantee great sound — or that you should pay a premium for a branded system. While automaker/audio-brand hookups continue to increase, awareness of better sound quality on the part of car shoppers has likewise grown. This rising audio tide has helped lift not only the quality of all stock stereo systems in general, but also the expectations of car buyers.
Listening With Purpose and for Pleasure
While we welcome a new era of quality OEM auto sound with open ears, at Edmunds we listen to systems with purpose as much as for pleasure. In the same way that we test vehicles to assess performance, comfort, styling, safety features and overall feel, we also evaluate them based on electronics amenities such as navigation, Bluetooth connectivity and, of course, audio.
As vehicles become more competitive and comparable in traditional key areas such as fuel economy and performance, in-cabin technology is growing increasingly important to many buyers — and as a way for manufacturers to set their vehicles apart.
But we don't just get into a car, crank it up and make a snap judgment on how a stereo system performs. We use techniques developed over decades of evaluating sound systems, and employ time-honored practices and procedures used by audio professionals, as well as those established by car audio competition sanctioning bodies such as the International Auto Sound Challenge Association (IASCA).
What We Listen for
While some audiophiles can go on endlessly about a system's sound characteristics in the same way that oenophiles can come up with all sorts of esoteric terms to describe wine, we use seven basic subjective sound-quality criteria to test an audio system, detailed below.
The range of human hearing is roughly 20 Hz to 20,000 Hz, although you feel more than hear the lowest frequencies, and people start to lose the ability to hear higher frequencies as they age. A good sound system should be able to not only re-create frequencies throughout this range, but do it in a balanced manner.
It's possible to objectively measure a stereo system's frequency response using an instrument called a real-time analyzer (RTA) and by playing test tones called pink noise, which have equal output throughout the audible frequency spectrum. This yields a visual representation of a system's frequency response "curve" on the RTA's display, so called because most systems have a response that has dips and peaks.
While a "flat" response is a theoretical ideal, this usually results in a sound that's, well, flat. Unlike pink noise, music is dynamic, with a rapidly changing range of frequencies. When we test a system we listen for tonal balance or how well it smoothly reproduces the frequency spectrum, without bass overpowering the midrange and highs and vice versa.
This is the ability of a system to reproduce music without distortion (unless it already exists in the listening material). While this is virtually impossible for any stereo system — and certainly most car audio systems — it's an ideal that system designers strive for and is readily apparent when a system falls short.
You've probably heard the fuzzing boom of bass from the neighborhood kid's Civic, but a good test for clarity is listening to cymbals and high-pitched female vocals. Cymbals will sound overly brassy and "sizzle" when distorted, whereas female vocals sound shrill.
This is a system's ability to re-create the lifelike sound of instruments. Acoustic instruments are good for assessing timbre since almost everyone knows what they sound like. When listening to a high-quality recording of an acoustic guitar, for example, it should have the warm, resonant quality that the instrument is known for — not sound like a low-resolution reproduction of that familiar sound. Vocals are also a good test of timbre since listeners know what a human voice sounds like.
This describes how faithful a system is in general to the original recording, and applies to instruments and vocals as well as a recording's ambience or the space in which the recording took place. While most modern recordings are done in a vacuum, with each instrument recorded individually and then mixed together to re-create the whole, older recordings, most live ones and some newer ones also capture the environment in which they were recorded.
Staging and imaging:
These two related concepts date back to the heyday of stereo and don't necessarily apply to modern recordings. The idea is that when you're listening to a stereo recording, the system should create the illusion of a stage in front of you on which the performance is occurring, and you also should be able to pinpoint the sonic images of the individual performers and/or instruments within the stage.
Ideally, a car should have a "sound stage" that is at least dash height, as wide or even wider than the car (in the best cars it extends beyond the A-pillars) and deeply layered so that it's almost three-dimensional. And you should be able to close your eyes — just not while driving — and pinpoint the location of the performers onstage.
Some systems reproduce the loudest parts of a piece of music with ease but have trouble with the softer passages, or vice versa. Think about when you're at an intimate live performance and you can hear the drummer lightly brush the cymbals just as well as you can feel the visceral thump of the kick drum, or the singer's whispered lyrics have the same emotional impact as when he or she belts out a tune. A great system can reproduce both extremes with equal ease, and dynamics refers to a system's ability to re-create this wide range for a vibrant, live-like sound.
Related to dynamics, this refers to a system's ability to retain detail despite the volume level; when a system is very linear, it has the same level of detail at low volume as it does at high volume, which is very difficult for most systems to achieve.
Together, these concepts relate to how "close" a system gets the listener to the actual performance and how the producer intended it to sound. Whether it's Miles Davis' trumpet, Jimmy Page's guitar, Norah Jones' voice, a Dr. Dre beat or the ambience of Carnegie Hall, how a stereo reproduces what went down in the studio or concert hall determines the difference between a system that's simply good and one that's exceptional.
For systems that play surround formats such as DVD-Audio, many of the same sound-quality concepts apply. But with surround formats we look for an enveloping, immersive sound.
How We Listen
When we critically evaluate an audio system, we listen to it under the following conditions:
- Sitting still with the engine at idle
- The volume set at moderately high levels
- All tone controls set to "flat" or at the center-detent position, including the fader and balance
- All surround or other signal processing turned off, if possible
While we also evaluate systems while driving and take into account road, wind and other noise and the overall quietness of the cabin, because of variables in road surfaces we test audio systems while sitting still to level the playing field. This also ensures that we give the listening task our full attention, without being distracted from the music while driving and vice versa. Likewise, when reviewing an audio system, we may comment on the presence and usefulness of the signal processing available, but for critical listening we switch it off.
What We Listen To
We have a library of roughly 50 music tracks we use to evaluate sound quality, as well as test tracks for staging, imaging, linearity and absence of noise. In addition to music tracks, we also use technical tracks on IASCA's Official Sound Quality Reference CD to test imaging, linearity and absence of noise.
Since the IASCA disc also has musical tracks as well as specific instructions on what to listen for when evaluating a system's sound quality, it's an excellent tool for novice and experienced listeners alike. As far as our personal library of test tracks, here are just a few we use and what we listen for with each one:
Bluesiana Triangle, Bluesiana Triangle (Windham Hill Jazz): I've listened to this 1990 disc on hundreds of systems over the years. Its well-recorded piano, horns and drums are an excellent test of timbre and tonal accuracy, while the jazzy jams reveal whether a system has good dynamics, tonal balance and sound-staging/imaging. In the track "Shoo Fly Don't Bother Me," a drum roll at 7:42 should seamlessly pan across the dash and extend beyond the vehicle's A-pillars, while the flute solo that starts at 2:20 should hover over the center of the dash and sound smooth and lifelike, not shrill.
Joan Armatrading, What's Inside (RCA): This disc starts out with a long, low bass note that causes all but the best systems to distort, and the vocal should be centered, with the instruments arrayed around it. In "Beyond the Blue," a drum break at 2:33 should sound accurate enough to hear the drum's "skin tone" and have palpable impact.
Red House Painters, Ocean Beach (4AD): I use the instrumental opening track on this disc, "Cabezon," to check for timbre, tonal accuracy, clarity and dynamics. The acoustic guitars have both a strong midbass resonance that wreaks havoc on weak midrange speakers and a high-pitched, trebly sound that causes many tweeters to spit and sputter. The track "San Geronimo" has intense upper bass and a thickly layered sound that most systems have a hard time accurately reproducing.
OutKast, Big Boi and Dre Present...OutKast (LaFace/Arista): To check for lack of distortion with powerful bass and whether a system's subwoofers can produce some serious thump, I cue up "Ain't no Thang" for its deep, rolling bass line.
iPod and MP3
Most people these days — including us — use a portable media player such as an iPod for day-to-day listening due to convenience. While we routinely test audio systems for iPod and portable-media compatibility and functionality — and recognize that the feature has become important to car buyers — we don't judge sound quality using "lossy" compression formats such as Apple's AAC and MP3.
When we critically evaluate a system we use CDs, DVD-Audio discs or uncompressed WAV files from tracks we've listened to for decades, using procedures that are equally time-tested. This is to assure you that when we give an opinion on a vehicle's stereo system, it's backed with the same expertise and experience that you've come to expect from Edmunds test-drives and model reviews and other information on our site.