If you’re like me, sometimes driving home from work, you glance in your car’s side mirror and are confronted with the phrase “Objects in Mirror are Closer than They Appear.” This safety warning appears because while it’s useful to have convex mirrors that give drivers a full view of what’s behind them, it’s downright dangerous to use such mirrors to estimate how close following traffic may be.
There’s a similar lesson to be learned with respect to measuring broadband speeds and ISP performance. If you follow this topic, you’ve probably heard of comScore, an Internet marketing research company that provides data to companies doing business online. One of the many things they test via online tools are the speeds that a user typically experiences over his or her Internet connection to surf the Web.
There’s nothing inherently wrong about measuring these speeds, and indeed, many cable companies make online speed tests, or links to such tests, available to their subscribers. Anyone who runs a blog or website or who surfs the Internet knows that it’s difficult to measure lots of things (such as site traffic), but we measure the best we can and try to be consistent about how we conduct those measurements, so that we can track changes over time. However, a problem arises when data used to measure one set of consumer experiences is used to support entirely different claims – in the case of user-generated speed tests, that such data demonstrates some sort of significant “gap” between the actual performance of ISP networks and advertised maximum speeds.
We’ve been pretty clear in noting our concern that the FCC’s National Broadband Plan mixes apples and oranges by relying on comScore data to draw conclusions about ISP performance (to note one example, the Plan asserts in Section 3.3 that “actual performance is approximately 45% of the ‘up to’ advertised speed,” citing the comScore database). We further detailed our concerns in a recent letter to the Commission:
. . . it is not entirely accurate to refer to a gap between “advertised” speed and “actual” speed as the Plan does. As NCTA has explained previously, because broadband speed is affected by so many factors beyond the control of the broadband provider, cable operators and other providers make clear in their advertising and other consumer materials that the speeds experienced by consumers will vary from the maximum available speed.
Moreover, in order to get a better read on how online speed tests might lead to faulty conclusions about ISP performance, one of NCTA’s member companies paid for the underlying comScore data and we commissioned NetForecast to prepare an analysis. You can read their report here which identifies certain data-gathering errors and data-interpretation errors. Ars Technica highlighted their findings as well. But perhaps the most important part of the review is focusing attention on what the data collected does — and does not — measure.
On this point, it’s helpful to read this blog post from NetForecast which also includes a response from comScore. Essentially, this response boils down to, “We measured what we intended to measure,” but it also includes a number of phrases that indicate agreement with a number of points in the NetForecast critique.
- “This is true and reflects the specific design of our study.”
- “…there are certainly device-level variables that can degrade speed.”
- “…this is a valid concern…”
The key issue is that comScore says “The goal of the test was to measure actual end user experience…” However, measuring “user experience” is not the same as measuring the network performance offered by an ISP. If an automobile manufacturer claims a new car is able to hit speeds of 120 mph, but you drive in traffic, in the rain or on country roads, then your “user experience” will be different.
At least two problems emerge when trying to shoehorn user experience data into conclusions about ISP performance.
First, we all know that there are a number of factors affecting performance outside the control of ISPs. Sluggish computer performance, home networking glitches, and distance travelled across the Internet can certainly affect speed that a user experiences, but most reasonable people would agree that such shortcomings should not be used to gauge ISP performance.
Second, methodological errors can affect results. For example, results can be skewed if testing fails to accurately designate the tier of service subscribed to by a user. For example, a subscriber purchasing 3Mbps monthly service but enjoying temporary “Powerboost” speeds of 6 Mbps might incorrectly be assumed to subscribe to a higher tier of service. Based on that assumption, one might conclude that the received speed is significantly less than the advertised speed for that tier of service when, in fact, that is not the case — there is no “gap” between what the subscriber generally receives and the advertised speed for that tier.
The point here is not to pick on comScore or the FCC. We fully agree that consumers need some way to assess performance, so that they can choose from among providers in the marketplace. The cable industry has indicated many times that we are eager to work with the FCC and other interested parties to develop such measurement tools for “on network” performance that provide consumers with meaningful data.
But, at the same time, we also ought to insist that speed data measuring “user experience” across the Internet be accompanied by appropriate disclaimers and educational qualifiers, so that consumers can know exactly what is being measured. And ISPs certainly should not be held responsible for allegedly failing to meet the maximum speeds advertised for particular service tiers because of factors outside their control.
We may not reach perfection, but by working together with industry, consumer groups, and policymakers, the cable industry is confident that we can improve on the accuracy, clarity, and transparency of speed data used today, and achieve a true “measure” of success.
