Android Browser Faster Than iPhone In Flawed StudyAndroid Browser Faster Than iPhone In Flawed Study

Blaze Software performed a huge study on browser performance, loading 1,000 different Web sites in the Android and iPhone browsers and taking 45,000 measurements. It says that the results conclusively prove that the Android browser is 52% faster than the Safari browser in the iPhone.

"We were very surprised by the results," Guy Podjarny, Blaze CTO and co-founder, said in a statement. "We assumed that it would be a closer race and that the latest JavaScript speed improvements would have a more material impact on performance. The fact that Android beat iPhone by such a large margin was not expected."

In order to perform the study, Blaze created its own custom applications -- one for the iPhone and one for Android -- to measure page load times on the smartphones. The apps ran on the actual devices, loaded a page on demand, and measured how long it took.

And therein is the problem with this study -- it didn't use the actual Web browsers. It used the browser's rendering engines, but through this custom application. That significantly undercuts the results.

As Gartner's Michael Gartenberg explained to me via Twitter, "For one thing, Mobile Safari was not used, so the new JavaScript engine wasn't being measured. Safari is how users browse the Web. More importantly, things like 52% faster are misleading when dealing with times of a second or less. In real world that's margin of error."

In a number of anecdotal tests I ran this morning, I couldn't see any appreciable difference in page load times between the iPhone and several Android devices (using the same Wi-Fi network). The Android browser loaded some Web pages faster than the iPhone and vice versa.

Gartenberg continued, "Real world mobile browsing is affected by connection signal, page load, ad being served, etc. Consumers do not make phone purchases based on stopwatches. If it's not a 'just noticeable difference' it doesn't matter."

In all the years I've spent testing phone performance, the number of variables that play a role here are numerous and will have a more material impact on page load times than the rendering engine. As Gartenberg notes, connection signal is important, but so is location, distance to the cell tower, cell site congestion, network congestion, and so on.

Sure, in a controlled lab using customized software, Blaze may have been able to get the Android browser's rendering engine to outperform the iPhone's rendering engine. What's seen in the lab, however, is not always how things work out in the real world.

UPDATE:

Apple has issued a response regarding its test. Apple said, "… they only tested their own proprietary app, which uses an embedded Web viewer that doesn’t actually take advantage of Safari’s Web performance optimizations. I'’s not just JavaScript, though. Safari on iOS 4.3 also has multi-threaded, asynchronous page-loading and some HTML5 caching."

Blaze then responded to Apple, "To the extent of our knowledge, this is the first time Apple has openly made such a statement. Given the information that various optimizations are not included in the embedded browser, it’s quite possible the iPhone page loads could be faster. We stand behind the statement that Android’s embedded browser is faster than iPhone's.

We hope Apple will help us enable those optimizations and repeat the measurement. Until then, for all we know the missing optimizations may not make a big impact."

Until such a time when Blaze can come up with a new test that takes advantage of the tools Apple says are embedded in the Safari browser, it stands by its results.