additions… #52
…web design…
results from a tough javascript test…
I have no idea what the various browsers' script-engines are optimized for and what they're capable of. The regular benchmark tests are popular – I see them referred to on many sites, but optimizing a browser for those shouldn't be too hard. Thus, when I wanted to find out what browsers were capable of, I discarded those tests.
I needed something heavier, something that could challenge those browsers a bit. Needless to say my choice pushed some of the older browser versions I also tested but did not include in this document, well beyond their capability. Old/slow browsers kept on hanging, and I also had to turn off “hang warnings” in the latest Firefox to be able to test it.
I ran the JavaScript Progressive Raytracer – Full Render – two consecutive times in each browser – 240 passes in total, without reloading the page or running anything else in between. This to make the test a little harder and the separation between browsers' performance clearer.
I then shut down all browsers, restarted one by one and let each have the respective machine for itself, and repeated the entire test to make sure I had reasonably stable and correct run-time data for each browser. Such fresh-start tests were repeated more than 10 times for the best/fastest browsers, to give them optimal conditions. The others were to slow to bother with more than 2 test-repetitions in.
The initial test (not including IETester) took around 10 hours from start to finish across browser-land – despite being able to test simultaneously on 4 machines, and the result was a bit surprising to say the least.
Below are my results for 4 separate machines/Operating Systems, with bars showing 1px/second to finish the second Full Render run-through for this test in each browser on each machine. Obviously: the shorter bar the better.
Not wanting to compress or cut the bars, this page became “a bit wide.” The spread is enormous, with some browsers being pretty quick about it and some obviously working very, very, hard and really taking their time.
on window XP…
Opera 10alpha 
Safari 3.2.1 Firefox 3.0.7 Chrome 1.0.154.48 IE7
Opera and Chrome: rounded up average of 10 fresh-start test runs – ± 0.4 sec.
IETester v0.3:
IE7 (Default) IE5.5 IE6
IE7 IE8rc1
Designers/developers may want to know if a test-tool like IETester is somewhat representative for the real
browsers. After a few runs I have to conclude that for javascript in IETester v0.3 it is
– just don't run such heavy cases in its IE5.5 
windows XP pro v.2002 SP3 - Acer Verison L460 - 2.2GHz dual CPU - 2GB RAM.
on windows Vista…
Opera 10alpha Safari 4 Public Beta
(528.16) Firefox 3.1.b3 Chrome 1.0.154.48 IE8
Opera and Chrome: rounded up average of 10 fresh-start test runs – ± 0.6 sec.
Safari 4 Public Beta (528.16): rounded up average of 10 fresh-start test runs (those without overshoot) – ± 4.5 sec.
IETester v0.3:
IE6 IE7 IE8rc1
IETester results are just a bonus.
windows Vista (32b) - Acer Verison L460 - 2.2GHz dual CPU - 2GB RAM.
on Ubuntu…
Opera 9.64 Konqueror 4.0.3 Firefox 3.0.7 Epiphany 2.22.2 Galeon 2.0.4
Opera and Konqueror: rounded up average of 10 fresh-start test runs – ± 0.4 sec.
Linux Ubuntu - Acer Verison L460 - 2.2GHz dual CPU - 2GB RAM.
on win2K…
Opera 9.64 Seamonkey 1.1.14 Firefox 3.0.7 Opera 8.0 IE6
Opera: rounded up average of 10 fresh-start test runs – ± 0.8 sec.
windows 2000 SP4 - Opteron - 2.0GHz dual CPU - 1GB RAM.
Note: Parhelia screencard slows down rendering on the Opteron slightly compared with on the other machines.
interpretation…
Those bars speak for themselves, and anyone can run the same test on their own machines and figure out how their preferred browser is doing. Life is short, and some of those browsers sure could render a little faster.
the list – good to bad:
- Chrome is really fast, and handles this kind of test quite well. A couple of apparent hangs/pauses
noticed during repeated test runs on Vista, was ommited, and they were not serious enough to pull this engine down from
the very top anyway.
— A definite Pass!
- Opera 9.64/10alpha is fast everywhere, but not necessarily the fastest anywhere (although 10alpha
on XP and Vista is impressingly fast). Opera runs and renders very stable and smooth on all Operating Systems
I have tested on, which is as expected since Opera is optimized for all Operating Systems.
— A definite Pass! - Konqueror 4.0.3 on Ubuntu is definitely optimized for this kind of test and the Operating System. It
renders all stages very smoothly, and gets in the lead both speed-wise and stability-wise.
— A definite Pass!
(Note that the older 3.5.xx versions hang on this test, so no use testing those.) - Safari 4pb is faster than all on windows, and apparently more than twice as fast as
Safari 3. However, Safari 4pb skips rendering for all but a couple of passes, and
since it isn't actually doing its job I'm not sure how to interpret its speed.
Safari 4pb also needs two clicks to start on the second round, and apart from showing quite a speed-spread it actually went beyond the last – the 240th – pass a number of times during repeated tests. Such instability, trickery and what looks like plain failure, is confusing.
— Very confused
- IE7 is doing quite well.
— Acceptable - IE6 isn't doing too bad speed-wise for such an old browser, but it can't really handle
the test.
— Acceptable - Firefox 3 is extremely slow everywhere – it certainly isn't optimized for the task at hand.
Epiphany is pretty much the same as Firefox, which isn't surprising.
— Complete Failure!
- Seamonkey 1.1.14 was tested as an older Gecko. Extremely slow. The same goes for all Gecko-based
browsers I've tested – those that didn't hang.
— Complete Failure! - Galeon appears to be slowest of them all when we correct for machine/OS speed. This wasn't all that
surprising either since this browser is “based on Mozilla.”
— Complete Failure! - IE8 (version: 8.0.6001.18702) is extremely slow, and the end-result – the image
– looks horribly jagged and unfinished.
— Complete Failure! - The IETester v0.3 reveals results similar to those seen in the real IE versions, an
indication that developers can do much of their script-performance testing in IETester.
Note that IETester v0.3 is not an entirely stable piece of software.
conclusion…
Apart from obviously having wasted too much time waiting for some of those browsers to finish their runs, I'm quite happy with the result. Only 3 browsers get a clear Pass, and which ones differs a bit from what's seen for regular benchmark tests.
Not sure if this JavaScript Progressive Raytracer will ever be recognized as a benchmark test, but it sure has given those browsers something to chew on.
My own preference, Opera, is doing ok everywhere, which is more or less as expected from everyday use. Opera will get a new script-engine in not too long anyway – in version 10.5 I think, so it can only get better.
I conclude that most javascript engines, not surprisingly, have become more efficient over time. The exception is Gecko's engine, which is lightyears behind the best and seems to be improving very slowly.
Safari 4pb is (maybe intentional) not suited for such cases. This is a beta version, so it'll be interesting to see what the final version will be capable of.
IE8 (released: 19.mar.2009) simply can't handle this test no matter how I interpret the result. Don't think it'll be too much of a problem for more regular scripts in use across the web though.
sincerely 
Hageland 13.mar.2009
14.mar.2009 - added IETester on XP.
15.mar.2009 - repeated test runs for fastest browsers. Revised relevant times and text.
17.mar.2009 - added IETester on Vista.
19.mar.2009 - tested IE8 final release.
20.mar.2009 - repeated test runs for fastest browsers on Vista. Revised relevant times and text.
12.apr.2009 - added note about Konqueror 3.5.xx versions under "interpretation".
last rev: 12.apr.2009
additions…
