Why does changing a car’s oil in take an hour in California? In Texas, this is a 15 minute process, in and out. I’ve never been a fan of wasting time, so while waiting on my car this morning, I started researching one of the most exciting test opportunities we have in the upcoming Gecko 1.9.2 platform: Graphics Hardware Acceleration.

Most computers that people use today have some amount of GPU acceleration under the hood. So, this should result in improved rendering performance, which we will need as the web becomes more graphically interesting.

Our current test suites do not have any notion of hardware accelerated versus software rendering. In fact, on Linux we run our test suites in an X virtual frame buffer, which never touches any graphics hardware. And the graphics hardware on our boxes is pretty bottom of the barrel. I’d bet that the chip rendering this text for you is more advanced.

So…how on earth are we going to test this? I’m mostly concerned about functional testing, but I recognize that whatever solution we come up with here might also be needed to aid the graphics team with their unit tests. My thoughts and research are just beginning on this effort, so this isn’t a complete plan, this is a preview to my plan and a sincere request for help.

First Steps

We need to determine what the top video cards/drivers are for each of our top three platforms: Windows, Linux, and Mac. We can then grab a small handful of machines that have those configurations. I’m thinking maybe just 3-6 machines here, nothing extravagant. I think we would still want to target middle-of-the-road hardware, not top of the line gaming systems. We want to test what the average web user will be running.

At the same time, we need to figure out what graphics benchmarks exist and whether or not these can be leveraged or re-implemented into a graphics benchmark for the web. Since I’m not interested in how we perform writing instructions into the XPCOM components that make up the underbelly of the graphics support, I’m thinking we should create these benchmarks/tests in JavaScript and canvas. We ought to use this new graphics support the way that real developers will be using it once it is released.

Further Thoughts

What we might use here would be a reftest-like framework that would be augmented to compare the test that is hardware accelerated with a reference image that is software rendered. I imagine there are pitfalls this way–with hardware support, the test image may be more detailed than the reference, and we would need a mechanism to detect this condition to flag it as a “pass” rather than a “failure”. I definitely need to think about this some more.

Vision

There are two basic parts here. First, we need a simple test infrastructure that can test our rendering in both accelerated and software modes. We need a set of representative machines to run these tests on and we need those machines automated into our normal build and test reporting structure.

However, our tiny sample of machines will not give us the coverage we really need. So, the other reason this test infrastructure must be simple is that I want to invite all interested users to run it for us. It must be something that the users can download and run on their nightly builds. Once the test finishes, the users should be able to send us the results data from that run with a click of a button. The anonymized results will help us understand how well our code is handling device/driver/OS combinations.

I wanted to start a conversation here, and throw a line out into the wide world of the net to see who might be interested in helping us figure it out. Thoughts and ideas are most welcome. I’ll keep y’all up to date as my plans progress.

Today, we are releasing Firefox 3.5 this morning, so we’re in the office really early.  Got to watch the sun rise over downtown Mountain View.

It’s going to be a fun day!

So,after a much more log parsing and number crunching than I had anticipated, I have some results from the Greener Tinderbox effort.  For those that don’t know, I’ve been running all the TUnit automated tests over and over on a set of knock-off Tinderboxen trying to hammer out which failures we are seeing, and which we aren’t.  My hope is keep doing this long term, debugging a failure or two every couple of weeks until we get the tinderboxes back to a  sane state.  But to get started, first I had to collect a bunch of data, parse the data, and determine what the data means.  That takes a while, but now, round 1 is done.

= Executive Summary =
The boxes displayed more noise than normal tinderboxes because of a couple of configuration issues described down in the “Quality of Results” section.  For the most part, many of the tests that failed have been designated as “intermittent” at one time or another or on different products (seamonkey, fennec, etc). You can grab the results in an Open Office spreadsheet:

== Next Steps ==
Looking over the top intermittent failures that were replicated from tinderbox, and comparing that to Ted’s new topfails reports, I’ve decided to attack three tests for debugging:
* browser chrome fuel test: browser_Browser.js (bug 458688)
* chrome test test_wheeltransaction.xul (bug 484853)
* mochitest jquery fx module tests (bug 484994)

My plan is to put in debugging information on these tests and run these three harnesses in tight loops for the next 24 hours to see if I can get the failures to replicate and figure out what is happening there.

On Friday, I want to update to a new changeset and re-run all the tests on the machines for another five days. And after getting that data, I’ll need to give the machines back, but I plan to keep the VMs.  Rolling up results should be quite a bit quicker next time as I won’t have any scripts to write for log crunching.

== The VM versus Machine Question ==
There is no doubt that the high end machines I’m using for this blow the VMs out of the water in terms of performance.  However, in terms of intermittent failures with tests, I don’t see any quantitative difference between VMs and machines.  For the second round, I hope to finally have my Linux machine ready and can compare it with the Linux VM.

= Details Details Details =
== Results ==
The pretty results are here (open office.org spreadsheet).  The raw logs are also available in that directory.  And I’ll be checking in all my log parsing/number crunching/test running code into http://hg.mozilla.org/qa/tunit-analysis once that repo is available.

== The Math ==
I was going to crunch the numbers, but once I started them (you can see some on the “Total” sheet in the spread sheet, the numbers didn’t look quite right to me, so I’m going to repost my “Counting Failures” blog post but with legible equations this time to make sure I’m going about this correctly.

== Quality of Results ==
There are two problems with the results.
1. The accessibility tests on Linux did not run, because I naively thought that –enable-accessible was the default on Linux as it is on windows.  So all the failures on mochitest-a11y on Linux should be disregarded.
2. On Windows, I set the color depth policy as specified in the reference platform document; however, I could not change the resolution or color depth on those machines.  I assumed that the policy would somehow take care of it (should have known better).  So a bunch of the color testing reftests failed on the windows machine.

Otherwise, the results seem to be good.  I have queried every failure (except the reftest failures as noted — many of them seem to be machine configuration issues) against bugzilla to find out if each failure is noise from the box or if it is a failure that has been reported as intermittent on Tinderbox, and I’ve noted those bugs.  If the bug field and it is blank then there is no mention of this failure in Bugzilla.

So, I’ve been trying to get some test results together over the last few weeks to try to determine why our Tinderboxes keep going orange.  To do that, I’ve been running the TUnit test harnesses constantly on the same change set, amassing quite a bit of data.  One of the things I’d like to be able to do here would be to determine why we get these random failures and how we can minimize them.  I’ve almost acquired my first set of full results, and to complement the investigation into each failure, I’d like to have a set of metrics I could apply to the entire run as a whole so that I could measure a couple of things.

• Measure the reliability of each test harness in the TUnit suite (i.e. XPCSHell, mochitest, mochitest-chrome, mochitest-browser, mochitest-a11y, Reftest, Crashtest).
• Measure how close my machines match that of Tinderbox in terms of random failures they encounter
• Measure how random each failure is.

So, to get each of these, I’ve come up with an idea on how to calculate them.  However, I’m no math whiz, and that’s why I turn to you to get your opinion before I start crunching numbers.  Here are my thoughts on how to measure this stuff.  I apologize in advance for the copies straight from my notes, I wanted to get feedback before spending time on making it extremely pretty.  I’ll do MathML next time, especially if these prove to be illegible.

Reliability of each Harness

To do this, I’m basing my analysis off the standard Mean Time Between Failure metric.  Here, it would be more aptly called mean tests per failure.  In other words, how many tests do we run before we hit a failure on a given test harness.

Mean Tests Per Failure from my notes

Here, my idea is to sum the delta of the number of tests ran in the harness and the number of failures in those test runs, then divide that by the total number of failures from all runs on this harness.  This does seem to make sense on the surface: if there are no failures your number of tests run per failure encountered approaches infinity.  If every test fails, the number of tests run per failure encountered approaches zero. One thing that bothers me about this metric is the amount of noise that will be introduced by upgrading to a new changeset – because with a new changeset, we’ll also have a new set of tests in the harness we are testing against.

How close are we to Tinderbox

To do this, I’m going to take a discreet view of the total known failures, both those that are known to be random on tinderbox, and those that I’m encountering:

Percent difference the test machines are from the true Tunit machines

This measurement is across all harnesses.  Essentially, the number of items in the set of observed failures on my machines (G) that are not in the set of known random failures on Tinderbox (T), divided by the number of the items in the set of known failures on Tinderbox (T).  This will give me a percentage value that indicates how closely we match Tinderbox.  If all the tests that fail on my machines are the same tests that are known to fail on tinderbox, then we have a 0% difference.  However if more tests fail on my machines than on tinderbox (which is what initial analysis is showing) then we will have a non-zero percentage difference.  One caveat here is if less tests fail on my machines (i.e. not all Tinderbox failures are displayed) then I would still have a 0% difference.  Since I’m only trying to determine “how different are my machines acting from Tinderbox’s behavior,” I think this might be OK.  Thoughts?

Measuring Randomness of a Failure

This is probably the most straightforward measurement that I am considering.

Measurement of how random a given failure is

This is simply the number of times a particular failure occurred (Cf), divided by the number of times we ran that harness (r).  This gives a simple percentage of how likely (l) it is that we will observe the failure in question.  The reason this calculation can be this simple is that during any given run A we can only hit any given failure X once.

Thoughts?

So, those are my first thoughts on how to measure this data.  I have a lot more questions than I have answers:

• I’ve been reading a lot, but it’s been a long time since I’ve done this kind of math — are these correct?
• Are there other metrics that would make sense for this issue? I don’t want to do a bunch of metrics for the sake of numbers, but I definitely want to have a good way to compare the data I get from this effort between runs on different changesets in order to determine if we are making the TUnit test suite more or less reliable.
• Ideas on how to measure the error in this process?

Thanks for the help!

In Austin, March means flowers, the beginning of the best season (the not-quite-so-hot-and-not-rainy-season, to be prescise) of the year and SXSW music.   I really miss SXSW; the way the town erupts into a giant party, the way that there is music in even MORE places than normal (it’s already everywhere in Austin–I’ve seen live bands in grocery stores!)…I wish I were heading home for it.

Alas.  But since I am stuck on a wind whipped coast with breakers crashing together in all kinds of crazy directions and rain coming down sideways, I thought I’d share one of my secrets to keeping warm.  Since much of the Mozilla world is hammering away on Firefox Beta 3 in cold climates, I figured this quick little dish would be welcome.  I found it in the The Islands in the Sun Cookbook: Culinary Treasures of the Italian Isles by Marlena Spieler, which is a great book for a bunch of reasons, not the least of them being this recipe.

Pasta Mista in Brodo alla Zenzaro

• 3 cups(715 ml) chicken broth (I use Pacific Natural Organic Veggie broth)
• 1 1/2 cups (350ml) diced tomatoes (canned work great)
• 1/2 cup (120 ml) tomato juice
• 1/2 to 1 tsp (1-5ml) finely chopped fresh ginger (you have to have fresh ginger, it’s the only hard part, I swear)
• 3 cloves garlic (chopped, minced, or pressed–as fine as you can make it, essentially)
• Some green onions
• Black pepper
• Sea Salt
• Dash of Tabasco sauce (I use several)
• 8 ounces (255g) of ravioli or tortellini (I use one package of the frozen stuff.  I find that sharper flavors work well like portabello mushroom and cheese.  This is actually kind of important, green or super strong flavors like spinach and Asiago or Gorgonzola really don’t work well with this dish).

1. Combine broth, tomatoes (dump ‘em in, juice and all), tomato juice, garlic, ginger, salt, and pepper into a big pot.  Bring it to a boil.  I usually let it come to a slow boil over medium heat, but if you’re hungry you can make it go faster.  Cook a minute or two then remove from heat (or just simmer it). Stir in your Tabasco sauce.
2. Cook up your pasta in another pot, drain that.
3. Put the pasta in a bowl, put the broth over the pasta, and drizzle green onions (and olive oil if you want, I never use it but it’s in the original recipe) over that.

Makes one awesome, warm, spicy dish.  I’ve been known to drink the tomato broth by itself–it’s just that good, and it warms you from the inside out.  I love this dish because you only have to chop the ginger (and the garlic if you opt to do that, I have a garlic press so it hardly counts as chopping).  You can prepare this in about 10 minutes and it will be ready to eat in about 10 more.  I like to let it cook longer to get the flavors to meld, but that still gets the job done in under 30.

When you’ve got a deadline to meet and a cold wind whipping at your door, this is exactly what the doctor ordered.  Now, back to coding.  Bon Appetit!

I was hoping this would pass me by.  But I do need to blog more, and while I haven’t had a bunch of time, I’ve enjoyed the “seven things” posts I’ve read on Planet.  I was tagged by Simon and Jane.

The rules.

1. Link back to your original tagger and list the rules in your post.
2. Share seven facts about yourself.
3. Tag some (none? ) people by leaving names and links to their blogs.
4. Let them know they’ve been tagged

The Seven Things.

1. I’ve been a writer all my life: my first word was “book”, I wrote my first story when I was five.  Now, I have four novels finished, hundreds of stories and a fifth novel in progress.  None of them are published because for many years I despised editing.  It was much more fun to write.  In 2007, I finally changed that perspective when a friend of mine published her first book.
2. I grew up in the swamps on the Texas and Louisiana border.  When I was fourteen, a buddy of mine and I captured an alligator using our bicycles and a rope. We took it to his house.  He was convinced his dad would let him keep it as a pet.  Needless to say, we were both grounded.
3. When I was very young, my best friend and I created an imaginary high fantasy world to play in.  We continued that story line and played in that world until we were about ten years old.  When we ended it, we held an imaginary hero’s funeral for the characters who had to die to save their world and so that we would be released from it.
4. When a land developer attempted to bulldoze the small bit of forested land across the street from my childhood home, I lead a loosely organized resistance of children to stop him.  We’d never read the Monkey Wrench Gang; we had no idea what we were doing.  We just knew we had to save Twin Hills (that was our name for it).  Ten years later, I met that very land developer face to face, standing upon the ruins of our forest.
5. I once lived outdoors for so long that when I first went back indoors (it was a restaurant, as I recall) it felt claustrophobic and unnatural.
6. In college, I befriended an entomology professor and spent a month living in the Caribbean studying bugs.  In the mountains of Dominica at that time, the water was so clean, you could stop at a stream, fill up your bottle and just start drinking.  And for the record, the “bottled spring water” that we are drowning in these days tastes nothing like the real thing.
7. My first computer was a Commodore 64 too, and my parents have a picture of me holding up a dot-matrix printout that is as tall as I am.  The printout was a text based adventure game that I’d programmed in BASIC (long live GOTO!).  I don’t have a scanner here so I can’t attach the pic.  Bummer.

Let’s see who shall I spread this thing to next.

* Stephend – who needs to blog more.

* Abillings – who always has an interesting story to tell.

I think this was a great conference.  Thanks to the folks from Mozilla Europe for organizing it.  I’ll have more to say about it later, but I wanted to dash down a few thoughts before running off into the internet free lands for vacation.  We had an amazing audience of people that came together.  The presentations were well attended, interesting and the audience had no qualms about getting involved and asking good questions.  It makes me happy when I get asked hard questions; it means I’m not just talking to the wall on the other side of the room.

We discussed quite a few things regarding test development.  There is a lot of interest in the Mozmill tool, as you might imagine.  Everything from the Mozilla Messaging team beating down the door to use it, to other folks interested to use it for their own testing of extensions and what not.  I have another blog post I want to write about that, so I’m not going to talk about it here.  Beyond that, we also talked about some other very interesting things:

• You don’t need a debug build to build with tests.  I don’t know when this changed and why our documentation hasn’t kept up, but –enable-tests is enough to build a build that will have Mochitests enabled.  However, it does not appear to run reftests if you build a release build this way.  Looking into that, and I have it on my list to change MDC. (Thanks Serge!)
• We talked about some serious printing tests.  We might be able to get some folks to volunteer some time to help with creating them once we have a plan in place for how to do this.
• Bernd and I talked about DOM Level 2 tests.  Too many of these are currently not enabled.  We need to look into getting these enabled on the tinderbox builds.
• It might be useful for developers to have a set of dedicated machines with debug environments that are the same structure as the tinderbox test machines.  These would be used to debug those failures that only occur on the tboxen.  I’ll float that idea around more formally when I get back.
• There is a patch that Jonathan (from SVG & Joost, didn’t catch his last name) has which will enable mochitests to run on IE.  I think this could be a big step to broadening our test framework to help increase interoperability across the web.  From talking to him, it sounded like several of the changes he made are the changes I am making to improve running mochitests on XULRunner applications (looking at you, Fennec).  So, he’s going to get me the patch and we’ll go from there. As you can imagine we also thought about how to get them to run on Webkit, he thinks he has an idea of how to do that too.
• I met a new contributor from France who’s interested in getting involved with QA.  We talked at length about ways to get started, things he found to be barriers to entry, and I tried to remove all those.  I think that he’ll start out by helping us translate some of the QMO documentation once our new site is ready.
• I learned also that you do in fact need to build with –enable-tests in order for Reftest to work.  I didn’t know that until someone asked us the question and I had a look at the makefile.

I did a new talk on Automated testing in general, and came up with an image of how to think about all our testing frameworks at Mozilla and how to understand where each one fits in relation to the others.  I’ll share that out, but it’s a blog post on its own.  Martijn did a great job with presenting Reftests and his “How to Reduce a Website to a Testcase” talk.  I also re-presented Shawn Wilsher’s Toronto Dev Day slides on Mochitests.  I think those are the best presentation to date on Mochitests, and they had an added benefit of the last slide stating “if you have any questions, email Shawn Wilsher”. 

I’m off for a week, hanging out in Eastern Europe and Germany.  I’ll see you on the happy internet when I get back.  This laptop is closing in 5-4-3-2-1 <poof>.

There was some response to my earlier post asking for volunteers to help with the QA Companion tool.  I’ve started to get the tools together that we’ll need in order to start working on this add-on.  There will be a project page on the up-coming final release of the quality.mozilla.org (QMO) site.  But for those of us already bristling with ideas on how to make the tool better, just check out our little Design Center.

If you want to find the code, build it, and see what’s up there, you’ll enjoy taking a look at the new QA Companion Hacking Guide.

As we get started, there will be other posts concerning design decisions and design quandries that we want feedback on.  But in the meantime, feel free to put your ideas on the lists in the Design Center page.

Calling all Add-On developers and would-be-Add-On developers!  The QA Companion Add-On could use your help!

Zach Lipton and Ben Hsieh created the QA Companion Add-On.  This is an extension that sits in a little window outside Firefox or Thunderbird and helps you to run litmus tests against the application.  It has helped hundreds of people when testing Firefox, and has become a central part of our QA Test Day events.

Both Zach and Ben are students and they’re busy with school, so maintaining the Add-On has fallen to me.  I can keep up with the maintenance on this, but that’s really not enough.  The QA Companion has great potential, lots of people have thought about ways to do more interesting mash-ups with it, including integrating some of the Nightly Tester Tool Functionality, integrating Mozmill automation UI etc. If we’re really going to take this little tool to the next level, then I can’t do it all by myself.

I’m asking for volunteers.  If you’ve always wanted to work on an Add-On, but it seemed like too big of an undertaking, this is a great opportunity.  The base code is already written, and it’s all pretty straightforward.  I’ll help you understand its ins and outs and will help you with the planning, new features, bug fixes, and ongoing maintenance.  Together, we can design a new direction for the tool and make it a really useful item for all the testers out there in Mozilla Land.

So often, getting involved with the Mozilla Project is like a scary leap into the dark water of the deep end of a pool.  This time, with a solid base of code, an installed base of users, and a willing mentor, you have a unique opportunity to step into the shallow end, and get deeper as you ramp up your skills.
Reply to this thread if you’re interested.  I look forward to working with you.  I’m on IRC as ctalbert (irc.mozilla.org, channel #qa).

We are organizing the Toronto Dev Days next week, Monday September 15 and Tuesday September 16 at Seneca College.  We are looking forward to the event, and the speakers are putting their finishing touches on their presentations.  I’ve been working on organizing the Testing Talks that will be on Tuesday, and I wanted to update everyone with the final schedule of how those would work.  We will be covering a few very broad areas of the Mozilla platform, areas that can be leveraged by extension developers, by other projects, and perhaps most importantly, by people that wish to jump into the Mozilla universe in the unique role of test development and automation.

* Test Talk I: XPCShell Testing with Ted Mielczarek

* Test Talk II: Mochitest Testing with Shawn Wilsher

* Test Talk III: Memory leak testing with Carsten Book

* Test Talk IV: Lightweight Mozilla UI automation with yours truly.

We are excited to see you in Toronto.  If you haven’t done so already, sign up!