TBPL

We’ve been working toward automation on TBPL for a while. This week we worked with Aki from releng to get mochitest, reftest, and emulator-based WebAPI tests running on Mozilla-Central, Try, Services-Central, Mozilla-Inbound, Fx-Team, Cedar, and Ash. We are set to add them to Mozilla-Aurora on Monday.

• Cedar is our staging area for mozilla-central. If you would like to turn on a new set of tests for mochitest or reftest, you can land that change here to see what happens. (If you break something, please back out)
• Ash is our staging area for mozilla-aurora. Here too, you can land things to help expand the automation destined for aurora. (If you break something please back out).
• Try support does exist, but the try chooser is not yet updated, so to run all B2G builds on try, use this try chooser syntax: try: -b o -p ics_armv7a_gecko -t none -u all

Gaia tests/QA Support

The Stephen Donner’s Web QA team is working on automating the Gaia smoketests. This week we pivoted hard to get those stood up in CI. We have them running in our Jenkins instance on an Unagi device and reporting to Autolog.

We met with Geo’s Web API team and made some fixes to the B2G mochitest suite so that they are unblocked and can continue creating webAPI mochitests.

Rob Wood and Dave Hunt continue working closely with the QA team to help them generate more tests. This week, Rob added ten more webSMS  emulator tests alone.

PandaBoards

Pandaboards are our on-change solution for automation that needs to run on device, like the Gaia automated smoketests. There are many moving parts to stand these up at scale. Here is a run down of what happened this week:

• Thomas Zimmermann worked with us and we now have a more reliable and stable pandaboard kernel.
• Chris Atlee from releng got our buildbot-provided panda builds to include the proper version of Gaia as well as this updated kernel which is going to be critical to the testing we’ll do next week.
• We have rolled out this new build to our test boards in IT’s colo, and are troubleshooting some networking issues now. Once those are resolved, we will be clear to start testing our pandaboard automation.

Eideticker Performance System

This week, Will Lachance got all the existing Eideticker benchmarks running using a pandaboard hooked to an Eideticker system. He has uploaded a screencast of the tasks.js panning test being performed on B2G here.

Next Steps

We’re not done yet. Here are the things we’re working on next:

• Turn on B2G reftest and mochitest on Aurora (target: Monday, November 12)
• Perform flash testing to ensure we can flash pandaboards per push (target: November )
• Create B2G specific Eideticker tests to measure specific B2G performance point (target: next week, by Nov 16)
• Fix the QA Automated Smoketest reliability issues (target: next week, by Nov 16)
• Add xpcshell to our B2G automated test suite (target: November 23)
• Get pandaboards running Gaia Smoketest and Gaia Integration tests per checkin (target: optimistically, end of November)
• Continue expanding the set of mochitest, reftest, and xpcshell tests that we are running in the B2G test automation (ongoing)

• Start Firefox with a URL ending with a query parameter like “start=”<current time in ms since EPOC>
• The page that the URL points to does a JavaScript “new Date().getTime();” as part of its onload handler and subtracts that from the value in the “start” query parameter
• The URL prints out the date to the console (because we can do that since we control the browser and the profile)
• Automation reads the console and puts the value in a database

Pretty simple.  Applying this to different browsers, you have to nix the “print to console” idea.  But, how hard could it be to POST to a web service that stuffs your result in a database?  Do that and the rest of it will all “just work”, right?

Well, not really.  Every browser implements the cross-origin access policy to a different degree, and since we did this on android, some of them don’t seem to support it at all.  Once we found a way around that, we realized that not all the data was making it into the database because the automation would kill the browser before it had a chance to POST its results.  So we slowed that down, forcing the automation to wait 20s before closing the browser.  Then our database crashed, this part we had nothing to do with, but Murphy’s law states that you can’t have an automation project without at least one bonfire igniting under your chair.

Add to this cross-browser headache that we’re automating this on multiple phones.  The older Nexus phones (Nexus One and Nexus S) will not stay connected to a wireless network after reboot (appears fixed with Galaxy-Nexus or with ICS, not sure which).  Even if you put these phones on an open network with no contention and they are set to “join automatically”, they will at some point boot into a state with their wireless disabled. We had to write some service code to ensure the wireless remained on and connected to our specific network on boot.  Our other phones (a Droid Pro and a Samsung Galaxy S2) have no problem staying connected to the network, but they alternately “freeze”.  I’m still trying to debug what this “freeze” actually is, but everything is functioning fine on the phone – network, logcat, process list etc are all normal.  However, the phone stops running the automation.  It’s interesting that the Nexus phones never encounter this issue and they are all running the same version of the automation code and browsers.

At long last, we have fought through enough of these issues so that we can start to see the results of the data coming into our database (select “2 months” or “all” to see data).  Because we are merely firing our “timing” function when the “onload” event happens for the page, we can see the different interoperability issues with measuring this event. We knew it wasn’t perfect, but the results we are seeing on Android make me call into question the usefulness of this as a cross-browser comparison tool at all.

• Opera seems to fire the onload event randomly.  I’m not sure what they are doing, but their timing is all over the place.  Note that this could be a fluke in the automation as the Samsung/Droid Pro hang usually occurs during the Opera test (which, by chance, is also the first test).  However, note that the Opera numbers for the Nexus phones are also wild, and they are not afflicted by this unusual hang.

• Dolphin and the stock Android browser are both webkit based browsers and we have always known that webkit tends to fire this event very early in the page-load sequence.  This is reinforced by the fact that the event always happens at roughly the same time regardless of the underlying phone hardware, especially on the stock Android browser

• Fennec – this automation measures the new native Fennec product.  Currently, the system contains results from the beginning of the project to the point at which we moved from the birch tree into the mozilla-central tree.  I have another set of jobs to run that will get us the last two weeks from the mozilla-central tree, once the phones finish their jobs from the previous two months.  Of the four browsers being measured, the only one changing versions is Fennec; therefore, you can see the effect of our developers’ work as they add features and battle regressions.  Native Fennec is still under heavy development, and this is why the Fennec number jumps around as much as it does.

The system is far from perfect.  Measuring onload is at best an artificial metric, and not at all indicative of what the user sees.  In desktop automation, we don’t even use onload, we use the “mozafterpaint” event notification.  For the next stage of the cross-browser test we are going to automate some visual comparison tests to get closer to measuring the metric that really matters: real-life user experience.  In the meantime, the onload tests will continue to give us a rough barometer of our regressions and performance, especially against our own historical data.  To that end, I am going to undertake the next few improvements to this automation:

• Understand what the hang is on the Galaxy S2 and Droid Pro phones and fix it
• Add more phones to the system so that it doesn’t take so long to run through a set of jobs (we only need these temporarily until the system catches up on old data).
• Experiment with lowering the timeout period between “results uploaded” and killing the browser under test. (This might work better now that we have changed database backends).
• Get a better front end UI for the results.  If you’d like to contribute to this, let me know, because this website could sure use your help!

However, week after week as I asked for each project’s status and listened to it, I wondered why on earth would anyone want to come to this?  And why were we spending an hour each week boring ourselves to tears when we could be doing something useful like being silly on IRC? So, the A-team and I talked about it, and we decided to do an experiment with the meeting.  Here’s what we’ve been doing for November:

• One person spends an hour or so a week collecting the status from everyone on the team.
• That person puts together the wiki page.
• At the meeting on Monday, that person is the emcee and does a five minute run down of the week’s highlights.  This is the toughest job.  We have a great team, and there are always a lot of highlights.
• After that, we raise any issues that need raising and discuss them, five to ten minutes.
• The emcee gets to pick the emcee for the following week.
• Then we remind people to check the wiki page for the schedule of project-specific meetings that week, and we’re done.

The entire thing takes no more than twenty minutes, and most weeks it takes less than ten. So far, I have to say I’m a fan of the new meeting.  I worried that I’d lose my ability to stay abreast of what is happening on our projects, but that hasn’t been the case.  In fact, if you compare the wiki pages from before with these new ones, you’ll see that our emcees do an amazing job pulling together the data and communicating the highlights.

The other benefit this gives us is that as we grow into a larger team, it’s harder for all of us to interact.  Our rotating emcee gives each person a chance to talk with everyone else on the team and learn something about everyone’s projects.

I don’t know if this would work well for other teams, but it has worked really well for us so far.  If you’d like to drop in, here’s the information about our meeting.  This week’s emcee is our illustrious maple-bacon-cake-baking, cowboy-boot-wearing intern, Tfair.

I worked at a company called SimDesk that decided to reuse the Thunderbird and Sunbird code bases and make a great email application–this was long before the Lightning extension came into being.  Like any good closed-source company, we stole the code and worked on it in secret until we had a shining example of an “Outlook killer” (well, more or less).

Then we started feeling like we should contribute some of that code back to Mozilla.  We had a bunch of very awkward meetings with Dan Mosedale and Mike Shaver as they tried to teach us how to do open source.  They kept saying, “just submit a patch”, we kept wondering which lawyers we’d have to get involved to do that.

Eventually, Mike Hovis (an old friend and superior developer) and I started writing those patches.  It became clear that our changes wouldn’t apply cleanly to the newly refactored “Lightning” source base.  We decided that I’d make it part of my job (20% of my time, as I recall) to make patches for functionality we cared about and get it to the Mozilla calendar team.

I started attending the calendar team’s public meetings, and during one, when they asked if anyone wanted to lead a calendar QA team, I volunteered.  I had no idea how to actually do this, but I wanted to try organizing online to see if some of my offline organizing skills would translate.  My contribution of time grew.  As SimDesk directed me to work on Outlook extensions rather than an Outlook killer, I spent more and more of my time working with my calendar team, writing patches, mentoring, and aiding volunteers as they found their roles as leaders and developers in the calendar project.

And one day, when I could plainly see the writing on the wall, I asked Dan if Mozilla would actually consider a resume from me.  After his enthusiastic “yes”, I applied, and the rest is history.

Starting in the calendar project was incredible.  It was smaller (of course so was Mozilla in those days–even though it felt huge to me at the time).  It was easier to see your impact in such a small space, easier to identify volunteers, and easier to mentor people through the process and watch them become leaders.

Starting in that small area was also fortuitous because there was so much that needed to be done and opportunities were everywhere.

I still think that there are small areas across Mozilla where people can start and have a similar experience.  However, I think that Mozilla seems so monolithic these days that it is daunting to even try to find those niches where you can start out as a volunteer.  It is up to us on our teams to identify those areas where people can start, publicize them, and help people make that leap from “casually interested party” to “volunteer”.  In that vein, I tried articulating the roles that we’d like to see people step up to fill on my team.  If you’re interested, you know where to find me.

Well, PandaBoards come with nothing, not even a power supply.  They can be powered off USB, but it’s pretty difficult to get adb working in that state (if you have steps, I’d love to hear them).  So, here are the steps to getting something usable working (See the official getting started too):

1. Order power cord, specifically the adapter and the cord
2. Order 8-16Gb SDCard
3. Ensure you have a mini USB cord
4. Ensure you have a CAT 5 network cable.

Once you have this, you can build or download a build onto your SDCard.  Oh yeah, you’ll need an SDCard writer/reader.  Most computers have them by default these days, thankfully.

Then, plug it all in, and it should work.  I’ve noticed a few oddities:

• Our SUTAgent had some difficulties at first, but now it seems to be working fine.  Still debugging this.
• ADB won’t work if the card is plugged in when it boots.  I think this is due to the build, as I seem to recall seeing an issue on it earlier.  I’ll keep researching and will try some different builds to find something more stable.  In the mean time, unplug when you reboot the card, plug in after the card is up and running.  Also, you won’t see the “USB” notification that you usually see in Android.  So, don’t expect that.
• There is something going on with the package manager.  I installed Fennec, but the pm doesn’t list it, and claims that it is not installed.  However, it runs fine, appears in the applications, and can’t be re-installed.  I just can’t uninstall it.  Still investigating that too, and like the other OS level issues, I’m wondering about this downloaded build.

But, today, Mozmill 1.5.3 is released.  It’s already on Pypi, and it was just submitted for review on AMO.

We fixed:

• Bug 643697 – JSBridge limits amount of data transferred (fixed on
  branch, fix still pending on master)
• Bug  645234 – Fix for addon compat prefs (short term).  More changes may be needed as we change our version numbers to accommodate faster releases.
• Bug 636746 – Add Mozmill version to test report
• Bug 646970 – Remove controller argument from _buildMenu call in Menu::open
• Bug 647030 – version bump
• Bug 648523 – Mozmill 1.5.x doesn’t remove temporary addon folders
• Bug 506760 – Mozmill tests executed with UTF-8 characters cannot be executed
• Bug 648523 – Make windows process handling more robust

Thanks very much to the tireless folk who’ve fixed all these bugs and verified them.

Mozmill 1.5.2 is now on pypi, you can easy_install it.  The Mozmill 1.5.2 add-on is currently under review on addons.mozilla.org (AMO), and will be posted live after the AMO editors complete their review.

There are getting to be way too many people to thank (which is an awesome problem to have), but I want to make sure to thank the following folk who definitely went above and beyond the call of duty:

• M.A. Darche
• Merike Sell
• Adrian Kalla
• Henrik Skupin
• Geo Mealer
• Anthony Hughes
• Aaron Train
• Heather Arthur
• Jeff Hammel
• Andrew Halberstadt
• And several others who’ve reviewed patches, looked at downstream bugs, and reported issues

Some of the big bugs that were fixed in this release were:

• Manifest support for easily disabling tests
• Fixes to compatibility checking for addons
• Adding the ability to forward command line parameters to the program under test (like the debugger for instance)
• Several improvements to the stopping/starting code to fix several hang bugs
• Drastically improved support for simulating drag and drop
• You can now get to elements using CSS selectors
• Syntax errors inside tests are now more clearly reported when running from the command line.

The full list of fixed bugs are here: http://bit.ly/e1Oe9f

If you want to get involved and help out with the Mozmill 2.0 effort, it’s not too late.  Stop by #mozmill on IRC (irc.mozilla.org).

We need some icons for our new Profile Manager application.  This is a profile manager that will work on any Gecko application (Thunderbird, Firefox, etc).  It does a whole lot more than the old, embedded profile manager ever thought of doing, but it has no icon, and no cool graphics for its web pages.

Beyond our gratitude, we’ll make sure you’re given attribution and recognition.  We’ll endorse your good work.  So, check out Jonathan’s blog post for more details, and get your pencils ready!

Specifically we’d like to see:

• 16×16 pixel icon
• 64×64 pixel icon
• 128×128 pixel icon
• Logo using the words “Profile Manager”

To submit your designs, please see bug 605576 for details on how to do that.

Thanks!

\o/ MOZMILL 1.5 \o/

It’s ready now on Pypi and (once it’s reviewed) on Addons.mozilla.org.  Tons of changes and many hours of work went into this release.  We couldn’t have done it without you!

• Heather Arthur
• Jeff Hammel
• Andrew Halberstadt
• Henrik Skupin
• Atul Varma
• Mark Banner
• M. A. Darche
• Brian Warner
• Anthony Hughes
• Aaron Train
• Adrian Kalla
• Geo Mealer

Thank you also to everyone who’s tried it out, commented in bugs, run tests, and reported issues.  We really appreciate your help in making this release as solid as we could make it.  There are a bunch of new features we are very excited about:

• New Bespin based editor UI
• Ability for tests to terminate the browser
• Ability to capture errors when the browser hangs
• Better error reporting and logging
• Handling accessing elements in content XUL
• Ability to handle non-ASCII characters in test scripts
• waitForEval replaced with general waitFor function to make tests easier to read
• Command line and robustness fixes for restart tests

You can see the fixed bugs here.

That said, making anything worth making involves some hard decisions.  There are several fixes that did not make it into 1.5 and we wished they had.  We’ll be collecting those for the next dot release, 1.5.1.  To nominate a bug for that release add the [mozmill-1.5.1?] whiteboard entry in bugzilla.

= What’s Next =
Well, we’ve been thinking a lot about it, and have already been working on it.  It’s Mozmill 2.0.  And it is going to be incredible.  You should check out the plan.  We’ll also be doing a bug-fix only set of dot releases as issues come up on the 1.5 branch.  We’d welcome you to get involved in Mozmill 2.0 by suggesting ideas or picking ideas to implement.  We’d love to help you with your patches.  You can always find our team in #qa on IRC or on this mailing list.

Thanks so much for your help.

Enjoy 1.5!

You can see the release notes for the add on on AMO and see the 1.4.2 specific bugs that went into this release.  The release is also available on pypi so you can get it through pip.

If you find any bugs please file them in the Mozmill component.

Thanks!