automation
Windmill Plugin for Hudson
May 10th
Over the last 6+ months, I have been using Hudson in conjunction with Windmill very heavily for continuous integration. For the most part using the build step specific to whatever the slave OS requires has worked sufficiently well until recently when my needs changed.
I use the ‘configuration matrix’ option to build a matrix of browsers to run the tests, this way I can have one job that represents a test run on multiple boxes and multiple browsers on each box. Drilling down allows me to see the results for each of these test runs within the job. (Configuration Matrix is awesome btw, except for one really annoying bug for, which there is a reasonable workaround)
Having many build resources, sometimes I want to move a job from one machine to another in order to equalize load on the different machines or to have a job running in ff2 and ie6 instead of ff3 and ie7. Thus the need for a uniform build step that will run the same way regardless of the machine or installed browsers, became necessary. Additionally the commands for running tests got bigger and less manageable all the time, so the time for a clean user interface finally came. This way I can automatically append arguments like ‘exit’ to keep the build step interface simple and clean.
The other simplification is that in the Hudson configuration page for the Windmill Test plug-in you can tell it to automatically call the contrib ‘clean_run.py’ script with the correct arguments (assuming clean_run.py can be found in your path).
As I am relatively new to the Java world, I struggled through the development process but thanks to some reasonable plug-in documentation, responses from the hudson dev mailing list and some IRC conversations with Kohsuke (thanks so much) I finally have something that sufficiently addresses my needs.
Screenshots
Project
The code is all available on GitHub and I welcome any improvements or input from the community as I know that continuous integration is a very important piece of the utility provided by Windmill and this pieces should be as easy as possible to setup.
The combination of this plug-in, and others like the amazon s3 plug-in or the hudson VMware plugin bring us that much closer to seamlessly creating test resources, running our functional web tests across all of our supported platforms and then throwing it away until it’s again needed (or using the CLOUD).I have also done some integration with virtual box that I have found to be very successful.
Download
Please log bugs and let me know what you think! If you are an avid Java and or Hudson plug-in developer and you are interested in contributing, please jump right in!
PyCon 2009 Recap
Apr 13th
Getting back in the swing of things after conferencing for weeks can be pretty painful, thus the lateness of the post. However I think it’s important to go over some thoughts still lingering in my brain as a result.
First off, I have to say that for those of you who don’t know, PyCon is a community organized event, and amazingly well done. I was impressed by the design of the conference, the way they had four talks going on at once and they tried to keep them in a similar interest track. Every talk I attended was at least “good”, and many were “great”. You could really feel a community vibe, and for a conference that had 800+ attendees in the middle of a major recession they had every right to be excited.
There were two major themes content wise that really impressed me, the first was an amazing amount of web framework focus. Django obviously being the twinkle in the eye of the community, but there were smaller communities for each of the other projects, Pylons-Turbo Gears, web2py and lots of tools built on top of them. One that struck me with some major promise is the Pinax Project. Their goal is to make it so that I don’t ever have to deal with building user registration and in site messaging… and all the other features expected for any site that has social network functionality.
The other major theme was a mini testing conference going on within PyCon, that I was very comfortable hanging around with. We had a hugely successful Birds Of a Feather, as well as a surprisingly active Open Space talk for Windmill.
The “Using Windmill” talk turns out to be pretty successful in every aspect that I really care about. I do wish that I had been able to get a little more sleep the night before, and I have to admit the size of the venue was a bit overwhelming. I now realize watching the footage that I used the word “UM” way too much, and the demo videos must have been hard to watch from the very back of the room. But barring those two things, I am quite happy (I shall learn and practice for the next round of shameless PR at OSCON 2009).
The “Functional Testing Tools in Python” panel was very successful, and a lot of fun. I always enjoy the friendly banter between the different project owners. Everyone has a different opinion on what they care about, focus on and feel they do the best. Obviously since the only two projects represented that focused on Web Testing were Windmill and Selenium, we got a lot of attention.
Watching that panel footage I definitely think that the introductions were too long, but I still think our Journey themed – mind blowing – Windmill demo video was a great intro. At the very least, the audience had a little entertainment before the geek droning began 
Slide had an awesome presence this year, a fun booth, huge banners everywhere and 6 attendees. It was fun to see all of the great responses I received about Slide from people out there in Python land.
Here are links to the videos:
- Using Windmill: http://pycon.blip.tv/file/1947236/
- Functional Testing Tools in Python Panel: http://pycon.blip.tv/file/1947342/
And some pictures:
If you are interested in seeing the new and improved version of the “Using Windmill” talk, please make it out to OSCON 2009, “Scheduled for 16:30 on 22 Jul 2009.” in San Jose, CA.
We are waiting to hear back from both Open Source Bridge and the AJAX Experience as to whether we will be participating in those conferences (fingers crossed)!
Skinning Windmill with JQuery UI Themes
Mar 24th
I have been doing a lot of UI work on the Windmill trunk, and over the past few months I have had multiple requests for the ability to apply skins. Of course my reaction up until this point has been… alter the CSS! Which is not exactly the answer people were looking for.
Here is your answer, and it is now easier than ever.
1. Go to: http://jqueryui.com/themeroller/ and create your theme (or pick a pre-defined one)
2. Download and unzip the file. (it doesn’t matter what you select for jQuery components)
- On a mac you will find something like this: jquery-ui-1.7.1.custom.zip Folder
3. Open the contained CSS folder there will be another folder, currently windmill uses “smoothness”
4. Find windmill/html/css folder in the windmill source directory
5. Copy the specified folder in step 3 into this folder
6. Edit windmill/html/remote.html, line 10 to read like the following:
1 | <link type="text/css" href="css/*your folder name*/jquery-ui-1.7.1.custom.css" rel="stylesheet" /> |
7. Load windmill!!
Here are a couple examples of pre-defined themes I tested out.
Humanity
UI Darkness
Happy skinning.
Meet lookupNode
Jan 20th
A few months ago I did a re-write of the DOM access functionality used by Windmill and various other projects of mine and the result was a wrapper around ElementsLib called lookupNode. I use this all over internally but until today didn’t realize that it could be hugely helpful for test developers and people trying to debug their applications.
The following list are the different options you have for looking up your node:
- link
- xpath
- id
- name
- value
- classname
- tagname
- label
Keep in mind that we iterate all iframes we can access in the window if the node isn’t found in the page document. lookupNode takes an object, like so:
var myNode = lookupNode({id:'myNodeId'});
The very interesting use case we had today was the need to access and parse data from a link on a page in a Python test, I believe the final result looked something like the following:
url = client.commands.execJS(code="lookupNode({link:'Link Text'}).href")["result"]
#url parsing in python goes here into needed_value
client.asserts.assertValue({id:'mycheckbox', validator=needed_value})
lookupNode in Action
I hope someone finds this useful, the abstracted standalone ElementsLib for the content space can be found here!
Windmill Gets a Facelift for 1.0Beta1
Nov 19th
Working up to the Windmill 1.0 Beta 1 Release, I finally had the opportunity to put some time into making the IDE (that a lot of you live in when in test writing mode) a little bit nicer to look at.
The IDE has been growing organically since 0.1 and there was a lot of functionality hacked into it that wasn’t in the original game plan, so I did what I could to improve the beauty of the CSS/Layout as well as the whole mess of code behind it.
Launching
If you have used our latest release, or are running trunk you know that we have significantly improved the load times for the Windmill IDE. By compressing the JavaScript when the service is instantiated we can simply hand the IDE window one file that contains the vast majority of the required code.
The reason that this makes such a huge performance difference is because we are loading the source via the local windmill proxy and the data size size had very little impact, the overhead was in the browser two connection limit. When you have to pull down ~30 files two at a time it takes its toll and made the IDE feel very sluggish and more like a web page loading than an IDE.
In the process of figuring out exactly what was slowing down the launch time we added some more informative messages and output so you don’t sit there staring at a twirling circle graphic wondering if anything is happening. And to make the experience even more fun, I couldn’t help but implement a progress bar.
General Layout
I removed the toolbar at the bottom of the screen, which I felt it was an irritation for test editing (especially with the drag and droppable actions). It is now in a drop down menu at the top right of the screen, with the rest of the UI access to IDE functions.
Settings and Firebug Lite Improvements
The settings dialog has continued to improve by implementing more useful defaults, adding new options, removing deprecated options and simply making it just look better. Thanks jQuery UI!
Firebug Lite has been a very popular feature since we first announced it, which has led to a handful of bug fixes over the last month. The most major of these was that the initial Windmill implementation of Firebug Lite required you to have Internet access as it was using resources that were hosted elsewhere.
These have since been copied to our source tree and are made available by the Windmill server so you can happily introspect your Web Apps JavaScript while writing tests on your Intranet.
Output and Performance
Instead of writing all the raw windmill output to the output and performance tabs there is now an array called windmill.errorArr, where all terrible errors and warnings about technical details are pushed in the case you are interested to see all that data. However, it’s more likely that you aren’t and scrolling through all that output data becomes tedious.
This is why we have implemented output in blocks with the background color representing pass/failure with green/red (white for performance). These blocks are expandable, clicking them will reveal all output (or performance information) we know about the action that was executed. This should give you a faster general overview of your results and allow you to quickly see the details you care about.
Other Worthwhile Mentions
We moved our XPath implementation from Ajax-Slt to JS-XPath, which has proven to be more accurate when it comes to resolving XPath generated in Firefox (or using Firebug) against non XPath native browsers such as IE.
Many bugs and improvements have been made to the DOM Explorer, which should now feel a lot more like the Firebug DOM inspector, but should work in any browser.
We have also put a lot of effort into improving the communication between the JavaScript Controller and the Python Service so that when a test fails you get as much detailed information in the service as you do in the IDE.
Timing and MozMill
The timing has lined up nicely as we are working on both a 1.0 release for Windmill and MozMill. MozMill is geared towards automated testing of all applications on the Mozilla Platform and functions in the trusted space providing lots of very useful flexibility.
You can currently try out MozMill 1.0rc1 as a Firefox Add-on, and keep your eyes pealed as some exciting new MozMill feature work is around the corner.
Participate
We are always trying to make life easier for the test writer, so please log your bugs and feel free to come chat with us in #windmill on FreeNode.
Zero to Continuous Integration with Windmill
Sep 18th
Following ‘automation’ and ‘continuous integration’ in the micro blogging world I have seen a major influx in people being super interested in functionally automating their web apps. I have seen a slew of things about Grid, and Selenium, and people hacking on Watir so I decided to show you from the ground up how incredibly easy it is to get automated test running setup using Windmill and Hudson. I am not going to walk you through every detail, this is much more high level but I do plan to start a ‘continuous integration’ page on getwindmill.com in the near future for those kinds of details.
The first step is to get a couple machines that you want use as slaves and a machine to run Hudson, our setup looks like this:
Each of the machines with a different OS has Windmill installed. To make them slaves you simply bring up the Hudson web page on the machine, and run the launcher.. now it’s a slave — crazy easy right?
Now to setup test runs for the machines, in Hudson you click: “New Job” on the left hand side and do something like the following:
Tie this job to the slave you want it to run on (we can’t have IE runs happening on MacOSX):
Tell this job to run 10 and 30 minutes after the hour:
The build steps to actually run the tests, the first kills any straggling processes (more details below):
On the Mac for the Safari job, I want to make sure there aren’t any instances of Safari left hanging, or windmill processes sitting around so we do:
ps -ax | grep windmill | awk '{ print $1 }' | xargs kill | true
ps -ax | grep Safari | awk '{ print $1 }' | xargs kill | true
Then we want to grab the latest test code from svn and launch the windmill test:
svn up /Users/adam/Documents/main_bt/windmill/
python /usr/local/bin/windmill safari http://www.facebook.com test=/Users/adam/Documents/main_bt/windmill/fb email=username@slide.com password=pass report=true exit
rm /Users/adam/Library/Cookies/Cookies.plist
I am telling windmill to run a test against facebook.com, with the test hierarchy in the windmill/fb directory in Safari, with the provided email and password, then to report it’s results and exit.
The only thing different on our windows test runs is the way we kill the processes:
Example:
taskkill /F /T /IM windmill.exe
taskkill /F /T /IM firefox.exe
You might be asking how do I use those variables, check it out in my setup module:
1 2 3 4 5 6 | def setup_module(module): client = WindmillTestClient(__name__) client.type(text=functest.registry['email'], id=u'email') client.type(text=functest.registry['password'], id=u'pass') client.click(id=u'doquicklogin') client.waits.forPageLoad(timeout=u'100000') |
You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here.
And that last line removing Cookies.plist makes sure that the next test run starts without any cookies set to cause problems.
Have Hudson keep you updated on Jabber:
Grab the generated XML output so you can view the test results in Hudson:
Do this for each of the test runs you would like to have, and boom — continuous integration:
This is obviously a simple scenario, and you can do way, way more customization.. but this should get you off the ground. Happy testing!
Windmill 0.9.0 Released
Sep 10th
Another round of productive bug fixes and feature additions have happened over the last few weeks. We are now positioning the project to approach a 1.0 release, which means that our goals for the coming releases is focused on shaking out all those blocker bugs before we push the big milestone.
New Features, since 0.8.0
- Command line option for changing service port
- Django Support
- Forwarding Conditionals
- Addition of firebug lite support in IDE and Testing Application
- Fully functional wxWindmill contrib application
- Enhanced ’smart’ recorder
- Auto element scroll into view to more closely simulate users
- Improved IDE user interface
- Improved frame recursive element look ups for IE
In addition to all the bug fixes, which you can see for yourself in trac.
We are really excited to see the exponential growth and exposure the project has been getting over the past couple weeks, thanks everyone!
Windmill Team
Bringing Windmill to Life
Sep 3rd
Project Status
I have spent nearly every day since July 7th working to bring the Windmill Project up to a level where it can be used reliably in a production environment. Our mission starts with “Windmill is a web testing framework intended for complete automation of user interface testing”, of course this refers to the web including everything and anything inside the browser window. This turns out to be a very large task, one that only an Open Source labor of love could possibly attempt to accomplish.
Windmill has slowly evolved as a project with user contributions, a moderately active IRC channel, and enough users to keep me from forgetting what a useful and powerful tool it is. When I was offered the opportunity to work on the project I quickly saw how much needed to be done in order to get to where we needed to be. We still aren’t quite there, and like most Open Source projects we might not ever get to the envisioned perfection, however recently we hit a very important milestone. The project is now fully hosted and run by the committers, and in many ways “Grown Up”, thanks to a lot of good advise and hard work. The milestone we have reached, is that Windmill is ready for YOU to use. This week we pushed 0.8.2, which is a release that has addressed all of the major issues that we know about and have discovered with heavy usage over the past months. Our hopes are that you will go install Windmill 0.8.2 and things will just WORK. If not, I can’t wait to get your issues in trac and see what we can do to fix them.
Priorities
The main things we care about when it comes to our web testing tools:
- Low barrier to entry, low learning curve, and ease of use
- Thorough documentation, community and project support
- Support for the big 3 platforms; Windows, MacOSX and Linux
- Support for the big 4 browsers; Firefox, IE, Safari and Opera
- Easy integration with continuous integration tools
- Reliability; developers aren’t going to pay attention if the failures aren’t real
- A really nice looking logo, and a web site that is easy on the eyes..
There are always more features to implement, but Windmill hasn’t needed new features for a very long time. What Windmill needed was some serious QA, some code cleanup and a whole mess of bug fixes. If you look through the Trac Timeline you will see the massive amounts of all of the above that have happened and I am proud as hell when I launch the application today and see all that it can do.
What can Windmill do?
Windmill offers the ability to build, write, record and run tests as well as aid in debugging and development. In addition, the framework provides the ability to create and maintain hierarchies of smart and thorough tests that will ensure the quality of your web applications over time. Not only can we save you hours creating and maintaing tests, but we can also help you see your web application as a growing feature rich product, instead of a QA nightmare.
Many tools out there provide ways to write tests, some even provide recorders and DOM explorers, but none that I have ever seen provide this rich functionality cross platform and cross browser, which is really what is required in order to build a thorough test repository that represents all your possible users.
The current set of major features can be found at the Windmill Features Page as well as more details about what is currently available. One of the more exciting new features is the full integration with Firebug Lite. Web developers rely on the existence of Firebug in order to quickly build and debug web applications, and Firebug Lite is the next best thing. It’s hard to even describe how useful it has been to instantly access the JavaScript Console and DOM inspector in IE to debug a failing test. As the Open Source community grows, and tools are improved and brought to light, I think it’s very important to do everything we can to utilize these tools and use them to enhance the Windmill Framework.
Keeping it Open
The Open Source aspect of Windmill has turned out to be it’s greatest asset. The project is almost entirely written in JavaScript and Python, which instantly gives us many advantages over the competition. The JavaScript community is constantly evolving and is most certainly the futures technology platform. Python has a very strong community as well and has given us immense amounts of functionality and flexibility right out of the box.
One of the most exciting things to me personally about this particular project is the immense potential user base out there, and the large impact the Windmill Tools can have on the daily work flow of it’s users. Windmill was obviously inspired with the hopes of minimizing the need for manual testing of rich web applications, and has grown to be much more than that.
The future of the work to be done on Windmill will primarily be driven by the needs of it’s users, the changes and development of the industry and the success of it reaching the goal, to make web automation better.
Moving Forward
Concluding this major push of work, testing, documentation and moving of infrastructure; we now need to see how the community feels. There are lots of choices out there for web automation and we have made many differentiating choices along the way. It is now time to get the word out and take in some real feedback.
Thanks you all for input, contributions, patience and valuable feedback. Those of you who spent many hours on Freenode in #windmill with us debugging and hunting down those spastic blockers are troopers and we really appreciate it.