Automation is really hard. It's difficult to create and very brittle, and thus difficult to maintain. I'd recommend you spend 75% of your time on unit tests, 15% of your time on integration tests, and 10% of your time on automation. Automate the happy path only. Don't automate anything that can be run with DUnit. Write your code so that you can easily do unit tests and integration tests.
As for an automation tool, I'd first recommend a scripting language. If that won't work for you, then Test Complete is a pretty good tool, and you can script it in Delphi.
Further to Nick's comment, I worked for two years in a shop which depended heavily on automation testing. They had four developers on Delphi, and three QA people on automation, writing and running the scripts. The automation took nearly 24 hours for a full cycle of tests, and that was only happy path. Coverage was not great, and apprently hard to assess.
I've been in a shop that used TesComplete and we eventually devolved to using screenshot comparisons in virtual machines, because the "read text out of controls" part didn't work with 90% of our controls (most ActiveX controls, anything from DevExpress, any grid, etc). We had more testers than developers and the ones testing manually found more bugs.
If you start with something primitive like AutoIt you can grind out basic scripts (in BASIC :) fairly quickly. The trick is, like all programming, to write a couple then start extracting common functionality into a library. That way later scripts become "StartProgram(); OpenReportScreen(); RunDailySalesReport("June", "2012", "HighlightsOnly");" then a bunch of reading values out and testing them. Otherwise you have 20 different versions of the same code in you 20 scripts and you are screwed when anything changes. Most testers will opt for the latter, BTW.
Thanks for all the comments. I truly appreciate everything you all have to say. We do use DUnit but not nearly as much as I would prefer. The rest of our testing is manual and our QA staff is interested in seeing what they could add in the way of automation.
You might want to check http://www.sikuli.org, it's an innovative since it uses OCR technology to recognize the UI elements on the screen. There is no other desktop automating tools that's as smart as sikuli, IMHO.
Automated test is much easier with web apps since Webpages has a DOM and script-able.
What's wrong with DUnit?
ReplyDeleteAutomation is really hard. It's difficult to create and very brittle, and thus difficult to maintain. I'd recommend you spend 75% of your time on unit tests, 15% of your time on integration tests, and 10% of your time on automation. Automate the happy path only. Don't automate anything that can be run with DUnit. Write your code so that you can easily do unit tests and integration tests.
ReplyDeleteAs for an automation tool, I'd first recommend a scripting language. If that won't work for you, then Test Complete is a pretty good tool, and you can script it in Delphi.
Further to Nick's comment, I worked for two years in a shop which depended heavily on automation testing. They had four developers on Delphi, and three QA people on automation, writing and running the scripts. The automation took nearly 24 hours for a full cycle of tests, and that was only happy path. Coverage was not great, and apprently hard to assess.
ReplyDeleteI thought DUnit was automated testing?
ReplyDeleteI've been in a shop that used TesComplete and we eventually devolved to using screenshot comparisons in virtual machines, because the "read text out of controls" part didn't work with 90% of our controls (most ActiveX controls, anything from DevExpress, any grid, etc). We had more testers than developers and the ones testing manually found more bugs.
If you start with something primitive like AutoIt you can grind out basic scripts (in BASIC :) fairly quickly. The trick is, like all programming, to write a couple then start extracting common functionality into a library. That way later scripts become "StartProgram(); OpenReportScreen(); RunDailySalesReport("June", "2012", "HighlightsOnly");" then a bunch of reading values out and testing them. Otherwise you have 20 different versions of the same code in you 20 scripts and you are screwed when anything changes. Most testers will opt for the latter, BTW.
Typically, "automated testing" refers to "GUI-driven automated testing". But yes, DUnit is automated.
ReplyDeleteThanks for all the comments. I truly appreciate everything you all have to say. We do use DUnit but not nearly as much as I would prefer. The rest of our testing is manual and our QA staff is interested in seeing what they could add in the way of automation.
ReplyDeleteDean -- I would have your QA continue with manual testing and expand your use of DUnit as much as possible.
ReplyDeleteYou might want to check http://www.sikuli.org, it's an innovative since it uses OCR technology to recognize the UI elements on the screen. There is no other desktop automating tools that's as smart as sikuli, IMHO.
ReplyDeleteAutomated test is much easier with web apps since Webpages has a DOM and script-able.