Blog post "The Story of the High Fix Rate of RAD Studio Bugs" at http://blog.marcocantu.com/blog/2015-august-quality-portal-rad-studio.html
Blog post "The Story of the High Fix Rate of RAD Studio Bugs" at http://blog.marcocantu.com/blog/2015-august-quality-portal-rad-studio.html
http://blog.marcocantu.com/blog/2015-august-quality-portal-rad-studio.html
http://blog.marcocantu.com/blog/2015-august-quality-portal-rad-studio.html
What was it again about 95% of the statistics? (;
ReplyDeleteMarco Cantù Thanks - however I have some remarks about what you wrote:
ReplyDelete"a significant number of bug reports that are not considered “errors” [..], but requests to extend a given capability are kept in the system. [..] they stay open and look like bugs not being addressed. In theory we could close them indicating the feature works as "currently" designed, and open a separate internal request for enhancements."
Then you could mark them as what they are: improvements - Jira has that category built-in, but some "smart" person decided to remove that from QP for no obvious reason - if you think this keeps people from not entering them, you are mistaken.
"70% of Reported Bugs Have Been Solved."
What the statistic shows is that 70% have been closed (solved is only one possible resolution available in JIRA). In theory half of them could have marked as "cannot reproduce", "test case error" or "won't fix" to name just a few of the standard resolutions that JIRA has.
If you look at the issues that are marked as closed in QP and filter by their resolution you can see that out of 853 closed issues only 386 are marked as fixed while 109 are duplicate, 161 are cannot reproduce and 119 are works as expected(!) to name just a few of the before mentioned resolutions.
So the bottom line for me is that you are processing the issues faster which is a good thing but it does not mean that you are fixing most of them.
Just a funny scenario that would improve your score: everyone here just report some bug so your QA can mark them all as duplicate and boom, hundreds of tickets resolved/closed in no time. So, as Jeroen said, statistics...
My personal reported/fixed ratio is ~25%. Out of 21 reported issues in QP there have been 5 resolved (resolution fixed in XE8 or XE8.1) but not closed yet (what are they waiting on?). 4 of the issues are duplicates when the QP had some hickup (which are still not marked as that - I guess they are waiting on the internal state to get transferred back) and the rest is open.
Actually, there's only one bug which needs to be resolved: the memory consumption of the IDE building "larger" projects. I am working on one which fails to build again and again and I have to restart the IDE. Many times a day. Since XE8 the situation is even worse. There's a huge entry in JIRA, many customers do have the same sort of problems but except some cargo cult hacking (use this parameter, remove that package etc.) nothing really works.
ReplyDeleteStefan Meisner Yep, that is top one....
ReplyDeleteStefan Glienke The fix rate of RAD Studio bugs is high. There has been a lot of mis-information (maybe not meant to be such) and I wanted to clarify the process and the status. Most of the features requests we end up with were still reported as bugs also in QC, but that's another story. There are some hiccups in the alignment of the quality portal, we are looking into it.
ReplyDeleteThe other think I want to mention is that researching a bug to see if it is real, checking the documentation if there is a discrepancy, and possibly closing it as "works as expected" is still an effort. Most of the "cannot reproduced" are in fact fixed, because they were originally reproducible, but were fixed along with some other bug fix or development work.
Not meaning to open a debate. You'd concur that the idea that "most bugs stay opened and are not even looked at for years" is totally out of the reality. And I agree there is room for improvement in the process, as I clearly mentioned at the end.
Stefan Meisner Absolutely. We could debate if this is a bug or a feature, but it is the most voted QP issue and it is going to get fixed.
ReplyDeleteAlmost all of my compiler bugs have been fixed over the last couple of years, so that's good. There's still a serious one in FMX which has been open for a long time, and some which are a bit more obscure and/or have workarounds. Hoping they'll get sorted too.
ReplyDeleteI do like the replication between the public and internal system. That's a nice improvement on the QC black hole.
ReplyDeleteJeroen Wiert Pluimers Don't forget: Lies, damn lies and statistics ;-)
ReplyDeleteNinja edit. Looks like the new feature request section mentioned here is only for the Community site. http://community.embarcadero.com/index.php/blogs/entry/you-can-now-post-community-issues-and-feature-requests-to-the-embarcadero-quality-portal
ReplyDeleteEli M Nope (to your original post--dear ninja), that's actually unrelated. It is a separate project on the Quality Portal
ReplyDeleteStefan Meisner
ReplyDelete+100
Marco Cantù Just add few more issue types (New Feature, Enhancement, like we had in QC) to RAD Studio QP and be done with it. I have been complaining about that since day one and honestly I cannot think of any good reasons why you haven't done it yet.
ReplyDelete"[...] when a bug is closed internally, its status is not immediately reflected in the public system. The reason is simple: Telling a customer the bug is closed but he or she cannot get the fix would be of no value. [...]"
ReplyDeleteI dont think this a good decision. You are hiding information for publicy from publicy. For the publicy it seems until a release all their problems are fully ignored. And when a new version is released someone checks which issues are (accidental) fixed and could be resolved, the rest are still ignored.
I know (hope^^) that is not true, but it looks like that to me.
Transparent information are something different :-(
Or in other words the value is that I see someone react on my reported issue, when reporting a issue I see I can change something for better. I get this information immediatly and when I report more than one issue, I get this information constantly until release. And not one message with n fixed issues.
Fabian S. Biehn "it seems until a release all their problems are fully ignored" this is not true, as there are other types of feedback reported, but they might not be able to have a fix until the next release, yes.
ReplyDelete"when a new version is released someone checks which issues are (accidental) fixed": so you thin we are accidentally fixing 70% of the reported bugs. We are really an incredibly lucky company! There is constant work on fixing bugs, the fact there is a single day a lot of these fixes are made available is totally unrelated to when they are fixed.
There are developers who are beta testers (and we did invite a large number of our update subscription customers as beta testers as well) and have earlier access to the same information, can see lists of bugs being fixed over type, and have much more information. But it comes as part of an NDA with our company.
When I wrote my comment I ignored your 70% because not everybody who visit quality.embarcadero.com is reading this post or your blog ;-)
ReplyDeleteI dont think embarcadero fixed 70% of reported bugs accidentally. I meant it looks like the bugs reported by customer to quality.embarcadero.com are fixed "accidentally" because they are also reported by interns and fixed long ago, maybe even before the issue on quality.embarcadero.com was created.
It looks like that quality.embarcadero.com is just a trashcan where all customer issues are collected and when a new version is released someone look at this trashcan and resolve these issues, which are fixed with this release because the bugs (or covered ones) were also reported from interns.
Fabian S. Biehn It might give that impression, but if you read the article there is a detailed description of the flow. When a bug is opened in quality portal, QA person for the area gets immediately notified and generally within a short time frame, it lands on the table of a developer, if this is a new genuine issue. All works happens on the internal system, the quality portal is just a customer facing front end, kept in sync (well, not always, still need to improve on that).
ReplyDeleteToday I know the workflow. Yesterday (and some days more^^) I knew some kind of this because a coworker told me. But how much people exists who dont know this workflow, dont read this post or your article?
ReplyDeleteIf you have the information "this bug is fixed and tested by qa and is expected to be released with the next version" why you dont share this with the reporter? I think every reporter has something to do with software development, so they should know if the bug is fixed and the resolution tested that I dont get the fix immediatly. I have to wait for the next patch/update/release/....
Marco Cantù I'll second the request to please get a 'Feature Request' category to file reports in QP with. Or at least, if something is determined to be a feature request, make that information flow back from the internal system to the public one. It's straight-out weird not to be able to sort by bug vs feature request.
ReplyDeleteOf my reports (20), I have about a 50/50 mix of bugs and feature requests, where some feature requests are arguably bugs. 1 bug was fixed. 1 bug was marked as 'as expected', although it was actually fixed in a later release. 18 items are open, of which 2 or 3 seem to me moderately important bugs, and 7 or 8 are (to me) fairly useful feature requests. The remainder were worth filing but I can understand if they don't make the cut as important enough to fix.
So that's a 10% success rate of report -> fix, and a 50% rate of reported -> not fixed or implemented, but (IMO) should be.
I know it's hard to triage bugs and decide if they should be fixed - I've done that, it's been my role to make that choice before. With that hat on, I can see why even all the important (!) bugs I filed have not been fixed. Yet, with my user hat on, the vast majority of what I filed are things that are important to me and that I'd really like to see done. I do hope many of those will make the cut next time you do a Quality release.
I have not reported it since it is not reproducible as intended but random IDE crashes are very annoying these days. I hope IDE stability will be improved in later versions.
ReplyDeleteIDE stability is currently a core focus for R&D.
ReplyDelete