I wanted to use the memory leak detection of DUnit (probably DUnitX would have the same problem) and suddenly all my tests failed because of supposed memory leaks.
I wanted to use the memory leak detection of DUnit (probably DUnitX would have the same problem) and suddenly all my tests failed because of supposed memory leaks.
After some digging through the code I found that the reason was using my mock implementation which internally uses the enhanced RTTI for obvious reasons.
Now comes the crux: TRttiContext uses caching for all TRttiXX objects it creates and when something of these are created during the run of the test (like when creating the mock during the Setup method even though the mock itself gets freed in TearDown) the memory leak detection complains (as it just compares the allocated memory before and after).
So basically this comes down to: how to prevent reporting these false leaks that are caused by things allocated during the test but outliving its duration (*cough* global state...).
After some digging through the code I found that the reason was using my mock implementation which internally uses the enhanced RTTI for obvious reasons.
Now comes the crux: TRttiContext uses caching for all TRttiXX objects it creates and when something of these are created during the run of the test (like when creating the mock during the Setup method even though the mock itself gets freed in TearDown) the memory leak detection complains (as it just compares the allocated memory before and after).
So basically this comes down to: how to prevent reporting these false leaks that are caused by things allocated during the test but outliving its duration (*cough* global state...).
Isn't there a way to register a class for being ignored?
ReplyDeleteOff-topic: EurekaLog has extensive leak detection.
Lars Fosdal I am not talking about the standard memory leak reporting that FastMM or other tools can do but about an approach to handle memory allocations outside the scope of the tested code and not consider them in the before/after memory consumption comparison during the run of unit tests.
ReplyDeleteAh, on a per test basis?
ReplyDeleteWould it be possible to somehow allocate these TRttiContext initiated objects prior to the actual tests?
Lars Fosdal Yes, but that would only solve the problem partially. If somewhere inside the code I am testing RTTI is used this causes the same problem because as soon as you have one global TRttiContext (which I usually have because of the caching) it holds the entire thing alive.
ReplyDeleteI could also force the TRttiContext to clear its state but that would probably blow up a framework like DUnitX which itself uses the RTTI or other code that works with the RTTI during startup.
For pragmatic reasons I usually disable memory leak detection on my workstation. Detection is only enabled on my build server. I use a simple trick to ignore these "one time memory leaks": I run the whole testsuite twice and evaluate only the second run.
ReplyDeleteThe implementation in the dpr file looks like this:
XMLTestRunner2.RunRegisteredTests();
if ParamStr(1) = 'IgnoreSingleTimeMemoryLeaks' then
XMLTestRunner2.RunRegisteredTests();
My build server still reports new memory leaks everytime I produce some in tested code.
It's not the nicest way to solve this problem but it works without headache.
Christopher Wosinski That's an approach I could use. But I want the leak reporting happen also on my workstation. TestInsight throwing that into my face is better than getting it later from the build server.
ReplyDeleteWhich would lead to my next question: is there a way to get the leaked things for the duration of the tests (just as you get with report memory leaks on shutdown)? The message: 4552 Bytes memory leak (Setup= -7392 TestProc= 17592 TearDown= -5648 ) is not that much helpful and I currently have to look into the Tests_MemoryManager_EventLog.txt it spits out.
I'm not familar with TestIsnight. Is there a way to inject TestInsight a TestRunner which executes each test twice? Then you could use this approach on your workstation with too.
ReplyDeleteUnfortunately I didn't find a way to get a more detailed memory leak report.
Hmm OK, you are the man behind TestInsight. Maybe it could be an approach to execute each test twice and evaluate the result after the second run. When a test leaks memory you could run only that particular test again via command line and attach the content of the memory leak file to the test result.
ReplyDeleteAs a user of TestInsight it would really feel like testing like a pro ;-)
Stefan Glienke Nasty cough you have there.... You should see someone about it. ;-)
ReplyDeleteI have been down this path and it is horrible. Especially when there is a *cough* (thanks - now I have caught it ;-) ) global state that has strings. If the string gets replace with a longer/shorter one, you get a (anti)leak reported.
Thanks to Honza Rameš for reminding me that his https://bitbucket.org/shadow_cs/delphi-leakcheck contains a mem leak monitor for DUnit which does way better than just telling me some stupid memory numbers before and after the test. Combined with running the tests twice (I just did a simple TestFramework.RunRegisteredTests regardless the selected tests in TI) I was able to nail down the memory leak and fix it (well actually I knew where it was before but I wanted to improve the unit test experience for not only finding failed tests because of a failing CheckWhatever call but also because of a memory leak during its execution.
ReplyDeleteStefan Glienke I failed to mention that the leak detector has its own mechanism for registering ignored leaks. You can easily ignore all TRttiObjects. I used this technique while testing spring4d code. Just check out the LeakCheck.Utils.
ReplyDeleteStefan Glienke Why do people still use DUnit?? Take a look at DUnit2 - it support the ability to set an "acceptable leak amount" for a specific test. For example:
ReplyDeleteprocedure TTestTIThread.TMultiReadSingleWriteSynchronizer;
begin
AllowedMemoryLeakSize:= 24;
UVCLSynchronizer.BeginRead;
UVCLSynchronizer.EndRead;
UVCLSynchronizer.BeginWrite;
UVCLSynchronizer.EndWrite;
Check(True);
end;
Graeme Geldenhuys Because DUnit2 brings nothing to the table and it does not run on all platforms while DUnit does.
ReplyDeleteThat solution is just not acceptable. Saying "oh, I accept 24 bytes" to be leaked is hardly anything I can live with. And tracking the memory usage has nothing to do with the test framework but with the mechanism you are using in the memory monitor implementation and what Honza Rameš did there is just amazing. Now if I can register stuff from System.Rtti as accepted leaks that perfectly solves my problem even without running the tests twice - shame on me for not remembering because he probably told me about it some while ago.
...points to his first comment and rolls eyes... ;)
ReplyDeleteGraeme Geldenhuys DUnitX has the same issues it was built on top of the notion that just counting leaked bytes is enough and while it gives you some information and was built on top of available memory manager functions it isn't really pointing you to a solution. dUnit on the other hand uses more abstract interface (but still with the hint that it only counts leaked bytes) that allowed me to exploit it a little bit and count not bytes but leaked pointer count. When there are some the snapshots can be used to determine what memory contents was left un-freed. With the help of some code from FastMM (thanks to Pierre le Riche) and the scanner I implemented, you are able to get the whole picture, together with the ignores you can limit scope to only your code not the RTL or other frameworks you have no chance of fixing ;-). The real problem here is that not all testing frameworks are that easy plugable and dUnit is (with minor patch that allowed me to use different scanning techniques if needed) but this is not the problem of the leak-check library but rather of the frameworks themselves, but if someone asks me I would be happy to cooperate and reintegrate (improve) other testing frameworks with leakchecking support.
ReplyDeleteHonza Rameš Since I "ported" DUnit2 memory management to DUnitX, I would ask that you port yours to DUnitX, as DUnit2 seems to be dead and DUnitX seems to be the way of the future... ;-)
ReplyDeleteNicholas Ring DUnit2 still gets some contributions, and a FPC specific port is still going too. So I wouldn't call it dead just yet. DUnit (on SF.net) hasn't been touched since 2012, and no major improvements since at least 2008. So that is more dead than DUnit2. ;-)
ReplyDeleteNicholas Ring I sent you a hangout so we can talk about specifics. If this is a not a good form of communication please let me know...
ReplyDeleteHonza Rameš From what you said, it sounds like DUnit had some recent code changes, but the project on SourceForge.net doesn't show any changes. So where is the latest DUnit code hosted now, and probably the next question, why the split from the one on SF.net?
ReplyDeleteGraeme Geldenhuys No there isn't I just added the modified version to my repository. There are minor changes really but I wasn't able to get in touch with dUnit guys to reintegrate it back to main repo. All the required information can be found here: https://bitbucket.org/shadow_cs/delphi-leakcheck. Please let me know if any of the information there is difficult to follow.
ReplyDeleteGraeme Geldenhuys We have had this discussion in the past - I don't know about the FPC version, I have submitted bugs to DUnit2 on SF (with fixes!!!) and they are still sitting there, close to a year later...
ReplyDeleteHonza Rameš Thanks for the info, I'll take a look at the code. On a quick review, it looks similar to what FPC's built-in heaptrc unit does (normally activated with compiler parameter -gh). Until I look at your code in more detail, I can't be 100% sure though.
ReplyDeleteVincent Parrett should read Honza Rameš's comments on leak sanitizing in this thread and see if DUnitX can be improved on.
ReplyDeleteLars Fosdal I've had Honza Rameš leakcheck page open in a tab in chrome for a while now.. it's slowly moving to the left as I get through reading other stuff (I know, I'm old school!). The problem with leak checking in code is it's all using the same memory manager.. and as others have pointed out.. strings just f*ck leak tracking! I will try and find some time to look at it soon. unless someone else beats me too it (happy to take contributions, I kinda busy trying to get FinalBuilder 8 finished).
ReplyDeleteVincent Parrett my leakcheck library is able to ignore some types of leaks (all strings for example), there is also an option to ignore a bunch of leaks between two points of execution and also all managed fields for given class.
ReplyDeleteJust a note: I managed to add the DUnitX support to my LeakCheck library. Will post about it in a minute.
ReplyDeleteHonza Rameš That is awesome!
ReplyDeleteHonza Rameš pull request merged thx.
ReplyDeleteVincent Parrett cool thanks
ReplyDeleteHonza Rameš Vincent Parrett Very nice, thanks a lot! Now I have to figure out how to write my tests in such a way that the individual tests don't leak strings. Some of the units that I test produce log output and these are counted as leaks by the LeakCheck library. Quoting from above: they "f*ck leak tracking" :)
ReplyDeleteLübbe Onken you can either ignore string leaks completely or modify your logging to write the output somewhere and cleanup the strings, you can also modify your logging to ignore separate leaks or decorate the test case (or the LeakCheck test plugin) to ignore them in a cleaner way (take a look how status is ignored both for DUnit and DUnitX). LeakCheck offers several ignoring options.
ReplyDelete