It seems I found another piece to the puzzle of generics causing big binaries and slow compile times:
It seems I found another piece to the puzzle of generics causing big binaries and slow compile times:
https://quality.embarcadero.com/browse/RSP-18080
It looks like the compiler puts more into a dcu than it should/have to. That causes some cascading effect if you have a project with many units that all suffer from this.
In my case I have a project with roughly 53k loc that compiles for 22sec and causes the IDE to go from 300MB to a little over 1GB RAM usage. But not only that it also causes the IDE to become sluggish right after that (yeah yeah, I can use msbuild to not let the IDE suffer but that's not the point). It goes away when I close the project but working with it and compiling even more projects makes it worse. The project is producing 73MB of dcu files that mostly contain the same stuff over and over (most of it being generic collection classes that are not even used but being pulled in because of the bug reported). Some of them being some couple hundred lines of trivial classes doing some list stuff becoming >2MB dcu files.
Sure, once the binary gets linked duplicates are being removed (not completely folded but if 2 units used a TList there is only one remaining) but I feel most of the performance here is being wasted for emitting useless code into the dcus (and compiler memory) that later is being thrown away again and is also causing stress for the linker (you can typically see that generic heavy projects hang on the linking phase way longer than "classic" code).
P.S. Yes, this is using Spring4D collections and I am working on some fix to mitigate this problem a bit but there is only so much I can do :(
https://quality.embarcadero.com/browse/RSP-18080
It looks like the compiler puts more into a dcu than it should/have to. That causes some cascading effect if you have a project with many units that all suffer from this.
In my case I have a project with roughly 53k loc that compiles for 22sec and causes the IDE to go from 300MB to a little over 1GB RAM usage. But not only that it also causes the IDE to become sluggish right after that (yeah yeah, I can use msbuild to not let the IDE suffer but that's not the point). It goes away when I close the project but working with it and compiling even more projects makes it worse. The project is producing 73MB of dcu files that mostly contain the same stuff over and over (most of it being generic collection classes that are not even used but being pulled in because of the bug reported). Some of them being some couple hundred lines of trivial classes doing some list stuff becoming >2MB dcu files.
Sure, once the binary gets linked duplicates are being removed (not completely folded but if 2 units used a TList
P.S. Yes, this is using Spring4D collections and I am working on some fix to mitigate this problem a bit but there is only so much I can do :(
Delphi Generics, good idea, badly implemented.
ReplyDeleteAttila Kovacs Unfortunately as so often you are missing the point. Put some methods into TTest and you will see how code will show up. The fact that Unit1.dcu contains TTest and TTest is the issue because it does nowhere call the method that contains them. Change the two TTest.Stuff overloads to StuffA and StuffB and you will see how one of them (the one you don't call) will go away from Unit1 as well as the TTest inside of it.
ReplyDeleteAttila Kovacs I don't need to convince you. I have the data and when some EMBT developer needs more information or a real world example I can show him.
ReplyDeleteStefan Glienke is the real world example inside that ZIP file?
ReplyDeleteAttila Kovacs maybe the bug does not make 100% of the clumsiness, but as Stefan said, it is a piece of a whole puzzle.
ReplyDeleteThe performance difference between Delphi XE and Delphi Berlin (that are the two latest Delphis I use) is remarkable, I am happy about every found bug which decreases the performance.
Nick Hodges The real world example is on my computer at work and cannot be put into some public issue tracker. However I put some other example using Spring4D that should make the problem more visible (although I think the original one shows the issues despite not producing a huge dcu)
ReplyDeleteStefan Glienke Does using DDevExtensions "Release compiler unit cache..." improve the IDE after compilation?
ReplyDeleteDavid Hoyle A bit but it makes debugging impossible because - well the dcu cache is cleared ;)
ReplyDeleteAttila Kovacs There is nothing new to find out. The thing is that the compiler seems to not do some linking steps that would be necessary on dcu level but only after linking the final binary which results in these enormous dcus causing all the problems I mentioned in the ticket.
ReplyDeleteI don't have enough knowledge about the compiler architecture to tell if that would be even possible or not. Maybe Allen Bauer or some developer at Embarcadero Technologies does.
Attila Kovacs TArray does not contain any executable code. And the code dealing with dynamic array is non generic and from System.pas and calls to those routines is automatically inserted by the compiler where being used.
ReplyDelete