Do all IBX variants have memory leaks with blob cache?
Do all IBX variants have memory leaks with blob cache?
We are using Mercury Database Objects, that is fork of IBX and I just noticed, that when using it to copy record to different database memory usage continued to grow and grow. And it was unidirectional dataset with "select * from tablename" query. And the table had 1 memo field. And checking around I discovered, that every time blob stream is created it checks record buffer in blob cache part for existing internal stream, that is used. And if it does not have it already, it creates new, adds reference to record buffer and adds it FBlobStreamList. And when fetching new record those references in record buffer are set to nil, but they remain in the FBlobStreamList, that is not cleared until closing dataset. So for every row in dataset new stream object is created.
I checked another fork of IBX and it seems to have same problem. And same is true for IBX from Delphi XE.
Obvious solution would be to call ClearBlobCache, when fetching new record. But there must be reason, why it was implemented like that.... So what am I missing.
We are using Mercury Database Objects, that is fork of IBX and I just noticed, that when using it to copy record to different database memory usage continued to grow and grow. And it was unidirectional dataset with "select * from tablename" query. And the table had 1 memo field. And checking around I discovered, that every time blob stream is created it checks record buffer in blob cache part for existing internal stream, that is used. And if it does not have it already, it creates new, adds reference to record buffer and adds it FBlobStreamList. And when fetching new record those references in record buffer are set to nil, but they remain in the FBlobStreamList, that is not cleared until closing dataset. So for every row in dataset new stream object is created.
I checked another fork of IBX and it seems to have same problem. And same is true for IBX from Delphi XE.
Obvious solution would be to call ClearBlobCache, when fetching new record. But there must be reason, why it was implemented like that.... So what am I missing.
Try committing transactions after reading for example dozen of blobs.
ReplyDelete"that is not cleared until closing dataset" - so, there is no leak. There is memory freeing happening in another moment, that one you imagine. But it is freed nonetheless.
"So for every row in dataset new stream object is created."
Unless you have bi-directional dataset where you go back and forth, this seems legit. Every new BLOB - every new stream.
Granted, for bi-directional fully in-memory cached dataset it would make sense to keep all the data ever fetched. And that would mean, the Buffers should grow large enough to NEVER (until closing or re-reading) be cleared actually. There are ActiveBuffers - the frame of all memory-cached rows - and there is RecordBuffer which is one current row of all those.
Arioch The Ok, I was checking the code more and record cache is probably one of the reasons. Although for unidirectional dataset it is only 2 rows. And it causes not just higher memory usage but also serious slowdown in case that every field needs to be read.
ReplyDeleteif you need topmost speed then BLOBs are not the best choice. They enforce several extra network roundtrips for reading. if their data size allows it, try to CAST them to VARCHAR in the select. Depending on their charset, max length of the VARCHAR can be from 32 KByte ro 8KChars
ReplyDeleteArioch The In normal use there is no issue. There are just some cases, when data is transferred to different database. And in that case there is a problem, because too many blobs are cached. And I don't think, that there is a good way to solve it universally. Best I can think of is using automatic clearing of blob cache in particular use case.
ReplyDeleteIBX was primary for-GUI library. But if you do not want to cache blobs - then you can call "Clear Cache" every row. or very 10th row, every 100th row, etc.
ReplyDeleteOr you can try some low-fat close-to-metal library like UIB. It provides minimal features, for example its primary dataset does not even have blob fields, but it is much closer to FB API and perhaps would give you - after learning curve - more performance.
I solved the problem by adding support for disabling blob caching (which causes external DSBlobStream to own internal BlobStream instead of adding it to blob cache). Should be safest solution. 40 minute table pumping decreased to less then 4 minutes.
ReplyDelete