Why is MaxListSize = MaxInt div 16? (Delphi 2007, see Classes.pas)
Why is MaxListSize = MaxInt div 16? (Delphi 2007, see Classes.pas)
I would have expected MaxInt div SizeOf(Pointer) or rather MaxCardinal div SizeOf(Pointer). With SizeOf(Pointer) = 4 for 32 bit applications.
I would have expected MaxInt div SizeOf(Pointer) or rather MaxCardinal div SizeOf(Pointer). With SizeOf(Pointer) = 4 for 32 bit applications.
What kind of list are you creating that needs more than 1 billion entries?
ReplyDeleteHm, that's not 1 billion but only 2^27 = 134,217,728 = 134 millions.
I'm guessing that the reason for this limit might be that ReallocMem (used in TList.SetCapacity) might fail much earlier than 1 Billion. Also, the rest of the program also needs some memory.
Ok, what are you doing that needs such large a list? Maybe there is a way to get away with a lot less.
I'm testing my dzLargeTextViewer tool with a dump of Wikipedia. It creates a List of Int64 (not a TList but my own TInt64List) to store the offsets of the line breaks. I have already used that tool to display an xml file > 2 GB but that file apparently had much longer and therefore fewer lines so it didn't hit that limit.
ReplyDeleteThe SetCapacity method (which I copied from TList) checks if the new capacity is > MaxListSize and raises an Exception "List capacity out of bounds." Since the list will always grow by 1/4 of the current capacity it bombs out even before it actually reaches that limit.
Since Int64 is 8 bytes, that limit makes more sense for your TInt64List than it does for TList. It's about half the available memory for 32 bit applications.
ReplyDeleteTo get around that limit, have you considered e.g. storing only every 16th line's offset and determine the offsets for the lines in between on demand? That would give you 2 billion lines. Still not infinite but it might be "enough for everybody" ;-).
You could even dynamically increase that "page size" the larger the file is or increase it with the number of lines. That would allow for nearly infinite file sizes.
And finally you could build that index in a file itself rather than in memory.
(Or you could just use an existing tool, but I guess you won't do that. ;-) )
Hm, some food for thought. Thanks. I'll sleep on it and continue tomorrow.
ReplyDeleteJFTR, at least in XE2 MaxListSize is marked as deprecated.
ReplyDelete