TStream does not allow files > 2 GB.
TStream does not allow files > 2 GB.
Why use an integer to count the length of the buffer?
Even in Win32 I can have buffers >= 3 GB.
Please vote for https://quality.embarcadero.com/browse/RSP-18866
I used the following code as a workaround to save data exactly 2GB in length.
procedure TForm75.AncestorCountsThreadsAllDone;
var
Stream: TFileStream;
i,x,y,z: integer;
FileName: string;
BytesWritten: cardinal;
Size: cardinal;
begin
//Wait for threads to finish
Filename:= TPath.GetDocumentsPath;
FileName:= Filename + '\UCT\ResearchStats';
ForceDirectories(Filename);
Label1.Caption:= 'Progress: writing results to disk';
Application.ProcessMessages;
Filename:= Filename + '\4x4_ancestor_counts_per_pixel.bin';
Stream:= TFileStream.Create(Filename, fmCreate);
Size:= Length(AncestorCountTable4x4) * SizeOf(Cardinal);
WriteFile(Stream.Handle, AncestorCountTable4x4[0], Size, BytesWritten, nil);
if (BytesWritten < Size) then begin
Size:= Size - BytesWritten;
WriteFile(Stream.Handle, PByte(@AncestorCountTable4x4[0])[BytesWritten], Size - BytesWritten, BytesWritten, nil);
end;
Stream.Free;
Label1.Caption:= 'All done';
end;
Just using the TFileStream as is causes a hang after saving the data. It looks like all the data has been saved, but I can't be sure.
https://quality.embarcadero.com/browse/RSP-18866
Why use an integer to count the length of the buffer?
Even in Win32 I can have buffers >= 3 GB.
Please vote for https://quality.embarcadero.com/browse/RSP-18866
I used the following code as a workaround to save data exactly 2GB in length.
procedure TForm75.AncestorCountsThreadsAllDone;
var
Stream: TFileStream;
i,x,y,z: integer;
FileName: string;
BytesWritten: cardinal;
Size: cardinal;
begin
//Wait for threads to finish
Filename:= TPath.GetDocumentsPath;
FileName:= Filename + '\UCT\ResearchStats';
ForceDirectories(Filename);
Label1.Caption:= 'Progress: writing results to disk';
Application.ProcessMessages;
Filename:= Filename + '\4x4_ancestor_counts_per_pixel.bin';
Stream:= TFileStream.Create(Filename, fmCreate);
Size:= Length(AncestorCountTable4x4) * SizeOf(Cardinal);
WriteFile(Stream.Handle, AncestorCountTable4x4[0], Size, BytesWritten, nil);
if (BytesWritten < Size) then begin
Size:= Size - BytesWritten;
WriteFile(Stream.Handle, PByte(@AncestorCountTable4x4[0])[BytesWritten], Size - BytesWritten, BytesWritten, nil);
end;
Stream.Free;
Label1.Caption:= 'All done';
end;
Just using the TFileStream as is causes a hang after saving the data. It looks like all the data has been saved, but I can't be sure.
https://quality.embarcadero.com/browse/RSP-18866
TStream have been upgraded, great!!
ReplyDeleteRead64/Write64 it's available from 10.2, internally its a chunk read/write of 512 MB.
If you are using large memory buffers and need persistence on disc try to use File Mapping for best performance.
ReplyDeleteYou can use this approach to map parts of the file to memory and use a class to access Indexes elements, mapping and unmapping a range of data, for example to use large bitmaps and save memory use and not only with images of more than >2GB
I fail to see why we need a separate Write64 method. Why not just update the standard Write method to use NativeUint. I don't need memory mapped files. TFileStream is fast enough for my purposes. I guess there is an issue with overloading that prevented the Emba team from just adding a Int64 overloaded version of write. Too bad overloading is semi-broken.
ReplyDelete