Discussions related to Visual Prolog
Loffy
Active Member
Posts: 47
Joined: 15 Aug 2019 11:32

Increase buffer size

Unread post by Loffy »

I am streaming (reading) a 2GB utf8 file in binary mode; doing some processing; and then streaming out the processed file.

Input "record size" is 1024 (8 bit) bytes as this suits my application.

Everything works, and the processing time is reasonable (294 seconds), though I think I can do better with minor tuning. Larger read and write buffers would probably help as a 2000 byte default buffer size does not fit well with my chosen "record" size.

See code below;

Code: Select all

        InStream = inputStream_file::openFileUtf8("File1Test.txt"),         InStream:setMode(stream::binary),         stdio::inputStream := InStream,         OutStream = outputStream_file::createUtf8("File9Test.txt"),         OutStream:setMode(stream::binary),         stdio::outputStream := OutStream,                 processing .......
I have tried searching for a predicate to increase the default buffer size, and I can see alternate file open predicates (non utf8) that can allocate custom buffer sizes (at least for the reading).

Can anyone point me to a way to increase my read and write streaming buffer sizes?

Regards,

Loffy
User avatar
Thomas Linder Puls
VIP Member
Posts: 1398
Joined: 28 Feb 2000 0:01

Re: Increase buffer size

Unread post by Thomas Linder Puls »

The default buffer size for both reading and writing to files is 8KB:

Code: Select all

class inputStream_file : inputStream     open core   constants     defaultBufferSize = 0x2000.
If your read in 8 bit text mode (e.g. utf8), the actual buffer size is only half, because we there is an approximate doubling when converting from 8 bit characters to utf16.

An NTFS disk has a sector size of 4KB and this is what has made us choose a default buffer size of 8KB, when reading in 8 bit mode you will read one sector at the time in utf16 and binary mode you will be reading 2 sectors at the time.

A corresponding story is the case for writing.

I doubt that you will gain much by changing the buffer size, and I am not sure that what seems better on one computer/disk will also be better on another.

Windows cashes file data, disc controllers may cache file data, disks them selves may cache file data. SSD disks behaves in an other way than old "spinning" disks.

But if you want to experiment then you will have to open the files using the predicates that takes "BufferSize" as argument. For reading:

Code: Select all

constructors     openFile : (string Filename, stream::mode Mode = unicode, fileSystem_api::accessPermit Access = fileSystem_api::permitRead,         byteCount BufferSize = defaultBufferSize).     % @short Opens file #Filename for input.     % @detail Opens file with specified name #Filename for input, setting stream position to the file start (skipping byte-order mark).     % #Mode defaults to unicode.<\br>     % #Access defults to permitRead.<\br>     % #BufferSize defaults to 0x2000 (it must be even)     % @exception The fileSystem_exception::cannotCreate exception is raised if the specified file cannot be created/opened     % @end
Regards Thomas Linder Puls
PDC
Loffy
Active Member
Posts: 47
Joined: 15 Aug 2019 11:32

Re: Increase buffer size

Unread post by Loffy »

Thomas,

Thank you for your prompt response. I misunderstood the 2000 implied buffer size. I am still a long way away from my final system concept and I will now go to the next step. If you do not think I will get much gain from playing with buffer sizes then I accept that advice.

If I ever get to the point that I need to gain every last second of performance I will go back and play later. That may be some time away as I am at best 20% of the way through my conceptual design, and if I believe the old mantra (and I do) that once you have a final concept that works, you are at best only 10% of the way to a "product". If I am honest, I reckon I am only 1% of the way.

Regards,

Loffy
Post Reply