basic_fstream and large files

[Eric Twietmeyer]

Hello,

I have code written using the basic fstream object on a Win32 system. This
is of course simply a typedef for basic_fstream<char, char_traits<char> >.

This object can not be used to write or read files that are larger than 2Gb
in size, as it would appear (from looking at the iosfwd header) that the
underlying streamoff and streamsize types are typedef'd to long and int. On
Win64 systems, these are typedef'd to __int64, so it seems that on Win64
systems fstream can in fact be used to read and write large files.

However, is there a way I can define a new type and define a char_traits for
this type so that I can use basic_fstream to read and write large files on
Win32? Looking into the std c++ iostream headers, it is not clear that this
is in fact doable, as many things seem to depend on these basic streamoff
and streamsize typedefs, which are not mutable in my own class.

I would like to know whether I am reading this correctly, or whether people
have had success in doing this, i.e. creating a char_traits type that when
used with basic_fstream will allow for large (>2Gb) file reading and
writing.

Thanks,

-Eric Twietmeyer

(MSDN no-spam alias: (e-mail address removed))

 

[Bo Persson]

Eric Twietmeyer wrote:
:: Hello,
::
:: I have code written using the basic fstream object on a Win32
:: system. This is of course simply a typedef for basic_fstream<char,
:: char_traits<char> >.
::
:: This object can not be used to write or read files that are larger
:: than 2Gb in size, as it would appear (from looking at the iosfwd
:: header) that the underlying streamoff and streamsize types are
:: typedef'd to long and int. On Win64 systems, these are typedef'd to
:: __int64, so it seems that on Win64 systems fstream can in fact be
:: used to read and write large files.
::
:: However, is there a way I can define a new type and define a
:: char_traits for this type so that I can use basic_fstream to read
:: and write large files on Win32? Looking into the std c++ iostream
:: headers, it is not clear that this is in fact doable, as many things
:: seem to depend on these basic streamoff and streamsize typedefs,
:: which are not mutable in my own class.
::
:: I would like to know whether I am reading this correctly, or whether
:: people have had success in doing this, i.e. creating a char_traits
:: type that when used with basic_fstream will allow for large (>2Gb)
:: file reading and writing.
::

You CAN read and write larger files, as long as you do it sequentially. The
size of streamoff limits the amount of positioning you can do. Do you really
do seeks with offsets >2GB?


Bo Persson

 

[Eric Twietmeyer]

Eric Twietmeyer wrote:
:: Hello,
::
:: I have code written using the basic fstream object on a Win32
:: system. This is of course simply a typedef for basic_fstream<char,
:: char_traits<char> >.
::
:: This object can not be used to write or read files that are larger
:: than 2Gb in size, as it would appear (from looking at the iosfwd
:: header) that the underlying streamoff and streamsize types are
:: typedef'd to long and int. On Win64 systems, these are typedef'd to
:: __int64, so it seems that on Win64 systems fstream can in fact be
:: used to read and write large files.
::
:: However, is there a way I can define a new type and define a
:: char_traits for this type so that I can use basic_fstream to read
:: and write large files on Win32? Looking into the std c++ iostream
:: headers, it is not clear that this is in fact doable, as many things
:: seem to depend on these basic streamoff and streamsize typedefs,
:: which are not mutable in my own class.
::
:: I would like to know whether I am reading this correctly, or whether
:: people have had success in doing this, i.e. creating a char_traits
:: type that when used with basic_fstream will allow for large (>2Gb)
:: file reading and writing.
::

You CAN read and write larger files, as long as you do it sequentially.
The size of streamoff limits the amount of positioning you can do. Do you
really do seeks with offsets >2GB?


Bo Persson

Yes, unfortunately, I do in fact need to be able to offset in the stream to
locations greater than 2Gb from the current point.

I'm also just curious how one may specialize the basic stream / buf objects
using different char_traits, whether that is actually doable given this
implementation provided with DevStudio, or whether it is expected that only
char_traits<char> and char_traits<wchar_t> would be used, and no others.

-Eric

 

[Bo Persson]

Eric Twietmeyer wrote:
:: ::: Eric Twietmeyer wrote:
::::: Hello,
:::::
::::: I have code written using the basic fstream object on a Win32
::::: system. This is of course simply a typedef for
::::: basic_fstream<char, char_traits<char> >.
:::::
::::: This object can not be used to write or read files that are larger
::::: than 2Gb in size, as it would appear (from looking at the iosfwd
::::: header) that the underlying streamoff and streamsize types are
::::: typedef'd to long and int. On Win64 systems, these are typedef'd
::::: to __int64, so it seems that on Win64 systems fstream can in fact
::::: be used to read and write large files.
:::::
::::: However, is there a way I can define a new type and define a
::::: char_traits for this type so that I can use basic_fstream to read
::::: and write large files on Win32? Looking into the std c++ iostream
::::: headers, it is not clear that this is in fact doable, as many
::::: things seem to depend on these basic streamoff and streamsize
::::: typedefs, which are not mutable in my own class.
:::::
::::: I would like to know whether I am reading this correctly, or
::::: whether people have had success in doing this, i.e. creating a
::::: char_traits type that when used with basic_fstream will allow for
::::: large (>2Gb) file reading and writing.
:::::
:::
::: You CAN read and write larger files, as long as you do it
::: sequentially. The size of streamoff limits the amount of
::: positioning you can do. Do you really do seeks with offsets >2GB?
:::
:::
::: Bo Persson
:::
:::
::
:: Yes, unfortunately, I do in fact need to be able to offset in the
:: stream to locations greater than 2Gb from the current point.

Too bad.

The Band-Aid solution is to do multiple seeks, 2 GB at a time. Works for
moderate file sizes.

::
:: I'm also just curious how one may specialize the basic stream / buf
:: objects using different char_traits, whether that is actually doable
:: given this implementation provided with DevStudio, or whether it is
:: expected that only char_traits<char> and char_traits<wchar_t> would
:: be used, and no others.

It is possible, but hard. It will probably not help you here, anyway.

Eventually, all the seeks will probably come down to a C level fseek(),
which also takes a long parameter. The real solution is to have a 'long'
type that is actually sort-of-long, and not just a different name for 'int'.
A bit late for that now, unfortunately.


Bo Persson