Need help understanding Stream.Read()

Question

I am a little confused as to the steps of reading a file into buffer gradually.

from the MSDN docs

public abstract int Read(
    byte[] buffer,
    int offset,
    int count
)

source from C# Examples

FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
try
{
    int length = (int)fileStream.Length;  // get file length
    buffer = new byte[length];            // create buffer
    int count;                            // actual number of bytes read
    int sum = 0;                          // total number of bytes read

    // read until Read method returns 0 (end of the stream has been reached)
    while ((count = fileStream.Read(buffer, sum, length - sum)) > 0)
        sum += count;  // sum is a buffer offset for next reading

can I say that the line fileStream.Read(buffer, sum, length - sum) reads as "Read fileStream from sum (offset) to length - sum (total bytes to read) into buffer". OK so at the start, when sum = 0, I will effectively read the whole fileStream into buffer in 1 short, but I think this is not the case. Maybe Read() reads whatever it possibly can into buffer? then returns so that you can Read() it again? I am a little confused.

Solution 1

Read will read whatever's available (blocking until something is available) but there may not be enough data ready to fill the buffer to start with.

Imagine downloading data over the network - there may be megabytes of data to download, but only some of it is available to start with. So you need to keep calling Read() until you've read as much as you want.

Stream.Read will read at most the number of bytes you've asked for, but can easily read less. Admittedly for local file streams I suspect it always reads as much as you've asked for unless the file is shorter, but that's not true of streams in general, and I don't believe it's guaranteed even for local file streams.

Solution 2

The Read method will read at least one byte, and at most the number of bytes specified.

The method will usually return all data that is currently available. If the stream is for example coming over the internet it will usually return what it has recieved to far, and for a file stream it will usually return the entire file.

However, it's up to the implementation to decide what the best behaviour is. It might for example first return what it can get from the file cache, which can be returned immediately, and let you do another call to get the data that requires an actual disk read.

When using the Read method, you should always use a loop so that you are sure to get all the data. It might not seem neccesary if the first call appears to always return all data, but there might be situations where it doesn't.

Comments

[Jiew Meng]

Hmm, then somehow, fileStream.Length will be set to the actual total length? not just what is available?

[Jiew Meng]

Same question posed to @Jon Skeet: somehow, fileStream.Length will be set to the actual total length? not just what is available?

[Jon Skeet]

@jiewmeng: Well, fileStream.Length will be the length of the file at the time you read the length. The file could change length between then and the time you read the data though.

[Jiew Meng]

also I wonder is the buffer something to do with reading the file progressively so that the app will not use too much memory? I doubt it tho ... just just checking

[Jiew Meng]

then i guess it will only increase? also I wonder is the buffer something to do with reading the file progressively so that the app will not use too much memory? I doubt it tho ... just just checking

[Jon Skeet]

@jlewmeng: It depends on how you've opened the file in terms of sharing, whether another app could truncate the file while you're reading it. The idea of streaming data does indeed allow you to read progressively without requiring everything to be either created or in memory at the same time. Don't just think on the level of files though - think of potentially infinite streams of data, etc.

[Guffa]

@jiewmeng: The Length property will be set to the total length if it's available. For a stream with unknown length the property is unavailable, and throws NotSupportedException. Yes, the reason for this way of reading a stream is that it can be read progressively, either because reading the entire file uses too much memory, or because the entire stream is not available yet.

[knocte]

@JonSkeet: you say blocking until something is available, but MSDN says ReadBlock is a blocking version of Read. which implies that Read() doesn't block, but ReadBlock() does block.

[Jon Skeet]

@knocte: That's just poor documentation IMO. StreamReader.ReadBlock (which is what I assume you're referring to) will block until either all the requested data has been read, or the end of the stream has been reached. Read will block until some data has been read - it still blocks.