Custom implementation of InputStream

Question

To send data to a file on my FTP server, I need to create a custom InputStream implementation that reads database data row by row, converts it to CSV and publishes it via its read() methods: from the database, I get a List<Application> object with the data. For each Application object, I want to create a line in the CSV file.

My idea is to load all the data in the constructor and then override the read method. Do I need to override all InputStream's methods? I tried googling for some examples but didn't succeed - could you eventually give me a link to one?

Solution 1

For possibly large data you can use com.google.common.io.FileBackedOutputStream from guava.

Javadoc: An OutputStream that starts buffering to a byte array, but switches to file buffering once the data reaches a configurable size.

Using out.getSupplier().getInput() you get your InputStream.

Solution 2

You only nead to implement the read() method without parameters. All other methods are implemented as calls to that method. For performance reasons (and even ease of implementation) it might be easier to implement the three-argument read() method instead and re-implement the no-args read() method in terms of that method.

Solution 3

Some very important points which I met when implementing my InputStream.

1. Override available(). As the Javadoc says:

   The available method for class InputStream always returns 0. This method should be overridden by subclasses.

   not overriding this method will causes that any tempt to test whether this stream is readable return false. For example, if you feed your inputStream to a inputStreamReader, this reader will always return false when you invoke reader.ready().

2. return -1 in the read(). The doc didn't emphasize it:

   If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.

   if you choose to block read() when no data is available, you have to remember to return -1 at some situations. Not doing this may causes that another read(byte b[], int off, int len) blocks for the following code in the source:

   for (; i < len ; i++) {// default len is a relative large number (8192 - readPosition)
       c = read();
       if (c == -1) {
           break;
       }
       b[off + i] = (byte)c;
   }
   

   And this causes some(if not all) high level read block, like a reader's readLine(), read() etc.

Solution 4

Why do you need a custon inputstream? why not just write the csv data as you generate it to the outputstream being written to the ftp server?

Solution 5

There's absolutely no need to create a custom InputStream. Use ByteArrayInputStream, something like this:

public static InputStream createStream(){
    final String csv = createCsvFromDataBaseValues();
    return new ByteArrayInputStream(csv.getBytes());
}

Especially given this quote:

My idea is to load all the data in the constructor and then override the read method.

If you do it like this, you gain absolutely nothing by implementing a custom InputStream. It's pretty much equivalent to the approach I outlined above.

View more solutions

Comments

[John Manak]

it's probably in tens of thousands of Application objects; each produces a line in the CSV file of roughly 100 characters. would it be a good idea to produce such a long string in memory or would it be better to make a temporary file and transfer it when it's done?

[Sean Patrick Floyd]

@John either way, I'd create a common interface that I'd pass to the Application objects, and I'd experiment with both StringBuilder- and File backed versions of this interface.

[John Manak]

so could I first use it as an output stream, submitting all my data, and then get an input stream providing all that data to my FTP client?

[maaartinus]

Yes and no, it should be easier than doing it on the fly. If there's any error when reading, nothing gets sent. You can display the whole content while debugging if you want to. It's up to you.

[Mr Ed]

According to the documentation, you need to implement the available() method also.

[Joachim Sauer]

Mr. Ed: it clearly says "should". And since you should not rely on available() to know how many bytes to ready anyway, I'd say that the stream will work just fine with the default implementation.

[Mr Ed]

The default implementation of available() returns zero. If some function relied on that to know if they could read and get something or not, that function won't work unless available() works.

[Joachim Sauer]

@Mr Ed: yes, that's true. But such a function would be broken: available() is only defined to be an estimate. And relying on an estimate to be accurate is a mistake, in my opinion.

[Mr Ed]

The latest documentation says: "...The available method for class InputStream always returns 0. This method should be overridden by subclasses. Returns: an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking or 0 when it reaches the end of the input stream." Anyway, the only reason I mentioned it in the first place was because my program didn't work because I didn't define that method.

[Brent Bradburn]

It's a nice approach if you can partition the reads and writes into separate threads.

[jtahlborn]

@nobar, that's rarely useful. about the only time that's a useful thing to do is if the producer and consumer can both arbitrarily block on some other resources.

[stu]

my actual experience here is if you only implement read() bufferedinputstream will return short reads if you try to read past 256 characters, when I implemented available() it started working correctly.