IOException: error=7, Argument list too long with large command line

Question

I have a requirement to call Unix command from Java. The code is as follows.

String strCmd = "iconv -f "+ strSrcEncoding+" -t "+ strTgtEncoding + " <<< "+"\""+InputMessage+"\"";

String commands[] = {"bash","-c",strCmd};
Process proc = Runtime.getRuntime().exec(commands);
String strData = null;

// Get the error Stream
BufferedReader brStdError = new BufferedReader(new
        InputStreamReader(proc.getErrorStream()));
StringBuilder sbError = new StringBuilder();


// read any errors from the attempted command
while ((strData = brStdError.readLine()) != null) {
    sbError.append(strData);
}

if(sbError.toString().isEmpty())
    return "success";
else
    return "failure"+sbError.toString();

When i pass a large data am getting an error

"bash": java.io.IOException: error=7, Argument list too long

I tried using echo instead as below

echo <<InputMessage>> | iconv -f utf8 -t Cp930

But was getting the same error

Am i missing anything?

Solution 1

There's a limit to much data you can pass to a program on the command line. If you have a lot of data you should pass it to iconv using its standard input stream, i.e. by writing it to proc.getOutputStream. Here's an example:

OutputStream os = proc.getOutputStream();
os.write(InputMessage.getBytes(strSrcEncoding));
os.close();

Unfortunately, for longer messages this will also fail because iconv will fill the buffer of the pipe it is connected to, and wait for the pipe to be read. The solution is writing data to iconv from one thread and reading the output from another. Dealing with external processes is a hassle because of all these pit-falls. You can read more about it here: http://www.javaworld.com/jw-12-2000/jw-1229-traps.html Apache commons exec helps you deal with some of them.

On the other hand, why are you using iconv? You know Java has good library support for most character encodings, cp930 included?

Solution 2

This happens, when the size of the arguments exceeds the allowed size. And this depends on the platform.

To verify the maximum limit, run

$ getconf ARG_MAX

For mac osx environment, I can see 262144 is the limit.

Limit for each platform is different, that can be found here: http://www.in-ulm.de/~mascheck/various/argmax/

To check the environment string:

$ env