Java HTTP server sending chunked response
13,158
Solution 1
Solved, not sure why, but removing the header:
Transfer-Encoding: chunked
And also the chunk lengths at the beginning of each chunk resolved the issue, I still write the data in 768 byte chunks. This works reliably and very well.
Not sure why I had to do this.
Final method to produce chunks from data string:
public static String[] arystrChunkData(String strData) {
int intChunks = (strData.length() / CHUNK_THRESHOLD_BYTESIZE) + 1;
String[] arystrChunks = new String[intChunks];
int intLength = strData.length(), intPos = 0;
for( int c=0; c<arystrChunks.length; c++ ) {
if ( intPos < intLength ) {
//Extract a chunk from the data
int intEnd = Math.min(intLength, intPos + CHUNK_THRESHOLD_BYTESIZE);
arystrChunks[c] = strData.substring(intPos, intEnd);
intPos = intEnd;
}
}
return arystrChunks;
}
Loop to write chunks, no lengths at the beginning and no 0 byte at the end of the chunks required:
String[] arystrChunks = arystrChunkData(strResponse);
for( String strChunk : arystrChunks ) {
if ( strChunk != null ) {
out.write(strChunk.getBytes());
}
}
Solution 2
As I commented already, there is NOT an official limit on HTTP response size. TCP does this work for you. However, you can always configure your web server to implement such a policy by setting Content-Length :: 32 bit Integer max size or 64 bit for modern browsers (see here).
Technically, you can have unlimited responses using Chunked Transfer as you state on your post. In theory, this is used to bypass the max Content-Length.
Most commonly and if there is such a requirement for a huge JSON file (at least some MBs in size), you can use some sort of pagination logic via sequential AJAX requests. In your case, you could split your big JSON data to chunks programmatically and send each one via another AJAX request. Then, let Javascript perform the merging task.
Typically, a JSON response of some MB in size will load successfully on any browser. I suggest you take a look on this article; it is 3-years old, but I guess things are even better nowadays.
In short, the above benchmark states that JSON of size less than 35 MB will probably load successfully on any modern desktop browser. This, however, may not be the case for mobile browsers. For instance, there are some reports for mobile safari limitations on >10MB Json files.
Comments
-
SPlatten almost 2 years
I am working on a Java application which has a built in HTTP server, at the moment the server is implemented using ServerSocketChannel, it listens on port 1694 for requests:
msvrCh = ServerSocketChannel.open(); msvrCh.socket().bind(new InetSocketAddress(mintPort)); msvrCh.configureBlocking(false);A thread is installed to manage requests and responses:
Thread thrd = new Thread(msgReceiver); thrd.setUncaughtExceptionHandler(exceptionHandler); thrd.start();The thread is quite simple:
Runnable msgReceiver = new Runnable() { @Override public void run() { try{ while( !Thread.interrupted() ) { //Sleep a short period between checks for new requests try{ Thread.sleep(DELAY_BETWEEN_ACCEPTS); } catch(Exception ex) { ex.printStackTrace(); } SocketChannel cliCh = msvrCh.accept(); if ( blnExit() == true ) { break; } if ( cliCh == null ) { continue; } processRequest(cliCh.socket()); } } catch (IOException ex) { ex.printStackTrace(); } finally { logMsg(TERMINATING_THREAD + "for accepting cluster connections", true); if ( msvrCh != null ) { try { msvrCh.close(); } catch (IOException ex) { ex.printStackTrace(); } msvrCh = null; } } } };The main bulk of the code for dealing with the response is in the function processRequest:
private void processRequest(Socket sck) { try { //AJAX Parameters final String AJAX_ID = "ajmid"; //The 'Handler Key' used to decode response final String HANDLER_KEY = "hkey"; //Message payload final String PAYLOAD = "payload"; //Post input buffer size final int REQUEST_BUFFER_SIZE = 4096; //Double carriage return marks the end of the headers final String CRLF = "\r\n"; BufferedReader in = new BufferedReader(new InputStreamReader(sck.getInputStream())); String strAMID = null, strHKey = null, strRequest; char[] chrBuffer = new char[REQUEST_BUFFER_SIZE]; StringBuffer sbRequest = new StringBuffer(); eMsgTypes eType = eMsgTypes.UNKNOWN; clsHTTPparameters objParams = null; int intPos, intCount; //Extract the entire request, including headers if ( (intCount = in.read(chrBuffer)) == 0 ) { throw new Exception("Cannot read request!"); } sbRequest.append(chrBuffer, 0, intCount); strRequest = sbRequest.toString(); //What method is being used by this request? if ( strRequest.startsWith(HTTP_GET) ) { //The request should end with a HTTP marker, remove this before trying to interpret the data if ( strRequest.indexOf(HTTP_MARKER) != -1 ) { strRequest = strRequest.substring(0, strRequest.indexOf(HTTP_MARKER)).trim(); } //Look for a data marker if ( (intPos = strRequest.indexOf(HTTP_DATA_START)) >= 0 ) { //Data is present in the query, skip to the start of the data strRequest = strRequest.substring(intPos + 1); } else { //Remove the method indicator strRequest = strRequest.substring(HTTP_GET.length()); } } else if ( strRequest.startsWith(HTTP_POST) ) { //Discard the headers and jump to the data if ( (intPos = strRequest.lastIndexOf(CRLF)) >= 0 ) { strRequest = strRequest.substring(intPos + CRLF.length()); } } if ( strRequest.length() > 1 ) { //Extract the parameters objParams = new clsHTTPparameters(strRequest); } if ( strRequest.startsWith("/") == true ) { //Look for the document reference strRequest = strRequest.substring(1); eType = eMsgTypes.SEND_DOC; } if ( objParams != null ) { //Transfer the payload to the request String strPayload = objParams.getValue(PAYLOAD); if ( strPayload != null ) { byte[] arybytPayload = Base64.decodeBase64(strPayload.getBytes()); strRequest = new String(arybytPayload); strAMID = objParams.getValue(AJAX_ID); strHKey = objParams.getValue(HANDLER_KEY); } } if ( eType == eMsgTypes.UNKNOWN && strRequest.startsWith("{") && strRequest.endsWith("}") ) { //The payload is JSON, is there a type parameter? String strType = strGetJSONItem(strRequest, JSON_LBL_TYPE); if ( strType != null && strType.length() > 0 ) { //Decode the type eType = eMsgTypes.valueOf(strType.toUpperCase().trim()); //What system is the message from? String strIP = strGetJSONItem(strRequest, JSON_LBL_IP) ,strMAC = strGetJSONItem(strRequest, JSON_LBL_MAC); if ( strIP != null && strIP.length() > 0 && strMAC != null && strMAC.length() > 0 ) { //Is this system known in the cluster? clsIPmon objSystem = objAddSysToCluster(strIP, strMAC); if ( objSystem != null ) { //Update the date/time stamp of the remote system objSystem.touch(); } //This is an internal cluster message, no response required return; } } } String strContentType = null, strRespPayload = null; OutputStream out = sck.getOutputStream(); byte[] arybytResponse = null; boolean blnShutdown = false; out.write("HTTP/1.0 200\n".getBytes()); switch( eType ) { case SEND_DOC: if ( strRequest.length() <= 1 ) { strRequest = HTML_ROOT + DEFAULT_DOC; } else { strRequest = HTML_ROOT + strRequest; } logMsg("HTTP Request for: " + strRequest, true); if ( strRequest.toLowerCase().endsWith(".css") == true ) { strContentType = MIME_CSS; } else if ( strRequest.toLowerCase().endsWith(".gif") == true ) { strContentType = MIME_GIF; } else if ( strRequest.toLowerCase().endsWith(".jpg") == true ) { strContentType = MIME_JPG; } else if ( strRequest.toLowerCase().endsWith(".js") == true ) { strContentType = MIME_JS; } else if ( strRequest.toLowerCase().endsWith(".png") == true ) { strContentType = MIME_PNG; } else if ( strRequest.toLowerCase().endsWith(".html") == true || strRequest.toLowerCase().endsWith(".htm") == true ) { strContentType = MIME_HTML; } File objFile = new File(strRequest); if ( objFile.exists() == true ) { FileInputStream objFIS = new FileInputStream(objFile); if ( objFIS != null ) { arybytResponse = new byte[(int)objFile.length()]; if ( objFIS.read(arybytResponse) == 0 ) { arybytResponse = null; } objFIS.close(); } } break; case CHANNEL_STS: strRespPayload = strChannelStatus(strRequest); strContentType = MIME_JSON; break; case CLUSTER_STS: strRespPayload = strClusterStatus(); strContentType = MIME_JSON; break; case MODULE_STS: strRespPayload = strModuleStatus(strRequest); strContentType = MIME_JSON; break; case NETWORK_INF: strRespPayload = strNetworkInfo(strRequest); strContentType = MIME_JSON; break; case NODE_STS: strRespPayload = strNodeStatus(strRequest); strContentType = MIME_JSON; break; case POLL_STS: strRespPayload = strPollStatus(strRequest); strContentType = MIME_JSON; break; case SYS_STS: //Issue system status strRespPayload = strAppStatus(); strContentType = MIME_JSON; break; case SHUTDOWN: //Issue instruction to restart system strRespPayload = "Shutdown in progress!"; strContentType = MIME_PLAIN; //Flag that shutdown has been requested blnShutdown = true; break; default: } if ( strRespPayload != null ) { //Convert response string to byte array arybytResponse = strRespPayload.getBytes(); System.out.println("[ " + strRespPayload.length() + " ]: " + strRespPayload); //HACK } if ( arybytResponse != null && arybytResponse.length > 0 ) { if ( strContentType == MIME_JSON ) { String strResponse = "{"; if ( strAMID != null ) { //Include the request AJAX Message ID in the response if ( strResponse.length() > 1 ) { strResponse += ","; } strResponse += "\"" + AJAX_ID + "\":" + strAMID; } if ( strHKey != null ) { if ( strResponse.length() > 1 ) { strResponse += ","; } strResponse += "\"" + HANDLER_KEY + "\":\"" + strHKey + "\""; } if ( strResponse.length() > 1 ) { strResponse += ","; } strResponse += "\"payload\":" + new String(arybytResponse) + "}"; arybytResponse = strResponse.getBytes(); } String strHeaders = ""; if ( strContentType != null ) { strHeaders += "Content-type: " + strContentType + "\n"; } strHeaders += "Content-length: " + arybytResponse.length + "\n" + "Access-Control-Allow-Origin: *\n" + "Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT\n" + "Access-Control-Allow-Credentials: true\n" + "Keep-Alive: timeout=2, max=100\n" + "Cache-Control: no-cache\n" + "Pragma: no-cache\n\n"; out.write(strHeaders.getBytes()); out.write(arybytResponse); out.flush(); } out.close(); sck.close(); if ( blnShutdown == true ) { String strSystem = mobjLocalIP.strGetIP(); if ( strSystem.compareTo(mobjLocalIP.strGetIP()) != 0 ) { //Specified system is not the local system, issue message to remote system. broadcastMessage("{\"" + JSON_LBL_TYPE + "\":\"" + eMsgTypes.SHUTDOWN + "\"" + ",\"" + JSON_LBL_TIME + "\":\"" + clsTimeMan.lngTimeNow() + "\"}"); } else { //Shutdown addressed to local system if ( getOS().indexOf("linux") >= 0 ) { //TO DO!!! } else if ( getOS().indexOf("win") >= 0 ) { Runtime runtime = Runtime.getRuntime(); runtime.exec("shutdown /r /c \"Shutdown request\" /t 0 /f"); System.exit(EXITCODE_REQUESTED_SHUTDOWN); } } } } catch (Exception ex) { } finally { if (sck != null) { try { sck.close(); } catch (IOException ex) { ex.printStackTrace(); } } } }I would like to implemented a chunked response, at present chunked responses are not supported by the code above.
[Edit] I've tried to implement a chunked response by adding the method:
/** * @param strData - The data to split into chunks * @return A string array containing the chunks */ public static String[] arystrChunkData(String strData) { int intChunks = (strData.length() / CHUNK_THRESHOLD_BYTESIZE) + 1; String[] arystrChunks = new String[intChunks]; int intLength = strData.length(), intPos = 0; for( int c=0; c<arystrChunks.length; c++ ) { if ( intPos < intLength ) { //Extract a chunk from the data int intEnd = Math.min(intLength - 1, intPos + CHUNK_THRESHOLD_BYTESIZE); arystrChunks[c] = strData.substring(intPos, intEnd); } //Advance data position to next chunk intPos += CHUNK_THRESHOLD_BYTESIZE; } return arystrChunks; }The modified processRequest now looks like this:
private void processRequest(Socket sck) { try { //AJAX Parameters final String AJAX_ID = "ajmid"; //The 'Handler Key' used to decode response final String HANDLER_KEY = "hkey"; //Message payload final String PAYLOAD = "payload"; //Post input buffer size final int REQUEST_BUFFER_SIZE = 4096; //Double carriage return marks the end of the headers final String CRLF = "\r\n"; BufferedReader in = new BufferedReader(new InputStreamReader(sck.getInputStream())); String strAMID = null, strHKey = null, strRequest; char[] chrBuffer = new char[REQUEST_BUFFER_SIZE]; StringBuffer sbRequest = new StringBuffer(); eMsgTypes eType = eMsgTypes.UNKNOWN; clsHTTPparameters objParams = null; int intPos, intCount; //Extract the entire request, including headers if ( (intCount = in.read(chrBuffer)) == 0 ) { throw new Exception("Cannot read request!"); } sbRequest.append(chrBuffer, 0, intCount); strRequest = sbRequest.toString(); //What method is being used by this request? if ( strRequest.startsWith(HTTP_GET) ) { //The request should end with a HTTP marker, remove this before trying to interpret the data if ( strRequest.indexOf(HTTP_MARKER) != -1 ) { strRequest = strRequest.substring(0, strRequest.indexOf(HTTP_MARKER)).trim(); } //Look for a data marker if ( (intPos = strRequest.indexOf(HTTP_DATA_START)) >= 0 ) { //Data is present in the query, skip to the start of the data strRequest = strRequest.substring(intPos + 1); } else { //Remove the method indicator strRequest = strRequest.substring(HTTP_GET.length()); } } else if ( strRequest.startsWith(HTTP_POST) ) { //Discard the headers and jump to the data if ( (intPos = strRequest.lastIndexOf(CRLF)) >= 0 ) { strRequest = strRequest.substring(intPos + CRLF.length()); } } if ( strRequest.length() > 1 ) { //Extract the parameters objParams = new clsHTTPparameters(strRequest); } if ( strRequest.startsWith("/") == true ) { //Look for the document reference strRequest = strRequest.substring(1); eType = eMsgTypes.SEND_DOC; } if ( objParams != null ) { //Transfer the payload to the request String strPayload = objParams.getValue(PAYLOAD); if ( strPayload != null ) { byte[] arybytPayload = Base64.decodeBase64(strPayload.getBytes()); strRequest = new String(arybytPayload); strAMID = objParams.getValue(AJAX_ID); strHKey = objParams.getValue(HANDLER_KEY); } } if ( eType == eMsgTypes.UNKNOWN && strRequest.startsWith("{") && strRequest.endsWith("}") ) { //The payload is JSON, is there a type parameter? String strType = strGetJSONItem(strRequest, JSON_LBL_TYPE); if ( strType != null && strType.length() > 0 ) { //Decode the type eType = eMsgTypes.valueOf(strType.toUpperCase().trim()); //What system is the message from? String strIP = strGetJSONItem(strRequest, JSON_LBL_IP) ,strMAC = strGetJSONItem(strRequest, JSON_LBL_MAC); if ( strIP != null && strIP.length() > 0 && strMAC != null && strMAC.length() > 0 ) { //Is this system known in the cluster? clsIPmon objSystem = objAddSysToCluster(strIP, strMAC); if ( objSystem != null ) { //Update the date/time stamp of the remote system objSystem.touch(); } //This is an internal cluster message, no response required return; } } } String strContentType = null, strRespPayload = null; OutputStream out = sck.getOutputStream(); byte[] arybytResponse = null; boolean blnShutdown = false; //Start the writing the headers String strHeaders = "HTTP/1.0 200\n" + "Date: " + (new Date()).toString() + "\n" + "Access-Control-Allow-Origin: *\n" + "Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT\n" + "Access-Control-Allow-Credentials: true\n" + "Keep-Alive: timeout=2, max=100\n" + "Cache-Control: no-cache\n" + "Pragma: no-cache\n"; out.write(strHeaders.getBytes()); strHeaders = ""; switch( eType ) { case SEND_DOC: if ( strRequest.length() <= 1 ) { strRequest = HTML_ROOT + DEFAULT_DOC; } else { strRequest = HTML_ROOT + strRequest; } logMsg("HTTP Request for: " + strRequest, true); if ( strRequest.toLowerCase().endsWith(".css") == true ) { strContentType = MIME_CSS; } else if ( strRequest.toLowerCase().endsWith(".gif") == true ) { strContentType = MIME_GIF; } else if ( strRequest.toLowerCase().endsWith(".jpg") == true ) { strContentType = MIME_JPG; } else if ( strRequest.toLowerCase().endsWith(".js") == true ) { strContentType = MIME_JS; } else if ( strRequest.toLowerCase().endsWith(".png") == true ) { strContentType = MIME_PNG; } else if ( strRequest.toLowerCase().endsWith(".html") == true || strRequest.toLowerCase().endsWith(".htm") == true ) { strContentType = MIME_HTML; } File objFile = new File(strRequest); if ( objFile.exists() == true ) { FileInputStream objFIS = new FileInputStream(objFile); if ( objFIS != null ) { arybytResponse = new byte[(int)objFile.length()]; if ( objFIS.read(arybytResponse) == 0 ) { arybytResponse = null; } objFIS.close(); } } break; case CHANNEL_STS: strRespPayload = strChannelStatus(strRequest); strContentType = MIME_JSON; break; case CLUSTER_STS: strRespPayload = strClusterStatus(); strContentType = MIME_JSON; break; case MODULE_STS: strRespPayload = strModuleStatus(strRequest); strContentType = MIME_JSON; break; case NETWORK_INF: strRespPayload = strNetworkInfo(strRequest); strContentType = MIME_JSON; break; case NODE_STS: strRespPayload = strNodeStatus(strRequest); strContentType = MIME_JSON; break; case POLL_STS: strRespPayload = strPollStatus(strRequest); strContentType = MIME_JSON; break; case SYS_STS: //Issue system status strRespPayload = strAppStatus(); strContentType = MIME_JSON; break; case SHUTDOWN: //Issue instruction to restart system strRespPayload = "Shutdown in progress!"; strContentType = MIME_PLAIN; //Flag that shutdown has been requested blnShutdown = true; break; default: } if ( strRespPayload != null ) { //Convert response string to byte array arybytResponse = strRespPayload.getBytes(); } if ( arybytResponse != null && arybytResponse.length > 0 ) { boolean blnChunked = false; if ( strContentType != null ) { strHeaders += "Content-type: " + strContentType + "\n"; } if ( strContentType == MIME_JSON ) { String strResponse = "{"; if ( strAMID != null ) { //Include the request AJAX Message ID in the response if ( strResponse.length() > 1 ) { strResponse += ","; } strResponse += "\"" + AJAX_ID + "\":" + strAMID; } if ( strHKey != null ) { if ( strResponse.length() > 1 ) { strResponse += ","; } strResponse += "\"" + HANDLER_KEY + "\":\"" + strHKey + "\""; } if ( strResponse.length() > 1 ) { strResponse += ","; } strResponse += "\"payload\":" + new String(arybytResponse) + "}"; //How big is the response? if ( strResponse.length() > CHUNK_THRESHOLD_BYTESIZE ) { blnChunked = true; strHeaders += "Transfer-Encoding: chunked\n\n"; out.write(strHeaders.getBytes()); //Slice up the string into chunks String[] arystrChunks = arystrChunkData(strResponse); for( int c=0; c<arystrChunks.length; c++ ) { String strChunk = arystrChunks[c]; if ( strChunk != null ) { String strLength = Integer.toHexString(strChunk.length()) + "\r\n"; strChunk += "\r\n"; out.write(strLength.getBytes()); out.write(strChunk.getBytes()); } } //Last chunk is always 0 bytes out.write("0\r\n\r\n".getBytes()); } else { arybytResponse = strResponse.getBytes(); } } if ( blnChunked == false ) { strHeaders += "Content-length: " + arybytResponse.length + "\n\n"; out.write(strHeaders.getBytes()); out.write(arybytResponse); } out.flush(); } out.close(); sck.close(); if ( blnShutdown == true ) { String strSystem = mobjLocalIP.strGetIP(); if ( strSystem.compareTo(mobjLocalIP.strGetIP()) != 0 ) { //Specified system is not the local system, issue message to remote system. broadcastMessage("{\"" + JSON_LBL_TYPE + "\":\"" + eMsgTypes.SHUTDOWN + "\"" + ",\"" + JSON_LBL_TIME + "\":\"" + clsTimeMan.lngTimeNow() + "\"}"); } else { //Shutdown addressed to local system if ( getOS().indexOf("linux") >= 0 ) { //TO DO!!! } else if ( getOS().indexOf("win") >= 0 ) { Runtime runtime = Runtime.getRuntime(); runtime.exec("shutdown /r /c \"Shutdown request\" /t 0 /f"); System.exit(EXITCODE_REQUESTED_SHUTDOWN); } } } } catch (Exception ex) { } finally { if (sck != null) { try { sck.close(); } catch (IOException ex) { ex.printStackTrace(); } } } }I've read several specifications for Chunked responses and as far as I can tell I am sending data in the correct format, however I don't receive anything in the browser.
I may have wrongly assume that the browser would correctly piece together the chunks into one, but I could be wrong. The client side handler looks like this:
this.responseHandler = function() { try { if ( mobjHTTP == null || !(mobjHTTP.readyState == 4 && mobjHTTP.status == 200) || !(mstrResponseText = mobjHTTP.responseText) || mstrResponseText.length == 0 ) { //Not ready or no response to decode return; } //Do something with the response } catch( ex ) { T.error("responseHandler:", ex); }};
This handler is set-up elsewhere in the object:
mobjHTTP.onreadystatechange = this.responseHandler;