Multi-Threading httpClient
Solution 1
I realised my folly! httpClient and httpRequest were both static. After i made them non-static it works fine! Executor Service gives me better control on managing threads and I was keen on using it.
Solution 2
Came here to say, MultiThreadedHttpConnectionManager is outdated. Currently (HttpClient version 4.*) this is the way: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e639
SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(
new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
ClientConnectionManager cm = new PoolingClientConnectionManager(schemeRegistry);
HttpClient httpClient = new DefaultHttpClient(cm);
Solution 3
You need define multi-threaded HttpConnectionManager e.g.
MultiThreadedHttpConnectionManager connectionManager =
new MultiThreadedHttpConnectionManager();
HttpClient client = new HttpClient(connectionManager);
For more details you can see http://hc.apache.org/httpclient-3.x/threading.html
Solution 4
In addition to bpgergo answer's - the Connection manager has updated yet again (as of HttpClient version >= 4.3), and now you should use PoolingHttpClientConnectionManager
instead. The default limitations of the PoolingHttpClientConnectionManager
are total of 20 connections, and 2 per route - but those can be overridden.
PoolingHttpClientConnectionManager cm=new PoolingHttpClientConnectionManager();
cm.setDefaultMaxPerRoute(40);
cm.setMaxTotal(500);
CloseableHttpClient client = HttpClients.createMinimal(cm);
RFT
Updated on July 09, 2022Comments
-
RFT almost 2 years
public class test { public static final int nThreads = 2; public static void main(String[] args) throws ExecutionException, InterruptedException{ // Runnable myrunnable = new myRunnable(); ExecutorService execute = Executors.newFixedThreadPool(nThreads); for (int i = 0; i < nThreads; ++i) { execute.execute(new MyTask()); } execute.awaitTermination(1000, TimeUnit.MILLISECONDS); execute.shutdown(); } } class MyTask implements Runnable { public static final int maxCalls = 10; public static final int sleepMillis = 500; private static HttpResponse response; private static HttpClient httpclient; public void run(){ int counter = 0; while (true) { if (counter >= maxCalls) { break; } try { Thread.currentThread().sleep(sleepMillis); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } execHttpRequest(); ++counter; } } private void execHttpRequest() { httpclient = new DefaultHttpClient(); HttpGet httpget = new HttpGet("My URL"); try { response = httpclient.execute(httpget); BufferedReader br = new BufferedReader(new InputStreamReader(response.getEntity().getContent())); String output; while((output=br.readLine())!=null){ System.out.println(Thread.currentThread().getName() +output); } br.close(); httpclient.getConnectionManager().shutdown(); //httpclient.getConnectionManager().shutdown(); } catch (ClientProtocolException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } finally{ httpclient.getConnectionManager().shutdown(); } } }
While running this code, I get the following exception:
Exception in thread "pool-1-thread-1" java.lang.IllegalStateException: Invalid use of SingleClientConnManager: connection still allocated. Make sure to release the connection before allocating another one. at org.apache.http.impl.conn.SingleClientConnManager.getConnection(SingleClientConnManager.java:216) at org.apache.http.impl.conn.SingleClientConnManager$1.getConnection(SingleClientConnManager.java:190) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) at MyTask.execHttpRequest(test.java:72) at MyTask.run(test.java:60) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) java.io.InterruptedIOException: Connection has been shut down at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:543) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) at MyTask.execHttpRequest(test.java:72) at MyTask.run(test.java:60) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.http.impl.conn.ConnectionShutdownException at org.apache.http.impl.conn.AbstractPooledConnAdapter.assertValid(AbstractPooledConnAdapter.java:86) at org.apache.http.impl.conn.AbstractPooledConnAdapter.getRoute(AbstractPooledConnAdapter.java:112) at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:740) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:577) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) ... 8 more
When I execute http request, then I see these Exceptions. It works perfectly fine for single-threaded. I am trying to call a specific URL (which works perfectly fine) but when I add more than one thread to it, it throws an illegal state exception.