Java 8: How can I convert a for loop to run in parallel?

10,346

Solution 1

I used the following code in Java 8 and it did the work. I was able to reduce the batch job to run from 28 minutes to 3:39 minutes.

IntStream.range(0, 100000).parallel().forEach(i->{
     restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
});

Solution 2

The standard call to parallel() will create a thread for each core your machine has available minus one core, using a Common Fork Join Pool.

If you want to specify the parallelism on your own, you will have different possibilities:

  1. Change the parallelism of the common pool: System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20")
  2. Use an own pool:

Example:

int allRequestsCount = 20;
int parallelism = 4; // Vary on your own

ForkJoinPool forkJoinPool = new ForkJoinPool(parallelism);
IntStream.range(0, parallelism).forEach(i -> forkJoinPool.submit(() -> {
  int chunkSize = allRequestsCount / parallelism;
  IntStream.range(i * chunkSize, i * chunkSize + chunkSize)
           .forEach(num -> {

             // Simulate long running operation
             try {
               Thread.sleep(1000);
             } catch (InterruptedException e) {
               e.printStackTrace();
             }

             System.out.println(Thread.currentThread().getName() + ": " + num);
           });
}));

This implementation is just examplary to give you an idea.

Share:
10,346
WowBow
Author by

WowBow

Updated on June 04, 2022

Comments

  • WowBow
    WowBow almost 2 years
    for (int i=0; i<100000; i++) {
        // REST API request. 
    restTemplate.exchange(url, HttpMethod.GET, request, String.class);
    }
    

    I have a situation where I have to request a resource for 100k users and it takes 70 minutes to finish. I tried to clean up my code as much as possible and I was able to reduce it only by 4 minutes).

    Since each request is independent of each other, I would love to send requests in parallel (may be in 10s, 100s, or even 1000s of chunks which every finishes quickly). I'm hoping that I can reduce the time to 10 minutes or something close. How do I calculate which chunk size would get the job done quickly?

    I have found the following way but I can't tell if the program processes all the 20 at a time; or 5 at a time; or 10 at a time.

    IntStream.range(0,20).parallel().forEach(i->{
         ... do something here
    });
    

    I appericiate your help. I am open to any suggestions or critics!!

    UPDATE: I was able to use IntStream and the task finished in 28 minutes. But I am not sure this is the best I could go for.

  • Imesha Sudasingha
    Imesha Sudasingha over 7 years
    His bottleneck is not in parallel processing. It is in blocking requests. Therefore, this won't do any good.
  • heaprc
    heaprc over 7 years
    I thought he is asking about how to split the requests in parallel maner
  • WowBow
    WowBow over 7 years
    @ImeshaSudasingha What do you mean by blocking requests ? I asked to split requests in parallel.
  • Imesha Sudasingha
    Imesha Sudasingha over 7 years
    What I'm asking is, put the code used to send the requests. There are 2 types of requests. synchronous and asynchronous. I want to know which one you were using because, as far as I see, the bottleneck is not with parallelism.
  • D.B.
    D.B. over 7 years
    @ImeshaSudasingha if the bottleneck is using synchronous rather than asynchronous requests then it absolutely has to do with parallelism. See this answer
  • Imesha Sudasingha
    Imesha Sudasingha over 7 years
    Yes, I know that parallelism matters. But my point is, since the requests are independent, he can use asynchronous requests to improve performance. Going for parallelism with synchronous requests is a premature optimization as I feel.