Java 8: How can I convert a for loop to run in parallel?
Solution 1
I used the following code in Java 8 and it did the work. I was able to reduce the batch job to run from 28 minutes to 3:39 minutes.
IntStream.range(0, 100000).parallel().forEach(i->{
restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
});
Solution 2
The standard call to parallel()
will create a thread for each core your machine has available minus one core, using a Common Fork Join Pool.
If you want to specify the parallelism on your own, you will have different possibilities:
- Change the parallelism of the common pool:
System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20")
- Use an own pool:
Example:
int allRequestsCount = 20;
int parallelism = 4; // Vary on your own
ForkJoinPool forkJoinPool = new ForkJoinPool(parallelism);
IntStream.range(0, parallelism).forEach(i -> forkJoinPool.submit(() -> {
int chunkSize = allRequestsCount / parallelism;
IntStream.range(i * chunkSize, i * chunkSize + chunkSize)
.forEach(num -> {
// Simulate long running operation
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + ": " + num);
});
}));
This implementation is just examplary to give you an idea.
WowBow
Updated on June 04, 2022Comments
-
WowBow almost 2 years
for (int i=0; i<100000; i++) { // REST API request. restTemplate.exchange(url, HttpMethod.GET, request, String.class); }
I have a situation where I have to request a resource for 100k users and it takes 70 minutes to finish. I tried to clean up my code as much as possible and I was able to reduce it only by 4 minutes).
Since each request is independent of each other, I would love to send requests in parallel (may be in 10s, 100s, or even 1000s of chunks which every finishes quickly). I'm hoping that I can reduce the time to 10 minutes or something close. How do I calculate which chunk size would get the job done quickly?
I have found the following way but I can't tell if the program processes all the 20 at a time; or 5 at a time; or 10 at a time.
IntStream.range(0,20).parallel().forEach(i->{ ... do something here });
I appericiate your help. I am open to any suggestions or critics!!
UPDATE: I was able to use IntStream and the task finished in 28 minutes. But I am not sure this is the best I could go for.
-
Imesha Sudasingha over 7 yearsHis bottleneck is not in parallel processing. It is in blocking requests. Therefore, this won't do any good.
-
heaprc over 7 yearsI thought he is asking about how to split the requests in parallel maner
-
WowBow over 7 years@ImeshaSudasingha What do you mean by blocking requests ? I asked to split requests in parallel.
-
Imesha Sudasingha over 7 yearsWhat I'm asking is, put the code used to send the requests. There are 2 types of requests. synchronous and asynchronous. I want to know which one you were using because, as far as I see, the bottleneck is not with parallelism.
-
D.B. over 7 years@ImeshaSudasingha if the bottleneck is using synchronous rather than asynchronous requests then it absolutely has to do with parallelism. See this answer
-
Imesha Sudasingha over 7 yearsYes, I know that parallelism matters. But my point is, since the requests are independent, he can use asynchronous requests to improve performance. Going for parallelism with synchronous requests is a premature optimization as I feel.