Windows batch files and multiple processors

Question

I have a Windows batch file that performs the same action multiple times, except on multiple files. Is there a way I can tell the computer to use different processor cores to speed up the process? Otherwise, it takes longer since the same processor is being used all the time.

Solution

Although a Batch file can start several parallel threads, there are multiple problems related to the synchronization of all of they. The usual synchro methods use the presence of a file like a semaphore, but the code that check for the semaphore-file waste CPU time. Although a delay can be inserted in this code, the result is usually cumbersome and tricky.

I developed an efficient multi-thread scheme for Batch files that use a pipe to get a synchro signal; this point allows the waiting code to not waste CPU time, so all CPU cores can be used for data processing. The program below is an example that process several files selected by a wild-card, but this code can easily be adapted to process any other set of data; this method is specifically designed for the case when just one thread ends its processing at same time and may fail if the processing of two or more files ends in a very short time interval.

@echo off
setlocal EnableDelayedExpansion

rem Multi-thread scheme in Batch files
rem Antonio Perez Ayala aka Aacini

set "myID=%~2"
if "%~1" neq "" goto %1

set "signal=X"
for /L %%i in (1,1,10) do set "signal=!signal!!signal!"

rem Create the list of start commands for the concurrent working threads
set "numThreads=%NUMBER_OF_PROCESSORS%"
set "threads="
del availableThread* 2> NUL
for /L %%i in (1,1,%numThreads%) do (
   set "threads=!threads! & start "" /B cmd /C "%~NX0" Thread %%i"
   > availableThread%%i echo %%i
)

echo Start of application
del requestForThread* 2> NUL
( %threads:~3% ) | "%~NX0" Main
echo End of application
goto :EOF


:Main

echo Main - started

rem Controlling code: process all files and assign each one to available threads
for /F "delims=" %%f in ('dir /A-D /B *.txt') do (

   rem Get the number of next available thread; waits for anyone, if necessary
   if not exist availableThread* (
      echo X > mainIsWaiting
      echo Main - waiting for an available thread
      set /P availableSignal=
      del mainIsWaiting
   )

   set "nextThread="
   for %%t in (availableThread*) do if not defined nextThread (
      set /P "nextThread=" < %%t
      del %%t
   )

   if not defined nextThread (
      echo SYNCHRO FAILED^^!
      goto waitThreadsEnds
   )

   rem Assign the next file to the available thread
   echo %%f> requestForThread!nextThread!
   echo Main - file "%%f" assigned to thread #!nextThread!

)

echo Main - all files sent to processing

rem Wait until all threads ends their tasks
:waitThreadsEnds
   set availableThreads=0
   for %%t in (availableThread*) do set /A availableThreads+=1
if %availableThreads% lss %numThreads% goto :waitThreadsEnds
del availableThread*

rem Send "exit" signals to all threads
for /L %%i in (1,1,%numThreads%) do echo exit> requestForThread%%i
echo Main - end
goto :EOF


:Thread

rem Wait until a request for this thread appear
if not exist requestForThread%myID% goto Thread
set "request="
set /P "request=" < requestForThread%myID%
if not defined request goto Thread
del requestForThread%myID%
if "%request%" equ "exit" goto :EOF

rem Process the file
echo %time% - Thread #%myID% start: "%request%" > CON
set /A seconds=6+%random:~-1%
ping localhost -n %seconds% > NUL
echo %time% - Thread #%myID% end:   "%request%" > CON

rem Inform that this thread is available for other task
> availableThread%myID% echo %myID%
if exist mainIsWaiting echo %signal:~0,1021%

goto Thread

Of course, in order for this method to work as expected, your Windows computer must process just this Batch file alone with no other concurrent process.

For a further explanation of the method used in this solution, see this post.