Number of tokens limit in a FOR command in a Windows batch script

Question

I was trying to process a text file in a Windows batch script and I ran into something that looks like a limitation to 31 tokens in a FOR loop. I isolated the issue in the code below:

@ECHO OFF
SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

FOR /F "tokens=31* delims= " %%i IN ("%DATA%") DO (
    ECHO [%%i]
    ECHO [%%j]
)
ECHO.
FOR /F "tokens=32* delims= " %%i IN ("%DATA%") DO (
    ECHO [%%i]
    ECHO [%%j]
)

The output is:

[31]
[32 33 34 35]

[01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35]
[%j]

and I was expecting this:

[31]
[32 33 34 35]

[32]
[33 34 35]

Hoping that I haven't been doing something wrong, I couldn't find this limitation documented in the help for the FOR command. I'm using Windows XP. Do you know any workaround for this, aside from chopping off parts of the data?

Thank you.

Solution 1

I came up with a solution. It's not elegant, but it solves my problem. When the commmand line interpreter cannot go further with the tokens, I pass the remaning of the data to a CALL :label command. Here is an example:

@ECHO OFF

SET DATA=01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO  1st token: %%i
    ECHO 31th token: %%j
    CALL :processdatatokens32-62 %%k
)

:processdatatokens32-62
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 32nd token: %%i
    ECHO 62th token: %%j
    CALL :processdatatokens63-83 %%k
)
GOTO :EOF

:processdatatokens63-83
SET DATA=%*
FOR /F "tokens=1,31* delims= " %%i IN ("%DATA%") DO (
    ECHO 63th token: %%i
    ECHO 93th token: %%j
)
GOTO :EOF

The output is:

 1st token: 01
31th token: 31
32nd token: 32
62th token: 62
63th token: 63
93th token: 93

Solution 2

From for /?:

%i is explicitly declared in the for statement and the %j and %k are implicitly declared via the tokens= option. You can specify up to 26 tokens via the tokens= line, provided it does not cause an attempt to declare a variable higher than the letter 'z' or 'Z'. Remember, FOR variables are single-letter, case sensitive, global, and you can't have more than 52 total active at any one time.

Comments

[Paul]

Thanks for the excerpt, Patrick, but it refers to another limitation, specifically about how many variables tied to tokens one can use inside the FOR loop. In my example I'm just using 2 variables, %%i and %jj, and the problem is about their values (the tokens in the data processed).

[SuPra]

I think the maximum limit increased in 2000/XP, etc.

[Patrick Cuff]

Sorry, I misunderstood your question.

[Paul]

Rob, thanks for the answer. I see what you mean, but this solution is based on picking up exactly 2 chars at different offsets in the DATA, which is not what I need. My tokens have various lengths. The ones I used here in the DATA are just for example purposes.

[dbenham]

See stackoverflow.com/a/8520993/1012053 for a more efficient way to access up to 64 tokens. The strategy can be extended to support more than 64, especially if extended character set is used.

[Paul]

Thanks, dbenham. The addendum in your answer there is really hardcore. :)