Maximum Array of Strings VisualBasic WSH

Question

I'm writing a WSH script in VB to read a massive directory listing generated with a redirected directory listing via .Run method.

The directory listing is about 8400 lines, but every time I run the script, the following loop

Do Until DirList.AtEndOfStream Redim Preserve arrData(i) arrData(i) = DirList.ReadLine i = i + 1 Loop

cuts out early, in a seemingly random range of 1800 to 3500 lines. Does this sound like an array size issue or a shell memory limit?

I have heard of people parsing LARGE log files, reading them all in at once like I have.

Solution 1

Would it not be better, in this instance, to cycle through the file first and count the number of lines, then Redim the array to the exact size required. Then close the file open it again this time you actually assign the lines to the array elements?

Solution 2

The size of arrays in VBScript is limited by a few different things, whichever comes first:

• A maximum of (2 ^ 31) - 1 elements (because the number of elements is stored internally as a Long value, and because there is no larger data type available to use as an indexer).

• A maximum of 60 dimensions.

• Available system memory.

Most of these limits are of no use beyond purely theoretical exercise, however, and I'd be very suspicious of any code written that had to be concerned with them.

Because you say that you only have 8400 lines to process, I doubt you're running into the theoretical limits placed on the size of an array. Instead, the biggest problem with your code is that you're using Redim Preserve inside of a loop.

The MSDN reference explains that Redim Preserve is used to resize the last dimension of the array dynamically, while preserving its existing contents. What it doesn't necessarily mention is how it works. Each time you use Redim Preserve, a new array is created with the number of elements that you specify, and the values of the elements in the previous array are copied into the new one. This should be sending up red flags immediately, because it means that on each iteration of your loop, you're allocating space for and filling an entirely new array. The problem only gets worse the more iterations you've made in the loop, because the size of each new array that is created is growing incrementally larger.

Thus, it's more likely that you're overflowing the stack space that VBScript allocates for local variables. (How appropriate—a stack overflow error?) Eventually those unused arrays will be garbage collected, but you're putting a giant amount of pressure on memory and resources when you do this in a tight loop.

You are far better off simply allocating enough space in your array to hold all of the directory listings that you'll need to hold. You don't necessarily have to get the maximum size exactly right. Simply allocating more than you'll need is still far cheaper than continually creating and destroying new arrays. If you're still concerned that this won't be enough, you can check in the loop if the current index is greater than the maximum amount of elements in the array, and if so, allocate a lot more space (by, say, doubling its current size). After you get finished, you can deallocate the excess space while retaining the good data using the Redim Preserve command, but this time only once! For a rough knock-up example:

Dim MyData()                           ''#declare the array
Dim MaxSize = 10000                    ''#guess an initial size
Redim MyData(MaxSize)                  ''#allocate an array of the initial size

Dim Size
Do Until DirList.AtEndOfStream
    Size = Size + 1                    ''#record the current index

    If Size > MaxSize Then
        ''#The array is full, so allocate more space
        MaxSize = MaxSize * 2          ''#determine a new size (doubling is a good guess)
        Redim Preserve MyData(MaxSize) ''#add more space in the array
    End If

    MyData(Size) = DirList.ReadLine    ''#store this data in the array
Loop

If Size < MaxSize Then
    Redim Preserve MyData(Size)        ''#deallocate the extra space in your array
End If

This just might save your memory from the abuse of the constant allocation and deallocation of arrays.