Multidimensional Array Comprehension in Julia

Question

I'm mucking about with Julia and can't seem to get multidimensional array comprehensions to work. I'm using a nightly build of 0.20-pre for OSX; this could conceivably be a bug in the build. I suspect, however, it's a bug in the user.

Lets say I want to wind up with something like:

5x2 Array
1 6
2 7
3 8
4 9
5 10

And I don't want to just call reshape. From what I can tell, a multidimensional array should be generated something like: [(x, y) for x in 1:5, y in 6:10]. But this generates a 5x5 Array of tuples:

julia> [(x, y) for x in 1:5, y in 6:10]
5x5 Array{(Int64,Int64),2}:
 (1,6)  (1,7)  (1,8)  (1,9)  (1,10)
 (2,6)  (2,7)  (2,8)  (2,9)  (2,10)
 (3,6)  (3,7)  (3,8)  (3,9)  (3,10)
 (4,6)  (4,7)  (4,8)  (4,9)  (4,10)
 (5,6)  (5,7)  (5,8)  (5,9)  (5,10)

Or, maybe I want to generate a set of values and a boolean code for each:

5x2 Array
1 false
2 false
3 false
4 false
5 false

Again, I can only seem to create an array of tuples with {(x, y) for x in 1:5, y=false}. If I remove the parens around x, y I get ERROR: syntax: missing separator in array expression. If I wrap x, y in something, I always get output of that kind -- Array, Array{Any}, or Tuple.

My guess: there's something I just don't get here. Anybody willing to help me understand what?

Solution 1

I don't think a comprehension is appropriate for what you're trying to do. The reason can be found in the Array Comprehension section of the Julia Manual:

A = [ F(x,y,...) for x=rx, y=ry, ... ]

The meaning of this form is that F(x,y,...) is evaluated with the variables x, y, etc. taking on each value in their given list of values. Values can be specified as any iterable object, but will commonly be ranges like 1:n or 2:(n-1), or explicit arrays of values like [1.2, 3.4, 5.7]. The result is an N-d dense array with dimensions that are the concatenation of the dimensions of the variable ranges rx, ry, etc. and each F(x,y,...) evaluation returns a scalar.

A caveat here is that if you set one of the variables to a >1 dimensional Array, it seems to get flattened first; so the statement that the "the result is... an array with dimensions that are the concatenation of the dimensions of the variable ranges rx, ry, etc" is not really accurate, since if rx is 2x2 and ry is 3, then you will not get a 2x2x3 result but rather a 4x3. But the result you're getting should make sense in light of the above: you are returning a tuple, so that's what goes in the Array cell. There is no automatic expansion of the returned tuple into the row of an Array.

If you want to get a 5x2 Array from a comprhension, you'll need to make sure x has a length of 5 and y has a length of 2. Then each cell would contain the result of the function evaluated with each possible pairing of elements from x and y as arguments. The thing is that the values in the cells of your example Arrays don't really require evaluating a function of two arguments. Rather what you're trying to do is just to stick two predetermined columns together into a 2D array. For that, use hcat or a literal:

• hcat(1:5, 6:10)
• [ 1:5 5:10 ]
• hcat(1:5, falses(5))
• [ 1:5 falses(5) ]

If you wanted to create a 2D Array where column 2 contained the result of a function evaluated on column 1, you could do this with a comprehension like so:

f(x) = x + 5
[ y ? f(x) : x for x=1:5, y=(false,true) ]

But this is a little confusing and it seems more intuitive to me to just do

x = 1:5
hcat( x, map(f,x) )

Solution 2

I think you are just reading the list comprehension wrong

julia> [x+5y for  x in 1:5, y in 0:1]
5x2 Array{Int64,2}:
 1   6
 2   7
 3   8
 4   9
 5  10

When you use them in multiple dimensions you get two variables and need a function for the cell values based on the coordinates

For your second question I think that you should reconsider your requirements. Julia uses typed arrays for performance and storing different types in different columns is possible. To get an untyped array you can use {} instead of [], but I think the better solution is to have an array of tuples (Int, Bool) or even better just use two arrays (one for the ints and one for the bool).

julia> [(i,false) for i in 1:5]
5-element Array{(Int64,Bool),1}:
 (1,false)
 (2,false)
 (3,false)
 (4,false)
 (5,false)

Solution 3

Your intuition was to write [(x, y) for x in 1:5, y in 6:10], but what you need is to wrap the ranges in zip, like this:

[i for i in zip(1:5, 6:10)]

Which gives you something very close to what you need, namely:

5-element Array{(Int64,Int64),1}:
(1,6) 
(2,7) 
(3,8) 
(4,9) 
(5,10)

To get exactly what you're looking for, you'll need:

hcat([[i...] for i in zip(1:5, 6:10)]...)'

This gives you:

5x2 Array{Int64,2}:
1   6
2   7
3   8
4   9
5  10

Solution 4

I kind of like the answer @fawr gave for the efficiency of the datatypes while retaining mutability, but this quickly gets you what you asked for (working off of Shawn's answer):

hcat(1:5,6:10)
hcat({i for i=1:5},falses(5))

The cell-array comprehension in the second part forces the datatype to be Any instead of IntXX

This also works:

hcat(1:5,{i for i in falses(5)})

I haven't found another way to explicitly convert an array to type Any besides the comprehension.

View more solutions

Comments

[Gastove]

Okay, this plus @ivarne's comment below makes sense. I'm coming from Python, but I clearly didn't grok the manual on my first pass. Thanks for helping clarify! My remaining question is this: I found [ 1:5 falses(5) ], which is perfect except that it encodes false as 0. The Julia documentation specifies that Julia doesn't use 0/1 for boolean encoding. 0 == false returns true, but if(0) throws a type error. Is there a way to hcat numbers with boolean values that returns booleans, not Int encodings?

[Gastove]

Yeah, I do understand that what I'm doing doesn't conform to how Julia is optimized -- however, at the moment, what I need is a mutable array, so I can flip false to true later on; using tuples doesn't allow for that, sadly. Using two separate arrays might actually work quite well, though! I'll look at that -- thanks for the suggestion/time!

[Sean Mackesey]

@Gastove Yeah what's happening here is that the falses are being promoted to Int64. I've been trying for a bit and I can't figure out a way to elegantly get the 5x2 Array with the falses. You can try typing the Array, but I can't get it to work: Any[ 1:5 falses(5) ] will return a 1x2 (neither the range or the falses array is expanded) and Union(Number,Bool)[ 1:5 falses(5) ] throws an error. Here is code that works though: a = Array(Union(Number,Bool),5,2); a[:,1] = 1:5; a[:,2] = falses(5)

[Gastove]

Well, thanks very much for the help, one way or the other. I have a hunch that making a cell array of appropriate dimensions, then populating it, will work, but I haven't messed with it enough to have it going yet. Still getting used to Julian Thinking.