Disable authentication for HTTP OPTIONS method (preflight request)

Question

I have pyspark dataframe like this

df = sqlContext.createDataFrame([
    Row(a=1, b=3),
    Row(a=3, b=2),
])

+---+---+
|  a|  b|
+---+---+
|  1|  3|
|  3|  2|
+---+---+

I tried self-join on it like this

df1 = df.alias("df1")
df2 = df.alias("df2")

cond = [df1.a == df2.b]
df1.join(df2, cond).show()

But it gives me error.

Ideally i want to find all pair where one neighbor is common. (3 is common to both 1,2)

+---+---+
| c1| c2|
+---+---+
|  1|  2|
+---+---+

Solution

You can limit the scope of Require valid-user by using Limit/LimitExcept :

<LimitExcept OPTIONS>
  Require valid-user
</LimitExcept>

See apache documentation on Limit

Comments

[ernest_k]

This seems to work: df2 = df.select(df.a.alias('aa'), df.b.alias('bb')) and then df.join(df2, df.a == df2.bb).select(['b', 'aa']).show()

[dcusmeb]

Thanks a lot <3, it worked. I think issue was with using df.a. Using col('df.a') works too.