Tag: spark

Weirdness when every function returns a Column: Chained when (Spark)

When when is chained, the chain breaks at the point that the test returns true. import org.apache.spark.sql.Column val isTrue = lit(true)def getWithChainedWhen():Column = { when(isTrue,"1st") .when(isTrue,"2nd") .when(isTrue,"3rd") }val df = sc.parallelize(List[(String)](("A"))) .toDF("a") .withColumn( "chained",getWithChainedWhen() ) .show(false) The results of running the above code is as follows: +---+-------+|a |chained|+---+-------+|A |1st |+---+-------+ Only the first when is … Continue reading Weirdness when every function returns a Column: Chained when (Spark)