When working with previous Spark Versions, I am always confused when it comes to specifying column names: should I use String or a col object.

Example of regexp_replace from 3.1.2:

pyspark.sql.functions.regexp_replace(str, pattern, replacement)[source]

I was running a cluster with Version 3.1.2 and both works:

df1.withColumn("modality",F.regexp_replace(F.col("name"),"i","")).display()
df1.withColumn("modality",F.regexp_replace("name","i","")).display()

From the docu I would have assumed that only a String is allowed, but both works. How can I see in the API docu, if also a col object is allowed (in the latest api this is pretty clear, but not in the previous ones).

When working with previous Spark Versions, I am always confused when it comes to specifying column names: should I use String or a col object.

Example of regexp_replace from 3.1.2:

pyspark.sql.functions.regexp_replace(str, pattern, replacement)[source]

I was running a cluster with Version 3.1.2 and both works:

df1.withColumn("modality",F.regexp_replace(F.col("name"),"i","")).display()
df1.withColumn("modality",F.regexp_replace("name","i","")).display()

From the docu I would have assumed that only a String is allowed, but both works. How can I see in the API docu, if also a col object is allowed (in the latest api this is pretty clear, but not in the previous ones).

• apache-spark
• pyspark

1 Answer 1

When you click on the source button of the 3.1.2 doc you find the source code of regexp_replace:

def regexp_replace(str, pattern, replacement):
    r"""Replace all substrings of the specified string value that match regexp with rep.

    .. versionadded:: 1.5.0

    Examples
    --------
    >>> df = spark.createDataFrame([('100-200',)], ['str'])
    >>> df.select(regexp_replace('str', r'(\d+)', '--').alias('d')).collect()
    [Row(d='-----')]
    """
    sc = SparkContext._active_spark_context
    jc = sc._jvm.functions.regexp_replace(_to_java_column(str), pattern, replacement)
    return Column(jc)

You see that the str argument is not used directly but wrapped within the _to_java_column function. The source code of _to_java_column clearly shows that it works with both column names (string) and column objects:

def _to_java_column(col: "ColumnOrName") -> "JavaObject":
    if isinstance(col, Column):
        jcol = col._jc
    elif isinstance(col, str):
        jcol = _create_column_from_name(col)
    else:
        raise PySparkTypeError(
            errorClass="NOT_COLUMN_OR_STR",
            messageParameters={"arg_name": "col", "arg_type": type(col).__name__},
        )
    return jcol

When browsing the source page of functions, you see that _to_java_column is omnipresent, which means that for most functions (or even all but I didn't check), both column names of column object can be used.