Pyspark mapinpandas

pyspark mapinpandas:

pyspark mapinpandas is a function that allows you to apply pandas operations to PySpark DataFrames. It is particularly useful when you need to perform complex operations that are difficult to express using built-in PySpark functions.

This function takes a pandas user-defined function (UDF) and applies it to each partition of a PySpark DataFrame, converting the partition into a pandas DataFrame. The UDF can then operate on the partition as if it were a regular pandas DataFrame, taking advantage of the extensive functionality provided by pandas.

Example:

from pyspark.sql import SparkSession
import pandas as pd
import pyspark
from pyspark.sql.functions import PandasUDFType, pandas_udf

# Create a PySpark DataFrame
spark = SparkSession.builder.getOrCreate()
data = [('John', 25), ('Alice', 30), ('Bob', 35)]
df = spark.createDataFrame(data, ['Name', 'Age'])

# Define a pandas UDF to calculate the squared age
@pandas_udf('int', PandasUDFType.SCALAR)
def calculate_squared_age(age: pd.Series) -> pd.Series:
    return age ** 2

# Apply the pandas UDF using mapInPandas
df_with_squared_age = df.mapInPandas(calculate_squared_age)

df_with_squared_age.show()
# Output:
# +-----+---+
# | Name|Age|
# +-----+---+
# | John|625|
# |Alice|900|
# |  Bob|1225|
# +-----+---+

In the above example, we create a PySpark DataFrame with two columns, “Name” and “Age”. We then define a pandas UDF called calculate_squared_age that calculates the squared age for each value in the “Age” column. Finally, we apply the pandas UDF to the PySpark DataFrame using mapInPandas, and the result is a new DataFrame with the squared ages.