Pandas keep_default_na

by

The pandas keep_default_na parameter is used to specify the default behavior of the pandas library for handling missing or null values (NaN) when reading a CSV or Excel file.

By default, when reading a file, pandas treats values like NA, NULL, N\A, nan, etc. as missing values and replaces them with NaN. This behavior is helpful in many cases as it allows you to easily detect and handle missing values.

However, in some cases, you may want to change this default behavior. This is where the keep_default_na parameter comes into play. It allows you to customize how pandas treats certain values as missing.

The keep_default_na parameter accepts either a boolean value or a list of strings. Here’s how it works:

  1. If keep_default_na is set to True (default), pandas will treat the default NaN values as missing. This means values like NA, NULL, N/A, nan, etc. will be considered missing and replaced with NaN.
  2. If keep_default_na is set to False, pandas will not treat the default NaN values as missing. Instead, it will keep them as regular values in the resulting DataFrame. In this case, you can customize how pandas treats missing values using other parameters like na_values.
  3. If keep_default_na is set to a list of strings, pandas will consider those strings as additional missing values. Any occurrence of those strings in the input data will be treated as missing and replaced with NaN. The default NaN values will still be treated as missing as well.

Let’s illustrate the usage of keep_default_na with a few examples: 


import pandas as pd

# Example CSV data: "data.csv"
# Column1,Column2
# 1,NA
# 2,NULL
# 3,N/A
# 4,nan

# Reading CSV with default behavior
df_default = pd.read_csv("data.csv")
print(df_default)
# Output:
#    Column1  Column2
# 0        1      NaN
# 1        2      NaN
# 2        3      NaN
# 3        4      NaN

# Reading CSV without treating default NaN as missing
df_no_na = pd.read_csv("data.csv", keep_default_na=False)
print(df_no_na)
# Output:
#    Column1 Column2
# 0        1      NA
# 1        2    NULL
# 2        3     N/A
# 3        4     nan

# Reading CSV with additional missing values
df_custom_na = pd.read_csv("data.csv", keep_default_na=["NA"])
print(df_custom_na)
# Output:
#    Column1  Column2
# 0        1      NaN
# 1        2      NaN
# 2        3      NaN
# 3        4      NaN

In the first example, we read the CSV file with the default behavior. The default NaN values (NA, NULL, N/A, nan) are treated as missing, and therefore, replaced with NaN.

In the second example, we read the CSV file without treating the default NaN values as missing by setting keep_default_na=False. As a result, the default NaN values are kept as regular values in the DataFrame.

In the third example, we read the CSV file while also specifying keep_default_na=["NA"]. This means that in addition to the default NaN values, the string “NA” will also be considered as missing and replaced with NaN.

Keep in mind that the keep_default_na parameter is applicable to other pandas read functions like read_excel as well. The behavior of these functions can be customized in a similar way.

Leave a comment