Missing data: transformations

Author

Nicky Wakim, Jessica Minnier, and Meike Niederhausen

From finding missing data to handling it

Before we start

  • Install and load the naniar package (a play on Narnia) if you haven’t already!

"Explore missing data in naniar!" Cute monsters with exploring gear looking around a landscape full of subtle (and not so subtle) NAs, hidden in trees, on mountain tops, as birds, and in flowers. Learn more about naniar.

Artwork by @allison_horst
  • We’ll be working with only a few variables in the HRS dataset in this lesson!
    • Mostly to help us view the tables and plots within the slides
hrs_00 <- hrs_data |>
  select(ed, race_original, srh, sex, act_vig, height, degree)

What do we do with missing data?

  • Filtering out missing values with drop_na()
  • A common gotcha: how filter() silently drops NAs

 

  • We will also use naniar functions to work with NAs in a more flexible way
    • Filling in missing values with replace_na() (and fct_na_value_to_level() for factors)
    • Replacing a values (numbers or levels) with NA using replace_with_na()

 

  • Not covered in this course (but very important in practice):
    • Imputation methods for filling in missing values
    • Weighting methods for adjusting for missingness

Filtering out missing values using drop_na()

Check missingness rates first before dropping NAs

  • We may want to remove rows that have missing data, coded as NA

  • First, summarize and visualize missingness using naniar

  • We’ll take a quick look at the counts

Create missingness sumamry table with miss_var_summary() 
hrs_00 |>
  miss_var_summary() |>
  gt() |>
  tab_options(table.font.size = px(40))
variable n_miss pct_miss
ed 507 18.6
race_original 486 17.8
degree 470 17.2
height 55 2.02
act_vig 9 0.330
srh 4 0.147
sex 0 0

drop_na(): removing missingness from all variables in the dataset

  • drop_na() removes rows with missing data
    • You can target specific columns, or drop any row with any missingness
hrs_01_na_rm <- hrs_00 |>
  drop_na()
tibble(hrs_01_na_rm)
# A tibble: 2,151 × 7
      ed race_original          srh       sex    act_vig     height degree      
   <dbl> <fct>                  <fct>     <fct>  <fct>        <dbl> <fct>       
 1    12 White/Caucasian        Good      Female Never         1.60 High school…
 2    16 White/Caucasian        Excellent Male   >1 per week   1.88 Master's de…
 3    16 Black/African American Good      Male   1 per week    1.75 Master's de…
 4    17 White/Caucasian        Poor      Female Never         1.55 Professiona…
 5    11 Black/African American Excellent Male   Never         1.78 None        
 6    13 Black/African American Fair      Male   Never         1.78 Associate's…
 7    16 White/Caucasian        Very Good Male   1 per week    1.73 Master's de…
 8    12 White/Caucasian        Good      Male   Never         1.88 High school…
 9    12 Black/African American Fair      Male   Never         1.73 None        
10    12 Black/African American Very Good Male   Never         1.73 None        
# ℹ 2,141 more rows

Pay attention to the number of rows for each of these results

drop_na(): removing missingness from all variables in the dataset

  • Remove missingness in one or more columns:
hrs_02_race_rm <- hrs_00 |> 
  drop_na(race_original, ed)

hrs_02_race_rm |> miss_var_summary() |>
  gt() |>
  tab_options(table.font.size = px(40))
variable n_miss pct_miss
height 47 2.13
act_vig 6 0.272
srh 4 0.181
ed 0 0
race_original 0 0
sex 0 0
degree 0 0

A warning about filter(): check for NA

Warning on filter(): it can silently drop your NAs

  • Suppose we want to remove the "Poor" category from srh (self-reported health)
hrs_00 |> tabyl(srh)
       srh   n     percent valid_percent
 Excellent 277 0.101539589    0.10168869
 Very Good 647 0.237170088    0.23751836
      Good 870 0.318914956    0.31938326
      Fair 743 0.272360704    0.27276065
      Poor 187 0.068548387    0.06864905
      <NA>   4 0.001466276            NA

srh has 4 missing values!

When we try to filter using !=, we lose our NAs!

hrs_03_rm_poor <- hrs_00 |> 
  filter(srh != "Poor")

When R checks NA != "Poor", it evaluates to NA, not TRUE, so that row gets dropped

hrs_03_rm_poor |> tabyl(srh)
       srh   n   percent
 Excellent 277 0.1091841
 Very Good 647 0.2550256
      Good 870 0.3429247
      Fair 743 0.2928656
      Poor   0 0.0000000

Use filter_out() to keep NAs

  • We can use filter_out() from the dplyr package to keep the NAs when we want to perform an exclusionary (!=) filter
hrs_04_rm_poor <- hrs_00 |>
  filter_out(srh == "Poor")

hrs_04_rm_poor |> tabyl(srh)
       srh   n     percent valid_percent
 Excellent 277 0.109012200     0.1091841
 Very Good 647 0.254624164     0.2550256
      Good 870 0.342384888     0.3429247
      Fair 743 0.292404565     0.2928656
      Poor   0 0.000000000     0.0000000
      <NA>   4 0.001574183            NA

Build this habit

Whenever you filter on a column that has missing values, stop and check: how many observations were removed and does that correspond to the number of values I expect (excluding NAs?)

Replace missing values with replace_na()

replace_na(): replace missing values with a specific value

  • Sometimes we want to fill in missing values with a specific value instead of removing the row

  • We can use the replace_na() function inside mutate() to set NAs to a specific value

  • replace_na() works on numeric and character variables, not factors  

  • Here’s the general version of the function:

data_filled <- data |>
  mutate(
1    variable_filled = replace_na(
2      variable,
3      value_to_fill_in
      )
  )
1
replace_na() swaps out any NA in variable for the value we provide. It’s good practice to assign this to a new variable, like variable_filled
2
variable is the column we want to fill in
3
value_to_fill_in is the value we want to use to replace the missing

Example 1: filling a numeric variable

  • Suppose if height is missing, we want to replace it with the average height across the dataset
    • This is not an advised solution, I just want to show you how the function works!
  • Here’s how we can use replace_na() to fill in the missing values:
hrs_05_height_filled <- hrs_00 |>
1  mutate(
2    height_filled = replace_na(
3      height,
4      mean(height, na.rm = TRUE))
    )
1
We need to wrap replace_na() in mutate() to create a new variable, height_filled
2
replace_na() swaps out any NA in height for the value we provide
3
height is the column we want to fill in.
4
We will replace the missing values with the mean of height
Check that the function worked 
hrs_05_height_filled |>
  select(contains("height")) |>
  arrange(height) |> tail()
     height height_filled
2723     NA      1.688729
2724     NA      1.688729
2725     NA      1.688729
2726     NA      1.688729
2727     NA      1.688729
2728     NA      1.688729

fct_na_value_to_level(): filling a factor variable

  • This is the forcats version of replace_na() for factor variables

 

  • We can use the fct_na_value_to_level() function inside mutate() to set NAs to a specific value

  • fct_na_value_to_level() works on factors!!

  • General version of the function is similar to replace_na():

data_filled <- data |>
1  mutate(
2    variable_filled = fct_na_value_to_level(
3      variable,
4      level = "new level to replace NAs"
      )
  )
1
We need to wrap fct_na_value_to_level() in mutate() to create a new variable, variable_filled
2
fct_na_value_to_level() swaps out any NA in variable for the level we provide
3
variable is the column we want to fill in
4
"new level to replace NAs" is added as a new level, and used to replace the missing values

Example 2: filling a factor variable

  • Suppose if race_original is missing, we want to replace it with Unknown
    • This is not an advised solution, I just want to show you how the function works!
  • Here’s how we can use replace_na() to fill in the missing values:
hrs_06_race_filled <- hrs_00 |>
  mutate(
1    race_filled = fct_na_value_to_level(
2      race_original,
3      level = "Unknown")
    )
1
fct_na_value_to_level() swaps out any NA in race_original for the level we provide
2
race_original is the column we want to fill in
3
"Unknown" is added as a new level, and used to replace the missing values
Check that the function worked 
hrs_06_race_filled |> tabyl(race_original, race_filled)
          race_original White/Caucasian Black/African American Other Unknown
        White/Caucasian            1000                      0     0       0
 Black/African American               0                    884     0       0
                  Other               0                      0   358       0
                   <NA>               0                      0     0     486

Replacing a factor level with NA

replace_with_na(): replace a factor level with NA

  • Sometimes there are data values that are not missing, but we want to treat them as missing
    • Example: you might see a numeric value as -99 or "999" to indicate missingness, but because it’s a valid number, R doesn’t treat it as NA
  • Note: there are no good examples of this in the HRS dataset, so I’m going to show you how to use the function with levels and values that should not be considered missing

 

  • General version of the function:
data_replaced_na <- data |>
1  replace_with_na(replace = list(
2    numeric_var1 = numeric_val1,
3    numeric_var2 = c(numeric_val2a, numeric_val2b, numeric_val2c),
4    factor_var1 = c("factor_level1", "factor_level2")
  ))
1
replace_with_na() takes a named list of columns and values to replace with NA
2
numeric_var1 is the name of a numeric column, and numeric_val1 is the value to replace with NA
3
numeric_var2 is the name of a numeric column, and numeric_val2a, numeric_val2b, and numeric_val2c are the values to replace with NA
4
factor_var1 is the name of a factor column, and "factor_level1" and "factor_level2" are the levels to replace with NA

Example 3: replace certain levels or values with NA (1/2)

  • We can replace a few values in the hrs_06_race_filled dataset with NA
hrs_07_other_na <- hrs_06_race_filled |>
1  mutate(
    sex_na = sex, 
    height_na = height,
    race_filled_na = race_filled
  ) |>
2  replace_with_na(replace = list(
3    sex_na = "Male",
4    height_na = 1.6002,
5    race_filled_na = c("Other", "Unknown")
  ))
1
I often make a new variable to hold the original values, so I can compare before and after
2
replace_with_na() takes a named list of columns and values to replace with NA
3
In the sex column, we will replace "Male" with NA
4
In the height column, we will replace 70 with NA
5
In the race_filled column, we will replace "Other" and "Unknown" with NA

Example 3: replace certain levels or values with NA (2/2)

  • We need to check our data to make sure we replaced the values we wanted to replace with NA
Check sex_na 
hrs_07_other_na |> 
  tabyl(sex, sex_na) |>
  gt() |>
  tab_options(table.font.size = px(35))
sex Female Male NA_
Female 1568 0 0
Male 0 0 1160
Check race_filled_na 
hrs_07_other_na |> 
  tabyl(race_filled, race_filled_na) |>
  gt() |>
  tab_options(table.font.size = px(35))
race_filled White/Caucasian Black/African American Other Unknown NA_
White/Caucasian 1000 0 0 0 0
Black/African American 0 884 0 0 0
Other 0 0 0 0 358
Unknown 0 0 0 0 486
Check height_na 
hrs_07_other_na |> 
  filter(is.na(height_na)) |>
  select(height, height_na) |>
  head()
  height height_na
1 1.6002        NA
2 1.6002        NA
3 1.6002        NA
4 1.6002        NA
5     NA        NA
6 1.6002        NA

Wrap-up

Wrap-up

  • We learned a few ways to handle missing data, once we know it’s there

  • drop_na() removes rows with missing data — target specific columns, or drop any row with any missingness

  • filter() silently drops NA rows unless you explicitly keep them with | is.na(...); filter_out() handles this for you automatically

  • replace_na() fills in missing values for numeric and character columns

  • fct_na_value_to_level() fills in missing values for factor columns, by adding a new explicit level

  • replace_with_na() goes the other direction: turns a specific value (or factor level) into NA

    • Always check your work afterward — compare before/after values with tabyl(), or compare miss_var_summary() before and after

  • There is no single “correct” way to handle missing data — the right choice depends on why the data is missing and what you’re trying to learn from it

Resources