Practice 9

PUBH 523/623

Overview

This practice is about missing data: first finding and describing it, then doing something about it. You will use the naniar package to count and visualize missingness in the HRS dataset, then use functions from tidyr, dplyr, forcats, and naniar to filter, fill in, and recode missing values.

Topics covered (Lessons 34-35):

• Missing data: summarization and visualization
• Missing data: transformations

Directions

Download this practice .qmd from the course GitHub repository, rename it, and save it in your course folder.

• Download practice_09.qmd from GitHub.
• Rename the file to Lastname_Firstinitial_Practice_09.qmd (or lastname_firstinitial_practice_09.qmd), replacing with your actual last name and first initial.
• Save it inside your practice folder. You may create a practice_09 subfolder if you prefer.

TipHow to use this file

The boxes with the yellow stripe explain what to do. Keep these in your submitted file. Do all your work below and outside of the yellow-striped boxes. You will need to create code chunks within this document to run your R code.

Please delete the following sections before submitting:

• Overview
• Directions
• Tip boxes (green stripe)
• Caution boxes / Needed Steps (orange/red stripe)
• Submission checklist
• Note on AI usage

ImportantNeeded setup

You should already have hrs_data.rds from Practice 4 (link on the practice page). Place it in your data folder if you haven’t already. You can check out the codebook here.

Load the tidyverse, here, rio, naniar, janitor, and gt packages. You’ll need to install naniar, janitor, and gt if you haven’t already (you should already have the rest from previous weeks).

Questions

Question 1: Missingness statistics

In Lesson 34, we saw how to get quick, high-level counts of missingness before doing anything else with a dataset.

Part A: Import dataset

WarningTask

Import hrs_data.rds and assign it to hrs_data.

Part B: Total missingness in the dataset

WarningTask

Use n_miss() on hrs_data to find the total number of missing cells across the entire dataset.

Part C: Proportion of missingness in the dataset

WarningTask

Use prop_miss() on hrs_data to find the proportion of all cells in the dataset that are missing.

Part D: Missingness in a single variable

WarningTask

Use n_miss() on just the srh column of hrs_data to find how many values are missing for self-reported health.

TipTip

You can pull out a single column with $, e.g. hrs_data$srh.

Part E: Missingness summary table

WarningTask

Use miss_var_summary() on hrs_data to get one row per variable with its count and percent missing. Pipe the result into gt() to make it a nicer table. Assign the table to hrs_01_summary_table and display it.

Which variable has the most missing values? Which has the least?

Write your answer here.

Question 2: Visualizing missingness

In Lesson 34, we also covered a few plotting functions from naniar that help us see where and how data is missing.

Part A: A bird’s-eye view with vis_miss()

WarningTask

Use vis_miss() on hrs_data to visualize missingness across the whole dataset. Assign the plot to hrs_02_plot_vis_miss and display it.

Part B: Customize the plot

WarningTask

vis_miss() returns a ggplot2 object, so you can add layers to it just like the plots you built last week. Take hrs_02_plot_vis_miss and add a theme() layer that rotates the x-axis text labels by 20 degrees (angle = 20) so they’re easier to read. Assign the result to hrs_03_plot_vis_miss_clean and display it.

Looking at your plot, is missingness concentrated in a few variables, spread across many observations, or both?

Write your answer here.

TipTip

theme(axis.text.x = element_text(angle = 20, hjust = 0)) adjusts the angle and alignment of the x-axis text.

Part C: Missingness by group with gg_miss_fct()

WarningTask

Use gg_miss_fct() on hrs_data, stratifying by degree (set fct = degree). Assign the plot to hrs_04_plot_fct and display it.

Does missingness look similar across degree, or does one group have noticeably more missing data for certain variables?

Write your answer here.

Part D: Missingness intersections with gg_miss_upset()

WarningTask

Use gg_miss_upset() on hrs_data, setting nsets = 10 and text.scale = 2. Assign the plot to hrs_05_plot_upset and display it.

Which combination of variables is missing together most often? Is there a variable that tends to be missing mostly on its own?

Write your answer here.

Question 3: Filtering out missing values

In Lesson 35, we covered how to remove rows with missing data, and a common gotcha when using filter() on a column with NAs.

Part A: Check missingness before you drop anything

WarningTask

Before dropping any rows, pipe hrs_data into miss_var_summary() and then gt() again, just like Question 1 Part E, so you have a “before” table to compare to later. Assign it to hrs_06_summary_before.

TipTip

Getting in the habit of checking missingness before and after you drop or fill in values helps you catch mistakes, like accidentally dropping more rows than you meant to.

Part B: Drop rows missing anything

WarningTask

Use drop_na() on hrs_data (with no arguments) to remove any row that has any missing value. Assign the result to hrs_07_dropna_all. How many rows are left compared to hrs_data?

Write your answer here.

Part C: Drop rows missing specific columns

WarningTask

Now use drop_na() on hrs_data again, but this time only target the act_vig and bp columns, so rows are only dropped if one of those two columns is missing. Assign the result to hrs_08_dropna_cols. How does the number of rows compare to hrs_07_dropna_all?

Write your answer here.

Part D: The filter() gotcha

WarningTask

Use tabyl() (from janitor) on hrs_data$smoke_ever to see how many missing values smoke_ever has and confirm its categories.

Then, use filter() on hrs_data to keep only rows where smoke_ever != "No". Assign the result to hrs_09_filter_gotcha. Use tabyl() on the smoke_ever column of hrs_09_filter_gotcha to check what happened to the missing values.

Part E: Fixing the gotcha with filter_out()

WarningTask

Redo Part D, but use filter_out() instead of filter() to exclude smoke_ever == "No" while keeping the NAs. Assign the result to hrs_10_filter_out_fixed. Confirm with tabyl() that the missing values are still there this time.

Question 4: Filling in and recoding missing values

In Lesson 35, we also covered how to fill in missing values instead of dropping them, and how to turn a specific value into NA when needed.

Part A: Fill in a numeric variable

WarningTask

Use mutate() and replace_na() to create a new column, cesd_filled, in hrs_data that replaces any missing cesd (depression symptom score) with the mean cesd (use na.rm = TRUE inside mean()). Assign the result to hrs_11_cesd_filled.

Is filling in a missing value with the overall mean always a good idea? Why or why not?

Write your answer here.

Part B: Fill in a factor variable

WarningTask

Starting from hrs_11_cesd_filled, use mutate() and fct_na_value_to_level() to create a new column, degree_filled, that replaces any missing degree with a new level called "Unknown". Assign the result to hrs_12_degree_filled.

TipTip

replace_na() does not work on factor columns. Use fct_na_value_to_level() instead whenever the column you’re filling in is a factor.

Part C: Check your work

WarningTask

Use tabyl() to compare degree and degree_filled in hrs_12_degree_filled (hint: tabyl() can take two column names to cross-tabulate them). Confirm that every row that was NA in degree now shows "Unknown" in degree_filled.

Part D: Turn a valid value into NA

WarningTask

Suppose, just for practice, we wanted to treat everyone who answered "No" to drink (whether they drink alcohol) as missing instead (this is not something you’d actually want to do with real data, but it shows how the function works). First, use mutate() to make a copy of the column called drink_na. Then use replace_with_na() to replace "No" with NA in the drink_na column. Assign the result to hrs_13_replace_with_na.

Use tabyl() to compare the original drink column to your new drink_na column and confirm the replacement worked as expected.

Part E: Summarize your final, cleaned dataset

WarningTask

Take hrs_12_degree_filled (from Part B, before Part D’s practice-only recoding) and run miss_var_summary() and gt() on it one more time. Assign it to hrs_14_summary_after. Compare it to hrs_06_summary_before from Question 3 Part A. Which variables now show less missingness, and why?

Write your answer here.

Submission Checklist

If having a checklist helps you stay organized, check off each item below as you complete it. You do not need to submit this checklist.

• hrs_data imported (Q1 Part A)
• Total missing count found with n_miss() (Q1 Part B)
• Total missing proportion found with prop_miss() (Q1 Part C)
• Missing count for srh found with n_miss() (Q1 Part D)
• hrs_01_summary_table created with miss_var_summary() and gt(), with question answered (Q1 Part E)
• hrs_02_plot_vis_miss created with vis_miss() (Q2 Part A)
• hrs_03_plot_vis_miss_clean created with an added theme() layer, with question answered (Q2 Part B)
• hrs_04_plot_fct created with gg_miss_fct(), with question answered (Q2 Part C)
• hrs_05_plot_upset created with gg_miss_upset(), with question answered (Q2 Part D)
• hrs_06_summary_before created as a “before” missingness table (Q3 Part A)
• hrs_07_dropna_all created with drop_na(), with question answered (Q3 Part B)
• hrs_08_dropna_cols created with targeted drop_na(), with question answered (Q3 Part C)
• hrs_09_filter_gotcha created and checked with tabyl() (Q3 Part D)
• hrs_10_filter_out_fixed created with filter_out() and checked with tabyl() (Q3 Part E)
• hrs_11_cesd_filled created with replace_na(), with question answered (Q4 Part A)
• hrs_12_degree_filled created with fct_na_value_to_level() (Q4 Part B)
• degree vs. degree_filled checked with tabyl() (Q4 Part C)
• hrs_13_replace_with_na created with replace_with_na() and checked with tabyl() (Q4 Part D)
• hrs_14_summary_after created and compared to hrs_06_summary_before, with question answered (Q4 Part E)
• Document renders to HTML with no errors
• Files named correctly: Lastname_Firstinitial_Practice_09.qmd and Lastname_Firstinitial_Practice_09.html (or lowercase versions)
• Both .qmd and .html submitted to Brightspace

Note on AI usage

I used GenAI (Claude) to help me draft this practice assignment. It helped me brainstorm ideas for the assignment, and I directed it to help you complete the needed tasks that I showed in the lessons.