Tidy work in Tidyverse

Note

Welcome to the hands-on workshop “Tidy Work in Tidyverse”. Most of the functions necessary to complete the tutorials and challenges were covered in the lecture. However, sometimes the tasks require that you check the docs or search online. Our solutions are not the only possible ones! Let us know if you can do better or solve things in a different way!

If stuck, look at hints, next do some google searches and, if still stuck, turn to a TA.

It is a lot of material, we know! Do not feel bad if you do not solve all the tasks. If you completed Challenge 3, you have used all the most important features of tidyverse! Good luck!

1 General exercises

Datasets are available here.

1.1 Pipes

Use the base R pipe |> throughout the lab. We use a few magrittr pipes (%T>% and %$%) only in exercises that specifically demonstrate side effects or exposition.

1.1.1 Chunk 1

Rewrite the following code chunks as pipes:

my_cars <- mtcars[, c(1:4, 10)]
my_cars <- my_cars[my_cars$disp > mean(my_cars$disp), ]
my_cars <- colMeans(my_cars)

This is our solution:

Code

mtcars |> 
  select(c(1:4, 10)) |>
  filter(disp > mean(disp)) |>
  colMeans() -> my_cars

What is wrong with our solution?

Tip

It is better to have the result assigned on the left hand side: result <- expression. In this case the expression is the whole pipe.
Our ‘expression -> result’ is correct but can easily be missed when reading the code.
Even better would be to replace colMeans with summarise as the former one drops the tibble structure.

my_cars <- mtcars |> 
  select(1:4, 10) |> 
  filter(disp > mean(disp)) |> 
  summarise(across(everything(), mean))

1.1.2 Chunk 2

The summary(x) function is a bit special: when you type summary(x) in the console, print is called in an implicit way. Pipe call does not do such implicite call, so you will have to invoke print in an explicit way. But the %T>% does unbranch for one call only, you will have to make printing of the summary a one single composed call using {}. Try to wrap your mind around this. If in doubt, turn to a TA.

summary(cars)
colSums(cars)

Code

cars %T>% {print(summary(.))} |> colSums()

There is also a way of solving this using so-called anonymous function \(x), that is a function without a name (anonymous) that is used only once:

Code

cars |> 
  {\(x) {print(summary(x)); x}}() |> 
  colSums()

1.1.3 Chunk 3

Rewrite the following correlations using pipes.

cor(mtcars$gear, mtcars$mpg)

Code

mtcars %$% cor(gear, mpg)
# or using modern tidyverse syntax and pull():
mtcars |> 
  summarise(cor = cor(gear, mpg)) |> 
  pull(cor)

cor(mtcars)

Code

mtcars |> cor()

1.1.4 Chunk 4

Given is the dim_summary(nrows, ncols) function which takes nrows and ncols as arguments and prints this info:

dim_summary <- function(nrows, ncols) {
  print(
    paste0('Matrix M has: ', nrows, ' rows and ', ncols, ' columns.')
  )
}

Rewrite each of the code chunks below using pipes:

distr1 <- rnorm(16)
M <- matrix(distr1, ncol = 4)
plot(M)
M <- M + sample(M)
dim_summary(nrows = nrow(M), ncols = ncol(M))

distr2 <- rnorm(16)
N <- matrix(distr2, ncol = 4)
colnames(N) <- (letters[1:4])
summary(N)
N <- N + 0

P <- M %x% t(N)
heatmap(P)
colnames(P) <- letters[1:dim(P)[2]]
cor(P[ ,'a'], P[ ,'i'])

Tip

Beware of a class of functions, called replacement functions. These beasts are of the form: function(arguments) <- value and rownames(x) <- c('a', 'b', 'c') is a good example of such beast. When writing pipes, we have bear in mind that whole function <- is the name of the replacement function and thus we have to use it as such in the pipe enquoted using backticks. Yes, we know… but you wont see this too often.

Tip

Sometimes, it may not be possible to put everything into one single pipe and the results of running two or more pipes have to be used in the final pipe.

Code

dim_summary <- function(nrows, ncols) {
  print(paste0('Matrix M has: ', nrows, ' rows and ', ncols, ' columns.'))
}

M <- rnorm(16) |>
  matrix(ncol = 4) %T>%
  plot() |>
  `+`(., sample(.)) %T>%
  {dim_summary(nrow(.), ncol(.))}

N <- rnorm(16) |>
  matrix(ncol = 4) |>
  `colnames<-`(letters[1:4]) %T>%
  summary() |> `+`(., 0)

P <- M |>
  `%x%`(., t(N)) %T>%
  heatmap() |>
  `colnames<-`(letters[1:dim(.)[2]]) |>
  as_tibble() %$%
  cor(a, i)

1.2 Tibbles

1.2.1 Task 1

Convert the mtcars dataset to a tibble vehicles.
Select the number of cylinders (cyl) variable using:
- the [[index]] accessor,
- the [[string]] accessor,
- the $ accessor.
Do the same selection as above, but using pipe and placeholders (use all three ways of accessing a variable).
Print the tibble.
Print the 30 first rows of the tibble.
Change the default behavior of printing a tibble so that at least 15 and at most 30 rows are printed.
What is the difference between the tibble.print_max and dplyr.print_min? Is there any? Test it.
Convert vehicles back to a data.frame called automobiles.

Code

# 1
vehicles <- mtcars |> as_tibble()

# 2
vehicles[['cyl']]
vehicles[[2]]
vehicles$cyl

# 3
vehicles %T>%
  {print(.[['cyl']])} %T>%
  {print(.[[2]])} |>
  .$cyl

# 4
vehicles

# 5
vehicles |> head(n = 30)

# 6
options(tibble.print_min = 15, tibble.print_max = 30)

# 7
# In theory there should be no difference. dplyr imports tibble from the tibble package
# and dplyr.width, dplyr.print_min and dplyr.print_min are passed down to the tibble.
# But test both behaviours. First with only the tibble package loaded, later with dplyr # loaded.

# 8
automobiles <- as.data.frame(vehicles)

1.2.2 Task 2

Create the following tibble using tribble():

id	event	date
1	success	24-04-2017
2	failed	25-04-2017
3	failed	25-04-2017
4	success	27-04-2017

Code

tab <- tribble(
  ~id, ~event, ~date,
  1, 'success', '24-04-2017',
  2, 'failed', '25-04-2017',
  3, 'failed', '25-04-2017',
  4, 'success', '27-04-2017'
)

1.2.3 Task 3

Compare the performance of as.data.frame(), as_data_frame() and as_tibble() on a 100 x 30 matrix filled with some random integers. Use package microbenchmark. Fill in your result here in the Tidyverse Lab sheet, Tibbles – performance.

Code

tst <- replicate(30, sample(100), simplify = TRUE)
colnames(tst) = paste0(rep('col', times = dim(tst)[2]), 1:dim(tst)[2])
microbenchmark::microbenchmark(
  as.data.frame(tst),
  as_data_frame(tst),
  as_tibble(tst)
)

1.2.4 Task 4

Do you think tibbles are lazy? Try to create a tibble that tests whether lazy evaluation applies to tibbles too.

Code

tibble(x = sample(1:10, size = 10, replace = T), y = log10(x))

1.3 Parsing

Parse the following vectors using parse_ functions:

vec1 <- c(1, 7.2, 3.84, -5.23) – parse it as double (any problems? why?).
Now, parse the same vector c(1, 7.2, 3.84, -5.23) as integer. What happens?
Can you still parse it as integer somehow?
Parse as double vec2 <- c('2', '3,45', '?', '-7,28')
Parse correctly vec3 <- c('2', '3,45', '?', '-7.28')
Parse the following guessing the parser: vec4 <- c('barrel: 432.7$', 'liter: 15.42PLN', 'gallon costs approx 32.1SEK', 'sunny, wind gusts up till 55m/s')
Can you parse vec4 as number? Do it if you can.
Parse vec5 <- "25 Dec 2015" as date (hint: ?parse_date()).
Parse 10_Jul_1410 as date.

Code

vec1 <- c(1, 7.2, 3.84, -5.23)
vec2 <- c('2', '3,45', '?', '-7,28')
vec3 <- c('2', '3,45', '?', '-7.28')
vec4 <- c('barrel: 432.7$', 'liter: 15.42PLN', 'gallon costs approx 32.1SEK', 'sunny, wind gusts up till 55m/s')
vec5 <- "25 Dec 2015"
parse_double(vec1)
parse_integer(vec1)
parse_integer(as.integer(vec1)) # Is it the best way? Hint: rounding.
parse_double(vec2, na = '?', locale = locale(decimal_mark = ','))
parse_number(vec3, na = '?', locale = locale(decimal_mark = '.'))
guess_parser(vec4)
parse_guess(vec4)
# Yes, you can:
parse_number(vec4)
parse_date(vec5, format="%d %b %Y")
parse_date("10_Jul_1410", format="%d%.%b%.%Y")

2 NYC flights Challenge

The nycflights13 package contains information about all flights that departed from NYC (i.e., EWR, JFK and LGA) in 2013: 336,776 flights with 16 variables. To help understand what causes delays, it also includes a number of other useful datasets: weather, planes, airports, airlines. We will use it to train working with tibbles and dplyr.

2.1 Task 1: Selecting column

Load the nycflights13 package (install if necessary).
Read about the data in the package docs.
Inspect the flights tibble.
Select all columns but carrier and arr_time.
Select carrier, tailnum and origin.
Hide columns from day through carrier.
Select all columns that have to do with arrival (hint: ?tidyselect).
Select columns based on a vector v <- c("arr_time", "sched_arr_time", "arr_delay").
Rename column dest to destination using:
- select() and
- rename()

What is the difference between the two approaches?

Code

install.packages('nycflights13')

library('nycflights13')

?nycflights13

flights

flights |> select(-carrier, -arr_time)

flights |> select(carrier, tailnum, origin)

flights |> select(-(day:carrier))

flights |> select(contains('arr_')) # or

v <- c("arr_time", "sched_arr_time", "arr_delay")
flights |> select(v) # ambiguous, or better
flights |> select(all_of(v))

flights |> select(destination = dest)
flights |> rename(destination = dest)
# select keeps only the renamed column while rename returns the whole dataset
# with the column renamed.

2.2 Task 2: Filtering rows

Filter only the flights that arrived ahead of schedule.

Code

flights |> filter(arr_delay < 0)

Filter the flights that had departure delay between 10 and 33.

Code

flights |> filter(dep_delay >= 10, dep_delay <= 33) # or
flights |> filter(between(dep_delay, 10, 33))

Fish out all flights with unknown arrival time.

Code

flights |> filter(is.na(arr_time))

Retrieve rows 1234:1258 (hint: ?slice).

Code

flights |> slice(1234:1258)

Sample (?sample_n()) 3 random flights per day in March.

Code

nycflights13::flights |> filter(month == 3) |>
  group_by(day) |>
  slice_sample(n = 3)

Show 5 most departure-delayed flights in January per carrier.

Code

nycflights13::flights |>
  filter(month == 1) |>
  group_by(carrier) |>
  slice_max(dep_delay, n = 5)

Retrieve all unique() routes and sort them by destination.

Code

nycflights13::flights |>
  select(origin, dest) |>
  unique() |>
  arrange(dest)

nycflights13::flights |>
  mutate(route = paste(origin, dest, sep="-")) |>
  select(route) |>
  unique()

Retrieve all distinct() routes and sort them by destination.

Code

nycflights13::flights |>
  distinct(origin, dest) |>
  arrange(dest)

2.3 Task 3: Trans(mutations) and `pick()`s

air_time is the amount of time in minutes spent in the air. Add a new column air_spd that will contain aircraft’s airspeed in mph.
As above, but keep only the new air_spd variable.
Use rownames_to_column() on mtcars to add car model as an extra column.
Use pick() on flights to select specific columns beginning with “air” and the “distance” column

Code

flights |> mutate(air_spd = distance/(air_time / 60))
flights |> transmute(air_spd = distance/(air_time / 60))
mtcars |> rownames_to_column('model')
flights |> mutate(air_spd = distance/(air_time / 60)) |> select(pick(starts_with("air")), distance)

2.4 Task 4: Groups and counts

Use .by or group_by(), summarise() and n() to see how many planes were delayed (departure) every month.

Code

flights |>
  filter(dep_delay > 0) |>
  group_by(month) |>
  summarise(num_dep_delayed = n())
# or even better:
flights |>
  filter(dep_delay > 0) |>
  summarise(num_dep_delayed = n(), .by = month)

Do the same but using count().

Code

flights |>
  filter(dep_delay > 0) |>
  count(month, name = "num_dep_delayed")

What was the mean dep_delay per month?

Code

flights |>
  summarise(mean_dep_delay = mean(dep_delay, na.rm = T), 
            .by = month)

Count the number of incoming delayed flights from each unique origin and sort origins by this count (descending).

Code

flights |>
  filter(arr_delay > 0) |>
  summarise(cnt = n(), .by = origin) |>
  arrange(desc(cnt))

Use summarise() to sum total dep_delay per month in hours.

Code

flights |>
 summarise(tot_dep_delay = sum(dep_delay/60, na.rm = T), 
           .by = month)

Use the wt parameter of count() to achieve the same.

Code

flights |>
 group_by(month) |>
 count(wt = dep_delay/60)

Run group_size() on carrier what does it return?

Code

flights |>
    group_by(carrier) |>
    group_size()

Use n_groups() to check the number of unique origin-carrier pairs.

Code

flights |>
    group_by(carrier) |>
    n_groups()

Note on ungroup

Depending on the version of dplyr, you may or may need to use the ungroup() if you want to group your data on some other variables. In the newer versions, summarise and mutate drop one aggregation level.

flights |>
  group_by(origin) |>
  mutate(mean_delay_orig = (mean(dep_delay, na.rm = T) + mean(arr_delay, na.rm = T)) / 2) |>
  ungroup() |>
  group_by(carrier) |>
  mutate(mean_delay_carr = (mean(dep_delay, na.rm = T) + mean(arr_delay, na.rm = T)) / 2) |>
  select(origin, carrier, mean_delay_orig, mean_delay_carr)

2.5 Task 5: Joins

Given the following tibbles set1 and set2:

set1 <- tribble(
  ~id, ~color,
  'id1', 'grey',
  'id1', 'red',
  'id2', 'green',
  'id3', 'blue'
)

set2 <- tribble(
  ~id, ~size,
  'id2', 'XL',
  'id3', 'M',
  'id4', 'M'
)

set1
set2

id	color
id1	grey
id1	red
id2	green
id3	blue

id	size
id2	XL
id3	M
id4	M

Perform joins on id that result in the grey area from the Venn diagrams below. We have not talked about all possible joins, so read the docs if you do not know which join to use.

Code

left_join(set1, set2, by = join_by('id'))
# or the old but still valid way:
left_join(set1, set2, by = 'id')

Code

right_join(set1, set2, by = join_by('id'))

Code

inner_join(set1, set2, by = join_by('id')) # or
semi_join(set1, set2, by = join_by('id')) # semi_join removes duplicates in x
# and also returns only columns from x.

Code

full_join(set1, set2, by = 'id') # or

Code

anti_join(set1, set2, by = 'id')

3 Tidying data

Now time to do some data tidying. First install a package with some untidy data:

#renv::install("rstudio/EDAWR")
library(EDAWR)

Tidy cases so that years are not in separate columns, but in the column called year containing a value per each year.

Code

tidy_cases <- cases |>
  pivot_longer(-country, names_to = "year", values_to = "count")

Now time for the pollution dataset. Tidy it so that there separate columns for large and small pollution values.

Code

tidy_pollution <- pollution |>
  pivot_wider(city, names_from = size, values_from = amount)

The storms dataset contains the date column. Make it into 3 columns: year, month and day. Store the result as tidy_storms.

Code

tidy_storms <- storms |>
  separate(col = date,
           into = c("year", "month", "day"),
           sep = "-")

Now, merge year, month and day in tidy_storms into a date column again but in the “DD/MM/YYYY” format.

Code

tidy_storms |> unite(col = "date", 4:6, sep = "/")

4 Wildlife Aircraft Strikes Challenge

Use the FAA report and tidyverse to learn more about aircraft incidents with wildlife. Use your imagination and NYC data science blog for inspiration!

5 Session

Click here

sessionInfo()

R version 4.6.0 (2026-04-24)
Platform: aarch64-apple-darwin23
Running under: macOS Tahoe 26.5.2

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/Stockholm
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] microbenchmark_1.5.0 here_1.0.2           EDAWR_0.1           
 [4] eulerr_8.1.0         bsplus_0.1.5         lubridate_1.9.5     
 [7] forcats_1.0.1        stringr_1.6.0        dplyr_1.2.1         
[10] purrr_1.2.2          readr_2.2.0          tidyr_1.3.2         
[13] tibble_3.3.1         tidyverse_2.0.0      ggplot2_4.0.3       

loaded via a namespace (and not attached):
 [1] gtable_0.3.6       jsonlite_2.0.0     compiler_4.6.0     tidyselect_1.2.1  
 [5] parallel_4.6.0     scales_1.4.0       yaml_2.3.12        fastmap_1.2.0     
 [9] R6_2.6.1           generics_0.1.4     knitr_1.51         htmlwidgets_1.6.4 
[13] rprojroot_2.1.1    pillar_1.11.1      RColorBrewer_1.1-3 tzdb_0.5.0        
[17] rlang_1.3.0        stringi_1.8.9      xfun_0.60          S7_0.2.2          
[21] otel_0.2.0         timechange_0.4.0   cli_3.6.6          withr_3.0.3       
[25] magrittr_2.0.5     digest_0.6.39      grid_4.6.0         hms_1.1.4         
[29] lifecycle_1.0.5    vctrs_0.7.3        evaluate_1.0.5     glue_1.8.1        
[33] farver_2.1.2       rmarkdown_2.31     tools_4.6.0        pkgconfig_2.0.3   
[37] htmltools_0.5.9

1 General exercises

1.1 Pipes

1.1.1 Chunk 1

1.1.2 Chunk 2

1.1.3 Chunk 3

1.1.4 Chunk 4

1.2 Tibbles

1.2.1 Task 1

1.2.2 Task 2

1.2.3 Task 3

1.2.4 Task 4

1.3 Parsing

2 NYC flights Challenge

2.1 Task 1: Selecting column

2.2 Task 2: Filtering rows

2.3 Task 3: Trans(mutations) and pick()s

2.4 Task 4: Groups and counts

2.5 Task 5: Joins

3 Tidying data

4 Wildlife Aircraft Strikes Challenge

5 Session

2.3 Task 3: Trans(mutations) and `pick()`s