Make sure tidyverse, magrittr and nycflights13 are loaded (library(tidyverse); library(magrittr); library(nycflights13)).

1 Introduction

Welcome to the hands-on workshop “Tidy Work in Tidyverse”. Most of the things necessary to complete the tutorials and challenges were covered in the lecture. However, sometimes the tasks require that you check the docs or search online. Not all our solutions are optimal. Let us know if you can do better or solve things in a different way. If stuck, look at hints, next google and if still stuck, turn to TA. It is a lot of material; sections marked Stretch are optional, do not feel bad if you do not solve all tasks. Good luck!

2 Pipes

  • Rewrite the following code chunk as one pipe (magrittr):
my_cars <- mtcars[, c(1:4, 7)]
my_cars <- my_cars[my_cars$disp > mean(my_cars$disp), ]
print(my_cars)
my_cars <- colMeans(my_cars)

my_cars <- mtcars %>%
  select(c(1:4, 7)) %>%
  filter(disp > mean(disp)) %T>%
  print() %>%
  colMeans()
  • Rewrite the correlations below using pipes.
cor(mtcars)

mtcars %>% cor()
Tip? You need to use different pipe for this one. (Check the cheatsheet at the end of the lab).
cor(mtcars$gear, mtcars$mpg)

mtcars %$% cor(gear, mpg)

3 Tibbles

  • Convert the mtcars dataset to a tibble vehicles.

vehicles <- mtcars %>% as_tibble()
  • Select the number of cylinders (cyl) variable using:
    • the [[index]] accessor,
    • the [[string]] accessor,
    • the $ accessor.

vehicles[['cyl']]
vehicles[[2]]
vehicles$cyl
##  [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
##  [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
##  [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
  • Do the same selection as above, but using pipe and placeholders (use all three ways of accessing a variable).
Tip?

1 - Which pipe allows you to print something but still pass the original data forward?
A. %>%
B. %T>%
C. %$%
D. |>

2 — Inside a pipe, which symbol refers to the current dataset?
A.$
B. {}
C. .
D. _

3 — Which structure lets you group multiple commands inside a pipe step?
A. () B. { ... }
C. [] D. <>

4 — Which extraction matches vehicles[['cyl']]?
A. .[2]
B. .$cyl
C. .[['cyl']]
D. [[2]]

5 — Which pipeline ending would extract the cyl column from the data?
A. %>% .$cyl
B. %T>% cyl
C. %>% cyl(.)
D. %$% cyl

vehicles %T>%
  {print(.[['cyl']])} %T>%
  {print(.[[2]])} %>%
  .$cyl
##  [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
##  [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
##  [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
  • Print the tibble.

vehicles
## # A tibble: 32 × 11
##      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  21       6  160    110  3.9   2.62  16.5     0     1     4     4
##  2  21       6  160    110  3.9   2.88  17.0     0     1     4     4
##  3  22.8     4  108     93  3.85  2.32  18.6     1     1     4     1
##  4  21.4     6  258    110  3.08  3.22  19.4     1     0     3     1
##  5  18.7     8  360    175  3.15  3.44  17.0     0     0     3     2
##  6  18.1     6  225    105  2.76  3.46  20.2     1     0     3     1
##  7  14.3     8  360    245  3.21  3.57  15.8     0     0     3     4
##  8  24.4     4  147.    62  3.69  3.19  20       1     0     4     2
##  9  22.8     4  141.    95  3.92  3.15  22.9     1     0     4     2
## 10  19.2     6  168.   123  3.92  3.44  18.3     1     0     4     4
## # ℹ 22 more rows
  • Print the 30 first rows of the tibble.

vehicles %>% head(n = 30)
## # A tibble: 30 × 11
##      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  21       6  160    110  3.9   2.62  16.5     0     1     4     4
##  2  21       6  160    110  3.9   2.88  17.0     0     1     4     4
##  3  22.8     4  108     93  3.85  2.32  18.6     1     1     4     1
##  4  21.4     6  258    110  3.08  3.22  19.4     1     0     3     1
##  5  18.7     8  360    175  3.15  3.44  17.0     0     0     3     2
##  6  18.1     6  225    105  2.76  3.46  20.2     1     0     3     1
##  7  14.3     8  360    245  3.21  3.57  15.8     0     0     3     4
##  8  24.4     4  147.    62  3.69  3.19  20       1     0     4     2
##  9  22.8     4  141.    95  3.92  3.15  22.9     1     0     4     2
## 10  19.2     6  168.   123  3.92  3.44  18.3     1     0     4     4
## # ℹ 20 more rows
  • Change the default behaviour of printing a tibble so that at least 15 and at most 30 rows are printed.

options(tibble.print_min = 15, tibble.print_max = 30)
  • Convert vehicles back to a data.frame called automobiles.

automobiles <- as.data.frame(vehicles)

Stretch: can a column of a tibble refer to a column defined just before it? Try to create a tibble with x (10 random integers 1-10) and y = log10(x). Does the same work in data.frame()?

tibble(x = sample(1:10, size = 10, replace = T), y = log10(x))
# data.frame(x = 1:3, y = log10(x)) fails: x is not found
## # A tibble: 10 × 2
##        x     y
##    <int> <dbl>
##  1     8 0.903
##  2     7 0.845
##  3     1 0    
##  4     5 0.699
##  5    10 1    
##  6     2 0.301
##  7     6 0.778
##  8     6 0.778
##  9     2 0.301
## 10     4 0.602

tibble() builds columns one after another, so later columns can use earlier ones.

4 NYC flights Challenge

The nycflights13 package contains information about all flights that departed from NYC (i.e., EWR, JFK and LGA) in 2013: 336,776 flights with 16 variables. To help understand what causes delays, it also includes a number of other useful datasets: weather, planes, airports, airlines. We will use it to train working with tibbles and dplyr.

4.1 Selecting columns

  • Load the nycflights13 package (install if necessary)

install.packages('nycflights13')
library(nycflights13)
  • Read about the data in the package docs

?nycflights13
  • Inspect the flights tibble.

flights
## # A tibble: 336,776 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      517            515         2      830            819
##  2  2013     1     1      533            529         4      850            830
##  3  2013     1     1      542            540         2      923            850
##  4  2013     1     1      544            545        -1     1004           1022
##  5  2013     1     1      554            600        -6      812            837
##  6  2013     1     1      554            558        -4      740            728
##  7  2013     1     1      555            600        -5      913            854
##  8  2013     1     1      557            600        -3      709            723
##  9  2013     1     1      557            600        -3      838            846
## 10  2013     1     1      558            600        -2      753            745
## 11  2013     1     1      558            600        -2      849            851
## 12  2013     1     1      558            600        -2      853            856
## 13  2013     1     1      558            600        -2      924            917
## 14  2013     1     1      558            600        -2      923            937
## 15  2013     1     1      559            600        -1      941            910
## # ℹ 336,761 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • Select all columns but carrier and arr_time

flights %>% select(-carrier, -arr_time)
## # A tibble: 336,776 × 17
##     year month   day dep_time sched_dep_time dep_delay sched_arr_time arr_delay
##    <int> <int> <int>    <int>          <int>     <dbl>          <int>     <dbl>
##  1  2013     1     1      517            515         2            819        11
##  2  2013     1     1      533            529         4            830        20
##  3  2013     1     1      542            540         2            850        33
##  4  2013     1     1      544            545        -1           1022       -18
##  5  2013     1     1      554            600        -6            837       -25
##  6  2013     1     1      554            558        -4            728        12
##  7  2013     1     1      555            600        -5            854        19
##  8  2013     1     1      557            600        -3            723       -14
##  9  2013     1     1      557            600        -3            846        -8
## 10  2013     1     1      558            600        -2            745         8
## 11  2013     1     1      558            600        -2            851        -2
## 12  2013     1     1      558            600        -2            856        -3
## 13  2013     1     1      558            600        -2            917         7
## 14  2013     1     1      558            600        -2            937       -14
## 15  2013     1     1      559            600        -1            910        31
## # ℹ 336,761 more rows
## # ℹ 9 more variables: flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## #   air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
  • Select carrier, tailnum and origin

flights %>% select(carrier, tailnum, origin)
## # A tibble: 336,776 × 3
##    carrier tailnum origin
##    <chr>   <chr>   <chr> 
##  1 UA      N14228  EWR   
##  2 UA      N24211  LGA   
##  3 AA      N619AA  JFK   
##  4 B6      N804JB  JFK   
##  5 DL      N668DN  LGA   
##  6 UA      N39463  EWR   
##  7 B6      N516JB  EWR   
##  8 EV      N829AS  LGA   
##  9 B6      N593JB  JFK   
## 10 AA      N3ALAA  LGA   
## 11 B6      N793JB  JFK   
## 12 B6      N657JB  JFK   
## 13 UA      N29129  JFK   
## 14 UA      N53441  EWR   
## 15 AA      N3DUAA  LGA   
## # ℹ 336,761 more rows
  • Hide columns from day through carrier

flights %>% select(-(day:carrier))
## # A tibble: 336,776 × 11
##     year month flight tailnum origin dest  air_time distance  hour minute
##    <int> <int>  <int> <chr>   <chr>  <chr>    <dbl>    <dbl> <dbl>  <dbl>
##  1  2013     1   1545 N14228  EWR    IAH        227     1400     5     15
##  2  2013     1   1714 N24211  LGA    IAH        227     1416     5     29
##  3  2013     1   1141 N619AA  JFK    MIA        160     1089     5     40
##  4  2013     1    725 N804JB  JFK    BQN        183     1576     5     45
##  5  2013     1    461 N668DN  LGA    ATL        116      762     6      0
##  6  2013     1   1696 N39463  EWR    ORD        150      719     5     58
##  7  2013     1    507 N516JB  EWR    FLL        158     1065     6      0
##  8  2013     1   5708 N829AS  LGA    IAD         53      229     6      0
##  9  2013     1     79 N593JB  JFK    MCO        140      944     6      0
## 10  2013     1    301 N3ALAA  LGA    ORD        138      733     6      0
## 11  2013     1     49 N793JB  JFK    PBI        149     1028     6      0
## 12  2013     1     71 N657JB  JFK    TPA        158     1005     6      0
## 13  2013     1    194 N29129  JFK    LAX        345     2475     6      0
## 14  2013     1   1124 N53441  EWR    SFO        361     2565     6      0
## 15  2013     1    707 N3DUAA  LGA    DFW        257     1389     6      0
## # ℹ 336,761 more rows
## # ℹ 1 more variable: time_hour <dttm>
  • Select all columns that have to do with arrival (arr_) (hint: contains, starts_with, matches, ends_with; more details here)

flights %>% select(contains('arr_'))
## # A tibble: 336,776 × 3
##    arr_time sched_arr_time arr_delay
##       <int>          <int>     <dbl>
##  1      830            819        11
##  2      850            830        20
##  3      923            850        33
##  4     1004           1022       -18
##  5      812            837       -25
##  6      740            728        12
##  7      913            854        19
##  8      709            723       -14
##  9      838            846        -8
## 10      753            745         8
## 11      849            851        -2
## 12      853            856        -3
## 13      924            917         7
## 14      923            937       -14
## 15      941            910        31
## # ℹ 336,761 more rows
  • Select columns based on a vector v <- c("arr_time", "sched_arr_time", "arr_delay")

v <- c("arr_time", "sched_arr_time", "arr_delay")
flights %>% select(all_of(v))
# select(one_of(v)) is the older (superseded) way
## # A tibble: 336,776 × 3
##    arr_time sched_arr_time arr_delay
##       <int>          <int>     <dbl>
##  1      830            819        11
##  2      850            830        20
##  3      923            850        33
##  4     1004           1022       -18
##  5      812            837       -25
##  6      740            728        12
##  7      913            854        19
##  8      709            723       -14
##  9      838            846        -8
## 10      753            745         8
## 11      849            851        -2
## 12      853            856        -3
## 13      924            917         7
## 14      923            937       -14
## 15      941            910        31
## # ℹ 336,761 more rows
  • Rename column dest to destination using select() and rename(). What is the difference between the two approaches?

flights %>% select(destination = dest) %>% head()
flights %>% rename(destination = dest) %>% head()
# select keeps only the renamed column while rename returns the whole dataset
# with the column renamed
## # A tibble: 6 × 1
##   destination
##   <chr>      
## 1 IAH        
## 2 IAH        
## 3 MIA        
## 4 BQN        
## 5 ATL        
## 6 ORD        
## # A tibble: 6 × 19
##    year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##   <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
## 1  2013     1     1      517            515         2      830            819
## 2  2013     1     1      533            529         4      850            830
## 3  2013     1     1      542            540         2      923            850
## 4  2013     1     1      544            545        -1     1004           1022
## 5  2013     1     1      554            600        -6      812            837
## 6  2013     1     1      554            558        -4      740            728
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, destination <chr>, air_time <dbl>,
## #   distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>

4.2 Filtering rows

  • Filter only the flights that arrived ahead of schedule

flights %>% filter(arr_delay < 0)
## # A tibble: 188,933 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      544            545        -1     1004           1022
##  2  2013     1     1      554            600        -6      812            837
##  3  2013     1     1      557            600        -3      709            723
##  4  2013     1     1      557            600        -3      838            846
##  5  2013     1     1      558            600        -2      849            851
##  6  2013     1     1      558            600        -2      853            856
##  7  2013     1     1      558            600        -2      923            937
##  8  2013     1     1      559            559         0      702            706
##  9  2013     1     1      559            600        -1      854            902
## 10  2013     1     1      600            600         0      851            858
## 11  2013     1     1      601            600         1      844            850
## 12  2013     1     1      602            610        -8      812            820
## 13  2013     1     1      606            610        -4      858            910
## 14  2013     1     1      606            610        -4      837            845
## 15  2013     1     1      607            607         0      858            915
## # ℹ 188,918 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • Filter the flights that had departure delay between 10 and 33

flights %>% filter(dep_delay >= 10, dep_delay <= 33) # or
flights %>% filter(between(dep_delay, 10, 33))
## # A tibble: 40,558 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      611            600        11      945            931
##  2  2013     1     1      623            610        13      920            915
##  3  2013     1     1      632            608        24      740            728
##  4  2013     1     1      743            730        13     1107           1100
##  5  2013     1     1      743            730        13     1059           1056
##  6  2013     1     1      851            840        11     1215           1206
##  7  2013     1     1      906            843        23     1134           1125
##  8  2013     1     1      912            900        12     1241           1220
##  9  2013     1     1      914            900        14     1058           1043
## 10  2013     1     1      920            905        15     1039           1025
## 11  2013     1     1      921            900        21     1237           1227
## 12  2013     1     1      930            905        25     1218           1209
## 13  2013     1     1      933            904        29     1252           1210
## 14  2013     1     1      953            921        32     1320           1241
## 15  2013     1     1     1011           1001        10     1133           1128
## # ℹ 40,543 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
## # A tibble: 40,558 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      611            600        11      945            931
##  2  2013     1     1      623            610        13      920            915
##  3  2013     1     1      632            608        24      740            728
##  4  2013     1     1      743            730        13     1107           1100
##  5  2013     1     1      743            730        13     1059           1056
##  6  2013     1     1      851            840        11     1215           1206
##  7  2013     1     1      906            843        23     1134           1125
##  8  2013     1     1      912            900        12     1241           1220
##  9  2013     1     1      914            900        14     1058           1043
## 10  2013     1     1      920            905        15     1039           1025
## 11  2013     1     1      921            900        21     1237           1227
## 12  2013     1     1      930            905        25     1218           1209
## 13  2013     1     1      933            904        29     1252           1210
## 14  2013     1     1      953            921        32     1320           1241
## 15  2013     1     1     1011           1001        10     1133           1128
## # ℹ 40,543 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • Fish out all flights with unknown arrival time

flights %>% filter(is.na(arr_time))
## # A tibble: 8,713 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1     2016           1930        46       NA           2220
##  2  2013     1     1       NA           1630        NA       NA           1815
##  3  2013     1     1       NA           1935        NA       NA           2240
##  4  2013     1     1       NA           1500        NA       NA           1825
##  5  2013     1     1       NA            600        NA       NA            901
##  6  2013     1     2     2041           2045        -4       NA           2359
##  7  2013     1     2     2145           2129        16       NA             33
##  8  2013     1     2       NA           1540        NA       NA           1747
##  9  2013     1     2       NA           1620        NA       NA           1746
## 10  2013     1     2       NA           1355        NA       NA           1459
## 11  2013     1     2       NA           1420        NA       NA           1644
## 12  2013     1     2       NA           1321        NA       NA           1536
## 13  2013     1     2       NA           1545        NA       NA           1910
## 14  2013     1     2       NA           1330        NA       NA           1640
## 15  2013     1     2       NA           1601        NA       NA           1735
## # ℹ 8,698 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • Retrieve rows 1234:1258 (hint: ?slice)

flights %>% slice(1234:1258)
## # A tibble: 25 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     2     1236           1238        -2     1446           1446
##  2  2013     1     2     1236           1240        -4     1403           1405
##  3  2013     1     2     1236           1200        36     1417           1330
##  4  2013     1     2     1239           1200        39     1429           1342
##  5  2013     1     2     1239           1220        19     1328           1320
##  6  2013     1     2     1240           1245        -5     1538           1600
##  7  2013     1     2     1240           1240         0     1556           1540
##  8  2013     1     2     1240           1245        -5     1521           1616
##  9  2013     1     2     1243           1229        14     1453           1428
## 10  2013     1     2     1244            900       224     1431           1104
## 11  2013     1     2     1245           1249        -4     1718           1800
## 12  2013     1     2     1249           1245         4     1555           1550
## 13  2013     1     2     1249           1258        -9     1535           1610
## 14  2013     1     2     1250           1245         5     1349           1350
## 15  2013     1     2     1251           1250         1     1552           1603
## 16  2013     1     2     1253           1300        -7     1505           1508
## 17  2013     1     2     1254           1300        -6     1545           1537
## 18  2013     1     2     1255           1259        -4     1501           1502
## 19  2013     1     2     1256           1252         4     1600           1555
## 20  2013     1     2     1256           1300        -4     1526           1518
## 21  2013     1     2     1256           1225        31     1409           1345
## 22  2013     1     2     1257           1303        -6     1356           1411
## 23  2013     1     2     1257           1300        -3     1529           1527
## 24  2013     1     2     1257           1257         0     1613           1607
## 25  2013     1     2     1258           1255         3     1537           1535
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • Sample (?slice_sample()) 3 random flights per day in March

flights %>% filter(month == 3) %>%
  group_by(day) %>%
  slice_sample(n = 3)
# older: sample_n(3)
## # A tibble: 93 × 19
## # Groups:   day [31]
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     3     1     2149           2145         4       55             34
##  2  2013     3     1     1418           1420        -2     1559           1620
##  3  2013     3     1     1930           1929         1     2149           2130
##  4  2013     3     2     1659           1700        -1     2154           2207
##  5  2013     3     2      621            630        -9      931            926
##  6  2013     3     2       NA           1247        NA       NA           1516
##  7  2013     3     3     2110           1900       130       49           2256
##  8  2013     3     3     1025           1030        -5     1147           1209
##  9  2013     3     3     1433           1436        -3     1554           1612
## 10  2013     3     4     1656           1700        -4     1919           1924
## 11  2013     3     4      558            600        -2      700            659
## 12  2013     3     4     2022           2000        22     2146           2136
## 13  2013     3     5     1732           1735        -3     2103           2115
## 14  2013     3     5     1508           1445        23     1754           1744
## 15  2013     3     5     2009           1936        33     2134           2100
## # ℹ 78 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • Show 5 most departure-delayed flights in January per carrier

flights %>%
  filter(month == 1) %>%
  group_by(carrier) %>%
  slice_max(dep_delay, n = 5)
# older: top_n(5, dep_delay)
## # A tibble: 77 × 19
## # Groups:   carrier [16]
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1    25       15           1815       360      208           1958
##  2  2013     1    26     1409            820       349     1528            937
##  3  2013     1    16     2013           1505       308     2136           1637
##  4  2013     1    25      123           2029       294      215           2140
##  5  2013     1     3     2056           1605       291     2239           1754
##  6  2013     1     2     1607           1030       337     2003           1355
##  7  2013     1     1     2205           1720       285       46           2040
##  8  2013     1    17     1130            715       255     1436           1045
##  9  2013     1    17     1012            610       242     1302            910
## 10  2013     1    22     2113           1720       233       23           2040
## 11  2013     1    20     2157           1815       222       46           2130
## 12  2013     1    31     2025           1815       130     2313           2130
## 13  2013     1    30     2006           1815       111     2247           2130
## 14  2013     1    28     1852           1815        37     2211           2130
## 15  2013     1    13     1846           1815        31     2132           2130
## # ℹ 62 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>
  • How many unique routes exists?

flights %>%
  mutate(route=paste(origin,"-",dest)) %>%
  distinct(route,.keep_all=T) %>%
  nrow()
## [1] 224
  • Which is the most frequent route?

flights %>%
  mutate(route=paste(origin,"-",dest)) %>% 
  count(route, sort = TRUE)

# JFK - LAX
## # A tibble: 224 × 2
##    route         n
##    <chr>     <int>
##  1 JFK - LAX 11262
##  2 LGA - ATL 10263
##  3 LGA - ORD  8857
##  4 JFK - SFO  8204
##  5 LGA - CLT  6168
##  6 EWR - ORD  6100
##  7 JFK - BOS  5898
##  8 LGA - MIA  5781
##  9 JFK - MCO  5464
## 10 EWR - BOS  5327
## 11 EWR - SFO  5127
## 12 LGA - DTW  5040
## 13 EWR - CLT  5026
## 14 EWR - ATL  5022
## 15 EWR - MCO  4941
## # ℹ 209 more rows

4.3 Trans(mutations)

  • air_time is the amount of time in minutes spent in the air. Add a new column air_spd that will contain aircraft’s airspeed in mph

flights %>% mutate(air_spd = distance/(air_time / 60))
## # A tibble: 336,776 × 20
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      517            515         2      830            819
##  2  2013     1     1      533            529         4      850            830
##  3  2013     1     1      542            540         2      923            850
##  4  2013     1     1      544            545        -1     1004           1022
##  5  2013     1     1      554            600        -6      812            837
##  6  2013     1     1      554            558        -4      740            728
##  7  2013     1     1      555            600        -5      913            854
##  8  2013     1     1      557            600        -3      709            723
##  9  2013     1     1      557            600        -3      838            846
## 10  2013     1     1      558            600        -2      753            745
## 11  2013     1     1      558            600        -2      849            851
## 12  2013     1     1      558            600        -2      853            856
## 13  2013     1     1      558            600        -2      924            917
## 14  2013     1     1      558            600        -2      923            937
## 15  2013     1     1      559            600        -1      941            910
## # ℹ 336,761 more rows
## # ℹ 12 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>, air_spd <dbl>
  • As above, but keep only the new air_spd variable

flights %>% transmute(air_spd = distance/(air_time / 60))
## # A tibble: 336,776 × 1
##    air_spd
##      <dbl>
##  1    370.
##  2    374.
##  3    408.
##  4    517.
##  5    394.
##  6    288.
##  7    404.
##  8    259.
##  9    405.
## 10    319.
## 11    414.
## 12    382.
## 13    430.
## 14    426.
## 15    324.
## # ℹ 336,761 more rows

4.4 Groups and counts

  • Use group_by(), summarise() and n() to see how many planes were delayed (departure) every month

flights %>%
  filter(dep_delay > 0) %>%
  group_by(month) %>%
  summarise(num_dep_delayed = n())
## # A tibble: 12 × 2
##    month num_dep_delayed
##    <int>           <int>
##  1     1            9662
##  2     2            9124
##  3     3           11209
##  4     4           10543
##  5     5           11291
##  6     6           12655
##  7     7           13909
##  8     8           11713
##  9     9            7815
## 10    10            8722
## 11    11            8239
## 12    12           13550
  • What was the mean dep_delay per month?

flights %>%
  group_by(month) %>%
  summarise(mean_dep_delay = mean(dep_delay, na.rm = T))
## # A tibble: 12 × 2
##    month mean_dep_delay
##    <int>          <dbl>
##  1     1          10.0 
##  2     2          10.8 
##  3     3          13.2 
##  4     4          13.9 
##  5     5          13.0 
##  6     6          20.8 
##  7     7          21.7 
##  8     8          12.6 
##  9     9           6.72
## 10    10           6.24
## 11    11           5.44
## 12    12          16.6
  • Count the number of incoming delayed flights from each unique origin and sort origins by this count (descending)

flights %>%
  filter(arr_delay > 0) %>%
  group_by(origin) %>%
  summarise(cnt = n()) %>%
  arrange(desc(cnt))
## # A tibble: 3 × 2
##   origin   cnt
##   <chr>  <int>
## 1 EWR    50099
## 2 JFK    42885
## 3 LGA    40020
  • Use summarise() to sum total dep_delay per month in hours

 flights %>%
   group_by(month) %>%
   summarize(tot_dep_delay = sum(dep_delay/60, na.rm = T))
## # A tibble: 12 × 2
##    month tot_dep_delay
##    <int>         <dbl>
##  1     1         4430.
##  2     2         4271.
##  3     3         6167.
##  4     4         6426.
##  5     5         6111.
##  6     6         9462.
##  7     7        10315.
##  8     8         6062.
##  9     9         3039.
## 10    10         2982.
## 11    11         2449.
## 12    12         7490.
  • Run group_size() on carrier what does it return?

flights %>%
    group_by(carrier) %>%
    group_size()
##  [1] 18460 32729   714 54635 48110 54173   685  3260   342 26397    32 58665
## [13] 20536  5162 12275   601
  • Use n_groups() to check the number of unique origin-carrier pairs,

flights %>%
    group_by(carrier, origin) %>% 
  n_groups()
## [1] 35

Note on ungroup Depending on the version of dplyr you may or may need to use the ungroup() if you want to group your data on some other variables. In the newer versions, summarise and mutate drop one aggregation level.

4.5 More transformations

  • Create a variable delay_class with case_when(): "early" if dep_delay < 0, "on time" if 0-15, "late" otherwise. Count flights per class.

flights %>%
  filter(!is.na(dep_delay)) %>%
  mutate(delay_class = case_when(dep_delay < 0 ~ "early",
                                 dep_delay <= 15 ~ "on time",
                                 .default = "late")) %>%
  count(delay_class)
## # A tibble: 3 × 2
##   delay_class      n
##   <chr>        <int>
## 1 early       183575
## 2 late         70774
## 3 on time      74172
  • Stretch: use across() to compute the mean of dep_delay and arr_delay per origin in one summarise().

flights %>%
  group_by(origin) %>%
  summarise(across(ends_with("_delay"), ~ mean(.x, na.rm = TRUE)))
## # A tibble: 3 × 3
##   origin dep_delay arr_delay
##   <chr>      <dbl>     <dbl>
## 1 EWR         15.1      9.11
## 2 JFK         12.1      5.55
## 3 LGA         10.3      5.78

5 Joins

The flights table only has a two-letter carrier code. The airlines, planes and airports tables hold the details.

  • Add the full airline name to flights (left_join). Which airline has the largest mean departure delay?

flights %>%
  left_join(airlines, by = "carrier") %>%
  group_by(name) %>%
  summarise(mean_dep_delay = mean(dep_delay, na.rm = TRUE)) %>%
  arrange(desc(mean_dep_delay))
## # A tibble: 16 × 2
##    name                        mean_dep_delay
##    <chr>                                <dbl>
##  1 Frontier Airlines Inc.               20.2 
##  2 ExpressJet Airlines Inc.             20.0 
##  3 Mesa Airlines Inc.                   19.0 
##  4 AirTran Airways Corporation          18.7 
##  5 Southwest Airlines Co.               17.7 
##  6 Endeavor Air Inc.                    16.7 
##  7 JetBlue Airways                      13.0 
##  8 Virgin America                       12.9 
##  9 SkyWest Airlines Inc.                12.6 
## 10 United Air Lines Inc.                12.1 
## 11 Envoy Air                            10.6 
## 12 Delta Air Lines Inc.                  9.26
## 13 American Airlines Inc.                8.59
## 14 Alaska Airlines Inc.                  5.80
## 15 Hawaiian Airlines Inc.                4.90
## 16 US Airways Inc.                       3.78
  • Which flights have no information in the planes table? (hint: anti_join). How many distinct tailnum is that?

flights %>% anti_join(planes, by = "tailnum") %>% distinct(tailnum) %>% nrow()
## [1] 722
  • Stretch: the key of airports is called faa, not dest. Join the destination airport names to flights and list the 5 most frequent destinations by name. (hint: by = c("dest" = "faa"))

flights %>%
  count(dest) %>%
  left_join(airports, by = c("dest" = "faa")) %>%
  select(dest, name, n) %>%
  slice_max(n, n = 5)
## # A tibble: 5 × 3
##   dest  name                                   n
##   <chr> <chr>                              <int>
## 1 ORD   Chicago Ohare Intl                 17283
## 2 ATL   Hartsfield Jackson Atlanta Intl    17215
## 3 LAX   Los Angeles Intl                   16174
## 4 BOS   General Edward Lawrence Logan Intl 15508
## 5 MCO   Orlando Intl                       14082

6 Tidy data

  • Look at table1, table2, table3 and table4a/table4b (all in tidyr). Which one is tidy? Why are the others not?

table1
# table1 is tidy: one row per country-year, one column per variable.
# table2: a variable (type) is spread across rows; table3: `rate` holds two values;
# table4a/4b: the year (a value) is in column names.
## # A tibble: 6 × 4
##   country      year  cases population
##   <chr>       <dbl>  <dbl>      <dbl>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583
  • Make table4a tidy (hint: pivot_longer).

table4a %>% pivot_longer(c(`1999`, `2000`), names_to = "year", values_to = "cases")
## # A tibble: 6 × 3
##   country     year   cases
##   <chr>       <chr>  <dbl>
## 1 Afghanistan 1999     745
## 2 Afghanistan 2000    2666
## 3 Brazil      1999   37737
## 4 Brazil      2000   80488
## 5 China       1999  212258
## 6 China       2000  213766
  • Make table2 tidy (hint: pivot_wider).

table2 %>% pivot_wider(names_from = type, values_from = count)
## # A tibble: 6 × 4
##   country      year  cases population
##   <chr>       <dbl>  <dbl>      <dbl>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583
  • Split rate in table3 into cases and population (hint: separate, convert = TRUE), then unite country and year into one column with unite.

table3 %>%
  separate(rate, into = c("cases", "population"), sep = "/", convert = TRUE) %>%
  unite("country_year", country, year, sep = "_")
## # A tibble: 6 × 3
##   country_year      cases population
##   <chr>             <int>      <int>
## 1 Afghanistan_1999    745   19987071
## 2 Afghanistan_2000   2666   20595360
## 3 Brazil_1999       37737  172006362
## 4 Brazil_2000       80488  174504898
## 5 China_1999       212258 1272915272
## 6 China_2000       213766 1280428583

7 Stretch: capstone

For each origin airport, find the carrier with the highest mean departure delay among carriers with at least 100 flights from that origin. Report the airline’s full name.

flights %>%
  group_by(origin, carrier) %>%
  summarise(n = n(), mean_dep_delay = mean(dep_delay, na.rm = TRUE), .groups = "drop") %>%
  filter(n >= 100) %>%
  group_by(origin) %>%
  slice_max(mean_dep_delay, n = 1) %>%
  left_join(airlines, by = "carrier")
## # A tibble: 3 × 5
## # Groups:   origin [3]
##   origin carrier     n mean_dep_delay name                    
##   <chr>  <chr>   <int>          <dbl> <chr>                   
## 1 EWR    EV      43939           20.2 ExpressJet Airlines Inc.
## 2 JFK    9E      14651           19.0 Endeavor Air Inc.       
## 3 LGA    F9        685           20.2 Frontier Airlines Inc.

8 Cheatsheet

8.1 Pipe

Make sure to load the relevant libraries.

Operator Package Short Explanation Example
%>% magrittr / tidyverse Forward pipe: passes the LHS as the first argument of the RHS function. Improves readability. mtcars %>% head(3)
%$% magrittr Exposition pipe: unpacks columns of a data frame so they can be used directly inside a function. mtcars %$% cor(mpg, hp)
%T>% magrittr Tee pipe: runs a function for side effects (e.g., plotting) but returns the original data. mtcars %T>% print() %>% summary()
\|> base R (R 4.1+) Base pipe: similar to %>% but stricter and simpler. LHS is inserted into _ placeholder if used. mtcars \|> head(3) or mtcars \|> (\(x) x[1:3, ])()

8.2 Placeholder

Placeholder Package Used With Short Explanation Example
. magrittr %>% pipes Insert LHS anywhere inside the RHS expression. mtcars %>% filter(mpg > mean(.$mpg))
.x purrr map(), walk(), etc. Represents the current element in purrr mapping. map(1:3, ~ .x^2)
.y purrr imap() Represents the index/name of the current element. imap(letters[1:3], ~ paste(.y, .x))
_ base R \|> Base-R placeholder: places the LHS at a named argument. mtcars \|> lm(mpg ~ hp, data = _)

8.3 Common dplyr, tidyr and tibble functions

Package Function What it does (short) Tiny example
dplyr filter() Keep rows that satisfy one or more logical conditions. flights %>% filter(dep_delay > 60)
dplyr select() Select or reorder columns by name, range, or helper functions. flights %>% select(year, month, day, dep_delay)
dplyr mutate() Add or modify columns while keeping all existing columns. flights %>% mutate(speed = distance / (air_time / 60))
dplyr transmute() Like mutate(), but drops all other columns, keeps only new ones. flights %>% transmute(speed = distance / (air_time / 60))
dplyr arrange() Reorder rows by one or more columns. flights %>% arrange(dep_delay)
dplyr arrange() + desc() Sort rows in descending order of a variable. flights %>% arrange(desc(dep_delay))
dplyr group_by() Define groups of rows for grouped operations (e.g. summarise). flights %>% group_by(carrier)
dplyr summarise() / summarize() Collapse each group to one row with summary stats. flights %>% summarise(mean_delay = mean(dep_delay, na.rm = TRUE))
dplyr distinct() Return unique rows based on selected columns. flights %>% distinct(carrier)
dplyr count() Shortcut for group_by(...) %>% summarise(n = n()). flights %>% count(carrier)
dplyr slice() Select rows by position (row numbers). flights %>% slice(1:5)
dplyr slice_sample() Randomly sample rows (replaces sample_n()). flights %>% slice_sample(n = 10)
dplyr slice_max() / slice_min() Keep the rows with the largest / smallest values (per group if grouped); replaces top_n(). flights %>% slice_max(dep_delay, n = 5)
dplyr rename() Rename columns while keeping all data. flights %>% rename(delay = dep_delay)
dplyr desc() Marks a variable for descending order in arrange() or top_n(). flights %>% arrange(desc(dep_delay))
dplyr between() Test if values lie between two endpoints (inclusive). flights %>% filter(between(dep_delay, 10, 30))
dplyr everything() Helper to select all remaining columns after those named. flights %>% select(year, month, day, everything())
dplyr contains() Select columns whose names contain a given string. flights %>% select(contains("delay"))
dplyr all_of() Select columns using a character vector of column names (replaces one_of()). vars <- c("year", "month"); flights %>% select(all_of(vars))
dplyr relocate() Move columns, by default to the front. flights %>% relocate(carrier)
dplyr case_when() Vectorised if/else-if for several conditions. mutate(cls = case_when(dep_delay < 0 ~ "early", .default = "late"))
dplyr across() Apply a function to several columns inside mutate/summarise. summarise(across(ends_with("delay"), ~ mean(.x, na.rm = TRUE)))
dplyr .by Group for one operation only (dplyr >= 1.1). flights %>% summarise(m = mean(dep_delay, na.rm = TRUE), .by = origin)
dplyr group_size() Returns the size (row count) of each group for a grouped data frame. flights %>% group_by(carrier) %>% group_size()
dplyr n_groups() Number of groups in a grouped data frame. flights %>% group_by(carrier) %>% n_groups()
dplyr inner_join() Join two data frames, keeping rows that match in both tables. inner_join(df1, df2, by = "id")
dplyr left_join() Keep all rows of the left table, add matching columns from the right. left_join(flights, airlines, by = "carrier")
dplyr anti_join() Rows of the left table with no match in the right table. anti_join(flights, planes, by = "tailnum")
dplyr join_by() Specify join keys, also with different names. left_join(x, y, by = join_by(id == key))
tidyr gather() (Superseded, use pivot_longer()) Convert wide columns into key–value long format. flights %>% gather(type, delay, dep_delay, arr_delay)
tidyr pivot_longer() Modern replacement for gather(): wide → long. flights %>% pivot_longer(c(dep_delay, arr_delay), names_to = "type", values_to = "delay")
tidyr pivot_wider() Long → wide, turning values of a column into new columns. delays %>% pivot_wider(names_from = type, values_from = delay)
tidyr separate() Split one column into multiple columns. df %>% separate(col = "x_y", into = c("x","y"), sep = "_")
tidyr unite() Combine multiple columns into a single one. df %>% unite("x_y", x, y, sep = "_")
tidyr complete() Ensure all combinations of given variables exist (fill in missing rows). df %>% complete(year, month)
tibble as_tibble() Convert an existing object (e.g. data.frame) into a tibble. as_tibble(df)
tibble tibble() Create a tibble (modern data frame) from scratch. tibble(x = 1:3, y = letters[1:3])