Make sure tidyverse, magrittr and nycflights13 are loaded (library(tidyverse); library(magrittr); library(nycflights13)).
Welcome to the hands-on workshop “Tidy Work in Tidyverse”. Most of the things necessary to complete the tutorials and challenges were covered in the lecture. However, sometimes the tasks require that you check the docs or search online. Not all our solutions are optimal. Let us know if you can do better or solve things in a different way. If stuck, look at hints, next google and if still stuck, turn to TA. It is a lot of material; sections marked Stretch are optional, do not feel bad if you do not solve all tasks. Good luck!
magrittr):my_cars <- mtcars[, c(1:4, 7)]
my_cars <- my_cars[my_cars$disp > mean(my_cars$disp), ]
print(my_cars)
my_cars <- colMeans(my_cars)
my_cars <- mtcars %>%
select(c(1:4, 7)) %>%
filter(disp > mean(disp)) %T>%
print() %>%
colMeans()
cor(mtcars)
mtcars %>% cor()
cor(mtcars$gear, mtcars$mpg)
mtcars %$% cor(gear, mpg)
mtcars dataset to a tibble vehicles.
vehicles <- mtcars %>% as_tibble()
cyl) variable using:
[[index]] accessor,[[string]] accessor,$ accessor.
vehicles[['cyl']]
vehicles[[2]]
vehicles$cyl
## [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
## [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
## [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
1 - Which pipe allows you to print something but still pass the original data forward?
A. %>%
B. %T>%
C. %$%
D. |>
2 — Inside a pipe, which symbol refers to the current dataset?
A.$
B. {}
C. .
D. _
3 — Which structure lets you group multiple commands inside a pipe step?
A. ()
B. { ... }
C. []
D. <>
4 — Which extraction matches vehicles[['cyl']]?
A. .[2]
B. .$cyl
C. .[['cyl']]
D. [[2]]
cyl column from the data?%>% .$cyl%T>% cyl%>% cyl(.)%$% cyl
vehicles %T>%
{print(.[['cyl']])} %T>%
{print(.[[2]])} %>%
.$cyl
## [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
## [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
## [1] 6 6 4 6 8 6 8 4 4 6 6 8 8 8 8 8 8 4 4 4 4 8 8 8 8 4 4 4 8 6 8 4
vehicles
## # A tibble: 32 × 11
## mpg cyl disp hp drat wt qsec vs am gear carb
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 21 6 160 110 3.9 2.62 16.5 0 1 4 4
## 2 21 6 160 110 3.9 2.88 17.0 0 1 4 4
## 3 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1
## 4 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1
## 5 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2
## 6 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1
## 7 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4
## 8 24.4 4 147. 62 3.69 3.19 20 1 0 4 2
## 9 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2
## 10 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4
## # ℹ 22 more rows
vehicles %>% head(n = 30)
## # A tibble: 30 × 11
## mpg cyl disp hp drat wt qsec vs am gear carb
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 21 6 160 110 3.9 2.62 16.5 0 1 4 4
## 2 21 6 160 110 3.9 2.88 17.0 0 1 4 4
## 3 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1
## 4 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1
## 5 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2
## 6 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1
## 7 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4
## 8 24.4 4 147. 62 3.69 3.19 20 1 0 4 2
## 9 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2
## 10 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4
## # ℹ 20 more rows
options(tibble.print_min = 15, tibble.print_max = 30)
vehicles back to a data.frame called automobiles.
automobiles <- as.data.frame(vehicles)
Stretch: can a column of a tibble refer to a column defined just before it? Try to create a tibble with x (10 random integers 1-10) and y = log10(x). Does the same work in data.frame()?
tibble(x = sample(1:10, size = 10, replace = T), y = log10(x))
# data.frame(x = 1:3, y = log10(x)) fails: x is not found
## # A tibble: 10 × 2
## x y
## <int> <dbl>
## 1 8 0.903
## 2 7 0.845
## 3 1 0
## 4 5 0.699
## 5 10 1
## 6 2 0.301
## 7 6 0.778
## 8 6 0.778
## 9 2 0.301
## 10 4 0.602
tibble() builds columns one after another, so later columns can use earlier ones.
The nycflights13 package contains information about all flights that departed from NYC (i.e., EWR, JFK and LGA) in 2013: 336,776 flights with 16 variables. To help understand what causes delays, it also includes a number of other useful datasets: weather, planes, airports, airlines. We will use it to train working with tibbles and dplyr.
nycflights13 package (install if necessary)
install.packages('nycflights13')
library(nycflights13)
?nycflights13
flights tibble.
flights
## # A tibble: 336,776 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 517 515 2 830 819
## 2 2013 1 1 533 529 4 850 830
## 3 2013 1 1 542 540 2 923 850
## 4 2013 1 1 544 545 -1 1004 1022
## 5 2013 1 1 554 600 -6 812 837
## 6 2013 1 1 554 558 -4 740 728
## 7 2013 1 1 555 600 -5 913 854
## 8 2013 1 1 557 600 -3 709 723
## 9 2013 1 1 557 600 -3 838 846
## 10 2013 1 1 558 600 -2 753 745
## 11 2013 1 1 558 600 -2 849 851
## 12 2013 1 1 558 600 -2 853 856
## 13 2013 1 1 558 600 -2 924 917
## 14 2013 1 1 558 600 -2 923 937
## 15 2013 1 1 559 600 -1 941 910
## # ℹ 336,761 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
carrier and arr_time
flights %>% select(-carrier, -arr_time)
## # A tibble: 336,776 × 17
## year month day dep_time sched_dep_time dep_delay sched_arr_time arr_delay
## <int> <int> <int> <int> <int> <dbl> <int> <dbl>
## 1 2013 1 1 517 515 2 819 11
## 2 2013 1 1 533 529 4 830 20
## 3 2013 1 1 542 540 2 850 33
## 4 2013 1 1 544 545 -1 1022 -18
## 5 2013 1 1 554 600 -6 837 -25
## 6 2013 1 1 554 558 -4 728 12
## 7 2013 1 1 555 600 -5 854 19
## 8 2013 1 1 557 600 -3 723 -14
## 9 2013 1 1 557 600 -3 846 -8
## 10 2013 1 1 558 600 -2 745 8
## 11 2013 1 1 558 600 -2 851 -2
## 12 2013 1 1 558 600 -2 856 -3
## 13 2013 1 1 558 600 -2 917 7
## 14 2013 1 1 558 600 -2 937 -14
## 15 2013 1 1 559 600 -1 910 31
## # ℹ 336,761 more rows
## # ℹ 9 more variables: flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
carrier, tailnum and origin
flights %>% select(carrier, tailnum, origin)
## # A tibble: 336,776 × 3
## carrier tailnum origin
## <chr> <chr> <chr>
## 1 UA N14228 EWR
## 2 UA N24211 LGA
## 3 AA N619AA JFK
## 4 B6 N804JB JFK
## 5 DL N668DN LGA
## 6 UA N39463 EWR
## 7 B6 N516JB EWR
## 8 EV N829AS LGA
## 9 B6 N593JB JFK
## 10 AA N3ALAA LGA
## 11 B6 N793JB JFK
## 12 B6 N657JB JFK
## 13 UA N29129 JFK
## 14 UA N53441 EWR
## 15 AA N3DUAA LGA
## # ℹ 336,761 more rows
day through carrier
flights %>% select(-(day:carrier))
## # A tibble: 336,776 × 11
## year month flight tailnum origin dest air_time distance hour minute
## <int> <int> <int> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl>
## 1 2013 1 1545 N14228 EWR IAH 227 1400 5 15
## 2 2013 1 1714 N24211 LGA IAH 227 1416 5 29
## 3 2013 1 1141 N619AA JFK MIA 160 1089 5 40
## 4 2013 1 725 N804JB JFK BQN 183 1576 5 45
## 5 2013 1 461 N668DN LGA ATL 116 762 6 0
## 6 2013 1 1696 N39463 EWR ORD 150 719 5 58
## 7 2013 1 507 N516JB EWR FLL 158 1065 6 0
## 8 2013 1 5708 N829AS LGA IAD 53 229 6 0
## 9 2013 1 79 N593JB JFK MCO 140 944 6 0
## 10 2013 1 301 N3ALAA LGA ORD 138 733 6 0
## 11 2013 1 49 N793JB JFK PBI 149 1028 6 0
## 12 2013 1 71 N657JB JFK TPA 158 1005 6 0
## 13 2013 1 194 N29129 JFK LAX 345 2475 6 0
## 14 2013 1 1124 N53441 EWR SFO 361 2565 6 0
## 15 2013 1 707 N3DUAA LGA DFW 257 1389 6 0
## # ℹ 336,761 more rows
## # ℹ 1 more variable: time_hour <dttm>
arr_) (hint: contains, starts_with, matches, ends_with; more details here)
flights %>% select(contains('arr_'))
## # A tibble: 336,776 × 3
## arr_time sched_arr_time arr_delay
## <int> <int> <dbl>
## 1 830 819 11
## 2 850 830 20
## 3 923 850 33
## 4 1004 1022 -18
## 5 812 837 -25
## 6 740 728 12
## 7 913 854 19
## 8 709 723 -14
## 9 838 846 -8
## 10 753 745 8
## 11 849 851 -2
## 12 853 856 -3
## 13 924 917 7
## 14 923 937 -14
## 15 941 910 31
## # ℹ 336,761 more rows
v <- c("arr_time", "sched_arr_time", "arr_delay")
v <- c("arr_time", "sched_arr_time", "arr_delay")
flights %>% select(all_of(v))
# select(one_of(v)) is the older (superseded) way
## # A tibble: 336,776 × 3
## arr_time sched_arr_time arr_delay
## <int> <int> <dbl>
## 1 830 819 11
## 2 850 830 20
## 3 923 850 33
## 4 1004 1022 -18
## 5 812 837 -25
## 6 740 728 12
## 7 913 854 19
## 8 709 723 -14
## 9 838 846 -8
## 10 753 745 8
## 11 849 851 -2
## 12 853 856 -3
## 13 924 917 7
## 14 923 937 -14
## 15 941 910 31
## # ℹ 336,761 more rows
dest to destination using select() and rename(). What is the difference between the two approaches?
flights %>% select(destination = dest) %>% head()
flights %>% rename(destination = dest) %>% head()
# select keeps only the renamed column while rename returns the whole dataset
# with the column renamed
## # A tibble: 6 × 1
## destination
## <chr>
## 1 IAH
## 2 IAH
## 3 MIA
## 4 BQN
## 5 ATL
## 6 ORD
## # A tibble: 6 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 517 515 2 830 819
## 2 2013 1 1 533 529 4 850 830
## 3 2013 1 1 542 540 2 923 850
## 4 2013 1 1 544 545 -1 1004 1022
## 5 2013 1 1 554 600 -6 812 837
## 6 2013 1 1 554 558 -4 740 728
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, destination <chr>, air_time <dbl>,
## # distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
flights %>% filter(arr_delay < 0)
## # A tibble: 188,933 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 544 545 -1 1004 1022
## 2 2013 1 1 554 600 -6 812 837
## 3 2013 1 1 557 600 -3 709 723
## 4 2013 1 1 557 600 -3 838 846
## 5 2013 1 1 558 600 -2 849 851
## 6 2013 1 1 558 600 -2 853 856
## 7 2013 1 1 558 600 -2 923 937
## 8 2013 1 1 559 559 0 702 706
## 9 2013 1 1 559 600 -1 854 902
## 10 2013 1 1 600 600 0 851 858
## 11 2013 1 1 601 600 1 844 850
## 12 2013 1 1 602 610 -8 812 820
## 13 2013 1 1 606 610 -4 858 910
## 14 2013 1 1 606 610 -4 837 845
## 15 2013 1 1 607 607 0 858 915
## # ℹ 188,918 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
flights %>% filter(dep_delay >= 10, dep_delay <= 33) # or
flights %>% filter(between(dep_delay, 10, 33))
## # A tibble: 40,558 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 611 600 11 945 931
## 2 2013 1 1 623 610 13 920 915
## 3 2013 1 1 632 608 24 740 728
## 4 2013 1 1 743 730 13 1107 1100
## 5 2013 1 1 743 730 13 1059 1056
## 6 2013 1 1 851 840 11 1215 1206
## 7 2013 1 1 906 843 23 1134 1125
## 8 2013 1 1 912 900 12 1241 1220
## 9 2013 1 1 914 900 14 1058 1043
## 10 2013 1 1 920 905 15 1039 1025
## 11 2013 1 1 921 900 21 1237 1227
## 12 2013 1 1 930 905 25 1218 1209
## 13 2013 1 1 933 904 29 1252 1210
## 14 2013 1 1 953 921 32 1320 1241
## 15 2013 1 1 1011 1001 10 1133 1128
## # ℹ 40,543 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
## # A tibble: 40,558 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 611 600 11 945 931
## 2 2013 1 1 623 610 13 920 915
## 3 2013 1 1 632 608 24 740 728
## 4 2013 1 1 743 730 13 1107 1100
## 5 2013 1 1 743 730 13 1059 1056
## 6 2013 1 1 851 840 11 1215 1206
## 7 2013 1 1 906 843 23 1134 1125
## 8 2013 1 1 912 900 12 1241 1220
## 9 2013 1 1 914 900 14 1058 1043
## 10 2013 1 1 920 905 15 1039 1025
## 11 2013 1 1 921 900 21 1237 1227
## 12 2013 1 1 930 905 25 1218 1209
## 13 2013 1 1 933 904 29 1252 1210
## 14 2013 1 1 953 921 32 1320 1241
## 15 2013 1 1 1011 1001 10 1133 1128
## # ℹ 40,543 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
flights %>% filter(is.na(arr_time))
## # A tibble: 8,713 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 2016 1930 46 NA 2220
## 2 2013 1 1 NA 1630 NA NA 1815
## 3 2013 1 1 NA 1935 NA NA 2240
## 4 2013 1 1 NA 1500 NA NA 1825
## 5 2013 1 1 NA 600 NA NA 901
## 6 2013 1 2 2041 2045 -4 NA 2359
## 7 2013 1 2 2145 2129 16 NA 33
## 8 2013 1 2 NA 1540 NA NA 1747
## 9 2013 1 2 NA 1620 NA NA 1746
## 10 2013 1 2 NA 1355 NA NA 1459
## 11 2013 1 2 NA 1420 NA NA 1644
## 12 2013 1 2 NA 1321 NA NA 1536
## 13 2013 1 2 NA 1545 NA NA 1910
## 14 2013 1 2 NA 1330 NA NA 1640
## 15 2013 1 2 NA 1601 NA NA 1735
## # ℹ 8,698 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
?slice)
flights %>% slice(1234:1258)
## # A tibble: 25 × 19
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 2 1236 1238 -2 1446 1446
## 2 2013 1 2 1236 1240 -4 1403 1405
## 3 2013 1 2 1236 1200 36 1417 1330
## 4 2013 1 2 1239 1200 39 1429 1342
## 5 2013 1 2 1239 1220 19 1328 1320
## 6 2013 1 2 1240 1245 -5 1538 1600
## 7 2013 1 2 1240 1240 0 1556 1540
## 8 2013 1 2 1240 1245 -5 1521 1616
## 9 2013 1 2 1243 1229 14 1453 1428
## 10 2013 1 2 1244 900 224 1431 1104
## 11 2013 1 2 1245 1249 -4 1718 1800
## 12 2013 1 2 1249 1245 4 1555 1550
## 13 2013 1 2 1249 1258 -9 1535 1610
## 14 2013 1 2 1250 1245 5 1349 1350
## 15 2013 1 2 1251 1250 1 1552 1603
## 16 2013 1 2 1253 1300 -7 1505 1508
## 17 2013 1 2 1254 1300 -6 1545 1537
## 18 2013 1 2 1255 1259 -4 1501 1502
## 19 2013 1 2 1256 1252 4 1600 1555
## 20 2013 1 2 1256 1300 -4 1526 1518
## 21 2013 1 2 1256 1225 31 1409 1345
## 22 2013 1 2 1257 1303 -6 1356 1411
## 23 2013 1 2 1257 1300 -3 1529 1527
## 24 2013 1 2 1257 1257 0 1613 1607
## 25 2013 1 2 1258 1255 3 1537 1535
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
?slice_sample()) 3 random flights per day in March
flights %>% filter(month == 3) %>%
group_by(day) %>%
slice_sample(n = 3)
# older: sample_n(3)
## # A tibble: 93 × 19
## # Groups: day [31]
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 3 1 2149 2145 4 55 34
## 2 2013 3 1 1418 1420 -2 1559 1620
## 3 2013 3 1 1930 1929 1 2149 2130
## 4 2013 3 2 1659 1700 -1 2154 2207
## 5 2013 3 2 621 630 -9 931 926
## 6 2013 3 2 NA 1247 NA NA 1516
## 7 2013 3 3 2110 1900 130 49 2256
## 8 2013 3 3 1025 1030 -5 1147 1209
## 9 2013 3 3 1433 1436 -3 1554 1612
## 10 2013 3 4 1656 1700 -4 1919 1924
## 11 2013 3 4 558 600 -2 700 659
## 12 2013 3 4 2022 2000 22 2146 2136
## 13 2013 3 5 1732 1735 -3 2103 2115
## 14 2013 3 5 1508 1445 23 1754 1744
## 15 2013 3 5 2009 1936 33 2134 2100
## # ℹ 78 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
flights %>%
filter(month == 1) %>%
group_by(carrier) %>%
slice_max(dep_delay, n = 5)
# older: top_n(5, dep_delay)
## # A tibble: 77 × 19
## # Groups: carrier [16]
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 25 15 1815 360 208 1958
## 2 2013 1 26 1409 820 349 1528 937
## 3 2013 1 16 2013 1505 308 2136 1637
## 4 2013 1 25 123 2029 294 215 2140
## 5 2013 1 3 2056 1605 291 2239 1754
## 6 2013 1 2 1607 1030 337 2003 1355
## 7 2013 1 1 2205 1720 285 46 2040
## 8 2013 1 17 1130 715 255 1436 1045
## 9 2013 1 17 1012 610 242 1302 910
## 10 2013 1 22 2113 1720 233 23 2040
## 11 2013 1 20 2157 1815 222 46 2130
## 12 2013 1 31 2025 1815 130 2313 2130
## 13 2013 1 30 2006 1815 111 2247 2130
## 14 2013 1 28 1852 1815 37 2211 2130
## 15 2013 1 13 1846 1815 31 2132 2130
## # ℹ 62 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>
flights %>%
mutate(route=paste(origin,"-",dest)) %>%
distinct(route,.keep_all=T) %>%
nrow()
## [1] 224
flights %>%
mutate(route=paste(origin,"-",dest)) %>%
count(route, sort = TRUE)
# JFK - LAX
## # A tibble: 224 × 2
## route n
## <chr> <int>
## 1 JFK - LAX 11262
## 2 LGA - ATL 10263
## 3 LGA - ORD 8857
## 4 JFK - SFO 8204
## 5 LGA - CLT 6168
## 6 EWR - ORD 6100
## 7 JFK - BOS 5898
## 8 LGA - MIA 5781
## 9 JFK - MCO 5464
## 10 EWR - BOS 5327
## 11 EWR - SFO 5127
## 12 LGA - DTW 5040
## 13 EWR - CLT 5026
## 14 EWR - ATL 5022
## 15 EWR - MCO 4941
## # ℹ 209 more rows
air_time is the amount of time in minutes spent in the air. Add a new column air_spd that will contain aircraft’s airspeed in mph
flights %>% mutate(air_spd = distance/(air_time / 60))
## # A tibble: 336,776 × 20
## year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <int> <dbl> <int> <int>
## 1 2013 1 1 517 515 2 830 819
## 2 2013 1 1 533 529 4 850 830
## 3 2013 1 1 542 540 2 923 850
## 4 2013 1 1 544 545 -1 1004 1022
## 5 2013 1 1 554 600 -6 812 837
## 6 2013 1 1 554 558 -4 740 728
## 7 2013 1 1 555 600 -5 913 854
## 8 2013 1 1 557 600 -3 709 723
## 9 2013 1 1 557 600 -3 838 846
## 10 2013 1 1 558 600 -2 753 745
## 11 2013 1 1 558 600 -2 849 851
## 12 2013 1 1 558 600 -2 853 856
## 13 2013 1 1 558 600 -2 924 917
## 14 2013 1 1 558 600 -2 923 937
## 15 2013 1 1 559 600 -1 941 910
## # ℹ 336,761 more rows
## # ℹ 12 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## # tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## # hour <dbl>, minute <dbl>, time_hour <dttm>, air_spd <dbl>
air_spd variable
flights %>% transmute(air_spd = distance/(air_time / 60))
## # A tibble: 336,776 × 1
## air_spd
## <dbl>
## 1 370.
## 2 374.
## 3 408.
## 4 517.
## 5 394.
## 6 288.
## 7 404.
## 8 259.
## 9 405.
## 10 319.
## 11 414.
## 12 382.
## 13 430.
## 14 426.
## 15 324.
## # ℹ 336,761 more rows
group_by(), summarise() and n() to see how many planes were delayed (departure) every month
flights %>%
filter(dep_delay > 0) %>%
group_by(month) %>%
summarise(num_dep_delayed = n())
## # A tibble: 12 × 2
## month num_dep_delayed
## <int> <int>
## 1 1 9662
## 2 2 9124
## 3 3 11209
## 4 4 10543
## 5 5 11291
## 6 6 12655
## 7 7 13909
## 8 8 11713
## 9 9 7815
## 10 10 8722
## 11 11 8239
## 12 12 13550
dep_delay per month?
flights %>%
group_by(month) %>%
summarise(mean_dep_delay = mean(dep_delay, na.rm = T))
## # A tibble: 12 × 2
## month mean_dep_delay
## <int> <dbl>
## 1 1 10.0
## 2 2 10.8
## 3 3 13.2
## 4 4 13.9
## 5 5 13.0
## 6 6 20.8
## 7 7 21.7
## 8 8 12.6
## 9 9 6.72
## 10 10 6.24
## 11 11 5.44
## 12 12 16.6
flights %>%
filter(arr_delay > 0) %>%
group_by(origin) %>%
summarise(cnt = n()) %>%
arrange(desc(cnt))
## # A tibble: 3 × 2
## origin cnt
## <chr> <int>
## 1 EWR 50099
## 2 JFK 42885
## 3 LGA 40020
summarise() to sum total dep_delay per month in hours
flights %>%
group_by(month) %>%
summarize(tot_dep_delay = sum(dep_delay/60, na.rm = T))
## # A tibble: 12 × 2
## month tot_dep_delay
## <int> <dbl>
## 1 1 4430.
## 2 2 4271.
## 3 3 6167.
## 4 4 6426.
## 5 5 6111.
## 6 6 9462.
## 7 7 10315.
## 8 8 6062.
## 9 9 3039.
## 10 10 2982.
## 11 11 2449.
## 12 12 7490.
group_size() on carrier what does it return?
flights %>%
group_by(carrier) %>%
group_size()
## [1] 18460 32729 714 54635 48110 54173 685 3260 342 26397 32 58665
## [13] 20536 5162 12275 601
n_groups() to check the number of unique origin-carrier pairs,
flights %>%
group_by(carrier, origin) %>%
n_groups()
## [1] 35
Note on ungroup Depending on the version of dplyr you may or may need to use the ungroup() if you want to group your data on some other variables. In the newer versions, summarise and mutate drop one aggregation level.
delay_class with case_when(): "early" if dep_delay < 0, "on time" if 0-15, "late" otherwise. Count flights per class.
flights %>%
filter(!is.na(dep_delay)) %>%
mutate(delay_class = case_when(dep_delay < 0 ~ "early",
dep_delay <= 15 ~ "on time",
.default = "late")) %>%
count(delay_class)
## # A tibble: 3 × 2
## delay_class n
## <chr> <int>
## 1 early 183575
## 2 late 70774
## 3 on time 74172
across() to compute the mean of dep_delay and arr_delay per origin in one summarise().
flights %>%
group_by(origin) %>%
summarise(across(ends_with("_delay"), ~ mean(.x, na.rm = TRUE)))
## # A tibble: 3 × 3
## origin dep_delay arr_delay
## <chr> <dbl> <dbl>
## 1 EWR 15.1 9.11
## 2 JFK 12.1 5.55
## 3 LGA 10.3 5.78
The flights table only has a two-letter carrier code. The airlines, planes and airports tables hold the details.
name to flights (left_join). Which airline has the largest mean departure delay?
flights %>%
left_join(airlines, by = "carrier") %>%
group_by(name) %>%
summarise(mean_dep_delay = mean(dep_delay, na.rm = TRUE)) %>%
arrange(desc(mean_dep_delay))
## # A tibble: 16 × 2
## name mean_dep_delay
## <chr> <dbl>
## 1 Frontier Airlines Inc. 20.2
## 2 ExpressJet Airlines Inc. 20.0
## 3 Mesa Airlines Inc. 19.0
## 4 AirTran Airways Corporation 18.7
## 5 Southwest Airlines Co. 17.7
## 6 Endeavor Air Inc. 16.7
## 7 JetBlue Airways 13.0
## 8 Virgin America 12.9
## 9 SkyWest Airlines Inc. 12.6
## 10 United Air Lines Inc. 12.1
## 11 Envoy Air 10.6
## 12 Delta Air Lines Inc. 9.26
## 13 American Airlines Inc. 8.59
## 14 Alaska Airlines Inc. 5.80
## 15 Hawaiian Airlines Inc. 4.90
## 16 US Airways Inc. 3.78
planes table? (hint: anti_join). How many distinct tailnum is that?
flights %>% anti_join(planes, by = "tailnum") %>% distinct(tailnum) %>% nrow()
## [1] 722
airports is called faa, not dest. Join the destination airport names to flights and list the 5 most frequent destinations by name. (hint: by = c("dest" = "faa"))
flights %>%
count(dest) %>%
left_join(airports, by = c("dest" = "faa")) %>%
select(dest, name, n) %>%
slice_max(n, n = 5)
## # A tibble: 5 × 3
## dest name n
## <chr> <chr> <int>
## 1 ORD Chicago Ohare Intl 17283
## 2 ATL Hartsfield Jackson Atlanta Intl 17215
## 3 LAX Los Angeles Intl 16174
## 4 BOS General Edward Lawrence Logan Intl 15508
## 5 MCO Orlando Intl 14082
table1, table2, table3 and table4a/table4b (all in tidyr). Which one is tidy? Why are the others not?
table1
# table1 is tidy: one row per country-year, one column per variable.
# table2: a variable (type) is spread across rows; table3: `rate` holds two values;
# table4a/4b: the year (a value) is in column names.
## # A tibble: 6 × 4
## country year cases population
## <chr> <dbl> <dbl> <dbl>
## 1 Afghanistan 1999 745 19987071
## 2 Afghanistan 2000 2666 20595360
## 3 Brazil 1999 37737 172006362
## 4 Brazil 2000 80488 174504898
## 5 China 1999 212258 1272915272
## 6 China 2000 213766 1280428583
table4a tidy (hint: pivot_longer).
table4a %>% pivot_longer(c(`1999`, `2000`), names_to = "year", values_to = "cases")
## # A tibble: 6 × 3
## country year cases
## <chr> <chr> <dbl>
## 1 Afghanistan 1999 745
## 2 Afghanistan 2000 2666
## 3 Brazil 1999 37737
## 4 Brazil 2000 80488
## 5 China 1999 212258
## 6 China 2000 213766
table2 tidy (hint: pivot_wider).
table2 %>% pivot_wider(names_from = type, values_from = count)
## # A tibble: 6 × 4
## country year cases population
## <chr> <dbl> <dbl> <dbl>
## 1 Afghanistan 1999 745 19987071
## 2 Afghanistan 2000 2666 20595360
## 3 Brazil 1999 37737 172006362
## 4 Brazil 2000 80488 174504898
## 5 China 1999 212258 1272915272
## 6 China 2000 213766 1280428583
rate in table3 into cases and population (hint: separate, convert = TRUE), then unite country and year into one column with unite.
table3 %>%
separate(rate, into = c("cases", "population"), sep = "/", convert = TRUE) %>%
unite("country_year", country, year, sep = "_")
## # A tibble: 6 × 3
## country_year cases population
## <chr> <int> <int>
## 1 Afghanistan_1999 745 19987071
## 2 Afghanistan_2000 2666 20595360
## 3 Brazil_1999 37737 172006362
## 4 Brazil_2000 80488 174504898
## 5 China_1999 212258 1272915272
## 6 China_2000 213766 1280428583
For each origin airport, find the carrier with the highest mean departure delay among carriers with at least 100 flights from that origin. Report the airline’s full name.
flights %>%
group_by(origin, carrier) %>%
summarise(n = n(), mean_dep_delay = mean(dep_delay, na.rm = TRUE), .groups = "drop") %>%
filter(n >= 100) %>%
group_by(origin) %>%
slice_max(mean_dep_delay, n = 1) %>%
left_join(airlines, by = "carrier")
## # A tibble: 3 × 5
## # Groups: origin [3]
## origin carrier n mean_dep_delay name
## <chr> <chr> <int> <dbl> <chr>
## 1 EWR EV 43939 20.2 ExpressJet Airlines Inc.
## 2 JFK 9E 14651 19.0 Endeavor Air Inc.
## 3 LGA F9 685 20.2 Frontier Airlines Inc.
Make sure to load the relevant libraries.
| Operator | Package | Short Explanation | Example |
|---|---|---|---|
%>% |
magrittr / tidyverse | Forward pipe: passes the LHS as the first argument of the RHS function. Improves readability. | mtcars %>% head(3) |
%$% |
magrittr | Exposition pipe: unpacks columns of a data frame so they can be used directly inside a function. | mtcars %$% cor(mpg, hp) |
%T>% |
magrittr | Tee pipe: runs a function for side effects (e.g., plotting) but returns the original data. | mtcars %T>% print() %>% summary() |
\|> |
base R (R 4.1+) | Base pipe: similar to %>% but stricter and simpler. LHS is inserted into _ placeholder if used. |
mtcars \|> head(3) or mtcars \|> (\(x) x[1:3, ])() |
| Placeholder | Package | Used With | Short Explanation | Example |
|---|---|---|---|---|
. |
magrittr | %>% pipes |
Insert LHS anywhere inside the RHS expression. | mtcars %>% filter(mpg > mean(.$mpg)) |
.x |
purrr | map(), walk(), etc. |
Represents the current element in purrr mapping. | map(1:3, ~ .x^2) |
.y |
purrr | imap() |
Represents the index/name of the current element. | imap(letters[1:3], ~ paste(.y, .x)) |
_ |
base R | \|> |
Base-R placeholder: places the LHS at a named argument. | mtcars \|> lm(mpg ~ hp, data = _) |
| Package | Function | What it does (short) | Tiny example |
|---|---|---|---|
| dplyr | filter() |
Keep rows that satisfy one or more logical conditions. | flights %>% filter(dep_delay > 60) |
| dplyr | select() |
Select or reorder columns by name, range, or helper functions. | flights %>% select(year, month, day, dep_delay) |
| dplyr | mutate() |
Add or modify columns while keeping all existing columns. | flights %>% mutate(speed = distance / (air_time / 60)) |
| dplyr | transmute() |
Like mutate(), but drops all other columns, keeps only new ones. |
flights %>% transmute(speed = distance / (air_time / 60)) |
| dplyr | arrange() |
Reorder rows by one or more columns. | flights %>% arrange(dep_delay) |
| dplyr | arrange() + desc() |
Sort rows in descending order of a variable. | flights %>% arrange(desc(dep_delay)) |
| dplyr | group_by() |
Define groups of rows for grouped operations (e.g. summarise). | flights %>% group_by(carrier) |
| dplyr | summarise() / summarize() |
Collapse each group to one row with summary stats. | flights %>% summarise(mean_delay = mean(dep_delay, na.rm = TRUE)) |
| dplyr | distinct() |
Return unique rows based on selected columns. | flights %>% distinct(carrier) |
| dplyr | count() |
Shortcut for group_by(...) %>% summarise(n = n()). |
flights %>% count(carrier) |
| dplyr | slice() |
Select rows by position (row numbers). | flights %>% slice(1:5) |
| dplyr | slice_sample() |
Randomly sample rows (replaces sample_n()). |
flights %>% slice_sample(n = 10) |
| dplyr | slice_max() / slice_min() |
Keep the rows with the largest / smallest values (per group if grouped); replaces top_n(). |
flights %>% slice_max(dep_delay, n = 5) |
| dplyr | rename() |
Rename columns while keeping all data. | flights %>% rename(delay = dep_delay) |
| dplyr | desc() |
Marks a variable for descending order in arrange() or top_n(). |
flights %>% arrange(desc(dep_delay)) |
| dplyr | between() |
Test if values lie between two endpoints (inclusive). | flights %>% filter(between(dep_delay, 10, 30)) |
| dplyr | everything() |
Helper to select all remaining columns after those named. | flights %>% select(year, month, day, everything()) |
| dplyr | contains() |
Select columns whose names contain a given string. | flights %>% select(contains("delay")) |
| dplyr | all_of() |
Select columns using a character vector of column names (replaces one_of()). |
vars <- c("year", "month"); flights %>% select(all_of(vars)) |
| dplyr | relocate() |
Move columns, by default to the front. | flights %>% relocate(carrier) |
| dplyr | case_when() |
Vectorised if/else-if for several conditions. | mutate(cls = case_when(dep_delay < 0 ~ "early", .default = "late")) |
| dplyr | across() |
Apply a function to several columns inside mutate/summarise. |
summarise(across(ends_with("delay"), ~ mean(.x, na.rm = TRUE))) |
| dplyr | .by |
Group for one operation only (dplyr >= 1.1). | flights %>% summarise(m = mean(dep_delay, na.rm = TRUE), .by = origin) |
| dplyr | group_size() |
Returns the size (row count) of each group for a grouped data frame. | flights %>% group_by(carrier) %>% group_size() |
| dplyr | n_groups() |
Number of groups in a grouped data frame. | flights %>% group_by(carrier) %>% n_groups() |
| dplyr | inner_join() |
Join two data frames, keeping rows that match in both tables. | inner_join(df1, df2, by = "id") |
| dplyr | left_join() |
Keep all rows of the left table, add matching columns from the right. | left_join(flights, airlines, by = "carrier") |
| dplyr | anti_join() |
Rows of the left table with no match in the right table. | anti_join(flights, planes, by = "tailnum") |
| dplyr | join_by() |
Specify join keys, also with different names. | left_join(x, y, by = join_by(id == key)) |
| tidyr | gather() |
(Superseded, use pivot_longer()) Convert wide columns into key–value long format. |
flights %>% gather(type, delay, dep_delay, arr_delay) |
| tidyr | pivot_longer() |
Modern replacement for gather(): wide → long. |
flights %>% pivot_longer(c(dep_delay, arr_delay), names_to = "type", values_to = "delay") |
| tidyr | pivot_wider() |
Long → wide, turning values of a column into new columns. | delays %>% pivot_wider(names_from = type, values_from = delay) |
| tidyr | separate() |
Split one column into multiple columns. | df %>% separate(col = "x_y", into = c("x","y"), sep = "_") |
| tidyr | unite() |
Combine multiple columns into a single one. | df %>% unite("x_y", x, y, sep = "_") |
| tidyr | complete() |
Ensure all combinations of given variables exist (fill in missing rows). | df %>% complete(year, month) |
| tibble | as_tibble() |
Convert an existing object (e.g. data.frame) into a tibble. | as_tibble(df) |
| tibble | tibble() |
Create a tibble (modern data frame) from scratch. | tibble(x = 1:3, y = letters[1:3]) |