R Roadmap

What Is R and R vs Python for Data Work Core

R is a language and environment for statistics, data analysis and graphics. It was designed by statisticians, so data frames, missing values, distributions and models are built in rather than added by libraries. Analysts in universities, health, finance and public bodies use it for reports, research and reproducible analysis.

Python and R overlap heavily, and the honest comparison is about fit. R is stronger for statistical modelling, specialised statistical methods, exploratory work, publication-quality graphics and reporting. Python is stronger as a general programming language, for production services and deep learning infrastructure. Many teams use both, and the skills transfer well.

In an interview, avoid taking sides. Say which you would choose for a given task and why, and be ready to work in either. The Python for Data Analysis Roadmap covers the Python side.

Practice. Write down three tasks, such as a survey analysis, a web API and an A/B test report, and say which language you would pick for each.

Installing R and Setting Up Core

R is installed from the Comprehensive R Archive Network. Download the build for your system, and on Windows also install Rtools if you plan to compile packages. On Linux, use your distribution's repository or a current build from the project.

R.version.string
sessionInfo()
getwd()
setwd("~/projects")   # avoid; use projects instead

Keep R reasonably current, since new releases arrive every year and packages target recent versions. Check the version with R.version.string. Rather than setwd, work in an RStudio or Positron project, which sets the working directory to the project folder and keeps paths relative. Since R 4.0 strings stay as characters by default, so old advice about stringsAsFactors no longer applies. Know where your user library is with .libPaths().

Practice. Install R, run `sessionInfo()` and paste the version and library paths into a notes file.

RStudio, Positron and VS Code Core

You can run R in a terminal, but an editor makes the work easier. RStudio is the long-standing IDE, with a source pane, console, environment viewer, plots and help in one window. Positron is a newer data-science IDE from the same company, built on the VS Code base, with an R and Python data explorer. VS Code works well with the R extension and a language server.

Learn the console and script split: try an idea in the console, then save it in a script. Useful keys are run line, run selection and restart session. Turn off "restore workspace on startup", so every session begins clean and your scripts, not hidden objects, define the results.

Create a project for each analysis. Projects make paths and history local, and fit neatly with version control.

Practice. Create a project, write a three-line script, run it by line and by file and restart the session to confirm it still runs.

Your First R Script and Basic Syntax Core

An R script is a text file of expressions run top to bottom. Assignment uses <-, and = works but is conventional only for arguments. Names are case-sensitive, and vectors start counting at one.

# a first analysis
heights <- c(162, 175, 181, 158)
mean(heights)
summary(heights)
cat("Tallest:", max(heights), "\n")
print(heights > 170)

Everything is a function call, including +. Comments start with #. A line is an expression and its value prints at the console automatically, but inside functions and loops you must call print or cat. Use ?mean or help("mean") to read documentation, example(mean) to run its examples and str(x) to inspect any object. Style: snake_case names, spaces around operators and lines under about 80 characters.

Practice. Write a script that stores five numbers, reports their mean, median and range, and prints a sentence with `cat`.

Installing Packages: CRAN, pak and Libraries Core

Packages extend R. CRAN is the main repository, with checked packages, and others come from Bioconductor or GitHub. Install once, then load in each session.

install.packages("dplyr")
library(dplyr)
dplyr::filter(mtcars, mpg > 25)   # use without loading
install.packages("pak")
pak::pkg_install("tidyverse")
pak::pkg_install("user/package")  # from a repository host

pak is a fast modern installer that resolves dependencies in parallel and handles system requirements and GitHub sources. library() loads a package and fails loudly if missing, whereas require() returns FALSE, which hides errors. Use pkg::function() for one-off calls and to avoid name clashes, where the later-loaded package masks the earlier one (the console warns you). Update with update.packages() or pak, and pin versions per project with renv, covered later.

Practice. Install the tidyverse with pak, load dplyr and explain the masking message you see about stats::filter.

Comments and Best Practices Core

Good habits make analysis repeatable and readable.

• Comment on the why, not the what. Section headers with # ---- Load data ---- give you an outline in the editor.
• Use one script or Quarto document per analysis, running from top to bottom in a fresh session.
• Use relative paths via a project, or the here package, never absolute ones.
• Keep raw data read-only and write cleaned data to a separate folder.
• Name objects clearly, avoid df, data and T/F (use TRUE and FALSE).
• Format with the styler package and lint with lintr.
• Set a random seed with set.seed() whenever randomness is involved.

set.seed(42)
sample(1:10, 3)

Check results with small sanity tests, such as row counts before and after a join. Version-control your code, and record the package versions with renv. These practices are what separate a throwaway script from a reproducible piece of analysis.

Practice. Tidy an old script: add section headers, remove absolute paths, style it with styler and set a seed.

Basic Data Types: Numeric, Integer, Character, Logical and Complex Core

R has six atomic types, and four of them cover most work. Numeric (double) is the default for numbers, integer takes an L suffix such as 5L, character is text and logical is TRUE or FALSE. Complex (3+2i) is for specialised maths, and raw holds bytes.

class(3.5); class(3L); class("a"); class(TRUE); class(2+3i)
typeof(1:3)           # "integer"
is.numeric(5L)        # TRUE
.Machine$integer.max
0.1 + 0.2 == 0.3      # FALSE
isTRUE(all.equal(0.1 + 0.2, 0.3))   # TRUE

Special values: NA for missing, NaN for undefined numbers, Inf and NULL for nothing at all. Floating-point comparison needs all.equal or a tolerance. Use class() for the high-level type and typeof() for the storage type, a distinction that interview questions touch.

Practice. Create one value of each type, print class and typeof for each, and explain why 0.1 + 0.2 is not equal to 0.3.

Creating Variables and Operators Core

Assign with <-; values are copied when changed (copy-on-modify), so changing one name never changes another.

x <- 10
y <- x
y <- y + 1       # x is still 10
5 %/% 2; 5 %% 2  # integer division, remainder
2 ^ 3; 2 ** 3
c(1, 2, 3) * 2   # vectorised
x == 10 & y > 5  # element-wise and
x == 10 && y > 5 # single-value and, short-circuits
5 %in% c(1, 5, 9)

Operators work on whole vectors at once. Know the difference between & and && (vectors versus a single condition in if), and between | and ||. Arithmetic recycles shorter vectors, with a warning when lengths do not divide evenly. Custom operators such as %in% are functions with percent signs, and you can write your own, for example ` %+% <- function(a, b) paste(a, b) `.

Practice. Predict then check the results of `c(1,2,3) + c(10,20)` and of `if (c(TRUE, FALSE)) 1` in your R version.

Type Conversion and Casting Types Core

R converts types automatically in some cases, called coercion, and explicitly with as. functions. In a vector, the order is logical, integer, numeric, character: anything mixed becomes the most general type.

c(1, "a", TRUE)         # all character
as.numeric("12.5")      # 12.5
as.integer("7")         # 7
as.numeric("abc")       # NA with a warning
as.character(1:3)
as.logical(c("TRUE", "yes", "T"))   # TRUE NA TRUE
as.Date("2024-03-01")
sum(c(TRUE, FALSE, TRUE))           # 2

Converting a factor straight to numeric returns the internal codes, so use as.numeric(as.character(f)). Conversion failures give NA and a warning rather than an error, so check anyNA() after parsing. Prefer typed readers such as readr::parse_number for messy text columns. Explicit casting after import is a standard cleaning step.

Practice. Convert a character column holding prices such as "£1,200" into numbers and report how many values failed.

Vectors Core

A vector is R's basic structure: an ordered sequence of values of one type. A single number is a vector of length one. Create vectors with c(), sequences with : and seq(), repetition with rep().

v <- c(5, 3, 8, 1)
v[2]            # one-based
v[-1]           # drop first
v[v > 3]        # logical filter
v[c(TRUE, FALSE)]   # recycling
names(v) <- c("a", "b", "c", "d")
v["c"]
sort(v); order(v); which(v > 3); rev(v)
seq(0, 1, by = 0.25); rep(c("x", "y"), times = 2)

Vectorised operations are the idiom: v * 2 instead of a loop. Indexing with logical vectors is the main way to filter, and negative indexes drop elements. Missing values spread: mean(c(1, NA)) is NA until you set na.rm = TRUE. Many bugs trace back to recycling and to forgetting that indexes start at one.

Practice. Without loops, find the values in 1 to 100 divisible by 3 or 5, and their sum.

Matrices and Arrays Recommended

A matrix is a two-dimensional vector of one type with rows and columns. An array extends this to more dimensions.

m <- matrix(1:6, nrow = 2, byrow = TRUE)
dim(m); t(m)
m[1, ]; m[, 2]; m[2, 3]
m %*% t(m)            # matrix product
apply(m, 1, sum)      # row sums
rowSums(m); colMeans(m)
a <- array(1:24, dim = c(2, 3, 4))
solve(matrix(c(2, 0, 0, 2), 2))   # inverse

Matrices underlie linear algebra, distance calculations and many modelling functions. Use * for element-wise and %*% for matrix multiplication, a common slip. Dimension names give readable output with dimnames. A data frame looks like a matrix but allows different types per column, so use a matrix only when everything is numeric or when an algorithm demands it.

Practice. Build a 3 by 3 matrix, compute its transpose and inverse, and verify that the product with the inverse is the identity.

Lists Core

A list holds elements of any type, including other lists. It is how R represents irregular or nested data, such as parsed JSON, model output and function results.

person <- list(name = "Ada", age = 36, langs = c("R", "C"))
person$name
person[["langs"]][2]
person["age"]         # still a list
person$email <- "ada@example.com"
str(person)
lapply(person, class)
unlist(list(a = 1, b = 2))

Single brackets return a list and double brackets or $ extract the element, which is a classic confusion. Use str() to explore nested lists. Lists work with the apply family and with purrr. A data frame is a list of equal-length vectors, which explains why df$col and df[["col"]] work the same way.

Practice. Parse a nested list representing three people and extract all their ages into a numeric vector.

Factors Core

A factor stores categorical data as integer codes with a set of labelled levels. It is used for groups in models, ordered categories and plot ordering.

size <- factor(c("small", "large", "medium", "small"),
               levels = c("small", "medium", "large"), ordered = TRUE)
levels(size); as.integer(size)
table(size)
size < "large"
droplevels(size[size != "large"])
forcats::fct_infreq(size)
forcats::fct_lump_n(size, 2)

Set the levels explicitly to control order in tables and plots. Dropped values leave unused levels, which droplevels() removes. The forcats package offers tools such as fct_reorder, fct_recode, fct_lump and fct_infreq for relabelling and ordering. Never convert a factor with as.numeric directly, because that returns the codes.

Practice. Create an ordered factor for survey answers, count each level and reorder a bar chart by frequency using forcats.

Data Frames and Tibbles Core

A data frame is a table: a list of equal-length columns of any type. It is the standard structure for analysis. A tibble is the tidyverse version with tidier printing, no partial matching and no surprise conversions.

df <- data.frame(id = 1:3, score = c(90, 85, NA),
                 group = c("a", "b", "a"))
df$score; df[df$score > 86, ]; df[["group"]]
nrow(df); ncol(df); names(df); str(df); summary(df)
head(df, 2)
library(tibble)
tb <- tibble(x = 1:3, y = x * 2)   # columns can use earlier ones
as_tibble(df)

Inspect with str, summary, glimpse and head. Subsetting with [rows, cols] and $ is base R, and the dplyr verbs are the modern route. A tibble never changes column names or types silently, and prints only the first rows. Convert with as_tibble() and as.data.frame().

Practice. Create a data frame by hand, add a computed column, filter rows with NA and print it as a tibble.

Data Tables and Working with Data Tables Recommended

data.table is a package for fast, memory-efficient work with large tables. It extends the data frame with a compact DT[i, j, by] syntax: filter rows in i, compute columns in j and group in by. It updates columns by reference with :=, which avoids copies.

library(data.table)
dt <- fread("sales.csv")
dt[region == "North", .(total = sum(amount)), by = month]
dt[, margin := revenue - cost]
setkey(dt, id)
merge(dt, other, by = "id")

fread reads files far faster than the base reader. Because updates occur in place, copying needs copy(dt). It shines with millions of rows, while dplyr is often easier to read. You can have both: dtplyr translates dplyr code to data.table. In interviews, say what each tool is for and choose by data size and team habit.

Practice. Load a large CSV with fread, compute grouped totals and add a derived column by reference.

Conditional Statements Core

if, else if and else test a single condition that must be TRUE or FALSE. For vectors, use ifelse or dplyr::if_else and case_when.

if (score >= 50) {
  result <- "pass"
} else {
  result <- "fail"
}
ifelse(x > 0, "pos", "non-pos")
dplyr::case_when(
  score >= 90 ~ "A",
  score >= 80 ~ "B",
  TRUE        ~ "C"
)
switch(type, mean = mean(x), median = median(x), stop("unknown type"))

if with a length-greater-than-one condition is an error in modern R. A condition containing NA also errors, so check is.na first. case_when evaluates in order and uses the first match; add a final default.

Practice. Write a function that grades a numeric vector with case_when and handles NA explicitly.

Loops Core

R has for, while and repeat loops. Use them when each step depends on the last or when you perform side effects such as writing files. For element-wise operations, prefer vectorised code or the apply family.

total <- 0
for (i in seq_along(x)) {
  if (is.na(x[i])) next
  total <- total + x[i]
}
results <- vector("list", length(files))
for (i in seq_along(files)) results[[i]] <- read.csv(files[i])
i <- 0
while (i < 3) i <- i + 1

Use seq_along(x) rather than 1:length(x), which breaks on empty input. Preallocate result containers rather than growing them with c() inside the loop, since growing copies the vector every pass. break and next control the flow. Loops in R are not inherently slow, but vectorised built-ins are shorter and faster.

Practice. Rewrite a loop that builds a vector by appending into one that preallocates, then into a vectorised expression.

Functions, Arguments and Defaults Core

A function is an object created with function (or \(x) shorthand). The last expression is returned, so return() is only for early exits. Arguments can have defaults, are matched by name or position, and are evaluated lazily.

normalise <- function(x, na.rm = TRUE, ...) {
  stopifnot(is.numeric(x))
  (x - mean(x, na.rm = na.rm)) / sd(x, na.rm = na.rm)
}
normalise(c(1, 5, NA, 9))
sapply(1:3, \(i) i^2)           # lambda shorthand
args(normalise)

The ... argument passes extra arguments on. Validate inputs with stopifnot, stop, warning and message. Functions should return one clearly typed result and avoid side effects. Write a function when you have copied code three times. Document with comments above, or roxygen comments if you build a package.

Practice. Write a function that returns the standardised values of a vector with an option to ignore NA, and test it on bad input.

Scopes Core

Scope is where R looks for a name. Inside a function, R searches the local environment, then the enclosing one and so on up to the global environment and attached packages. This is lexical scoping, and it makes closures possible.

x <- 10
f <- function() {
  x <- 1          # local, does not change global x
  x
}
f(); x             # 1, then 10
make_counter <- function() {
  n <- 0
  function() { n <<- n + 1; n }
}
cnt <- make_counter(); cnt(); cnt()   # 1, 2

The <<- operator assigns in an enclosing environment and should be used sparingly. A function modifies only its local copy of its arguments, since R copies on modify. Reading globals inside functions leads to hard-to-track bugs, so pass values as arguments. environment() and exists() help when debugging name lookup.

Practice. Write a counter factory with closures, and a version that wrongly depends on a global, and explain the difference.

Recursion Optional

A recursive function calls itself on a smaller problem until a base case stops it. R handles modest depths, and the default expression limit (5000) stops runaway calls with an error.

fact <- function(n) if (n <= 1) 1 else n * fact(n - 1)
fib_memo <- local({
  cache <- c(1, 1)
  function(n) {
    if (n <= length(cache)) return(cache[n])
    v <- fib_memo(n - 1) + fib_memo(n - 2)
    cache[n] <<- v
    v
  }
})

Recursion suits tree-shaped data such as nested lists, directory walks and divide-and-conquer algorithms. Because R has no tail-call optimisation, iterate for deep problems, and memoise to avoid repeated work. For everyday analysis, vectorised functions and Reduce replace most recursion.

Practice. Write a recursive function that flattens an arbitrarily nested list and compare it with `unlist`.

Apply Family and Map Functions Core

The apply family replaces many loops. lapply returns a list, sapply simplifies when it can, vapply checks the result type (the safest), mapply iterates over several inputs, apply works over matrix margins and tapply groups.

lapply(1:3, function(i) i * 2)
vapply(mtcars, mean, numeric(1))
mapply(function(a, b) a + b, 1:3, 4:6)
tapply(mtcars$mpg, mtcars$cyl, mean)
library(purrr)
map(1:3, \(i) i * 2)
map_dbl(mtcars, mean)
map2_chr(c("a", "b"), c("x", "y"), paste0)
mtcars |> split(~cyl) |> map(\(d) lm(mpg ~ wt, d)) |> map(coef)

The purrr map functions have a consistent name pattern: map_dbl, map_chr, map_lgl and map_dfr state the output type, so they fail loudly if the type is wrong. Prefer vapply or purrr over sapply, whose return type can change with the input. Use purrr's walk for side effects.

Practice. Fit a regression for each group of a data set with split and map, and collect the slopes in a tibble.

Error Handling Core

R signals problems with errors, warnings and messages. Handle them with tryCatch, which can run code on each condition and a finally block.

result <- tryCatch(
  {
    as.numeric("abc")          # warns
  },
  warning = function(w) { message("bad input: ", conditionMessage(w)); NA },
  error   = function(e) stop("wrapped: ", conditionMessage(e)),
  finally = message("done")
)
safe_read <- purrr::possibly(readr::read_csv, otherwise = NULL)
try(log(-1), silent = TRUE)

stop() raises an error, warning() continues and message() informs. purrr::safely and possibly wrap a function so one failing item does not end a whole map. Write validation at the start of functions with stopifnot. Log failures instead of hiding them, and never use an empty handler that swallows errors.

Practice. Read a list of ten files where two are missing, keep the good ones and report the failures at the end.

Debugging Core

Read the error message and call stack first; traceback() shows the calls that led to the error. Then isolate with a small example. In RStudio, set a breakpoint, or insert browser() to pause and inspect.

f <- function(x) {
  browser()          # opens an interactive prompt here
  x / 0
}
debug(f)             # step through next call
undebug(f)
options(error = recover)
str(x); print(head(x))

At the browser prompt, n runs the next line, c continues, Q quits and any object can be printed. debugonce(f) steps through one call only. Strategies that work: shrink the input, check types with str, check lengths and NA counts and bisect the pipeline by printing intermediate results. The reprex package makes minimal, shareable examples for asking for help.

Practice. Debug a function with an off-by-one error using browser(), without adding print statements.

Importing Data: readr and CSV Core

The readr package reads delimited files quickly, guesses column types from a sample, and returns tibbles. Base read.csv still works but is slower and less predictable.

library(readr)
sales <- read_csv("sales.csv",
  col_types = cols(id = col_integer(), date = col_date("%d/%m/%Y"),
                   amount = col_double(), .default = col_character()),
  na = c("", "NA", "n/a"))
problems(sales)
write_csv(sales, "clean/sales.csv")
read_delim("data.txt", delim = "|")

Always check spec() or problems() after reading, and state col_types for important files, because guessing can misread codes with leading zeros as numbers. Use locale() for decimal marks and encodings, n_max to peek and skip for header notes. Write with write_csv and, for faster binary storage, arrow::write_parquet or saveRDS.

Practice. Import a messy CSV with a European date format and thousands separators, and force correct types.

Excel, JSON and Other Formats Core

Spreadsheets and nested files are common sources.

library(readxl)
excel_sheets("book.xlsx")
read_excel("book.xlsx", sheet = "Q1", range = "B2:F100", skip = 1)
library(jsonlite)
raw <- fromJSON("api.json", flatten = TRUE)
toJSON(head(mtcars, 2), pretty = TRUE)
haven::read_sav("survey.sav")
arrow::read_parquet("big.parquet")

readxl needs no Excel installation; use range or skip to ignore decorative headers, and col_names to set your own. jsonlite::fromJSON turns arrays of records into data frames and nested objects into list columns, which tidyr::unnest flattens. haven reads SPSS, Stata and SAS files. For writing Excel, writexl is simple and openxlsx offers formatting. Prefer text or Parquet for exchange, since Excel silently changes types, dates and gene names.

Practice. Read one workbook sheet with a decorative header and one nested JSON file and flatten both into tidy tibbles.

APIs and Web Scraping Recommended

Use httr2 to call web APIs and rvest to read HTML pages. Respect terms of service, robots files and rate limits.

library(httr2)
resp <- request("https://api.example.com/items") |>
  req_url_query(page = 1) |>
  req_headers(Authorization = paste("Bearer", Sys.getenv("API_TOKEN"))) |>
  req_retry(max_tries = 3) |>
  req_perform()
items <- resp_body_json(resp, simplifyVector = TRUE)
library(rvest)
page <- read_html("https://example.com/table")
page |> html_element("table") |> html_table()

Keep secrets in environment variables (Sys.getenv) or the keyring package, never in scripts. Add delays between requests, cache responses on disk and handle pagination in a loop. Scraping breaks when pages change, so prefer an official API or a downloadable data set when one exists.

Practice. Call a public JSON API with pagination, store the responses and bind them into one tibble.

Data Cleaning and Handling Missing Values Core

Cleaning usually takes more time than modelling. Start by inspecting, then fix types, names, duplicates, impossible values and gaps.

library(dplyr); library(janitor)
df <- df |> clean_names() |> distinct()
colSums(is.na(df))                      # missing per column
df |> filter(!is.na(income))
df |> mutate(income = coalesce(income, median(income, na.rm = TRUE)))
df |> tidyr::fill(region, .direction = "downup")   # forward / backward fill
df |> tidyr::drop_na(age)

In R, missing means NA; there is no isnull, isna pair as in other tools, you use is.na() and anyNA(). Dropping vs. imputing is the main choice. Dropping rows is simple but can bias results; imputing keeps data but invents values, so record what you did. Choose by how much is missing and why: completely random, related to other variables or related to the value itself. Use fill for ordered data such as time series, and flag imputed values in a new column.

Practice. Take a data set with 10 percent missing values, compare dropping and imputing and show how the mean changes.

String Manipulation, stringr and Regular Expressions Core

stringr offers consistent functions that all start with str_ and take the string first, with ICU regular expressions behind them.

library(stringr)
x <- c("  Ada Lovelace ", "alan TURING", NA)
str_trim(x) |> str_to_title()
str_detect(x, "(?i)turing")
str_extract("order-4521-uk", "\\d+")
str_replace_all("a-b_c", "[-_]", " ")
str_split("a,b,c", ",", simplify = TRUE)
str_sub("abcdef", 2, 4)
str_c("id_", 1:3, collapse = ", ")
str_glue("Total: {n}")

Base R equivalents are grepl, sub, gsub, strsplit and regmatches. Strip, replace, split are the three jobs behind most text cleaning: str_trim or str_squish, str_replace, and str_split. Escape backslashes twice in an R string, or use a raw string r"(\d+)". Test regexes on a handful of tricky examples, and use str_view to see matches.

Practice. Clean a column of messy phone numbers with regular expressions into a single international format and flag the unfixable ones.

R Dates & Times: lubridate and Parsing Dates Core

Base R has Date and POSIXct classes; lubridate makes them easier to parse and calculate with.

library(lubridate)
d <- ymd("2024-03-01")
dmy("01/03/2024"); parse_date_time("Mar 1, 2024 10:30", "bdY HM")
year(d); month(d, label = TRUE); wday(d, label = TRUE)
d + months(1); d + days(30)
interval(d, ymd("2025-01-01")) / days(1)
floor_date(now(), "week")
with_tz(now(), "Europe/London")
as.Date("01/03/2024", format = "%d/%m/%Y")

Strings and dates are linked: parsing functions take the order of the parts in their name (ymd, dmy, mdy). Parsing ambiguous formats such as 03/04/2024 needs an explicit order. Store timestamps in UTC and convert for display. difftime and the period and duration classes differ: a month period varies in length, a duration is a fixed number of seconds. Check for NA after parsing, which shows unparseable values.

Practice. Parse a column of dates in two mixed formats, compute the weekday and the number of days from each to the end of the year.

Encoding Categories and Working with Factors in Data Prep Recommended

Models need categories as numbers or sets of indicator columns. In R, a factor is usually enough because modelling functions create dummy variables automatically. For manual control, use recipes or model.matrix.

df$grade <- factor(df$grade, levels = c("low", "mid", "high"))
model.matrix(~ grade, df)            # one-hot with a reference level
library(forcats)
df |> mutate(city = fct_lump_prop(city, 0.05), grade = fct_relevel(grade, "mid"))
library(recipes)
recipe(y ~ ., df) |> step_dummy(all_nominal_predictors()) |> step_zv(all_predictors())

Rare levels cause unstable estimates, so lump them with fct_lump. The first level is the reference in regression, so choose it deliberately. Ordered categories can be encoded as integers only if the spacing is meaningful. Use the same levels for training and new data, or predictions fail. Never encode with information from the test set.

Practice. Encode a categorical predictor with five levels, lump the rare ones and fit a model showing the chosen reference level.

The Tidyverse, Pipe and Tidy Data Core

Data manipulation in R is now mostly done with the tidyverse, a family of packages sharing one design: dplyr for transformation, tidyr for reshaping, ggplot2 for graphics, readr for import, purrr for iteration, stringr, forcats and tibble. Install with install.packages("tidyverse") and load with library(tidyverse).

Tidy data means each variable is a column, each observation is a row and each value is a cell. Most tools work best on tidy data.

The pipe passes the left value as the first argument of the right function. R 4.1 added the native pipe |>, and the older magrittr %>% still works.

mtcars |>
  filter(cyl == 4) |>
  mutate(kpl = mpg * 0.425) |>
  summarise(avg = mean(kpl))

Write pipelines of five to ten steps that read like a recipe, and name intermediate results when a chain grows. The native pipe is slightly faster and needs no package, but %>% offers a . placeholder, which the native pipe writes as _.

Practice. Rewrite a nested function call as a pipe and explain in a sentence why the data are or are not tidy.

dplyr: Filter, Query, Arrange Core

dplyr verbs each do one job and return a data frame. filter selects rows, select picks columns, arrange sorts, distinct removes duplicates and slice_* picks rows by position.

flights |>
  filter(month == 1, dep_delay > 60 | is.na(arr_time)) |>
  select(carrier, origin, ends_with("delay")) |>
  arrange(desc(dep_delay)) |>
  slice_head(n = 10)
df |> filter(between(age, 18, 30), name %in% c("Ada", "Alan"))
df |> filter(str_detect(city, "^Lon"))
df |> select(-id) |> relocate(name)

Comma-separated conditions in filter mean AND. filter drops rows where the condition is NA, a common surprise. Helpers such as starts_with, contains and where(is.numeric) select columns. Because filtering and sorting are query operations, the same code works with databases through dbplyr, which converts it to SQL. See the SQL Roadmap for the equivalent queries.

Practice. From a flights data set, list the ten longest delays for one airline in January, showing only four columns.

Mutate & Transform with Across Core

mutate adds or changes columns while keeping every row. transmute keeps only the new ones, and across applies a function to many columns.

df |>
  mutate(
    bmi = weight / height^2,
    band = case_when(bmi < 18.5 ~ "under", bmi < 25 ~ "healthy", TRUE ~ "over"),
    across(where(is.character), str_to_lower),
    across(c(score_a, score_b), \(x) x / max(x, na.rm = TRUE), .names = "{.col}_rel")
  )
df |> rename(height_m = height) |> mutate(id = row_number())

Columns created earlier in the same mutate can be used by later ones. Use case_when for multiple branches, if_else for two (strictly typed) and coalesce for defaults. across replaces the old mutate_at family. Write column names as plain symbols in a pipeline, and pass names in variables with .data[[var]] or the curly-curly operator {{ var }} inside your own functions.

Practice. Add a BMI column, a category column and rescale three numeric columns in one mutate call.

Group By & Summarize, and Window Functions Core

group_by splits data and summarise collapses each group to one row. Since dplyr 1.1 you can also group inside a single verb with .by.

sales |>
  group_by(region) |>
  summarise(n = n(), revenue = sum(amount), avg = mean(amount, na.rm = TRUE),
            .groups = "drop")
sales |> summarise(top = max(amount), .by = product)
sales |>
  group_by(region) |>
  mutate(share = amount / sum(amount),
         rank = min_rank(desc(amount)),
         running = cumsum(amount),
         prev = lag(amount)) |>
  ungroup()
sales |> count(region, sort = TRUE)

A window function returns a value for every row but uses the group or ordering around it: lag, lead, cumsum, row_number, min_rank and ntile. Always ungroup() when you are finished, since leftover grouping alters later calculations. Compare with SQL window functions with OVER (PARTITION BY ...); the ideas are the same.

Practice. Compute revenue per region, each sale's share within its region and a running total ordered by date.

Joins Core

Joins combine tables on key columns. left_join keeps all rows on the left, inner_join keeps matches only, full_join keeps everything and anti_join finds rows with no match. semi_join filters rows by existence in another table.

orders |>
  left_join(customers, by = join_by(customer_id == id)) |>
  anti_join(refunds, by = "order_id")
bind_rows(a, b); bind_cols(a, b)
orders |> left_join(items, by = "order_id", relationship = "one-to-many")

Check the key before joining: duplicates in the right table multiply rows. Recent dplyr warns about many-to-many matches, and the relationship argument asserts the shape you expect. Compare nrow before and after, and use anti_join to find mismatches. Joins on different key names use join_by(a == b), and inequality and rolling joins are possible too. Key types must match, so convert character and integer ids first.

Practice. Join orders to customers, find orders with no matching customer and explain the row count before and after.

Reshaping with tidyr Core

pivot_longer turns columns into rows (wide to long), and pivot_wider does the reverse. Long format suits plotting and grouping; wide suits reports and some models.

library(tidyr)
scores |>
  pivot_longer(cols = starts_with("q"), names_to = "quarter", values_to = "sales")
long |>
  pivot_wider(names_from = quarter, values_from = sales, values_fill = 0)
df |> separate_wider_delim(name, " ", names = c("first", "last"))
df |> unite("full", first, last, sep = " ")
df |> unnest_longer(tags)
df |> complete(month, region, fill = list(sales = 0))
df |> nest(.by = region)

separate_* splits one column into several, unite joins them and unnest expands list columns, such as parsed JSON. complete adds missing combinations. Before pivoting wider, make sure each id and name pair is unique, or you get list columns. Reshaping is a core skill because data rarely arrives in the shape the next step needs.

Practice. Reshape a wide table of monthly sales into long form, summarise it and pivot it back to wide with zeros for gaps.

Exploratory Data Analysis and Summary Statistics Core

Exploratory data analysis is visual inspection and summary before modelling: sizes, types, missing values, distributions, outliers and relationships. Start wide and then narrow.

glimpse(df)
summary(df)
skimr::skim(df)
df |> count(category, sort = TRUE)
df |> summarise(across(where(is.numeric),
       list(mean = mean, sd = sd, med = median), na.rm = TRUE))
quantile(df$income, c(.1, .5, .9), na.rm = TRUE)

Summary statistics cover centre (mean, median, mode), spread (range, variance, standard deviation, interquartile range) and shape (skew). A mean far from the median signals skew or outliers. Pair numbers with plots, since very different data can share the same summary. Write down every surprise you find, such as impossible ages and duplicate ids, because it drives the cleaning plan.

Practice. Run a complete first-pass exploration of a new data set and write five observations that affect your next steps.

Descriptive Statistics and Distribution Analysis Core

Descriptive statistics summarise a sample without inferring beyond it. Distribution analysis asks what the values look like: symmetric or skewed, one peak or several, thin or heavy tails.

mean(x); median(x); var(x); sd(x); range(x)
table(x); prop.table(table(x))
hist(x, breaks = 30)
ggplot(df, aes(x)) + geom_histogram(bins = 30) + geom_density()
qqnorm(x); qqline(x)
shapiro.test(x)       # small samples; interpret with care
moments::skewness(x)

Histograms, density plots, box plots and Q-Q plots show shape. A Q-Q plot compares your quantiles with a normal one. Normality tests are overly sensitive for large samples and weak for small ones, so judge by plots first. Log transforms often tame right-skewed positive data such as income. Report the median and interquartile range for skewed data.

Practice. Plot a right-skewed variable, test normality, apply a log transform and compare the plots before and after.

Outliers: IQR and Z-score Core

An outlier is a value far from the rest. It might be an error, a rare real event or the most interesting row, so investigate before removing.

The IQR rule flags values beyond 1.5 times the interquartile range outside the quartiles, which is what box plot whiskers show. The Z-score measures distance from the mean in standard deviations, with 3 or more a common flag.

q <- quantile(x, c(.25, .75), na.rm = TRUE); iqr <- diff(q)
outlier_iqr <- x < q[1] - 1.5 * iqr | x > q[2] + 1.5 * iqr
z <- as.numeric(scale(x))
outlier_z <- abs(z) > 3
boxplot.stats(x)$out

Z-scores assume roughly normal data and are themselves distorted by the outliers, so the IQR is sturdier for skewed data. Options: keep and use outlier-resistant methods, cap values (winsorise), transform or remove with a documented reason.

Practice. Flag outliers in a numeric column using both methods, compare the lists and decide what to do with each one.

Correlation and Relationship Analysis Core

Correlation measures how two variables move together, from -1 to 1. Pearson measures linear association, and Spearman uses ranks, which suits monotone, non-linear or ordinal data. Correlation is not causation.

cor(df$height, df$weight, use = "complete.obs")
cor(df$x, df$y, method = "spearman")
cor(select(df, where(is.numeric)), use = "pairwise.complete.obs")
cor.test(df$x, df$y)
ggplot(df, aes(x, y)) + geom_point() + geom_smooth(method = "lm")
chisq.test(table(df$a, df$b))      # two categorical variables

Relationship analysis picks the tool by variable types: numeric with numeric uses scatter plots and correlation; numeric by category uses box plots and group means; category by category uses cross tables and a chi-squared test. Always plot first, because Anscombe's quartet shows four very different data sets with one correlation. Look for confounders, and beware of restricted ranges and outliers.

Practice. Build a correlation matrix for a data set, plot the strongest pair and explain one plausible confounder.

Probability Distributions and Sampling Core

R names distribution functions with four prefixes: d (density), p (cumulative probability), q (quantile) and r (random draws), followed by the distribution: norm, binom, pois, unif, t, chisq, exp.

dnorm(0); pnorm(1.96); qnorm(0.975)
dbinom(3, size = 10, prob = 0.5)
ppois(2, lambda = 4)
set.seed(1); x <- rnorm(1000, mean = 50, sd = 10)
sample(1:100, 5); sample(c("A", "B"), 10, replace = TRUE)
dplyr::slice_sample(df, n = 100)
replicate(1000, mean(sample(x, 30)))   # sampling distribution

Sampling draws part of a population. Simple random, stratified and cluster sampling suit different cases. The central limit theorem says sample means become roughly normal as size grows, which underlies confidence intervals. Always set a seed for reproducible simulations, and use bootstrap resampling to estimate uncertainty when formulas are awkward.

Practice. Simulate 5000 sample means from a skewed distribution and show with a histogram that they are close to normal.

Hypothesis Testing and rstatix Core

Statistical analysis often begins with a hypothesis test, which asks whether data are consistent with a null hypothesis, such as "no difference in means". The p-value is the probability of data at least this extreme if the null is true; it is not the probability the null is true. State the hypothesis, alpha and effect size before testing.

t.test(score ~ group, data = df)             # two groups
wilcox.test(score ~ group, data = df)        # rank-based alternative
aov(score ~ group, data = df) |> summary()
chisq.test(table(df$smoker, df$disease))
library(rstatix)
df |> group_by(site) |> t_test(score ~ group) |> adjust_pvalue(method = "BH")
df |> cohens_d(score ~ group)

rstatix returns tidy tibbles from common tests and pipes well. Check assumptions such as independence, equal variance and rough normality; use non-parametric tests when they fail. Adjust for multiple comparisons when running many tests. Report effect sizes and confidence intervals, not just p-values. A very large sample can make trivial differences significant.

Practice. Compare two groups with a t-test and a Wilcoxon test, report effect size and write a one-paragraph plain-language conclusion.

Linear and Logistic Regression with stats Core

The stats package, loaded by default, includes lm for linear regression and glm for generalised models, such as logistic regression for a yes-or-no outcome.

fit <- lm(price ~ area + bedrooms + factor(city), data = homes)
summary(fit)
confint(fit)
plot(fit)                        # residual diagnostics
predict(fit, newdata = new_homes, interval = "prediction")
logit <- glm(churn ~ tenure + plan, data = cust, family = binomial)
exp(coef(logit))                 # odds ratios
broom::tidy(fit); broom::glance(fit)

Coefficients show the average change in the outcome for a one-unit change in a predictor, holding the others fixed. Check residuals for pattern, non-constant variance and influential points. For logistic regression, coefficients are log-odds, so exponentiate for odds ratios and judge a classifier with a confusion matrix, ROC and AUC. Watch for multicollinearity and overfitting, and evaluate on held-out data.

Practice. Fit a linear model on a housing data set, check the residual plots and write a sentence interpreting two coefficients.

Base R Plotting Recommended

Base graphics draw on a canvas one layer at a time. They are quick for exploration and need no packages.

hist(mtcars$mpg, main = "Fuel economy", xlab = "mpg", col = "grey80")
plot(mtcars$wt, mtcars$mpg, pch = 19, col = "steelblue")
abline(lm(mpg ~ wt, mtcars), lwd = 2)
boxplot(mpg ~ cyl, data = mtcars)
barplot(table(mtcars$cyl))
par(mfrow = c(1, 2))      # two panels
legend("topright", legend = c("a", "b"), pch = 19)

Functions such as plot, hist, boxplot and barplot draw a new chart; lines, points, abline, text and legend add to it. par() sets global options and layouts. Save with png() or pdf() followed by dev.off(). Base plots are fine for diagnostics such as plot(model). For anything you share, ggplot2 gives more control and consistency.

Practice. Draw a scatter plot with a fitted line and a histogram side by side in base R and save it to a PNG file.

Grammar of Graphics and ggplot2 Core

ggplot2 builds charts from the grammar of graphics: data, aesthetic mappings (which variable controls which visual property), geometric objects, scales, coordinates, facets and themes, added with +.

library(ggplot2)
ggplot(mpg, aes(x = displ, y = hwy, colour = class)) +
  geom_point(alpha = 0.7) +
  geom_smooth(aes(group = 1), method = "lm", se = FALSE) +
  scale_colour_brewer(palette = "Dark2") +
  labs(title = "Fuel economy falls with engine size",
       x = "Engine size (litres)", y = "Highway mpg", colour = "Class") +
  theme_minimal()

Think in layers: what is the data, what is mapped, what shape draws it. Mapping inside aes() links a variable to a property, and setting outside aes() applies a constant. A chart is an object you can store, extend and save with ggsave("plot.png", width = 7, height = 4, dpi = 300). Write a clear title that states the finding.

Practice. Recreate a chart from a news article in ggplot2 using the same data, with a clear title and labelled axes.

Aesthetic Mappings Core

Aesthetics are the visual channels: x, y, colour, fill, size, shape, alpha, linetype and group. A variable mapped to one of these gets an automatic scale and legend.

ggplot(penguins, aes(bill_length_mm, body_mass_g)) +
  geom_point(aes(colour = species, shape = sex), size = 2) +
  scale_colour_manual(values = c(Adelie = "#1b9e77", Chinstrap = "#d95f02", Gentoo = "#7570b3")) +
  scale_y_continuous(labels = scales::comma)

colour is the outline or point and fill is the interior of areas such as bars. Map categories to colour or shape and numbers to position, size or a continuous colour scale. Pick a colour-blind-safe palette (viridis or Brewer) and avoid encoding too many variables at once. Use scale_* functions to change breaks, limits, labels and palettes, and coord_cartesian to zoom without dropping data.

Practice. Make one scatter plot that encodes four variables, then simplify it to two and say which version communicates better.

Common Plots Core

Choose the chart from the question. Distribution of one variable: histogram, density, box or violin. Comparison between categories: bar or dot plot. Relationship between numbers: scatter plot with a smooth line. Change over time: line chart. Part of a whole: stacked bar, rarely a pie.

ggplot(df, aes(score)) + geom_histogram(bins = 30)
ggplot(df, aes(group, score)) + geom_boxplot() + geom_jitter(width = .1, alpha = .3)
ggplot(df, aes(fct_infreq(category))) + geom_bar() + coord_flip()
ggplot(ts, aes(date, value)) + geom_line()
ggplot(sum_df, aes(group, mean, ymin = lo, ymax = hi)) + geom_pointrange()
ggplot(grid, aes(x, y, fill = z)) + geom_tile()

Start bars at zero. Order categories by value, not alphabet. Show uncertainty with intervals when you summarise. Label directly where possible, and remove chart junk such as heavy gridlines and 3D effects.

Practice. Show the same data as a bar chart, a dot plot and a boxplot, and write which suits an audience of managers.

Faceting and Themes Core

Faceting splits a plot into small multiples by a variable, which is often clearer than many colours on one panel.

ggplot(mpg, aes(displ, hwy)) + geom_point() +
  facet_wrap(~ class, scales = "free_y", ncol = 4)
ggplot(mpg, aes(displ, hwy)) + geom_point() + facet_grid(drv ~ cyl)
theme_set(theme_bw(base_size = 12))
my_theme <- theme_minimal() +
  theme(legend.position = "bottom", plot.title = element_text(face = "bold"),
        panel.grid.minor = element_blank())

facet_wrap wraps one variable across panels and facet_grid crosses two. Fixed scales help comparison and free scales help reading detail. Themes control non-data elements: fonts, grids, legends and backgrounds. Build one team theme and apply it to every chart, so a report looks consistent. Packages such as ggthemes and patchwork extend themes and combine plots.

Practice. Facet a chart by two variables and create a reusable theme that you apply to three different charts.

Overplotting Techniques Recommended

When thousands of points overlap, patterns disappear. Fixes include transparency, smaller points, jitter, binning and summarising.

ggplot(df, aes(x, y)) + geom_point(alpha = 0.05, size = 0.5)
ggplot(df, aes(x, y)) + geom_hex(bins = 40)
ggplot(df, aes(x, y)) + geom_density_2d_filled()
ggplot(df, aes(x, y)) + geom_bin2d()
ggplot(df, aes(grp, y)) + geom_jitter(width = 0.2, height = 0)
ggplot(df, aes(x, y)) + geom_point() + facet_wrap(~ grp)

Alpha blending shows density through darkness. Hexagonal or rectangular binning counts points per cell and scales to millions of rows. Jitter helps discrete variables, but add only as much as needed and say so. Another route is to sample or to aggregate before plotting. Pick the technique that keeps the message and mention it in a caption.

Practice. Plot one million random points, then fix the overplotting in three different ways and compare them.

Interactive Plots with plotly and Leaflet Recommended

Interactive charts let readers hover, zoom and filter. plotly turns a ggplot into an interactive widget with one call and also builds charts directly.

library(plotly)
p <- ggplot(mpg, aes(displ, hwy, colour = class, text = model)) + geom_point()
ggplotly(p, tooltip = c("text", "x", "y"))
plot_ly(mpg, x = ~displ, y = ~hwy, color = ~class, type = "scatter", mode = "markers")
library(leaflet)
leaflet(cities) |> addTiles() |> addCircleMarkers(~lon, ~lat, popup = ~name)

Widgets work in Quarto and R Markdown HTML output and Shiny apps, but not in PDF or Word, where you need a static image. Interactivity adds value for dense data, maps and dashboards, and adds clutter for simple messages. Provide a static summary plus the interactive one when accessibility matters, with alt text.

Practice. Convert a static ggplot into an interactive version with custom tooltips and embed it in an HTML report.

R Markdown and Quarto Core

A literate document mixes prose, code and results, and re-running it rebuilds every table and chart. R Markdown is the original format and Quarto is its successor, working with R, Python and Julia, and rendering to HTML, PDF, Word, slides and websites.

---
title: "Quarterly sales"
format: html
execute:
  echo: false
---

```{r}
sales |> count(region) |> knitr::kable()
```

Code chunks take options such as echo, eval, warning, fig-width and cache. Inline code, written with a backtick and r, puts values into sentences so that numbers never go stale. Parameters allow one template to generate many reports, for example one per region. Render with quarto render report.qmd. Keep heavy computation outside the document, and run it from a fresh session.

Practice. Create a report with a parameter for region, render it for three regions and confirm the numbers in the text update.

Shiny and Dashboards Recommended

Shiny builds web apps in R without writing JavaScript. An app has a user interface and a server function. Inputs trigger reactive expressions, and outputs update automatically.

library(shiny)
ui <- fluidPage(
  sliderInput("n", "Sample size", 10, 500, 100),
  plotOutput("hist")
)
server <- function(input, output, session) {
  data <- reactive(rnorm(input$n))
  output$hist <- renderPlot(hist(data()))
}
shinyApp(ui, server)

The core idea is reactivity: reactive() caches a computed value, observeEvent() runs side effects and eventReactive() waits for a button. Split large apps into modules, validate inputs and keep slow work out of the reactive path. flexdashboard or Quarto dashboards give simple layouts without a server, and bslib improves styling. Deploy to a Shiny server, a container or a hosted service. For large deployments, test with shinytest2.

Practice. Build a Shiny app with a dropdown and a slider that filters a data set and updates a chart and a table.

Power BI / Tableau with R Optional

Business intelligence tools are often the final stop for analysis in organisations. Power BI can run R scripts as data sources and visuals, and Tableau connects to R through an external service. Both let analysts without code explore results.

A common workflow: R cleans and models data, writes a tidy table to a database or a file, and the BI tool visualises it with refresh schedules, row-level security and sharing built in. Choose R for repeatable analysis and statistical methods, and BI tools for broad, self-service dashboards. Knowing where each fits is useful in interviews, since many roles combine them. The Power BI Roadmap covers the dashboard side in depth.

Practice. Export a modelled data set from R as a tidy table and build a one-page dashboard from it in a BI tool.

Reproducibility with renv and Positron Workflows Core

A reproducible analysis gives the same answer on another machine and next year. Three things change: your data, your code and your package versions.

renv records and restores a project's package versions.

renv::init()          # create a project library and lockfile
renv::snapshot()      # record current versions
renv::restore()       # reinstall them elsewhere
renv::status()

Commit renv.lock with your code. Combine it with Git, relative paths, a fixed seed and a scripted pipeline: the targets package builds steps in order and reruns only what changed. For full isolation use a Docker image pinned to an R version. In Positron or RStudio, use projects so each analysis is self-contained and set the editor to start a clean session. Document how to run the whole thing in a README.

Practice. Initialise renv for a project, snapshot, delete the library and restore it successfully from the lockfile.

Machine Learning with tidymodels Core

tidymodels is a consistent set of packages for modelling: rsample for splits and resampling, recipes for preprocessing, parsnip for model specifications, workflows to bundle them, tune for hyperparameters and yardstick for metrics.

library(tidymodels)
split <- initial_split(churn, strata = left, prop = 0.8)
rec <- recipe(left ~ ., data = training(split)) |>
  step_impute_median(all_numeric_predictors()) |>
  step_dummy(all_nominal_predictors()) |> step_normalize(all_numeric_predictors())
rf <- rand_forest(trees = 500, mtry = tune()) |> set_engine("ranger") |> set_mode("classification")
wf <- workflow(rec, rf)
res <- tune_grid(wf, resamples = vfold_cv(training(split), v = 5), grid = 10)
final <- finalize_workflow(wf, select_best(res, metric = "roc_auc")) |> last_fit(split)
collect_metrics(final)

Split before preprocessing, so no information leaks from the test set. Use cross-validation to tune, and the test set once, for the final estimate. Pick metrics that fit the cost of errors, such as precision, recall and ROC AUC for imbalanced classes. Start with a simple baseline.

Practice. Build a classification workflow with a recipe and cross-validation, and compare it with a plain logistic regression baseline.

Time Series Analysis Recommended

Time series data are ordered in time, so observations are not independent. Look for trend, seasonality and noise before modelling.

library(tsibble); library(feasts); library(fable)
ts_df <- sales |> as_tsibble(index = month)
ts_df |> autoplot(revenue)
ts_df |> model(STL(revenue)) |> components() |> autoplot()
fit <- ts_df |> model(arima = ARIMA(revenue), ets = ETS(revenue))
fit |> forecast(h = "12 months") |> autoplot(ts_df)
accuracy(fit)

The tidyverts family (tsibble, feasts, fable) fits into the tidy workflow, and the older forecast package is still common. Split by time, never at random, and evaluate on the latest period. Check residuals for leftover pattern, and use a naive or seasonal-naive forecast as a baseline. Handle gaps and time zones before anything else.

Practice. Forecast twelve months of a seasonal series with two models and compare their errors on a hold-out period.

Text and NLP: tidytext and quanteda Optional

tidytext treats text as tidy data with one token per row, so dplyr and ggplot2 work on it. quanteda builds document-feature matrices for larger corpora and has its own tools for dictionaries and models.

library(tidytext)
reviews |>
  unnest_tokens(word, text) |>
  anti_join(stop_words, by = "word") |>
  count(word, sort = TRUE)
reviews |> unnest_tokens(word, text) |> inner_join(get_sentiments("bing"))
library(quanteda)
dfm(tokens(corp, remove_punct = TRUE)) |> dfm_tfidf()

Typical tasks are word counts, TF-IDF (which words distinguish documents), sentiment with a lexicon and topic modelling. Clean text first by lowercasing, removing stop words and handling encoding. Lexicon methods are quick but miss sarcasm and context. For modern language tasks, call a language model through an API or use embedding packages, and treat lexicons as a baseline.

Practice. Count the most distinctive words in two sets of reviews using TF-IDF and chart them.

Network Analysis: igraph and tidygraph Optional

A network has nodes and edges, as in social links, transport routes or dependencies. igraph is the main engine and tidygraph wraps it in a tidy interface, with ggraph for plotting.

library(tidygraph); library(ggraph)
g <- tbl_graph(nodes = people, edges = links, directed = FALSE)
g <- g |> activate(nodes) |>
  mutate(degree = centrality_degree(), community = group_louvain())
ggraph(g, layout = "fr") + geom_edge_link(alpha = .3) +
  geom_node_point(aes(size = degree, colour = factor(community)))
igraph::shortest_paths(g, from = 1, to = 5)

Key measures: degree (number of links), betweenness (control over paths), closeness and clustering. Community detection finds tightly linked groups. Edge lists and node tables are the usual input. Layouts only suggest structure, so do not read meaning into positions in a hairball plot, and filter or aggregate large networks first.

Practice. Build a network from an edge list, find the most central node by two measures and plot the communities.

Geospatial Analysis: sf, terra, leaflet and tmap Optional

Spatial data pairs shapes with attributes. sf handles vector data (points, lines, polygons) as data frames with a geometry column, and terra handles raster grids such as elevation and satellite images. leaflet and tmap draw maps.

library(sf); library(tmap)
regions <- st_read("regions.gpkg")
st_crs(regions)                           # coordinate reference system
pts <- st_as_sf(stores, coords = c("lon", "lat"), crs = 4326)
st_join(pts, regions)                     # which region is each store in
regions |> st_transform(27700) |> st_area()
tm_shape(regions) + tm_polygons("population")
library(terra); r <- rast("elevation.tif"); global(r, "mean", na.rm = TRUE)

The coordinate reference system matters most: transform to a suitable projected system before measuring distances or areas, and use the same one for joined layers. Spatial joins and buffers answer questions such as nearby or inside. Always check the CRS first when shapes do not line up.

Practice. Place a set of stores on a map, work out which region each lies in and shade regions by store count.

Big Data Tools: Sparklyr and Arrow Optional

When data no longer fits in memory, move the work to where the data lives. arrow reads Parquet and other columnar files lazily and runs dplyr verbs out of memory; duckdb runs SQL-fast analysis on local files; sparklyr connects dplyr to an Apache Spark cluster.

library(arrow); library(dplyr)
open_dataset("events/") |>
  filter(year == 2024) |>
  group_by(country) |> summarise(n = n()) |> collect()
library(sparklyr)
sc <- spark_connect(master = "local")
spark_read_parquet(sc, "events", "events/") |> count(country) |> collect()

Work lazily: filter and aggregate before collect(), which pulls the result into R memory. Choose columnar formats such as Parquet over CSV. For a few gigabytes, data.table, arrow or duckdb on one machine usually beats a cluster; Spark is for data too big for one node. Mention the trade-off in interviews.

Practice. Convert a large CSV to partitioned Parquet and compare aggregation time with the original file.

Parallel Computing Optional

R runs one process by default. For independent tasks, run them in parallel on several cores using the parallel, future and furrr packages.

library(future); library(furrr)
plan(multisession, workers = 4)
results <- future_map(1:100, \(i) slow_simulation(i), .options = furrr_options(seed = TRUE))
parallel::detectCores()
parallel::mclapply(1:10, slow_fun, mc.cores = 4)    # not on Windows

future separates what you compute from how it runs, so changing plan() moves the same code between sequential, multicore and cluster. Set seeds properly for random numbers, send only the data each worker needs and expect overhead, so parallelism pays only when each task takes a while. Profile first with system.time or profvis, since vectorising often gives more than adding cores.

Practice. Run a 200-iteration simulation sequentially and with four workers, and report the speed-up.