R for Data Scientists: Audit a Model and Its Uncertainty

An audit is stronger when it does not call the function that produced the reported metric. The Python stage exports row-level observations and predictions; this guide uses base R to reconstruct the errors, compare the model with its baseline and resample complete days. The audit can therefore detect a changed schema, row count, metric definition or numerical result without importing scikit-learn.

What enters the audit

The Python guide writes artifacts/predictions.csv. Each row contains:

• the immutable source identifier;
• observation timestamp and date;
• validation or test partition label;
• observed hourly rentals;
• the matching calendar-baseline prediction; and
• the selected ridge-pipeline prediction.

The R script checks the exact column order, rejects missing values, confirms that only validation and test records are present, and requires 2,168 test rows. It does not load the fitted Python model. This keeps the audit focused on the claims made from predictions rather than repeating training.

The 2011–2012 observations remain a historical teaching benchmark. The audit does not turn them into current transport evidence or a causal study.

Run the exact audit

Install R 4.6.1 using the official R installation guidance. The audit uses base R only; no CRAN package installation is required.

Run the upstream stages and audit from the repository root:

examples/programming-for-data-science/.venv/bin/python \
  examples/programming-for-data-science/sql/build_features.py
examples/programming-for-data-science/.venv/bin/python \
  examples/programming-for-data-science/python/train.py
Rscript examples/programming-for-data-science/r/audit.R

The script stops if the interpreter is not exactly R 4.6.1. That strict check is appropriate for a publication baseline; exploratory use may support a broader version range, but it cannot be recorded as this reproduction without comparison.

Recalculate before interpreting

For the test period, define the row-level signed errors:

e_model,i = prediction_model,i − actual_i

e_baseline,i = prediction_baseline,i − actual_i

R reconstructs MAE, RMSE and mean signed model error directly:

baseline_error <- test$baseline_prediction - test$actual
model_error <- test$model_prediction - test$actual

baseline_mae <- mean(abs(baseline_error))
model_mae <- mean(abs(model_error))
model_rmse <- sqrt(mean(model_error^2))
model_mean_error <- mean(model_error)

The golden checkpoints are:

Metric	Expected value
Baseline MAE	79.049
Model MAE	49.894
Model RMSE	78.457
Model mean signed error	−0.927

R compares its full-precision values with the stored checkpoints. A disagreement larger than 1e-5 rentals fails the audit; the table is rounded to three decimals. This narrow allowance absorbs negligible floating-point differences between equivalent linear-algebra builds. Changing the tolerance or expected values requires an explanation of whether the difference came from data, dependencies, preprocessing, model selection or numerical execution.

Preserve the paired comparison

Both predictors are evaluated against the same actual value for each hour. The comparison should retain that pairing:

d_i = |e_model,i| − |e_baseline,i|

The point estimate is the mean of d_i. It is approximately −29.16 rentals per recorded hour, matching the difference between the two MAEs. A negative value favours the ridge pipeline.

Calculating separate uncertainty intervals for each MAE would discard the covariance between methods. The paired difference is more direct because the decision question is whether the candidate’s absolute error is lower on the same observations.

Why the bootstrap resamples days

Hourly errors within one day can share commuting patterns, weather and operational conditions. Treating all 2,168 hours as independent would ignore that clustering and can produce an interval that is too narrow.

The audit groups the paired differences into 92 dates. With seed 20260815, it draws 92 dates with replacement, retains every hourly error from each selected date, calculates the mean paired difference and repeats the process 2,000 times:

day_values <- split(paired_difference, test$observation_date)
day_names <- names(day_values)

set.seed(20260815)
bootstrap_differences <- replicate(2000L, {
  sampled_days <- sample(day_names, length(day_names), replace = TRUE)
  mean(unlist(day_values[sampled_days], use.names = FALSE))
})

interval <- quantile(bootstrap_differences, c(0.025, 0.975), type = 7)

The executable gate requires a finite interval, the point estimate to lie inside it, and the upper bound to remain below zero. Those checks establish that the model’s lower absolute error is stable across resampled days in this holdout. They do not show that the predictors caused demand, that the model will transfer to another period, or that the interval accounts for model-selection uncertainty and data drift.

The block definition is a modelling choice. A week-block bootstrap could better preserve weekly dependence but would provide only about thirteen blocks in this quarter. A moving-block time-series procedure or rolling-origin evaluation may be preferable for a fuller study. The day block is a transparent compromise for this first reproducible audit.

Diagnose where the average comes from

An overall MAE can hide operationally important failures. The script writes r_diagnostics.csv with baseline MAE, model MAE and model mean signed error for:

• every hour from 0 to 23; and
• each test month from October to December 2012.

It asserts that all 24 hourly groups and all three monthly groups are present. Review these rows for peak-period underprediction, deterioration late in the quarter, small groups and cases where the baseline wins. A favourable overall interval does not excuse a systematic failure during the period that controls fleet or staffing decisions.

Further audits could stratify by working-day status and weather, but subgroup claims need adequate observations and should be declared before extensive searching. Repeatedly examining small groups and reporting only the most extreme one creates a different selection problem.

Preserve machine-readable evidence

The audit writes two files:

• r_audit.csv with the four reconstructed metrics and the paired-error point estimate and 95% percentile interval; and
• r_diagnostics.csv with hour- and month-level error summaries.

Expected console output begins with:

audit_ok test_rows=2168 model_mae=49.893855 paired_delta=-29.155307 ...

The exact bootstrap bounds appear in the completed output and are checked for their declared properties. Machine-readable tables allow CI and later review to compare values without scraping prose.

What this audit does not certify

Passing the script means that the pinned R environment reproduced the stated calculations from the supplied prediction file. It does not certify the upstream data as complete, the feature availability as operationally realistic, the model as fair, the decision process as cost-effective or the artefact as safe to deploy.

Before production use, add:

• provenance checks between the prediction file and registered model run;
• uncertainty that reflects model selection and temporal retraining;
• cost-weighted metrics tied to the operational decision;
• subgroup definitions and minimum sample rules set before inspection;
• drift tests on current inputs and residuals; and
• a review of who bears the cost of over- and underprediction.

The historical dataset cannot answer those governance questions on its own.

When R adds value here

R provides a compact statistical audit that is independent of the Python training library. Base R is sufficient for the first release: metric reconstruction, diagnostics, block resampling and machine-readable output all remain executable without a second package lock. A later report layer may render these outputs, but it should consume the tested files rather than replace the underlying audit.

Return to the Programming for Data Scientists learning path for the complete contract and proposed extensions.

Primary references

• R Core Team (2026), An Introduction to R, version 4.6.1.
• R Core Team (2026), R statistical functions reference, version 4.6.1.
• Fanaee-T, H. (2013), Bike Sharing, UCI Machine Learning Repository, CC BY 4.0.

AI assistance helped draft the guide and code. Independent recalculation, explicit resampling and executable thresholds make the numerical claims auditable. Clean-environment reproduction was verified on 15 August 2026, and human editorial review followed; the historical-data, causal and deployment limitations remain in force.