Cleaning

Goals of this notebook

The steps we’ll take to prepare our data:

  • Download the data
  • Import it into our notebook
  • Clean up data types and columns
  • Export the data for next notebook

Set up

The first step is to load the library packages that will help with certain functions in this assignment.

library(tidyverse)
library(janitor)

Importing the data

To get data into our notebook, we can either download it from the internet or import it from our computer. In this case, the data I downloaded from Climate Data Online is already in my files’ data_raw folder, so the next steps would be to import it, assign it to an R object, and glimpse at it.

For future: updated data can be downloaded from Camp Mabry station directly.

To lowercase the columns into code-friendly format, we will use the clean_names() function.

weather_raw <- read_csv("data_raw/3796405.csv") |> clean_names()
Warning: One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
Rows: 31510 Columns: 10
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (2): STATION, NAME
dbl  (6): PRCP, SNOW, SNWD, TAVG, TMAX, TMIN
lgl  (1): TOBS
date (1): DATE

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
weather_raw

The warning notification above is from suspected inconsistency in the tobs column, which we will later remove.

Before proceeding further, I want to make sure the data set I’m working with:

  • Has a single header row with well-formed column names (and “_” instead of spaces).
  • Each column has the same kind of data: numbers(dbl), words(chr), etc.
  • The file has no notes or comments.
weather_raw |> glimpse()
Rows: 31,510
Columns: 10
$ station <chr> "USW00013958", "USW00013958", "USW00013958", "USW00013958", "U…
$ name    <chr> "AUSTIN CAMP MABRY, TX US", "AUSTIN CAMP MABRY, TX US", "AUSTI…
$ date    <date> 1938-06-01, 1938-06-02, 1938-06-03, 1938-06-04, 1938-06-05, 1…
$ prcp    <dbl> 0.00, 0.00, 0.00, 0.40, 0.02, 0.00, 0.00, 0.00, 1.60, 0.01, 0.…
$ snow    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ snwd    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ tavg    <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ tmax    <dbl> 91, 94, 94, 90, 94, 92, 95, 92, 87, 90, 92, 91, 91, 91, 89, 89…
$ tmin    <dbl> 72, 67, 70, 68, 68, 70, 70, 76, 64, 76, 75, 71, 70, 68, 71, 70…
$ tobs    <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…

Fixing the date

For the date to not show up as a numerical value, I will use one of the lubridate functions, mutate(), to modify it. I referred to lubridate documentation for the right format.

Just in case, I also arranged the dates in chronological order from old to new.

weather_date <- weather_raw |>
  mutate(date = ymd(date)) |>
  arrange(date)

weather_date |> glimpse()
Rows: 31,510
Columns: 10
$ station <chr> "USW00013958", "USW00013958", "USW00013958", "USW00013958", "U…
$ name    <chr> "AUSTIN CAMP MABRY, TX US", "AUSTIN CAMP MABRY, TX US", "AUSTI…
$ date    <date> 1938-06-01, 1938-06-02, 1938-06-03, 1938-06-04, 1938-06-05, 1…
$ prcp    <dbl> 0.00, 0.00, 0.00, 0.40, 0.02, 0.00, 0.00, 0.00, 1.60, 0.01, 0.…
$ snow    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ snwd    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ tavg    <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
$ tmax    <dbl> 91, 94, 94, 90, 94, 92, 95, 92, 87, 90, 92, 91, 91, 91, 89, 89…
$ tmin    <dbl> 72, 67, 70, 68, 68, 70, 70, 76, 64, 76, 75, 71, 70, 68, 71, 70…
$ tobs    <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…

Reviewing the data

Let’s review the data in table form to ensure the numbers and names are in proper format.

weather_raw |> head(10)

I will also check the summary stats of the 30,000 rows we are working with.

weather_raw |> summary()
   station              name                date                 prcp        
 Length:31510       Length:31510       Min.   :1938-06-01   Min.   :0.00000  
 Class :character   Class :character   1st Qu.:1959-12-25   1st Qu.:0.00000  
 Mode  :character   Mode  :character   Median :1981-07-19   Median :0.00000  
                                       Mean   :1981-07-19   Mean   :0.09114  
                                       3rd Qu.:2003-02-11   3rd Qu.:0.00000  
                                       Max.   :2024-09-06   Max.   :7.55000  
                                                                             
      snow               snwd               tavg            tmax       
 Min.   :0.000000   Min.   :0.000000   Min.   : 0.00   Min.   : 20.00  
 1st Qu.:0.000000   1st Qu.:0.000000   1st Qu.:60.00   1st Qu.: 70.00  
 Median :0.000000   Median :0.000000   Median :73.00   Median : 82.00  
 Mean   :0.002054   Mean   :0.002539   Mean   :70.36   Mean   : 79.59  
 3rd Qu.:0.000000   3rd Qu.:0.000000   3rd Qu.:83.00   3rd Qu.: 92.00  
 Max.   :6.500000   Max.   :6.000000   Max.   :95.00   Max.   :112.00  
 NA's   :7          NA's   :4          NA's   :28857                   
      tmin         tobs        
 Min.   :-2.00   Mode:logical  
 1st Qu.:47.00   NA's:31510    
 Median :61.00                 
 Mean   :58.44                 
 3rd Qu.:72.00                 
 Max.   :93.00

Aside from station identification and name, the summary shows maximum and minimum record of precipitation, temperature, snowfall, and snow depth levels in Austin, Texas. The most recent data is from September 6th, 2024.

Selecting the columns

For the sake of brevity, we will remove the tavg and tobs columns since they contain incomplete and unnecessary info.

weather_clean <- weather_raw |>
  select(!c(tavg, tobs))

weather_clean |> glimpse()
Rows: 31,510
Columns: 8
$ station <chr> "USW00013958", "USW00013958", "USW00013958", "USW00013958", "U…
$ name    <chr> "AUSTIN CAMP MABRY, TX US", "AUSTIN CAMP MABRY, TX US", "AUSTI…
$ date    <date> 1938-06-01, 1938-06-02, 1938-06-03, 1938-06-04, 1938-06-05, 1…
$ prcp    <dbl> 0.00, 0.00, 0.00, 0.40, 0.02, 0.00, 0.00, 0.00, 1.60, 0.01, 0.…
$ snow    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ snwd    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ tmax    <dbl> 91, 94, 94, 90, 94, 92, 95, 92, 87, 90, 92, 91, 91, 91, 89, 89…
$ tmin    <dbl> 72, 67, 70, 68, 68, 70, 70, 76, 64, 76, 75, 71, 70, 68, 71, 70…

Exporting the data

Last but not least, we will export the cleaned data in a native R format called .rds.

weather_clean |>
   write_rds("data_processed/01-weather.rds")