---
title: Data Import and Transformation
date: 2025-07-10
date-modified: last-modified
---

```{r}
library(tidyverse)
```

# Import

## Explicitly define variables and types

```{r}
cols <- cols_only(
  Area = col_double(), Perim. = col_double(), Width = col_double(),
  Height = col_double(), Major = col_double(), Minor = col_double(),
  Circ. = col_double(), Feret = col_double(), MinFeret = col_double(),
  AR = col_double(), Round = col_double(), Solidity = col_double(),
  L_BlackBackground = col_double(), a_BlackBackground = col_double(),
  b_BlackBackground = col_double(), L_GreenBackground = col_double(),
  a_GreenBackground = col_double(), b_GreenBackground = col_double(),
  Color_sd = col_double(), ColorNature = col_factor(),
  Color_Name = col_factor(), Color_Category = col_factor(),
  Group = col_factor(), RF_use = col_logical(),
  File_Name = col_character(), Particle_Num = col_character()
)
```

## Load all data sets and combine them

```{r}
data <- list.files("data", pattern = "*.csv$", full.names = T) |> 
  map(\(path) read_csv(path, col_types = cols)) |>
  list_rbind()
```

```{r}
data <- data |> janitor::clean_names()
```

All variables rewritten in *snake_case*.

# Transformations

## Duplicated items

```{r}
data |> count(file_name, particle_num, sort = TRUE)
```
There are **2 duplicated items** from SFS220112_VNG_T1_1_25000. 

```{r}
data <- data |> distinct(file_name, particle_num, .keep_all = TRUE)
```

They are filtered out from the database.

## Unique identifiers

```{r}
data <- data |> mutate(
  id = str_c(file_name, "_", particle_num), .keep = "unused", .before = 1
)
```

## Factor variables

```{r}
data <- data |> mutate(
  color_nature = color_nature |> fct_recode(
    opaque = "Opaque",
    translucent = "TT"
  ),
  
  group = group |> fct_collapse(
    turf = c("Artificial turf", "Artificial.turf"),
    filament = c("Filament"),
    film = c("Film.Sheet", "Film/Sheet"),
    foam = c("Foam"),
    fragment = c("Fragment"),
    pellet = c("Pellet"),
    spherule = c("Spherule.Microbead", "Spherule/Microbead"),
    other_level = "other"
  )
)
```

## Writing database

```{r}
data |> write_rds("data/data.rds")
```

