This manual aims to highlight the importance of soil spectral data in advancing soil science and underscores the central role of R software in this transformation. Whether applied in academic research, agricultural consultancy, or environmental conservation, this manual may be useful to understand and effectively utilize soil spectral data in predicting soil properties. However, new users often find it difficult to navigate R’s extensive functionality, especially when it comes to soil spectral analysis, as comprehensive resources and codes for all types of spectral analysis are not readily available in one place.
The book Soil Spectral Analysis in R aims to bridge this gap by providing a comprehensive guide with ready-to-use codes for most operations related to spectral data analysis. This book is designed to give the user a guided tour of the R platform for spectral data modeling using machine learning, with a focus on methods used predominantly in scientific publication. There are 5 chapters which mainly deal with Introduction to R, soil spectral data handling in R, spectral data pre-processing, and Spectral data modeling.
Soil analytical data is crucial for understanding the physical, chemical, and biological properties of soil, which can significantly impact agricultural productivity and environmental health. As global challenges such as food security, climate change, and the demand for sustainable land use escalate, the need for accurate, efficient, and cost-effective soil analysis has never been more pressing. Traditional soil testing methods, while reliable, are often labour-intensive and time-consuming. Soil spectral data, by contrast, presents a transformative solution—enabling rapid, non-destructive assessment of multiple soil properties through high-throughput techniques. Spectral data is typically acquired using instruments like diffuse reflectance spectroscopy (DRS), near-infrared (NIR), or mid-infrared (MIR) sensors. These instruments capture the interaction between soil and electromagnetic radiation, providing a rich dataset that reflects a wide array of soil attributes, including organic carbon content, moisture levels, texture, nutrient availability, and mineral composition. By leveraging this spectral information, researchers can develop predictive models that are not only highly efficient but also scalable across varied agricultural and ecological landscapes. The use of R software in soil spectral data modelling has further revolutionized this field. R offers a robust, open-source platform for statistical analysis, machine learning, and data visualization, making it a preferred tool for soil scientists, agronomists, and environmental analysts. Its vast ecosystem of packages tailored to large datasets, combined with the flexibility to build custom models, empowers users to handle complex analyses such as multivariate regression, machine learning, and geospatial data processing. These capabilities are crucial for unlocking the full potential of soil spectral data. This manual aims to highlight the importance of soil spectral data in advancing soil science and underscores the central role of R software in this transformation. Whether applied in academic research, agricultural consultancy, or environmental conservation, this manual may be useful to understand and effectively utilize soil spectral data in predicting soil properties.
R has become the world-wide language for statistics, predictive analytics, and data visualization. R is not limited to statistics; it provides a wide range of tools for spectroscopy. It offers a wide range of methodologies for spectral data preprocessing, spectral analysis, multivariate analysis of the spectral data, machine leraning models for prediction of properties, and visualization. As an open source project, it’s freely available for a range of platforms, including Windows, Mac OS X, and Linux. An integrated development environment (IDE) for R, R Studio, allows interactive execution of R functions with user interface. RStudio is available in two formats - RStudio Desktop and RStudio Server - both available in free and fee-based (commercial) editions. 1.1. Download and Install R The R software setup files can be downloaded from any of the widely distributed Comprehensive R Archive Network (CARN). To download R, go to url: https:// www.r-project.org/ and choose your preferred CRAN mirror, a location close to you and follow the steps to download and installation of R and Rtools. 1.2. Download and Install R Studio Download and Install the R Studio once the base R is installed. Go to https://www. rstudio.com/products/rstudio/download/ and click on download R Studio Desktop with Open Source license. 1.3. Updating R To update the R, we need the package “installr” and use function “updateR( )”
This chapter deals with data importing, exporting and data manipulations in R. R can import almost all kinds of data format. The basic data format of .txt and .csv and data from other softwares like MS office .xls and .xlsx, spss ……, stata ……., etc. however, we will discuss in this chapter to read and write the basic formats and the .xlsx data which we use the most. Further, we will also discuss the basic data manipulations. For a detailed reading on data manipulation, readers can refer to the first book of the series namely “Data Handling and Manipulation in R”. 2.1. Primary Spectral Data Processing The original spectra consisting of relative reflectance values of 2151 points (at 1 nm interval) between 350 and 2500 nm were averaged at every tenth-nanometer wavelength interval from 360 to 2490 nm by integration technique. Thus, the new spectra contained reflectance values at 360, 370, 380, 2490 nm, are inclusive. For example, the new reflectance value for 360 nm represented an integrated value of the original reflectance values from 356 to 365 nm, inclusive. Similarly, the reflectance at 400 nm was an integrated reflectance values from 396 to 405 nm, while the reflectance at 2490 nm was an integrated value of reflectance from 2486 to 2495 nm. This was done to reduce the volume of data for analysis and to match it more closely to the spectral resolution of the instrument (3 to 10 nm). This procedure resulted in a new dataset, with reflectance values specified for 214 wavelengths for all samples, which were further processed for noise removal. 2.2. Importing Data in R We can rerad many format of tabular data in R such as excel, text, csv, etc. In thid book we will be using the csv format. Before reading the data, arrange data in excel/CSV sheet so that spectral data in column is followed by chemical data/ lab data. This will ensure that if any soil property or properties added in next column there is no need to change code for selection of spectral data in R environment.
Data pre-processing is a crucial step in data analysis and machine learning that involves preparing raw data for further processing and analysis. Proper preprocessing helps in improving the quality of the data, which in turn enhances the performance of machine learning models. Smoothening VIS-NIR spectra enhances signal quality by reducing noise, crucial for better spectral data interpretation. Key methods include Savitzky-Golay and moving average filters, wavelet transform (WT), and frational derivatives. Savitzky-Golay Filter: The Savitzky-Golay filter is widely used for smoothing spectral data. It preserves the original shape and features of the signal better than simple averaging filters.Savitzky-Golay Filter fits low-degree polynomials to successive subsets of data points using linear least squares, effectively preserving the signal’s original shape and features, making it ideal for spectroscopy. Moving Average:Moving Average is a simple method that reduces random noise by averaging neighboring data points. While effective, it can sometimes blur sharp features. Wavelet Transform: Wavelet transform is effective for denoising spectral data, especially when dealing with non-stationary signals.Wavelet Transform (WT) decomposes spectral data into multiple scales, providing both time and frequency information. It is particularly useful for identifying features like peaks and trends. WT includes Continuous Wavelet Transform (CWT), Discrete Wavelet Transform (DWT), and Maximal Overlap Discrete Wavelet Transform (MODWT). Fourier Transform Filtering: Fourier transform can be used to remove highfrequency noise components from the spectra. Median Filter: Median filtering is useful for removing outliers and preserving edges in the spectral data. Fractional derivative:Fractional derivative generalizes traditional derivatives to non-integer (fractional) orders, allowing for arbitrary order derivatives. This provides a flexible, nuanced approach to modeling and analyzing complex, nonlinear, or memory-dependent behaviors.
4.1. Data Set Generation for Modelling We have generated 54 transfomed spectra by applying different combination of parameter. Now this spectrum is to be tagged with respective soil sample and desired soil properties for which spectral modeling has to be done. Here we are generaring data tagged with CaCO3. For that we have to use cbind() fuction. Here we are keeping soil properties in first column and spectra after that column. We are storing data in R memeory and naming data in such way that we can identify properties and transfoemed spectra attached with that data. After generating data for modeling, data is divided into calibration and validation data set by using random sampling method or Kedard- Stone method # Data set for CaCO3 with raw spectra #### Data.CaCO3.pc.113=cbind(data10nm_$CaCO3_equivalent_per, First_ derivative10nm113) #case 1st Here in case 1st, CaCO3 indicating properties and 113 indcating without smoothed 1st derivative, 1st order polynomial 3rd point spectra is tagged with these soil samples. Same was followed with other cases also
For applying spectral data models with other machine learning and regression algorithms we have to follow all steps upto Chapter 4 till the partition of whole datainto calibration and validation data set. After that different algorithms is used to develop others models so that,the best model among the different algorithms can be identified and used for better perdiction of soil properties. 5.1 Principal Component Regression (PCR) Principal Component Regression (PCR) is a regression technique that combines Principal Component Analysis (PCA) and multiple linear regression. It is particularly useful when dealing with multicollinearity in the predictor variables, as it transforms the predictors into a set of uncorrelated components before performing the regression. set.seed(123) pcr.CaCO3.pc.223SGS<- train(CaCO3.pc ~., Train_Data.CaCO3.pc.223SGS, method =”pcr”, trControl = custom, tuneLength=25, importance = TRUE) Explaination for code train(CaCO3.pc ~ ., data = Train_Data.CaCO3.pc.223SGS, ...) Formula (CaCO3.pc ~ .): Specifies the model formula where CaCO3.pc is the dependent variable and . indicates that all other columns in Train_Data.CaCO3. pc.223SGS are independent variables. Data (data = Train_Data.CaCO3.pc.223SGS): The dataset used for training the model. method = “pcr”:Specifies that the model to be used is Principal Component Regression (PCR). trControl = custom:Specifies the training control method. custom should be an object of class trainControl created earlier, defining the resampling method (e.g., cross-validation).This example sets up 10-fold cross-validation. tuneLength = 25:Specifies the number of different values for the number of principal components to consider. The package caret will try 25 different values to find the optimal number of components.
