Data Cleaning Pipeline Documentation and Instructions

Compiled by: Rosie Howard, 2024
Edits and contributions: Zoran Nesic, Sara Knox, June Skeeter, Paul Moore, Karolina Bajda, and other EcoFlux Lab members and affiliates.

Revisions:
- June 2025: Further detailed “full” documentation added, all moved to lab website.
- March 2025: Full documentation added including pipeline features and further information to supplement the tutorial.
- February 2025: “Recently Added Features” section 7.2 added. Full documentation still in progress.
- October 2024: These are the “quick-start” instructions for new users of the data-cleaning pipeline. The full documentation will be made available at a later date.

Useful links

• Recently added pipeline features
• Link to Overview of Data Cleaning Pipeline slides presented during FLUXNET-CH4 V2.0 Workshop (on Day 1: Afternoon, and Day 2: Morning and Afternoon).
• Link to YouTube playlist of ALL presentations from FLUXNET-CH4 V2.0 Workshop.

Documentation outline

1. Motivation: The Importance of Flux Data Standardization and Reproducibility
    1.1  Note on EddyPro Processing of High Frequency Data

2. Software Installation
    2.1  Install Software: Git, and Create Github account (optional)
    2.2  Download Biomet.net Library
    2.3  Install Software: Matlab
    2.4  Configure Matlab for Biomet.net
    2.5  Install Software: R/RStudio
    2.6  Install Software: Python (optional)

3. Data Cleaning Principles

4. Quick Start Tutorial - Recommended for First-Time Users
    4.1  Quick Start Tutorial: Project Directory Structure and Matlab Configuration
    4.2  Quick Start Tutorial: Create Database and Visualize
    4.3  Quick Start Tutorial: Create First Stage INI File
    4.4  Quick Start Tutorial: Create Second Stage INI File
    4.5  Quick Start Tutorial: Third Stage and Ameriflux Output

5. Full Documentation: Features, Details, and Other Useful Information for Advanced Users
    5.1  Full Documentation: Project Directory Structure and Matlab Configuration
    5.2  Full Documentation: Create Database and Visualize
    5.3  Full Documentation: First Stage INI Files
    5.4  Full Documentation: Second Stage INI Files
    5.5  Full Documentation: Third Stage and Ameriflux Output

6. Data Visualization
    6.1  Matlab plotApp
    6.2  R-Shiny App
    6.3  Other Biomet Plotting Tools

7. Troubleshooting and FAQ
    7.1  Recommended Software Versions
    7.2  Recently Added Features