Table of Contents

ARCSIX Field Processing

FIXME - Make the ingest log how much space is free on the disk each time it syncs

FIXME - TODO:

Aircraft Data

  1. Backup lvisf drive
  2. Backup camera drive
  3. Backup rtp drive except for 'rtlvis' folder as it is a duplicate
  4. Create proc folder for the new mjd
    1. Create subfolders
    2. Copy the ql2 files from aircraft RTP drive into v2 directory
  5. Generate tiles

Ground GPS

Specifics

# required variables
export mjd=60461
export lvisf_path=$(find /mnt/archive/lvisf/ -type d -name "${mjd}")
ln -s $lvisf_path /mnt/archive/mjd/

Copy

Copy lvisf External Drive

Ingest bash script: ingest_lvisf_external.zip

  1. Attach drive to computer (do not let it mount)
  2. Run the following script 
    ingest_lvisf_external

Copy 510 card

ingest_lvisf_card

Copy 610 usb

ingest_lvisf_usb

Copy Camera drive

ingest_lvisf_camera

Copy RTP drive

ingest_lvisf_rtp
create_archive_links.sh ${mjd}

GPS Ground Data

  1. Laptop out to shack
  2. run 
    javad_get_bases
  3. Plug ethernet into switch
  4. Copy data from laptop via rsync to raid (on laptop) (open cygwin shell) 
    rsync -vaP ~/gndnav/ lvis@192.168.0.48:/mnt/archive/gndnav/
  5. Sync archive to external GROUNDNAV ssd 
    rsync -vaP /mnt/archive/gndnav/ /media/lvis/GROUND_NAV/gndnav/

Care Package Creation

MJD Skeleton

create_mjd_care_package.sh ${mjd}

Quality

Generate KMZs

On Ground If RTP Fails
  1. Create directory structure and symbolically link RTP ql2 files from aircraft 
    # MOUNT proc drive
sudo mount /dev/disk/by-label/LVISF_PROC /mnt/proc
mkdir -p /mnt/proc/lvisf/${mjd}/v2
cd /mnt/proc/lvisf/${mjd}
ln -s /mnt/archive/rtp/mjd/${mjd}/v2/*.ql2 /mnt/proc/lvisf/${mjd}/v2
  2. Kick off processing 
    time ~/field_tools/local/rtp_local_proc.sh ~/field_tools/local/config_610i_arcsix2024.cfg ${mjd}
  3. Channel mixed processing 
    time ~/field_tools/local/rtp_local_proc.sh ~/field_tools/local/config_610i_arcsix2024_chanmix.cfg ${mjd}

If you need to start over 

cd /mnt/proc/lvisf/${mjd}
~/field_tools/local/clean.sh all

If you need to use a different trajectory 

cd /mnt/proc/lvisf/${mjd}/traj/
find /mnt/archive/usb/APP610I-2/ -type f -newermt 2024-06-06 -exec cat {} \; > /mnt/proc/lvisf/${mjd}/traj/traj${mjd}_610i.bin

If you need to run the kmz generation manually 

inst=LVISF
tiff2kml.py height "$inst""$mjd"_Field_Height.kmz "$inst""$mjd"_Field_Height height.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py elevation "$inst""$mjd"_Field_Elevation.kmz "$inst""$mjd"_Field_Elevation elevation.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py elevation_block "$inst""$mjd"_Elevation_5m.kmz "$inst""$mjd"_Elevation_5m ${HOME}/field_tools/local/elev_block2.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py energy "$inst""$mjd"_Field_Energy.kmz "$inst""$mjd"_Field_Energy energy.png ${HOME}/field_tools/local/LVIS.png coords.txt
Copy RTP Results

This is the standard, only do the above if the onboard RTP generation didn't produce results

  1. Create a directory in the proc drive 
    mkdir -p /mnt/proc/lvisf/${mjd}
  2. Copy the files from the RTP archive (which is a copy from the plane) (excluding rtlivs just in case it was somehow copied over) 
    rsync -vaP --exclude='rtlvis' /mnt/archive/rtp/mjd/${mjd}/ /mnt/proc/lvisf/${mjd}/
  3. Copy all kmz files to external drive (optional, but a good idea assuming GROUND_NAV is mounted!) (max depth was because there is an older folder in one of the days to avoid copying those) 
    find /mnt/proc/lvisf/ -maxdepth 2 -name "*.kmz" -type f -exec cp -a {} /media/lvis/GROUND_NAV/kmz/ \;

Summary Plots

  1. Scrape the ql2 files 
    mkdir -p /mnt/proc/summaryplots/${mjd}
scrape_ql2 /mnt/proc/lvisf/${mjd}/v2 /mnt/proc/summaryplots/${mjd}/${mjd} -s 17
  2. Generate the plots 
    cd /mnt/proc/summaryplots/${mjd}/
~/field_tools/rtp_summaryplots/runall.sh ${mjd}
  3. Copy to external drive
    1. Mount GROUND_NAV
    2. Sync the files to it 
      rsync -vaP /mnt/proc/summaryplots/ /media/lvis/GROUND_NAV/summaryplots/
  4. Sync back to incoming at GSFC

Mission Meta

ln -s /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/arcsix2024.cfg ~/field_tools/mission_meta/
  1. Modify the configuration file so that the last mjd is captured
    1. Edit the config 
      nano /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/arcsix2024.cfg
  2. Generate mission meta data 
    cd ~/field_tools/mission_meta
time ./make_mission_data_summary.py arcsix2024.cfg
  3. Copy to external drive
    1. Mount GROUND_NAV
    2. Sync the files to it 
      rsync -vaP /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/ /media/lvis/GROUND_NAV/coverage/

Results end up here: 

/mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage

External Drive MJD Tar

  1. Attached external drive and mount with Ubuntu manager
  2. Run this script based on MJD 
    sleep 2h ; create_mjd_external_archive.sh ${mjd} /media/lvis/A6_BACKUP03/
  3. Copy the camera folder to the external drive.

Copy Applanix Logs from External Camera Backup

find /media/lvis/A6_BACKUP04 -maxdepth 2 -type f -iname '*_log.txt'  -exec cp "{}" /media/lvis/GROUND_NAV/applanix_logs/ \;

# to make sure you get ALL txt files (Rob and John named them differently)
find /mnt/archive/camera/id/ -maxdepth 3 -type f -iname '*.txt'  -exec cp "{}" /tmp/applanix_logs/ \;

Raw Data Copy to Gaia

  1. Copy card data 
    rsync -vazP /mnt/archive/card/ lvis@gaia.lattice.net:incoming/card/
  2. Copy usb data 
    rsync -vazP /mnt/archive/usb/ lvis@gaia.lattice.net:incoming/usb/

Verification

Final Raid Inventory