• ARCSIX-2 Data Summary
• ARCSIX-2 Alignment Summary
• Backup Drives “/media/lvis/A6_BACKUP01”
• Backup Drives “/media/lvis/A6_BACKUP02”

- Make the ingest log how much space is free on the disk each time it syncs

- TODO:

• Change the IMU names to 2024
• Pluck out some low altitude data for Michelle
  • Low level (1500' & 4000')
  • Spiral down reasonable ice

    IDL> print,min(c.bctime)
           48708.312
    IDL> print,max(c.bctime)
           48878.315

  • Descent during porpoise with clouds?
  • East vs West area
• Find photo that matches the data that Michelle generated the image for (lat/lon)
  • Lat 84 42 15 N Lon 65 41 W ( -65.68333333 84.70416667 )

    x=[-65.68333333,-65.68333333] & y = [84.70416667,84.70416667]
    ; zm in 52690 (14:38:10) there abouts and 53350 (14:49:10)

  • File is: lvis_20240528_143800_00000306.dat
• Add final scripts to make symbolic links for 510i, CAM1, P1
• Fix this page: How straight forward is this? (schedule with dinner included)
• PDF / Print this so you have a hard copy and a local version

• ARCSIX-1 Data Summary (ARCSIX-1 Data Summary original page)

1. Backup lvisf drive
2. Backup camera drive
3. Backup rtp drive except for 'rtlvis' folder as it is a duplicate
4. Create proc folder for the new mjd
   1. Create subfolders
   2. Copy the ql2 files from aircraft RTP drive into v2 directory
5. Generate tiles

• Javad Basestation Script

# required variables
export mjd=60461
export lvisf_path=$(find /mnt/archive/lvisf/ -type d -name "${mjd}")
ln -s $lvisf_path /mnt/archive/mjd/

Ingest bash script: ingest_lvisf_external.zip

1. Attach drive to computer (do not let it mount)
2. Run the following script

   ingest_lvisf_external

ingest_lvisf_card

ingest_lvisf_usb

ingest_lvisf_camera

ingest_lvisf_rtp

create_archive_links.sh ${mjd}

1. Laptop out to shack
2. run

   javad_get_bases

3. Plug ethernet into switch
4. Copy data from laptop via rsync to raid (on laptop) (open cygwin shell)

   rsync -vaP ~/gndnav/ lvis@192.168.0.48:/mnt/archive/gndnav/

5. Sync archive to external GROUNDNAV ssd

   rsync -vaP /mnt/archive/gndnav/ /media/lvis/GROUND_NAV/gndnav/

create_mjd_care_package.sh ${mjd}

1. Create directory structure and symbolically link RTP ql2 files from aircraft

   # MOUNT proc drive
   sudo mount /dev/disk/by-label/LVISF_PROC /mnt/proc
   mkdir -p /mnt/proc/lvisf/${mjd}/v2
   cd /mnt/proc/lvisf/${mjd}
   ln -s /mnt/archive/rtp/mjd/${mjd}/v2/*.ql2 /mnt/proc/lvisf/${mjd}/v2 

2. Kick off processing

   time ~/field_tools/local/rtp_local_proc.sh ~/field_tools/local/config_610i_arcsix2024.cfg ${mjd}

3. Channel mixed processing

   time ~/field_tools/local/rtp_local_proc.sh ~/field_tools/local/config_610i_arcsix2024_chanmix.cfg ${mjd}

If you need to start over

cd /mnt/proc/lvisf/${mjd}
~/field_tools/local/clean.sh all

If you need to use a different trajectory

cd /mnt/proc/lvisf/${mjd}/traj/
find /mnt/archive/usb/APP610I-2/ -type f -newermt 2024-06-06 -exec cat {} \; > /mnt/proc/lvisf/${mjd}/traj/traj${mjd}_610i.bin

If you need to run the kmz generation manually

inst=LVISF
tiff2kml.py height "$inst""$mjd"_Field_Height.kmz "$inst""$mjd"_Field_Height height.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py elevation "$inst""$mjd"_Field_Elevation.kmz "$inst""$mjd"_Field_Elevation elevation.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py elevation_block "$inst""$mjd"_Elevation_5m.kmz "$inst""$mjd"_Elevation_5m ${HOME}/field_tools/local/elev_block2.png ${HOME}/field_tools/local/LVIS.png coords.txt
rm files/*
tiff2kml.py energy "$inst""$mjd"_Field_Energy.kmz "$inst""$mjd"_Field_Energy energy.png ${HOME}/field_tools/local/LVIS.png coords.txt

This is the standard, only do the above if the onboard RTP generation didn't produce results

1. Create a directory in the proc drive

   mkdir -p /mnt/proc/lvisf/${mjd}

2. Copy the files from the RTP archive (which is a copy from the plane) (excluding rtlivs just in case it was somehow copied over)

   rsync -vaP --exclude='rtlvis' /mnt/archive/rtp/mjd/${mjd}/ /mnt/proc/lvisf/${mjd}/

3. Copy all kmz files to external drive (optional, but a good idea assuming GROUND_NAV is mounted!) (max depth was because there is an older folder in one of the days to avoid copying those)

   find /mnt/proc/lvisf/ -maxdepth 2 -name "*.kmz" -type f -exec cp -a {} /media/lvis/GROUND_NAV/kmz/ \;

1. Scrape the ql2 files

   mkdir -p /mnt/proc/summaryplots/${mjd}
   scrape_ql2 /mnt/proc/lvisf/${mjd}/v2 /mnt/proc/summaryplots/${mjd}/${mjd} -s 17

2. Generate the plots

   cd /mnt/proc/summaryplots/${mjd}/
   ~/field_tools/rtp_summaryplots/runall.sh ${mjd}

3. Copy to external drive
   1. Mount GROUND_NAV
   2. Sync the files to it

      rsync -vaP /mnt/proc/summaryplots/ /media/lvis/GROUND_NAV/summaryplots/

4. Sync back to incoming at GSFC

ln -s /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/arcsix2024.cfg ~/field_tools/mission_meta/

1. Modify the configuration file so that the last mjd is captured
   1. Edit the config

      nano /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/arcsix2024.cfg

2. Generate mission meta data

   cd ~/field_tools/mission_meta
   time ./make_mission_data_summary.py arcsix2024.cfg

3. Copy to external drive
   1. Mount GROUND_NAV
   2. Sync the files to it

      rsync -vaP /mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage/ /media/lvis/GROUND_NAV/coverage/

Results end up here:

/mnt/archive/mjd/60410_60550_ancillary/mission_meta/coverage 

1. Attached external drive and mount with Ubuntu manager
2. Run this script based on MJD

   sleep 2h ; create_mjd_external_archive.sh ${mjd} /media/lvis/A6_BACKUP03/

3. Copy the camera folder to the external drive.

find /media/lvis/A6_BACKUP04 -maxdepth 2 -type f -iname '*_log.txt'  -exec cp "{}" /media/lvis/GROUND_NAV/applanix_logs/ \;

# to make sure you get ALL txt files (Rob and John named them differently)
find /mnt/archive/camera/id/ -maxdepth 3 -type f -iname '*.txt'  -exec cp "{}" /tmp/applanix_logs/ \;

1. Copy card data

   rsync -vazP /mnt/archive/card/ lvis@gaia.lattice.net:incoming/card/

2. Copy usb data

   rsync -vazP /mnt/archive/usb/ lvis@gaia.lattice.net:incoming/usb/

• Run md5deep on LVIS raw files
• Run teqc on GPS raw files
• Run applanix time check on raw IMU files
• Camera?

• Ensure each drive has a meta and inventory entry
• Copy inventory and meta files to three places drive_meta/year/campaign/
  • camera, card, lvisf, nav, rtp, usb
  • Box, Teams and Wiki

    # manual creation of inventory
    time_date=$(date '+%Y%m%d')
    time_time=$(date '+%H%M%S')
    inventory_output="${camera_basepath}/inventory/${time_date}_${time_time}_${camera_diskid}_${linkname}_inventory.txt"
    meta_output="${camera_basepath}/meta/${linkname}.txt"
    mkdir -p ${camera_basepath}/inventory/ ${camera_basepath}/meta/
    meta_path="${camera_basepath}/meta/"
    drivemeta=$(df -h | grep ${camera_mountpoint})
    echo "${time_date} ${time_time} ${camera_diskid} ${drivemeta}"
    echo "${time_date} ${time_time} ${camera_diskid} ${drivemeta}" >> ${meta_output}
    echo "find ${camera_mountpoint} -type f -exec ls -l {} \; >> ${inventory_output}"
    find ${camera_mountpoint} -type f -exec ls -l {} \; >> ${inventory_output}

• md5deep will follow symbolic links, so run this

  time sudo find /mnt/archive/ -type f -exec md5deep -z {} \; > ~/md5deep_archive2024_all.txt
  
  # 68 hours for 44TB not bad
  # /dev/sda2        44T   43T  970G  98% /mnt/archive
  real    4033m5.615s
  user    1616m48.209s
  sys     429m25.964s
  
  # and make a sorted version of the file
  cat ~/md5deep_archive2024_all.txt  | sort -k 3 > ~/md5deep_archive2024_all_sorted.txt