code:work:lvisf:2022:fieldarchive:start
This is an old revision of the document!
Table of Contents
Bash Style Archive
Field Script
Prerequisites
sudo apt-get install hfsprogs ## NOTE is saw that we should disable journaling on the external drive before writing with Linux sudo mount -t hfsplus -o force,rw /dev/sdXY /media/mntpoint # or sudo mount -t hfsplus -o remount,force,rw /mount/point
HowTo
- Attach flight drive to archive CPU
- Attach mirror drive to rsync to (either HFS+ journaling disabled or EXT4)
- Attach archive drive which will get the compressed mjd tar file for each day
- rsync the flight data to the mirror drive (directory by directory at top level)
- run compress script (see below) to generate a MMMMM_0.tar.gz for each flight day on the archive drive
later you can add MMMMM_[1-z].tar.gz for any additional files that were missed?
Compress All Dirs To A Folder
lvis@lvis-dell-xps:~/bin$ more compress_all_mjd.sh #!/bin/bash echo "Usage $0 <input dir> <output dir>" M_INPUTDIR=$(echo $1 | sed '/\/$/! s|$|/|') # add a slash if necessary M_OUTPUTDIR=$(echo $2 | sed '/\/$/! s|$|/|') # found online echo Input: $M_INPUTDIR echo Output: $M_OUTPUTDIR cd $M_INPUTDIR for f in *; do if [ -d "$f" ]; then M_BASENAME=$(basename $f) re='^[0-9]+$' # check if this is a just a number (day mjd) (found online) if [[ $M_BASENAME =~ $re ]] ; then M_TARFILE="${M_OUTPUTDIR}${M_BASENAME=}_raw_data.tar.gz" echo "Run on $f which is $M_BASENAME" time tar cf - $f | pigz > $M_TARFILE fi fi done
Mount Tar As Filesystem
- https://csatlas.com/mount-tar-file-ubuntu/ ← archivemount HOWTO
Options we might be interested in
-o readonly disable write support -o nobackup remove archive file backups
- Install if not already on the system
sudo apt install -y archivemount
- Create a mount point
mkdir /tmp/myfiles
- Mount the tar file as read only
# read / write mount which makes a .orig file archivemount myfiles.tar.gz /tmp/myfiles # read / write mount without any backup archivemount -o nobackup myfiles.tar.gz /tmp/myfiles # read only archivemount -o readonly myfiles.tar.gz /tmp/myfiles
- Read data out of archive as if it were a file
- Remove the mount
cd && umount /tmp/myfiles
code/work/lvisf/2022/fieldarchive/start.1649439416.txt.gz · Last modified: 2022/04/08 17:36 by david