May 2017 LVIS Upgrades

May 2017 LVIS Upgrades

Processing

~~Waveforms on B that are just WRONG?~~
- Identify where this is happening, how often? How to identify?
- Scan through all the data to see if it exists in bursts or on other channels
- FYI ALL three channel RF cables were only finger tight (the input to the bias tee) 2017.05.23
- FYI channel B was physically loose inside receiver box tightened and tested and looks good again 2017.05.24

Ground Station

Auto copy data when an external drive is plugged in
- User
  - Plug in both external archive drives (red and blue)
  - Plug in flight data drive with label LVISF_EXTERNAL and have the data rsync automagically
- Links:

Hardware Purchases

Ground computer setup (on order, needs to be tested when it gets here)
- Ordered two new laptops that have USB3 and eSATA
- Ordered a cable that converts eSATA to USB3 to attempt to enable windows to properly move data at high speed
~~Just get more of the tough books, they're fast enough and will copy stuff… and are $1000 each~~ $250 refurbished laptops, ought to do it 2017.05.24

Mini computer does not need to be encrypted, go with a NUC or other small form factor PC (ARK!)
Buy an off the shelf NAS with ZFS for BTFS based files and lots of big drives
Can use existing Mac Mini computers or Panasonic Toughbooks
Drive Cost $600 + $400×5 = $2600 for 30TB of backed up storage
Synology DS916+ (8GB) Diskless System Network Storage
Seagate IronWolf 10TB NAS Hard Drive 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive ST10000VN0004 x 5
Mini computer:
- https://www.newegg.com/Product/Product.aspx?item=N82E16856102096 NUC $368
- https://www.newegg.com/Product/Product.aspx?Item=N82E16820104476 16 GB Ram $125
- https://www.newegg.com/Product/Product.aspx?Item=N82E16820147374 1TB ssd $350
- Will still need a monitor / keyboard / mouse and it is MINI-HDMI but under $1000 for a transfer machine
- Can get a laptop for about that, maybe not with 1tb drive though…

GUI / System Tweaks

~~Add system off button that shuts down everything (closes out everything gracefully)~~
Digitizer calibration buttons (short and long)
~~Digitizer temperatures are not being displayed~~
Display channel threshold above each channel on the waveform screen since it could be changing with Ed mod.
Display waveform delta between even / odd sample mean on each channel (watch for temperature drift in ADC calibration)
Status Summary Window Showing
- Applanix position with ROLL ANGLE shown in
  - red if abs(roll) > 16 degrees
  - blinking red if abs(roll) > 20 degrees
- Applanix status (all the fail bits that get sent)
- Approximate range to the surface in meters (weighted mean of actual returns? or just mean? Weight towards farther away but not PAST GPS altitude, since more likely that's the ground)
- Disk space in GB free (yellow if under 100GB) (red if under 32GB)
- Fiber peak values versus the attenuation settings (and if peak gets above 3500 go red if peak is below 1000 go red)
- File MJD being written to (gives you today's MJD)
- File name currently being written (lvis and innovative at this point.. Dave will need to send you)
- Laser status
- LVIS GPS Timer card status (GPS locked, card locked) (in raw data packet, need to interpret)
- MET (Mission Elapsed Time), GPS Time, Local Time, Date
- Percentage of data seeing triggers in RX window for A B and C
- System repetition rate (should be around 4000)
- Temperatures are within bounds (just values… but can color code good (green) and bad (red))
  - Digitizer ADC0 and ADC1 yellow if > 60C
  - Digitizer ADC0 and ADC1 red if > 70C
  - TX Galv temperatures (limits once we get heaters heating properly around 25C or so)
- Time since last packet (of each kind you're expecting)
- Watch for clipping (how to do this? This is kind of tough)

Hardware Upgrades

Camera cables connectorized
Improve CPU and digitizer cooling (mostly done)
- Replaced under chassis three fans with higher CFM versions of the fans we have 2017.05.19
- Re-routed cables to open up inlet under chassis 2017.05.23 (verified ambient steady state operations in the lab show 58C primary ADC temperature which is probably 10 degrees cooler than before) (box not assembled, this is only with back panel installed)
- ~~More fans are on order to replace on the top of the computer~~
- Fans have arrived and will be wired and installed this week (week of June 05, probably Thursday)
~~Improve drive copying speed from external ext2 data partition~~ 2017.06.01
- Will try ext2 reading with windows computer next
- TRY the Panasonic using a USB to boot into Ubuntu and attempt the same disk copy and check total time.
- Writing from ARK-1550 USB3 port in ubuntu linux to external SSD Camera Drive is 380MB/sec
```
time sh -c "dd if=/dev/zero of=ddfile bs=8k count=800000 && sync"
6553600000 bytes (6.6 GB, 6.1 GiB) copied, 17.0243 s, 385 MB/s
```
- Writing from ARK-1550 USB3 port in ubuntu linux to exteral ext2 drive was 520MB/sec
```
time sh -c "sudo dd if=/dev/zero of=ddfile bs=8k count=800000 && sync"                                                                                                                                            
6553600000 bytes (6.6 GB, 6.1 GiB) copied, 12.5258 s, 523 MB/s
```
- Plugging USB3.0 cable into my new ASUS laptop and reformatting a Samsun 840 drive, I could not get more than 40 MB/sec throughput reading from the USB3 external camera SSD… so it is being limited by the OS… maybe my hardware just doesn't play nice with the Addonics firmware?! - 2017.05.19
- DONE! USB3 to eSATA cable on ARK got me 224G in 31 minutes or 7.2 GB/sec or a full TB in 142 minutes so about two hours to move the entire disk to a fast storage disk
```
sent 239,704,196,084 bytes  received 1,973 bytes  126,259,783.02 bytes/sec
total size is 239,645,682,099  speedup is 1.00

real    31m37.963s
user    25m48.620s
sys     6m8.856s
```

Software Upgrades

~~Applanix packets only every other second now?~~ -Dan was making up for the fact we were sending him two sets of data 2017.05.26
Automate the gain settings on the fibers
- Look at temperatures versus start pulse energy
Automated threshold setting based on noise and standard deviation
- ~~We can do this in software: make the OK's go quiet~~ - Dave Y. 2017.05.24
- We can do this in FPGA: Ed / Sigma to modify the flight code on noise calculations
Camera software get it up and running
~~Clean shutdown (let got of Spectrum digitizers properly)~~ - Dave Y. 2017.05.24
~~Digitizer single edge vs dual edge clock~~ - Ed says we are single edge (1.6 GS) 2017.05.23
Digitizer temperature sensitivity
Document base map creation
Document operations manual v1.0
Directory structure creation on startup to write in archive layout
- Startup script! So it can be interactive: aka… disk is full, want to delete? etc….
  - check for free disk space
  - clear the internal drive (prepare for writing)
  - create the data structure on the external drive
  - generate symbolic link to data directory
  - synchronize applanix and javad data before flight (or clear?)
    - How long does it take to wget mirror from scratch? (20 minutes per Applanix and 5 minutes for a Javad (about))
    - Store the applanix and javad data internally and then rsync to new external drive during startup script
Disk failure needs to be gracefully handled and logged
~~Enable recalibration of the digitizers while flying / operating~~ 2017.06.01 - Dave Y. worked with Lie at Spectrum and they enabled commands that allow a long and short calibration that can take place while the system has the cards open. The short calibration takes only a few seconds.
- Initial thoughts are to stop laser / close spectrum / open spectrum / fire laser
- Dave Y. tried this, and it does work to do reset, but have to re-init cards afterwards and it breaks merge at the moment. I think resetting merge is pretty straight forward, just pop all data from the circular buffers.
Galvanometer pattern needs to better match ground speed of aircraft to get rid of gaps
Galvanometer table will need to carry parameters used to generate it in ASCII and binary forms… no reason to have to reverse engineer the table!
Galvanometer trigger timing for position digitzer (SDF board)
GPS status flag enabled (GPS good or bad)
GUI show shots, filename percentage missed, etc…
~~Add a laser shutdown that stops data collection and then turns off the laser (keeps triggers going)~~ - Dave Y. did this a while back, works great, but Dan needs to add controls
~~Javad stopping and Applanix stopping~~ - Dan wrote this in Python and it appears to work well!
- How to download everything
Log all status packets and applanix raw data
Log file needs to be standardized and used! Log everything to the file not to the display
Log FPGA version on startup to verify build that is uploaded on both digitizers
~~md5sum in real time if possible (TEST and validate)~~ - 2015.05.25 works great! Cleaned up output so only file name comes through (had path before)
~~password free login from GUI to backend~~ 2015.05.25 done using this link
~~Send ADC temperatures to GUI as a status packet~~ - Dave Y. 2017.05.24 All 5 temps for both digitizer are sending over, Dan needs to get new packet
~~Send only primary Applanix packets to Dan, not everything we see (filter out by IP source)~~ - Dave Y. looks at the IP address and only sends the one on the instrument to GUI.
Send swath characterization to Dan for real time map geolocation
- Dave Y. and Dan C. got the communications working, need to fill with meaningful information
Dan needs galv value vs time for the edges and middle position
- Packet should be shot return list or something to that effect
Startup as a single script, can we get there? I think so with launched “screens” and a bash script
~~Post flight script to download the flight Javad and Applanix data.~~ - 2017.06.05 Dave R. works well at the moment, probably some corner cases where it might not…
- Multi-thread to download all the javad data at once? Why not… the javads are the limit, not drive and network
Record raw Applanix packets to disk (maybe just the primary bench mounted Applanix for now)
~~Record CPU temperature (this looks to be as easy as reading a system file:)~~ - Dave Y. Fixed last week.
```
[lvis@lvisf ~]$ cat /sys/class/thermal/thermal_zone*/temp # shows all the cores
27800
29800
```
Waveform packet display, RX bleeds into GUI FR, is that on data system side or display?
~~Zero file for first file~~ Dave Y. 2017.05.25

Polishing the system

flow rate monitoring
signal level monitoring
auto attenuation on the TX and FR
~~real time md5sum OK or a bridge too far for data system?~~ works! 2015.05.25
better range display
data system needs to send real timie moving map information to GUI
bring a “little john” bathroom for the boys and girls
1. boyz
2. ladies
3. or is this better? http://www.sportys.com/pilotshop/travel-john-pack-of-18.html
need a percentage of shots triggering (0 to 100% for all three channels)
GUI, HOWTO use the basemap with no flight plan?
need accurate mean and standard deviation values on the GUI display
auto thresholding (FPGA modification to set the multiplier of SD + noise mean)
tx heaters don't appear to be on, the temperature is 13C
single drive fail or full should only stop the system from writing out to that drive, not stop completely
automate downloading of javad and applanix data (wget mirror before and after)
1. each directory will have md5sum of all files
2. will need to stop the applanix and javads in order to download the last / latest files
modify mounting plate to allow us to swap camera lenses? - maybe just a small scallop/notch somewhere
pee bottles and general plane discomfort kit - check Sporty’s pilot shop (what for the ladys?)
flashlights - all shapes and sizes
portable battery powered vacuum - Dyson? with an extra battery or one that uses the same drill batteries we have now (not necessarily on the plane - but at least on the ground for before/after flights)
Curtain between TX and RX FOVs

Long term

move to TEC only - no liquid laser cooling (in your best movie promo voice - Imagine, a world without liquid coolant… or at least a true closed loop system so we don't have to deal with fluids at all)
1. talk to Fibertech
2. design our own
3. heat pipes? or heat spreaders

Table of Contents

May 2017 LVIS Upgrades

Processing

Ground Station

Hardware Purchases

GUI / System Tweaks

Hardware Upgrades

Software Upgrades

Polishing the system

Long term