feat(abr-testing): ABR video capture

[munanwugo-opentrons]

Overview

This PR adds video capture functionality to the ABR suite. While a version of this functionality existed before, the way ffmpeg processed m3u8 meant that when trying to record the 30 seconds before an event, the video returned would display the 30 seconds after. This PR fixes that issue by using a 30-clip-long linear buffer of 1-second clips (essentially just 30 seconds) generated by a persistent ffmpeg stream. These clips are all saved in a directory, and this directory is accessed to find all of the clips. The clips are then stitched together when needed.

Test Plan and Hands on Testing

No testing as of yet, waiting for the ABR suite to calm down a little. There's already enough going wrong, I don't need to add another potential point of failure just yet.

Changelog

background_helpers.py

Error Recovery checks were updated to poll every 30 seconds instead of 300 (5 minutes). This is to line up with the size of the buffer.

A new change_robot_video_length() function was added. This was simply to change the length of the video that the robots saved locally on their machines. May remove, as this had no effect.

a new video_capture_buffer() function was added. This launches the persistent ffmpeg stream, tracks the files that the stream generates, and enforces the buffer's maximum length.

run_background.sh

Lockfile logic updated to handle multiple processes within it.

2 new processes launched:

• change_robot_video_length [discrete]
• video_capture_buffer [persistent]

run_helpers.py

A new access_live_stream() function was added to access the files stored in the buffer directory, stitch them together, and output a new file that stores the full 30-second mp4.

This function replaces the "get_livestream_video()" function. Changes were made to functions who called that function to now call this one. x

check_robot_status.py

in get_run_details_from_robot(), when error recovery is detected, it uses access_live_stream() from run_helpers.py to create a video of the most recent thirty seconds. It then sends this video alongside the error recovery message.

Review requests

I'm really worried about one particular race condition here. The way I access the buffer is by collecting the names of all the video files and feeding them to ffmpeg to stitch together. But in the event that one of these files gets removed from the buffer after the name is processed but before ffmpeg actually processes it, the system falls apart. I may be over thinking this, and tried to mitigate by making sure the list of files is sorted so that the first video that the accessor processes would also be the one that's most likely to get thrown out by the buffer, but I'm not sure. Anyways, any advice in this regard would be much appreciated.

Will polling every 30 seconds be too hard on the system?

Risk assessment

Low. Only affects ABR-suite.

[rclarke0]

the blind spot is fine, you can remove these comments

[rclarke0]

remove this comment - code is self explanatory

[rclarke0]

how will you know if the deletion process is not working?

[munanwugo-opentrons]

This is a great point, I'll add something that says if there are more than like 40 files in the directory then the deletion isn't working and we should put something in the terminal to say that. Then when testing I can watch the terminal to see if anything pops up.

[rclarke0]

This will be super awesome to see once it is tested! Having a video of error recovery will be very helpful. I left some comments on the some of the code comments. In general, only using comments in the code when it is not self explanatory and a note on if the old video deletion process is not working.