Difference between revisions of "BindSight"

From Hack Manhattan Wiki
Jump to: navigation, search
(Spew Spigot)
(Current Project Plan)
(One intermediate revision by the same user not shown)
Line 63: Line 63:
  
 
<onlyinclude>
 
<onlyinclude>
 +
 +
==== Refine ====
 +
* inform match only >2
 +
* refactor get_env to common
 +
* revise get_register_name to use list
 +
* common spigot behavior
 +
 +
==== Spew Prefix Reservation ====
 +
* prefix reservation GenServer
 +
* pull name digit count from config
 +
* reservation stash and recovery cycle
 +
* start_link checkin method called from spigot
 +
* release name from reservation on linked exit
  
 
==== Spew Spigot ====
 
==== Spew Spigot ====
* spew spigots don't need names -- use agent to store tap pid
 
 
* unit tests for spew spigot
 
* unit tests for spew spigot
 
* broadcast consumer-producer to make room for snoops
 
* broadcast consumer-producer to make room for snoops

Revision as of 16:01, 11 August 2019

Github icon.jpg Github Repository | Bricodash screenshot.png A Bricodash Sub-Project | Beads Land-Trujillo.jpg A Beadsland Creation

Concurrent frame-scrubbing webcam broadcast gateway daemon.

Devoted service to stream doorcam and spacecam to Bricodash and public gateway, respectively, while tracking activity and performance of these and other webcams at the space. Will be more efficient and reliable than spawning PHP and Python processes on an as-they-come basis.

Written in Elixir, will be taking advantage of various new features of the language, building on the strengths of Erlang/OTP, including Mint (web client), GenStage (backpressure event pipelines) and ultimately mix release (build-time deployment packaging).

Background

Presently, webcam-related features of Bricodash are provided on a two-prong basis.

Spacecam

The public-facing Camera showing current activity in the space is presently relayed via a PHP script. This script is configured to relay various HM cameras, including those for the exterior door, the CR-10 and (when online) the hydroponics lab. In the case of the spacecam, all relayed requests are tracked to feed the sous veil Creepy Eyeball: snapshot requests touch of a file identifying the requesting pid; stream requests trigger touch two files, one on initiation of the relay, one at process termination. The Creepy Eyeball feature uses the modification times on all such files to transition the opacity of the eyeball animated gif that indicates observers are viewing the camera feed.

The Apache server hosting the Bricodash back-end imposes a cap on the number of executing PHP processes, returning a 504 Gateway Timeout error when this cap is exceeded. Under normal usage, this isn't an issue, but PHP scripts are meant to assemble dynamic content and deliver it, not to provide a data stream as a continuously running process of indeterminate duration. The current setup is thus not ideal.

A legacy gateway for the same Camera remains on Bo.x0.rs. When users of our Slack type "Who's at the space?", the snapshot that our resident slackbot posts in response is drawn from the Bo.x0.rs gateway. Bo.x0.rs also tracks camera access, exposing it via the sousveillance Chrome notification mechanism. However, the sous veil Creepy Eyeball and the sousveillance Chrome notifications system do not, at present, interact or share information.

Doorcam

Bricodash displays a feed of the exterior door camera. Rather than the PHP script, the doorcam display is drawn from a Python3 CGI script.

Our current setup for the door camera feed is prone to stalling and data corruption. The CGI script inspects individual frames, discarding those that are revealed to be invalid JPG format or, while valid, have corruption that renders as a grey field on the lower half of the image. A Slack webhook regime has also been explored for tracking stalling of the feed. However, even after dropping corrupt frames between the webcam and the server, additional frame corruption has appeared between server and client. Thus client-side Javascript has also been implemented to drop bad frames and trigger webhooks on stalls. Performance of the feed was such that the webhooks proved too noisy and so it is currently disabled.

Due to hardware limitation of both Bricodash clients, the standard webcam MJPEG push stream proved too much of a firehose to accept without risk of the client lagging or freezing entirely. For this reason, the camera feed on both clients is implemented not as a stream but a throttled frames-per-second snapshot flipbook. Initially, this was managed by the CGI script. However, with the introduction of Javascript-invoked Slack webhooks to detect stalls and corruption client-side, the feed is now an entirely pull-based snapshot flipbook.

These flipbooks are executed as upward to 10 snapshot requests per second, from each client, all of which are relayed directly to the webcam. It is possible that this has lead to greater instability on the external door camera that could be resolved by extracting frames from a single stream rather than pounding the server with near two dozen requests per second.

The CGI script also tracks delays on flipbook snapshot requests from the Bricodash clients, reading such delays as evidence of a stalled client. This triggers a restart of the Bricodash client on the Chromecast, and would trigger a webhook alert for the pishop client (if webhooks were not currently disabled). This trigger does not distinguish between delays in flipbook snapshot requests due to stalls of the client and delays due to timeouts on the client while waiting for a stalled doorcam feed. Which means that stalls on the webcam can presently result is a restart of the Chromecast that is not, itself, stalled.

Functional Specification

BindSight will replace both the PHP and Python gateways, and will be designed to allow for integration between the sous veil Creepy Eyeball observer-tracking system and the Bo.x0.rs sousveillance view-tracking and Chrome notifications system.

Features will include:

  • a GenStage pipeline to broadcast from a single stream from each camera source
  • concurrent frame corruption checking to minimize latency
  • high performance under stream and flipbook stress testing
  • plug-in architecture for performance monitoring and notifications
  • running as an independent system service (daemon)
  • configurable support for using domain-specific SSL cert
  • distributed plug-in for integration with Bo.x0.rs

BindSight will leverage Elixir's [GenStage] behavior to convert a single MJPEG stream into a series of frames that will then be served as any number of snapshots or reconstituted MJPEG streams on demand. This will be implemented as two half-pipelines, or spigots. For each webcam, as slurpspigot will consume an MJPEG stream, releasing it as individual frames, initiate frame corruption checking, track camera performance, and broadcast to sinks. For each client request, a spewspigot will subscribe to the appropriate slurpspigot, scrub out corrupt frames, watermark the last available image in the event of a camera timeout, and serve the resulting snapshot or stream via HTTPS.

Custom plug-ins (including several local plug-ins) will be installable per camera (slurpspigot) or per client/request (spewspigot). These will operate out-of-band as GenStage processes running concurrently with the throughput of the main pipeline. Likewise, frame corruption checks will run asynchronously, allowing frames to be passed down the pipeline independent of JPEG decompression and pixel inspection. Concurrency is the core strength of the Erlang virtual machine ([BEAM]), which is designed to obtain exceptional performance by spinning up large numbers of lightweight threads.

Erlang, and thus Elixir, is also designed for distributed systems. We will take advantage of this for integration with Bo.x0.rs, relying on Erlang's native node-discovery to manage message passing between souseveillance and sous veil, despite these systems running on different hardware.

Current Project Plan

Refine

  • inform match only >2
  • refactor get_env to common
  • revise get_register_name to use list
  • common spigot behavior

Spew Prefix Reservation

  • prefix reservation GenServer
  • pull name digit count from config
  • reservation stash and recovery cycle
  • start_link checkin method called from spigot
  • release name from reservation on linked exit

Spew Spigot

  • unit tests for spew spigot
  • broadcast consumer-producer to make room for snoops
  • ripcord consumer-producer that dies on unsubscribe

Validation

  • retool Validate stage to send tuple {:ok, binary} or {_status, binary}
  • consumer-producer in spew to filter out :corrupt/:greytoss messages
  • consumer-producer in slurp to introduce :corrupt/:greytoss on :test stream

Web Consumer

  • snoop supervisor component for both slurp and spew
  • slurp snoop to flush broadcast
  • web consumer as filter-spigot Task
  • web consumer poolboy

Streaming

  • implement stream as snapshot flipshow
  • ensure recovery when flipshow fails
  • reimplement to consume source stream
  • ensure recovery when source stream fails

Advanced

  • provide for timeout watermarking
  • implement greytoss checking
  • replace async batching with task/agent batching, for JIT evaluation
  • review supervision tree for batch tasks & agents
  • integrate certificate used by Apache
  • configure to launch as daemon
  • bootstrap to obtain dependencies and compile cold

Subsystem Integration

  • spin off as dedicated project
  • slurp snoop to track performance on each camera
  • slurp snoop to webhook on corruption/timeouts
  • spew snoop to swap out CGI for upt/chk touch points
  • spew snoop to swap out PHP for sous veil touch points
  • distributed sous veil client for cross-platform data exchange
  • integration with sousveillance watch-back system

Wrap-up

  • add robots.txt route
  • configure public ipcam as :test
  • flesh out documentation
  • investigate periodic multi-hit events: server severing connections prematurely? badly behaved client device?
  • investigate specific issue of pishop launch causing multi-hit failures on all other clients
  • stress test