Difference between revisions of "BindSight"

From Hack Manhattan Wiki
Jump to: navigation, search
(Current Project Plan)
Line 63: Line 63:
==== Oroborous ====
* 404 misreports at :test after correctly reporting twice
* try to get mint to log/inspect while eating own tail
==== Firefox ====
==== Firefox ====

Revision as of 04:33, 29 August 2019

Github icon.jpg Github Repository | Bricodash screenshot.png A Bricodash Sub-Project | Beads Land-Trujillo.jpg A Beadsland Creation

Concurrent, extensible, frame-scrubbing webcam gateway.

Web API service to stream doorcam and spacecam to Bricodash and public gateway, respectively, while tracking activity and performance of these and other webcams at the space. Will be more efficient and reliable than spawning PHP and Python processes on an as-they-come basis.

Written in Elixir, will be taking advantage of various new features of the language, building on the strengths of Erlang/OTP, including Mint (web client), GenStage (backpressure event pipelines) and ultimately mix release (build-time deployment packaging).


Presently, webcam-related features of Bricodash are provided on a two-prong basis.


The public-facing Camera showing current activity in the space is presently relayed via a PHP script. This script is configured to relay various HM cameras, including those for the exterior door, the CR-10 and (when online) the hydroponics lab. In the case of the spacecam, all relayed requests are tracked to feed the sous veil Creepy Eyeball: snapshot requests touch a file identifying the requesting pid; stream requests trigger touch two files, one on initiation of the relay, one at process termination. The Creepy Eyeball feature uses the modification times on all such files to transition the opacity of the eyeball animated gif that indicates observers are viewing the camera feed.

The Apache server hosting the Bricodash back-end imposes a cap on the number of executing PHP processes, returning a 504 Gateway Timeout error when this cap is exceeded. Under normal usage, this isn't an issue, but PHP scripts are meant to assemble dynamic content and deliver it, not to provide a data stream as a continuously running process of indeterminate duration. The current setup is thus not ideal.

A legacy gateway for the same Camera remains on Bo.x0.rs. When users of our Slack type "Who's at the space?", the snapshot that our resident slackbot posts in response is drawn from the Bo.x0.rs gateway. Bo.x0.rs also tracks camera access, exposing it via the sousveillance Chrome notification mechanism. However, the sous veil Creepy Eyeball and the sousveillance Chrome notifications system do not, at present, interact or share information.


Bricodash displays a feed of the exterior door camera. Rather than the PHP script, the doorcam display is drawn from a Python3 CGI script.

Our current setup for the door camera feed is prone to stalling and data corruption. The CGI script inspects individual frames, discarding those that are revealed to be invalid JPG format or, while valid, have corruption that renders as a grey field on the lower half of the image. A Slack webhook regime has also been explored for tracking stalling of the feed. However, even after dropping corrupt frames between the webcam and the server, additional frame corruption has appeared between server and client. Thus client-side Javascript has also been implemented to drop bad frames and trigger webhooks on stalls. Performance of the feed was such that the webhooks proved too noisy and so it is currently disabled.

Due to hardware limitation of both Bricodash clients, the standard webcam MJPEG push stream proved too much of a firehose to accept without risk of the client lagging or freezing entirely. For this reason, the camera feed on both clients is implemented not as a stream but a throttled frames-per-second snapshot flipbook. Initially, this was managed by the CGI script. However, with the introduction of Javascript-invoked Slack webhooks to detect stalls and corruption client-side, the feed is now an entirely pull-based snapshot flipbook.

These flipbooks are executed as upward to 10 snapshot requests per second, from each client, all of which are relayed directly to the webcam. It is possible that this has lead to greater instability on the external door camera that could be resolved by extracting frames from a single stream rather than pounding the server with near two dozen requests per second.

The CGI script also tracks delays on flipbook snapshot requests from the Bricodash clients, reading such delays as evidence of a stalled client. This triggers a restart of the Bricodash client on the Chromecast, and would trigger a webhook alert for the pishop client (if webhooks were not currently disabled). This trigger does not distinguish between delays in flipbook snapshot requests due to stalls of the client and delays due to timeouts on the client while waiting for a stalled doorcam feed. Which means that stalls on the webcam can presently result is a restart of the Chromecast that is not, itself, stalled.

Functional Specification

BindSight will replace both the PHP and Python gateways, and will be designed to allow for integration between the sous veil Creepy Eyeball observer-tracking system and the Bo.x0.rs sousveillance view-tracking and Chrome notifications system.

Features will include:

  • a GenStage pipeline to broadcast from a single stream from each camera source
  • concurrent frame corruption checking to minimize latency
  • high performance under stream and flipbook stress testing
  • plug-in architecture for performance monitoring and notifications
  • running as an independent system service (daemon)
  • configurable support for using domain-specific SSL cert
  • distributed plug-in for integration with Bo.x0.rs

BindSight will leverage Elixir's [GenStage] behavior to convert a single MJPEG stream into a series of frames that will then be served as any number of snapshots or reconstituted MJPEG streams on demand. This will be implemented as two half-pipelines, or spigots. For each webcam, as slurpspigot will consume an MJPEG stream, releasing it as individual frames, initiate frame corruption checking, track camera performance, and broadcast to sinks. For each client request, a spewspigot will subscribe to the appropriate slurpspigot, scrub out corrupt frames, watermark the last available image in the event of a camera timeout, and serve the resulting snapshot or stream via HTTPS.

Custom plug-ins (including several local plug-ins) will be installable per camera (slurpspigot) or per client/request (spewspigot). These will operate out-of-band as GenStage processes running concurrently with the throughput of the main pipeline. Likewise, frame corruption checks will run asynchronously, allowing frames to be passed down the pipeline independent of JPEG decompression and pixel inspection. Concurrency is the core strength of the Erlang virtual machine ([BEAM]), which is designed to obtain exceptional performance by spinning up large numbers of lightweight threads.

Erlang, and thus Elixir, is also designed for distributed systems. We will take advantage of this for integration with Bo.x0.rs, relying on Erlang's native node-discovery to manage message passing between souseveillance and sous veil, despite these systems running on different hardware.

Current Project Plan


  • 404 misreports at :test after correctly reporting twice
  • try to get mint to log/inspect while eating own tail


  • test if conforming to non-standard EOLs fixes problem
  • figure out way to compare mjpg-streamer and bindsight wireshark activity
  • firefox appears to have memory leak on bindsight but not other mjpeg sources
  • determine why firefox has flicker/decentering issues when viewing stream via bindsight

Mint/Cowboy Interaction

  • recast all shortnames with bindsight prefix
  • test how major browsers other than chrome handle extraneous charset declaration following boundary string definition
  • try to fix mint failure by double quoting boundary string (per RFC1341) to demarcate as separate from ; and following postfix charset
  • submit issue concerning mint failure to flush data_buffer when multipart content-type boundary designation doesn't terminate at EOL
  • figure out why mint doesn't flush data_buffer if reading from cowboy (reading code suggests it always would)

Code Cleanup

  • investigate doing per-connection idle timeouts
  • refactor side-by-side concatenations to bulk joins
  • rebalance chunk load to other genstages
  • chunk: clean up handle_data pipe chain
  • digest: refactor and clean up

Error Recovery

  • confirm LFCR is result of non-compliance by mjpg-streamer
  • fix EOL check to verify in case of dropped bytes
  • test logging on down camera
  • don't nag when connection down (periodic warn)
  • ensure recovery when source stream fails


  • review whether batch requires task
  • consumer-producer in spew to filter out :corrupt/:greytoss/fail messages
  • retool Validate stage to send tuple {:ok, binary} or {_status, binary}
  • retool Digest stage to sent tuple {:ok, binary} or {:fail, error}
  • retool polling functions to send tuple {:fail, error} when such occur
  • consumer-producer in slurp to introduce :corrupt/:greytoss on :test stream

Deep Validation

  • implement greytoss checking
  • replace async batching with task/agent batching, for JIT evaluation
  • provide for timeout watermarking


  • review supervision tree for batch tasks & agents
  • review memory usage to mitigate ProcBin leaks
  • integrate certificate used by Apache
  • configure to launch as daemon
  • bootstrap to obtain dependencies and compile cold


  • type guards on get_env
  • @impl on all OTP callbacks
  • type guards all public functions
  • type checks on unstructured opts
  • typespecs and dialyze throughout

Subsystem Integration

  • slurp snoop to track performance on each camera
  • slurp snoop to webhook on corruption/timeouts
  • spew snoop to swap out CGI for upt/chk touch points
  • spew snoop to track client fps performance

Sousveillance Integration

  • spew snoop to swap out PHP for sous veil touch points
  • distributed sous veil client for cross-platform data exchange
  • integration with sousveillance watch-back system


  • add robots.txt route
  • configure public ipcam as :test
  • flesh out documentation
  • investigate periodic multi-hit events: server severing connections prematurely? badly behaved client device?
  • investigate specific issue of pishop launch causing multi-hit failures on all other clients
  • stress test