Bricodash

Github Repository | A Beadsland Creation

An idlying chromecast is a bad chromecast. Cast a dashboard instead.

Bricodash is an informational dashboard displayed on an overhead TV in the main work area and on a small monitor mounted by the coat-racks next to the machine shop. The server back-end is housed on a black tower named bricolage—literally, "Something constructed using whatever was available at the time". See also: Add Events to Community Calendar Register Member Repositories with HM Cloud Log Future Ideas to be Implemented   Overview In October of 2017, Hack Manhattan installed a TV by the workspace tables. To make it easily accessible to everyone in the space, we added a Chromecast. To make that more useful, we're casting a "default" dashboard now. The same default dashboard is also displayed on a screen over the intercom at the entrance to our shop. This screen is driven by an old Raspberry Pi. Bricodash represents a major revision to last year's defaultcast, originally developed by mz, providing more functionality while using a thin-client architecture. If the early version Raspberry Pi we're using is low powered, the Chromecast is especially so. The goal has been to provide as much functionality as possible while keeping client load to a minimum. With this in mind, we rely heavily on a server-side cron hierarchy to update various HTML components, allowing the browser to poll for those files at appropriate intervals. At last count, Bricodash incorporates eight (8) different APIs. Design Philosophy I'm an old-school programmer, and prefer to avoid bloat and wall-of-text source files, in favor of clean, single-purpose, Small-is-Beautiful modules that do what they do with minimal byte-tonage. That said, the source files hacked together to get the system off the ground are just that, hacks. They're ugly and experimental and will hurt the eyes and brains of anyone accustomed to the regimented, framework-heavy best practices of contemporary Web design. Hic sunt dracones. This is not meant to be an example of professional, full-stack Web development. It is meant, instead, to demonstrate what can be accomplished when working without a net, allowing creativity and resourcefulness to drive the writing process, rather than convention and established practices. Moreover, that this system even uses a Web architecture is an artefact of the prototyping process. The ultimate intention—once all proposed features have been addressed—is to rewrite the entire system in Elixir to run as an embedded system on a single-board dongle that will drive its own display, rather that rely on Web protocols. Error Handling Each panel polls HTML files built periodically by back-end scripts on the bricodash server. Each file includes a timestamp tag indicating when the file was last generated by its respective script. If a script fails to complete successfully, the client will merely receive the last successfully written version when the file is next polled. As part of the polling step, each panel checks the age of the HTML file received against its polling interval. When age exceeds polling interval by a small multiple, the file is deemed stale and a yellow warning icon () is displayed in the center of the panel. Intermittent appearances of yellow warning icons can indicate network issues or excessively short polling cycles. Persistent indicators are a sign that the relevant back-end script should be executed at the console to determine why it might be failing. Calendar Errors The community and building calendars are compiled via a two-stage process. For each calendar data source, there is a back-end script that polls that source at an interval appropriate to its API and/or volatility. The results of these polls are written to json files for later use. A separate, master script then collates the contents of these variously updated JSON files to produce consolidated HTML files for display on the dashboard. The master script does not check for freshness of the intermediate JSON files. Rather, each successful run of the master script will show as having a fresh timestamp, regardless of the age of the source JSON files. No yellow warning icon will be shown if any of the data source scripts fail. Instead, a failing source script will become apparent if either the calendar is seen to be listing events that have passed days prior and/or fails to show current event listings that one would expect from the source. In either case, an error condition can then be confirmed by checking the system timestamps on all intermediate calendar JSON files. (It should be noted that there are some events, notably events that start at a specific time, that may continue to be listed on the day immediately following their start time. Usually these will be holidays on lunar or lunisolar calendars, astronomical events, or events that last beyond midnight. Such yesterday events do not represent errors or script failures, but rather are by design.) Panels The dashboard is comprised of five panels, or panes, and two overlays, described below starting clockwise from the top: Door Cam A cleaned up stream from our door camera graces the upper-right corner of the dashboard. The dashboard pulls an MJPG stream from a proxy gateway. Rather than relay the MJPG stream raw to our display devices, the gateway accepts sequential snapshot requests from the client devices, which are displayed by said client devices as a flipshow, allowing the chromecast and shoppi to control their own frame rates as a function of local load, rather than attempt to consume the firehose of a full mjpeg stream. This is done to avoid stalling out the Chromecast and Pi devices. Additionally, our script discards broken frames to reduce the number of grey-bottom artefacts appearing in the displayed feed. To be clear, a Chromecast seems perfectly capable of handling a full speed MJPG stream if that's all it's being asked to do. Once that MJPG feed begins sharing cycles with Javascript setInterval triggers, things get less stable. The more the Chromecast is asked to repeatedly do in addition to displaying the stream, the more that stream starts to look like a firehose. Eventually, the device freezes, becoming unresponsive to requests to stop the dashboard or start playing other content, and must by powercycled to get working again. Lowering the frame rate below full stream allows us to both stream a camera feed and actively update other content on the dashboard without overtaxing the Chromecast. NOTE: That all said, the current gateway architecture is to be replaced by BindSight—a dedicated MJPG consumer-broadcaster written in Elixir, now under development. The upper-right corner of the feed, in turn, displays regularly updated weather information, including wind speed and "feels like" temperatures, when appropriate. A time clock, meanwhile, is superimposed over the lower-right corner of the camera feed. Future Ideas Buzzer-Activated IR Remote Subway Status The lower-right corner of the dashboard displays subway line status information, as provided by our local transit authority. The widget is a variation of that intermittently displayed on the digital information kiosks now becoming common in the city. As provided, however, the transit authority's widget provides only for real-time status at time of page load. Thus, our back-end polls the widget, and caches the result for polled retrieval via a gateway script. This allows for any number of dashboard display devices to show minute-by-minute updates to subway status while keeping traffic to the information provider constant. Additionally, our back-end script strips live components unnecessary to a non-interactive display and munges colors in styles and images to conform to the rest of the dashboard. That all said, the script providing this functionality is currently an imperative hack, and could do with some refactoring. Future Ideas: The polling module is ripe for refactoring. Slack Channel Centered on the bottom half of the dashboard are recent posts to the #hackerspace channel on Hack Manhattan's Slack. This panel does not attempt to emulate the content layout of a Slack client. Instead, various strategies are used to concisely and crisply represent the content of the channel while making optimal use of the limited screen real estate available. Originally implemented via Slack's deprecated "Integration" mechanism, the back-end script for this panel has recently been migrated to a Slack's newer "App" model. The new model implements a finagley permissions system consisting of scopes and paired API keys (one for the app proper, one for a bot associated with the app), where certain methods available to one API context are not available to the other. At this time, it appears that the migration was successful. But it is possible that certain obscure features not regularly used will, when introduced into the channel stream, result in a missing_scope exception that will temporarily crash the back-end script, resulting in the display of yellow warning icon () within the panel. Future Ideas: Draft a Privacy Policy HM Cloud Log In the lower-left corner of the dashboard is a panel with a logographic heading that translates as "Hack Manhattan Cloud Write Scroll", i.e., "Hack Manhattan Cloud Log". This panel provides a running log of recent changes to the Hack Manhattan community's online activities. This includes changes to pages on this mediawiki server and to event listings on our Meetup group, as well as a activity on Hack Manhattan's Github repository. In addition to listing recent activity for all projects under the the github.com/hackmanhattan, members may optionally register their own repository services accounts via Bricodash/Extra:Repos. Meanwhile, a thumbnail gallery feature indicates whenever new images are uploaded to this mediawiki server or our Meetup group. (Integration of our Instragram and Twitter accounts is pending.) Future Ideas Zendesk — count of unread/open inquiries Status updates on donations: Patreon, Paypal, Amazon Smile... Instagram Integration Twitter Integration Event Calendars The upper-left hand side of the dashboard alternates between a calendar showing events of interest to the Hack Manhattan community and a calendar showing events occurring within the Rat Park building. Event listings are polled periodically from a number of different cloud services and data sources, each by way a source-specific script that extracts relevant event information then writes formatted and decorated output to an intermediary JSON file. A master script then regularly collates these files to generate three files, two HTML files for polling by the dashboard, and a third file formatted for posting to Slack. Community Calendar The community calendar draws events from the following sources: Meetup Hack Manhattan — Meetup Group Math for Math's Sake — Meetup Group (co-hosted by Hack Manhattan) Hacktoberfest — general event search (annual festivities around town) Google Calendar Babycastles — neighboring organization (with ties to our community) Hack Manhattan — private events at space (not listed on Meetup) Local Text Files birth.list — birthdays of notable makers, hackers, cultural figures and scientists geek.cal — various holidays mathy, sciencey, silly and makery local.cal — recurring events of regional interest usnat.cal — national public holidays and related events trivia.cal — tech anniversaries and natural phenomena multi.py — module for calculating multi-day, lunar, lunisolar and astronomical events (STILL UNDER DEVELOPMENT) Bricodash/Extra:Calendar — member-supplied events of interest Without line wrapping, the calendar panel has room for listing seven (7) upcoming events on the main screen. The smaller screen by the shop can display eight (8) lines of events, but as lines are shorter, lines wrap more often. To privilege events happening at Hack Manhattan's space, the number of non-space events are capped at four (4). Within this constrained line count, more trivial sources are further filtered to only appear on the day of or immediately before, to avoid (somewhat) crowding out of more salient upcoming events like national holidays. Future Ideas flesh out documentation now that community and building have been broken out into separate calendars various community calendar updates in Bricodash/Future#Current_Tasks_Outstanding Draft a Privacy Policy Building Calendar Although relevant to our community, the Rat Park calendar is not of primary interest. For this reason, the Building Calendar is displayed only one out of every five minutes, just long enough to get a general glance of events happening in the building. The event calendar draws events from the following sources: Meetup Hack Manhattan — Meetup Group Math for Math's Sake — Meetup Group (co-hosted by Hack Manhattan) 137 W 14th Street — general search (non-HM events hosted in our building) Google Calendar Hack Manhattan — private events at space (not listed on Meetup) Eventbrite Secret Loft — DIY event space that shares our floor Offside Tavern — sports bar on building's ground floor autogenerated Offside Tavern — one recurring event listed on their Website but not otherwise pollable Events hosted by Secret Loft and Offside Tavern more often than not involve loud music, and in the case of some Secret Loft events, heavy use of the hallway leading to our space. The listings for these events are formatted and decorated to indicate the possibility of noise. In addition, Eventbrite's API license requires that event listings obtained from their service include a link to their event page on Eventbrite's site. As our dashboard is non-interactive, a URL shortener is used to provide a useable link as a superscript following each such listing. Meanwhile, Offside Tavern often serves as a venue for various Meetup Groups, more often than not in some manner tech-related. Future Ideas: Draft a Privacy Policy Slack Calendar The remaining calendar is generated by combining all of the above sources, and stripping away much of the formatting and decoration used in the dashboard's HTML panels. Twice a day, the Bricodash bot posts all events from both of the above calendars due to occur in the next 72 hours to the channel #thehaps on our Slack, first deleting its prior post of the same. Also planned are Slack Events API bot functionality, such that the same calendar data can feed bah-style responses to user questions like "What's happening?" Future Ideas: Slack Calendar Response Bot Creepy Eyeball The same feature that supports the dashboard door camera feed also allows for viewing Hack Manhattan's Space Cam. When viewed access the MJPG feed through the dashboard's bricolage gateway, an animated GIF of a disembodied eyeball appears as an overlay on the upper right corner of the Calendar panel. Unlike the bo.x0.rs-hosted sousveillance system that inspired it, the Bricodash sous/veil system tracks both the connect and disconnect time of live streams. The eyeball animation appears and fades quickly for snapshots, appears and fades slowly for live streams while they are active, and then fades away more swiftly after a disconnect. Multiple simultaneous feeds have an additive effect, such that the eyeball will remain opaque longer and fade less rapidly, than for a single feed. It should be noted that the Creepy Eyeball is only activated when our Space Cam is viewed via the Bricodash-bundled gateway. Snapshots and streams accessed via the bo.x0.rs gateway—such as link provided for viewing the Space Cam via VLC Media Player, or snapshots provided by the "Who's at the space?" trigger on our Slack—do not register as Creepy Eyeball activity. Future Ideas MJPEG Gateway Server Full conversion of former sousveillance gateway references to sous/veil gateway would be ideal. Weird Photos At random intervals, a photo overlay will fade-zoom out from the center-left of the dashboard, remain briefly, and then fade--zoom back toward the center and disappear. Due to limited screen real-estate, there is no ready way to share photos related to our space on the dashboard. This may change once an door buzzer IR remote solution is implemented. Until then, the Weird Photos mechanism is a compromise solution. Photos are displayed long enough to grab attention, but not so long as to obscure calendar or slack panel content for a prolonged period. Further, photos, when they do fade-zoom into view, are offset to the left to avoid obscuring the Door Cam feed. Presently, photos are only drawn from the 100 most recently posted images on our Meetup group. (Instagram integration is pending.) One of our members was conducting a year-long project involving representing holidays and notable events with tableaus involving rubber chickens and rubber ducks—often blue screened into appropriate settings. Many of these photos were posted to the albums for then-upcoming Open House events on the Meetup Group, hence the prevalence of rubber fowl in many of the photos that fade-zoom into and out of view on our dashboard. Future Ideas Instagram Integration Twitter Integration   Caveats and Policies Bricodash incorporates features from a wide variety of APIs, many of which have specific requirements for use under their respective licenses. In particular, some of the APIs used by Bricodash require that any public Web site that integrates with each such API must have a published privacy policy congruent with that governing such API's Web service. At present, no such privacy policy has been created for Bricodash. Therefore, to remain in compliance with those licenses, Bricodash is only to be used for the on-site displays within Hack Manhattan, and not displayed on any public Web site. (At least until such time as an appropriate privacy policy has been published for Bricodash). Future Ideas: Draft a Privacy Policy			Bricodash screenshot
			Pending Tasks Performance and Maintenance Fixes fix early voting -- comma not second plus sign resolve v3/events/search failure (email reply from Eventbrite API team 22 Oct: solution being explored) explore having bricodash spawn in new tab then close old tab -- to force GC of JS code snippets Repository recreate repository as non-fork readme link to defaultcast research and select a code of conduct flesh out issues linking to project plan Long Overdue Calendarical Overhaul Reverse Engineer FCNA/ECFR Calculations relearn trigonometry refactor astronomical month calculations to new repository see if we can solve sunrise (sunset) equation for pair(?) of latitudes on a given meridian (at a given datetime) see if any altitude delta patterns are apparent with coarse (non-continuous) iteration over enlongation-bounded datetime range work out if we can iteratively narrow down latitude pairs to solve for altitude deltas within intersecting sunsets determine how elevation complicates all of this (i.e., intersecting sunset clusters rather than intersecting sunsets) examine if there's a solution for westward obstructions (e.g. mountain ranges) in determining sunsets explore performance gains of reimplementing search traversal via elixir calls to python calculations Islamic Calendar refactor islamic calendarical library to new repository incorporate for astronomical calculation as used by some in North America confirm safe astimezone use (not erroneously masking local time as UTC) refactor to combine observational and astronomical date sets add Eid al-Adha to calendar Hindu Calendar refactor hindu calendrical library to new repository new module to calculate enumerations of tithi for a given month new module to calculate enumerations of months for a given year (rather than lisp ordinal month) reimplement lisp functions for hindu calendar in python3 Calendrical Umbrella spin off jewish calendarical library as dedicated project spin off master festivals project for drawing together specific calendars BindSight :2020 See also: BindSight project page Concurrent, extensible, frame-scrubbing webcam gateway. Web API service to stream doorcam and spacecam to Bricodash and public gateway, respectively, while tracking activity and performance of these and other webcams at the space. Will be more efficient and reliable than spawning PHP and Python processes on an as-they-come basis. Written in Elixir, will be taking advantage of various new features of the language, building on the strengths of Erlang/OTP, including Mint (web client), GenStage (backpressure event pipelines) and ultimately mix release (build-time deployment packaging). Fault Tolerance refactor spigot as behaviour insert exit monitor into children exit monitor crashes on exit received; supervisor strategy of rest_on_one webapi exit monitor reconnects to new spew and keeps running Refactoring no reason to split on doubledash reconfigure profile to run when profile_seconds > 0 refactor adhere/hold to own Adhere genstage rebalance chunk load to other genstages digest: refactor handle_events/3 case statement to function clauses digest: drop EOL determination digest: refactor further if possible Error Recovery deferred spigot spinup in event camera offline at startlink fix EOL check to verify in case of dropped bytes test logging on down camera don't nag when connection down (periodic warn) ensure recovery when source stream fails Validation review whether batch requires task consumer-producer in spew to filter out :corrupt/:greytoss/fail messages retool Validate stage to send tuple {:ok, binary} or {_status, binary} retool Digest stage to sent tuple {:ok, binary} or {:fail, error} retool polling functions to send tuple {:fail, error} when such occur consumer-producer in slurp to introduce :corrupt/:greytoss on :test stream Deep Validation implement greytoss checking replace async batching with task/agent batching, for JIT evaluation watermark with source timestamp configurable agpl watermark provide for timeout watermarking Advanced review supervision tree for batch tasks & agents review memory usage to mitigate ProcBin leaks integrate certificate used by Apache configure to launch as daemon bootstrap to obtain dependencies and compile cold Robustness type guards on get_env @impl on all OTP callbacks type guards all public functions type checks on unstructured opts typespecs and dialyze throughout Subsystem Integration slurp snoop to track performance on each camera slurp snoop to webhook on corruption/timeouts spew snoop to swap out CGI for upt/chk touch points spew snoop to track client fps performance Sousveillance Integration spew snoop to swap out PHP for sous veil touch points distributed sous veil client for cross-platform data exchange integration with sousveillance watch-back system Wrap-up add robots.txt route configure public ipcam as :test flesh out documentation investigate periodic multi-hit events: server severing connections prematurely? badly behaved client device? investigate specific issue of pishop launch causing multi-hit failures on all other clients stress test explore sobelow security checking Webhook Notifications Javascript client-side triggers to deliver messages to our Slack channels to provide alerts when the camera feed or client devices stall. migrate sleep macro to general purpose library migrate from webhooks integration to webhooks feature of bricodash app (and update notice) server-side webhook on stall of any given device when wonkiness addressed: downthrottle chromecast reboots to 1x per day; stream swap-in to never Server Standardize Web server setup and fully integrate BindSight. migrate to standard web port (now that relayd installed on router) configure X-Forwarded-For logging in apache enable SSL for camera feeds (requested for full conversion to new sous veil system) enable SSL directly via daemon's listening port (i.e., elixir rather than apache) transfer slackbot snapshots to bricolage gateway Documentation Flesh out project documentation—after fixing existing problems with our Mediawiki server. recent edits monitor to trigger cache purge on transcluded Bricodash documentation updates help get mediawiki server 504 issues resolved (currently an obstacle to Bricodash documentation) broach discussion of absent Copyrights page on Wiki -- do we want to adopt a license?? redraft lists in calendar sections as tables flesh out Bricodash Future Ideas, explaining known requirements for each Cloud Log Incorporate remaining sources into rolling update log shown in lower left-hand corner. User Repositories refactor cloud/github to abstract away github/gitlab/etc allow more than most recent pushed repository to be shown add boxrs-based git repositories to Cloud Log panel Other Cloud Services Mediawiki — pending new account requests Zendesk — count of unread/open inquiries Status updates on donations: Patreon, Paypal, Amazon Smile... write up requirements for Instagram Integration write up requirements for Twitter Integration Retire IFFF Wiki Updates bot to post daily cloud log update to #general on our slack Calendars Incorporate additional calculated and externally sourced calendar events. Fixes almanac alt for almanac events add enum of event sources for sorting priority (when same start time) move HM anniversary dates to special .cal that takes precedence on HM calendar (and also appears on NYC) swap out icalendar (buggy repetition exception handling) in favor of true API Calculation new module to calculate hindu lunar tithi => closing muhurat => solar day revise diwali/deepawali calculation using tithi rather than lisp library approximations correctly calculate dates of International Astronomy Day (email out to Astronomical League for better calculation rules) calculate dates of Manhattanhenge (email out to library of American Museum of Natural History for formula) add eclipses and other significant astronomical events to holidays finish fleshing out remainder of movable feasts for Community Calendar (June 2019 forward) Integration use google search to find Offside Tavern Meetup events add placeholder events for non-Meetup Offside Tavern events refactor brite search by location routine to common library add nyc mesh, nyc resistor, fat cat fab lab, makerbar, makerspace nyc (SI), futureworks (BK/SI), genspace to Community Calendar also add grow with google (if still in NYC), 2600, Off the Hook (if still broadcasting), blick, michaels, local library branches, makerish meetup events, NYC street fairs bot to post RSVPs for upcoming events to Slack Advanced revise #thehaps formatting to provide headers per day embed hotlinks in #thehaps channel event listings concise calendar response for interactive slackbot auto-capture of #thehaps posted events to Bricodash/Extra:Calendar automation/streamlining tools for cross-posting HM Meetup events to Spingo enable toggling between Welcome header and holiday greetings ^ Cleanup Remaining features and code maintenance before calling it a major release. integrate citymapper API for local subway stations refactor source files from initial launch find owners to adopt data sourcing projects that require more than passive API calls (see sections below) squish and push fork to upstream Upgrades Once built, streamline and rewrite as a truly beautiful application. migrate from bricolage to less power-hungry hardware combined community slack calendar to wiki page (requires privacy policy under brite license) reimplement bricolage to run as embedded system under elixir/scenic/nerves explore future development ideas