In the video, I am using a webcam located on top of the main screen. Anytime the webcam sees the marker, a 3d cube is drawn in the camera space and rendered in the camera video feed on the main screen.

Everything in this demo is being done in browser using the AR.js library and newer HTML5 camera APIs.  I am using an iPad at the beginning of the video to show the marker image, but then towards the end I switched to a print out of the marker to make clear there is no technology component from the ipad.  The demo would be more compelling if shown using a phone/mobile device (which would use the camera as a pass-through to the display containing the 3D image), but my android device camera is dying. 

 The demo does not work with current iOS browsers, but it might be able to work in iOS 11.  UPDATE:[2017-10-21] Confirmed this does now work with iOS 11 Safari browser.

This video was done as a live youtube stream from a foscam pan/tilt webcam, streamed using VLC.

In this experiment, an RHT03 humidity and temperature sensor is connected to the Photon. There’s also a NeoPixel 8xRGB LED stick connected:

Photon breadboarded with RHT03 and RGB LED Stick

All programming, compiling, and updates were done using the browser based IDE:

Particle Build IDE

The particle SDK supports declaring certain variables and functions to be accessible from the particle web console. Particle provides a generic browser based UI to view these variables and functions, and the same functions are also accessible via a simple HTTP API for integration with other software.

Particle Console WebUI

In this case, the neopixel was connected for visual effect only. It is simply cycling through colors. The neopixel products are interesting because each LED can be individually addressed to display any RGB color and brightness.

Where’s my Internet connected toaster?

I started following Particle when it was still named “spark.io”, and I was fortunate enough to have a beer with their lead developer in San Francisco during their early days. I bought a photon soon after their first kick starter where they garnered ~$500k from backers. Since then they have released a similar cellular enabled device called Electron, making the platform accessible anywhere there’s cellular coverage. Very cool.

The Photon and Electron are great matches for Internet-of-Things (IoT) type ideas. IoT might be a buzzword, but even so, the related platforms and devices coming about to support IoT are opening up a whole new level of possibilities.

This was an ad hoc project using OpenCV to detect faces in real-time and drive a simple pan/tilt platform with a camera attached.

A built-in laptop webcam ran the face detection and controlled the pan/tilt servos. A separate webcam attached to the pan/tilt platform did the actual recording.

Adobe Media Encoder and YouTube Live (in beta) were used to stream the webcam while simultaneously recording it.  The delay between live and streaming was 20+ seconds, but a benefit was that it was watchable on all devices, including iOS.

Behind the Scenes

Resulting video from webcam: https://youtu.be/3FT468u_prM

To improve on this, servo movements could be smoothed and averaging applied to the face detection results.  Also, these particular servos were really noisy, especially when working to hold a position. In a real world scenario, a Dynamixel based pan/tilt platform might be quieter.

Additional Face Tracking Tests

These videos show experiments using OpenCV to track a face in a typical single performer stage performance. The green circles are from using a haar frontal classifier and the blue circles are from using a haar profile classifier.

• • opencv-test-008-good
    • a near best case scenario

• • opencv-test-011-bad
    • a near worst case scenario due to over exposed lighting, glasses, mic in front face, and frequent 3/4 face profile presentation.

This is my fourth iteration creating a pan/tilt video demo to test out new software ideas. The construction (and availability of the off-the-shelf components) has changed quite a bit since the days of version 1 & version 2. I don’t know what happened to version 3.

Photos

Video

The implementation is far enough along that I’ve tested end-to-end functionality through all the layers described below. The joystick controlled RGB LED project I wrote about previously is using this framework with all of the specific technologies and tools that I have baselined here.

Layers

(Graphic created on the iPad using Omnigraffle.)

Purpose

At first glance this framework might seem over engineered. However, I have thought a lot about the layers along with some of things I hope to accomplish eventually.  My constant goal has been to keep things as simple as possible, but no simpler.  Some of my ideas involve doing things that aren’t easily possible when using off-the-shelf components.  For example, I want to have enough control that I can custom program ease-in/out movements of servos (both RC Servos and Dynamixel Servos). Most of the servo controllers on the market do not have that capability. In earlier experiments I tried programming these custom movements in the PC layer using an SSC-32 for actual servo control, but the communications latency (RS232, buffers, microcontroller, etc) made the end result unreliable and not as smooth as I had hoped for.

Controller/Keyboard

A controller could be a joystick, a mouse, a Belkin Nostroma n52te, a computer keyboard, or any combination. The controller layer could also eventually include a head tracking device like the Natural Point TrackIR.

My current joystick is a Cyborg FLY5 Stick. It’s notable only because it has two independent throttle slide controls.

User Interface

The computer provides the GUI and all visuals. In all the other layers of this framework I have decided on specific technology and product choices for a baseline. However, I am still undecided as to which GUI framework/platform I want to use as a baseline. Options for the GUI layer include:

• JavaFX 2.0 (very different from JavaFX 1.0)
• Netbeans Platform (Swing. Not same thing as the Netbeans IDE.)
• HTML/JavaScript (possibly wrapped in QtWebkit Window)
• Qt Framework
• Flash/Flex/AIR
• Eclipse RCP

I keep holding off because I only have so much time to learn new things. The state of GUI development for business is a bit up-in-the-air presently.

Keeping the GUI as its own self-contained layer means it’s possible to make use of the remaining parts of this framework without any GUI at all. That’s what I’ve been doing mostly to-date because GUI’s take so much time to develop and generally require a clear goal.

Most recently I’ve been experimenting by creating a Java wrapper with an embedded Groovy engine. This lets me prototype console applications quickly using a dynamic language, but with all the power of the JDK and many available Java libraries.

High Level Logic on PC

Hobbyists, and even many professionals, seem to want to do as much as possible on the on-board computers or microcontrollers. I am guessing many of them come from an EE background. I am of the opposite mind though. I want to do as much as possible on the PC. My reasons for preferring to develop on PC vs. microcontroller:

• More advanced development tools available
• Often using same tools I already use for business app development
• More resources available (CPU and Memory)
• More language/platform options &
• Easier to test and debug.

The only reason not to do something on the PC is that it would be too slow. As it turns out in robotics, that is often the case, and so highly optimized routines and lower-level microcontroller programming is required. Still, being able to do as much of the high-level, non real-time, logic on the PC as possible opens up a lot of possibilities.

TCP/IP Server and Serial Port Proxy

The TCP/IP server is a custom C++ server that listens on a generic network socket for clients. Its primary purpose is to convert network messages to serial messages that get sent to the microcontroller and vice-a-versa.

The C++ server was written in cross-platform C++, making significant use of Boost.  Boost provides the cross-platform threading, network socket abstraction, and serial port abstraction.  (The network and serial port programming share the same paradigm in the  Boost::ASIO library).  Everything in Boost is impressive, but the ASIO library especially strikes me as being written by the elite of C++ craftsmen.)  If you want to get a feel for the type of programming involved, you can see snippets of my code in a question I posted to StackOverflow while working on this a while ago: “How should I delete a child from parent’s slot?“.

The C++ server is very generic in its base from.  As part of the initial setup protocol when a client connects (all of which is custom), the client instructs the server which serial port to open and provides port configuration parameters. The server is meant to be an executable piece of code requiring little to no configuration that can be run almost anywhere to provide network to serial conversion.  The ability to run it almost anywhere is important because it means that the server can be run on the PC workstation, or it can be run on an on-board computer (perhaps even one as small as a Linux Gumstix.)  C/C++ is not the easiest of languages, but it can be one of the most cross-platform in source code form.  After developing the C++ server, I came across a few open source projects that are essentially doing the same thing. (i.e. TCP/IP to Serial proxying). However, I didn’t see anything open source that could be easily made cross-platform or that looked as robust and as flexible as the server I developed myself using ASIO.

Instead of C++, I could have used Java. I have used RXTX for serial port access and Bluecove for Bluetooth access in Java successfully. However, there were minor issues with both. (RXTX wasn’t super reliable. Bluecove experienced random delays when connecting.) More importantly though, using Java would have severely limited options for running the serial port proxy on embedded devices.

Native Capabilities

Another benefit to using a custom C++ server is that it provides the option to write very low-level routines if needed. Most of the better GUI frameworks and platforms I’m familiar with (Java, Flex/Flash/AIR, Java) have limited capabilities for interacting with low-level OS functions and hardware directly. (Java provides JNI/JNA, but that would mean yet another conversion layer.) There are ways of integrating native C/C++ code with these platforms, but the techniques and/or libraries are specific to each platform. By providing a smartly designed network interface, the native interfacing layer is common and sharable with all the front-end platforms.

If needed, I can implement  C/C++ routines when and where needed, which can then be easily called by any client via the uniform network interface.  The network latency might be a problem for certain types of problems, but this is still a great addition to the framework’s capabilities.

Microcontroller

The microcontroller is where the programming hits the metal.  This type of programming is often referred to as embedded programming or realtime programming.

There are significant differences in the various types of microcontrollers and related programming distinctions, but these differences would seem arcane to anyone outside of the embedded space. Keep in mind that a company might be producing hundreds of thousands of devices with an embedded microcontroller and being able to select a microcontroller that costs even a few cents less at bulk quantities can be a significant cost difference. For this reason many microcontrollers are highly specialized and limited in their capabilities. Luckily, hobbyists don’t have to make the same trade-offs and the sorts-of microcontrollers I’m interested in tend to be more powerful and feature rich.

One of the more popular microcontroller platforms for hobbyists currently is the Aurdino.  However, I wanted something more powerful, so my baseline is a Parallax Propeller microcontroller. (Parallax is probably best known for creating the BasicStamp line of microcontrollers.)  The propellor is available in various forms. I started with the education kit (breadboard and 40-pin DIP) so I could learn the Propellor’s Spin language. I then purchased a PropStick USB:

PropStickUSB

This is a smaller version of the chip with supporting voltage regulators and USB interface already soldered on.  I put this onto a custom board where I soldered headers for connecting servos. The end result is quite powerful (for a microcontroller), easy to program (via USB), and yet is smaller than most dedicated servo controllers.

My custom board compared to SSC-32

The only technical difficulty I’ve run in to so far with the propellor that I probably wouldn’t have with the Arduino, is that the Propellor is 3.3v, whereas the Arduino is 5.0v.  Because of that I had to be careful with the power rails to the servos.  (I think I need to redo my current power connection setup regardless. I didn’t count on how just how much amperage could be needed if all servos are active at the same time.)

A key feature of the Propellor design is the notion of “Cogs”…a paradigm for multithreaded like programming:

Propeller Block Diagram courtesy of Parallax  (click to enlarge)

The Propellor design does not use interrupts which is a big difference as compared to many other microcontrollers.  The cog design is similar to having 8 CPUs each being given a slice of (real)time to operate with relatively safe access to shared resources.  I have used this capability to simplify the programming such that one cog handles communications to/from the serial connection, two more cogs are used to track and drive PWM signals to the connected servos.

On the robot/device there could be other components and addons, such as a dedicated servo controller like the Lynxmotion SSC-32, sensors, motor drivers, and other hardware interfacing components. The expectation as that my custom microcontroller brokers all of these devices to the higher level layers of the framework. Common protocols and related terms at this level include PWM, I2C, RS232, RS485, TTL, SPI, 2-wire, Relays, H-Bridges, MOSFET, ESCs, BECs, etc.

Next

I think the next step will be to use all of this to make my Lynxmotion BRAT derived chassis take some smoother steps. (Mentioned in an Alibre 3D case study previously.) Maybe. Or perhaps I’ll design a physical graphing apparatus so I that can accurately plot the motion curves of servos. Maybe. Or perhaps it’s time to start on that FPV RC Truck Platform with head tracking camera project I have had in mind. Maybe.

The PC reads the joystick position and communicates messages to a Parallax Propellor microcontroller. The microcontroller processes the messages and then controls the BlinkM RGB LED using it’s I2C interface.

Video

(I also created a much longer explanation video. It even has a background soundtrack. I finished it, watched it, and then realized it was so boring and inane that I would feel bad asking anyone to watch it.)

Photos

Description

This is a simple project for controlling a BlinkM RGB LED with joystick. The joystick is attached to a PC, which communicates the inputs to a Microcontroller with the BlinkM LED attached. This project is way over engineered for simply controlling an RGB LED. I’m using it as an early test for a larger architecture I’ve been working on for more advanced robotic control.

The joystick is a little different than most because it has two slider controls. I have the controls setup as follows:

• Left Slider: RED component
• Right Slider: GREEN component
• Stick Y-Axis: BLUE component
• Trigger Button: Decreases BLUE component by one-half

I did the video mainly to experiment shooting video in the dark.