In this experiment, an RHT03 humidity and temperature sensor is connected to the Photon. There’s also a NeoPixel 8xRGB LED stick connected:

Photon breadboarded with RHT03 and RGB LED Stick

All programming, compiling, and updates were done using the browser based IDE:

Particle Build IDE

The particle SDK supports declaring certain variables and functions to be accessible from the particle web console. Particle provides a generic browser based UI to view these variables and functions, and the same functions are also accessible via a simple HTTP API for integration with other software.

Particle Console WebUI

In this case, the neopixel was connected for visual effect only. It is simply cycling through colors. The neopixel products are interesting because each LED can be individually addressed to display any RGB color and brightness.

Where’s my Internet connected toaster?

I started following Particle when it was still named “spark.io”, and I was fortunate enough to have a beer with their lead developer in San Francisco during their early days. I bought a photon soon after their first kick starter where they garnered ~$500k from backers. Since then they have released a similar cellular enabled device called Electron, making the platform accessible anywhere there’s cellular coverage. Very cool.

The Photon and Electron are great matches for Internet-of-Things (IoT) type ideas. IoT might be a buzzword, but even so, the related platforms and devices coming about to support IoT are opening up a whole new level of possibilities.

This was an ad hoc project using OpenCV to detect faces in real-time and drive a simple pan/tilt platform with a camera attached.

A built-in laptop webcam ran the face detection and controlled the pan/tilt servos. A separate webcam attached to the pan/tilt platform did the actual recording.

Adobe Media Encoder and YouTube Live (in beta) were used to stream the webcam while simultaneously recording it.  The delay between live and streaming was 20+ seconds, but a benefit was that it was watchable on all devices, including iOS.

Behind the Scenes

Resulting video from webcam: https://youtu.be/3FT468u_prM

To improve on this, servo movements could be smoothed and averaging applied to the face detection results.  Also, these particular servos were really noisy, especially when working to hold a position. In a real world scenario, a Dynamixel based pan/tilt platform might be quieter.

Additional Face Tracking Tests

These videos show experiments using OpenCV to track a face in a typical single performer stage performance. The green circles are from using a haar frontal classifier and the blue circles are from using a haar profile classifier.

• • opencv-test-008-good
    • a near best case scenario

• • opencv-test-011-bad
    • a near worst case scenario due to over exposed lighting, glasses, mic in front face, and frequent 3/4 face profile presentation.

This post describes my opinion as to the best platform for new web application projects. And while “best” is nebulous, I mean it with some of the following goals in mind:

• quick to develop with, yet robust
• does not place arbitrary restrictions on application capabilities
• does not hold things back when needing to scale
• maintainable over time, by one or many programmers
• has decent community and future outlook
• supports building all kinds of cool things

Before I continue, let me be clear about something. The best platform choice (out of a selection of good platforms) is usually going to be the platform for which you and/or your team have the most experience.  That fact alone trumps almost every benefit I’ll describe below in my choices and often tends to get lost in the “my platform is better than yours” discussions. Most of the good platforms can be used to build most any web system that can be imagined.  This post answers the question, “If ignoring existing experience and time-to-learn, what would be the best platform?”.

Baseline

I have been fortunate as a mostly independent software developer in that I’ve been able to work with a number of companies and technology platforms.   When I build my own web application projects though, these are the major components that I would chose:  (as of January 2015):

• Backend
  • Java 8+
  • Sprint Boot
    • Spring Framework 4+
      • Spring MVC
      • Spring Data/JPA
      • Spring Security
    • Embedded Tomcat 8
  • JPA2 with Hibernate 4+
  • PostgreSQL
  • Maven 3+
  • Freemarker (if backend html/xml generation is necessary)
  • Jackson (for JSON in/out on a Restful API)
  • Spring WebSockets (if async is required)

•  Frontend
  • HTML5/CSS3/JavaScript
  • BootStrap or Foundation (or for mobile hybrid, Ionic Framework)
  • AngularJS  (still debating this choice because I find AngularJS very problematic)
  • Gulp, Bower

• Tools
  • IntelliJ
  • JRebel for hot reloading
  • Vagrant for development environment services

• Deployment and Hosting
  • Amazon Web Services, with configuration via OpsWorks or CloudFormation
    • Or possibly with Docker via new EC2 Container Service
  • For server provisioning, still undecided, but probably Chef or Ansible.

For anyone sharing my way of thinking, be sure to check out the jHipster project. It combines all of this and more with a yeoman generator.

Reasoning

The astericks indicate platforms for which I have real world experience on production systems.  I am not going to list out of what I do and don’t like about each platform.  There are entire books worth of forum posts and opinions written about the various merits.  All of these platforms can be, and have been, used to create anything from small personal websites to enterprise class systems (even PHP).

Pros and Cons of Dynamic Languages on the Backend

Greater productivity at the beginning is why so many people like the newer platforms, such as RoR, and there is something valid to be said about that. However, when it comes to evolution over time and maintenance, dynamic languages start looking not so great.

Although controversial, I also question if productivity is greater in dynamic languages for non-beginners.  My reasoning is that when doing software development, the majority of the time is not spent writing code.  It’s thinking through problems, devising and testing solutions, and refactoring. A dynamic language might make the code more succinct, but in the end, I wonder if that is worth the trade-offs.  An example trade-off is the very powerful automatic refactoring capabilities that are possible with a typed language like Java.  While some refactoring is possible with dynamic languages and IDE capabilities are growing, the majority of refactoring when using dynamic languages often comes down to having a very good text search & replace tool with decent regular expression support.

Java has a bad (and maybe deserved) reputation for being overly verbose.  Again though, I know for myself the majority of my time is not spent in writing code. Rather, it’s doing all the things around writing code.  I think it’s also interesting that the C/C++ language usage is steady, if not on the rise, in many polls/indexes, and C/C++ makes Java’s verbosity look like Cliff Notes.  There are (almost) always trade-offs.

Negating some benefits of the dynamic platforms

Some of the benefits of more dynamic platforms are negated when modern Java development options. For example:

• JRebel – Removes need to restart after every change. This is a huge productivity booster and I wouldn’t want to do large app Java development without it.  There are alternatives to JRebel (many much less expensive), but JRebel is still the best AFAIK.

• Embedded Scripting Engines – Java makes it easy to embed a scripting engine. So it becomes easy to write glue code using JavaScript, Groovy, or any number of other dynamic languages.  Although polygot programming is problematic IMO, being able to have parts of system be written in a dynamic language can solve a number of otherwise hard to solve problems.  (For example, arbitrary business logic that changes often and should not require recompiling/redeploying the system.)

• Alternatives Languages on JVM – Although I don’t have personal experience in this, some leading developers that I respect seem to highly recommend using Scala. Other options include Jython, JRuby, Groovy, and Clojure.  As long as the alternative language allows calling in to existing Java libraries, then I can see this being beneficial when there’s a benefit to using one language over another for a certain problem domain. That said, I sort-of agree with Lawrence Kesterlot’s post “Java for Everything?“.

As far as getting a backend project bootstrapped, I find Maven makes it very easy, especially when combined with Spring Boot.  And projects like jHipster can bootstrap a whole stack in minutes.

 Conclusion

This post is just a collection of random thoughts around my own baseline.  I started the post back in 2013 (two years ago) and kept it as a draft because it never seemed worth publishing.  I’m not sure it is worth much even now, but decided to get it out of my drafts folder. It only required a few updates to bring it up-to-date.

I find that being a more experienced modern programmer requires walking a fine line between using what works reliably, and constantly seeking out ways of doing things ever better.

The only way I know of, currently, is to provide a custom WebMvcConfigurer. Various examples on the Internet and StackOverflow show how to provide an explicit mapping, but I wanted to extend the built-in WebMvcConfigurer instead of replacing it.

I decided to try extending the Spring Boot WebMvcAutoConfigurationAdapter, and that seems to have worked.

@Configuration
@ConditionalOnWebApplication
@ConditionalOnClass({Servlet.class, DispatcherServlet.class, WebMvcConfigurerAdapter.class})
@ConditionalOnMissingBean(WebMvcConfigurationSupport.class)
@Order(Ordered.HIGHEST_PRECEDENCE + 10)
@AutoConfigureAfter(DispatcherServletAutoConfiguration.class)
public class CustomWebMvcAutoConfig extends WebMvcAutoConfiguration.WebMvcAutoConfigurationAdapter {

  @Override
  public void addResourceHandlers(ResourceHandlerRegistry registry) {
    String myExternalFilePath = "file:///C:/Temp/whatever/m/";

    registry.addResourceHandler("/m/**").addResourceLocations(myExternalFilePath);

    super.addResourceHandlers(registry);
  }

}

The call to `super.addResourceHandlers` sets up the default handlers.

Also:

• Note the trailing slash on the external file path. (Depends on your expectation for URL mappings).
• Consider reviewing the source code of [WebMvcAutoConfigurationAdapter][2]

With this in place, static content from myExternalFilePath is served to the /m path, and the default static URL mappings continue to work as well.

Related StackOverflow Q/A: How do I use Spring Boot to serve static content located in Dropbox folder?

This is my fourth iteration creating a pan/tilt video demo to test out new software ideas. The construction (and availability of the off-the-shelf components) has changed quite a bit since the days of version 1 & version 2. I don’t know what happened to version 3.

Photos

Video

“No developer that is proficient in both modern Java and Microsoft .NET, would choose to use Microsoft .NET for building new backend type systems.”  – Me

So it’s perplexing to me why there seem to be so many .NET based companies, at least in the local area.  I understand that using what the development team already knows and has experience with will trump most other aspects, especially in an environment that has already standardized on the Microsoft platform.  However, I don’t understand why more .NET developers don’t even try looking outside-the-box. I think they’d be amazed.

Not that long ago when interviewing and trying to win new consulting work, I would say that I was ambivalent to actual platform.  That was of course key to finding work since most environments already had chosen their platform, but I was truly proud of that way of thinking. I really didn’t care. I just wanted to build cool things and either platform made that possible to the same extent.

That is no longer the case. In fact, now if a team tells me they are using .NET, I will likely try to avoid working there.  .NET gets in the way of building cool things.  It’s not that it can’t be done.  (See stackoverflow for example.)  It just can not be done as quickly or as efficiently or to the same degree of excellence when using more modern techniques and tools.

It’s difficult to have a real discussion on this because very few developers have worked on both platforms in any real capacity (much less the alternatives like RoR, Grails, Node.js, etc).  It is also developer nature to have a strong bias towards whatever it is that was learned first and/or known best.  I consider myself one of the lucky(?) few to have toggled back and forth on these platforms a few times over the past 10 years, but I know I am biased as well no matter how hard I’ve tried to evaluate .NET objectively lately.

Here are just few of the reasons that I’ve become more set in my ways:

• Visual Studio is not nearly as productive as Eclipse, IntelliJ, or Netbeans.
  • I’ve been using VS 2012. From a pure UI performance perspective, 2010 and 2012 seemed like a step back.
  • With VS, it’s basically required to also purchase ReSharper. Even with ReSharper, the refactoring support doesn’t compare to that in the major Java IDEs.
  • VS is simply slow. This still surprises me because Java GUIs are not known for their responsiveness, and yet they are more responsive than VS for many operations, especially when using contextual navigation.
  • It really irks me when the VS UI becomes unresponsive (which happens often) while it does some other task. Some other task that should’ve been a background task not blocking the UI.
  • Coming from Eclipse especially, with it’s constant compiler, having to constantly save and build in VS is painful.
  • The VS continue-and-edit capability is weak when debugging ASP.NET apps. It provides a fraction of the features available with the standard JDT.  And much, much, less capability and efficiency if used to using JRebel with Java …especially on larger web type systems that take a while to start.
• All of the useful .NET development utility type libraries feel rough, incomplete, and generally just way behind the Java equivalents.  Some examples include NuGet vs. Maven, Hibernate vs. Entity Framework, and Spring IoC vs. Ninject.
• The .NET development community as a whole (and I’m generalizing heavily here), does not seem to have enough experience or sufficient background in building robust systems.  While there are bad developers everywhere (and probably even more in the Java camp due to larger community size, age of the platform, and a bias towards stagnant enterprise environments), what I’m referring to is the difficulty in finding good information and best practices online.
  • As just one example, it is very difficult to find definitive information on how to use transactions correctly in Entity Framework on ASP.NET MVC project.  Most .NET developers I’ve come across seem to assume that everything works out-of-the-box.  And in the case of not knowing your actual database transaction boundaries, it does work …most of the time.

Looking over this list so far, I’m not sure it’s really indicative of why it is I no longer enjoy using .NET personally though.  Maybe it’s the  little things. Like being able to attach a remote debugger to a runtime instance and profile with jVisualVM, or that there’s access to all sorts of excellent libraries like Apache HttpClient and Google’s Guava. Or that the Java concurrency package is light years ahead of anything in the .NET framework.

I’m not an Oracle fan.  I actually despise the company and their database. And I’m no Microsoft hater.  I have a significant interest and likes for many things coming out of Microsoft.  Also, note that I really have no concern for costs. (Though why, given all the above, companies choose to pay large amounts to Microsoft without at least a long-term objective intending to move off that development platform is beyond my intellect.)

If the developer horde did read this blog, then I’m sure I’d gets lots of comments of the sort “java sucks almost as bad as .NET you old timer. use <new-awesome-platform> instead”.  I’d refer those people to the excellent post by Zef, “Pick your battles“.

I simply want to build cool things that get used by real people who probably don’t care about the underlying technology. Better. Faster. Reliably.  For that reason, I no longer want to work with Microsoft .NET.

Alas, on Monday, I will once again head in to an office to do, you guessed it, work using .NET.

RealTerm

Initially RealTerm 2.0.0.70 displays errors when installing and running.  The fix is simply to run as administrator.  (Having administrator privileges is not sufficient.) At some point something changed though and now I no longer have to run as administrator.  I’m not sure if it’s only required on the initial install/run, or if changes I made to get com0com working also affected RealTerm.

com0com (windows 8)

Installing com0com 3.0.0.0 required more effort. Initially it showed multiple errors during install. Something like:

(SetupCopyOEMInf(C:\Program Files (x86)\com0com\com0com.inf) ERROR: 0xE000022F)”.

The virtual com ports displayed in DeviceManager, but after reboot were labeled with a warning about not having digital signature verification. I tried a number of things before I got it working. In the end, I think this is all that’s required:

1. Enable test mode per the readme.txt
   1. i.e. bcdedit.exe -set TESTSIGNING ON
   2. In windows 7 (x64), that was sufficient for me.  It seems some people also had to disable verification via that same tool, but that no longer works in Windows 8 regardless.
2. In windows 8, you also need to A) bypass/disable driver signature verification or B) sign the com0com driver.
   1. To bypass temporarily,  you can go in to the windows 8 startup options menu and disable driver signature verification. Once disabled, com0com installed and worked fine. However, I could not figure out how to disable the verification permanently at the time, so it reverted after every reboot. The drivers would be installed, but disabled in device manager.
      1. I later did read about ways to permanently disable the driver verification. I did not try this though since signing the driver was sufficient (and would seem to be slightly more secure.)
   2. To sign the com0com.sys driver (in the “Windows/System32/Drivers” directory), I used this tool: “Driver Signature Enforcement Overrider 1.3b” at ngohq.com. Once signed (and still in windows test mode), com0com works as it did in Windows 7 x64.

[Update 2014-06]

com0com (windows 8.1)

In windows 8.1, signing the driver as described above no longer seems to work.  However disabling driver signature verification via windows 8.1 recovery troubleshooting restart menus did allow com0com to install and function.  (I’m not sure if it will continue to function after a normal restart though.)

public class GZipToJsonHandler : DelegatingHandler
{
	protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
	{
		// Handle only if content type is 'application/gzip'
		if (request.Content.Headers.ContentType == null ||
			request.Content.Headers.ContentType.MediaType != "application/gzip")
		{
			return base.SendAsync(request, cancellationToken);
		}

		// Read in the input stream, then decompress in to the outputstream.
		// Doing this asynronously, but not really required at this point
		// since we end up waiting on it right after this.
		Stream outputStream = new MemoryStream();
		Task task = request.Content.ReadAsStreamAsync().ContinueWith(t =>
			{
				Stream inputStream = t.Result;
				var gzipStream = new GZipStream(inputStream, CompressionMode.Decompress);

				gzipStream.CopyTo(outputStream);
				gzipStream.Dispose();

				outputStream.Seek(0, SeekOrigin.Begin);
			});

		// Wait for inputstream and decompression to complete. Would be nice
		// to not block here and work async when ready instead, but I couldn't
		// figure out how to do it in context of a DelegatingHandler.
		task.Wait();

		// This next section is the key...

		// Save the original content
		HttpContent origContent = request.Content;

		// Replace request content with the newly decompressed stream
		request.Content = new StreamContent(outputStream);

		// Copy all headers from original content in to new one
		foreach (var header in origContent.Headers)
		{
			request.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
		}

		// Replace the original content-type with content type
		// of decompressed data. In our case, we can assume application/json. A
		// more generic and reuseable handler would need some other
		// way to differentiate the decompressed content type.
		request.Content.Headers.Remove("Content-Type");
		request.Content.Headers.Add("Content-Type", "application/json");

		return base.SendAsync(request, cancellationToken);
	}
}

Added to the MessageHandlers in WebApiConfig.cs with:

    config.MessageHandlers.Add(new GZipToJsonHandler());

Using this approach, existing controllers which normally work with JSON content and automatic model binding, continue to work without any changes. Related StackOverflow question with my answer here.

Using Java from within Matlab

The java integration with Matlab is straightforward and works well.  (The same seems to be true for .NET integration, although I didn’t test it.)   No additional toolboxes or addons are required, beyond the core Matlab product, in order to call java classes from within Matlab.

However, there is one significant caveat; the java classpath of any custom code must share the classpath with matlab, and matlab includes a number of well-known libraries. My proof-of-concept project also used a lot of libraries and so there were a number of conflicts that made it difficult to reuse my code in Matlab directly. I almost got it to work eventually and probably could’ve if I continued, but I ended up taking a different approach using a remote network interface rather than direct Java library calls. The network based approach is much more flexible and doesn’t suffer from the classpath conflict issues.

• http://www.mathworks.com/help/matlab/using-java-libraries-in-matlab.html
• http://www.mathworks.com/matlabcentral/answers/5892-use-java-classes-in-matlab
• http://undocumentedmatlab.com/blog/matlab-and-the-event-dispatch-thread-edt/

In my proof-of-concept I published an RMI service as an interface in to my existing code base (which required minimal amount of code & setup since I was already using Spring Framework). I then built a small JAR library for use by Matlab that provided an RMI client proxy to the remote services. I used RMI, but could easily have used any other remoting protocol as well, such as Spring’s HTTP remoting. This setup worked well, and is flexible because it provides a dedicated area where matlab specific conversions and handling can be done with impact on existing code. It is also negated issues with classpath conflicts. Yet another benefit was that the Matlab client could be running on a different machine than the server code base.

Using .NET from within Matlab

Similar to Java integration, no additional toolboxes or addons are required in order to call .NET assemblies from within Matlab.

• http://www.mathworks.com/help/matlab/using-net-libraries-in-matlab.html

Calling Matlab from Java/.NET

Calling in to matlab from Java/.NET requires addons. The official approach to is to purchase the matlab compiler and then a standalone toolbox specific to each platform. This combination lets you compile standalone executables (and libraries?).

Pricing

I’m not sure what the rules are around publishing pricing information, but suffice to say, the official Matlab approach requires a relatively expensive initial investment.  Required components include:

• Matlab Core
• Matlab Compiler
• Matlab Builder for Deployment  (per platform)

The initial pricing I received would’ve required >$10k to get started. There are no runtime distribution costs though.

Alternative Unofficial Approaches

• https://code.google.com/p/matlabcontrol/
• http://www.cs.virginia.edu/~whitehouse/matlab/JavaMatlab.html
• http://www.cs.utoronto.ca/~bowen/code/code.html#matjav
• http://jamal.khadkevich.org / https://github.com/hutm/JAMAL

Additional Research

The remaining part of this post is primarily a collection of links and notes I generated while researching.

I don’t have much experience using external math engines in recent projects, however, based on what I’ve researched, I would not lean towards using Matlab for my own projects these days. It might be different if I had extensive knowledge of Matlab already, or lots of code that depended on it.

Matlab Alternatives

• SciPy: http://en.wikipedia.org/wiki/SciPy
• GNU Octave :http://en.wikipedia.org/wiki/GNU_Octave
• Scilab: http://www.scilab.org/
• Sage: http://www.sagemath.org/ (more symbolic math oriented? e.g. More like Mathematica vs. Matlab)

NumPy/SciPy

• http://www.scipy.org/more_about_SciPy
• http://docs.scipy.org/doc/
• http://stackoverflow.com/questions/6200910/relationship-between-scipy-and-numpy

Matlab to NumPy

• http://www.scipy.org/NumPy_for_Matlab_Users
• http://mathesaurus.sourceforge.net/matlab-numpy.html

NumPy/SciPy Development

Dedicated SciPy IDEs

• http://enthought.com/products/epd.php
• http://code.google.com/p/spyderlib/
• http://code.google.com/p/pythonxy/

Generic Python IDEs

• http://www.jetbrains.com/pycharm/
• http://wingware.com/
• http://ipython.org/

Plotting

• http://matplotlib.org/
• http://www.packtpub.com/matplotlib-python-development/book?utm_source=matplotlib.sourceforge.net&utm_medium=link&utm_content=pod&utm_campaign=mdb_002124
• http://code.enthought.com/projects/mayavi/

Misc Links & Articles

Matlab vs PySci

• http://stevetjoa.com/305/, good info including the comments
• http://vnoel.wordpress.com/2008/05/03/bye-matlab-hello-python-thanks-sage/

Antecdotal

• http://news.ycombinator.com/item?id=363096
• http://stackoverflow.com/a/11144102/123378
• http://stackoverflow.com/questions/5063037/should-i-switch-to-python
• http://www.reddit.com/r/math/comments/tl77g/which_one_of_these_gpl_gnu_general_public_license/

Mathematica vs Matlab vs Sage vs Octave

• http://math.stackexchange.com/questions/153/what-should-i-learn-first-mathematica-or-matlab
• http://sagemath.blogspot.com/2007/12/why-isnt-sage-just-part-of-octave.html

Python & Qt, PySide/PyQt

• http://askubuntu.com/questions/140740/should-i-use-pyqt-or-pyside-for-a-new-qt-project
• http://www.pyinstaller.org/
• http://www.stat.washington.edu/~hoytak/blog/bestlibraries.html
• http://code.google.com/p/guiqwt/
• http://www.pyqtgraph.org/
• http://pythonhosted.org/PyQt-Fit/

Matlab & Python

• http://stackoverflow.com/questions/1707780/call-python-function-from-matlab
• http://www.mathworks.com/matlabcentral/newsreader/view_thread/290116

Matlab, Misc

• http://stackoverflow.com/questions/2326609/how-to-organize-matlab-code
• http://www.datatool.com/prod02.htm (matlab programming style guidelines)

• Important for Regression
  • http://www.mathworks.com/help/matlab/math/interpolating-gridded-data.html
  • http://www.mathworks.com/support/solutions/en/data/1-15NDN/index.html?product=ML&solution=1-15NDN

• Undocumented Matlab
  • http://undocumentedmatlab.com/blog/category/java/

• Especially interesting:
  • http://undocumentedmatlab.com/blog/extending-a-java-class-with-udd/
  • http://undocumentedmatlab.com/blog/using-groovy-in-matlab/
  • http://undocumentedmatlab.com/blog/using-java-collections-in-matlab/

No one that has used and learned IDEs like IntelliJ or Eclipse would choose Visual Studio if given the choice.  (The same is true for .NET development in general).  But alas, the world does not always operate at 100% efficiency, and so the shortcuts above are of interest to me currently.