Predictions for 2014 - Part #1

"The most exciting phrase to hear in science... is not 'Eureka!' (I found it!) but 'That's funny.'" ~ Isaac Asimov

"One ring to rule them all, one ring to find them, one ring to bring them all and in the darkness bind them." ~ JRR Tolkien

1. The Big Easy — Big Data gets smaller part #1

I started with big data solutions back in 2008, not (unlike Twitter or Facebook) because I needed a solution to escape the CAP limitations of SQL solutions, but in search of new value from data that we’d otherwise have discarded. CouchDB came first as an experiment in moving off MySQL for Rails apps. MongoDB came next, and persisted because of the following features:

• Easy structure and protocols for SQL-trained DBAs to adopt
• Mongoid and MongoMapper data modeling gems for Rails
• JSON syntax and conventions

This got things started, as time went on the Hadoop environment has produced richer and richer toolsets for bigger and bigger data. These are great for global web-scale companies, but might miss the point for the rest of us. I made the point a couple of years ago that, for the rest of us, the key thing was NOT that big data was BIG — it was that big data was FAST. Now in 2014 we’re ready to take the next step forward: in 2014 everything is FAST, and so big data now needs to be EASY.

I’ve also written before that “there are only 2 kinds of problem that big data solves”:: "Hindsight" (where something has happened and you want to know what in your pile of data might have predicted it) and "Foresight" (where you have a pile of data and want to know what it leads to). Foresight solutions probably outnumber Hindsight 10:1, and being 2014 everybody should be familiar with recommendation engines. So here’s what we can expect in 2014:

• Seeking insight from all forms (web, orders, social, searches) of social data has moved from Innovative to to Best Practice
• All that data is unstructured and needs structure. Hadoop-as-ETL rules the day — leading to…
• Return of the Jedi — SQL databases and reporting tools were always good but couldn’t handle unstructured data. Hadoop is magical with unstructured data, but doesn’t easily provide real-time results, reporting, and support hands-on analysis. So…

2. Business Intelligence is BACK - Big Data gets smaller part #2

The ubiquity of Y2K-spawned ERP and enterprise data systems led to a golden age for BI, but those implementations are mossbacked now, and more than 65% of the new data generated today is unstructured. Standard BI solutions, before the Hadoop boom might run $50K per terabyte in licensing fees alone, and cost and structure made them a tough fit for vast, sloppy piles of interaction data. MapReduce to the rescue, with the operation part of the term is Reduce — as petabytes of document-data crystalize like diamonds into gigabytes of nice, reduced, rows and columns in conventional data stores. Manage your data right, and the "Spreadsheets for the New Millennium" that I’ve written about previously here, here and here become exactly that:

Spreadsheets!

3. JavaScript everywhere - MEAN and Meteor

Much of my earlier big data work started with Ruby on Rails — Rails was a great DSL for the web, and it provided a couple of wonders that are still wonders today, and were absolutely magical in 2005:

• A web domain-specific-language. Write for the Web in the language (thinly-veneered http/html) of the web
• Active Record — object-relational management for the rest of us — escaping the nasty .Net antipattern of starting with brittle stored procedures first and subsequently coding your way out into the user domain
• Full-stacked-ness — with Active Record and similar patterns it only took one language (Ruby) to create your entire application — front end, back end, databases and presentation layer and all. Seemingly gone was the need for distinct (and non-cross-communicating) teams of DB-developers, middleware developers, and front-end developers.

I still love Rails, but by Version 4 Rails has left its simple past. Gone are the days when DHH (and everyone like him) could produce a good demo of a blog application, crafted from scratch in the course of a 15-minute video. As Rails has gotten bigger and richer, its universality has declined because it got cumbersome and subsequent programming models started with assumptions Rails pioneered.

I still love full-stack development and tools, and much as I like Rails as a replacement for sweet-but-overgrown J2EE, it’s now time to see what might advance us beyond sweet-but-overgrown Rails. Such an überstack might feature:

• Full-stack: one programming language and model, top-to-bottom
• Fast: I’ve always loved Smalltalk (Alan Kay’s gift to programming languages), but the cobra still bites at anything > 100 ms.
• Universal: Separate teams with separate development languages is SO 1994! Even worse, desktops are so 1981 and modern code needs to expect to run on everything from handsets to big-screen displays.

Can any language and platform meet all these requirements? Fortunately, there is a solution!

Here’s what we need — power and consistency at the database, application, and presentation layers, with a common language and syntax across all layers. In Rails we covered the layers with Ruby from Active Record up to ERB, and if we’re going to get better and faster, we’ve got to get MEAN:

• MongoDB (the database)
• Express (presentation framework)
• AngularJS (presentation language extensions)
• Node.js (the web server)

Meteor and Ember.js are great emerging framework as well, and they all lead to full stack development with JavaScript everywhere. As I’ve written before, Node.js is wicked fast with modern JS engines. There are other nice frameworks rising in the JS world — Meteor.js, Derby.js and others — and JavaScript is the world’s most popular programming language, and has moved as near to ubiquity as a programming language can. For a great introduction to JavaScript and what makes it good, and a nice intro to MapReduce you might look here: Can Your Programming Language Do This?. To paraphrase William Gibson, the future is already here, and it's about to become more evenly distributed...

We’ll need these tools and popularity for the yin and the yang of the modern web age: The Internet of Things and BubblePop, which I’ll cover next time.

Comments

[Jim]

node.js is awesome but I'm not sure I'm going to leave my day job for it anytime soon. Maybe, I just haven't written enough software in it yet.

[Jim]

I should have said in the earlier comment that I do believe Rails has kind of lost it's way. It's ridiculously slow now, maybe it's been that way all along and we ignored it in the pursuit of developer productivity. 4.0 brings even more changes to something that should have matured by now. I'm currently looking into faster languages/frameworks, Elixir, Node, Go. All of the above seem to be just as productive for programmers with much greater speed.