My view from the Rose Colored Glasses at AWS re:Invent 2015

Another year, another AWS re:Invent conference,  Changes, innovation, surprises are now the expected, and the expected, trendy, new and exciting are mundane or mature or yesterdays news for last weeks generation.

I'll leave the highlights and summaries to others - or follow the trail starting here https://reinvent.awsevents.com/

Instead, here's my take on the the impact and 'meta' view of how AWS and the fairly new but now mature conference and related technology relates to, drives, foreshadows and reflects trends in technology.  Not large by SalesForce standards,  but what was 5000, then 10,000 now 20,000 attendees at the dual hotel/conference mecca The Palazzo/Venetian has reached conference capacity and overreached sanity.   The finale "Party" aka "Re:Play" was too big for the location and had to be *built* out of a parking lot at the Linq -- I got a good view from atop the "High Roller" the next day of the disassembly.   Hundreds of shipping containers stacked like legos covered the night before by cloth or plastic still remained as the steel bones of the scaffolding were dismantled.

The "Party" mirrors the Company which mirrors the Technology ... huge, efficient, geeky.  Fun for some, hard work for others - but not really much like a "Party".  Not for lack of substance or glamour, free food and drinks, an empty "VIP" section - logos of name-your-brand Fortune 500 companies - and I'm not sure who is paying for what.  But under the hood its pure business, with a decent attempt at fun and show but not trying to fool anyone that its all about the business of Getting Things Done.

A quick skim over the expected 'new and improved' --

• "Snowball" - Proof still (and horribly re-'Invented' quote which I will spare you -- a closer to original being "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."   and presciently satired in https://what-if.xkcd.com/31/
  500 TB of military grade storage you can drop from 5 feet to concrete that includes its own e-Ink shipping label to get your data to and from AWS faster then a speeding ... FedEx Truck.
  Not new but very Amazon.  Efficient.
• The Biggest and the Smallest EC2 instance type to date ( X1 and nano )
• A few new services - but mostly incremental (but long needed) enhancements, features, and 'next' generation services that make it easier to manage the growing range of services.
  
  AWS is so far ahead of any so-called "competition" in every way - the only value I can see in the 'competition' being able to claim there is one so lock-in wary people can squint really hard and convince themselves that Open Stack, Google Apps, Microsoft Azure are all viable replacements if the need arises ... Good luck with that.
  It keeps AWS's prices dropping and their services increasing.  
  I do hope the others try to pretend to keep pace so if/when AWS turns Evil others can pick up a few years behind and a few trillion systems short and keep the world from falling apart.

In the mean time -- I'm more certain then ever were are in the Singularity.   Quite literally.

As an imaginary space traveler falling into a black hole -- things don't look any different then the entire universe does to us here 'at rest' in the center of the unchanging universe.   The wonderful paradox of exponential progress and cyclic patterns -- more of the same and 'you didn't believe me last year but now its old news' -- and you still wont believe it.  It doesn't matter - "The Future Is Now" is way too mundane to describe things.

A sample of the Paradox of the day ... 

• Microservices
  Not long ago ( days ? years ?) they were a nice theory - and quite useful for startups and 'small toy projects'.  REST to the Rescue will help you distribute your monolithic app into a thousand (hundred?) "Micro" services -- Not to be confused with every day "Services" ... no.
  "Micro" services ... A matter of scale -- but still "Services".   Client/server kind of services that live a long time and sit on a port waiting to answer a "micro" request.  Stateless, redundant.   Scale up and down.  They solve all the problems of the web and fickle unpredictable and predictable demands of users -- no "consumers" -- we don't "use" anymore we "consume" -- as fast as we possibly can on every device -- no not "device" -- "IoT"  -- ( Internet of Things).
  Its a good thing too we're "consumers" again -- "Users" are picky and want what they want -- "Consumers" fit the publishing market much better -- sure they want specialized feeds (not firehoses) ... and you need to keep information about them to keep them fed.  But "Consumers" -- Much easier for the corporate world to comprehend then "Users" ...
  
  But wait ... "keep information?"
  
  Microservices are all about "Stateless" that's how they can work.  They are all armies of unattached clones with no bottlenecks -- no "state" ... don't look too close -- maybe "microservices" aren't quite the answer --   Just as were starting to get a hang of what it means to "Dockerize" a legacy application into a thousand stateless pieces --
• Serverless Applications 

All that managing of Docker containers and caches and smart thisandthats -- that's a lot of work.  For what ? To keep track of a million 'consumers' constantly -- but can come and go as unannounced as the servers that feed them.   Will Bill's watch actually ping back for that map sub-segment rendered 5 seconds ahead of where his phone's GPS pinged the Train station Wifi?  Or maybe hes at the pub now and well into a Pint.  You'll not hear from Bill for a long long time -- minutes maybe.   Wasted minutes holding on to open sockets and relatively expenisve huge resource hungry transient micro servers.  In aggregate. 

To do that you have to keep track of them all !   Who wants that.

Just keep it all static on S3 or your favorite distributed file system, load up the client with a supercomputer load of JavaScript and no one will ever know.

Seriously.  Major applications entirely based on "static" resources and device-side logic.

Add a bit of SSO (provided by our humble service provider via Federated SAML or OATH authentication) -- were back in the 90's !  

Just code up some static web pages, and by the way, the code is static web pages too.

But maybe that's not *quite* good enough ... 

Enter the Distributed Event Driven Asynchronous "Near the data" programmable IT oriented Logic Processors.   Ya those things that I said would replace the 'standard' paradigm of "Big" servers and "Central Programs" (aka Docker Hosts + Microservices) -- Ya replace *those* before they were even a sparkle in a young geeks eyes.  Before VMWare was scared into brown pants and along with the other Big Boys started playing "Were MORE Docker then Docker" ... Yes before even ECS ... The stuff of scifi that everyone knows is way way in the future -- of scifi -- and never really going to happen.  

But it snuck up and bit us and now its yesterday's news, and today's commodity.  Literally commodity.   Software as a service, by the nibble, not the bite, not the meal.   Individual functions you upload to the Infrastrure that get executed on your behalf and changed by the *millisecond* -- and the first million calls are free.  Yes, Johny, a MILLION calls are free.  And when you do pay -- you dont need those big clunky docker containers and "swarms" and service discovery, managers and load balancers -- all that hard stuff you had to *program* last week, and *wire up* last year.  na.

• Lambda's Matured

Last year the most incredible, yet inexplicable service from AWS,  "Lambda" appeared.  Like most AWS services, deceptively simple and quite underpowered -- at first.   JavaScript by the line, invoked within a MS of only a handful specific events -- interesting but not much more -- or maybe not even close to -- a stored procedure.  Except ... its *outside* the database.   That's not close.  There is no database.   No, that's not close.  The entire IT Infrastructure of AWS is the 'database' for this "stored procedure" -- and you don't have to manage a thing except throw it over the wire and say "go" ... ( and you better also say when to "stop" -- because now its loose.  you cant "shut down" the VM -- because there isn't on.  Or rather its all over..)

But still ... I'm just barely getting a grip on what they can do, what they should do, and if its insanity or fantasy or hell or heaven.   And just when I either figure it out, or find out its not quite featureful enough, or the edge cases are not "edge" etc... a golden egg drops from the AWS Duck and voila !
Don't like JavaScript ? Ok heres' Java.  Java 8 *and any library you want to upload with it.
DynamoDB Events not enough, ok here's SNS, S3 lifecycle events, and Cloud Watch triggers.
Hard to program Asyncronous distributed bits of functions spewed all over the world ?   How about universal logging, metering, synchronous  calls calls that *return* data, and we'll throw in a complete REST API Front end for -- well --- EVERYTHING.  If you want it -- no pressure.   But just in case you dont like writing secure, enterprise hardened DOS protected, fault tollerant, scalable, dynamic data model mapping front ends to ... everything ... ( All of AWS's thousands of APIS,  any HTTP exposed service on the Internet, and , well, Lambdas ... ) ... here you go.  And for fun we'll toss in a CDN, cached responses, dynamic programmatic DNS, load balancing, DoD protection and WTF integrate with a few Open Source REST modeling tools.

That was the last few months. But breakfast is over, and AWS re:Invent is over ... and they couldnt let a multi bizillion dollar event go without another feature now could they ?
Don't like JavaScript or Java ?  Well here's Python.  And every python library you can find just zip it up and we'll run it for you. By the millisecond,  by the millions.    Oh and sure, here's a cron-like scheduler for those recurring events (the one in a million that don't already occur).
What else ? Honestly I cant keep up.  But if its not there now, I'm sure it will be tomorrow.

So whats the irony ? The paradox ? Whats the Big New Thing ?
Its so big its small.   It so small it huge.
Serverless Distributed applications implemented with IT event driven triggers, Stateful clients and servers built on top of swarms of stateless micro services sharing fast ubiquitous infinitely scalable memory -- or database -- or storage -- or networking -- or -- what is it ?
Best you don't think too hard about that ... Because by the time you do think you have it figured out,
It will be next week and you'll be a micron closer to the center of the singularity.  It will all be so completely different -- that you can't tell it apart for the normal.   In the mean time -- That old dusty program -- the monolithic one that kids scoff at as "Not Modern" and "So 2000's", the languages and markup and documents it grinds through at light speed  ... That sucker is *fast* and *hungry*  and better then ever -- On or off AWS -- well if its not 'on' AWS -- something touching it is ...
and those kids ... their credit cards, the bank's CEO's investments, the media they consume ... the sensors in the  oil wells that feed the electric generators powering their electric cars.
Its all riding along in those "Ancient" apps that have gotten so good they are invisible -- but you wouldst want to live without them.    And AWS wouldn't dream of missing out on some of that revenue ... bring em on ... there's room for everyone.  But dont look behind the green curtain.

Enter Tomorrow, Yesterday -- and hold your hat, the ride hasn't even started yet.

-David