Sunday, October 11, 2015

My view from the Rose Colored Glasses at AWS re:Invent 2015

Another year, another AWS re:Invent conference,  Changes, innovation, surprises are now the expected, and the expected, trendy, new and exciting are mundane or mature or yesterdays news for last weeks generation.

I'll leave the highlights and summaries to others - or follow the trail starting here https://reinvent.awsevents.com/

Instead, here's my take on the the impact and 'meta' view of how AWS and the fairly new but now mature conference and related technology relates to, drives, foreshadows and reflects trends in technology.  Not large by SalesForce standards,  but what was 5000, then 10,000 now 20,000 attendees at the dual hotel/conference mecca The Palazzo/Venetian has reached conference capacity and overreached sanity.   The finale "Party" aka "Re:Play" was too big for the location and had to be *built* out of a parking lot at the Linq -- I got a good view from atop the "High Roller" the next day of the disassembly.   Hundreds of shipping containers stacked like legos covered the night before by cloth or plastic still remained as the steel bones of the scaffolding were dismantled.

The "Party" mirrors the Company which mirrors the Technology ... huge, efficient, geeky.  Fun for some, hard work for others - but not really much like a "Party".  Not for lack of substance or glamour, free food and drinks, an empty "VIP" section - logos of name-your-brand Fortune 500 companies - and I'm not sure who is paying for what.  But under the hood its pure business, with a decent attempt at fun and show but not trying to fool anyone that its all about the business of Getting Things Done.

A quick skim over the expected 'new and improved' --

  • "Snowball" - Proof still (and horribly re-'Invented' quote which I will spare you -- a closer to original being "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."   and presciently satired in https://what-if.xkcd.com/31/
    500 TB of military grade storage you can drop from 5 feet to concrete that includes its own e-Ink shipping label to get your data to and from AWS faster then a speeding ... FedEx Truck.
    Not new but very Amazon.  Efficient.
  • The Biggest and the Smallest EC2 instance type to date ( X1 and nano )
  • A few new services - but mostly incremental (but long needed) enhancements, features, and 'next' generation services that make it easier to manage the growing range of services.

    AWS is so far ahead of any so-called "competition" in every way - the only value I can see in the 'competition' being able to claim there is one so lock-in wary people can squint really hard and convince themselves that Open Stack, Google Apps, Microsoft Azure are all viable replacements if the need arises ... Good luck with that.
    It keeps AWS's prices dropping and their services increasing.  
    I do hope the others try to pretend to keep pace so if/when AWS turns Evil others can pick up a few years behind and a few trillion systems short and keep the world from falling apart.
In the mean time -- I'm more certain then ever were are in the Singularity.   Quite literally.
As an imaginary space traveler falling into a black hole -- things don't look any different then the entire universe does to us here 'at rest' in the center of the unchanging universe.   The wonderful paradox of exponential progress and cyclic patterns -- more of the same and 'you didn't believe me last year but now its old news' -- and you still wont believe it.  It doesn't matter - "The Future Is Now" is way too mundane to describe things.

A sample of the Paradox of the day ... 

  • Microservices
    Not long ago ( days ? years ?) they were a nice theory - and quite useful for startups and 'small toy projects'.  REST to the Rescue will help you distribute your monolithic app into a thousand (hundred?) "Micro" services -- Not to be confused with every day "Services" ... no.
    "Micro" services ... A matter of scale -- but still "Services".   Client/server kind of services that live a long time and sit on a port waiting to answer a "micro" request.  Stateless, redundant.   Scale up and down.  They solve all the problems of the web and fickle unpredictable and predictable demands of users -- no "consumers" -- we don't "use" anymore we "consume" -- as fast as we possibly can on every device -- no not "device" -- "IoT"  -- ( Internet of Things).
    Its a good thing too we're "consumers" again -- "Users" are picky and want what they want -- "Consumers" fit the publishing market much better -- sure they want specialized feeds (not firehoses) ... and you need to keep information about them to keep them fed.  But "Consumers" -- Much easier for the corporate world to comprehend then "Users" ...

    But wait ... "keep information?"

    Microservices are all about "Stateless" that's how they can work.  They are all armies of unattached clones with no bottlenecks -- no "state" ... don't look too close -- maybe "microservices" aren't quite the answer --   Just as were starting to get a hang of what it means to "Dockerize" a legacy application into a thousand stateless pieces --
  • Serverless Applications 
All that managing of Docker containers and caches and smart thisandthats -- that's a lot of work.  For what ? To keep track of a million 'consumers' constantly -- but can come and go as unannounced as the servers that feed them.   Will Bill's watch actually ping back for that map sub-segment rendered 5 seconds ahead of where his phone's GPS pinged the Train station Wifi?  Or maybe hes at the pub now and well into a Pint.  You'll not hear from Bill for a long long time -- minutes maybe.   Wasted minutes holding on to open sockets and relatively expenisve huge resource hungry transient micro servers.  In aggregate. 
To do that you have to keep track of them all !   Who wants that.

Just keep it all static on S3 or your favorite distributed file system, load up the client with a supercomputer load of JavaScript and no one will ever know.
Seriously.  Major applications entirely based on "static" resources and device-side logic.
Add a bit of SSO (provided by our humble service provider via Federated SAML or OATH authentication) -- were back in the 90's !  
Just code up some static web pages, and by the way, the code is static web pages too.

But maybe that's not *quite* good enough ... 
Enter the Distributed Event Driven Asynchronous "Near the data" programmable IT oriented Logic Processors.   Ya those things that I said would replace the 'standard' paradigm of "Big" servers and "Central Programs" (aka Docker Hosts + Microservices) -- Ya replace *those* before they were even a sparkle in a young geeks eyes.  Before VMWare was scared into brown pants and along with the other Big Boys started playing "Were MORE Docker then Docker" ... Yes before even ECS ... The stuff of scifi that everyone knows is way way in the future -- of scifi -- and never really going to happen.  


But it snuck up and bit us and now its yesterday's news, and today's commodity.  Literally commodity.   Software as a service, by the nibble, not the bite, not the meal.   Individual functions you upload to the Infrastrure that get executed on your behalf and changed by the *millisecond* -- and the first million calls are free.  Yes, Johny, a MILLION calls are free.  And when you do pay -- you dont need those big clunky docker containers and "swarms" and service discovery, managers and load balancers -- all that hard stuff you had to *program* last week, and *wire up* last year.  na.

  • Lambda's Matured
Last year the most incredible, yet inexplicable service from AWS,  "Lambda" appeared.  Like most AWS services, deceptively simple and quite underpowered -- at first.   JavaScript by the line, invoked within a MS of only a handful specific events -- interesting but not much more -- or maybe not even close to -- a stored procedure.  Except ... its *outside* the database.   That's not close.  There is no database.   No, that's not close.  The entire IT Infrastructure of AWS is the 'database' for this "stored procedure" -- and you don't have to manage a thing except throw it over the wire and say "go" ... ( and you better also say when to "stop" -- because now its loose.  you cant "shut down" the VM -- because there isn't on.  Or rather its all over..)

But still ... I'm just barely getting a grip on what they can do, what they should do, and if its insanity or fantasy or hell or heaven.   And just when I either figure it out, or find out its not quite featureful enough, or the edge cases are not "edge" etc... a golden egg drops from the AWS Duck and voila !
Don't like JavaScript ? Ok heres' Java.  Java 8 *and any library you want to upload with it.
DynamoDB Events not enough, ok here's SNS, S3 lifecycle events, and Cloud Watch triggers.
Hard to program Asyncronous distributed bits of functions spewed all over the world ?   How about universal logging, metering, synchronous  calls calls that *return* data, and we'll throw in a complete REST API Front end for -- well --- EVERYTHING.  If you want it -- no pressure.   But just in case you dont like writing secure, enterprise hardened DOS protected, fault tollerant, scalable, dynamic data model mapping front ends to ... everything ... ( All of AWS's thousands of APIS,  any HTTP exposed service on the Internet, and , well, Lambdas ... ) ... here you go.  And for fun we'll toss in a CDN, cached responses, dynamic programmatic DNS, load balancing, DoD protection and WTF integrate with a few Open Source REST modeling tools.

That was the last few months. But breakfast is over, and AWS re:Invent is over ... and they couldnt let a multi bizillion dollar event go without another feature now could they ?
Don't like JavaScript or Java ?  Well here's Python.  And every python library you can find just zip it up and we'll run it for you. By the millisecond,  by the millions.    Oh and sure, here's a cron-like scheduler for those recurring events (the one in a million that don't already occur).
What else ? Honestly I cant keep up.  But if its not there now, I'm sure it will be tomorrow.

So whats the irony ? The paradox ? Whats the Big New Thing ?
Its so big its small.   It so small it huge.
Serverless Distributed applications implemented with IT event driven triggers, Stateful clients and servers built on top of swarms of stateless micro services sharing fast ubiquitous infinitely scalable memory -- or database -- or storage -- or networking -- or -- what is it ?
Best you don't think too hard about that ... Because by the time you do think you have it figured out,
It will be next week and you'll be a micron closer to the center of the singularity.  It will all be so completely different -- that you can't tell it apart for the normal.   In the mean time -- That old dusty program -- the monolithic one that kids scoff at as "Not Modern" and "So 2000's", the languages and markup and documents it grinds through at light speed  ... That sucker is *fast* and *hungry*  and better then ever -- On or off AWS -- well if its not 'on' AWS -- something touching it is ...
and those kids ... their credit cards, the bank's CEO's investments, the media they consume ... the sensors in the  oil wells that feed the electric generators powering their electric cars.
Its all riding along in those "Ancient" apps that have gotten so good they are invisible -- but you wouldst want to live without them.    And AWS wouldn't dream of missing out on some of that revenue ... bring em on ... there's room for everyone.  But dont look behind the green curtain.

Enter Tomorrow, Yesterday -- and hold your hat, the ride hasn't even started yet.

-David

Monday, December 9, 2013

Security - What is it and what is it good for ?

Security - What is it and what is it good for ?


Disclaimer, I am NOT a security expert.  This commentary is more focused on the societal and social concepts then any particular technology - from the perspective of a normal reasonably educated and informed human.  In fact I suggest that you miss the forest through the trees if you focus on security-as-technology.

"Security" is a simple sounding word that most people think has an obvious meaning.  It doesn't.  "Security" is complicated, its not black and white, and there aren't even enough colors in the rainbow to fully describe it.   The Industry tosses around the word as if it had tangible meaning because, well, who doesn't want to be "Secure"?   Security is good, right ? Everyone wants it, everyone needs it so if you sell a product or service slapping "Secure" on it is a no-brainer.   Its good.

Possession is ownership

First off, generally physical items can be characterized by their tangible nature.  You can touch them, you can hold them, you can own them.   Generally physically possessing something is synonymous with ownership.  So if you can secure possession you can secure ownership.  And if you secure ownership you secure value.
But what is "Security" ?  Security is often used as a placeholder for "Protection".  But protection from what?  For physical things, security usually means protection from loss of ownership.   Since there is only one of a particular thing, loss means taking something away depriving you of it.  Loss of possession becomes loss of ownership which becomes loss of value.

Protection

  A "Bank Vault" is a typical image of physical security.  You put things in it so people can't take them.   But if something is in a vault you can't use it.    If you put your car in a vault you couldn't drive it.   If your money is in a vault you can't spend it.  If your physical life needs to be secure should you lock yourself in a vault? 
Physical security is at direct odds with usability.    If something is perfectly secure its generally completely unusable.   So what good is owning or possessing something if you can't use it?   Sometimes securing something for future use is sufficient, such as saving money to spend later.  But often you want to secure something AND use it.   And in either case the moment you remove your item from the valut it is no longer protected so it may simply be reducing the window of opportunity for loss not total protection from it.

Usability

This leads to comprises in security in order to have a measure of usability.   Instead of putting your car in a vault, you put locks on the doors so you can drive it.  Its not as secure, but its more usable.   Another way of compromising security with usability is using a proxy for the item.  You put your real jewlery in the vault but wear glass and knowing you actually own the real thing you feel like you're getting the benefit of it without putting it at risk.   You can associate trust with a proxy vendor.  For example  you create Bank Notes, Checks, Credit Lines, Certificates of Ownership  etc which you can use to buy things without actually putting at risk your actual money (or gold or gems).   This lets your money stay safe but still be used.   But now the recipient has lost some security.  How do they know the check you wrote is good since its not 'the real thing'?  This is where the Proxy comes in.  You are trading on a promise based on a shared faith (the proxy).   Both parties have faith that you really have the money in the bank and the check is a proxy for transferring money from one party to the next.    You both have faith in the proxy vendor (the bank in this case) to honor the transaction - to mirror the transaction of the proxy with an equal transaction of the real thing.
Getting complicated isnt it?

Trust

But now not only have you added a layer of trust (and hence risk) but because the instrument of the proxy (the check) is not the real thing it can be copied!  How do you know your getting the original ? In the case of money which itself is yet another proxy for value, it itself could be copied (counterfeited) so now we have multiple levels of trust and risk and even less security all in the name of usability.

Insurance

How is this solved ?  Insurance.   You add another party to the transaction which is neither the of the parties of the transaction nor the proxy for value.  This party ”insures" the system still works by guarenteeing a replacement in kind or value if the system breaks down.   You leverage insurance at many levels.  At the bank, if your money is stolen it is insured against theft by the bank.  If the currency itself is copied (counterfeit) you are insured by the treasury.  If a check is duplicated you are insured against loss by the bank or a seperate insurance agency.   If your car gets stolen you are insured by car insurance for replacement.
By now the system is so complicated you have absolutely no security of the actual physical items themselves but only their value (declared in some currency or unit).   None is promising to give your your same car back only something of equivalent value.   
And what if the original item is truely irreplaceable - say a one of a kind painting, or your life, or health.  These things have no intrinsic value; you must declare one and agree on the replacement.  How much is your life worth to you ? How about that rare coin ?

Cost Compromise

None of this comes for free.   All the physical security, third party proxies, records of transactions of ownership and insurance all cost.   The cost could conceivably be more than the value of the item you want secured.  Or maybe just more then you are willing to spend.   Maybe it simply isnt worth the cost or effect and you take your chances and you weight the cost vs benefit of "security" and decide  what cost is worth the benefit to you.

Intellectual Property

What about "Intellectual Property" ... Thats a funny term.  Think of IP as a thought, or rather the recording of a thought.  It might be a recipe, a picture, a movie, software.    Like physical, tangible, items, IP has a value.   Unlike Physical items however, the cost of copying the property is generally small compared to the cost of creating the original.   A copy of intellectual property may be  completely indistinguishable from the original.  This is an interesting property that some tangible items share.  An extremely good copy of a say a gold coin may be just as valuable as the original.   But then there is no such thing as a copy of your life.   There is also the issue of theft vs copying.   If I take a physical item from you, I deprive you of it.    If I *copy* an item I may or may not deprive you of its value.    If I even *see* something I may deprive you of its value (for example a photo of you committing a crime).
So what is "security" of intellectual property mean?  Is it similar to tangible items ? 

Start with Nothing

Lets take for example a simple IP - an event.  For example you ran a red light.   If you want to secure this event whats the best way?   Dont do it.   If you dont do something then it can't be taken from you.  It is secure.    "Freedom is nothing left to lose".   And if you do run the red light certianly dont take a picture of it!  Learn from our congressmen.    Any events that dont occur you dont need to secure.  Any events that arnt recorded are secure.
Simple right ?  Or is it.
What about the traffic cam you didnt see ? Simply not recording something yourself doesnt secure you, it can be recorded by others.   So not recording something yourself is not sufficient.  You must make sure it is not recorded by anyone.  Ever. 
But what if there was no recording?
Suppose the police *claimed* you ran the red light.  You are not secure!  You have no proof that you didn't do it.  But if you had a picture of the light at the time claimed and it was green, you are secure.  Or if you have a record of you at a different place at the same time you are secure.
So for security against events that never even occurred you need IP!  This is why many people in Russia have camera recorders going all the time.  To protect themselves from events that didn't happen.
So to secure an event, you need both records of it happening (to assert the positive) AND records of it not happening (to assert the negetive).   This may ultimately be why "life bloging" devices start getting used - Not just to record events we want to remember and share later, but to protect ourselves (and others) against accusations of events that didnt occur or occurred to others who were not currently recording them.

Pictures

What is the classic example of a record of an event?  A  Picture.   How would you protect it ? First off, what are you protecting against ?  Do you want to make sure noone gets it without your consent? Or do you want to make sure it is never lost?   Or both?
To make sure a picture is never acquired the best solution is to never take it in the first place.  Second best is to destroy it and all evidence of it and that it ever was taken.   If there is evidence that a picture exists or existed that may be less secure then either the picture itself or nothing at all.  Amazing, but consider this.  If during the security checkpoint on a trip the boarder patrol finds encrypted files on your disk, you may be under more suspicion (less secure) then if they found the pictures and they were clearly of your sunny vacation from Mexico.     The absence of a murder weapon from its case may be more suspicious than finding the weapon itself (unused).

Privacy

The above examples of the modern world lead to some uncomfortable conclusionsThe same with tangible items where you trade value and usability for security, for your life You may willingly trade security for privacy.   In fact you already are giving up your privacy, likely unknowingly or unwillingly, by the existence of omnipresent surveillance.  Every time you go into public, make a purchase (even with cash), travel, almost every action leaves a trail of electronic bits.    You simply cannot prevent this.  Even trying to "leave the grid" will leave a trail of your absence.
Since you cannot stop the collection of data about your every life, one way to combat some of the negative consequences is to collect your own.   The more evidence you collect about your daily life  and control  yourself  the more secure you are against misuse of information and the more control you have over your digital life.    Securing yourself against events you are willingly involved in which you dont want others to find out about is a different topic entirely.   But protecting your life against misuse of information collected by others can be improved by collecting as much information you can yourself and securing  it.  And at the same time you may also be protecting others by providing an independent information source.

The Tradeoff for Securing IP

For whatever reasons you have collected a lot of intellectual property (pictures, videos, emails, software, documents, sales reciept etc).    And for whatever reason you've decided that securing this IP is important.  As with physical items there is always a tradeoff, always a cost, and always side effects.

Delete it

As mentioned before, one way of protecting  IP is getting rid of it.  None can take what you dont have. For some, that is a good idea.  It takes a lot of effort to do however, and actually its impossible to completely eliminate your e-trail.   Its even extremely hard to do a half-decent job.  But suppose you could, and you delete all your emails, all your photos, even destroy your old hard drives, phones etc.  Lets forget for the moment that copies of most of this stuff is still lying around somewhere on internet servers and the NSA's private vault.   But for your purposes you have secured your  IP by deleting it.  But now it is unusable.   If you've done your best to get rid of things so noone can use them, then certainly you cant either.  Thats one way to go, but in fact you have not really protected yourself against dedicated e-hungers but you have completely removed all value to yourself.   Thats generally not the compromise most people want.

Encryption

Encryption is a false god.  It is often touted as a panacea of IP security.   The idea is good.  You encrypt your data then no one else can get it but you.    The problem is in the details.  In one way encryption is like a safe.   Only you (with the key) can get in.  But also like safes, there are safe crackers and there are lost keys.   In order to use the data you need the key.  The harder the key is to find (or remember) the harder it is to use the data.   Roughly, encryption can be thought of like locks.  The better the encryption the harder the lock to open but the less accessible the data.  Some locks, like your car or front door,  are to prevent casual thieves but wont deter a persistent professional.  On the other hand, if you lose the key you can call a locksmith or look under the mat for the spare.   Really good locks can be like Fort Nox.  Really hard to use, really expensive and noone is going to get your data - probably not even you because if you lose your key your toast.
Consider also what is the purpose of securing your data? Is it to prevent it from being accessed ? or is it prevent it from being lost ?  Encryption can be a good tool, but needs to be used with care because it actively makes it harder to get at your data and a higher risk of losing it all together.

Duplication

Sometimes the  best kept secret is the one you tell everyone.     This is actually used by security agencies if they discover a skeleton in your closet.  You write a letter to everyone you care about admitting to it then you no longer are at risk for blackmail.   Your secret is no longer a security problem because its not a secret.
IP can be the same.  If you copy your data all over the place you are less likely to lose it.  You are secure against data loss.   Interestingly you can actually improve ownership by giving it away.   If you embed in your data your claim of ownership (for example a copywrite notice)  then it becomes harder for anyone to claim it was theirs.  They can remove the notice themselve  but it is very hard to remove it from all copies all over the world.    Similar to Life Event, by trading privacy (hiding) for security you are actually increasing your security against misuse of information.    The security you gain is both security against physical loss and at the same time the security against misuse of information.
But like Life Events, you probably shouldn't use this technique to secure IP  you dont want people to know about.

Insurance and IP Law

Like physical property, IP can be secured with insurance.   Like physical items, complete loss may not be recoverable by identical data, but loss of value may be recovered by forms of insurance.  This might be insurance from a IP provider against loss, copyright infringement lawsuits for stolen data or actual IP insurance from an insurance agency.    Loss by either physical loss or loss of value can be mitigated.  And like with physical property everything has a cost and it is up to you to decide if your IP is worth the cost of the security or insurance and how to make the right compromise.

Conclusion

There is a lot more to security then can be covered in this discussion but there are some important takeaways.
·         Security is not a simple concept.
·         Security can be protection from loss or protection from discovery. These are often at odds with each other.  
·         There are intrinsic tradeoffs of security vs privacy vs costs vs usability.
·         Information security has similar properties to physical security.
·         Collecting information yourself and managing it can help protect against misuse of information collected about you by others, adding to your personal security.
·         There is no single solution but there are methods and considerations to maximize protection and mitigate loss.


Monday, August 13, 2012

Big Changes in the Markup World

This year I attended and Presented at Balisage (http://balisage.net/).   I have been going to (and presenting at) this Geekfest for years although a small slice of the time of many attendees.   For those who don't know, Balisage is THE Markup and XML conference of north America.    As things do, over the years things have changed ... but this year I really felt a groundswelling of change rather than surface changes.

A few years ago this started with a sense of "Is XML Dead" ... then came the "JSON Is Taking over" ... But now I get a feeling the tips of the changeberg have melted and merged and matured.   The "fight" of HTML vs XML vs JSON seems largely over and subsumed by "How can we work together".  Many discussions of how to compromise or change data models to work better with different formats rather than to force the other side over.  XML seems as solid as ever ... or even more so ... after seeing what big agencies like the US Navy and the CDC are doing ... XML is no longer a new technology, it is the foundation of huge complex systems - but by that it seems not particularly interesting in itself but rather the complex systems built out of them.

Purity of format doesn't seem as dominate as much as the need to adapt to complex and heterogeneous systems where XML is a core part but often not the beginnings or the ends.   XML has found its place strongly in the complex document area and the tools are widely used.   The change I feel is a shift of focus from XML as a Technology, to Processes, Integration and Complexity management.   I think the next few years will see some deep shifting sands in things like data models and tools - not cataclysmic changes but rather adoptive changes as competing technologies, processes and people work out what it will take to handle the next level of complexity.   People seem less religious about the shape of the angle brackets and more interested in how to get work done.

At least that is my view.


(fyi my presentation is here :  http://www.balisage.net/Proceedings/vol8/html/Lee01/BalisageVol8-Lee01.html )


Pictures care of Syd Bauman
 https://www.dropbox.com/sh/8jsm7zom62gs88c/_u_to-acyS?m#/





Wednesday, May 30, 2012

Speaking at Balisage 2012

Just in ! I will be speaking and presenting a paper at Balisage 2012.
http://www.balisage.net

Details to follow after the public announcements are published.

Come join markup geeks for the best conference in the world !

Tuesday, March 6, 2012

Be Excellent


A few months ago I was at a concert.

A mind-blowing, incredible awe-inspiring amazing kind of concert.  The kind that sends you to another world you didn't know existed. The kind that you're embarrassed to describe to people because they think you must be insane or on drugs. The kind I suspect few ever experience or know could possibly exist.

But its true.  Hit back if you don't believe.  There is nothing more for you here.

At some point I realized I was disconcerted,  nearly depressed.  Why?  Because while I think of myself as a kind of closet musician;  there is no way I could achieve this. The level of excellence so far surpasses my ability as to humble and shame me.  The excellence was so far beyond what I could ever achieve even if I were to pursue music and art full-time I would never come close.  Ever.  It was the Mozart to my Amadeus.  To my pretense of Amadeus.  A diamond to my cheap fake glass.  I felt bad, inferior, ugly, insufficient, ashamed, a sham, a fraud.  I  couldn't tell anyone because admitting such inner feelings was too shameful.

Yet here I was at an incredible event with incredible people, both performers and audience.  And both enjoying it beyond description and envious and sad because I could never achieve what I was experiencing.
Like Lucifer was tempting me with the joy of the impossible hoping I would succumb to the sins of envy and shame.

Then I realized:  Be Excellent.  How  can I possibly feel bad in the presence of such masterpiece? This is a crack in the doorway to heaven and the light shines so brightly so I am blinded and yet I feel bad because its not me? Because I could not do that?
What an amazing tower of egotism such that I cannot appreciate the accomplishments of others because I am incapable !

Then I realized:

Be Excellent.

I can be Excellent.

It may not be in this form or expression, but in everything I do I can be as excellent as I possibly can.  It may not have the rare majesty and skill of performance art, but the beauty is in the creator as well as the creation.  We all can be excellent even if no-one but us experiences it. But more.  We *must* be Excellent.  What is the point of life if not to be the best we possibly can be. Anything less is a waste of life's miracle.  Every second that passes where we are not the absolute best we can possibly be is a waste of the most magical event that has ever occurred in the universe. Whether you believe in the creator or the emerged ... life and consciousness is the most amazing thing ever and we waste every second of its precious minuscule finite mortality unless we are excellent in every way. Always. Every Moment.  Every thought.  Be Excellent.

I left with the best understanding ever, and I can only hope to hold onto it.

Be excellent.

You can do it.  you must.

Saturday, April 23, 2011

Failure is the new Success

This week was a real eye opener in the IT world. Amazon's EC2 cluster suffered a massive meltdown (just Google Amazon EC2 I wont bother with the links). The media was all over it about how half the internet went down and major sites were DOA for 24-48 hours ... Massive Collapse. Like the Japanese nuclear reactors (except the part where the internet being down didn't actually hurt anyone except financial ... or did it .. but thats another story).

But like the Japanese disaster, there is a hidden success story. The world didn't come to an end. The Japanese reactors didn't actually explode in an apocalyptic meltdown and kill everyone on earth turning it into the green glowing dawn of the living dead. The system was actually contained and it was mainly the news media which focused on the drama while ignoring the real tragedy of 12,000+ dead due to a tsunami. I guess thousands dead due to a natural disaster isn't as exciting as no one actually dead in a nuclear reactor that didn't actually explode. "But it could have! " ... well it didn't. The hidden success in the Japan Disaster is that in fact the reactors stood up to 9+ quake AND a tsunami.

Similar with the Amazon story. The media wants us to think the cloud suffered a meltdown taking down the internet and we cant trust it anymore. The reality is that Amazon worked exactly as advertised ... well maybe off by a few 9's but whats statistics. The reality is that a single geographical region suffered a 'network meltdown' taking down the whole region for a while, then within 12 hours only a single 'availability zone' As of now (2011-04-23 19:39:00 EST) Amazon is mostly but not entirely up. But it actually worked ! Other zones were unaffected. A single geographic zone went dark for 48+ hours but the rest kept chugging on. And in fact it seems that all data is restored, nothing lost.

So who suffered ? those vast number 'mainstream internet sites taken down' ? Well the ones that suffered were the ones that didn't plan for failure. Netflix is based on Amazon but kept on chugging because it was designed for the conditions Amazon advertises. That is a single geographic region might actually die so don't put all your chickens in one basket. The people that didn't understand that were hurt. Those that embraced failure rode the wave. Even those that didn't embrace the failure were just down for a while and amazon restored their data and servers from redundant storage ... it just took a while.

So whats to learn ? The fact is there is no totally perfect system. I don't think any company could build a data-center better then Amazon ... Google and Microsoft might equal it but no ones perfect. Failures are going to happen. Period.
Amazon was designed for geographic redundancy and exposes the necessary API's to take advantage of it. If you don't, its your fault.
well maybe ...

Maybe not. The whole Cloud Computing concept is based on redundancy and trust in large scale distributed systems. Its the outsourcing of the IT department. Why should individual developers who subscribe to a cloud system be required to manage this ? While I agree that Amazon performed exactly as it advertised, and that if you planned for it and took advantage of its capabilities you'd have rode out the storm just fine ... it points to a weakness in the system. I argue that cloud computing should hide this from you. Its an artificial artifact that a "virtual machine" actually resides somewhere physically and that you have to care. The next generation of cloud computing should hide this from the users much like the Amazon S3 storage (which hides its physical location), the EC2 and EBS system should be able to migrate to different geographic locations without the programmer having to architect dynamic load balancing, fault torrence and hot swap failover. Isn't that the whole point of "the cloud" ? To let the 'big boys' figure that out and leave us to writing apps ?

Maybe thats the real point of clouds like Google Cloud Computing ? Its time Amazon wake up and accept its awesome but still needs to go the extra mile. If the biggest web sites crashed because they failed to make use of the advanced features of cloud computing, maybe its time to make those features less "advanced".

Tuesday, March 22, 2011

Patience



What is Patience ? I've noticed a pattern when I show off my new hobby, model trains (http://trains.calldei.com).
Almost universally the comment is "Wow, that must take a lot of Patience, I could never do that". At first I thought I understood, as certainly it takes a lot of *time* and *effort*. But reflecting on this, I cant understand the term "Patience".

This takes no "Patience" whatsoever. To contemplate, how can a hobby or an art or work or similar thing take "Patience" ?. More so then say Jogging, or watching TV or reading. Raising children, now THAT takes patience ! But doing something which every second is enjoyable and at the end builds something. That takes no patience. It doesn't matter how long it takes. The longer the better. I actually worry that I'll complete it and have no more to do.

Watching Television, 40 years later and you've accomplished nothing. That doesn't take patience ? but building something does ? I guess I don't understand what people are thinking. Perhaps its as ancient and almost cliché as eastern philosophy about the journey not the destination.

I look back at my life at the things which I have a problem with Patience. I consider myself *very impatient*. I cant stand sitting around at "parties" while people invoke "small talk". Give me a gun I could shoot myself. Waiting for a delayed plane. Waiting for my life and career to progress to the point I could afford a house. Driving in traffic. Waiting for a shipment of a new train part! Trying to persuade someone who doesn't agree with you. All those things take "Patience". But actually doing something ? Immeshed completely in the act of doing something that takes enough of your senses to distract and absorb the ongoing ramble of an overactive brain ? Thats not patience.