I’ve just moved my blog to a new location http://johnny.io, you can see, how, why and when by reading my last post on “Jekyll is rocking my new blog”

I won’t be posting here anymore, so if you want to keep reading me point your rss feed to http://feeds.feedburner.com/johnny-halife or bookmark http://johnny.io

bye,
~johnny

As I would expect people that attended the talk were senior geeks which engaged with the topic and (hopefully) enjoyed a from-scratch writing of a Testing Framework =).

I want to thank everyone that assisted and Miguel Saez for the organization of the Event which as usual was flawless.

As promised here are the materials from the talk:

• Download the presentation from here
• Download the samples from here (same code I shown)

About a month ago, I left my home in Buenos Aires to start working with James Conard and his team on the “Cloud Convergence” Demo and some other PDC 09′ Stuff. Today since Ray Ozzie touched the stage we (backstage) were waiting for our demo to become true. Before that were a couple of demos, and specially (and I’ll talk about this later on this post) “Tailspin Travel”.

When we got to Redmond area, we met Jonathan Carter (a.k.a JC) who was working with PC (a fellow southy), they were working on the “Tailspin Travel” demo for the keynote that was shown by Cameron Skinner. Tailpin’s demo represents most of the “best practices” that you should apply when doing an ASP.NET MVC Web Application that has features like back-end support with Workflow and other WCF Services, and integrates with your Enterprise Identity using WS-Federation Protocol.

Imagine how good the demo was, that we took that same application to do our work on the “Future of the Platform”. Additionally, they are doing something great with the application that since the keynote was available for you to download it and learn the “goodies” of .NET 4/WIF/AppFabric development.

Tailspin Travel

The Tailspin Travel application covers a pretty substantial set of functionality, but ultimately seeks to provide a holistic perspective of how Visual Studio 2010, .NET Framework 4,

and the server platform can be used together. Among other technologies it leverages: Visual Studio 2010 (with the new fancy diagrams), .NET 4.0 (MVC2, WIF, WF, WCF, and so) andshow the hosting layer of your services using the brand-new “AppFabric”

I guess at this point of the post you should be rushing to get the bits, and if you don’t go new to get them at http://tailspintravel.codeplex.com/.

Meet me at PDC

I’m on PDC with other fellow Southies, we are all the time around, and if you want to talk and share a coffee, just look for me or DM message at @johnnyhalife on twitter and we will meet for sure.

Today I’ve been interviewed by Jean-Christophe Cimetiere about the work I’m doing with Ruby and Azure. You should stay tuned for more information that I’ll be announcing soon on these things, like Azure 2009-09-19 support of the API on the Ruby SDK.

Acknowledments

Although Matías , PC and I where backstage, there were lots of Southies that made these PDC events work great, I sincerely appreciate the work all the Southies were doing, and how do they helped us.

hope to see you around,

~johnny

Symptom
While trying to connect to Windows Azure (Storage at least) from Argentina, you get 403 errors even dough the message seems to be formatted correctly.

Issue
Argentina was supposed to change the time by adjusting it to DTS (Daylight Time Saving), one day before that happens the Argentinean Government decided not to make that change, generating a lots of implications for computer systems adjusted to DTS.

Since Windows Azure relays on UTC Timestamp for making an assertion on the signature, the current time for GMT -3 isn’t what is expected from the server side causing the whole message to fail after the an assertion of the signature.

Workaround
There’s no apparent solution yet, but there are a couple of workarounds in order to properly develop against Windows Azure from a not changed GMT -3 time zone.

• Change your time zone to GMT -4. I switched my computer back to Halifax – Canada that has GMT -4 for an Windows Azure started accepting my requests
• Keep your computer with the clock one hour ahead . Keep the current time zone without tweaking the Date/Time, but remember that you will be out of sync with the country

Both of the workarounds listed above proved to be successfully working, I will stick with GMT -4 since I guide myself (eating, sleeping, and all that) in my computer clock.

Hope it helps,
~johnny

I started thinking and looking around some samples from DataMapper and other ORM frameworks that have the ability to override the current context by calling a method or specifying connection name. I looked deeply into it, but end up thinking that it will be better to have a little stack of connections in order to handle different contexts.

I end up writing a new method to WAZ::Storage::Base that enables you to scope your code to some specific context and once that context is left you get back to the default connection.

Consider the following sample in order to better understand what I did

  WAZ::Storage::Base.establish_connection!(:account_name => 'name', :access_key => 'key')

  # any operation performed here will be on the default context expressed above. e.g.
  container = WAZ::Blobs::Container.find('my-container')

  # So, now let's suppose that I need to perform an operation on another context,
  # but those are just a few, and very scoped that isn't worth switching the whole
  # context to that. Leveraging the new method, I'll just do:
  WAZ::Storage::Base.establish_connection(:account_name => 'another', :access_key => 'key') do
     # any operation that I perform here, will be on this new context. e.g.
     another_container = WAZ::Blobs::Container.find('my-container')
     # Note that the operation above is performing a look-up of the container on another
     # storage account.
  end

  # Now that I've left the block, I'm back to my original context.

This new feature has been added to project at github.com/johnnyhalife/waz-storage and the new documentation is also available at waz-storage.heroku.com.

You can upgrade your gem, in order to have this new feature or just using the bleeding edge by cloning the repository on github (git://github.com/johnnyhalife/waz-storage.git)

Another small change

If you are (or tried) using waz-storage on Heroku.com without rails, you will realize that some exceptions about ‘String.start_with?’ being undefined may appear. I’ve added an snippet to implement this method, on the rails way, when loading the waz gem (if it was previously undefined).

thanks,
~johnny

For the project source code and pretty basic documentation go to http://github.com/johnnyhalife/waz-storage.

For the whole API documentation refer to http://waz-storage.heroku.com.

What is Ruby Windows Azure Storage SDK (a.k.a waz-storage)?

A simple implementation of Windows Azure Storage API for Ruby, inspired by the S3 gems and self experience of dealing with queues. The major goal of the whole gem is to enable ruby developers, like me =), to leverage Windows Azure Storage features and have another option for cloud storage.

The whole gem is implemented based on Microsoft’s specs from the communication and underlying service description and protocol (REST). The API is for Ruby developers built by a ruby developer. I’m trying to follow idioms, patterns and fluent type of doing APIs on Ruby.

This work isn’t related at all with StorageClient Sample shipped with Microsoft SDK and written in .NET, the whole API is based on my own understanding, experience and values of elegance and ruby development.

Full documentation for the gem is available at waz-storage.heroku.com

Scenario Idea

The gem was thought having heroku.com in mind as it is my main Ruby hosting vendor. The basic idea can be expressed with the following diagram:

As you can see there’s no need (and no personal desire) of hosting your application in Windows Azure or even writing it on .NET (or Windows needed at all).

How does this differ from waz-blobs and waz-queues?

Well, this is a sum up of the whole experience of writing those gems and getting them to work together to simplify end user experience. Although there’re some breaking changes, it’s pretty backward compatible with existing gems.

Open Standards

The waz-storage model is built upon the REST API published by Microsoft for Windows Azure Storage Services following the HTTP Standard. It’s basically a bunch of HTTP Requests with a special signature and some custom headers. The same API can be ported to any platform supporting a decent HTTP Stack.

So, what?

As in the latest posts, I won’t include too much detail here since it’s all explained at github.com/johnnyhalife/waz-storage and the whole API is detailed (and documented) at waz-storage.heroku.com.

The future is bright, I strongly appreciate all your feedback, your tweets and the time spent on (at least) reading the code. I really love this whole experience, but I won’t stop here, I’ll be relaxing from this SDK for a week or so (maybe less) until I come back for doing Tables support and fixing the issues that users may have encounter.

TODOs’

There should be some samples or at least one Reference Implementation of the API, probably I’ll be requesting help from other people in order to accomplish this, but I promise that some sample will be soon online so you can see the unveiled power of the Windows Azure Storage when running on Ruby.

Also there’s some support for Blocks that will be added, and some other sugar features for the application.

Time will tell, meanwhile you can comment, tweet or say whatever you want since I’ll be listening to your feedback.

Acknowledgment

This time I’ll make an special acknowledgment, besides my team, teammates, friends, and collegues at Southworks, Ezequiel Morito and Juampi Garcia. I want to thank and recognize that this work won’t be online today if I didn’t receive the advice from Martin Salias who encouraged me to go further on spiking and publishing my spikes.

Thanks Martin, I really appreciate your support and guidance.

Pro’s

• Blobs and Queues handle the same type of security, connection and headers. They share the core service called WAZ::Storage::SharedKeyCoreService
• While I was doing some work on Queues I found some bugs on connection management and URI management on the CoreService, so those changes weren’t propagated to Blobs (yet).
• 99% use case of the gem is that people handle a single WAZ Account for everything (Queues, Blobs and Tables) so have to connect, configure and consume them from different gems seems lot of overhead.
• Maintainability Index really drops on having them split.
• This is a single man’s work and even worse is doing over my spare time (so I cannot keep them both up-to-date always).
• Total gem size is 50KB compared to two gems of 30KB (so storage isn’t a problem here)

Con’s

• At some point seemed cool and very aligned with YAGNI that you need to install only what you need (queues or blobs)
• Shared behavior could be inherited from a third gem called ‘waz-storage-core’ but that will require more maintainability work.

Those are the things that came to my mind while working on this, so I decided to glue them together, refactor them and generate a consisten API for both services (and the tables soon enough).

Towards the SDK

Ruby devs aren’t usually required to have SDKs that’s more of the Windows World. SDK is the name I decided the whole thing will have: documentation, sample, gems and source code (Yes! I’ll do more docs). I’m looking for a consumable way for Ruby Devs to get involved on Windows Azure (since it’s worth doing it) but also there’re a lot of .NET Developers that may want to have a look at Ruby’s features and opportunities outside Windows World.

This is work in progress

Although I didn’t officially released anything related to this new SDK, I started my own work towards completing the API, so every piece of valuable feedback you might have will be really appreciated! I really enjoy while doing this and probably will do better as time goes by but as you can see on the pictures, I already started, merged and have a pretty decent TODO’s to apply to this thing and probably after that I’ll be releasing another version.

From the picture on the right you can infer, that I’ve already merged the test suite, and the TODO’s. The whole project is now a single thing. These are the things that I’m considering for next version

As you can see, there are things that still need to be figured out, but with time (no much) , they will emerge and I’ll be releasing as soon as I have them.

Help wanted!

As I mentioned now and on the previous posts, this is a one man’s work but you can change it =). I’m looking for Developers (juniors, amateurs, pro’s, whatever) to contribute on the project as they wish. If you want to test the code base, write a sample, write some docs, or whatever you’re invited! Drop me a message at @johnnyhalife and we should get going soon.

that’s it, stay tuned for more! (very soon)

thanks,
~johnny

Like the previous API, it’s 100% pure ruby, relays on RestClient written by Adam Wiggins and was written and tested on Mac and Ubuntu.

As on the previous post I wrote about the Blobs API,  this post isn’t about API documentation. It’s about all those things that forced me to write the API and some other random thoughts about where this whole thing is going.

Oh, you did it again!

Yes! I love it, interoperability is my passion, I love to write things to demonstrate that they can work with other very different things, and given the fact that Microsoft did a great job when they published the Windows Azure Storage Queue API on line, I decided I’ll spend some time building this. Queues are great, async programming is cool and REST is something that people do not believe until they see it (for those skeptical devs this is fully implemented over HTTP Stack on pure Ruby)

There’s nothing worse than waiting inline. Bummer!

Welcome to the twentieth one century, we live surrounded by the buzz and the current one is called “Cloud Computing”. Although I won’t discuss the topic here, it relays on a very simple principle “pay-to-play”. If you want to handle more requests just increase the number of workers and decrease the number of money you’ve got on your pockets.

Not all the operations on a System need to be always consistent, Amazon cart is an example of that. When you don’t want to pay more just get a queue, do some work async, and voila! your problems are solved.

Some hosting platforms like Heroku o Windows Azure have a limited number of requests that they can handle with a single machine/worker/dyno/you_name_it, when the backlog becomes to big they start to deflect request with HTTP 500 messages. If you relay on a queue you will also have a way to increase responsiveness on your app, sites like digg and twitter.com already do something similar.

Why do I need Azure Queues?

It’s an extremely powerful alternative to having a File System Queue or archaic Queue Systems like MSMQ, it’s on the cloud, it’s unlimited storage growth and can be used from either your current hosting or Windows Azure.

The whole Queue Service provided by Windows Azure it’s really simple but not least powerful. Possibilities are endless, and you can do whatever you want and create a multi-party system that has your web app hosted on Heroku written in pure ruby, a back end service running on Windows Azure written on C# and a python app hosted on Google Apps engine that uses Windows Azure Queues to communicate with each other. There’s no limit! (Even your mainframe can handle WAZ-Queues with some C HTTP Lib + playing nice with sockets).

Ok, I buy it, now show me your magic

The API is pretty simple, simpler than anything I’ve written before. Here’s a snippet that illustrates basic API interaction (or in other words, everything you need from a queue)

Getting started

sudo gem install waz-queues --source http://gemcutter.org

Reference Code

  require 'waz-queues'

        service = WAZ::Queues::Base.establish_connection!(:account_name => account_name,
                                                          :access_key => access_key)

        # excepts that the metadata for the queue changes this method behaves as PUT
        # remarks: it performs a validation whether metadata changed or not (HTTP 409 conflict)
        queue = WAZ::Queues::Queue.create('my-queue')

        10.times do |m|
          # enqueue a receives string. Message content can be anything up to 8KB
          # you can serialize and send anything that serializes to UTF-8 string (JSON, XML, etc)
          queue.enqueue!("message##{m}")
        end

        while(queue.size > 0) do
          # Since WAZ implements the peek lock pattern we are locking messages (not dequeuing)
          # it has two parameters how many messages and for how long they are locked
          messages = queue.lock(10)

          puts "dequeued message: #{messages.size}"

          # deletes the message from the queue so other clients do not pick it after
          # visibility time out expires
          messages.each {|m| m.destroy!}
        end

What’s next

While doing this I figured out that there’s a common behavior shared across blobs and queues, you can see it on the code that I started doing some merging while creating a ruby module called WAZ::Storage::SharedKeyCoreService that handles everything related to connections, requests, headers and signing messages. I’ll be merging the whole thing into ‘waz-storage’ gem that I’m planning to start over the weekend and create a small reference application to illustrate better what does this do? (if you wanna join ping me via twitter @johnnnyhalife)

That’s it, I won’t say I’ll do tables API too, I guess that after this week you can figure it out by yourselves (although it might take more time)

Special Thanks

I’d like to thank the following people that help me with something more important and painful than designing the API or writing the code that this time was an individual experiment. The following people are listening to me in every spare time I have at work, talking about why I design something like I did, how cool it is, or how do I enjoy this, which can be even hard than putting this code together!

These people are: Matias Woloski, Ezequiel Jadib, Federico Boerr, Ezequiel Morito, Sebastian Renzi, Martín Salias and Pablo Costantini.

That’s all folks. Thanks,
~johnny

Yes, it’s 100% organic Ruby code, there’s no strange Microsoft library that you need to consume and even better it was written and tested on Mac OS X and Ubuntu 8.10.  This post is about the motivation, and design process I’ve taken. Here you will find a minimal reference to the code, if you’re looking for the bits go straight to http://github.com/johnnyhalife/waz-blobs/ that includes the whole API documentation.

Motivation

Interoperability, fun and inspiration! Those three words are the base of this whole experience, from those OSS Fan thinking “why would I go for Windows Azure Storage? I’ve have my App written 100% ruby and runs on my custom built linux distro!”, here is the summary of the thinking I’ve been doing when I was trying to understand whether to write the API or not:

• It’s solid. Although it’s on CTP, it has been out for almost a year and during the different version it has demonstrated some basic fundamentals covered on Distributed Storage and the speed it’s really good.

• Ruby’s alternatives for serving static files isn’t great. I love Ruby, it has slowly became the language of my preference on the last year, but we have to admit that serving static files it’s better when outside our Ruby App server.

• It’s interoperable. As I started this post, the code available at http://github.com/johnnyhalife/waz-blobs/ is 100% organic ruby code, it’s based on the RestClient API written by Addam Wiggins from Heroku that can be found on http://github.com/adamwiggins/rest-client/. The code is based on the WAZ Rest API available at http://msdn.microsoft.com/en-us/library/dd135733.aspx

• It works great from Heroku. I spend my spare time hacking on pure Ruby and most of my project end up on Heroku (by far the best Ruby Hosting Platform on the Clouds). As you can see on Heroku Docs, their recommendation is to go for AWS-S3, and I don’t like when my choices are limited, I wrote the gem just to have an alternative for S3 when hosting on Heroku.

Getting started with the waz-blobs API

First of all, if you just want to use it just get it from the gems repository like the following:

gem install waz-blobs --source http://gemcutter.org

And here’s some basics

  require 'waz-blobs'

  # Save your configuration info
  WAZ::Blobs::Base.establish_connection!(:account_name => "name", :account_key => "key")

  # creates a container
  container = WAZ::Blobs::Container.create('my-container')

  # stores a blob with custom properties (metadata)
  blob = container.store('my-blob.txt',  'this is the content of my blob', 'plain/text')

  # return a specific blob from a container
  blob = container['my-blob.txt']

  # retrieves a blob value
  blob.value

The complete API reference and source code can be found at http://github.com/johnnyhalife/waz-blobs/, feel free to download, fork it, and submit patches.

Next Steps

During the next days, I’ll continue using this and probably refactoring or bug fixing. In the near future (don’t know how quick yet), I’ll get the WAZ Queues API too.

While I work on Queues, fork me on GitHub and consider yourself invited to submit as many patches as you want.

thanks and read you soon,
~johnny

Software Development was also different then, it was a pretty streamlined process where a Requirements Engineer went to an office, analyzed how people behave on their daily basis, and then some programmer put together a couple of lines of code just to ease those proceseses.

That was an application, people thought about computer programs as a bunch of CRUD screens, that eased their daily work by cutting the clutter of working with papers. But, of course, it had problems since people was limited on what they can do with it, and innovation did not come from that side.

Not everything was bad, since it was the beginning of a new era, what we call today “Emergent Behavior” started then not as we look at it today but those fixed forms application made people find a workaround for fulfilling their on business needs.

Emergent Behavior

Let’s start easy by defining it, according to wikipedia:

An emergent behavior or emergent property can appear when a number of simple entities (agents) operate in an environment, forming more complex behaviors as a collective. If emergence happens over disparate size scales, then the reason is usually a causal relation across different scales. In other words there is often a form of top-down feedback in systems with emergent properties. The processes from which emergent properties result may occur in either the observed or observing system, and can commonly be identified by their patterns of accumulating change, most generally called ‘growth’.

Now let’s analyze the definition by applying it to Software Engineering:

Peter and Sally work on a company, both of them are sales team. They have a system that let’s them load their sales and then have a report of sales or a subgroup of them by just making a search and pressing the report button.

One day, Sally decides that it would be great if they can have the same sales report they do daily but independently for each distribution hub of their company. Their system doesn’t have that ability but they figured out that if they label each sale with tag like “[phone-sale]” or “[store-pickup]” they will be able to easily perform that report leveraging the search + export functionality (previously mentioned).

From now on, they have enhanced their process, added value to the business by just innovating with the existing features of the application, thew will simply label sales, search for that label and make their report.

On the previous sample, we have seen how people may react to a system and how they can outperform by just using whatever is already on the system.

We as software people (architects, developers, managers, whatever) tend to think that our system is complete and has 1:1 mapping with the real business. But that’s often (if not always) misleading. If we were the architects of the system on the sample, what we would do? 80% (just an estimation) will say: “Oh that’s something that the sales team didn’t tell us we should probably go and build that feature into the software so we ease their life”. In my opinion that’s terribly wrong.

Nowadays the rate of change of the equation of people + business + information moves faster than never, we sometimes confuse the domain of the problem with a particular problem of an application. In the case above we should be probably look for a solution that helps people to manage their data in an unstructured way as they want (like tagging). Instead of adding a new layer of complexity to the system, we will just enable people to adapt the system to their needs.

Enabling people and that’s what this is all about

It started on the early days with the Apple II and the VisiCalc, a piece of software the enabled people to read and write tabular data, then it was Microsoft Excel that enables people like my father to run their own software to run his own business. And the star of this is twitter, enabling communications that are valid (or at least important for a short period of time).

Why twitter has become so important?

It was the early days of the Web 2.0 when the creators of twitter published it, it didn’t have lot of sense to have an online application that enable people to write up 140 characters and select people that will be able to read it and also having the ability to read others tweets.

It did not make sense until the iPhone came out, and that was a boom, that 140 characters enable people to say where they were, what they were looking at, or how they were feeling. Lots and lots of services came out like posting pictures or geo-location references, +100 of clients were written for it, and twitter stayed the same.

Twitter pals started looking at what people were doing, at the results of the collective intelligence, for example the hash-tagging for trending topics or people looking for those trends on the web. And they just worked out those parts.

Now twitter also servers as the #1 Information Delivery Mechanism for people all over the worlds, digital media is into twitter and even stock prices are notified via twitter.

What did twitter do? They just enhanced application aspects like performance, searching and API tweaks not modifying their core functionality for sake of having more functionality,  just changed anything needed to enable more for end-users.

So what?

We live on the era where information flows too fast and too furious for us, more than we can manage and more than we can imagine. Back on the 90′s as I said on the beginning of this post probably projects like twitter might end up having a “Twitter Finance” or “Twitter Media” feature.

What’s wrong on adding features based on what people do? There’s nothing wrong about it, but what is often confused is what people is doing. For example, what do I, Twitter Stock Exchange and NY Times Publishing on twitter have in common?

From a 90′s Requirements Engineering point of view absolutely nothing. From a collective intelligence point of view, everything we are all trying to put information that has value for a short period of time and that sometimes falls under an specific trend or category. If I were twitter’s architect I will probably look into making twitter easy for the general use case (sharing information) instead of looking for adding an specific feature built-in to the system.

Getting started with Emergent Behaviors Design

“We are what we do”, and that’s it. No matter what we thought for the application intend it will end up being what people are doing with it. A system is only the result of what their users do.

Having the previous paragraph, let’s create a small list of things that you have to consider when designing for emergent behaviors aware applications:

1. Know your domain. Excel knows that it’s domain is cells and columns and everything must be optimized for working on that domain and keep it as small as possible.
2. Do not confuse domain with features. If you looking into something, and trying to solve a problem try to differentiate what people is looking forward doing with it instead of solving the actual problem.
3. Do not map features and use cases 1:1. Try to enable more and do less, invest on what is worth and what will keep your system alive, there’s one principle that we cannot avoid when thinking of a software product “whatever becomes useless end up on the trash can”.
4. Define what you wanna do. Software lifespan is really reduced nowadays, enable the more for the time being “alive” and know that at some point in time will be better to back off and come back with a new product.
5. Constraints, constraints, constraints. Define a model, define a language, and define operations over the model. The rest will be open for end users but have definitions, everything isn’t a plain text (or formatted text file) business value comes from this constraints, and this is what will difference you from having your whole enterprise using Microsoft Excel for every application.
6. Be open minded. Once you set up your system constraints, do not try to interfere with how people interact with your application telling them “oh that wasn’t supposed to be that way”.
7. Open your API. Having people accessing your model will constraint it even more, and the result will be people leveraging your model, your language and the operations you defined in a whole different way, and that what generates business values (and evolves your business).
8. Keep a consistent interface to your data. Learned from Gmail’s label operator where you can search for email that is unread or fall under an specific category.
9. Have logging and tracing mechanisms. Understand how do your users interact with the system is what will help you make it better.
10. Plant a seed and watch them grow. Have an agile process, get your system on the field as soon as you can and start watching it, people may not realized that they have changed the requirements but your job will be to understand those behaviors and combined with the previous point will let you enhance your product as whole, sticking always to the reality.

Remember, there’s no silver bullet on software design, there’s no one-size-fits-all, and there’s no product that can last forever. When facing your next development challenge think in terms of what may people end up doing with this, which is my domain, or how will I enable people to restructure the data. Then just work on

Read you soon,
~johnny