Johnny Halife’s Blog

I was working on the Ruby Windows Azure SDK and I started receiving HTTP/1.1 403 errors, I figured out which is the problem so here you can get a simple issue analysis

Symptom
While trying to connect to Windows Azure (Storage at least) from Argentina, you get 403 errors even dough the message seems to be formatted correctly.

Issue
Argentina was supposed to change the time by adjusting it to DTS (Daylight Time Saving), one day before that happens the Argentinean Government decided not to make that change, generating a lots of implications for computer systems adjusted to DTS.

Since Windows Azure relays on UTC Timestamp for making an assertion on the signature, the current time for GMT -3 isn’t what is expected from the server side causing the whole message to fail after the an assertion of the signature.

Workaround
There’s no apparent solution yet, but there are a couple of workarounds in order to properly develop against Windows Azure from a not changed GMT -3 time zone.

• Change your time zone to GMT -4. I switched my computer back to Halifax - Canada that has GMT -4 for an Windows Azure started accepting my requests
• Keep your computer with the clock one hour ahead . Keep the current time zone without tweaking the Date/Time, but remember that you will be out of sync with the country

Both of the workarounds listed above proved to be successfully working, I will stick with GMT -4 since I guide myself (eating, sleeping, and all that) in my computer clock.

Hope it helps,
~johnny

Today I was discussing with my friend Juampi Garcia who is writing a nice sample application that leverages the waz-storage gem and some other interesting things. While we were talking about his application, he asked me for some way of overriding the default connection for performing a bunch of operations as he needed it for his application.

I started thinking and looking around some samples from DataMapper and other ORM frameworks that have the ability to override the current context by calling a method or specifying connection name. I looked deeply into it, but end up thinking that it will be better to have a little stack of connections in order to handle different contexts.

I end up writing a new method to WAZ::Storage::Base that enables you to scope your code to some specific context and once that context is left you get back to the default connection.

Consider the following sample in order to better understand what I did

  WAZ::Storage::Base.establish_connection!(:account_name => 'name', :access_key => 'key')

  # any operation performed here will be on the default context expressed above. e.g.
  container = WAZ::Blobs::Container.find('my-container')

  # So, now let's suppose that I need to perform an operation on another context,
  # but those are just a few, and very scoped that isn't worth switching the whole
  # context to that. Leveraging the new method, I'll just do:
  WAZ::Storage::Base.establish_connection(:account_name => 'another', :access_key => 'key') do
     # any operation that I perform here, will be on this new context. e.g.
     another_container = WAZ::Blobs::Container.find('my-container')
     # Note that the operation above is performing a look-up of the container on another
     # storage account.
  end

  # Now that I've left the block, I'm back to my original context.

This new feature has been added to project at github.com/johnnyhalife/waz-storage and the new documentation is also available at waz-storage.heroku.com.

You can upgrade your gem, in order to have this new feature or just using the bleeding edge by cloning the repository on github (git://github.com/johnnyhalife/waz-storage.git)

Another small change

If you are (or tried) using waz-storage on Heroku.com without rails, you will realize that some exceptions about ‘String.start_with?’ being undefined may appear. I’ve added an snippet to implement this method, on the rails way, when loading the waz gem (if it was previously undefined).

thanks,
~johnny

Hey, after talking, talking and talking about the possibilities of building a Windows Azure Storage SDK for Ruby, I finally got together a couple of ideas and develop the v.0.5. I’m still needing to write a decent sample application but now it has more resources that previous gems had.

For the project source code and pretty basic documentation go to http://github.com/johnnyhalife/waz-storage.

For the whole API documentation refer to http://waz-storage.heroku.com.

What is Ruby Windows Azure Storage SDK (a.k.a waz-storage)?

A simple implementation of Windows Azure Storage API for Ruby, inspired by the S3 gems and self experience of dealing with queues. The major goal of the whole gem is to enable ruby developers, like me =), to leverage Windows Azure Storage features and have another option for cloud storage.

The whole gem is implemented based on Microsoft’s specs from the communication and underlying service description and protocol (REST). The API is for Ruby developers built by a ruby developer. I’m trying to follow idioms, patterns and fluent type of doing APIs on Ruby.

This work isn’t related at all with StorageClient Sample shipped with Microsoft SDK and written in .NET, the whole API is based on my own understanding, experience and values of elegance and ruby development.

Full documentation for the gem is available at waz-storage.heroku.com

Scenario Idea

The gem was thought having heroku.com in mind as it is my main Ruby hosting vendor. The basic idea can be expressed with the following diagram:

As you can see there’s no need (and no personal desire) of hosting your application in Windows Azure or even writing it on .NET (or Windows needed at all).

How does this differ from waz-blobs and waz-queues?

Well, this is a sum up of the whole experience of writing those gems and getting them to work together to simplify end user experience. Although there’re some breaking changes, it’s pretty backward compatible with existing gems.

Open Standards

The waz-storage model is built upon the REST API published by Microsoft for Windows Azure Storage Services following the HTTP Standard. It’s basically a bunch of HTTP Requests with a special signature and some custom headers. The same API can be ported to any platform supporting a decent HTTP Stack.

So, what?

As in the latest posts, I won’t include too much detail here since it’s all explained at github.com/johnnyhalife/waz-storage and the whole API is detailed (and documented) at waz-storage.heroku.com.

The future is bright, I strongly appreciate all your feedback, your tweets and the time spent on (at least) reading the code. I really love this whole experience, but I won’t stop here, I’ll be relaxing from this SDK for a week or so (maybe less) until I come back for doing Tables support and fixing the issues that users may have encounter.

TODOs’

There should be some samples or at least one Reference Implementation of the API, probably I’ll be requesting help from other people in order to accomplish this, but I promise that some sample will be soon online so you can see the unveiled power of the Windows Azure Storage when running on Ruby.

Also there’s some support for Blocks that will be added, and some other sugar features for the application.

Time will tell, meanwhile you can comment, tweet or say whatever you want since I’ll be listening to your feedback.

Acknowledgment

This time I’ll make an special acknowledgment, besides my team, teammates, friends, and collegues at Southworks, Ezequiel Morito and Juampi Garcia. I want to thank and recognize that this work won’t be online today if I didn’t receive the advice from Martin Salias who encouraged me to go further on spiking and publishing my spikes.

Thanks Martin, I really appreciate your support and guidance.

Hey, after giving lots of thoughts of what am I going to do with the whole set of Ruby APIs for Windows Azure Storage, I figured out that might be better to have them merged, tested and refactored as a whole thing. I thought about lot’s of pro’s and con’s regarding whether to do this or not and I came out with a the following list:

Pro’s

• Blobs and Queues handle the same type of security, connection and headers. They share the core service called WAZ::Storage::SharedKeyCoreService
• While I was doing some work on Queues I found some bugs on connection management and URI management on the CoreService, so those changes weren’t propagated to Blobs (yet).
• 99% use case of the gem is that people handle a single WAZ Account for everything (Queues, Blobs and Tables) so have to connect, configure and consume them from different gems seems lot of overhead.
• Maintainability Index really drops on having them split.
• This is a single man’s work and even worse is doing over my spare time (so I cannot keep them both up-to-date always).
• Total gem size is 50KB compared to two gems of 30KB (so storage isn’t a problem here)

Con’s

• At some point seemed cool and very aligned with YAGNI that you need to install only what you need (queues or blobs)
• Shared behavior could be inherited from a third gem called ‘waz-storage-core’ but that will require more maintainability work.

Those are the things that came to my mind while working on this, so I decided to glue them together, refactor them and generate a consisten API for both services (and the tables soon enough).

Towards the SDK

Ruby devs aren’t usually required to have SDKs that’s more of the Windows World. SDK is the name I decided the whole thing will have: documentation, sample, gems and source code (Yes! I’ll do more docs). I’m looking for a consumable way for Ruby Devs to get involved on Windows Azure (since it’s worth doing it) but also there’re a lot of .NET Developers that may want to have a look at Ruby’s features and opportunities outside Windows World.

This is work in progress

Although I didn’t officially released anything related to this new SDK, I started my own work towards completing the API, so every piece of valuable feedback you might have will be really appreciated! I really enjoy while doing this and probably will do better as time goes by but as you can see on the pictures, I already started, merged and have a pretty decent TODO’s to apply to this thing and probably after that I’ll be releasing another version.

From the picture on the right you can infer, that I’ve already merged the test suite, and the TODO’s. The whole project is now a single thing. These are the things that I’m considering for next version

As you can see, there are things that still need to be figured out, but with time (no much) , they will emerge and I’ll be releasing as soon as I have them.

Help wanted!

As I mentioned now and on the previous posts, this is a one man’s work but you can change it =). I’m looking for Developers (juniors, amateurs, pro’s, whatever) to contribute on the project as they wish. If you want to test the code base, write a sample, write some docs, or whatever you’re invited! Drop me a message at @johnnyhalife and we should get going soon.

that’s it, stay tuned for more! (very soon)

thanks,
~johnny

Hey Pals! As you might recall two days ago I’ve published an API called “waz-blobs” that enabled Ruby Developers to get the best from Windows Azure storage straight from their pure ruby code. I told you I won’t stop there, so now I’m proud to announce that I’ve created waz-queues. As you can imagine and given the fact that I wasn’t creative at the time I named this things (If I had a dog I’d name it dog), this API exposes Windows Azure Queues to Ruby Developers (and IronRuby Devs too!).

Like the previous API, it’s 100% pure ruby, relays on RestClient written by Adam Wiggins and was written and tested on Mac and Ubuntu.

As on the previous post I wrote about the Blobs API,  this post isn’t about API documentation. It’s about all those things that forced me to write the API and some other random thoughts about where this whole thing is going.

Oh, you did it again!

Yes! I love it, interoperability is my passion, I love to write things to demonstrate that they can work with other very different things, and given the fact that Microsoft did a great job when they published the Windows Azure Storage Queue API on line, I decided I’ll spend some time building this. Queues are great, async programming is cool and REST is something that people do not believe until they see it (for those skeptical devs this is fully implemented over HTTP Stack on pure Ruby)

There’s nothing worse than waiting inline. Bummer!

Welcome to the twentieth one century, we live surrounded by the buzz and the current one is called “Cloud Computing”. Although I won’t discuss the topic here, it relays on a very simple principle “pay-to-play”. If you want to handle more requests just increase the number of workers and decrease the number of money you’ve got on your pockets.

Not all the operations on a System need to be always consistent, Amazon cart is an example of that. When you don’t want to pay more just get a queue, do some work async, and voila! your problems are solved.

Some hosting platforms like Heroku o Windows Azure have a limited number of requests that they can handle with a single machine/worker/dyno/you_name_it, when the backlog becomes to big they start to deflect request with HTTP 500 messages. If you relay on a queue you will also have a way to increase responsiveness on your app, sites like digg and twitter.com already do something similar.

Why do I need Azure Queues?

It’s an extremely powerful alternative to having a File System Queue or archaic Queue Systems like MSMQ, it’s on the cloud, it’s unlimited storage growth and can be used from either your current hosting or Windows Azure.

The whole Queue Service provided by Windows Azure it’s really simple but not least powerful. Possibilities are endless, and you can do whatever you want and create a multi-party system that has your web app hosted on Heroku written in pure ruby, a back end service running on Windows Azure written on C# and a python app hosted on Google Apps engine that uses Windows Azure Queues to communicate with each other. There’s no limit! (Even your mainframe can handle WAZ-Queues with some C HTTP Lib + playing nice with sockets).

Ok, I buy it, now show me your magic

The API is pretty simple, simpler than anything I’ve written before. Here’s a snippet that illustrates basic API interaction (or in other words, everything you need from a queue)

Getting started

sudo gem install waz-queues --source http://gemcutter.org

Reference Code

  require 'waz-queues'

        service = WAZ::Queues::Base.establish_connection!(:account_name => account_name,
                                                          :access_key => access_key)

        # excepts that the metadata for the queue changes this method behaves as PUT
        # remarks: it performs a validation whether metadata changed or not (HTTP 409 conflict)
        queue = WAZ::Queues::Queue.create('my-queue')

        10.times do |m|
          # enqueue a receives string. Message content can be anything up to 8KB
          # you can serialize and send anything that serializes to UTF-8 string (JSON, XML, etc)
          queue.enqueue!("message##{m}")
        end

        while(queue.size > 0) do
          # Since WAZ implements the peek lock pattern we are locking messages (not dequeuing)
          # it has two parameters how many messages and for how long they are locked
          messages = queue.lock(10)

          puts "dequeued message: #{messages.size}"

          # deletes the message from the queue so other clients do not pick it after
          # visibility time out expires
          messages.each {|m| m.destroy!}
        end

What’s next

While doing this I figured out that there’s a common behavior shared across blobs and queues, you can see it on the code that I started doing some merging while creating a ruby module called WAZ::Storage::SharedKeyCoreService that handles everything related to connections, requests, headers and signing messages. I’ll be merging the whole thing into ‘waz-storage’ gem that I’m planning to start over the weekend and create a small reference application to illustrate better what does this do? (if you wanna join ping me via twitter @johnnnyhalife)

That’s it, I won’t say I’ll do tables API too, I guess that after this week you can figure it out by yourselves (although it might take more time)

Special Thanks

I’d like to thank the following people that help me with something more important and painful than designing the API or writing the code that this time was an individual experiment. The following people are listening to me in every spare time I have at work, talking about why I design something like I did, how cool it is, or how do I enjoy this, which can be even hard than putting this code together!

These people are: Matias Woloski, Ezequiel Jadib, Federico Boerr, Ezequiel Morito, Sebastian Renzi, Martín Salias and Pablo Costantini.

That’s all folks. Thanks,
~johnny

Hey Folks, today I’m proud to announce my first release of the waz-blobs ruby gem, for interacting with Windows Azure Blob Storage from Ruby programming language.

Yes, it’s 100% organic Ruby code, there’s no strange Microsoft library that you need to consume and even better it was written and tested on Mac OS X and Ubuntu 8.10.  This post is about the motivation, and design process I’ve taken. Here you will find a minimal reference to the code, if you’re looking for the bits go straight to http://github.com/johnnyhalife/waz-blobs/ that includes the whole API documentation.

Motivation

Interoperability, fun and inspiration! Those three words are the base of this whole experience, from those OSS Fan thinking “why would I go for Windows Azure Storage? I’ve have my App written 100% ruby and runs on my custom built linux distro!”, here is the summary of the thinking I’ve been doing when I was trying to understand whether to write the API or not:

• It’s solid. Although it’s on CTP, it has been out for almost a year and during the different version it has demonstrated some basic fundamentals covered on Distributed Storage and the speed it’s really good.

• Ruby’s alternatives for serving static files isn’t great. I love Ruby, it has slowly became the language of my preference on the last year, but we have to admit that serving static files it’s better when outside our Ruby App server.

• It’s interoperable. As I started this post, the code available at http://github.com/johnnyhalife/waz-blobs/ is 100% organic ruby code, it’s based on the RestClient API written by Addam Wiggins from Heroku that can be found on http://github.com/adamwiggins/rest-client/. The code is based on the WAZ Rest API available at http://msdn.microsoft.com/en-us/library/dd135733.aspx

• It works great from Heroku. I spend my spare time hacking on pure Ruby and most of my project end up on Heroku (by far the best Ruby Hosting Platform on the Clouds). As you can see on Heroku Docs, their recommendation is to go for AWS-S3, and I don’t like when my choices are limited, I wrote the gem just to have an alternative for S3 when hosting on Heroku.

Getting started with the waz-blobs API

First of all, if you just want to use it just get it from the gems repository like the following:

gem install waz-blobs --source http://gemcutter.org

And here’s some basics

  require 'waz-blobs'

  # Save your configuration info
  WAZ::Blobs::Base.establish_connection!(:account_name => "name", :account_key => "key")

  # creates a container
  container = WAZ::Blobs::Container.create('my-container')

  # stores a blob with custom properties (metadata)
  blob = container.store('my-blob.txt',  'this is the content of my blob', 'plain/text')

  # return a specific blob from a container
  blob = container['my-blob.txt']

  # retrieves a blob value
  blob.value

The complete API reference and source code can be found at http://github.com/johnnyhalife/waz-blobs/, feel free to download, fork it, and submit patches.

Next Steps

During the next days, I’ll continue using this and probably refactoring or bug fixing. In the near future (don’t know how quick yet), I’ll get the WAZ Queues API too.

While I work on Queues, fork me on GitHub and consider yourself invited to submit as many patches as you want.

thanks and read you soon,
~johnny