The book is still in its final phase, but several drops have been done to the community in order to get early feedback.

If you’re a enthusiastic software developer or an architect looking for the best practices when building a highly scalable data driven application I recommend you to take a look on this Data Access Guidance at CodePlex site.

Keep in touch for future releases!

The following concepts are first class citizens within the Fakes ecosystem:

• A stub replaces another class with a small substitute that implements the same interface. To use stubs, you have to design your application so that each component depends only on interfaces, and not on other components.

• A shim modifies the compiled code of your application at run time so that instead of making a specified method call, it runs the shim code that your test provides. Shims can be used to replace calls to assemblies that you cannot modify, such .NET assemblies.

Basically stubs are what you find in many other mocking frameworks for .NET, but shims provides extra value by allowing you to modify the compiled code (not targeting the classes you want to test, but the classes those ones rely on). You can read more about Fakes and usage here.

Here I’ll show you how you can mock base classes within the ASP.NET pipeline as HttpContextBase, HttpRequestBase and HttpResponseBase in order to set up everything MVC needs to resolve a route.

So you can add a private method to your fixture setting up these mocks like the following one:

private IDisposable MocksForMVCRouteTesting(string requestUrl, out HttpContextBase contextBase)
{
  var shimsContext = ShimsContext.Create();
  var requestBase = new System.Web.Fakes.StubHttpRequestBase
  {
    ApplicationPathGet = () => "/",
    AppRelativeCurrentExecutionFilePathGet = () => requestUrl
  };

  contextBase = new System.Web.Fakes.StubHttpContextBase
  {
    RequestGet = () => requestBase,
    ResponseGet = () => new System.Web.Fakes.StubHttpResponseBase()
  };

  return shimsContext;
}

Ok, now you got your mocks to test the routes, but how to use it? Easy, following the next pattern:

// Act
HttpContextBase context;
using (this.MocksForMVCRouteTesting("{mvc_route_to_text_here}", out context))
{
  var routeData = routes.GetRouteData(context);

  // Asserts here...
}

And what about a real like example?

  // Arrange
  var routes = new RouteCollection();
  RouteConfig.RegisterRoutes(routes);

  // Act
  HttpContextBase context;
  using (this.MocksForMVCRouteTesting("~/account/register/", out context))
  {
    var routeData = routes.GetRouteData(context);

    // Assert
    Assert.IsNotNull(routeData);
    Assert.AreEqual("Account", (string)routeData.Values["controller"], true);
    Assert.AreEqual("Register", (string)routeData.Values["action"], true);
    Assert.IsTrue(string.IsNullOrWhiteSpace(routeData.Values["id"].ToString()));
  }

Hope that you can find this useful .

This turns to be a Document Oriented store working as a key-value DB when interacting with its HTTP RESTful API. It stores JSON documents, and support sharding, replication, and many other things that we can find these days in most NoSql databases. You can go to the Couchbase home site and learn more about it.

As I said before, in order to try some of its features, I’ve created a console application (the client driver is very simple to add to your solution – just using NuGet).

First at all you need to get a configured client instance before interact with the DB. So let’s start by downloading the client NuGet package into your project, and add a new section within your .config file for Couchbase:

<configSections>
   <section name="couchbase" type="Couchbase.Configuration.CouchbaseClientSection, Couchbase"/>
</configSections>

<couchbase>
   <servers bucket="default" bucketPassword="">
      <add uri="http://127.0.0.1:8091/pools/default"/>
   </servers>
</couchbase>

Now you can get a client instance from your app just this easy:

var client = new CouchbaseClient();

Let’s perform some simple CRUD operations like adding and getting a document. Having the following Book class defined:

[Serializable]
public class Book
{
 public string Title { get; set; }
 public string[] Authors { get; set; }
 public decimal Price { get; set; }
 public bool Recommended { get; set; }
}

How do we create and add a new book to the store? Simple, just like the following:

var bookId = "f1637bb1-cd16-40bd-9ae7-e267d58ff62f";
client.StoreJson(StoreMode.Add, bookId, new Book
{
   Title = "For Whom the Bell Tolls",
   Authors = new string[] { "Ernest Hemingway" },
   Recommended = true,
   Price = 15.64M
});

In this example I have chosen the key to be a Guid, but you can use any string-set you want to define your own keys domain. I have also used a StoreJson() method that is not present at the CouchbaseClient class interface; well you can take advantage of it by using the Couchbase.Extensions namespace static classes containing some extension methods for this class.

And what’s the code to retrieve? Some kind of GetJson()? Exactly! Actually a GetJson<T>() extension method:

var book = client.GetJson<Book>(bookId);

Note: the CouchbaseClient class contains simple Store() and Get<T>() methods, so why I didn’t use them before? Well, if you use these methods the client will store the document instance as binary within a JSON document in the server. Having a JSON readable document stored in the server will be helpful to perform server-side operations over them (basically MapReduce operations).

But now, how you can retrieve or perform a custom operation, and get its results, over a set of documents? Well, as you may guessed, the answer here (as in many other NoSql databases) is to recall the MapReduce concept I’ve mentioned before. In Couchbase you have a concept that is very similar to relational DB’s Views to resolve this kind of things, and as happens in a RDBMS, it allows you to obtain a subset or full set of documents from the store (besides performing aggregation operations). Views in Couchbase iterate over a specific bucket, and they can be expensive to build if the dataset you selected is big, but once created they are automatically updated whenever data changes (having low performance impact). To store the views this DB has a design document concept (as we can find it in CouchDB – where you can store your views within a design document or run temporary views, resulting in performance degradation as temporary views execute when you sent them to the server – refer to CouchDB technical documentation for more info).

To create a new view you can go to your administrative console at http://localhost:8091/ and click in the Views tab to create a new design document and add some views to it, but I’ll show you how you can achieve this from your client app later.

Let’s first create a view that will be useful to search using book’s titles as I did on previous articles, a BookByTitle view/index. I’ve created a JavascriptFunctions class within my application to contain (as strings) the Javascript functions I need for the MapReduce operations, although you can store them whenever you want. Here it is, containing the mapping function for getting books by title:

public static class JavascriptFunctions
{
 public static string Dev_Books_ByTitle_Map
 {
   get
   {
      return @"function (doc, meta) {
         if (doc.title && doc.authors) {
            emit(doc.title, null);
         }
      }";
   }
 }
}

As you can see within the function I’m checking for documents that contain a title and authors, as the documents within the bucket can be of any kind and the map function will receive all the documents in the bucket (at least if they contain title and authors there is a big chance that they are books – you can split your documents within different buckets or choose to have a kind of ‘document type‘ identifier).

If you add this view to the server you can use it from the client to get a range of books by title for example, as shown below:

var booksByTitle = client.GetView<Book>("dev_books", "bytitle", true);
foreach (var book in booksByTitle.StartKey("S"))
{
   Console.WriteLine(string.Format("* Book retrieved: {0}", book));
}

Using the GetView<T>() method you can instruct the server that you want to get the results from the view (ie. the map or reduce functions results), but setting the last method parameter (shouldLookupDocById) to true you will get the documents either. And there I’m showing the books which titles start with ‘S’ until the end of the collection (remember that you have an index built below).

But, what if I want now to count books by authors as I did in previous articles? I’ll just need to add another view that maps the authors names with their book count from each document, and reduce these results grouping by the key (the author in these case).. as with other NoSql DBs I worked with, obtaining this result is a piece of cake. So let’s add two new ‘functions’ to my JavascriptFunctions class that will map and reduce authors with their books count:

public static class JavascriptFunctions
{
 public static string Dev_Books_ByTitle_Map
 {
   get
   {
      return @"function (doc, meta) {
         if (doc.title && doc.authors) {
            emit(doc.title, null);
         }
      }";
   }
 }

 public static string Dev_Books_AuthorWithBookCount_Map
 {
   get
   {
      return @"function (doc, meta) {
         if (doc.authors && doc.recommended) {
            for (var i in doc.authors) {
               emit(doc.authors[i], 1);
            }
         }
      }";
   }
 }

 public static string Dev_Books_AuthorWithBookCount_Reduce
 {
   get
   {
      return @"function (keys, values, rereduce) {
         if (!rereduce) {
            return sum(values);
         }
      }";
   }
 }
}

Now I have two new Javascript functions defined at this class: Dev_Books_AuthorWithBookCount (map & reduce versions). If you pay attention to the Map version, you can see that I’m only interested in books that are ‘recommended‘ (an again, to know that they are books I’m querying its authors property). From the client app you can get its results in the following way:

var countPerAuthor = client.GetView("dev_books", "authorwithbookcount").Group(true);
foreach (var row in countPerAuthor)
{
   Console.WriteLine("* Author {0} has {1} books", row.Info["key"], row.Info["value"]);
}

Please note that I’m using Group() method from the IView interface there to indicate the server that the reduce function should operate on each one of the keys. You can get more information here (and see there how to handle rereduce parameter – something that I’m not doing here since I don’t expect any).

Well… up to now the client code seems pretty simple in order to get view results, but how can we test it so far? Until now you could have been creating the respective views within the administration console (under the Views tab).

At first glance, it seems the C# SDK doesn’t have an API to manage views. Nevertheless, as the design document that contains views details is just another document within the DB, I’ll show you how you can create with a PUT request. You can achieve the ‘trick‘ easily with the following code that relies on the WebClient class from .NET framework to do the job, but first let’s add the design document’s body (JSON) to the JavascriptFunction class to have it at hand:

public static string Dev_Books_DesignDocument
{
   get
   {
      return @"{""views"":{
         ""bytitle"":{
            ""map"":""function (doc, meta) {\n if (doc.title && doc.authors) { \n   emit(doc.title, null); \n }\n}""
         },
         ""authorwithbookcount"":{
            ""map"":""function (doc, meta) {\n  if (doc.authors && doc.recommended) {\n    for (var i in doc.authors) {\n      emit(doc.authors[i], 1); \n    }\n  }\n}"",
            ""reduce"":""function (keys, values, rereduce) {\n  if (!rereduce) {\n    return sum(values); \n  }\n}""}}
         }";
   }
}

Note: This is a temporary solution to have the design document body, but I recommend you to create a C# class instance that can be mapped to that JSON body and transform it before make the request. That will be a better, and more elegant, way to do this.

Anyway, with the JSON body there we can make the PUT request now:

var url = "http://localhost:8091/couchBase/default/_design/dev_books";
var webRequest = new WebClient();
try
{
   webRequest.DownloadData(url);
}
catch (WebException ex)
{
   if (ex.Response is HttpWebResponse && ((HttpWebResponse)ex.Response).StatusCode.Equals(HttpStatusCode.NotFound))
   {
      var data = Encoding.ASCII.GetBytes(JavascriptFunctions.Dev_Books_DesignDocument);
      webRequest.Headers.Add("Content-Type", "application/json");
      webRequest.UploadData(url, "PUT", data);
   }
   else
      throw;
}

Where are making a PUT request to the default bucket with the _design/dev_books id. If you go to your Views tab within the administration console you should see the design document and its views. At the code I’m trying to get the design document and if doesn’t exist within the DB (404 – not found) I’m creating it.

You can find the sample code here. As usually, I recommend you go to the DB home site, download and try it yourself (you can do it from C#, using any other language found there or simply by using the REST interface the server has).

That’s all folks!!

The workarounds in which this article is based can be found online, so here I’m going to keep it simple and just write a list of steps (you can refer to these articles, found at the end of this post, if you need more information). So after install the package you downloaded, a CouchbaseServer service (Windows Service) instance should be running as shown below:

Execute the following steps in order to get a working instance (if it’s not working yet):

1. Stop the service
2. Edit the your Windows hosts file with Administrator rights
3. Add the following line: 127.0.0.1    couchbase-node1
4. Save and close the hosts file
5. Now relative to the root folder where you installed Couchbase edit the .\Server\bin\service_register.bat with Administrator rights
6. Update the line that is setting the NS_NAME var to be: set NS_NAME=ns_1@couchbase-node1
7. Update the line next to it to also set NS_NAME=ns_1@couchbase-node1
8. Save and close the file
9. From your Couchbase root folder delete the contents of .\Server\var\lib\couchbase\mnesia
10. Now run the .\Server\bin\service_unregister.bat as Administrator (from your root folder)
11. Next run the .\Server\bin\service_register.bat file you edited
12. At this point if you run the service and check the listening sockets (netstat -a) you should seen ports 8091 & 8092
13. To run the memcached healthy replace the libtcmalloc_minimal-4.dll (at .\Server\bin) with an previous version (this works for me)
14. And finally, start the service

After all these steps have been performed you should have access to your administrative console under IE (http://localhost:8091/) and see everything running OK as shown below:

Related articles:

IP Address related

Allocation library related (article1, article2)

Besides both databases have very similar features, RavenDB supports ACID transactions under the unit or work pattern. For a fully set of features that both databases support, and a comparison, I suggest you to go to their respective home sites and read more about them.

So, in order to try some of its features, I’ve created a console application using RavenDB (after installing the server, the client driver is very simple to add to your solution – just using NuGet).

First at all you need to get a document store instance and configure it before using sessions, after you downloaded the client NuGet package you can do it like this:

var documentStore = new DocumentStore { Url = "http://localhost:8080/" };
documentStore.DefaultDatabase = "mauro";
documentStore.Conventions.FindTypeTagName = (t) => t.Equals(typeof(City)) ? "Cities" : string.Format("{0}s", t.Name);
documentStore.Initialize();

Now you can perform for example a book adding operation. You will have to do it under a session (that will track all your changes on objects, as an ORM does, and commit or discard the whole thing):

using (var session = documentStore.OpenSession())
{
 session.Store(new Book
 {
   Title = "For Whom the Bell Tolls",
   Authors = new string[] { "Ernest Hemingway" },
   Price = 15.64M
 });
 session.SaveChanges();
}

Where your Book class is defined as:

public class Book
{
 public string Title { get; set; }
 public string[] Authors { get; set; }
 public decimal Price { get; set; }
}

How we can iterate over all our book collection? Well, as RavenDB is ‘Safe by Default‘ meaning, among other things, that you can NOT do a ‘SELECT * from Books‘ you will need to paginate through your collection or take another approach for custom queries and/or aggregation operations.

For this purpose is that RavenDB uses indexes for everything. And what is an index under this DB? Well, in the most simple concept an index here is a MapReduce operation. If you don’t have an existing, or static in RavenDB language, index for a specific query, the server will create a temporary one for you when needs to resolve a query (I recommend you to read more about at the home site documentation as indexes are the core of the inner working of queries).

So, how we can define a static index to resolve a specific query? This is pretty simple from the C# client and using LINQ. You can set for example this index to query all the books in the book collection (anyway remember always the ‘safe’ behavior):

using (var session = documentStore.OpenSession())
{
 if (session.Advanced.DocumentStore.DatabaseCommands.GetIndex("allBooks") == null)
 {
   session.Advanced.DocumentStore.DatabaseCommands.PutIndex("allBooks", new IndexDefinitionBuilder<Book>
   {
      Map = documents => documents.Select(entity => new { })
   });
 }
}

You can choose to create an IndexDefinition with your map/reduce function as strings within it, or use the IndexDefinitionBuilder helper (as here) and make things easier as you can use code-insight help to write the LINQ queries and you have all your domain objects at hand. As we are returning an empty object here for each one of the iterated books we are creating like a ‘mock index’ over the whole collection.

Tip: you can use this index to delete the book collection with the DeleteByIndex() command as it is shown next:

session.Advanced.DocumentStore.DatabaseCommands.DeleteByIndex("allBooks", new IndexQuery());

You can create any index you want to this way, just using the PutIndex() command as shown above and the IndexDefinitionBuilder class, but what is the best practice to create an index?

Well, another way to define and create a static index within the DB is to inherit AbstractIndexCreationTask<TDocument> class and set the index properties and behavior within the new class ctor. Let’s use this approach to create an index per book’s title:

public class BooksByTitleIndex : AbstractIndexCreationTask<Book>
{
 public BooksByTitleIndex()
 {
   this.Map = books => books.Select(book => new { Title = book.Title });

   //will analyze title to be available on search operations
   this.Indexes.Add(x => x.Title, FieldIndexing.Analyzed);
 }
}

Besides being the recommended approach to create static indexes, this one is also good for use them later on queries where you can explicitly tell the engine what index should use or leave it to be decided by the query resolver. In the following code you have two queries, the first one does not specify which index should use to resolve, and the second one does, anyway both queries in this case will use the same index: ‘BooksByTitleIndex‘.

Why? Because the resolver sees that your are using the Title property of your class and it has an index for it.

var book1 = session.Query<Book>()
      .Where(b => b.Title.Equals("Seven Databases in Seven Weeks"))
      .FirstOrDefault();

var book2 = session.Query<Book, BooksByTitleIndex>()
      .Where(b => b.Title.Equals("Seven Databases in Seven Weeks"))
      .FirstOrDefault();

But how do we create the index in the DB before executing the queries? Well, you can simple instantiate and call the Execute() method for your index, or use a static index creation helper class that takes all types that inherited from the AbstractIndexCreationTask class and creates the indexes for you if they don’t exist (it is recommended to do this when initializing):

// create from index classes
IndexCreation.CreateIndexes(typeof(Program).Assembly, documentStore);

Now let’s say that you want to sum the prices of all books in your collection. Well that’s something that you can very easily do with just a couple of lines in C# using LINQ and workaround the ‘safe’ behavior by retrieving pages of books (or change the ‘safe’ behavior settings - although this is NOT recommended!).

The problem with this? You are transferring all the data to the client, need to paginate over all the books in your collection and it’s bizarre. The cost will be prohibitive if this collection is huge and you will be going against the ‘safe’ behavior the DB has – that will be for sure a very bad design decision.

So let’s create a new index for this using the MapReduce features RavenDB has for us. Again, we are going to define a class for the new index to follow best practices and, in addition, simplify our code after:

public class SumBookPricesIndex: AbstractIndexCreationTask<Book, SumBookPricesIndex.ReduceResult>
{
 public class ReduceResult
 {
   public string SetOfBooks { get; set; }
   public decimal SumOfPrices { get; set; }
 }

 public SumBookPricesIndex()
 {
   this.Map = books => books.Select(book => new ReduceResult { SetOfBooks = "all", SumOfPrices = book.Price });

   this.Reduce = results => results
      .GroupBy(result => result.SetOfBooks)
      .Select(group => new ReduceResult { SetOfBooks = group.Key, SumOfPrices = group.Sum(groupCount => groupCount.SumOfPrices) });
 }
}

Look how a new ReduceResult class is defined within the new index class. This is to make things easier later also, and because as the MapReduce engine could call reduce with mapped or reduced lists you will need to have a logic that can handle both type of items (this is the simplest way). Refer to the documentation for more information.

How do you get the sum of  all the book prices using MapReduce then? It is very simple after you have the index shown above created, just like this:

var sumOfBooksPrices = session
   .Query<SumBookPricesIndex.ReduceResult, SumBookPricesIndex>()
   .FirstOrDefault();

Here we are also using the ReduceResult class definition to indicate the query the type of the returned objects.

What if for example we want to group the authors in our DB and count their books? The driver does not support the GroupBy() operation (at least using the Query() method exposed by the LINQ interface – but it is supported within the LuceneQuery() advanced method)… So? Just create another index!

public class GroupBookAuthorsIndex : AbstractIndexCreationTask<Book, GroupBookAuthorsIndex.ReduceResult>
{
 public class ReduceResult
 {
   public string Author { get; set; }
   public int NumberOfBooks { get; set; }
 }

 public GroupBookAuthorsIndex()
 {
   // be aware with linq queries [here SelectMany(authors,new) is not the same as SelectMany(authors).Select(new)]
   this.Map = books => books
      .SelectMany(book => book.Authors, (book, author) => new ReduceResult { Author = author, NumberOfBooks = 1 });

   this.Reduce = results => results
      .GroupBy(result => result.Author)
      .Select(group => new ReduceResult { Author = group.Key, NumberOfBooks = group.Sum(groupCount => groupCount.NumberOfBooks) });
 }
}

To finish this article I’ll show you a cool thing you can do by using RavenDB and taking advantage of the Lucene engine it relies on (in this particular case Lucene.NET): the “Did you mean?” feature (suggestions).

Let’s say that you want to look for a book title but remember just an approximate word or you did a typo when entering your search:

var query = session.Query<Book, BooksByTitleIndex>().Where(book => book.Title.Equals("sells"));
var searchedBook = query.FirstOrDefault();

And you have the following titles in your DB:

• For Whom the Bell Tolls
• Cat’s Cradle
• Slaughterhouse-Five
• Seven Databases in Seven Weeks

What will be the result (searchedBook variable)? Well, as there is no match for ‘sells‘ the result will be null.

At this point you can show suggestions to the user (using several comparison algorithms, the user’s input and your index’s tokens [BooksByTitleIndex]) calling the Suggest() method from the LinqExtensions class:

if (searchedBook == null)
{
   Console.WriteLine("Did you mean:");
   foreach (var suggestion in query.Suggest(new SuggestionQuery { Accuracy = 0.4f, MaxSuggestions = 5 }).Suggestions)
   {
      Console.WriteLine("\t. {0}?", suggestion);
   }
}

Being the output something like this:

Did you mean:

• tools?
• bell?
• seven?

You can find the sample code here. Anyway, this is just the top of the iceberg, there is a lot of features (document relationships, polymorphic indexes, transformation of results, lazy operations, and many many more not just related to querying) RavenDB offers not used within this article. I recommend you to download the DB and give it a try.

]]> 0 http://blogs.southworks.net/mkrikorian/2013/01/29/using-mongodb-from-c/ http://blogs.southworks.net/mkrikorian/2013/01/29/using-mongodb-from-c/#comments Tue, 29 Jan 2013 21:44:11 +0000 mkrikorian http://blogs.southworks.net/mkrikorian/?p=311 read more]]> Today I’ve been playing a little with MongoDB, and after enjoyed triggering a few commands from Mongo’s console, I decided to download, build and try the C# driver. Build it and use it is pretty straightforward, and in a couple of minutes you can be playing with your local database from a C# console application.

So I created a simple console application, that inserts and gets a couple of books from a DB collection in minutes with just very few simple steps. First at all you need to get a client instance and a server reference from C#, and using the driver you can do it like this:

// connect to localhost and get server
var server = new MongoClient().GetServer();
var db = server.GetDatabase("mauro");

// get book collection and clean it
var books = db.GetCollection("books");

From now on you can start performing operations in your books collection, for example adding a new book:

books.Insert(
 new BsonDocument
 {
   { "Authors", new BsonArray { "Ernest Hemingway" } },
   { "Title", "For Whom the Bell Tolls" },
   { "Price", 31.53 }
 });

To learn about the BsonDocument, BsonArray and other types related to this DB, browse its information here.

If you are wondering why the keys for the JSON object are capitalized, you will find the answer in just a few  seconds, when a C# Book class will be introduced and used with the driver. This is an easy way to work with your domain objects as you would do if you were using an ORM. In this case I decided to follow C# convention when naming class properties (except for the _id used in Mongo) as you can see below:

public class Book
{
 public BsonObjectId _id { get; set; }
 public string Title { get; set; }
 public string[] Authors { get; set; }
 // decimal is not well-converted by the driver
 public double Price { get; set; }
}

Now that we have a Book class defined we can use it within the driver calls and work with this typed domain object. If you use the db.GetCollection<TDocumentType>() getter when instancing a collection of entities you will tell the driver that you want to wrap the BSON objects (managed under the hood) as TDocumentType objects:

var typedBooks = db.GetCollection<Book>("books");

Now you can make iterate all the elements in this collection; each element returned as a Book instance:

Console.WriteLine("\nRetrieving all books (typed collection)");
foreach (var book in typedBooks.FindAll())
{
   Console.WriteLine(string.Format("Book retrieved: {0}", book));
}

And, of course, you can also add a new Book instance to the collection in a more easy way that having to deal with ‘JSON’ in C#:

typedBooks.Insert(new Book
 {
   Title = "Seven Databases in Seven Weeks",
   Authors = new string[] { "Eric Redmond", "Jim R. Wilson" },
   Price = 35.67
 })

At this point all these things are pretty much what any DB can offer (in addition to the ORM features we have from the driver), but what about using a little of MapReduce?

Can I use it from C#? Yes!

Let’s say that you want to sum the prices of all books in your collection. Well that’s something that you can very easily do with just a single line in C# using LINQ:

var allPrices = typedBooks.FindAll().Sum(b => b.Price);
Console.WriteLine(string.Format("Sum of all prices [linq]: ${0}", allPrices));

The problem with this? You are transferring all the data to the client, all the books in your collection. Think what will be the cost if this collection is huge! In addition, this is a simple process that just sum values, but what if the logic would be more complex and require more processing power to obtain the result in a reasonable time?

Fortunately, you can take advantage of the MapReduce engine MongoDB provides, besides being this the correct way to perform this kind of operations (over all your dataset). So let’s try MapReduce from C#.

First at all, I created a static class to put there my Javascript functions (as MongoDB talks Javascript):

public static class JavascriptFunctions
{
 public static string MapBuildAllPricesKey
 {
   get
   {
      return "function(obj) { return 'all'; }";
   }
 }

public static string MapAllPrices
 {
   get
   {
      return @"function() { emit(buildAllPricesKey(this), this.Price); }";
   }
 }

public static string ReduceAllPrices
 {
   get
   {
      return @"function(key, values) {
         var allPrices = 0;
         for (var i=0; i<values.length; i++) {
            allPrices += values[i];
         }
         return { key: key, books: values.length, total: allPrices };
      }";
   }
 }
}

How can I use this functions now?

Well, the MongoCollection provides a MapReduce() method that you can use to instruct the server what are the map and reduce operations you want to perform over a collection.

In addition to just try MapReduce, I’ve also decided to take advantage of the possibiliy to store a function in the server (kind of stored procedure in RDBMS) and, as you can see, I’m planning to use it within the MapAllPrices() js function. Here’s the code to store a function in the server that can be called from other functions you upload:

db.GetCollection("system.js").Save(
 new BsonDocument
 {
   { "_id", "buildAllPricesKey" },
   { "value", new BsonJavaScript(JavascriptFunctions.MapBuildAllPricesKey) }
 });

But how do we finally sum all prices using MapReduce? Easy, just like this:

var mr = books.MapReduce(JavascriptFunctions.MapAllPrices, JavascriptFunctions.ReduceAllPrices);
Console.WriteLine(string.Format("Sum of all prices [map/reduce]: ${0}", mr.GetResults().First()["value"]["total"]));

The mr variable contains the results of your MapReduce call, you can iterate its elements using LINQ (if the result values are many), here we are just taking the First() as we have only one.

You can find the complete sample here.

I enjoy watching how complex things can be resolved with just a simple idea, or a couple of them, and this is the case in the DB world where the integration of all the things is more than the sum of individual parts. So, let’s start with the things that gladly caught my attention:

One of the things I liked most and never gone into detail before, more than just heard its name and vaguely known what it could be used for, are the Bloom filters. I gladly understood how some of the databases analyzed during the book take advantage of them in order to perform fast lookups when need to get a specific value. It is a good structure to manage a not accurate index using significantly less space than an accurate index would use and, with a certain probability and error rate, can tell you if the thing you are looking for is maybe there, or answer that is not there (being 100% sure – i.e. it guarantees no false negatives). One interesting thing about its theory is that you can calculate these probabilities according to how many bits you are using to track things, how many hash functions are used to evaluate each piece of data and how big the set you are indexing is getting (or will be). Using this and having an approximate idea of how your data will grow you can know the number of bits you need to maintain a filter with an error rate below a specific bound.

A good practical example is how this is used within HBase. As this database stores columns in chunks that are split among several servers according to some random sharding schema (decided by the servers), you can know what is the chunk you need to go looking for something but you don’t know it that something could be there or not (here a Bloom filter can avoid you to search at all if it answers that what you are looking for is not there). You can find an interactive example and related stuff here.

Another interesting concept used in many NoSql databases to resolve query and aggregation demands, among others, is the usage of the MapReduce algorithm. Since the idea is very simple to understand, it is a powerful concept and unleashes you to perform whatever logic you want to execute over your distributed and probably huge sets of data in several ways. Its foundations is the divide and conquer principle, where a task (or big problem) is split and treated as subtask (or small problems) returning partial results, and at the end a final step is performed to merge them all (this approach sometimes helps to simplify computational complexity and demanded resources, where both can be analyzed theoretically). Besides, being a very old concept in computer science, the MapReduce new boom is related to how Google (who patented it) uses it to resolve its huge query demands. In this case this is just a particular scenario that happens on nodes within a cluster so it introduces parallelism as well. Each of the participants containing part of the data you need to process, and waiting for the function (or logic) each one must execute (over its owned set) in order to return its results to the caller. An overall view consists of: a first step (map) where you possibly evaluate and convert set of elements to another domain, and a merge step (reduce) that operates with the list of results from different nodes to generate the final result.

A practical example of this can be found on Riak DB where map/reduce function is sent to the cluster and distributed over the participant nodes in the “Riak Ring”. The map function is ran in each of the nodes in the cluster closer to where the data is (without the need to transfer ‘raw’ data through the network). So, besides avoiding network latency issues (by moving big sets of data), you don’t also need high centralized processing power as this architecture evaluates (in parallel) each subset in each individual node. Finally the reduce function is executed after all the partial results from nodes has been collected. You can find more details about how this works here.

A (very) interesting theorem, from theoretical computer science and related to distributed computing, is the CAP theorem. This basically asserts that if you have a distributed system, you can have simultaneously two of the following properties: Consistency, Availability or Partition Tolerance, but not the three of them. Taking into consideration the environment and the many factors that act against a distributed system is not possible to fulfill these three features without falling into the fallacies of distributed computing.

Beyond what the theorem says, it’s more interesting to see how some NoSql databases, that are also distributed, deal with it and allow ‘playing’ with this assertion. If we get Riak into scene again, we will find that this DB has an interesting way of parameterize its inner workings while reading/writing data within the cluster. Riak’s ring architecture distributes, and replicates, data within the nodes in its cluster. This clearly sets the DB as Partition Tolerance but other features are still available for you to play with: Consistency & Availability. Nevertheless, in the case of Riak you will never achieve full consistency if you use it in a real distributed environment (Riak has something that it calls ‘Eventual Consistency’).

This database allows you to freely select the values for: the total of nodes (N) that will participate in replicating data, how many nodes should be written (W) before a successful write is returned, and how many nodes should be read (R) to get the latest value for a key. With these parameters you can for example achieve high availability by setting a high value for N, nevertheless, if what you want to achieve is consistency you can try setting W=N or R=N (leaving the other value R or W to 1), where the first set of settings is called consistency by writes and the last one consistency by reads. You can think that having a high N and one of two choices you can achieve both consistency and availability, but this is not the case as having a high number of replica nodes leads to the following scenario: if you choose to configure consistency by writes, the number of endpoints where an update can hit is bigger leading to inconsistent versions of data for the same key (that the user must resolve); in addition to degrading the performance when writing to multiple nodes. If you try to set consistency by reads you will finally find a similar scenario, showing that you can’t achieve both. I encourage you to read more of how Riak allows you to play with it and its internals.

This is how Riak stands in front of the CAP theorem, and it’s nice to see how tweaking some parameters and playing a little with the cluster configuration, you can set Riak as {c}AP having sometimes (or eventually) consistency. Other databases achieve consistency or availability exclusively, but not both.

There are also another interesting structures used in the web distributed DB world that you can find, and if you like algorithms, data structures and architectures like me, I recommend you to dive into them: inverted indexes, vector clocks, write-ahead logging, distributed file systems, DB sharding, and many more.

Thanks for reading, happy diving!!

Below is an abstraction of the datamodel I had that time (two tables related using an alternative key in the Master):

Besides the primary key of each one of the tables (composite for Master, and not defined for Detail), you can see the relation among both tables by the AlternativeKeyQ field (by taking advantage of this unique field you don’t have to propagate the composite PK from Master to the Detail). Also the Detail table has a foreign key to another entity called ForeignKeyN in this case.

So, how do you setup the NHibernate mappings to make it work?

First at all you’ll need to define your domain object for Detail. Of course, you want to have a Master and a ForeignKeyN instances to allow easy navigability from Detail to both entities. So here is your base class defined in C#:

public class Detail
{
public virtual Master Master { get; set; }
public virtual ForeignKeyN ForeignKeyN { get; set; }
}

Now you have two navigation, and entity, properties in your Detail domain object. You need to define your mapping:

<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2">
<class name="MyNamespace.Detail, MyAssembly" table="Detail" lazy="true">
<many-to-one name="Master" column="AlternativeKeyQ" property-ref="AlternativeKeyQ" class="MyNamespace.Master, MyAssembly" />
<many-to-one name="ForeignKeyN" column="ForeignKeyN" class="MyNamespace.ForeignKeyN, MyAssembly" />
</class>
</hibernate-mapping>

You can see how the relation to Master is defined there, using the ‘property-ref’ attribute we let NHibernate know that we are using that field on the Master table to establish the relation. But if you try to parse this NHibernate mapping within a session you’ll receive an error saying that the entity doesn’t contain an identifier (basically).

So we’re going to fix this by adding an identifier to the entity, as we need to add this field to both (class and mapping) we are going to define it in a way that only NHibernate makes use of it:

public class Detail
{
public virtual Master Master { get; set; }
public virtual ForeignKeyN ForeignKeyN { get; set; }
protected virtual int Id
{
get { return this.Master.AlternativeKey; }
set { // do nothing.. }
}
}

Now we added a protected field to the Detail class that we are going to use as main identifier within the NHibernate mapping (but don’t expose it in the object’s interface – i.e. it won’t be visible to users).

<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2">
<class name="MyNamespace.Detail, MyAssembly" table="Detail" lazy="true">
<id name="Id" column="AlternativeKeyQ" />
<many-to-one name="Master" column="AlternativeKeyQ" property-ref="AlternativeKeyQ" class="MyNamespace.Master, MyAssembly" insert="false" update="false" />
<many-to-one name="ForeignKeyN" column="ForeignKeyN" class="MyNamespace.ForeignKeyN, MyAssembly" />
</class>
</hibernate-mapping>

As you can see in the mapping, we are using the Id property of the class as the identifier, and mapping it to the AlternativeKeyQ table field. We also disabled insertion or edition (insert=”false” – update=”false”) via the many-to-one Master property as otherwise it will clash with the Id update when NHibernate tries to update the entity.

In the Master mapping to add a bag with the Detail entities you have to use again the ‘property-ref’ attribute to let NHibernate know that you are linking these entities by AlternativeKeyQ field in the Master (if NHibernate requires you to override Equals() and GetHashCode() within your Detail domain object, use your new Id property to compare and return hash values).

The problem to solve is how to allow the user have its custom UDFs within entities we’re storing at Azure Table Storage as the client API works with models inherited from TableServiceEntity (types that are well known at design time). Our core logic will not be aware of UDFs created by the customer, and in consequence, will not know their models. This is where reflection comes in!

So, let’s suppose the customer defines in ‘CustomerModels.dll’ assembly its own model as the following:

public class MyCustomEntityFields : TableServiceEntity
{
    public int MyCustomId { get; set; }
    public string MyCustomText { get; set; }
    public override string ToString()
    {
        return string.Format("CustomId: {0} - CustomText: {1}", this.MyCustomId, this.MyCustomText);
    }
}

Then you could associate by convention or configuration this model to your main entity, as a customer’s extension of main fields. Things are simple when you add a new entity as you will associate the PartitionKey and RowKey to main entity’s keys and you will fill other properties by reflection (for example asking for them in the UI) and the Add() method supports any object. So you can have an adding code like this:

var customType = customerAssembly.GetType("CustomerNamespace.MyCustomEntityFields");
var udfInstance = Activator.CreateInstance(customType) as TableServiceEntity;

// fill instance custom properties (interacting via UI?) { ... }

udfInstance.PartitionKey = entityPartitionKey;
udfInstance.RowKey = entityRowkey;

// save instance using the TableServiceContext.Add() method

Let’s suppose that after the user filled the instance properties in a ‘new entity’ page (you can map its properties to a dictionary via reflection, gather their values and set them back), and you do udfInstance.ToString() you get:

But what when you need to retrieve these fields to show their data in the UI for edition or details? Well, as the TableServiceContext.CreateQuery() has a generic <T> parameter you don’t know (as the class/type is defined by the customer), is at this point when things get complicated and need to be solved using reflection. A practical approach to handle this is to create a ‘QueryWrapper’ class that you will instantiate via reflection and use it to retrieve this unknown instance data. You can create a simple query wrapper like the one shown below that is going to help you to retrieve this unknown objects:

public class UDFQueryWrapper where T : TableServiceEntity
{
    private readonly CloudStorageAccount account;
    private readonly string tableName;

    public UDFQueryWrapper(CloudStorageAccount account, string tableName)
    {
        this.account = account;
        this.tableName = tableName;
     }

    public IQueryable Query
    {
        get
        {
            var context = new TableServiceContext(this.account.TableEndpoint.ToString(), this.account.Credentials);
            return context.CreateQuery<T>(this.tableName).AsTableServiceQuery();
        }
     }
}

And you can create a static class with no generic parameter to make things even easier:

public static class UDFQueryWrapper
{
    public static IQueryable<TableServiceEntity> BuildQuery(CloudStorageAccount account, string tableName, Type udfType)
    {
        var queryWrapper = typeof(UDFQueryWrapper<>).MakeGenericType(new Type[] { udfType });
        var queryWrapperInstance = Activator.CreateInstance(queryWrapper, account, TableName);
        return (queryWrapper.GetProperty("Query").GetValue(queryWrapperInstance, null) as IQueryable<TableServiceEntity>);
    }
}

Then, when you need to get additional data for an entity you can take advantage of the UDFQueryWrapper.BuildQuery() method and you will use the PartitionKey and RowKey of the main entity to retrieve the correct instance data:

var udfInstance = UDFQueryWrapper.BuildQuery(account, TableName, customType)
    .Where(e => e.PartitionKey.Equals(entityPartitionKey) && e.RowKey.Equals(entityRowKey))
    .FirstOrDefault();

You can get the udfInstance data via reflection and if you call the ToString() overridden method you will get:

Just what you stored, demonstrating that the instance’s type and data binding were resolved correctly during retrieval .

In this post I’ll show you with just a few steps how you can use it to secure a resource and shared only with a couple of partners. First at all you need to get the X509 certificates for the actors that are going to be involved. You should have both: the .CER format having only the public part of the RSA key and the .PFX format having the complete RSA key. To be used with the Trust Services SDK these certificates should have a key length of 1024 bits and should be able to encrypt and sign data.

You can use the MakeCert tool for create the certificates you’re going to need, for example:

By executing these commands you will create three new self-signed certificates which are valid to be used with the Trust Services SDK. You can then export the .PFX files needed from the certificate store (LocalMachine/My).

Next step is to provision a Trust Server. You can do this by going to the Trust Services site and click on the ‘Use the Service’ link, sign in with your Live ID and create a new Trust Server establishing as TSPA’s certificate the Admin.cer created before.

The main scenario for using this service is when you upload sensitive data to the Cloud and want to protect it. Nevertheless, lets going to suppose that you have a shared repository in the file system where you store sensitive data and want only to be seen by authorized partners (but letting access to the resource to everyone). So first at all you have to define and ID for the resource or set of resources you want to protect. This is a simple task as you only need to define a resource Uri like the following one:

The next step is to use the management API via the SDK to authorize Partner1 and Partner2 to access the resources:

By doing this you are establishing an encryption policy for the Resource Uri and allow accessing it to both partners only. Now let’s suppose that Partner1 wants to update data and store it (protected) in the shared repository, it can easily do that by using the SDK as shown below:

By executing the code shown above Partner1 is storing the PlainData.txt file at a shared repository but encrypted. Performing this operation through the Trust Server ensures that: the data will no be accessed by unauthorized parties and no one has the key to decrypt it. So, if the Administrator decides to exclude a partner from the list of authorized people at the Trust Server then that partner will not be able to read the file content anymore.

When the authorized Partner2 wants to read the data stored in that shared resource he can do that by performing the following operations:

Partner2 is able to read the original data (decrypted) while he belongs to the authorized partners list at the Trust Server, if the Administrator revokes its priviledges then he will not be able to decrypt CipherData.enc anymore as it does not know the encryption key.

To understand which is the mechanism that secures the data and how the keys are managed in a way that no one knows them I recommend you to see the videos at the web site mentioned before.