Linq to Google

Updated my artile on CodeProject:

• Added support for parametized query and projection.
• Refactored source code and namespaces.
• Support for Google Groups query.

You can download the source code here.

Are the CEP guys looking at LINQ?

CEP (Complex Event Processing) has become a very hot topic in the capital market as Algorithmic Trading gains popularity. TIBCO and Apama are two of the biggest players in the CEP engine market. There are also a handful of other vendors, including Coral8, Gemfire, StreamBase, etc).

One problem in this domain is the standardization of front end language of choice: different products tend to choose different front end approaches, ranging from SQL-like syntax, OO class library to drag-n-drop visual user interface.

I wonder whether any of them are working towards using LINQ as its .NET front end.

Linq To Google Image 0.1

It seems getting to the bottom of LINQ isn't as difficult as I anticipated. With two days of experimenting, I have got a crude Linq To Google Image implementation, and have learnt quite a lot about Linq's inner working along the way.

I have put together a summary on CodeProject here. And you can download the source code (at your own risk) here.

I shall probably take one step back, get away from C# 3.0 for a while and take a quick look into WCF, as that's also of great interest to me and my knowledge is pretty much still stops two years short when it was still Indigo...

Google Linq

The ultimate way to understand LINQ, of course, is to write your own IQueryable magic. I have choosed Google Image as my first pet project for this: The reason for selecting Image over Web search is that Image results are more structured. For example, we can filter on size, type and color. Implementing all these would provide a more complete view of the inner working of LINQ/IQueryable.

Base on the principles of TDD, I shall first define the test cases for the project, for example:

var test = from img in

               MChen.Linq.Google.ImageSearch.Instance

           where img.Description.Contains("linq")

               && img.Size == ImageSize.Small

           select img;

I intend to define ImageSearch as a singleton because, well, there is only one Google… Obviously this can be further extended to Web, Group and Froogle in the future.

As the very first step, I have put togather a crude implementation of IQueryable that simply dumps the Expression tree in CreateQuery function. The output for the query expression above looks like:

Some references that are very useful along the way:

• Use Reflector to exam the implementation in DLINQ.
• Linq To Amazon by Fabrice Marguerie
• Linq To Flickr by Bashmohandes

More on DLINQ and Projection Operator

I posted my observation on DLINQ and Projection Operator on MSDN Forum and soon received a response from Keith J. Farmer at Microsoft.

The answer was somewhat expected: DLINQ intends to translate everything to SQL to avoid any type of client side query for performance reasons. I can understand this for where and orderby clause, but not for select and projection operator I was using.

On the other hand, Keith did recommend a practical solution: put the .Net method call in SQL/CLR so that it can be called as a user defined function in SQL. Then map the UDF in DLINQ so that it can be used from the query:

• First, create a SQL Server Project in Visual Studio and create the UDF. This is really nothing new and can be done in VS2005.
  
  

      [Microsoft.SqlServer.Server.SqlFunction]

      public static SqlString StringFormat(

          SqlString format,

          SqlString o1,

          SqlMoney o2)

      {

          return new SqlString(string.Format(

              format.Value, o1.Value, o2.Value)

          );

      }

• Deploy the UDF in SQL Server. This can be done inside VS2005 (Build -> Deploy [project name]) or manually (see this post for steps).
• You probably don’t want to manually write the mapping code for the UDF this time. So use SqlMetal, which is provided as part of .NET 3.5 Beta, to generate the database mapping file:
  
  

  sqlmetal /server:localhost /database:northwind

           /code:northwind.cs /functions
           /namespace:MyDLinqTest

• Now we have mapped the entire Northwind database with all functions, we can use its tables and UDFs in DLINQ query like this:
  

  Northwind db = new Northwind(ConnStr);

  var list = from p in db.Products

             where p.UnitPrice > 30

             select new {

                p.ProductID,

                Description 
                  = db.StringFormat("{0} - {1}",

                    p.ProductName, p.UnitPrice)

             };

DLINQ and Projection Operator

Seems I was impressed too early. Projection operation on DLINQ is giving me a bit of grief:

var list = from p in prods

       where p.UnitPrice > 30

       select new {

          p.ProductID,

          Description = string.Format("{0} - {1}",

              p.ProductName, p.UnitPrice)

      };

I got the following exception:

  

Yeah, I know String.Format can’t be translated to SQL, but I’m expecting it to be executed as .NET code. DLINQ is already using CodeDom to generate database access code anyway. Why can’t it generate the SQL that retrieves only directly mapped columns/fields, then produce the rest fields in the anonymous type, like this:

class _AnonymousType

{

    //map to database columns

    private int     _productID;

    private string  _productName;

    private decimal _unitPrice;

 

    //...

    //generate Description as

    public string Description {

        get {

            return string.Format("{0} - {1}",

              _productName,

              _unitPrice);

        }

    }

}

Sure, this would require the Projection Operator be part of the expression tree and the anonymous type being generated at runtime, rather than by the compiler, as it currently is. But limiting the projection in DLINQ to only these operations that can be translated in SQL is probably too much of a sacrifice… In the example above, we can change string.Format to:

     Description = p.ProductName + "-" + p.UnitPrice

 But what if the projection involves more complicated function calls?

Digging DLINQ

Just gave DLINQ a quick shot and it indeed is quite impressive. The way Table and Column attributes are used for database mapping is similar to that of XML Serialization:

    [Table(Name="Products")]

    public class Product

    {

        [Column]

        public int ProductID { get; set; }

 

        [Column]

        public string ProductName { get; set; }

 

        [Column]

        public decimal UnitPrice { get; set; }

    }

Now, you can use LINQ syntax to perform strong typed database query:

public static void DLinqQuery()

{

    string ConnStr =

        @"Server=localhost;Database=Northwind;";

 

    DataContext ctx = new DataContext(ConnStr);

    Table<Product> prods = ctx.GetTable<Product>();

 

    //this is COOL!

    var list = from p in prods

               where p.UnitPrice > 30

               select p;

 

    foreach (var p in list)

    {

        Console.WriteLine("{0,5}\t{1,-32}\t{2,-6}",

            p.ProductID, p.ProductName, p.UnitPrice);

    }

}

Pretty neat, agree? I only wish this were available eailier so that I didn’t have to deal with typed dataset. That was aweful… Some further point of interests:

• Is the underlying SQL generated by DLINQ effecient? (IQueryable and the expression tree do deserve their own blog entries).
• Mapping between foreign key relationship to object graphs.
• Stored Proc and Transaction support

Breaking Changes in Linq, Syntactic Sugar? Syntactic Heroin?

Breaking changes in Linq

Two days into the Linq jungle, I have noticed quite a few breaking changes in C# 3.0 Beta compare to the CTP release last year. The majority of sample code off the web can no longer compile under the new version. The poor documentation (well, to be fair, it's documentation for Beta...) made it a lot worse:

• System.Query namespace is now System.Linq.
• System.Expressions is now System.Linq.Expressions
• Extension methods ToQueryable and ToEnumerable are now AsQueryable and AsEnumerable
• Sequence class has been substituted with Enumerable, although this likely has no impact on code compilation.

What's "Syntactic Sugar" anyway?

All C# 3.0 new features are described as "Syntactic Sugar" by someone at some point. This leads me to think what exactly is Syntactic Sugar: Is OOP just a web of Syntactic Sugar(think Objective C, for example)? "Member Method" covers up the underlying function with an added this pointer, and the almighty "Polymorphism" is just a short hand for function pointer tables... So where do we end with this stripping process? "MOV EBP, ESP"? Oh, wait, that's just "Syntactic Sugar" for a bunch of 0s and 1s...

Syntactic Heroin

Some called operator overloading "Syntactic Heroin" because of the potential abusive use. I have the same concern for Extension Method: I see the legit use in certain situations, but it's so easy to abuse the concept by using it to quickly "hack up" a incorrectly designed class contract. It's not hard to see the code would become virtually unmaintainable when extension method is overused.

Ways to skin a LINQ query

Quite naturally, the very first step to understand LINQ is to go behind curtain and see how the magic select/where statement works. Although this has been explained many, many, many times, I still couldn't resist the temptation:

    public class Customer
    {
        public string Name { get; internal set; }
        public int Age { get; internal set; }
    }

    //...
            List<Customer> list = ...;
            var ret = from s in list
                      orderby s.Age descending
                      where s.Name == "Ming"
                      select s;

First, let's take out the syntax sugar out of the query. It's then equivalent to:

            Func<Customer, bool> p1
                = c => c.Name == "Ming";
            Func<Customer, int> p2
                = c => c.Age;
            return list.Where(p1).OrderByDescending(p2);

Still, this uses Lambda Expression. Taking that out, we have:

            var ret = list.Where<Customer>(
                new Func<Customer, bool>(
                    delegate(Customer c) {
                        return c.Name == "Ming";
                    }
            ));
            ret = ret.OrderByDescending<Customer, int>(
                new Func<Customer, int>(
                    delegate(Customer c) { return c.Age; }
            ));

And finally, without the Extension Method and var magic, the C# 2.0 way to write the query:

            IEnumerable<Customer> ret =
                System.Linq.Enumerable.Where<Customer>(
                    list,
                    new Func<Customer, bool>(
                        delegate(Customer c) {
                            return c.Name == "Ming";
                        }
            ));
            ret = System.Linq.Enumerable.
                OrderByDescending<Customer, int>(
                    ret,
                    new Func<Customer, int>(
                        delegate(Customer c) {
                            return c.Age;
                        }
            ));

It's now clear that LINQ is a compiler feature plus corresponding library support (System.Linq.Enumerable, for example). There really isn't anything new from CLR point of view.

The Orcas is here

• Visual Studio "Orcas" (Beta) is now available for download (need MSDN Subscription).
• Or you can download "Orcas" express editions here (Community Technology Preview).