Observing the Elements passed through an Expression of IQueryable<T>

[Marcel Müller]

I have a query provider and I want to track usage statistics of query
expressions.

Input is as usual some expression tree.

The IQueryable<T> in the innermost Where expression is then replaced by
some data source that contains a set of potentially matching objects.
But the number ob objects is still too large (only indices are applied
so far), so the filtering by the original Where expression is still
required.
Everything is working fine so far.

But now I want to track the selectivity of the .Where Expression
relative to the number of objects retrieved by the query provider. E.g.:
var objects = new [] { 1,2,3,4,5 };
objects.AsQueryable(); // fake
var result = objects.Where(obj => obj >= 3 && (obj & 1) = 0);
Let's say the query provider only understands the sub expression "obj >=
3" but not "(obj & 1) = 0". It will then retrieve the objects 3 to 5,
effectively executing
result = new [] { 3,4,5 }
.AsQueryable()
.Where(obj => obj >= 3 && (obj & 1) = 0); // OK obj >= 3 could be removed
(Of course, I needed to rewrite the expression tree to this one.)

So to track the counts I want to add visitor functions:
int count_before_where = 0;
int count_after_where = 0;
result = new [] { 3,4,5 }
.Select(obj =>
{ ++count_before_where;
return obj;
})
.AsQueryable()
.Where(obj => obj >= 3 && (obj & 1) = 0)
.Select(obj =>
{ ++count_after_where;
return obj;
});
The first one works fine.
Unfortunately the second one does not compile because of the Type
IQueryable<T> and I failed to create an equivalent expression tree
manually with Expression.Lambda and so on.

And to make things a bit more complicated I also need to track
Enumerator.Dispose at the second visitor, because that's the only point
where I know that count_before_where and count_after_where are complete.

Any ideas how to get that working?

I should note that executing immediately is not an option because not
all enumerators are read to the end and reading object could take long
(web services, several seconds per object). So I need deferred execution.


Marcel

 

[Arne Vajhøj]

I have a query provider and I want to track usage statistics of query
expressions.

Input is as usual some expression tree.

The IQueryable<T> in the innermost Where expression is then replaced by
some data source that contains a set of potentially matching objects.
But the number ob objects is still too large (only indices are applied
so far), so the filtering by the original Where expression is still
required.
Everything is working fine so far.

But now I want to track the selectivity of the .Where Expression
relative to the number of objects retrieved by the query provider. E.g.:
var objects = new [] { 1,2,3,4,5 };
objects.AsQueryable(); // fake
var result = objects.Where(obj => obj >= 3 && (obj & 1) = 0);
Let's say the query provider only understands the sub expression "obj >=
3" but not "(obj & 1) = 0". It will then retrieve the objects 3 to 5,
effectively executing
result = new [] { 3,4,5 }
.AsQueryable()
.Where(obj => obj >= 3 && (obj & 1) = 0); // OK obj >= 3 could be
removed
(Of course, I needed to rewrite the expression tree to this one.)

So to track the counts I want to add visitor functions:
int count_before_where = 0;
int count_after_where = 0;
result = new [] { 3,4,5 }
.Select(obj =>
{ ++count_before_where;
return obj;
})
.AsQueryable()
.Where(obj => obj >= 3 && (obj & 1) = 0)
.Select(obj =>
{ ++count_after_where;
return obj;
});
The first one works fine.
Unfortunately the second one does not compile because of the Type
IQueryable<T> and I failed to create an equivalent expression tree
manually with Expression.Lambda and so on.

And to make things a bit more complicated I also need to track
Enumerator.Dispose at the second visitor, because that's the only point
where I know that count_before_where and count_after_where are complete.

Any ideas how to get that working?

I must admit that I am confused, but that is most likely because
I know too little about LINQ and query providers.

Anyway according to my limited understanding then a queryable select
does not get executed as C# code, but get translated to something by the
query provider.

If it is a database provider it gets translated into the select
list.

What is a query provider like a database provider supposed to do
with a captured C# variable?

So to me it is not just a compiler/syntax problem but a logical
problem about what semantics should be.

I think you have to move the counting down into your provider code.

Arne

 

[Marcel Müller]

Am 29.11.2013 02:33, schrieb Arne Vajhøj:

So to track the counts I want to add visitor functions:
int count_before_where = 0;
int count_after_where = 0;
result = new [] { 3,4,5 }
.Select(obj =>
{ ++count_before_where;
return obj;
})
.AsQueryable()
.Where(obj => obj >= 3 && (obj & 1) = 0)
.Select(obj =>
{ ++count_after_where;
return obj;
});
The first one works fine.
Unfortunately the second one does not compile because of the Type
IQueryable<T> and I failed to create an equivalent expression tree
manually with Expression.Lambda and so on.

This one I got working now by constructing the expression tree manually:

E := element type

Expression query_expression = innermoste_where_expression;
Counter counter = new Counter();

var objparam = Expression.Parameter(typeof(E));
var callincout = Expression.Call(Expression.Constant(counter),
new Func<E,E>(counter.IncOut).Method,
objparam);
var inclambda = Expression.Lambda<Func<E,E>>(callincout, objparam);
var queryable_select =
new Func<IQueryable<E>,Expression<Func<E,E>>,IQueryable<E>>
(Queryable.Select).Method;
query_expression = Expression.Call(null, queryable_select,
query_expression, Expression.Quote(inclambda));

class Counter
{ int InCount = 0;
int OutCount = 0;
public E IncIn(E obj)
{ ++OutCount;
return obj;
}
public E IncOut(E obj)
{ ++OutCount;
return obj;
}
public void Finish()
{ Root.Statistics<E>.Instance.AddQuery(InCount, OutCount);
}
}


But here I got no solution so far.

I think you have to move the counting down into your provider code.

In fact the Code /is/ in the query providers Execute method. My provider
translates only parts of the query into another language. These part
returns an IEnumerable<E>. But there are further parts of the expression
tree that are simply passed to
Expression.Lambda(expression, null).Compile().DynamicInvoke()
i.e. the .NET compiler.

Maybe I need to compile the expression in two steps. First compile the
additional .Where conditions, apply them to the providers internal
result set. Then apply the counter, but as Compiled code rather than as
LINQ expression - still with deferred execution. And then replace the
entire .Where sub expression by the result and compile the remaining part.


Marcel

 

[Arne Vajhøj]

Am 29.11.2013 02:33, schrieb Arne Vajhøj:

So to track the counts I want to add visitor functions:
int count_before_where = 0;
int count_after_where = 0;
result = new [] { 3,4,5 }
.Select(obj =>
{ ++count_before_where;
return obj;
})
.AsQueryable()
.Where(obj => obj >= 3 && (obj & 1) = 0)
.Select(obj =>
{ ++count_after_where;
return obj;
});
The first one works fine.
Unfortunately the second one does not compile because of the Type
IQueryable<T> and I failed to create an equivalent expression tree
manually with Expression.Lambda and so on.

This one I got working now by constructing the expression tree manually:

E := element type

Expression query_expression = innermoste_where_expression;
Counter counter = new Counter();

var objparam = Expression.Parameter(typeof(E));
var callincout = Expression.Call(Expression.Constant(counter),
new Func<E,E>(counter.IncOut).Method,
objparam);
var inclambda = Expression.Lambda<Func<E,E>>(callincout, objparam);
var queryable_select =
new Func<IQueryable<E>,Expression<Func<E,E>>,IQueryable<E>>
(Queryable.Select).Method;
query_expression = Expression.Call(null, queryable_select,
query_expression, Expression.Quote(inclambda));

class Counter
{ int InCount = 0;
int OutCount = 0;
public E IncIn(E obj)
{ ++OutCount;
return obj;
}
public E IncOut(E obj)
{ ++OutCount;
return obj;
}
public void Finish()
{ Root.Statistics<E>.Instance.AddQuery(InCount, OutCount);
}
}


But here I got no solution so far.

I think you have to move the counting down into your provider code.

In fact the Code /is/ in the query providers Execute method. My provider
translates only parts of the query into another language. These part
returns an IEnumerable<E>. But there are further parts of the expression
tree that are simply passed to
Expression.Lambda(expression, null).Compile().DynamicInvoke()
i.e. the .NET compiler.

Maybe I need to compile the expression in two steps. First compile the
additional .Where conditions, apply them to the providers internal
result set. Then apply the counter, but as Compiled code rather than as
LINQ expression - still with deferred execution. And then replace the
entire .Where sub expression by the result and compile the remaining part.

Are you sure that LINQ is the right solution for your problem?

Arne

 

[Marcel Müller]

Am 29.11.2013 22:08, schrieb Arne Vajhøj:

Are you sure that LINQ is the right solution for your problem?

I don't know any alternatives.

However, the hack to compile the Expression tree in two steps seem to
work correctly. First compile the .Where expression to IEnumerable<E>,
then apply a visitor enumerator that keeps track of the traffic and, of
course, Dispose() and then compile the remaining parts of the expression
that are /not/ handled by the custom query provider with LINQ to Objects
(AsQueryable).


Marcel

 

[Arne Vajhoej]

Am 29.11.2013 22:08, schrieb Arne Vajhøj:

I don't know any alternatives.

So your problem could not be solved in C# before LINQ?



I would expect some more traditional code to be able to do it!

However, the hack to compile the Expression tree in two steps seem to
work correctly. First compile the .Where expression to IEnumerable<E>,
then apply a visitor enumerator that keeps track of the traffic and, of
course, Dispose() and then compile the remaining parts of the expression
that are /not/ handled by the custom query provider with LINQ to Objects
(AsQueryable).

Is that something that is easy to maintain?

Arne

 

[Marcel Müller]

Am 04.12.2013 03:23, schrieb Arne Vajhoej:

So your problem could not be solved in C# before LINQ?

Exactly. It simply makes absolutely no sense without LINQ, because it is
a LINQ query provider. ;-)

I would expect some more traditional code to be able to do it!

If you look at the entire application, of course, yes. But I will for
sure not rewrite hundrets of existing LINQ statements because of a small
change.

Is that something that is easy to maintain?

No more complicated than any other query provider too. Writing providers
is always an advanced task.


Marcel

 

[Arne Vajhoej]

Am 04.12.2013 03:23, schrieb Arne Vajhoej:

Exactly. It simply makes absolutely no sense without LINQ, because it is
a LINQ query provider. ;-)



If you look at the entire application, of course, yes. But I will for
sure not rewrite hundrets of existing LINQ statements because of a small
change.

Ah - so you already have the code using LINQ and you need to do some
provider magic to make it work.

I certainly see the point.

I would still be concerned about the long term support cost.

The first thing to do to get up of a hole is to stop digging it deeper.

No more complicated than any other query provider too. Writing providers
is always an advanced task.

Yes.

And that is a good argument to try to avoid writing one.

Arne