re-mix (formerly known as re-motion mixins) has been published to CodePlex. The link is http://remix.codeplex.com
Similar to re-linq, the source code repository will still be part of the re-motion project. The source code and binaries of a “mixin release” will be published to CodePlex.
The re-mix page on CodePlex will also contain additional documentation to Mixins.
Today’s blog-post is about the recent namespace change in re-linq.
About a month ago, we restructured re-motion’s folders and namespaces to make re-linq a top-level project within re-motion. This move had been a long time coming but the time was never right until now. It is also the first step towards pushing the re-linq source code into the CodePlex repository entirely. This second move will happen some time in the future.
What does this mean for re-linq’s…
New namespaces and assemblies (Remotion.Linq instead of Remotion.Data.Linq). This is going to be the last foreseeable change in the namespace and assembly structure that will affect re-linq’s frontend (Remotion.Linq.dll).
The source code is now better isolated from the rest of re-motion, making it easier to work with.
As of this week, the re-motion project has a publicly available issue tracker, courtesy of Atlassian’s JIRA and Patrick Größ, who took it upon himself to tame the beast. Aside from setting it up and integrating it with our DMZ, this also meant integrating it with our internal processes here at rubicon. I’ll get to this at the end of the post. But first, I’d like to spend a couple of lines on our versioning policy and outlining our development process.
re-motion’s version numbers are designed to follow the classic trunk/stable version scheme used by various open source projects, not the least, the Linux kernel. This means we use the minor number to indicate the level of support a version receives.
- Uneven minor version numbers (1.11, 1.13, 1.15, etc) indicate that the version was built from the trunk. This is the main line of development in re-motion, and we don’t make any guarantees beyond our test-coverage for those versions. This means the APIs are more likely to break repeatedly on the trunk than they are once an API is considered stable. Also, we do not offer hotfixes for the trunk, no matter what. If there is a problem, it will get fixed but you do have to upgrade to the next version that contains the fix.
- Even minor version numbers (1.12, 1.14, 2.0) indicate that the version has been declared stable and resides in a distinct branch. Here, it is possible to receive a hotfix if there is a serious bug in the code. On the downside, you won’t get new features until we release another stable version and you upgrade again.
So, now that that’s out of the way, let’s talk about the third part of the version number, often referred to as the build number.
For the trunk, this number represents the individual XP iterations of re-motion’s trunk and gets incremented every week. You can also check out what’s planned for the week via the “re-motion current iteration” filter result displayed on the JIRA’s dashboard. Right now, we’re working on version 1.13.70, the ‘70’ indicating this is the 70th iteration on the 1.13 trunk.
For stable versions, the build number indicates a hotfix release. So 1.12.3 is the third hotfix for the 1.12 branch.
The following issue types are available to all users:
- New Feature: Represents a feature request by a user. Basically, any sort of feedback that represents the wish for added, improved, or changed functionality of re-motion. They’re also the highlight of our release notes. After all, who wants to read about tons of bug-fixes in a release? I prefer my features to be bug-free the first time around.
- Sub-Feature: In case you envision a really big feature, you can break it down into sub-features. Typically, though, that’s only done by re-motion’s developers after spending a lot of time in front of a whiteboard.
- Performance: In case you find an issue with re-motion that hampers your application’s performance, you can explicitly declare it as such.
- Bug: Represents feedback on a piece of re-motion that does not work as documented (or obviously intended). Please keep in mind that we reserve the right to change a bug report to a feature request if it’s not a bug but a feature 😉
The other issue types are only intended to be used by re-motion’s developers as they represent artifacts of the previous issue types.
- Task and Sub-Task: Those are to-dos for re-motion’s developers, directly derived from a reported feature request or sometimes a bug report. They’re the only issue types excluded from our release notes and contain the technical information only relevant to the poor soul having to implement it. They also show up in the Subversion log messages and link a change-set to a task and subsequently to a feature.
- Breaking Change: This is the big one. The one nobody likes. But since we don’t guarantee everlasting APIs like Microsoft does, introducing the odd breaking change is acceptable if it keeps the design clean, makes the API better, or is plain and simple necessary to implement a feature request.
We’ve looked at what other projects do. We’ve looked at the defaults provided by JIRA. And after much debate, we decided to offer only two priority levels when dealing with issues:
Normal, I guess, should be pretty much self-explanatory. So, what about critical? Well, that’s easy. In case you stumble across a bug that stops your project cold and for which there aren’t any (sensible) workarounds, please assign it a critical level of priority. We do strive not to have any of those, at least in the unresolved category.
The resolution type is the distilled reason why an issue has been closed.
- Fixed: This one’s easy. The task was fully implemented.
- Won’t Fix: Sometimes, an issue reported cannot be resolved by us because it would break some other invariant of re-motion. In such a case we like to make our point clear and close the issue as won’t fix. Usually, though, we try to explain our reasoning first and only close it as ‘Won’t Fix’ after a consensus has been reached with the reporter about why we won’t fix the issue.
- Duplicate: There’s already an issue in the system referring to the same bug or missing feature. In this case, the two issues get linked and one of the issues gets closed as being the duplicate. Which issue this is depends on lots of things, such as the amount of detail in the respective issue, whether the other issue is already scheduled, and, maybe, on the tea leafs left on the bottom of the cup on the developer’s desk.
- Obsolete: In case fixing one issue invalidates another issue completely, this issue will get closed as obsolete.
- Cannot Reproduce: Last but not least, in case we can’t make heads or tails of a bug-report and the reporter hasn’t updated the issue with more information, we will close it as ‘Cannot reproduce’. Another reason would be if the reported behavior is the result of an application error instead of a bug in re-motion.
The issue status describes the present state of affairs for any open issue.
- Open: Nobody is working in this issue right now.
- More information needed: We need more information about the task, usually from the reporter. This request always includes a comment and the task being assigned back to the reporter. Please make sure to check on your issues either via e-mail notification or by visiting our JIRA regularly because if we don’t have the information needed to reproduce or solve a problem and the reporter doesn’t seem to care, the issue will get closed as ‘Cannot Reproduce’.
- Closed: The issue is done and the resolution type is set to anything but ‘Unresolved’.
- In Progress: A status only relevant to developers. It indicates that the assignee is currently doing work on this issue.
- Review needed: Another one of those internal statuses. The assignee is done with the task and would like the reporter to check the work.
- MarkAsDeleted: This one’s needed in case an issue has to be deleted from the system. While we strive not to do that with user-reported issues, we don’t have the same scruples when it comes to obsolete developer tasks.
- MarkAsClosed: This one’s basically the same as the ‘Closed’ status and it only exists because our infrastructure behind the scenes needs it. Normally, no issue has this status for more than a minute or so, and if we had the option of hiding it from the UI, we would.
Registration and Account Management
Our JIRA is integrated into the rest of re-motion project’s community platform. This means, primarily, that you can use the same user account when reporting issues, participating in our forum, or our Wiki.
Unfortunately, those two are still work in progress, but the context is relevant since you can’t register directly with JIRA but instead have to visit our portal site. Afterwards, you can use the same user credentials across the entire re-motion web-site. [Update 2012: You can register on the login-page of jira and use the same credentials for wiki and the forum]
What’s going on behind the scenes? From time to time, you will notice that the JIRA Sync user has updated a task, sometimes even re-opening and then closing it. This activity originates in our backend service and is used to sync re-motion’s JIRA with our internal issue tracker here at rubicon. We tried to make it as unobtrusive as possible, but unfortunately, we can’t hide its activity completely. So, if you have activated e-mail notifications, you can safely ignore those sent by the “JIRA Sync” user. [Update 2012: The sync-process has been retired. Instead, we’re now using Atlassian’s application-link feature.]
So, I guess that’s it for today,
I’m happy to announce that re-linq is going to support partial trust environments out of the box starting from build 1.13.63, which is due at the end of this week. Once it’s ready, you can get it here: http://www.re-motion.org/builds/. The SVN trunk already includes the partial trust feature.
re-linq can also be used from ASP.NET medium trust, but some LINQ query features (such as anonymous types created by the C# compiler or access to local variables from a query) require reflection permissions not available in default medium trust. Note that the C# compiler often automatically uses anonymous types when you write a LINQ query with multiple from clauses or joins.
In order to be able to parse such queries using re-linq (or any other LINQ provider, I assume), the ReflectionPermission must explicitly be granted with the MemberAccess permission flag in addition to the default ASP.NET medium trust permissions.
Copied from our mailing list. Please post any questions about this feature to the list.
The re-motion team has worked on this for quite some time: Michael Ketting gathered the requirements, Patrick Größ wrote a JIRA integration tool based on Michael’s instructions, I integrated that tool into our build system. And now we’re finally “done” – we finally have release notes for our weekly re-motion builds.
Here is a partial screenshot of the ones for version 1.13.43:
The re-linq build packages also have dedicated release notes which only include re-linq issues:
The release notes are automatically generated from our JIRA issues; we try to create descriptive issues for each change we are making. This involves new features and bugfixes as well as breaking changes, performance improvements, and so on. We started doing so some time ago, in anticipation of having auto-generated release notes, and so we’ve been able to generated release notes for older builds as well. This means if you’re currently using 1.13.23 and want to know what has changed since then, you can simply look up all the release notes starting from 1.13.24 (be aware, though, that older release notes might not be as complete as the new ones). The downloadable ZIP packages contain the release notes of all the builds created for the respective minor version (1.13, currently).
Here’s what the download package for build 1.13.43 contains:
All in all, I hope this will make it easier for you to determine when to upgrade your re-motion versions; and it might also give some meaning to the “COMMONS-1234” keys we’re always referring to in our SVN commit messages 🙂
I’ve already alluded to it on our mailing list, and now it’s official: starting with build 1.13.41, the re-linq assembly Remotion.Data.Linq.dll is now a stand-alone DLL. Over Christmas, Matthias Görtler spent quite a few hours rewriting those parts of re-linq that used to depend on the rest of re-motion. Here’s the result:
1.13.40: What a difference a reference makes…
1.13.41: Only standard libs, please
As you can see, the reference to Remotion.dll also added indirect dependencies on Remotion.Interfaces.dll, log4net, and Castle DynamicProxy 2. By removing that reference, re-linq now only depends on .NET framework assemblies. That should facilitate interoperability for projects that already use a specific version of log4net or DynamicProxy.
If you have any questions, just post them to our discussion group.
Fabian had the news first, but here it is: We wanted to take re-motion to CodePlex already, but LGPL v3 is not a valid license there. We modified the license headers of re-motion from LGPL v3 to “LGPL v2.1 or later”, so we should be cool with CodePlex.
Here’s the link: relinq.codeplex.com
All the stuff, including the Subversion repository, is still here on re-motion.org, and CodePlex has only a few links and downloads. That’s a first step, and we believe that some people will learn about re-linq only through its presence on CodePlex.
Later, we plan to decouple re-linq from the rest of re-motion, since there is not a lot of shared code: re-linq uses almost nothing from re-motion, and it should therefore be a separate project, with a separate release cycle.
re-store (re-motion’s ORM) uses re-linq and will continue to do so. In upcoming releases, re-store will be treated just like any other downstream project from re-linq’s perspective.
As Fabian recently posted, re-linq just got its body parts reassembled. This is good news for everyone who wants to create a capable LINQ provider. Fabian has the details, but let me phrase it for mere mortals like myself.
Creating LINQ providers used to be a pain. It looked easy from a distance, since LINQ is similar to most query languages in many ways (including SQL, NHibernate’s HQL, Entity Framework’s Entity SQL or even the querying part of XQuery). How hard can it be to transform a LINQ abstract syntax tree (AST) to any of those? The problem is, LINQ won’t give you such a tree, but something entirely different that is really hard to translate to anything.
The main trick of re-linq is that it gives you exactly what you’d have expected in the first place: an AST that resembles the LINQ statement you wrote. Plus some infrastructure to make your transformations (including visitors and a few stock transformations you’re likely going to need) and other stuff (like creating additional queries for eager fetching).
Now finally creating a LINQ provider is as simple as you would think. You don’t know what the hack transparent identifiers are and how they make your live harder? Now you don’t have to. We believe it’s perfectly possible to create a very powerful LINQ provider in a few weeks. If you want to support every conceivable LINQ query, it’s going to be a bit more, depending on how different your target language is from LINQ. But still, there’s a lot in re-linq that helps you achieve even that.
This probably matters most for the NHibernate team. They just released a LINQ provider that supports basic query scenarios, but they plan to switch to a new one that uses their new HQL AST model. Using re-linq, they will hopefully be able to quickly create a provider that surpasses the old one’s capabilities by far. Steve Strong is already working on it.
As I explained in my last post, the idea of re-linq is to transform ugly IQueryable-based ASTs into nice query expression ASTs (from … select).
A discussion with Frans Bouma, distinguished LINQ tamer, brought up a point we discussed for quite a while internally: does this approach limit our ability to express every conceivable LINQ query? And if not, will it still provide value, or will this result in ASTs that are just as hard to use as IQueryable ASTs, only different?
Frans brings up the following example:
from c in md.Customer.Where(x=>x.Country=="Germany") join o in md.... select c.Orders;
Initially we were thinking that re-linq would just lead to a 80+% solution, because using Queriable’s extension methods just lets you do things that cannot be expressed directly using query expressions. (That was good enough for us anyway, since all we wanted to do back then was to get serious – but not necessarily complete – LINQ support for re-store, our own ORM. So we just went ahead.)
When we thought about turning re-linq into a much more generic beast, we revisited this assumption and discovered that for any direct use of Queryable there is not only a way to express the same using query expressions (from … select notation). It is also easier to process these expressions, because when we get down to it, using a full LINQ provider is only useful if we transform the query to a powerful enough query language, like SQL, HQL or Entity SQL. And those languages usually support sub-queries, but none of them supports anything even remotely similar than direct invocations on Where(), SelectMany() etc.
For Frans’ example, that would be:
from c in (from x in md.Customer where x.Country=="Germany" select x) join o in md.... select c.Orders;
The transformation result might not be shorter, but it’s much easier to process and transform because
- it is structurally similar to the output model,
- you can use existing transformations in a modular way for sub-queries, and
- everything strange (like transparent identifiers) has already been dealt with at that point.
A more intelligent transformation could transform the whole thing to just
from c in md.Customer join o in md.... where x.Country=="Germany" select c.Orders;
These queries are semantically identical (i.e., there should only be a difference if you execute these statements imperatively using LINQ to objects). This is a very interesting transformation that does a nice optimization. For any simple SQL-emitting back-end, this would also lead to much cleaner SQL queries without the back-end even being aware of this particular optimization).
At the end of the day, this is a matter of how many special cases you want to support. LINQ to SQL is known to go out of its way to produce the simplest SQL possible even for frightening LINQ queries. But some optimizations might be even easier to achieve using re-linq, because the patterns are more easily recognizable.
Either way, moving this kind of optimizations into a shared OSS project sounds more promising to me than solving it for every single provider. It doesn’t only reduce the overall effort, it also means that all optimizations that are done on the query model itself can be shared between providers. Everyone who thinks that re-linq cannot do some trick that they absolutely need is welcome to contribute to re-linq. (For instance, we here at rubicon are not particularly interested in optimizing LINQ joins, because joins are usually implicit in ORMs, and a good DB like SQL server would recognize the pattern anyway and come up with identical execution plans for both queries. But if someone else builds this optimization (and it won’t cost us an arm and a leg in terms of performance), we will gladly use it.)
Now we’re not there yet, and I cannot promise that we’re not going to discover situations that are more difficult to solve than we anticipated, but we’re confident that this is not a dead-end approach that will just fail for more complex scenarios. The design behind re-linq’s transformations is not trivial, and we are quite confident that it is a solid ground to build on. I’m sure Fabian will find some time to go into internals when he’s back at work.
I’m pretty sure that everyone who has ever tried to create a serious LINQ provider has suffered through the same stages:
- Fascination: The way C# 3’s new features were combined to create query comprehensions are cool. Lambdas, anonymous types, extension methods, and all the rest are really useful language features on their own, and the query expression syntax (from … select) is really just a very thin layer on top of it (although they somehow managed to get Monads into the game).
- Excitement: Via IQueryable and Expression<Func<…>>, we even get ASTs for the queries we write, and we can process them to generate SQL!
- Hope: LINQ query expressions look much like SQL, so there must be a simple way to transform one into the other.
- Frustration: The Queryable class makes a big mess out of those nice-looking query expressions. As we look at more complex queries and read the C# specs, we realize that the transformation from query expressions into method invocations follows some strange rules. They are very necessary for creating expressions that can be compiled and executed in C#, but unfortunately, we get the same bloated expressions for our ASTs. Now transformation to SQL doesn’t look so simple anymore. Transparent identifiers anyone?
- Despair: System.Core does not contain any useful code to deal with those ASTs. LINQ to SQL does, but it doesn’t expose it (unfortunately, because LINQ to SQL is one oft the few LINQ providers that can deal with anything you throw at it).
- Capitulation: We settle with a simple LINQ provider that understands just a primitive subset of LINQ (and throws exceptions for everything else), or we find that other LINQ providers might be good enough.
OK, now the last point is an exaggeration. Not everyone surrenders to the complexity of LINQ, or we would not have any useful LINQ providers. But few people actually get past this point, just have a look at Frans Bouma’s 15-part series Developing Linq to LLBLGen Pro. And honestly, it’s a real PITA that everyone would have to do this from scratch.
If we look at the nice syntax that query expressions have, it’s really a pity. Why can’t we have ASTs that resemble this beautiful syntax? After all, they are so much more like anything we’d want to transform them to (except for compiled code). SQL, XQuery, NHibernate’s HQL, Entity Framework’s Entity SQL, you name it.
Starting last year, we created a LINQ infrastructure that supports our own O/R mapper (re-store), and we started from exactly this point: To reduce complexity, we’d transform LINQ ASTs back into ASTs that resemble the original query expression syntax. (In some cases this is not the original syntax, since the extension methods of Queryable can also be called directly. But in compiled code this makes no difference, so we transform everything into query expressions.)
The result is re-linq. If you are thinking about writing a LINQ provider that supports more than just from/where/select, I’d like to encourage you to take a close look at re-link. Also, do not hesitate to contact us about missing features or chances to participate. (Unfortunately, providing public access to our issue tracker and a mailing list are still on our to-do list, but supply will follow demand!)
Fabian has written a short paper about re-linq (including open issues), and we’re going to publish more information in this blog soon. Stay tuned.