Getting your priorities right

I want to take a look at implementing prioritisation in redis.workflow. Currently, all workflows have the same priority; redis.workflow is all about throughput. If all tasks are going to be completed by some SLA-agreed time then it doesn’t technically matter which order everything is run in, and we can run through workflows in any order, as long as there’s no slack time in task turnover.

However, there contexts in which pure throughput isn’t enough. Sometimes its social: someone arbitrarily wants to see a graph showing a steady completion of recognizable sets of workflows. Sometimes it’s based on operational information: not all workloads are heterogeneous, and might have long-tail task duration distributions; and those long runners will breach your delivery SLA if you start them too late.  Sometimes you have combined workloads: the system might be responsible for both batch throughput calculations and ad-hoc calculations issued directly by someone who’s waiting for an answer, and wants some guarantee that they’re not wedged behind a freight train of batch work. Sometimes it’s simply SLA based: you have 24 hours worth of work, but some of it needs to be delivered by 8am, and some of it by 5pm.

In many ways then a pure throughput system might not meet all requirements, and in each of the scenarios above a decision needs to be made about each item of work: do I need to get this work done first, or that? The system needs to understand work priority.

In detail it’s also worth considering where the priority lies. Currently I’m considering prioritising workflows only relative to each other. That said it’s as easy to implement task-level prioritisation as it it to implement workflow-level prioritisation; doing the former gives me a little more flexibility, and I (or any client) could replicate workflow-level prioritisation by giving every task in a workflow the same priority.

Prioritisation itself can be simple or sophisticated, but will definitely have consequences for the structures used in redis.workflow.

Currently redis.workflow uses a queue of submitted tasks, which is fine for a simple throughput scenario. I could extend it to make practical binary priority decisions by enqueuing high priority tasks at the front of the queue, and low priority tasks at the back, without the need for a new structure.

If finer grained priorities are required then things get a little harder. If the number of levels of priority is small then having a queue per priority might be feasible; in fact, combined with the enqueue-to-front-or-back trick you only need half the number of queues as you have priority levels.

If very fine-grained or arbitrary priorities are needed, then I’d need to start thinking about maintaining relative order in the queue or list. Insertion at a given priority could happen between work of higher and lower priority, which would require iterating through the list to work out where to put it, or just enqueuing, then sorting the list. That’s likely to perform underwhelmingly; a more sophisticated solution would be to add a lookup that identifies priority boundaries by queue position.

Each of these is a reasonable way of managing prioritised work. Clearly enabling the use of arbitrary priorities would allow clients to choose, and even use simpler (say binary or even fall back to unary) prioritisation, but come at a cost in terms of complexity.

The binary high/low priority solution – enqueuing to front or back – is relatively easy to imagine in my existing code, as it’s already placing work at the back. Jumping to arbitrary priorities is harder, but it’s worth considering what would need to be done.

To manage arbitrarily prioritised work, I’d need to sort the work submitted in some way. Redis does have a sorted set, which offers O(log N) lookup and insertion by score. I’d need to be able to remove the highest-scoring task, which I would be able to do with

zremrangebyscore submitted:<type> <minPriority> <maxPriority> limit 0 1

except that there’s no limit out-of-the-box for zremrangebyscore. Instead, I’d have to use zrange to return a sorted list of items (although this sorts from lowest to highest score, meaning I need to decide whether 0 is low or high priority) then use zrem to remove the task.

I implemented this sorted-set-based prioritisation and ran it thorugh the benchmark app, giving each task the same priority. It was noticeably slower, taking about 485 seconds to run through the benchmark (a churn of about 1.8k tasks per second). Task and’ actual workflow completion seemed to keep pace with submission, but handling of the messages announcing workflow completion seems to fall behind, suggesting that it’s not scaling well in this example.

The benchmark needs to be extended to test workflows with varying numbers of tasks, and tasks with varying priorities.

For this moment I think that’s enough; there’s a wide range of ways of tackling the prioritisation problem, but the implementation of each is fairly straightforward in Redis. The way I’ve implemented it gives it a certain amount of flexibility – a client can configure priority at task or workflow level, but does slow things down. It’s possible the other, simpler methods have a less detrimental effect on performance; as the difference in performance is so significant it might make sense to consider making use of prioritisation configurable, although that’ll be difficult as it has fundamental consequences for the structure of the data in Redis.

Task type differentiation and looking a bit more SoA

At the moment all tasks are considered the same. If a client wanted to process only “widget painter” tasks it wouldn’t be able to, unless it pulled tasks from the queue then decided whether to process them or not. If not – there’s not much it could do, as the tasks are off the queue. It could throw them away, which isn’t very constructive.

More practically, the client could hand tasks off to components suited to dealing with them. This complicates the client architecture, as it introduces the need for a broker component that can do a fine-grained distribution of work, along with internal queuing based on task type. That can quickly become a full-blown project in itself.

To avoid the need for a broker component (but also not to obviate them entirely) it makes sense to handle this queueing of tasks by type in the workflow management domain; to advertise submission of tasks of each type using a distinct channel, and allow clients to subscribe to channels that advertise submission of tasks that they can handle. In my workflow system that means having a dedicated submitted queue for each arbitrary task type, and in turn, submission messages would need to be published on dedicated channels.

The supporting functionality would all need to become task-type aware – popping a task would involve knowing what task type to pop; pause and release would need to know which queues to place tasks back in; and so on.

Not only do we need dedicated submitted:type queues, but if we’re going to ask questions about all the submitted tasks then I’ll need a submittedTypes set just so I can locate all the submitted:type queues that are in use.

Finally; what should the task type specified be? The most straightforward immediate answer is for it to be an arbitrary string. That means we’ll be creating an arbitrary subschema of lookups that map a set of client-defined types with tasks. That looks like the beginning of a general system I have in mind that allows the client to associate any number of arbitrary properties with tasks, not just a type; but to meet this use case – allowing the client to differentiate tasks by a given type – this should be enough.

Pausing for thought

This post relates to code changes in this commit to my redis workflow project on GitHub, under this issue .

In my workflow project it’s time to consider how to pause a given workflow, and the first thing is to decide on what pause actually means.

At first sight it means “hold everything that’s running” and “stop anything new from starting”, but that turns out to be difficult to implement effectively generally, and has some particular wrinkles when using Redis.

It looks easy to stop anything new from starting: I can move all tasks for a given workflow to a state that means they won’t get picked up; let’s say that state is labelled paused, because frankly, this gets complicated enough. To release the tasks all I’ll need to do is move all the paused tasks for this workflow back to the submitted state.

Except: the fact I’m asking for all the submitted (or paused) tasks for a specific workflow means we have a decision to make. At the point of time this functionality was added all these tasks were together in a global submitted state. I could loop through every id in the submitted set, use those ids to lookup tasks and fetch their workflow ids, then decide whether each one needs to be acted on.

Alternatively I could populate an additional set when tasks transition to each new state, keyed by workflow id. When tasks in workflow 1 move to the submitted state, for example, we could add them to a distinct set with key submitted:1. That makes finding tasks in a specific state for a specific workflow much easier, and is the approach I’m taking. It means S x W new sets at worst, where S is the number of states and W the number of workflows. At the scale I’m aiming for that means ~600,000 sets, assuming no clean-up after workflows.

So, when someone asks to pause a workflow we’ll just shift all tasks from submitted state to paused, yes? And on release, we should just move everything from paused to submitted? Not quite.

I’m deciding to let running tasks run to completion. As currently running tasks complete (or fail) they might cause further tasks to be submitted, which isn’t the behaviour we want. It makes sense then to have the completion logic know that if the completed task has moved to the paused state then it shouldn’t cause its dependencies to be submitted, although it can be transitioned to is final desired end state (complete, or failed).

However, when it comes to releasing the workflow again, what should happen to the children of those completed tasks, if all their parents have completed? If I don’t do something then they won’t get picked up when someone asks to release the workflow – there’s nothing left to complete and trigger their submission.

Instead, if a paused task completes and its children become eligible for submission they should be placed in the paused state themselves. Then, when someone requests that the workflow be released, they’ll shift back to the submitted state.

On release, then, that takes care of ensuring I don’t lose track of any tasks, assuming that the release command comes after all outstanding running tasks have completed.

If the release request comes while tasks are still running down, however, releasing the workflow  will be difficult. We won’t know whether to move the tasks to the submitted or running state, because there’s no differentiation in source state for tasks in the paused set. We’ve lost that information. If we shift a task that’s being executed somewhere to the submitted state on release then it might get run twice in parallel, or get confused when it comes to submit and finds it’s not in an expected state.

The (hopefully) final puzzle piece is then to record the last state of each task, so when a release request is issued we can move it back to the appropriate state. Running tasks should get moved back to running, submitted tasks to submitted. Tasks that have completed already won’t be in the paused state, so won’t get considered for release, and we’ll need to tag newly runnable tasks with a fake last state of submitted.

That should means we don’t lose any information about the state of the workflow during pause and release operations. For such a simple thing it’s still quite possible to get wrong; making a few assumptions and decisions explicitly helps a great deal. I decided to:

1. Stop anything new from starting.

2. Let executing tasks run to completion.

3. Create lookups to answer specific questions.

4. Avoid duplicate task execution.

all of which seem reasonable currently; let’s see how the next chunk of functionality changes that.

Loading SOS in windbg. Why is it never quite as easy as you hope it’ll be?

I started analysing a production crash dump at my desk, with a set of libraries that don’t match those installed on the production server. Typically windbg would complain:

> !clrstack
The version of SOS does not match the version of CLR you are debugging.  Please
load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.1026
SOS Version: 4.0.30319.18444
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is 
                in the version directory or on the symbol path
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks___.dll is on your symbol path.
            4) you are debugging on supported cross platform architecture as 
                the dump file. For example, an ARM dump file must be debugged
                on an X86 or an ARM machine; an AMD64 dump file must be
                debugged on an AMD64 machine.
 
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
 
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

The first step is to get hold of the correct libraries from the server:

cp \\server\c$\windows\microsoft.net\Framework64\v4.0.30319\sos.dll c:\temp
cp \\server\c$\windows\microsoft.net\Framework64\v4.0.30319\mscordacwks.dll c:\temp
cp \\server\c$\windows\microsoft.net\Framework64\v4.0.30319\clr.dll c:\temp

In theory at that point, I should just be able to run

> .load c:\temp\sos.dll
> !clrstack

but sadly still no joy. If I run

> .chain

it tells me that it knows about more than one version of SOS

…
Extension DLL chain:
    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos: image 4.0.30319.18444, API 1.0.0, built Wed Oct 30 21:40:20 2013
        [path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos.dll]
    c:\temp\sos.dll: image 4.0.30319.1026, API 1.0.0, built Thu Jul 03 07:58:50 2014
        [path: c:\temp\sos.dll]
…

I can unload the non-useful one

> .unload C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos

and then run SOS commands happily.

Paint dries, kettles boil, threads block; waiting can be hard to do

Today I added a simple wait-then-retry mechanism to a piece of code that occasionally bites my current client. The existing code tried an action and just dumped out if an exception was thrown. In this case, some of the thrown exceptions weren’t fatal -- the command could be retried in a short while (it’s actually an attempt to update a database under transient load).

At first I implemented the wait-before-retry simply, with a deliberate trade-off between complexity and accuracy – as a series of Thread.Sleep steps, so that after each step the thread could yield to another. It looked a little like this:

class ActWithSimpleRetry
{
    private int _allowedAttempts;

    private int _delay;

    private Action _customAction;

    public ActWithSimpleRetry(int allowedAttempts, int delay, Action customAction)
    {
        _allowedAttempts = allowedAttempts;
        _delay = delay;
        _customAction = customAction;
    }

    public void Run()
    {
        var attempt = 0;

        while(attempt++ < _allowedAttempts)
        {
            try
            {
                // _customAction should throw a CustomException to simulate 
                // the behaviour I wanted to retry on.
                _customAction();

                return;
            }
            catch (CustomException)
            {
                if (attempt == _allowedAttempts) throw;
            }

            YieldingWait(_delay);
        }
    }

    private void YieldingWait(int delay)
    {
        var elapsed = 0;
        var step = 500;

        delay = 1000 * delay;

        while(elapsed < delay)
        {
            System.Threading.Thread.Sleep(step);

            elapsed += step;
        }
    }
}

There we go, professional job done. Looks like I’ve thought about it, it’s not just sat there blocking the thread for delay milliseconds. But hang on: do I even need to yield?

Constructing the wait like this is more complex than it need be: a single Thread.Sleep would do, as I’m not worried about freezing a UI; I’m not checking on “Cancelled” state; and I’m not waiting before synchronizing threads.

If I had been worried about synchronizing threads then this implementation would be clunky, leading me to poll on some shared state. Moreover, the waiting thread could always yield to another, and it might be a while before the waiting thread is able to execute its next wait step. The way I had implemented the wait – with no reference to actual time elapsed – is naive.

A simple Thread.Sleep would be a more honest solution to the problem.

What about situations where you do want to worry about timing accuracy and thread freezing – and coordinating with a cancellation signal on another thread? The alternative is to use a Timer running independently of the main thread. A first attempt might look like this:

public void Run()
{
    var attempt = 0;
    var timerEvent = new EventWaitHandle(true, EventResetMode.AutoReset);

    var timer = new System.Timers.Timer(_delay * 1000);
    timer.Elapsed += (sender, args) => timerEvent.Set();

    while (attempt++ < _allowedAttempts && timerEvent.WaitOne())
    {
        try
        {
            _customAction();

            return;
        }
        catch (CustomException)
        {
            if (attempt == _allowedAttempts) throw;
        }

        timer.Start();
    }
}

The Timer is created with the desired wait, and configured to signal a WaitHandle on expiry. The main thread then Waits on that handle (and checks the number of attempts) before trying again.

Here I’ve gained some control over the duration of the wait at the cost of some complexity. However – I’m still blocking on the main thread at timerEvent.WaitOne! If I want to entertain the possibility of a clean shutdown, I’ll need to also take a look at waiting on a cancellation signal. That would look something like this:

public void Run()
{
    var attempt = 0;
    var timerEvent = new EventWaitHandle(true, EventResetMode.AutoReset);
    
    // in practice this would be a member field that could be Set
    // from a separate "cancellation" thread
    var cancelEvent = new EventWaitHandle(false, EventResetMode.ManualReset);

    var timer = new System.Timers.Timer(_delay * 1000);
    timer.Elapsed += (sender, args) => timerEvent.Set();

    while (attempt++ < _allowedAttempts && WaitOnSignals(timerEvent, cancelEvent))
    {
        try
        {
            _customAction();

            return;
        }
        catch (CustomException)
        {
            if (attempt == _allowedAttempts) throw;
        }

        timer.Start();
    }

    // handle the drop through here; probably a cancellation
}

private bool WaitOnSignals(WaitHandle timer, WaitHandle cancel)
{
    var handleActions = new[]
        {
            new { Handle = cancel, Result = false },
            new { Handle = timer,  Result = true  },
        };

    var handles = handleActions.Select(x => x.Handle).ToArray();

    int signalled = WaitHandle.WaitAny(handles);

    return handleActions[signalled].Result;
}

Which is better (and reminds me that Chris Oldwood is always reminding me to wait on two events, not one): it’ll break out of the processing if there’s a cancellation. However, it still blocks at WaitHandle.WaitAny. If, finally, I want to yield to other threads periodically then I’ll have to break out of the wait periodically:

private bool WaitOnSignals(WaitHandle timer, WaitHandle cancel)
{
    var handleActions = new[]
        {
            new { Handle = cancel, Result = false },
            new { Handle = timer,  Result = true  },
        };

    int signalled;
    var handles = handleActions.Select(x => x.Handle).ToArray();

    while((signalled = WaitHandle.WaitAny(handles, 10)) == WaitHandle.WaitTimeout)
    {
    }

    return handleActions[signalled].Result;
}

Finally, here I get actual elapsed time before the timer expires and allows work to continue, and can avoid blocking on the timer wait by using Waithandle.WaitAny to automatically handle both timer and cancellation signals. There is a risk that it won’t react to a cancellation or complete signal as soon as possible because it yields control after 10 milliseconds – but if you want to allow other threads a look-in you have to accept that they’ll take some processing time, and that on top of the time taken for context switches.

Bearing in mind the added complexity of this wait and the loose restrictions around thread freezing and cancellation for the original app – I’d be sorely tempted to go for a simple, honest Thread.Sleep instead.

C# performance counters impact performance? No, but it seems that Chart.SaveImage does.

A little thought later and it looks like no, they don’t impact performance. What I was seeing was the effect of prepping and drawing a chart – on about the same timescale that I was querying for information.

Diagnosing the problem was straightforward; I crosschecked the processor usage for the tool with perfmon. After configuring it to avoid saving down a chart the processor usage dropped to zero.

It is a good example of misleading yourself while looking at performance stats. I could see usage spiking on a period that looked suspicious. I used the VS 2010 in-built profiler and saw that PerformanceCounter.NextValue() was by far the biggest contributor to CPU time, and put two and two together to make five.

I ran the app for a few minutes, rather than the one minute I originally tested with, and the stats inverted. Chart.SaveImage() was overwhelmingly responsible for CPU consumption, with the calling method, Agent.Graph(), taking up ~71% of samples.

Schoolboy misinterpretation, but as always a bit of thought and investigation soon shows up the problem.

Now. Why is Chart.SaveImage() so expensive, and is there a cheaper alternative?

Creating a chart programmatically in C# using DataVisualization.Charting

Means you have to reference System.Windows.Forms and System.Windows.Forms.DataVisualization, even though you might be doing this in a console application. That said, it also includes a set of data manipulation functions, some of which are even finance-specific:

using System;
using System.Drawing;
using System.Windows.Forms.DataVisualization.Charting;

namespace Sandbox
{
    class Program
    {
        static void Main(string[] args)
        {
            // set up some data
            var xvals = new[]
                {
                    new DateTime(2012, 4, 4), 
                    new DateTime(2012, 4, 5), 
                    new DateTime(2012, 4, 6), 
                    new DateTime(2012, 4, 7)
                };
            var yvals = new[] { 1,3,7,12 };

            // create the chart
            var chart = new Chart();
            chart.Size = new Size(600, 250);

            var chartArea = new ChartArea();
            chartArea.AxisX.LabelStyle.Format = "dd/MMM\nhh:mm";
            chartArea.AxisX.MajorGrid.LineColor = Color.LightGray;
            chartArea.AxisY.MajorGrid.LineColor = Color.LightGray;
            chartArea.AxisX.LabelStyle.Font = new Font("Consolas", 8);
            chartArea.AxisY.LabelStyle.Font = new Font("Consolas", 8);
            chart.ChartAreas.Add(chartArea);

            var series = new Series();
            series.Name = "Series1";
            series.ChartType = SeriesChartType.FastLine;
            series.XValueType = ChartValueType.DateTime;
            chart.Series.Add(series);

            // bind the datapoints
            chart.Series["Series1"].Points.DataBindXY(xvals, yvals);

            // copy the series and manipulate the copy
            chart.DataManipulator.CopySeriesValues("Series1", "Series2");
            chart.DataManipulator.FinancialFormula(
                FinancialFormula.WeightedMovingAverage, 
                "Series2"
            );
            chart.Series["Series2"].ChartType = SeriesChartType.FastLine;

            // draw!
            chart.Invalidate();

            // write out a file
            chart.SaveImage("chart.png", ChartImageFormat.Png);
        }
    }
}

Gives the following result:

RRDTool, and C# .NET bindings

RRDTool is pretty much ubiquitous in Linux/Unix cluster environments – or at least, it is in those I’ve worked with. It provides simple but highly reliable round-robin database services into which you can dump pretty much any metric you like. It’ll keep a running dataset at the granularity and duration you define; enable you to export data in chart and xml form; and even perform some useful analysis.

These days I’m typically working in Windows environment, and typically most clients I work for need some sort of dashboard to aggregate information about their clusters and grids. For a long time RRDTool was a pain to build on Windows, and there was no single .NET wrapper. The front runner – NHawk – actually spawned rrdtool processes rather than wrapping the underlying libs direct.

Now it turns out that someone’s contributed .NET bindings to the trunk of RRDTool. I figured I’d see if I could get it working, on 64bit Win7 (although actually x86 is fine for my app), under Visual Studio 2010.

I checked trunk out from SVN and took a look.

The first thing is to set up the dependencies – I tried to take a short cut and begin with a GTK+ package, but found it’s best to start with the files mentioned explicitly in the WIN32_BUILD_TIPS file.

There were a few apparently non-serious build errors in the rrdlib project. In several places explicit casts from void* were required on malloc, realloc. Also, there was some skipping of variable initialization in switch statements – latter can be fixed just by enclosing in a block.

Running through the tutorial with rrdtutorial raised one problem – an error claiming “No positional legend found”. The error was generated in rrd_graph_helper.c. The log showed that this was a work in progress, so I updated to the preceding version, built and it worked apparently fine.

After that I built the .NET bindings – basically fine, although I wrangled everything to x86 in the end just to get started. But then – nothing was being exported. There’s a .def file in the lib solution, but it doesn’t appear to exist any more. So: I wrapped a copy of the public method declarations in #ifdef WIN32, and prefixed with __declspec(dllexport), then rebuilt the .NET binding solution and: success.

All looks good to go. Next step is to grab metrics from windows performance counters in a C# app and see what I can present through RRDTool.

Out (and ref) parameters in Actions, Funcs, delegates and lambdas

I ran into a couple of wrinkles I wasn’t aware of before when coding up a facade for the ThreadPool for tests today, along the lines of the Path, Directory etc facades I’ve done before.

I wanted to insert a replacement ThreadPool so I could test the GetAvailableThreads method, which takes two out parameters.

You can’t quite specify the lambda as you might expect for the method implementation. This initial attempt, for example, doesn’t work:

public static class ThreadPool
{
    public static Action<int, int> GetAvailableThreadsImpl =
        System.Threading.ThreadPool.GetAvailableThreads;

    public static void GetAvailableThreads(out int workerThreads, out int ioThreads)
    {
        GetAvailableThreadsImpl(out workerThreads, out ioThreads);
    }
}

I suspect this is to do with the calling style here -- but need to understand why it happens in more detail. You can work around it by just creating your own delegate class:

public delegate void GetAvailableThreadsDel(out int x, out int y);
 
public static class ThreadPool
{
    public static GetAvailableThreadsDel GetAvailableThreadsImpl =
        System.Threading.ThreadPool.GetAvailableThreads;

    public static void GetAvailableThreads(out int workerThreads, out int ioThreads)
    {
        GetAvailableThreadsImpl(out workerThreads, out ioThreads);
    }
}

Then follows a small wrinkle when defining a replacement behaviour in your test. You can, of course, create your own method with the right signature:

[Test]
public void Blah()
{
    int callCount = 0;
    ThreadPool.GetAvailableThreadsImpl = GetAvailableThreadsStub;
}

private int callCount = 0;
private void GetAvailableThreadsStub(out int x, out int y)
{
    if (callCount == 0)
    {
        x = 5;
        y = 5;
    }
    else
    {
        x = 10;
        y = 10;
    }

    callCount++;
}

It turns out you can define out parameters in a lambda however -- but you must also explicitly state the parameter type as well, unlike normal:

[Test]
public void Blah()
{
    int callCount = 0;
    ThreadPool.GetAvailableThreadsImpl = (out int x, out int y) =>
        {
            if (callCount == 0)
            {
                x = 5;
                y = 5;
            }
            else
            {
                x = 10;
                y = 10;
            }

            callCount++;
        };
    }
}

Again, I'm not entirely sure how it works here, and need to investigate further. For now, just a wrinkle to remember.

Wiping the state clean: static stateful classes in NUnit tests

I’m automating some tests that are currently run manually in a batch script that spins up the same executable multiple times, with varying options. As always, I’d like to get the tests started without modifying existing code, and preferably without adding any new code.

Unfortunately the application poses a few problems. The entry class has a curiously-implemented singleton behaviour to ensure that only one is running in a process. It also keeps a static ref to a set of command line options, but doesn’t give the client any way of clearing those options to restart processing. I also need to inject some behaviour that would allow the test to sense that the correct thing had been done. The class looks a bit like this:

namespace AppDomainInNUnit
{
    internal class BaseApplication
    {
        public static BaseApplication Instance = null;

        public BaseApplication()
        {
            if(Instance == null)
            {
                Instance = this;
            }
            else
            {
                throw new InvalidOperationException("ctor fail");
            }
    }

    class Application : BaseApplication
    {
        static int Main(string[] args)
        {
            var app = new Application();
            return app.Run();
        }

        int Run()
        {
            _options.Add("option", "added");
            foreach(var option in _options.Keys)
                Console.WriteLine("{0} {1}", option, _options[option]);
            return 0;
        }

        public static Action<string> PostFinalStatus = s => Console.WriteLine(s);

        private static readonly Dictionary<string, string> _options = new Dictionary<string, string>();
    }
}

I could add methods to the existing code to clear the options and to handle the singleton behaviour, but that means changes. In principle there’s no need, as I can instantiate the classes I need in a separate AppDomain, and unload that AppDomain (and dump any static state) when I’m done.

To get this under test I need to do several things. First, I need to create a facade for the Application class that can be marshalled across AppDomain boundaries. Second, I need to consistently create and unload AppDomains in NUnit tests – I want to unload at the end of each test to dump the class’ static state. Finally, I want to inject some behaviour for the PostFinalStatus delegate that will allow the test to sense the state posted by the class.

So, first I need a facade for Application. This is partly because I need to create a class that can be marshalled across AppDomain boundaries. To use marshal-by-reference semantics, and have the instance execute in the new AppDomain, it’ll need to inherit MarshalByRefObject. Unfortunately I need to break the no-changes constraint, and add an internal hook to allow the facade to kick the Application class off.

The TestMain bootstrap class is a basically a copy of the existing Main( … ):

        internal static int TestMain(string[] args)
        {
            return Main(args);
        }

Then I have a simple Facade:

    public class Facade : MarshalByRefObject
    {
        public Facade()
        {
            Console.WriteLine("Facade default ctor in {0}", 
                        Thread.GetDomain().FriendlyName);
        }

        public void Run(string[] args)
        {
            Console.WriteLine("Calling to {0}.", 
                        Thread.GetDomain().FriendlyName);
            
            Application.TestMain(args);
        }
    }

Creating an AppDomain in an NUnit test is fairly straightforward, although it confused me for a while until I realised that the tests are run under a different application – in my case either the JetBrains TestRunner, or nunit-console. In either case the code is not actually available under the executable’s application base directory. The solution is to explicitly set the application base directory for the new test AppDomain:

        [Test]
        public void Test()
        {
            var callingDomain = Thread.GetDomain();
            var callingDomainName = callingDomain.FriendlyName;
            
            Console.WriteLine("{0}\n{1}\n{2}", 
                callingDomainName, 
                callingDomain.SetupInformation.ApplicationBase, 
                callingDomain.SetupInformation.PrivateBinPath);

            var domain = AppDomain.CreateDomain("test-domain", null, null);
            
            Console.WriteLine("{0}\n{1}\n{2}", 
                domain.FriendlyName, 
                domain.SetupInformation.ApplicationBase, 
                domain.SetupInformation.PrivateBinPath);
            
            AppDomain.Unload(domain);

            var setup = new AppDomainSetup() { 
                ApplicationBase = callingDomain.SetupInformation.ApplicationBase };
            
            domain = AppDomain.CreateDomain("test-domain", null, setup);
            
            Console.WriteLine("{0}\n{1}\n{2}", 
                domain.FriendlyName, 
                domain.SetupInformation.ApplicationBase, 
                domain.SetupInformation.PrivateBinPath);

            var assembly = Assembly.GetExecutingAssembly().CodeBase;

            var facade = domain.CreateInstanceFromAndUnwrap(
                assembly, 
                "AppDomainInNUnit.Facade") as Facade;
            
            facade.Run(new [] { "some", "args" });

            AppDomain.Unload(domain);
        }

There’s a lot of cruft in there, but the intent is to show which AppDomain the class is actually being instantiated in, for clarity.

Finally, I want my tests to be informed of the Application’s final status when it posts it. This is tricky; I’ve been able to do it by passing in a delegate that sets some property data on the test AppDomain, then querying the AppDomain property data during the test assert stage. First the facade needs a method that allows me to set the post behaviour for Application:

        public void SetPostBehaviour(Action<string> behaviour)
        {
            Application.PostFinalStatus = behaviour;
        }

Then I need to pass a delegate with the right behaviour in during the test, and assert on the value of that data before unloading the test AppDomain:

            ...
            var assembly = Assembly.GetExecutingAssembly().CodeBase;

            var facade = domain.CreateInstanceFromAndUnwrap(
                assembly, 
                "AppDomainInNUnit.Facade") as Facade;
            
            var posted = false;
            facade.SetPostBehaviour(s => Thread.GetDomain().SetData("posted", true));
            
            facade.Run(new [] { "some", "args" });

            Assert.IsTrue((bool)domain.GetData("posted"));
            ...

So, it's possible to work around these static classes without too many intrusive changes to the existing code. Generally:

1. To instantiate and execute a class in another AppDomain the class should derive from MarshalByRefObject (see Richter, CLR via C#). You’ll get a proxy to the instance in the new domain.
2. Don’t tag with Serializable instead of deriving from MarshalByRefObject – you’ll get a copy of the instance in the other domain, and will still run in the original domain.
3. You can, but you shouldn’t invoke static methods or fields on your class that derives from MarshalRefByObject. It won’t give you the effect you expect: all static calls are made in the calling domain, not the separate domain. To make calls on static methods you need to create a facade or adapter that can be marshalled across the domain boundary, and have that make the static calls for you.
4. In the various CreateInstance… methods on AppDomain remember to use a fully qualified type name for the class you want to instantiate. This generally isn’t clear from examples like that in Richter, who dump all their classes in the global namespace (tsk).
5. If you’re creating an AppDomain in an NUnit test, remember that you might need to set the ApplicationBase for the domain to the directory holding your test code.
6. I wanted to get AppDomain.ExecuteAssembly(string, string[]) working, which would mean that I wouldn’t need a TestMain hook in the existing code. I’ve not managed to get it working yet, though. In any event, I’d still need a facade that I could marshal across the boundary in order to set the status post behaviour.

Preventing inadvertent changes to static classes

This is a followup to these posts, where I developed a way of opening up a seam along filesystem access in C#.

The last post highlighted a problem with the approach: when behaviour is changed by injecting a new delegate it is changed for all subsequent users of the class.

This is a small problem in tests – you just need to make sure that you reset your static class state in test setup and tear down.

In a production system however it could be a much larger problem. A developer could change the behaviour of the class without realising the consequences, and change the behaviour of the system in an unexpected way. Equally, they could deliberately subvert the behaviour of the system – but if they’re going to do that, they’d probably find a better way of doing it.

You could avoid this by treating this technique as only a short term measure to open a seam while refactoring legacy code, and rapidly refactor again to something more robust, perhaps using a factory or IOC container to inject different implementations for test or production code. This is entirely reasonable; the technique gets very quick and easy results, and you could definitely use it while characterizing legacy code and rapidly refactor away from it.

Alternatively you could make it somewhat lockable, and have it warn if code tries to change behaviour without. I’m musing now, because I tend to think about this technique in the sense I’ve just described. In principle though you could guard against careless changes in behaviour by having the class “locked” by default, and only allow changes to behaviour after first unlocking the class.

Tangentially this raises a question – is a physical lock a form of security by obfuscation and awkwardness?

To give an example, consider this test code for a notional File class:

    [TestFixture]
    public class FileTests
    {
        [SetUp]
        public void SetUp()
        {
            File.Unlock();
            File.CopyImpl = null;
            File.Lock();
        }

        [Test]
        [ExpectedException("System.InvalidOperationException")]
        public void File_ShouldWarn_IfBehaviourChangesWhenLocked()
        {
            File.CopyImpl = null;
        }

        [Test]
        public void File_ShouldAllow_BehaviourChangeWhenUnlocked()
        {
            File.Unlock();
            File.CopyImpl = (src, dst, ovrt) => { };
        }
    }

I don't see the lock as a strong lock -- just a flag to make a developer stop and think about what they're doing, and to allow a mechanism to raise an alert if it's done without thought. An implementation that matches these tests could look like:

    public static class File
    {
        private static Action<string, string, bool> _copyImpl = System.IO.File.Copy;

        private static bool _locked = true;

        public static Action<string, string, bool> CopyImpl
        {
            get
            {
                return _copyImpl;
            }
            set
            {
                if(_locked)
                {
                    throw new InvalidOperationException("IO.Abstractions.File is locked to disallow behaviour changes");
                }
                _copyImpl = value;
            }
        }

        public static void Unlock()
        {
            Trace.WriteLine("IO.Abstractions.File now unlocked for behaviour changes");
            _locked = false;
        }

        public static void Lock()
        {
            Trace.WriteLine("IO.Abstractions.File now locked for behaviour changes");
            _locked = true;            
        }

        public static void Copy(string source, string destination, bool overwrite)
        {
            CopyImpl(source, destination, overwrite);
        }
    }

The underlying implementation is the same as I presented before, with a couple of changes. There are now obvious Lock and Unlock methods; I'm using these to just set or unset an internal flag. I'm wrapping the point of injection of new behaviour for CopyImpl in a property, and using the set method to check the locking status. If the class is locked, it'll throw an exception to indicate that you're not meant to be changing it. The class is locked by default.

Exactly how useful this is in the long term I'm not sure. It's tempting to keep the Abstractions in place, as the class structure is simpler than the normal interface-and-factory style approach that you might use out of choice. It's possible that an IOC container might be abused in the same way as this technique, by a developer who wants a specific local behaviour but doesn't understand that their changes might have global consequences.

And I've not even touched on any possible threading problems.

Using static classes in tests

This is a followup from my previous post on mocking out static classes and injecting behaviour.

The disadvantage to injecting test specific behaviour into a static class is that if you’re not careful you can leak behaviour from test to test. This certainly happens in NUnit, in which a static class will be instantiated once for a test session – not once per test instance.

Take these psuedo-tests for example:

[Test]
public void FirstTest()
{
    IO.Abstractions.Path.CombineImpl = (path1, path2) => "some\\default\\path";

    Assert.AreEqual("some\\default\\path", IO.Abstractions.Path.Combine("some", "path"));
}

[Test]
public void SecondTest()
{
    // assume we're using the default implementation -- System.IO.Path.Combine

    Assert.AreEqual("some\\path", IO.Abstractions.Path.Combine("some", "path"));
}

FirstTest will succeed. If FirstTest runs first, then SecondTest will fail; if SecondTest runs first, SecondTest will succeed. the problem is that FirstTest sets the behaviour for Path.Combine and SecondTest doesn't override it.

The trick is to ensure you reset the behaviour in test setup and teardown. You can just reset to default behaviour in either:

[SetUp}
public void SetUp()
{
    IO.Abstractions.Path.CombineImpl = System.IO.Path.Combine;
}

or you might want to set the behaviour to null, and just set the defaults on teardown:

[SetUp}
public void SetUp()
{
    IO.Abstractions.Path.CombineImpl = null;
}

[TearDown}
public void TearDown()
{
    IO.Abstractions.Path.CombineImpl = System.IO.Path.Combine;
}

This latter approach has the advantage that -- if you null out all possible behaviour across the static classes used -- you immediately get to sense which methods are being hit in any given test, and where in the code under test it's being hit from.

Constructing a task occupancy time series from discrete task start and end times

When finding my way around a new distributed system I often want to see a chart of the number of tasks running on a system at the same time but only have discrete task start and end time data. Even if time series data already exists, it’s often not broken down in a meaningful way – say by task type, or final status.

So: how do I build a time series like this from a set of discrete start and end times?

I tackle this in three steps. First, I build and sort lists of start and end times. Secondly, I process these two lists to get a task event dataset: every start takes a running count of tasks up, and every end takes it down. Finally I process that event dataset to build a time series based on discrete time points.

Building sorted lists of task start and end times is pretty straightforward. I typically dump start and end time data from the database, which might look like this:

2011-07-19 04:02:45.0730000,2011-07-19 04:03:21.6000000
2011-07-19 04:02:45.0730000,2011-07-19 04:03:44.7030000
2011-07-18 22:58:52.1800000,2011-07-18 22:58:52.6670000
2011-07-18 22:58:53.3500000,2011-07-18 22:58:57.7700000
2011-07-18 22:58:52.1800000,2011-07-18 22:58:52.7200000
2011-07-18 22:58:53.3500000,2011-07-18 22:58:58.4030000
2011-07-18 22:58:52.1800000,2011-07-18 22:58:52.7100000
2011-07-18 22:58:53.3500000,2011-07-18 22:58:57.7700000
2011-07-18 22:58:52.1800000,2011-07-18 22:58:52.7630000
2011-07-18 22:58:53.3500000,2011-07-18 22:58:57.7700000

and then use the excellent FileHelpers library to load them in. I make some assumptions about the data – the earliest start is earlier than the first and point, at least.

After that, sort the lists in place in ascending time order.

var starts = new List<DateTime>();
var ends = new List<DateTime>();
foreach(var task in tasks)
{
	starts.Add(task.start);
	ends.Add(task.end);
}

starts.Sort((a, b) => 
	Convert.ToInt32(new TimeSpan(a.Ticks - b.Ticks).TotalMilliseconds));
ends.Sort((a, b) => 
	Convert.ToInt32(new TimeSpan(a.Ticks - b.Ticks).TotalMilliseconds));

Merging the two lists to create an event dataset is a little trickier. I do it by processing either the start or the end list in order at any time, swapping between them when the timestamps in one jump ahead of the timestamps in the other. I keep a running count of tasks (the occupancy) – every start takes it up, and every end takes it down.

int currentIndex = 0;
int otherIndex = 0;
int runningCount = 0;
var currentList = starts;
var otherList = ends;
var incdec = 1;

var timestamp = new List<DateTime>();
var value = new List<int>();

while (currentIndex < currentList.Count - 1)
{
	runningCount += incdec;
	timestamp.Add(currentList[currentIndex]);
	value.Add(runningCount);

	currentIndex++;
	if (currentList[currentIndex] > otherList[otherIndex])
	{
		var tempIndex = currentIndex;
		currentIndex = otherIndex;
		otherIndex = tempIndex;
		var tempList = currentList;
		currentList = otherList;
		otherList = tempList;

		incdec *= -1;
	}
}

This generates a relative time series, with offsets from the number of tasks running at the start of the dataset. To constrain this I’d need to work out what that initial number of tasks is. Generally I’d count the number of tasks that started but didn’t end before my time series starts.

Finally, create a discrete timeseries with evenly spaced points. I just generate a list of timestamp points and for each take the nearest previous recorded occupancy value. Generally I’ll also take a highwater mark, as otherwise you lose a feel of any peaky behaviour within between points.

var index = 0;
var dt = new TimeSpan(0, 5, 0);
var current = timestamp[0];

while (current < timestamp[timestamp.Count - 2])
{
	var next = current.AddTicks(dt.Ticks);

	var highwater = 0;
	while (index < timestamp.Count - 1 
		&& timestamp[index + 1] < next)
	{
		index++;
		if (highwater < value[index]) 
			highwater = value[index];
	}

	Console.WriteLine(current + "\t" + value[index] + "\t" + highwater);
	current = next;
}

You’ll notice this is quite loose; I’m not really worrying about interpolating to get values at exact time series points, and so on.

And that’s basically it. When building your series you can fold in extra information – about the task type for example – and use that to show how the system treats different task types.

Comparing fluent interfaces for guard clauses [part 4]

In the last post on fluent guard checks I showed an interface that aggregates state, removing the trapdoor effect associated with multiple guards. It works, but might in some environments be an unnecessary contributor to memory pressure.

Performance-tweaked fluent chained guards

Rick Brewster came up with an alternative syntax that only instantiates a state object on the first fail. Otherwise, it just passes null. Again, the guard checks are extension methods that hang off the state object – even (I didn’t realise this worked) if the state object is null:

Validate.Begin()
    .IsNotNull(message, "message")
    .SomeOtherCondition(message, "message")
    .Check();

The state object looks like (in this case, a simple equivalent – Rick’s is more sophisticated):

public class Validation
{
    private _exceptions = new List<exception>(1);
    public List<exception> Exceptions
    {
        get { return _exceptions; }
}

Validate.Begin() looks like:

public static Validation Begin()
{
    return null;
}

An example guard check looks like:

public static Validation IsNotNull(
    this Validation state, 
    object value, 
    string name)
{
    if(value == null)
    {
        if(state == null)
        {
            state = new Validation();
        }
        state.Exceptions
            .Add(new ArgumentNullException(name));
    }
    return state;
}

Check() just checks for any exceptions, and throws a ValidationException if there are any, which wraps the Exception list -- just like in part 3.

So, because the argument value and name aren't in the state object they need to be explicitly passed into each guard check; the check code is also more complicated because it does more than one thing.

This makes the syntax a little less clear to me, with a would-be-good-if-it-were-redundant Validate.Begin() always starting a guard chain, and with argument value and name sometimes appearing repeatedly.

Roundup

So, the decision about whether to avoid creating an unnecessary state object determines to some extent how clear that syntax is. Deciding whether to break fast or accumulate guard state just means that you might need to have something to terminate the chain.

Whether you really need to avoid the state object creation is really app-specific; you’d need to think and test. I’ve hit the need for chained guard checks a number of times though, so I’d definitely look to implement that. My ideal solution is somewhere between the two.

Also interesting: www.bikeshed.org.

Comparing fluent interfaces for guard clauses [part 3]

A fluent interface helps simplify the expression of multiple guard checks, but doesn’t avoid the trapdoor effect in which the first guard fail causes a break – because none of the guard state is propagated from check to check.

Aggregating state

To avoid the trapdoor problem and aggregate the guard result state as it’s passed down the chain, then break if needed in some chain terminator.

The state object could just contain a list of exceptions to add to:

public class GuardState
{
    private _exceptions = new List(1);
    public List Exceptions
    {
        get { return _exceptions; }
    }

    public object Value { get; set; }
    public string Name { get; set; }
}

Typical guards would then just add an exception to the list and pass it on:

public static GuardState IsNotNull(
    this GuardState state)
{
    if(state.Value == null)
    {
        state.Exceptions
            .Add(new ArgumentNullException(state.Name));
    }
    return state;
}

Finally, we need to terminate the guard chain and take action:

public static void Check(this GuardState state)
{
    if(state.Exceptions.Count > 0)
    {
    	var ex = new GuardStateException(state.Exceptions);
	throw ex;
    }
}

This means you can now gather all useful state in one go before throwing, which then means you have a chance of solving multiple problems before releasing a fix. This is a rough draft – for example you might want to only instantiate the exception list on the first fail to limit memory usage – but it’s usable.

If it’s used in a high throughput environment memory pressure might be important, even though the guard state objects wouldn’t make it out of gen0. Rick Brewster came up with a syntax that sacrifices a little clarity to tweak the memory use of his guard interface – which I’ll talk about briefly in the next post.

Comparing fluent interfaces for guard clauses [part 2]

So Microsoft provide a Guard class that simplifies guard checks. It has a simple interface, but is a bit restricted and can look a bit clunky if you need to do multiple checks -- assuming you've extended the Guard class:

Guard.ArgumentNotNull(message, "message");
Guard.Contains("XYZ", message, "message");
// ...

We can simplify these chains a lot by using a fluent interface.

Fluently chaining guards

A fluent interface can be used to chain checks. It’d look something like this:

Guard.Argument(message, "message")
    .IsNotNull()
    .SomeOtherCondition();
Guard.Argument(count, "count")
    .IsPositive();

The guard checks are extension methods hanging off the underlying state object that’s passed from check to check. The state object is populated by Argument(…), and holds the argument value and name:

public class GuardState
{
    public object Value { get; set; }
    public string Name { get; set; }
}

and an example guard check looks like:

public static void IsNotNull(this GuardState state)
{
    if(state.Value == null) 
    { 
        throw new ArgumentNullException(state.Name);
    }
}

So with this technique I could simplify chained guard checks, but the code would break on the first guard fail. CuttingEdge.Conditions is a freely-available library that does much the same thing, in, I’d guess, the same way.

In some sense there’s not much difference between using this, and just using repeated calls to specific IsNotNull(value, “name”)  methods. To make it more useful, we’d need to start aggregating state and avoiding the guard trapdoor effect.

Comparing fluent interfaces for guard clauses [part 1]

Urgh, not again:

public void SomeMethod(string message)
{
    if (message == null)
    {
        throw new ArgumentNullException("message");
    }
}

Guard clauses are clunky. They’re also like a series of trap doors – you break on the first guard, patch and release only to find that you break on the next.

For the clunkiness, we could Extract Method, making things clearer:

public void SomeMethod(string message)
{
    IsNotNull(message, "message");
}

public void IsNotNull(string argumentValue, string argumentName)
{
    if (message == null)
    {
        throw new ArgumentNullException(argumentName);
    }
}

Eventually we might collect a few and gather them under a static class, as they're not state-dependent:

Guard.IsNotNull(message, "message");

And helpfully, there is a lightweight simplifying Guard class in the Microsoft.Practices.Unity app block: the Guard static class provides a series of methods for basic guard checks.

Guard.ArgumentNotNull(message, “message”);

The Guard class only provides a limited range of checks – although it’s partial so could be extended. The interface is pretty simple though, and provides a good starting point.

However, if you need to make a series of guard checks the interface gets cluttered with a series of Guard references and reused arguments. That can be made a bit clearer with a simple fluent interface.

A simple paged IList, arrays of doubles and the LOH, part 3

In these posts I looked at creating a paged IList after finding that large arrays of doubles are stored on the LOH rather than the normal heap, and gave a simple first-cut implementation. Here I thought I’d compare the add-item behaviour with a comparable List<double>.

There’s a lot of data here but there’s a couple of things I want to draw out.

Where the page size is large compared to the array size, performance is comparable to List<double>. Conversely, small page sizes are really not suited to large arrays, even though the PagedList has an advantage in that it doesn’t need to recopy all the data, as a List would.

I want is for this chart to be flat, with similar performance for all PageList total sizes. I want to knock out that big peak to the left hand side and make it perform more consistently.

One way to handle this is to follow List’s lead an allow the amount of capacity added to vary, doubling in size every time it fills, achieving additions in constant amortized time. PagedList can do the same by doubling the pageSize every time the last page is filled.

The fundamental change comes in the AddPage method, where I double the size of _pageSize after adding each new page.

private void AddPage()
{
    var t = new T[_pageSize];
    _underlyingList.Add(t);
    _currentPage++;
    _currentPageHighwatermark = 0;
    CurrentPage = t;
    _pageSize *= 2;
}

A new constructor lets the PagedList be instantiated with a default page size of 4.

private const int DefaultPageSize = 4;
private int _pageSize = DefaultPageSize;

public PagedList(int initialPageSize)
{
    _pageSize = initialPageSize;
    _underlyingList = list;
    _underlyingList.Add(new T[_pageSize]);     
    _currentPage = _underlyingList.Count - 1;  
    CurrentPage = _underlyingList[_currentPage];
}

Various consequences then ripple through the code, leading to changes in Count() and the indexer this[], which needs to map a global index to a page location and page index. I’ve left this as a pretty simple implementation for now:

private PageLocation PageLocationForIndex(int index)
{
    var l = new PageLocation();
    l.desiredPage = 0;

    int lower = 0;
    int upper = 0;
    foreach(var page in _underlyingList)
    {
        upper = lower + page.Length;
        if (index >= lower && index < upper)
        {
            break;
        }
        l.desiredPage++;
        lower = upper;
    }

    l.desiredPageIndex = lower == 0 ? index : index % lower;

    return l;
}

where PageLocation is simple struct holding ints desiredPage and desiredPageIndex.

Looking at the performance then:

That’s significantly better. Even in debug the peak is knocked down from ~10 to ~2 times, and in a release build it’s more comparable to List<double>.

However, I’m a little concerned that the relative times are increasing with array size for the PagedList<double>, which might indicate some O(n) or worse behaviour that needs looking at.

Also, I’m no longer meeting my main requirement; the PagedList<double> will, at some stage, store a double array larger than 1000 elements. I could fix that by capping – but I’ll look at that in a later post.

A simple paged IList, arrays of doubles and the LOH part 2

So, first of all it might be interesting to check that large double arrays are put on the LOH, in case the internet lies. Bring on CLR Profiler to check allocations.

Tangentially: >=1000? Not >1024?

So, if I allocate a number of double arrays of size = 100 everything allocated on the normal heap (in these CLR Profiler diagrams, System.Double[] allocations are in red):

… and with array size = 1000 everything is allocated on the Large Object Heap.

Finally, I can show that PagedList<double>s of size 1000 but a page size of 100 use space on the normal heap, not on the LOH:

Not sure about all those System.Double[][] – that might be something to look into; but with the allocations briefly confirmed, I want to do some very brief performance comparisons in the next post.

Mocks and dependencies

On rationalizing class dependencies -- John Sonmez is exploring a number of these ideas at the moment, and is building a useful list of patterns for dealing with common scenarios.