There are many reasons to create a workflow system. It could be that you have users that want custom flows that they can manage. It could be that you want to create a processing pipeline for data. Or it may be as simple as wanting to write code that is more encapsulated and easier to manage, skipping some of the boilerplate needed in common applications.

All of these are valid use-cases (and there are many, many more). The reality is that everything in life works around "activities." We all deal with processes and procedures at work, at home, and even in our thoughts. Each step, or activity, in these processes, may work the same every time you activate them. So why rewrite the code, or do them manually, when you can create a workflow engine to do it for you.

Now, you cannot just apply this to everything in life (obviously). But you can take complex topics in code and translate them into a workflow engine. The benefit is we can avoid many nested, or configurable, if-then statements in our code. Can you imagine writing a dynamic workflow that requires each step to be checked directly in our core code, where there are nested if statements 5 deep? This would be a nightmare (of course, most developers wouldn't do this, to begin with).

But go back and look at code, or services, you've created in maybe a Web API. Could it have been implemented with a workflow of some kind? Does it have pretty complex processing? Are you using parts of that pipeline in multiple places, but it is hardcoded in a way that you'll have to change it in multiple spots?

Workflows also tend to be more readable, long-term. The underlying system itself is usually complex, but utilizing the frameworks are easy. This is where we save our time.

There are a number of great workflow systems already out there. My goal isn't to make these obsolete, or suggest, using this approach below over these frameworks. You should investigate these and determine which one fits your needs best. Each approach the idea of workflow very differently and have unique advantages and disadvantages

  • Workflow Core: Light weight embeddable workflow engine targeting .NET Standard
  • Elsa Workflows: Library that enables workflow execution in any .NET Core application and can be defined in code or using a visual designer.
  • Automatonymous: A state machine library for .NET that provides fluent syntax for events, states and activities

There are many more than this. These are just ones I've worked with and enjoyed. It'll be your choice to decide if these are overkill for your use-case or have the features you need. But if not, or in doubt, build your own that meets your needs!

Let's walk through it.

Note: I made mention of workflows and provided a very similar example in C# Simple Interfaces: Containers. This article will be more thorough in explanation and implementations.

Activities

For any workflow, activities are the key. We want to perform a task, or set of tasks, depending on certain conditions. The complexity isn't necessarily in the activity themselves, but when conditions come into play and how we want developers to interface with the engine's contract.

Let's start by defining some details about activities themselves,

This set of interfaces provides us a very simple framework for activities and will be the core of our workflow system (although it will change later in the article). In fact, if we want, we can add some extensions and then stop, if this is good enough for your needs.

We're only adding a single extension type of Then which will simply run 1 activity after another.

Processing the Activities

Now that we have our activities, we need to process them. There are a lot of ways to do this, but we'll just create another interface and an implementation for the system.

And then an implementation,

This service will basically go through each activity in a collection and run them back-to-back in a loop, until completion, or until canceled by the activity itself. We will either create an activity based on type or run the provided instance directly. We are not performing any optimizations here, like caching activity creation, but this can be easily added.

Note: Custom activities can have services injected into them. This is why I use ActivatorUtilities. It provides this service for us. However, it also means that the activity must be registered with DI.

Let's not forget we need a basic implementation of IActivityCollection.

It doesn't have to be much more complex than this, although you may want to customize it a bit. Up to you really.

The last point to make is how we use this simple engine. Let's assume that you have the following defined activities:

  • ProcessDatabaseRecordsActivity: Goes to the database and processes any available records.
  • DataWarehouseActivity: Takes current data in the database and pushes to a data warehouse.
  • EmailActivity: Emails you at the end of the process that it was completed successfully.

You can customize these activities through constructor injection, but no data is being passed between them. We'll address this later.

Let's show a simple usage example,

And that's it! We have created a basic workflow engine that will actually work. However, I would be the first to admit this isn't EXACTLY a flexible implementation, yet. There are some issues with this approach:

  1. Data cannot be "shared" between steps. What if we don't want to have a central source, like a database, that coordinates this information?
  2. We have to create a custom activity for even the simplest of tasks. Can we find a way to use lambdas for very basic steps, like altering a property in the data?
  3. We have no conditional logic of any kind. How can we implement if-then logic based on data in our workflow?

Let's solve some of these issues.

Passing Data and The Context Object

I've seen three primary approaches to sharing data between steps in a workflow process:

  1. Shared Context Object: A single object is created and any mutations to the object is seen by all steps.
  2. Paired Input / Output Matching: Each step takes a specific input object type and outputs a specific return type. This approach requires steps to match input to output otherwise an impedance mismatch occurs.
  3. Global Step-Based Data: As each step is completed, it can set output data that can then be accessed by subsequent steps by referencing a prior steps output property. Most useful in UI-based interfaces where steps are well known by the design.

Like anything else, there are trade-offs. Shared Context Object is the easiest to implement in a strongly typed language than the others but debugging can be tricky when trying to determine who is mutating the object when issues arise. Paired Input / Output Matching is appropriate for ETL applications where data is being transformed and you want to control step building (Think functional programming). It is much more limiting in terms of available use-cases and largely can be resistant to change. Global Step-Based Data is great in UI-based interfaces where the user wants to directly access data, knowing their process, but is less useful for dynamic workflows, or code-based workflows.

I am going to take the Shared Context Object approach due to its simplicity and varied use cases. I believe it will provide the most value in the majority of developer use-cases.

Changes to the Structure and Contracts

We'll need to make some adjustments to our activity definitions to make this work. Let's show the changed interfaces and structures with added context.

Then the altered extensions,

We'll need another implementation for the collection itself,

And without going too much further, a sample usage,

Now we can provide a specific context object to our activities, as long as the object implements IActivityContext. We can customize it, add functions. Anything. However, each set of steps can only have one context object as it runs (another limitation).

I want to pause here and talk about an issue with this. Before, we didn't have to specify the Context in the extension methods. Instead, they were pretty simple and straight forward. However, extension methods fall short here. It's not really the extension methods themselves, but the nature of method generics.

Without getting too deep in the weeds, when specifying generics, you either need to specify ALL the generic parameters, or none of them (when they can be implied). There isn't any in-between, or partial support. So, in the case where we want to specify the type, we run into a bit of a problem. I really don't like this from a usability standpoint.

How can we make this better?

Collection Builder

We're going to have to create a specific builder interface instead.

Ignoring the implementation of this interface for a minute, let's look at its usage now,

Now, this feels more right to me. This also means that the extension methods we created for Then previously are no longer needed. Instead, we're going to have to use the builder to get the syntactic sugar we desire. Which means, that we'll also have to use it for any future extensions as well. So instead of extensions on IActivityCollection most will be on IActivityCollectionBuilder here on out.

Changes to the Runner

We still don't have a runner for this new structure, with the context object. So, let's create that for completeness.

First the interface needs to change,

Then our runner implementation,

Not much has changed, except that we now have to pass in a context object to run the activities. Maybe you want to pass in a creator lambda? Or maybe your context object is a DTO and you want it to be created automatically for you. We can create a couple of extension methods to make this a bit easier.

Again, these are quality-of-life extensions. They are not necessary.

For completeness, I'm going to also provide a base ActivityContext that can be inherited. For now, it will only make it so you don't have to reimplement the Cancel property. But in your case, there may be more than just a single property.

Simple Tasks

Now that we have a core framework down for our workflow engine, maybe we should make some quality-of-life adjustments for simple tasks. For example, after a step, you might want to mutate your context based on some simple logic, or even just set a property. Let's create a couple of activity implementations, and associated extension methods, that will help us out here.

First the activities,

All these activities provide is a holder for the lambda that will be executed at runtime. There are cases, where you will have some synchronous code, so we added both an async version and a void version.

To ease their usage, we create associated extension methods,

The usage of these methods is pretty straightforward,

There will be times when the custom logic will make sense to be defined in this flow. However, anything bigger than a few lines will make this less readable. So be cautious on when you use it. Either way, it does help prevent the need to create a large number of custom activities for simple tasks.

Conditional Logic, Loops, and Exception Handling

How about conditions in the workflow (sometimes known as decision points)? How about a scenario where we need to run a process multiple times in a row? Or how do we handle exceptions?

Although, these are all unique situations, we're going to use a similar technique for all of them, where we create an aggregate activity and use a sub-builder to extend them.

Default Builder

At this point, we haven't really defined our builder. Before it was just assumed. However, we need to define it now.

The builder defines a private class for the collection. There are cases where this isn't desired, but we allow this to ensure the implementation is what we want. However, there are cases, where custom collection types might be desirable. In that case, there would be a need for an IActivityCollectionFactory of some sort.

Everything else, should be self-explanatory. We add the appropriate methods for Then. Then we return the collection during build. I make sure that we replace the backing field, so that the builder cannot alter it anymore after this point.

Normally, I would make this builder an internal class. This protects its usage and also allows me to expose only the pieces required for the developer to properly use it. Here, it is public for ease of use in our examples. But always try to keep classes internal until they need to be public. EN-CAP-SU-LATION.

We're going to run into a little problem…

We're about to hit a road bump. And unfortunately, it will require some refactoring. See, we're going to want to use IActivityCollection as the input for our activities. However, it's a collection of ActivityDescriptor. This means that the creation of activities, and running them, is going to cause us some pain, especially at runtime. We MAY have access to an IActivity to run, but maybe not. We might have a Type that we have to create. To create these types, we need IServiceProvider.

It's getting complicated.

So maybe, let's create an activity that will do this creation instead, allowing us to only have an IActivityCollection that inherits IList<IActivity>.

Sorry, if you feel like we're going back and forth on the implementation. This is the process of refactoring. I'm hoping it provides some level understanding as to my process.

First let's change the collection interface,

We simply change from ActivityDescriptor<TContext> to IActivity<TContext>. Not any more complicated than that.

We have to alter our builder to have the provider injected. We'll also need to change our collection implementation here (Glad we put it here. Hid the changes from the user, almost).

The builder interface changes,

We added a StartNew method that will act as a factory for sub-systems that want to start over with a clean collection.

Probably good to show the ServiceProviderActivity itself,

Our runner will also need to be changed,

We were able to simplify the class a bit and remove the GetActivity portion.

Now when it comes to usage,

The biggest change here is the use of the DefaultActivityCollectionBuilder and passing in the IServiceProvider. At this point, we can move forward with conditionals.

Condition Logic

We're going to create a very simple If-Then-Else conditional logic activity. We will optionally be able to provide just the If-Then logic or the If-Then-Else in the same activity.

What are we doing? We're passing in two sets of activities and a condition. The condition is what will determine which collection to run. The collections are then run appropriately based on said condition.

Where are those quality-of-life improvements, you say? Those are next!

We're using the Then methods to create the conditional activities. The question becomes what this looks like to the programmer now? Let's look at usage,

After we have processed the data, we will check if it was successful with a flag that is set. If we are successful, then we will run a success email activity, otherwise, a failure.

Loops

I won't go through all possible implementations of various loops, but I will show one, a Repeat Loop.

The goal with the repeat activity is to run the same set of activities multiple times either with a static count or based on a property on the context.

Simple enough. We can pass in a function that will be calculated based on context, or we can pass in the count directly. Let's create those extensions,

And finally, it's usage,

I would encourage you to add whatever other extensions that may be useful. Maybe you want to just pass in a single activity. For instance, you can have an extension that takes count and IActivity<TContext> instead.

Exception Handling

The last common issue that can arise is the handling of exceptions. You can do this in the activities yourself. But what if you want to alter course based on a specific exception? Similar to the last two approaches, we'll create a special activity for this.

Let's show the activity first,

I kept this somewhat simple. You pass in a set of activities for the try clause and optionally the finally clause. For the catch clause we use a list of Type and activity collections. When catching the exception, we want to use similar logic as the true try-catch. So, we find the first exception in the collection that is assignable. If none, then we re-throw.

Note, best practice would say that we should have an interface for the CatchActivityCollection. I left this out for brevity.

Let's look at the extensions,

And finally usage,

I will admit, from a readability standpoint, this could be better. I think we could find some ways to improve it, but hopefully you get the idea.

Conclusion and Sign-Off

Well, that sure was a lot of code. We created a simple workflow engine that can be used for endless use-cases. However, it does have some downsides,

  1. Saving / Loading: It would be difficult to save / load this structure dynamically from a file or database. Everything is created as instances and not descriptors.
  2. Configuration Limitations: Configuration is limited to what can be injected into the constructor of activities. Maybe this is ok, but dynamically changing configuration could be a useful feature.
  3. Lack of State Machine: There isn't really a state machine being used here. You could theoretically run a state machine with the context object, but what if you don't want everything to run at once? What if you want to continue a workflow later?

This is by no means a perfect workflow system. This version can be used in many ways, but not in every situation. At the very least I hope you learned some different techniques that you can implement when designing your next system.

If you have any thoughts or suggestions, please comment below. And be sure to follow me on medium, as I release new topics all the time!

Until Next Time!