I come from a strange background with AI. My first program was a chess computer, and I’ve been in/around the space longer than I have been in industry. In that time, I’ve seen technologies that were once cutting edge AI become commoditised, turn into brands or products, and cease to be considered AI (Google Maps navigation, Google Translate, Grammarly).

My early experience in industry was skewed by the tail end of the last AI Winter, and was an era where people had to hide their usage of AI in order to secure funding for projects.

I’m familiar with the cycle of excitement and overinvestment, followed by the pullback of funding and I see the current time as just another phase in the AI history cycle. In the words of AC/DC, Money talks, everything else walks - and once investors say they don’t want AI, then that will be the driver. For now - Money says AI, and that’s what we’re going to do.

At some point “We use LLMs in our pipeline” will be the same as declaring that you use containers, Postgres or have CI/CD. The thing that matters will be what you do with it, and whether it is stable and secure.

Until that time comes - we as engineers are all figuring out our place in the world, and what our relationship with our tools is going to look like.

I have shaped an AI policy that reflects my values and should be workable in the real world, published here.

How I use AI

• A simple Agents.md for common rules works well for most code bases - avoid bloat.
• Keep the skills files to a minimum - Only add core operations.
• Keep the context small and focused on task - My top two commands are /new and esc.
• Steering is key - watch it run, provide guidance into the dialogue stream.
• A code base should be navigable by humans. As a side effect, this will also be navigable by bots. We still need to pass audit - investors will want to see what is being done.
• Code tags are great for the bot to grep/discover. These are signposts in your code base. E.g. //BOT: Extend here for additional jobs.
• Sometimes I still write code by hand!! Refactor a bit, make a commit, run “Apply last commit to all classes implementing IDocumentFactory.” poof magic.

Test and Verify

I have always driven a strong testing culture. Most of my engineering effort has been focused on defining what correct looks like, and then proving that it does actually work that way.

From my experience, when we tackle our large engineering problems, code and code generation have never been the challenge. 90% of the engineering effort has always gone into proving that you have delivered what you have said you have delivered. No amount of automation makes that part go away, so how fast we can move with automation is limited by how fast we can verify it is correct.

In an age of AI, rapid development and equally rapid change - having robust test harnesses, test suites and observability is even more critical in enabling us to move safely at speed.

Why keep blogging?

Sure - I could make a content factory loop, schedule a job running a bot, it’s not a complicated process:

• Schedule a job
• Generate 5 subject ideas based off current tech trends for a persona (Eccentric bald lead engineer who likes hats)
• Pick the best idea and generate an extended script of the subject following the persona.
• Generate an audio track of the script from a sample voice pack.
• Generate a video from the script/audio using an image asset pack.
• Publish to YouTube once a week.

Content is easy, it’s become noise. Now noise is easy, signal is hard. So mostly I write for me - as a journal of things I have built, a record of things I explored and learned, and ways of approaching problems that I find useful or interesting.

I will speak with my own voice and aim to be a home of something genuine, and maybe a signal in the sea of noise.

I’m keeping my old blog, and the old format, although I’ll work on modernising the stack, as it provides history and content from the “before times” and a bridge into the modern AI era.

Credits

I have always credited the photographer / artist behind images I used on my blog - but in the 4 years since my last post, finding anything made by a human has become… challenging. Header photo by Ryutaro Tsukata

Uptime

Here at Checkout.com, we process payments — and as the Wolf of Wall Street taught us, money never sleeps. It’s always flowing, 24 hours a day, 365.25 days a year.

As the engineering department for a company that processes payments, we cannot remain locked in stasis while the system runs. We have to be able to replace components and subsystems without interrupting the flow of traffic passing through the system.

Blue/Green deployments allow us to hot-swap systems already in use. Here, I cover the evolution of a simple server setup to a modern managed cluster and how Blue/Green deployments fit into the various systems configurations.

In the beginning

Let’s start with this basic setup. It’s a classic. The greybeards call it: Client/Server.

This… isn’t good enough to run a critical business. We can’t build a resilient, scalable system using this pattern. We’re limited to a single node that could fail, and a good rule of thumb is we should always run a minimum of two physical systems. With all the modern cloud systems we deploy to, it’s also pretty much impossible to avoid deploying to a cluster/load balancer setup anyway. Let’s represent that:

Great! Our app got deployed to a cluster, and now it’s being used. We’re running a successful business, money is rolling in, and everything is good. Now the business needs a new feature. We’ve developed it, tested it (it works on my machine!), and it’s all ready to go… let’s roll it out!

(I’ve used Octopus in our examples as it’s what we use; other deployment tools are available).

We’ve gone to our deployment system and pressed “deploy”. It’s pushed our package/container to the cluster, replacing our old app with new instances. And now we’ve got a couple of problems — at least in our completely contrived scenario:

• As we’re replacing our apps, some inbound connections will find the server they are trying to reach is in the process of an update, and we return some kind of failure, such as a 503 response.
• In the old app, some apps were not gracefully shut down. We have open transactions and locks on resources that were not released, so the new app may be locked out until these expire.
• In our new deployment, after we start it up, it doesn’t have the exchange rates (FX) in memory to process requests, and the API that provides them is not available. We can’t begin servicing requests.
• While we’re replacing the multiple app nodes, we also have no control or guarantee whether we are processing a given request with V1 or V2 of the system.
• At this point, our alerting system is screaming at us, and there’s nothing we can do about it. If we put the old system back, it will have the same problem with the FX API on start, so we’re still burning a hole in our service level agreement.

How do we avoid affecting our primary system while we deploy?

First, we need to have something to switch to — a secondary version of the app running in isolation. What does this system look like?

When we talk about Blue/Green deployment, we’re really talking about a pair of systems that can hand over to each other. How do we build a setup so we can handle hot-swapping our two systems? First, we need to bring both the original Green and the new Blue system under a single entry point, which will manage traffic between the two app clusters.

Now we have two separate systems behind a single entry point, which is invisible to the client. So how does our deployment system handle pushing to this new configuration, and how do we get our hot swap?

Magic!

Can we do anything to improve this? Currently, our swap is immediate, and we may still find a bug in our new app when we deploy to an environment and start running it at maximum load. Can we add any more confidence to our deployment?

Indeed we can. In the current setup, we instruct the traffic manager that acts as routing between our two systems to send all new connections to our secondary app at once — we can make this smoother.

To do so, we can gradually transfer traffic to the new service, keeping an eye on our logging system (such as DataDog) and confirming at each level of load that we’re not getting any errors until all traffic has been migrated.

This process trades off deployment speed for a slower, more risk-averse approach. In critical areas of the business, this tradeoff makes sense as we can deploy the new version of our app with greater confidence.

Simplifying the System

As it stands, our system is composed of two clusters with their own load balancers, plus an additional traffic manager to handle routing between the two. Network hops are one of the more expensive considerations in a distributed system: how can we reduce these?

As all three components are responsible for handling network traffic, we could fold all the responsibilities for these together and make it run in a single hybrid cluster. If we do, we have a single controller for the cluster that handles both DNS routing and traffic management. If you’ve worked with Azure, the below setup used to support slot-based deployments will look familiar:

More magic!

Modern Cluster

This brings us to where we are today with modern cluster management techniques used for ECS, function apps, lambdas, and automatic scale sets. They can support hybrid app hosting and slot-based deployments. Now, we live in a magical wonderland of containers and shared infrastructure that just scales up and down on-demand. We only pay for what we use a̶n̶d̶ ̶w̶e̶ ̶n̶e̶v̶e̶r̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶w̶o̶r̶r̶y̶ ̶a̶b̶o̶u̶t̶ ̶c̶o̶-̶h̶o̶s̶t̶e̶d̶ ̶s̶e̶r̶v̶i̶c̶e̶s̶ ̶e̶a̶t̶i̶n̶g̶ ̶a̶l̶l̶ ̶o̶f̶ ̶o̶u̶r̶ ̶s̶h̶a̶r̶e̶d̶ ̶r̶e̶s̶o̶u̶r̶c̶e̶s̶ ̶a̶n̶d̶ ̶D̶O̶S̶i̶n̶g̶ ̶u̶s̶…

Alternative Implementations

We can take Blue/Green deployment principles and apply them in a couple of other ways.

Canary deployment

Rather than deploying a complete cluster/node set and swapping, we can deploy to a single active node, observe how it runs under incremental load and replace the remaining nodes one at a time.

Canary Consumers

The same principle can be applied to queue consumers. A single throttled queue consumer (thread limits and delays) can be deployed and then monitored, with a policy in place to ramp up the consumer and then deploy the remainder.

Index/Database swaps

We can also use this approach to publish a new version of a data store/index. Create a new read-only database/index under an alias and change the default alias used by the read cluster — for example, changing the “main” database role to point to a new DB/Elasticsearch index.

The queries continue without interruption, but the new store is now returning results.

Limitations / Considerations

One of the key principles in this approach is backward compatibility. It is critical we do not deploy breaking changes to contracts or storage schemas/models, as this will cause errors when used by different versions. This is why when adding a new column or field, it’s important to ensure it is optional and, when removing one, to ensure that it’s not validated or a required field for serialization.

This is doubly important for data that has a temporal element and may live for a longer period of time outside a service — for example, scheduled messages — as it’s entirely possible we could have progressed multiple versions of an API or system ahead of where the message originated.

One strategy for managing this is to use versioned contracts and feature toggles where appropriate.

Wrap up

These techniques are part of the toolkit for building resilient systems and remain responsive through the development/deployment cycle. It’d be great to see more of these in production, particularly the local council waste collection site, which is usually down when I need to check the bin days…

Credits

Header photo by Tima Miroshnichenko from Pexels

Diagrams.net is a great tool for diagramming. I find it vastly superior to Miro, exclusively down to its soothing pastel color palette and selection of line end arrows. Aesthetics aside, taking screen shots, clipping them, sorting them, and updating your post, and once it’s live spotting a spelling mistake…. only to have to do it all over again… it just leaves my crying into my tea.

There must be an easier way!

Can we embed diagrams directly in blog posts?

via GIPHY

Hmmmm…

Yes we can!

With diagrams.net’s handy viewer tool, we can host our diagram and export an IFrame wrapper around the viewer tool to get a nifty embedded diagram.

Dump the diagrams somewhere publicly accessible, I’m using a Diagrams repo

Open your diagram and go to File > Embed > HTML (or an IFrame, but I found you have more control with the HTML side).

And dump the link / html into your HTML / markdown.

via GIPHY

Caveat

I still haven’t figured out how to style them properly :)

Credit

Image by Clker-Free-Vector-Images from Pixabay

By now, Command and Event based systems in software are ubiquitous and well understood. We can see them in WPF / React with UI commands and state change events, and we see it in distributed systems with Command and Event queues and the publish / subscribe model.

These systems have two types of messages, and are composed of three main concepts.

State

The system state as it currently stands can be stateful and in memory, such as a React state / WPF model, or stateless and out of process such as a database or document store tied to a set of function apps / lambdas.

Commands

Commands are an instruction to modify state: they have not yet happened, but are an expression of intent to make something happen in the future. Commands have one recipient and they can fail, but when they succeed they have side effects.

Events

Events are indications that the state has changed. The state has been updated and the event provides a description and timestamp for the thing that occurred in the past. They can be broadcast to multiple recipients / subscribers.

Putting it together

In this diagram, flowing from left to right, you have commands on the left, that are modifying the current state in the middle, which are raising events on the right.

Making it real

While this is good, it’s all still a bit abstract. Let’s make it real - who likes food? We’re all on the same page? Great! So let’s go with a restaurant kitchen. We’re going to map the above concepts to our kitchen:

• State: the kitchen’s current ingredients, available chefs (threads) and in-progress dishes.
• Commands: the orders coming in from the waiters. You’ll see these queuing up on the shelf.
• Events: the dishes output by the kitchen. The Ack part is slapping the order ticket on the completed orders spike when the dish is picked up.

Bonus: Our kitchen analogy comes with built in queues; I don’t even need to explain them.

What else is there?

We’ve got two messages here: Commands / Orders that express intent / hunger, and Events / Dishes that express a side effect / satisfied customer. What else is there in our Kitchen system?

Ingredients

These aren’t commands and they don’t do anything on their own - a tomato is not a command, and neither is a steak. We’ll receive them and use them when we execute an order/command and may issue low stock alerts when they run low, but the ingredients themselves are inert.

When we receive ingredients, we have a couple of options:

• We could prepare them and freeze them
• We could put them in the fridge raw
• We could prepare them in a sauce as an ingredient for another dish
• We could prepare a dish to be served immediately.

Whatever choice we make we’ll do based on our own context, not because of any specific instructions attached to our raw ingredient.

Recipies

Recipes look kind of like commands at first glance, but the key difference is that we’re not running them now - they’re a pre-defined set of steps we want to store as a template, and execute when we run a command.

We may unpack those steps into individual commands to be run inside the context of the kitchen, and we may skip some steps depending on the context of the order (I’m not fond of mushrooms, so we’ll leave them out), but as far as front of house is concerned these are invisible.

Behind the scenes

This is what’s going on behind the scenes of our KitchenAPI:

Wait, what’s that?

Our orders and dishes are all we want expose to our front of house - or in other words, our commands and events are all we want to surface from our KitchenAPI.

But there are clearly other things happening behind the scenes, because guacamole doesn’t appear out of thin air. These aren’t commands/orders, as we are not representing intent. Nor are they events/dishes, as we are not indicating something has happened. What we receive as the payload is the thing that is happening, and that thing can have side effects in its own right.

What are they in a software context, these concepts that aren’t commands or events?

Just a message

They’re just a plain message, a document, an inert packet of information or data that carries no context of its own. What the recipient does with the payload is up to them. A couple of real world examples:

• Data points: instead of sensor information as a sequence of events, they can be viewed as a series of sampled absolute values.
• Frames: a video frame is not an event, but it is part of a feed that provides you with hours of entertainment.
• Documents: the complete works of Shakespeare are not an event. Unless someone performs it, it’s just a payload of data.
• Document fragment: if a document is large and needs to be broken down, the parts of that document that are transmitted are not events.
• Workflow: a self-contained workflow can be a payload in its own right, and when processed may generate commands and events as part of its transformation while not be either itself.

Shoulders of giants

And of course, nothing new here.

Bottom line

We’re all building first-class command and event-based systems. When thinking about our command sources and event stores, and while running our user action and event storming workshops, it’s always worth bearing in mind that it’s still valid to pass messages between systems that are neither commands nor events - and it’s worth having room to express these concepts in your system designs.

Finally

I’m hungry, so I’m going to run

kitchen exec prepare cheeseburger-taco

Aaaand….

(No, there is no clean way to eat these)

Credits

Art work - All stick man artwork is masterfully created by me in Draw.io… probably the wrong tool for the job.

Main photo by Rodion Kutsaiev from Pexels

Over my last few posts I’ve covered pattern matching with regex, how to break down a sentence into recognized parts using a lexer/tokeniser, creating the basic building blocks of a combinatorial parser from an OO perspective and migrated it all to a purely functional approach.

We used a two phase process, firstly a Lexer/Tokeniser, where we turned a string into higher order tokens such as Day, ClockTime and Reserved Words, and a combinatorial expression Parser to process the structure of phrase:

And we applied this process to the following phrases to generate an object:

// Separate two part element context => DayTime range
"Pickup Mon 08:00 dropoff wed 17:00"

// Range elements with different separators => Open Days range and Hours Range
"Open Mon to Fri 08:00 - 18:00"

// Repeating tokens => List of tour times
"Tours 10:00 12:00 14:00 17:00 20:00"

// Repeating complex elements => List of event day times
"Events Tuesday 18:00 Wednesday 15:00 Friday 12:00"

Why two phase

In some of the projects I worked on the goal with the separate tokenisation phase was to keep the language constructs separate from the syntax rules. This would enable us to hot-swap the language pack without touching the syntax rules, and on the flip side, convert any compiled syntax rules into a local language.

I put together a similar example implementation here based off a modified language aware tokeniser.

Additionally, it’s entirely possible that you may be Tokenising from a different source, maybe a number of images are being processed as words, or you are processing an audio stream and tokenising that before you analyze it for expression.

What else can a token be?

The first phase of our two phase process is to turn our strings into an array of objects / symbols that represent complete concepts, but what constitutes a complete concept is flexible.

At the end of the day, we need an array of things to work with, and currently they are Token objects. A string is also an array of Char[], and if we consider Character, Digit, Symbol, and Whitespace as our tokens then we can use a string directly as the Token source and build all of our actual word logic via combinatorial parsers.

Scannerless Parsing

The single phase parsing process is known as Scannerless Parsing and there is already an excellent .Net implementation Sprache (The word is “Language” in German) which has been going for a good decade now.

Sprache

Sprache is a great library that has a full blown suite of combinatorial parsers and the ability to embed regex matches at any phase, consider some of the following we never even got near:

• DelimitedBy
• Repeat
• XOr
• Optional
• Once
• Not

There are a couple of other neat tricks they have, rather than returning a single object, it returns an array of 0 or 1 (and possibly in some scenarios more, I have not looked into concatenation), which means that it can be used directly with Linq and it can be written using the Linq Query syntax:

Parser<string> identifier =
    from leading in Parse.WhiteSpace.Many()
    from first in Parse.Letter.Once().Text()
    from rest in Parse.LetterOrDigit.Many().Text()
    from trailing in Parse.WhiteSpace.Many()
    select first + rest;

I’ve done a third implementation of the original expression parser suite in Sprache, with the parsers split between the Token Parsers and Expression Parsers

We can look at the different ways to implement DayTime expression parser side by side:

// Linq Delegates Syntax
public static Parser<DayTime> DayTimeDelegate =
    TokenParsers.DayOfWeek.Then(day =>
        Parse.WhiteSpace.Then(_ =>
            TokenParsers.LocalTime.Select(time =>
                new DayTime(day, time))));

// Linq Query Syntax                        
public static Parser<DayTime> DayTimeLinqQuery =
    from day in TokenParsers.DayOfWeek
    from _ in Parse.WhiteSpace
    from time in TokenParsers.LocalTime
    select new DayTime(day, time);

Whitespace

It’s worth noting that we have to pay more attention to white space in the patterns. Scannerless parsing does not pre-process and package tokens, so they have to be sifted out of the whitespace / punctuation.

The tokeniser we have been using so far has been very basic and not taken into account white space or phrases that contain white space. A feature complete implementation would be able to shave regex matches off a string and move a cursor forward the length of the match.

Up Next

I’m going to do something completely different…

Credit

I’d like to credit Sprache for being a source of inspiration, and helping me bridge a couple of the concept gaps when trying to build both equivalent OO and Functional version of the same combinatorial parsers.

Header image by Gerd Altmann from Pixabay

In my previous post, we created a basic set of object orientated parser implementations to handle converting some simple expressions.

// Separate two part element context => DayTime range
"Pickup Mon 08:00 dropoff wed 17:00"

// Range elements with different separators => Open Days range and Hours Range
"Open Mon to Fri 08:00 - 18:00"

// Repeating tokens => List of tour times
"Tours 10:00 12:00 14:00 17:00 20:00"

// Repeating complex elements => List of event day times
"Events Tuesday 18:00 Wednesday 15:00 Friday 12:00"

In Action

The System

The system is two phase, with an initial Lexer/Tokeniser that splits a string into identified parts, and a combinatorial parser system that matches a complex pattern and converts this into a higher order object. We’ve been following an Object Orientated style so far with interfaces and implementation classes.

// Converts words into typed concepts
public interface ITokenParser
{
    TokenisationResult Tokenise(string token);
}

// Converts the array of typed objects into higher order objects
public interface IParser<T>
{
    ParseResult<T> Parse(Position position);
}

public static class ParserExtensions
{
    // Extension method to enable us to enter into the IParser interface with just a Token array.
    public static ParseResult<T> Parse<T>(this Parser<T> parser, Token[] tokens) =>
        parser(Position.For(tokens));
}

Implementations

• Is
• Then
• Or
• Select
• ListOf
• End

Going Functional

Now we’re going to drop the interface implementations, make them all delegates and take an entirely functional approach.

public delegate TokenisationResult TokenParser(string token);
public delegate ParseResult<T> Parser<T>(Position position);

Disclaimer

While I use functional programming regularly, I don’t come from a functional background, so I may be butchering some terms. I’ve tried to be as accurate as possible, but the concepts feel kind of fuzzy and share many similar attributes with each other, so I may blur the boundaries a bit.

Main Players

There are a couple of concepts that we need words for in order to properly describe whats going on.

• Functors
• Monads
• Combinators / Combinatory Logic

Now, if you click on any of those links, you’ll be confronted with a lot of mathsy looking stuff that’s heavy on symbols and steeped in category theory. Monads also seem to have this logical fallacy around them that once you understand Monads, you cannot explain them to people who don’t. But I think I’m safe, because I still don’t understand them…

Functor

Wikipedia:

In functional programming, a functor is a design pattern inspired by the definition from category theory, that allows for a generic type to apply a function inside without changing the structure of the generic type.

Me:

A function that turns one thing into a another thing.

Combinator

Wikipedia:

A combinator is a higher-order function that uses only function application and earlier defined combinators to define a result from its arguments.

Me:

One or more functions used to generate a result inside another function.

Monads

Wikipedia:

In functional programming, a monad is a type that wraps another type and gives some form of quality to the underlying type. In addition to wrapping a type, monads define two functions: one to wrap a value in a monad, and another to compose together functions that output monads (these are known as monadic functions).

Me:

A function that you wrap in another function of the same type to decorate / extend its behavior.

Final Note: All monads are functors, but not all functors are monads.

Functional Flow

via GIPHY

Let’s start with two functions, baseValue() which returns 0, and add(), which is a function that runs baseValue() and returns the result + 1.

[Fact]
public void BasicNestedFunctionTest()
{
    // Function that returns a starting value of 0
    Func<int> baseValue = () => 0;

    // Function that returns a function that adds 1
    // to the result of the basevalue function.
    Func<int> add = () =>
        baseValue() + 1;

    // Run the add function, returning the result of basevalue() + 1
    add()
        .Should().Be(1);
}

When we run add, we get 1 added to the result of another function. Not too useful, but both the baseValue() and add() are Func<int>, meaning they are functions that take no parameters and return an int.

This means that, as they have the same function footprint, we can apply a combinator pattern and put one Func<int> inside another Func<int>, and what we have is still a single Func<int> that acts as a bridge between the other two.

[Fact]
public void NestedFunctionTest()
{
    Func<int> baseValue = () => 0;

    // A Function that takes a function and returns a function that adds 1 to 
    // the result.
    Func<Func<int>, Func<int>> add = (Func<int> baseValue) =>
        () => baseValue() + 1;

    // A function returning a baseValue() inside an Add(), inside an Add()
    Func<int> result = 
        add(
            add(
                baseValue));

    // Because we ran add() twice, we get a value of 2.
    result()
        .Should().Be(2);
}

I believe this is now a fully qualified Monad. We have a bridge that can nest and chain our functions and total the result, but we need to be able to do more than just add 1, we need a value parameter.

[Fact]
public void MonadNestingTest()
{
    Func<int> baseValue = () => 0;

    // A Function that takes value and a function and returns a function that 
    // adds the value to the result of the function.
    Func<int, Func<int>, Func<int>> add =
        (int value, Func<int> baseValue) =>
            () => baseValue() + value;

    // A function returning a baseValue, inside an add(1), inside an add(2), 
    // inside an add(3)
    Func<int> result =
        add(3,
            add(2,
                add(1,
                    baseValue)));

    result()
        .Should().Be(6);
}

Now let’s say we want to take our number and turn it into a monetary string, so what we want is a Func<string> that is going to be wrapping/converting a Func<int>, this is where we’re going to apply our combinator pattern against a functor. (I believe that while a Monad always chains functions, a Functor is just a converter of types, and those types can be functions.)

[Fact]
public void MonadNestingFunctorTest()
{
    Func<int> baseValue = () => 0;

    // A Function that takes a value and a function and returns a function that 
    // adds the value to the result of the function.
    Func<int, Func<int>, Func<int>> add =
        (int value, Func<int> baseValue) =>
            () => baseValue() + value;

    // Function that takes a Func<int> and returns a Func<string> that runs the Func<int>,
    // turns the result into a string and prefixes it with the &pound; symbol
    Func<Func<int>, Func<string>> asGBPString =
        (Func<int> baseValue) =>
            () => $"&pound;" + baseValue();

    // A function returning a baseValue, inside an add(1), inside an add(2), 
    // inside an add(3), inside an asString()
    Func<string> price =
        asGBPString(
            add(3,
                add(2,
                    add(1,
                        baseValue))));

    price()
        .Should().Be("&pound;6");
}

So now we’re composing nested additions with our values and converting the result into a formatted string, but the configuration is still fiddly and unintuitive. We can tidy this up with some fluent helpers.

public static class Beginning
{
    public static Func<int> With(int value) =>
        () => value;
}

public static class MonadExtensions
{
    public static Func<int> Add(this Func<int> baseValue, int value) =>
        () => baseValue() + value;

    public static Func<string> AsGBPString(this Func<int> baseValue) =>
        () => $"&pound;" + baseValue();
}

And our updated code makes our nesting come out as a flat chain:

[Fact]
public void MonadChainFluentTest()
{
    Func<string> price =
        Beginning
            .With(0)
            .Add(1)
            .Add(2)
            .Add(3)
            .AsGBPString();

    price()
        .Should().Be("&pound;6");
}

Much cleaner. So the equation here is

Functions + Fluent API = Magic

Wait a second

If this feels familiar to any C# developers, it should, because this pattern is the backbone of Linq and the Lambda based composable query system. Linq extends and chains IEnumerable, and in my last post we were chaining functions that returned IParsers and using extension methods on the interface. This time, we don’t have an interface, so we’re putting extension methods onto a Delegate.

via GIPHY

Putting it together

But what does this mean, and why is it useful to us?

Let’s go back to OO thinking and start with one of our original IParser implementations, IsToken:

public class IsToken<T> : IParser<T>
{
    public ParseResult<T> Parse(Position position)
    {
        return position.Current.Is<T>() ?
            ParseResult<T>.Successful(position.Next(), position.Current.As<T>()) :
            ParseResult<T>.Failure(position);
    }
}

What if we want to add a condition? We’ve got our OO hat on, so let’s create a base class!

public abstract class IsTokenConditionBase<T> : IParser<T>
{
    public ParseResult<T> Parse(Position position)
    {
        return position.Current.Is<T>() && check(position.Current.As<T>()) ?
            ParseResult<T>.Successful(position.Next(), position.Current.As<T>()) :
            ParseResult<T>.Failure(position);
    }

    public abstract bool check(T entity);
}

Now we’ve got a base class we can use to implement conditions, let’s create a check for ints over a certain value.

public class IsIntegerOver : IsTokenConditionBase<int>
{
    int _value;

    public IsIntegerOver(int value) =>
        _value = value;

    public override bool check(int entity) =>
        entity > _value;
}

By now, your spider senses should be tingling. This route is the route to great pain, we’re going to be creating a looooot of base classes and classes that use configurations of them and it feels like a trap. What about if we make it flexible by supplying our check as a delegate?

public class FlexibleIsTokenCondition<T> : IParser<T>
{
    Func<T, bool> _check;

    public FlexibleIsTokenCondition(Func<T, bool> check) =>
        _check = check;

    public ParseResult<T> Parse(Position position)
    {
        return position.Current.Is<T>() && _check(position.Current.As<T>()) ?
            ParseResult<T>.Successful(position.Next(), position.Current.As<T>()) :
            ParseResult<T>.Failure(position);
    }
}

We don’t need to make a new implementation each time, we’ve got something configurable. We can run that in a test, and it all works.

[Theory]
[InlineData("1", false)]
[InlineData("10", false)]
[InlineData("101", true)]
[InlineData("1000", true)]
public void FlexibleParser(string text, bool succeed)
{
    var tokens = Tokenise(text);
    new FlexibleIsTokenCondition<int>(i => i > 100)
        .Parse(tokens)
        .Success
        .Should().Be(succeed);
}

What we’ve just done here to make it more flexible is to put a function inside an object. Going back to earlier in this post, we talk about the Interface being replaceable with a Function, if we make that change now, we will instead be putting a function inside a function, and that brings functional composition of functors and monads into reach.

public delegate ParseResult<T> Parser<T>(Position position);

public static Parser<T> IsToken<T>() => (Position position) =>
{
    return position.Current.Is<T>() ? 
        ParseResult<T>.Successful(position.Next(), position.Current.As<T>()) :
        ParseResult<T>.Failure(position);
};

public static Parser<T> IsToken<T>(Func<T, bool> check) => (Position position) =>
{
    return position.Current.Is<T>() && _check(position.Current.As<T>()) ?
        ParseResult<T>.Successful(position.Next(), position.Current.As<T>()) :
        ParseResult<T>.Failure(position);
};

And now we can implement each complete parser in a declarative static property, no interfaces, no classes, no inheritance, no new().

[Theory]
[InlineData("1", false)]
[InlineData("10", false)]
[InlineData("101", true)]
[InlineData("1000", true)]
public void DelegateParser(string text, bool succeed)
{
    var tokens = Tokenise(text);
    Parsers
        .IsToken<int>(i => i > 100)
        .Parse(tokens)
        .Success
        .Should().Be(succeed);
}

Functional Parsers

I’ve migrated all the original OO implementations of the Parsers and Token Parsers.

There are a couple of interesting additions worth noting. This FromRegex() method can be used to fluently build a Parser by combining a regex pattern and a function that processes the successful Regex Match.

public static TokenParser FromRegex(string pattern, Func<Match, TokenisationResult> resolver)
{
    var regex = new Regex(pattern);
    return (string token) =>
    {
        var match = regex.Match(token);

        if (!match.Success)
            return TokenisationResult.Fail();

        return resolver(match);
    };
}

This enables us to separate the Regex match step from the actual processing of the successful match, and use them together via a combinator.

public static TokenParser JoiningWord = Tokeniser
    .FromRegex(@"^[Tt]o$", match =>
        TokenisationResult.Success(new JoiningWord()));

Monad Parsers vs Functor Parsers

If we have a closer look at the original Parser implementations, we’ll see that they aren’t all specifically Monads or Functors. I’m not confident enough to accurately categorize them, but have a think about where they sit relative to the functional concepts described above:

• Is - Parser<T> IsToken<T>()
• Then - Parser<U> Then<T, U>(Parser<T> first, Func<T, Parser<U>> second)
• Or - Parser<T> Or<T>(params Parser<T>[] children)
• Select - Parser<U> Select<T, U>(Parser<T> child, Func<T, U> converter)
• ListOf - Parser<List<T>> ListOf<T>(Parser<T> child)
• End - Parser<T> End<T>(Parser<T> child)

Functionally Parsing DayTime

Now we can go back to the starting scenario from the previous post which was initially solved with the OO approach:

public static IParser<DayTime> DayTimeParser =
    new Then<DayOfWeek, DayTime>(
        new IsToken<DayOfWeek>(),
        dow => new Select<LocalTime, DayTime>(
            new IsToken<LocalTime>(),
            lt => new DayTime { Day = dow, LocalTime = lt }));

This can now be assembled with functions:

public static Parser<DayTime> WeekDayTimeParser =
    Parsers.Then(
        Parsers.IsToken<DayOfWeek>(),
        dow => Parsers.Select(
            Parsers.IsToken<LocalTime>(),
            lt => new DayTime { Day = dow, LocalTime = lt }));

And we can apply our fluent API on top of this:

public static Parser<DayTime> DayTimeFluentParser =
    Parsers.IsToken<DayOfWeek>().Then(dow =>
        Parsers.IsToken<LocalTime>().Select(lt =>
            new DayTime { Day = dow, LocalTime = lt }));

And now we can do the same for all of the other combinations of parsers that we have, pure functions and functional combinators.

Code

The code is all available here where I put together a bunch of different ways to do parsing. We’ve just covered the functional part, so check out the Functional folder.

Note

It’s interesting to note that when you apply the fluent API, both the OO and Functional code is now identical on the surface. The OO pattern is still packaging object => delegate => object => delegate, but at least it’s tidy. In the new functional implementation it’s just delegates all the way.

The reason they are so similar is that I cheated in my last post and the “Object Orientated Approach” is actually a hybrid, but I find it helps bridge the gap between the OO and functional concepts.

Credits

Header Image by jacqueline macou from Pixabay

// Separate two part element context
"Pickup Mon 08:00 dropoff wed 17:00"

// Range elements with different separators
"Open Mon to Fri 08:00 - 18:00"

// Repeating tokens
"Tours 10:00 12:00 14:00 17:00 20:00"

// Repeating complex elements
"Events Tuesday 18:00 Wednesday 15:00 Friday 12:00"

Expression Trees

Selecting exactly two values into a single object is simple, but when things become combinations of conditional, repetitive or optional rules, the patterns cannot be easily configured in a single phase pattern match. We need a better understanding of the pattern structure in order to begin to solve it with code. For this, we are going to need to define our sentences as expression trees, with the root being the complete concept we are parsing.

You read an expression tree from left leaf to right leaf, with each branch node being a syntax rule and each leaf element being a token extracted from the original sentence. We can start by describing our original parsing subject, DayTime:

This is simple enough, but our sentences are more complex concepts than this, a higher order object can be composed of other higher order concepts, and what is a root for some scenarios may be a child for others. Consider the expression trees for the scenarios above:

Pickup / Dropoff	Range seperators	Repeating Tokens	Repeating Complex Children

Basic Parser Rules

In order to begin to process the above scenarios, we’re going to need something to handle our expression rules, let’s stick with those needed just for handling DayTime for now:

• IsToken - A parser that matches a single token. (These would be the blue Token nodes in the tree)
• Then - A parser that matches two consecutive parsers together in a chain.
• Select - A parser that converts the result of one Parser into a different result type. (These would be the purple higher order nodes in the tree)

Parser Chaining

How do you start to combine these together? We are in a situation where we have a list of tokens we want to provide to a parser, but we also need to put a parser inside another parser and do some magic… We’re missing some key concepts. Let’s go back to our naive parser implementation:

public interface IParser<T>
{
    ParseResult<T> Parse(Token[] tokens);
}

public class ParseResult<T>
{
    public bool Success { get; init; }

    public T Value { get; init; }
}

We can’t really chain this. Our input is always a Token[] array, and our output is always a result of processing the complete input. But what if an instance of a parser only consumed the tokens that were relevant to itself and its children, and returned the remaining tail? How would we do that?

We can create a Position object that encapsulates our Tokens and Ordinal:

public class Position
{
    public Position(Token[] source, int ordinal)
    {
        Source = source;
        Ordinal = ordinal;
    }

    public Token[] Source { get; }

    public int Ordinal { get; }

    public Token Current => !End ? Source[Ordinal] : throw new ApplicationException("At End");

    public bool End => (Ordinal == Source.Length);
}

From here, if we change the input to Position and we add a Position to the response, then when we take the a successful Parse attempt that has moved the Ordinal forward, we can feed that relative Position into the next parser in the expression tree, and we’ve opened up the possibility of chaining and composition.

public interface IParser<T>
{
    ParseResult<T> Parse(Position position);
}

public class ParseResult<T>
{
    public bool Success { get; init; }

    public T Value { get; init; }

    public Position Position { get; }
}

Now let’s look at some implementations to get an idea of what they do individually.

Basic implementations

IsToken

We can start with the most basic check, see if the current position is a match for the Token, and if it is, return the next position and the result, otherwise return a failure at the current position. We’ll use this to check for Day and LocalTime.

public class IsToken<T> : IParser<T>
{
    public IsToken() { }

    public ParseResult<T> Parse(Position position)
    {
        return position.Current.Is<T>() ?
            ParseResult<T>.Successful(position.Next(), position.Current.As<T>()) :
            ParseResult<T>.Failure(position);
    }
}

Then

The Then implementation contains a first parser, representing the syntax on the left, and a function to generate the second parser, which we run and use to evaluate the syntax on the right but only if the first parser is successful. In our scenario, we will be looking for an Is<DayOfWeek> on the left, followed by Is<LocalTime> on the right.

public class Then<T, U> : IParser<U>
{
    IParser<T> _first;
    Func<T, IParser<U>> _second;

    public Then(IParser<T> first, Func<T, IParser<U>> second)
    {
        _first = first;
        _second = second;
    }

    public ParseResult<U> Parse(Position position)
    {
        var result = _first.Parse(position);
        position = result.Position;

        if (!result.Success)
            return ParseResult<U>.Failure(position);

        var thenResult = _second(result.Value)
            .Parse(position);
        position = thenResult.Position;

        return thenResult.Success ?
            ParseResult<U>.Successful(position, thenResult.Value) :
            ParseResult<U>.Failure(position);
    }
}

Select

Select takes a core Parser as an argument and a converter function. When the core parser is successful, the result is fed into the converter function and the converted result is returned. Here, our converter method will pass in the previous two values and generate our DayTime.

public class Select<T, U> : IParser<U>
{
    IParser<T> _core;
    Func<T, U> _converter;

    public Select(IParser<T> core, Func<T, U> converter)
    {
        _core = core;
        _converter = converter;
    }

    public ParseResult<U> Parse(Position position)
    {
        var result = _core.Parse(position);
        position = result.Position;

        return result.Success ?
            ParseResult<U>.Successful(position, _converter(result.Value)) :
            ParseResult<U>.Failure(position);
    }
}

Using this pattern, we can build a tree of parsers to represent our expression and run this over our Token array.

Putting it together

Let’s take our original naive parser for DayTime, and replace it with our new parser setup.

public static IParser<DayTime> DayTimeParser =
    new Then<DayOfWeek, DayTime>(
        new IsToken<DayOfWeek>(),
        dow => new Select<LocalTime, DayTime>(
            new IsToken<LocalTime>(),
            lt => new DayTime { Day = dow, LocalTime = lt }));

Ok - it works, it’s got the Is<DayOfWeek>(), Then(), Is<LocalTime>(), Select<DayTime>() pattern, but this isn’t particularly fun to work with. There are lots of new keywords and the code reads like Polish notation.

Making it fluent

To make things easier, we can create a bunch of static helper methods to configure our root parsers, and a set of extension methods to combine our parsers together:

public static class Parsers
{
    public static IParser<T> IsToken<T>() =>
        new IsToken<T>();

    public static IParser<List<T>> ListOf<T>(IParser<T> parser) =>
        new ListOf<T>(parser);

    public static IParser<T> Or<T>(params IParser<T>[] options) =>
        new Or<T>(options);
}

public static class ParserExtensions
{
    public static IParser<U> Then<T, U>(this IParser<T> core, Func<T, IParser<U>> then) =>
        new Then<T, U>(core, then);

    public static IParser<U> Select<T, U>(this IParser<T> core, Func<T, U> select) =>
        new Select<T, U>(core, select);

    public static IParser<T> End<T>(this IParser<T> core) =>
        new End<T>(core);
}

Then we can re-write our original implementation using a fluent approach to compose our parser tree, with the result being much cleaner:

public static IParser<DayTime> DayTimeFluentParser =
    Parsers.IsToken<DayOfWeek>().Then(dow =>
        Parsers.IsToken<LocalTime>().Select(lt =>
            new DayTime { Day = dow, LocalTime = lt }));

But this is only a parser for our basic scenario, right? We’ve added a whole bunch of code, complexity and fluff, with no new functionality, and we’re still left with only this?

Composition

Well… not quite. Now we can create parsers from other parsers; that’s what we did when we created our DayTimeParser from Is<DayOfWeek> and Is<LocalTime> checks, so let’s take our pickup / dropoff scenario above, where our DayTimeParser can point to either of our fluent/non-fluent implementations:

// This parser returns a DayTime where it is preceded by a PickupFlag
public static IParser<DayTime> PickupDayTime => Parsers
    .IsToken<PickupFlag>()
        .Then(_ => DayTimeParser);

// This parser returns a DayTime where it is preceded by a Dropoff flag
public static IParser<DayTime> DropOffDayTime => Parsers
    .IsToken<DropoffFlag>()
        .Then(_ => DayTimeParser);

// This parser returns a PickupDropOff object that is derived from a 
// sequential Pickup/Dropoff match. Note that each of the Pickup/Dropoff
// parsers return a DayTime.
public static IParser<PickupDropoff> PickupDropOff => 
    PickupDayTime.Then(pu => 
        DropOffDayTime.Select(dr => 
            new PickupDropoff { Pickup = pu, DropOff = dr }))
    .End();

And that’s it, we’ve got our text with our pickup DayTime and our dropoff DayTime mashed together into a single object, using composition and some tidy fluent snippets.

Completing the set

We’re still pretty limited in terms of what we can do with our original Is/Then/Select parsers, we can’t cater to some of the more complex scenarios. We’re not done yet.

Or

A parser that matches one of multiple child parsers of the same result type. Let’s say we want to treat two different Tokens the same way; we need something conditional in our expression to indicate that one expression must be a match. Let’s take an expression where we want to use “to” or “-“ to indicate that we are matching a term describing a range between two values.

public static IParser<RangeMarker> RangeMarker =
    Parsers.Or(
        Parsers.IsToken<HypenSymbol>().Select(r => new RangeMarker()),
        Parsers.IsToken<JoiningWord>().Select(r => new RangeMarker())
    );

We can then use this as part of a combination to satisfy either condition:

public static IParser<Range<LocalTime>> TimeRangeParser =
    Parsers.IsToken<LocalTime>().Then(from =>
        RangeMarker.Then(_ =>
            Parsers.IsToken<LocalTime>()
                .Select(to => new Range<LocalTime> { From = from, To = to })));

And we can match 12 or 24 hour clocks, separated by a range marker of “to” or “-“.

ListOf

A parser that matches a number of items against a single child parser and returns a list of the child parser values:

ParseResult<List<DayTime>> result = Parsers
    .ListOf(DayTimeParser)
    .Parse(tokens);

End

A parser that determines if the result is the end of the Token array. This prevents a match from being successful when there are unconsumed tokens from the complete parsing expression.

public static IParser<DayTime> ExplicitDayTimeParser =
    DayTimeParser.End();

All of these lovely Parser implementations are available here, with the configurations for the original sentence structures here. There might even be some tests in there somewhere.

Turtles

And after that… it all gets a bit, Turtles… all the way down…

So if you want to take this thought experiment a bit further, try thinking about (or implementing) lists of lists of complex objects…. or lists of lists of lists… or just n+1. Once you solve the problem at one level, you can solve it at any level.

Code

More code turtles are available here where I put together a bunch of different ways to do parsing. We’re currently still in the slavishly Object Orientated part, so check out the OO folder.

Next Up

So far, we’ve been good little object orientated boys and girls, we’ve got our Interface, we have our little implementation classes, we’ve got good SOLID principles, but it’s still pretty heavy… this interface is only a single method.

public interface IParser<T>
{
    ParseResult<T> Parse(Position position);
}

Does it need to be an interface? What else could it be?

public delegate ParseResult<T> Parser<T>(Position position);

via GIPHY

Credits

Header Image by Glen Carrie on Unsplash

In my last post, I talked about using Regex to match the pattern of a WeekDay or ClockTime string and unpacking this into a meaningful object. This is known as Parsing, and we can see plenty of examples in the .Net framework, int.Parse(), Guid.Parse(), plus others. But how do we deal with parsing more complicated bodies of text that are made up of multiple words / parts?

"Monday 08:30am"
"tue 18:30"
"thurs 12:30pm"

We’re going to tackle this in two phases firstly we’re going to split our text into parts and parse them into tokens (Tokenisation/Scanning phase), then we’re going to check the tokens match a correct syntax pattern and turn that into an object (Parsing phase).

Tokeniser

To start with, we’re going to need a Tokeniser for converting a complete string into tokens. This is going to need a split pattern and a number of ITokenParsers to identify and convert the individual string fragments into sentence primitives.

public class Tokeniser
{
    Regex _splitPattern;
    IList<ITokenParser> _tokenisers;

    public Tokeniser(string splitPattern, params ITokenParser[] tokenisers)
    {
        _splitPattern = new Regex(splitPattern);
        _tokenisers = tokenisers.ToList();
    }

    public IEnumerable<Token> Tokenise(string inputString)
    {
        var parts = _splitPattern.Split(inputString);
        foreach (var part in parts)
        {
            var match = _tokenisers
                .Select(t => t.Tokenise(part))
                .Where(t => t.Successful)
                .FirstOrDefault();

            yield return match == null ?
                Token.Create(part) :
                match.Token;
        }
    }
}

public interface ITokenParser
{
    TokenisationResult Tokenise(string token);
}

public class TokenisationResult
{
    public Token Token  { get; init; }

    public bool Successful  { get; init; }
}

We will now need to port the logic from the original post for matching and parsing DayOfWeek and LocalTime and implement our ITokenParser interface, then drop them all into the Tokeniser…

Now we have an array of Tokens generated from a string, with tokens that match a specific Tokeniser rule being captured and converted, note that I’ve also included an integer converter, and any unmatched tokens are returned as String tokens. But we still need something that checks if these tokens are a recognizable configuration, and then converts this into a meaningful object.

“Parsing” a complete DayTime

While we’re only writing one parser for one pattern right now, it’s a common concept, so we can start with an interface. We’ll also follow the Regex response pattern, which returns a Match object with a Success flag and result information, but we will be returning a typed object.

public interface IParser<T>
{
    ParseResult<T> Parse(Token[] tokens);
}

public class ParseResult<T>
{
    public bool Success { get; init; }

    public T Value { get; init; }
}

Our DayTime object is composed of two parts, the Day and the Time. All we need now is to create a simple expression parser that takes an array of Token[] and populates the fields in a DayTime object.

public class DayTime
{
    public DayTime(DayOfWeek day, LocalTime time)
    {
        Day = day;
        LocalTime = time;
    }

    public DayOfWeek Day { get; }
    public LocalTime LocalTime { get; }
}

public class SimpleDayTimeParser : IParser<DayTime>
{
    public ParseResult<DayTime> Parse(Token[] tokens)
    {
        if (tokens.Length != 2)
            return ParseResult<DayTime>.Failure();
        if (!tokens[0].Is<DayOfWeek>())
            return ParseResult<DayTime>.Failure();
        if (!tokens[1].Is<LocalTime>())
            return ParseResult<DayTime>.Failure();

        var result = new DayTime(
            tokens[0].As<DayOfWeek>(),
            tokens[1].As<LocalTime>());

        return ParseResult<DayTime>
            .Successful(result);
    }
}

And that’s it, we can convert our two part string into a higher order object.

Expression Trees and Parsers

Our “parser” implementation is a bit of an overstatement here, this is the most naive logic we can get. It doesn’t have any flexibility for supporting more complex structures and is only going to support a very explicit match. If we wanted to extend this to support more complex patterns, we’re going to be quite limited. Consider trying to solve the following patterns:

// Separate two part element context
"Pickup Mon 08:00 dropoff wed 17:00"

// Range elements with different separators
"Open Mon to Fri 08:00 - 18:00"

// Repeating tokens
"Tours 10:00 12:00 14:00 17:00 20:00"

// Repeating complex elements
"Events Tuesday 18:00 Wednesday 15:00 Friday 12:00"

To solve these problems, we’re going to need a parser with the ability to process an expression tree, and we’ll get to that next.

All the code is available in a small sandbox project I’m building to go along with a couple of posts here.

Note

Post updated 7th January to line up with work done on demo project.

Credits

Header Image by Tim west from Pixabay

Because I went through a phase of being monumentally dumb recently, I felt it was appropriate to perform some kind of penance, and prove that I’m not actually an idiot when it comes to using Regex.

So this is an introductory post on using regex to perform basic tokenisation on a couple of common concepts, weekdays and clock times.

Regex for a Weekday

Let’s say we want to figure out if a string of characters represents the day Monday. Well, we need to figure out all the ways we can define Monday, then define that in a pattern. Mon, mon, Monday, monday are the 4 we’ll use to start. So, how do we create a pattern to find that?

Firstly, we’ll note that we want the beginning and the end of the string to line up with the beginning and the end of the word. To do this, we’ll need to use Anchors. These ensure that the text being matched begins at the beginning, and ends at the end of a string.

• ^ => This symbol anchors the beginning of a string
• $ => This symbol anchors the end of a string

Then we need the word itself. Let’s start with a single literal text - Monday.

^Monday$

This will match the start and end of a string matching the phrase ‘Monday’. ‘ Monday’ and ‘Monday ‘ would not match.

But what about our other permutations?

Well, we can handle the casing at the beginning of the word by using a range of characters.

• [Mm] => this represents a match against either an upper case M or a lower case m, so we can add this to the beginning.

^[Mm]onday$

This gets us Monday and monday, but we have no abbreviation. So we need to add something to indicate that the last section is optional.

• ()? => wrapping braces and placing a question mark after an expression (or a single character) makes it optional, so we can add this to indicate that the suffix (day)? may or may not exist.

^[Mm]on(day)?$

Broken down it looks like this:

And that’s it, we’ve got a nice little regex expression to check for Monday in various forms.

You can see the whole test suite here.

Groups

Many versions of regex have the ability to retrieve named groups from captures, in the .Net Regex library, this is achieved by wrapping the target in (?<name>pattern), and when you retrieve the result of a Regex.Match(), you will be able to look in the Match.Groups[“name”] property to retrieve the value.

• (?<name>pattern) => Capture a named group where the pattern is met.

This is useful when we have a number of related patterns that we want to group together, we can use the group names, combined with the | operator, which functions as an OR condition.

• this|that => match “this” or “that”

Full Week Regex

Now if we take all the regex for the individual days of the week, we can use the regex Group syntax (?<dayName>dayPattern), combined with the or pipe |. This enables us to set up a named match that covers each day, and when that match is met, we return the named group for the match.

”^(?<Monday>[Mm]on(day)?)|(?<Tuesday>[Tt]ue(sday)?)|(?<Wednesday>[Ww]ed(nesday)?)|(?<Thursday>[Tt]hu(rs(day)?)?)|(?<Friday>[Ff]ri(day)?)|(?<Saturday>[Ss]at(urday)?)|(?<Sunday>[Ss]un(day)?)$”

With this regex, we can now validate a string token and convert it from a range of input values to the correct matching Enum using some basic mapping logic.

DayOfWeek? Convert(string token)
{
    var match = Regex.Match(token, FullWeekRegex);
    if (!match.Success)
        throw new NotSupportedException($"Bad Format: {token}");

    if (match.Groups["Monday"].Success)
        return DayOfWeek.Monday;

    if (match.Groups["Tuesday"].Success)
        return DayOfWeek.Tuesday;

    if (match.Groups["Wednesday"].Success)
        return DayOfWeek.Wednesday;

    if (match.Groups["Thursday"].Success)
        return DayOfWeek.Thursday;

    if (match.Groups["Friday"].Success)
        return DayOfWeek.Friday;

    if (match.Groups["Saturday"].Success)
        return DayOfWeek.Saturday;

    if (match.Groups["Sunday"].Success)
        return DayOfWeek.Sunday;

    throw new NotSupportedException($"Bad Format: {token}");
}

Now we can run our word through this text converter and get back an Enum for DayOfWeek. This process is known as Tokenisation and is the first step in building a text processing system.

The whole setup is available here.

Clock Time

Let’s have a look at doing the same with a clock time. What’s the anatomy?

• Starts with 1-2 digits to indicate the hour part.
• Colon character ‘:’ to separate the hour from the minute.
• 2 digits to indicate the minute part.
• An optional am/pm indicator which, if omitted indicates a 24 hour clock.

Based on this, we’re going to need to match the following patterns:

// 12 Hr
"8:30am"
"08:30am"
"8:30pm"
"08:30pm"

// 24 Hr
"20:30"
"10:30"
"1:30"

So what does the pattern look like?

This gives us a pattern match, so we can detect a time pattern (at least approximately), but we still need to convert it into something useful, so we have to pick it apart. To do this, we’re going to go back to groups.

^(?<hr>\d{1,2}):(?<min>\d{2})((?<am>am)|(?<pm>pm))?$

Here we pull apart our clock time string and take the hour digits, the minute digits and the am/pm indicator group. Then we convert that into local time with a sprinkling of number range validation, and we’re done.

public LocalTime? MatchTime(string token)
{
    var match = Regex.Match(token, @"^(?<hr>\d{1,2}):(?<min>\d{2})((?<am>am)|(?<pm>pm))?$");
    if (!match.Success)
        throw new NotSupportedException($"Bad Format: {token}");

    var hour = int.Parse(match.Groups["hr"].Value);
    var min = int.Parse(match.Groups["min"].Value);
    var am = match.Groups["am"].Success;
    var pm = match.Groups["pm"].Success;
    var twentyFourHr = !am && !pm;

    if (min > 60)
        throw new NotSupportedException($"Bad Format: {token}");

    if (twentyFourHr && hour < 24)
    {
        return new LocalTime(hour, min);
    }
    else if (am && hour < 12)
    {
        return new LocalTime(hour, min);
    }
    else if (pm && hour < 12)
    {
        return new LocalTime(hour + 12, min);
    }

    throw new NotSupportedException($"Bad Format: {token}");
}

And we can see that running against our range of formats here:

Now we have the logic to determine if a string matches a Weekday or ClockTime pattern, and return the appropriate typed object we’ve got the first steps of a system to tokenise strings. I’ve spun up a small repo to play with these concepts, it’s available here, and I’m going to use it as a sandbox to dive further into the really cool powerful things you can do with regex and tokens.

If you want to pull the repo and have a play, there are a couple of exercises you can do:

• Can you make the day converter process say, German/French as well as English interpretations of days?
• What other international clock formats are there? Can you add support them?
• Create a basic tokeniser for an enum representing Hotel Room types, or Retail Stores.

I use a lot of regex. I’ve written parses and tokenisers for a wide range of DSL’s and data formats. So one day, I reached for my Regex-fu to validate a credit card number, and it was gone. Like superman finding he suddenly couldn’t fly, I found myself glued to the earth, taking the stairs…

What’s going on?

Does this look right to you? Because it sure didn’t look right to me…. and I spent a long time staring at it.

What’s really going on?

So… what is this dastardly bug that has cropped up? Why are my tests failing, and how did I get there? Well… I started with the simplest scenarios. This seems like a solid start right? We’re all good?

That all works right? Regex magic all good. But when I added the new scenario above, it failed. And this confused me, because I knew it worked on the simpler scenario, that assumption was now in my white list. My test pass, they prove the regex is correct, but a new scenario failes then it has to be a bug, right?

Well… not quite. See… here’s the thing…

via GIPHY

Regex… has TWO STRING PARAMETERS. Do you know what happens if you put them in the wrong order? It don’t work right. So what was my assumption? That the Regex pattern was on the left.

But… why didn’t we catch that? Surely the tests would have picked up that the Regex and the check are in the wrong order? We validate both the positive and the negative… But…

This check string matches the valid pattern, and the pattern matches the valid check string, so it works either way round.

So yeah, assumptions creep in, and you forget to go back to basics and re-validate.

Sick Note

I totally want to play my sick note card here. I had food poisoning and spent hours peering through heavy brain fog.

via GIPHY

But I don’t really have any excuse… this is just one of those experiences that keeps you humble.

So in a note to other devs… sick or not, check your bloody assumptions before you do something really stupid like filing a report for a non existant bug.

Credit

Header Image by OpenClipart-Vectors from Pixabay