Our new development approach was born out of a desire to overcome the headaches in the development workflow between front-end and back-end developers working in different technologies.

At the same time, we had long desired the dynamic user interface provided by a Single Page Application, but were unwilling to make the necessary compromises in SEO indexability and initial page load time. With our front-end tightly coupled to MVC’s server-side view engine, any experimentation was difficult.

Our Goals

  • Decouple the front-end and back-end to create a clear separation of concerns and reduce duplication of efforts
  • Combine the benefits of a Single Page App with the search-engine indexability and the speed of server rendering
  • Create a platform-agnostic REST API that could be consumed not only by our web-application, but future native applications too
  • Give front-end developers the freedom to autonomously develop and deploy front-end features

Our development team came together at the start of the year to think about how we could solve these problems collectively. We didn’t have a clear concept of where our goals would take us, but we began by looking into each problem in more detail.

Workflow Frustations

Since the emergence of specialised front-end and back-end roles in web development, a commonly seen workflow has front-end developers working in conjunction with design and UX team members to produce HTML templates, while back-end developers concoct the database and behind the scenes application logic. At some point of course, the two worlds collide — the templates must be merged with server side code to create a dynamic application.

This process became a major pain point for our development team — a never entirely satisfactory workflow whereby pristine HTML was broken up into Razor templates, at the discretion of the back-end developer.

The final dynamic templates never quite matched the intention of the front-end developer — the breaking up of HTML templates into partials was usually driven by the back-end view of the world — and the inherent freedom of the Razor templating language meant that it was simply too tempting to put what should be server logic into templates. Logic-heavy templates indicated a poor separation of concerns, and the resulting mangling of C# and HTML became unapproachable to our front-end team should additional changes in markup need to be made.

There began the problem of duplication of effort: Front-end templates are built in isolation, but only Razor templates go into production. Every enhancement or bug fix needed to be made back within the static HTML environment before repeating the entire process to put it back into production.

This process became a major pain point — a never entirely satisfactory workflow whereby pristine HTML was broken up and inserted into Razor templates, at the discretion of the back-end developer

With a .NET back-end, a front-end developer using a Mac could only achieve a full-stack development environment by running Visual Studio through a sluggish Windows virtual machine, in addition to needing either a local or shared remote database in order to see their templates running in a production-like environment. This never proved a reliable workflow, and would often result in front-end code changes being pushed before they had been properly tested in the belief that it would be quicker for back-end to fix any issues than it would be to continually maintain the front-end developer’s VM setup each time configuration or DB changes had been made to the .NET solution.

Overall, we estimated that this duplication in effort was costing us between a half a day to a day per significant feature, while general frustration arising from the workflow did little to contribute to team morale.

Shared Components

We knew that reducing duplication of effort within a C#/JavaScript stack would lead us toward certain elements of isomorphism, so we began to think about which elements in stack could be “shared” between the front-end and back-end. The term “isomorphism” is most commonly used when referring to a full JavaScript stack, but we wanted to see how we could apply the concept to what are typically viewed as separate and incompatible technologies. We had several options:

  • Use technologies with implementations in both JavaScript and C#
  • Use language-agnostic data formats wherever possible (e.g. JSON)
  • Look into methods of converting JavaScript into C# or vice-versa

Shared Logicless Templates

The Handlebars templating language was in many ways the starting point for our whole solution. Our lead front-end developer had worked with it before in previous efforts to modularise bloated PHP templates in WordPress, and enjoyed the strictness that it enforced through its intentionally limited set of available logic (#if, #unless, and #each), the idea being that if you find yourself needing to do anything more complex, the template is not the place for it.

Handlebars is also very clear in its goal — it simply renders templates against data. It is not a view engine, and it is not closely coupled to any particular language (although original written for JavaScript). It can therefore be integrated into any environment for which an implementation exists.

Thankfully, we were able to find a well maintained and up-to-date Handlebars implementation for .NET by Rex Morgan on GitHub called Handlebars.Net, which when used alongside its sister implementation in JavaScript was crucial to us being able to mirror the same view rendering behaviour on both sides of our stack.

Using Handlebars, we broke up our design into distinct reusable "modules", which could function in any context and in any order. Another level down, smaller reusable pieces of markup such as buttons and SVGs were turned into “snippets” for minimum duplication of code.

A Handlebars Module

<section class="bucket partner-bucket" data-behavior="bucket">
    <header class="bucket-header wrapper">
        <h2>{{module.title}}</h2>

        {{>bucket-controls}}
    </header>

    <div class="slider-track">
        <div class="slider wrapper">
            {{#each entry.partners}}
                <a href="{{this.linkUrl}}" class="partner">
                    <span class="image">
                        {{#if this.logo}}
                            <img src="{{this.logo.defaultUrl}}"
                                alt="{{this.name}}"/>
                        {{/if}}
                    </span>

                    <h4>{{this.name}}</h4>
                </a>
            {{/each}}
        </div>
    </div>
</section>
An example of a Handlebars “partner-bucket” module, referencing a “bucket-controls” snippet

A Global Data Structure

Handlebars is completely unopinionated about data structure and therefore allows complete freedom in the data models which are passed to it.

One of the drawbacks to our previous Razor templating solution was difficulty in accessing data that may not have originally been exposed or intended for a particular template. To make sure this problem was accounted for early on, we designed a global structure for our view models that would ensure that data remained organised and globally accessible at any time, by any module.

Consider the following object, made available to a Handlebars template:

var data = {
    site: {...},
    entry: {...},
    state: {...},
    user: {...},
    module: {...}
};
We organised the data for our view model in to clearly seperated domains

The site object contains all site-wide configuration data and text that isn’t specific to a particular entry or view (i.e. title, meta description, Google Analytics ID, feature toggles, etc).

The entry object contains the data for the current resource that is being viewed. For example, the home page, or a particular film. As the user navigates around the site, its data is updated to reflect the currently viewed resource.

The state object contains data pertaining to the state of the application, i.e. the currently active page or tab, or whether or not a modal is currently open and so on.

The user object contains non-sensitive data about the currently signed-in user, such as their name, avatar, purchase history, etc.

The module object allows us to import arbitrary data into a particular module without exposing it to any other modules. In other words, it is restricted to a module’s local scope. This is particularly useful within a loop of modules or when we want a generic module to be able to function in various different contexts by importing data into it.

We ensured that data over the API followed the exact same structure, which resulted the sharing of a single data service between the view controllers and API controllers on the back-end. Any data received via the API by the front-end would therefore be exactly the same as that used by the back-end when rendering a view.

Transpiled View Models

Our C# view models had always been the responsibility of the back-end team with the front-end team communicating any changes needed as new features were added (e.g. a new property being added, or existing data being made available where it wasn’t before).

This wasn’t particularly efficient and needed careful attention from both back-end and front-end to ensure that the desired properties were made available with the correct data and named consistently. It also broke our development goal of allowing front-end to push new features in their entirety when possible.

Retrospectively, it seemed obvious that front-end should be responsible for designing view models when creating or updating templates, and more generally responsible for determining the structure and semantics of data delivered to the front-end.

We decided therefore that view models should be created in the front-end in JavaScript. After initially attempting to manually recreate these view models in C#, we realised that this process could be automated if only we had the right tools. We decided to try our hand at creating a “transpiler” that would take a JavaScript view model in the form of a constructor function or ES6 class and recreate it in C# with the help of a small Node.js app and ironically, a Handlebars template of a C# class. Both languages being fundamentally C-like, we were able to convert from one to the other comfortably through enumeration, type checking and a not-insignificant amount of regex.

Both languages being fundamentally C-like, we were able to convert from one to the other comfortably through enumeration, type checking and a not-insignificant amount of regex

The final C# view models are then transferred into the .NET solution. When a view is requested, the appropriate data is mapped into them before being combined with Handlebars modules for rendering.

A Javascript View Model

class BoltOn extends Product {
    constructor() {
        this.bucketTitle    = '';
        this.assets         = [new Asset()];
        this.hasOpenAsset   = false;
        this.hasTrailer     = false;
        this.isPhysical     = false;

        Object.defineProperties(this, {
            isMultiAsset: {
                get() {
                    return this.assets.length > 1;
                }
            },
            hasTrailer: {
                get() {
                    return this.trailer !== null;
                }
            },
            hasCertificate: {
                get() {
                    return !this.isPhysical && this.certificate !== '';
                }
            }
        });
    }
}

export default BoltOn;
A JavaScript “Bolt-on” view model demonstrating getter properties and inheritance from a “Product” view model

A C# View Model

namespace Colony.Website.Models
{
    using System.Collections.Generic;

    public class BoltOn : Product
    {
        public string bucketTitle { get; set; }

        public List<Asset> assets { get; set; }

        public bool hasOpenAsset { get; set; }

        public bool isPhysical { get; set; }

        public bool hasTrailer
        {
            get
            {
                return this.trailer != null;
            }
        }

        public bool isMultiAsset
        {
            get
            {
                return this.assets.Count > 1;
            }
        }

        public bool hasCertificate
        {
            get
            {
                return !this.isPhysical && this.certificate != string.Empty;
            }
        }
    }
}
The same view model after transpilation into C#

Layout Files

We now had to find a way of dictating which modules should be rendered for a particular view and in what order, a task previously done by Razor “master” pages. Our first attempt was to hard-code arrays of modules into our view controllers, but this meant duplication of code between our back-end and front-end controllers, which again prevented the front-end from having complete ownership of the presentation layer of the application.

As these lists contained only data and not functionality, we decided that JSON would be the ideal format allowing the layouts to be easily consumed by either side of the stack. At this point we had basic JSON “layout” files that looked something like this:

[
    "head",
    "global-header",
    "banner",
    "list",
    "global-footer"
    "foot"
]
A simple layout file describing a home page

However, we soon realised that if we added a certain amount of simple logic to our layout files, code duplication elsewhere could be drastically reduced and rendering performance improved.

Firstly, we wanted to render certain modules only if specific conditions in our data model were met. For example, a user menu should only be rendered if the user is signed in:

[
    ...
    {
        "name": "user-sidebar",
        "if": [
            "user.isSignedIn"
        ]
    }
    ...
]
We created a declarative syntax to describe simple logic via JSON

We also wanted the ability nest arrays of modules within other modules, and import arbitrary data into a module, which when combined with basic Handlebars-inspired logic such as if, unless and each would enable a powerful and expressive way of describing view structure and logic using a tree data structure.

The following example showcases the full range of functionality that evolved throughout the process of our rebuild:

A JSON Layout File

[
    {
        "name": "head",
        "layout": [
            {
                "name": "emergency-banner",
                "if": ["site.emergencyText"]
            },
            {
                "name": "user-sidebar",
                "if": ["user.isSignedIn"]
            }
        ]
    },
    {
        "name": "global-header"
    },
    {
        "name": "editorial-preview",
        "if": [
            "user.isEditor",
            "site.isPreviewEnvironment"
        ]
    },
    {
        "name": "companion-overview"
    },
    {
        "name": "companion-coming-soon",
        "if": ["entry.userAccess.isComingSoon"]
    },
    {
        "name": "companion-presale",
        "if": ["entry.userAccess.isPresaleable"]
    },
    {
        "name": "companion-nav",
        "unless": [
            "entry.userAccess.isBlocked",
            "entry.userAccess.isComingSoon",
            "entry.userAccess.isSupportersOnly"
        ]
    },
    {
        "name": "asset-bucket",
        "if": ["state.isExploreTab"],
        "each": "entry.assetBuckets"
    },
    {
        "name": "global-footer"
    },
    {
        "name": "foot"
    }
]
A more complex layout file with various logical directives

It’s worth noting that in the case of if and unless, we could just have easily wrapped the entire HTML of the module in question within a Handlebars #if or #unless block, but apart from feeling like a hack that would reduce the readability of our modules, we realised that evaluating this logic earlier during the layout file parsing stage would avoid more expensive template evaluation later on by Handlebars, increasing rendering performance considerably.

Page Maker

We now had our Handlebars modules, layout files, and structured data. The next challenge was to combine these three elements into fully rendered views — a task which had to produce identical results both in C# on the server, and JavaScript in the browser.

We designed a class which became known as the “Page Maker” to perform this task. We knew at this point that code duplication between C# and JavaScript was unavoidable, so we took the opportunity to write both implementations as group programming exercises between front-end and back-end in the hope of sharing knowledge and hopefully shedding some light on each environment for the benefit of the other team.

The final class that we arrived at receives a JSON layout file along with data from a controller in the aforementioned structure, and iterates through the layout file rendering out modules, resulting in a the concatenation of a single, fully rendered HTML string.

We took the opportunity to write both implementations as group programming exercises between front-end and back-end in the hope of sharing knowledge

As the Page Maker iterates through each module in the layout file, it performs the necessary logic, loops and imports as instructed. If a module contains nested modules, it will recursively drop down a level and evaluate those modules before continuing with the original list.

In both the JavaScript and C# implementations, a non-blocking, asynchronous pattern is used to achieve the fastest possible rendering, with all modules being sent off to be rendered in parallel and the final HTML string being concatenated together only on the completion of all render tasks, which can occur in any order.

In the C# version, module templates are loaded from the file system, whereas in the JavaScript version modules are loaded over the API. In each case they are requested once and then cached for performance.

Front-end Single Page Application

Our team had combined experience in Angular, Backbone, Knockout, and Durandal as potential candidates for a framework that could handle the various aspects needed for our front-end Single Page Application (SPA). This was however not long after the infamous Angular 2.0 announcement, and investing time to tie us to a third-party framework that could soon become obsolete was now very much a questionable decision.

Our lead front-end developer had recently began work on building an SPA framework from scratch as part of a side-project and the various lessons learned in routing, controllers, dynamic user interfaces and data-binding could easily be applied to our new architecture. We decided therefore to forge ahead with our own solution, with the peace of mind that fully understanding all aspects of our stack would lead to faster development in the long term, and the ability to swiftly adapt the framework the needs of an evolving platform — with minimal hacking.

We decided therefore to forge ahead with our own solution, with the peace of mind that fully understanding all aspects of our stack would lead to faster development in the long term

Much of our front-end SPA framework follows the typical MVC pattern, with the exception that the server has already rendered the view in its entirety before the application takes over.

  • After the initial server rendering of a view, the application starts up in the background
  • An identical data “scope” is created (via the API) to what has just been rendered by the server
  • All link clicks are intercepted by a History API-enabled router
  • The router then assigns a controller, coupled to a specific layout file
  • The controller updates the entry scope via the API, and instantiates a Page Maker to render the view

A system of “behaviors” define and manage dynamic, replaceable user interfaces with isolated event-binding and garbage collection (similar to React), and also encompassing the functionality to data-bind elements to respond to changes in the scope, reusing the Handlebars logic already in the templates. DOM events are used to signal data changes rather than using observables or digest loops.

We were conscious to maintain a clean separation of concerns between HTML markup and JavaScript behavior, staying true to the now somewhat forgotten "Unobtrusive JavaScript" approach. This was achieved primarily by not tightly coupling our Handlebars markup to our JavaScript behaviors in any way (unlike React or Angular). When and if data is needed by a JavaScript behavior, it has full access to the global data scope, API (if needed), as well as the ability to pull in module-specific data from markup via data-attributes.

We were conscious to maintain a clean separation of concerns between HTML markup and JavaScript behavior staying true to the now somewhat forgotten “Unobtrusive JavaScript” approach

To maximize performance on the front-end, we implemented extremely aggressive caching (and pre-caching) of API resources including the data itself as well as all other components necessary to render views (layouts, modules and snippets). We also took advantage of HTML5 Storage to persist this entire cache between sessions, with the ability to flush specific elements on-demand when updates are made either to the database, or the front-end codebase.

Front-end & Back-end Development Workflow

We now had a fully independent front-end application containing the various shared components such as modules, layouts and view models. The problem remained of how to integrate those shared components into the .NET solution so that the full application could be deployed into production.

Initially we simply copied and pasted all relevant files into the solution, committed the code and deployed — but the inefficiency and error-prone nature of this workflow led us toward the idea of making the front-end a NuGet package that could be pulled in via our continuous integration server on TeamCity.

When front-end code is pushed, TeamCity creates a nuget package from the front-end Git repository. The files to be packaged are defined by a .nuspec file which also lives in the respository. TeamCity then hosts the files on its own NuGet server, ready to be pulled down.

Whenever front-end changes are pushed and ready to be integrated, the package can be pulled in via the Nuget tool within Visual Studio. Once installed, a powershell script is run automatically to copy files into all relevant folders within the solution and remove any previous versions.

Node.js Development Environment

A crucial part of solving the initial workflow problems was the creation of a front-end development environment that was as close to the production back-end as possible, while avoiding use of Visual Studio or Windows virtual machines.

When we first prototyped the Handlebars module rendering concept, we built a very lightweight Node.js application using Express.js to provide some basic routes, data and view rendering.

As development progressed we expanded this application to mimic more and more features of the back-end (in a very stripped-down way), so that we could fully test things like API calls and routes, without having to commit unfinished code just to test integration with the back-end.

Of course this “psuedo-application” could have been written in any language, but using Node and Express allowed for minimal code duplication. For example, our Node.js app is also largely isomorphic, sharing the same controller and route files across the stack, as well as many of the same helper files and utility libraries. Therefore the overall size of the application is much smaller than it would be were we using a different server-side language, and much easier to maintain.

Additionally, using Node as a server allowed us to use to Gulp and all the great build tools available to it, resulting in a fully integrated development environment with cross-stack watch, lint and build tasks.

This “psuedo-application” could have been written in any language, but using Node and Express allowed for minimal code duplication

The Final Architecture

Request Route Back-end C# View Controller DB Transpiled View Models Shared Data Service C# Page Maker Handlebars Modules JSON Layouts HTML Route Front-end JS View Controller APIController JS Page Maker
This schematic illustrates the full lifecycle of both the initial server-side request and all subsequent client-side requests, with the components used for each

Separation, Not Isolation

We were very lucky to have an unusually quiet month at the start of the year before launching our public beta which enabled us to focus almost exclusively on the rebuild. Had things been busier, we may never have got past the initial early experiments.

With that said, the overall time saved in our development processes as a result has been considerable, and 10 months later, has more than vindicated our decision to invest time in what started off as a somewhat experimental and potentially risky endeavour.

There were some initial reservations (and a fair amount of world-domination related jokes) from the back-end team around relinquishing control of certain features entirely to the front end, but in hindsight these decisions now feel like obvious and logical choices which have not only helped to free up resources, but have also furthered each team’s understanding and appreciation of the other’s technology and methods.

Tackling platform architecture as a team effort has formed the basis for a much healthier knowledge sharing process, with developers now able to frequently dip into the each other’s teams to help out when needed. We have found that a decoupling and separation of concerns need not mean an isolation of teams and responsibilities, but leads instead to increased collaboration, understanding, and ultimately innovation.