May 10, 2023

I've spent the last four years building and maintaining a project called SaaS Pegasus. Pegasus is a SaaS boilerplate or SaaS starter kit. It provides a starting codebase for new SaaS projects with a bunch of features like user accounts, teams, and billing already included.

Most boilerplates are regular codebases. You clone a repository from Github, follow the getting started instructions, and you're up and running. What makes Pegasus different is that every project's codebase is unique. You enter information about your project and the tech stack you want to use, and Pegasus makes a codebase just for you. In other words, Pegasus is a codebase that makes other codebases.

Making—and more importantly, maintaining—a codebase-making codebase has posed some unique challenges, and I've had to get pretty creative to solve them. In this post I'll outline how Pegasus is built, the problems I've run into, and the tooling that holds it all together.

But first, I need to address the obvious question.

Why generate codebases at all?

I mentioned that most boilerplates just provide starting code and don't bother with any code-generation business. You may be thinking...wouldn't that be much easier? Why take on all this extra complexity?

The problem is my inner perfectionist.

When I started working on Pegasus, I had two important design goals:

The first was that it should be flexible. I knew every project would be different and wanted to make Pegasus flexible enough to handle a lot of different use cases.

The second was that I wanted developers to love it. Pegasus is a product geared towards developers—notoriously an opinionated and finicky bunch of people. For Pegasus to be successful, I needed it to create codebases that developers would be happy to use. For this constraint, I used myself as a litmus test. Anything that I personally wouldn't use was an instant no-go.

Anyway—let's first focus on flexibility. In order for a boilerplate to be flexible it has to be configurable.

As an example, many SaaS projects want to collect monthly payments from their users. But many others—internal tools or hobby projects—won't. To facilitate both of these use cases you need to have some piece of configuration that tells the app whether billing is enabled or not, and then have everything behave accordingly.

If the boilerplate is "just a codebase", this configuration has to be a runtime setting. You'd set USE_BILLING = True in your settings file, and that would dynamically enable the billing module, UI, and so on. This is typically how configurable open-source libraries work, and it works reasonably well.

if settings.USE_BILLING:
    show_billing_module()

Unfortunately, from a developer experience perspective—this sucks.

In this world, all generated codebases include the billing code (and all its dependencies)—even the codebases that never use it. It also means lots of "if USE_BILLING" statements sprinkled throughout the codebase that—for any individual project—only add unnecessary complexity. It would be hard for developers working on the code to know what could safely be deleted, what libraries were never used, and so on. This might be acceptable for libraries—since they are largely treated as a black boxes—but for the starting code for a new app it felt...sloppy.

So I decided runtime configuration was out. The purist in me wouldn't allow it.

What quickly became clear was that if Pegasus was going to generate code that satisfied developers like me, the generated codebase itself needed to be configurable. Then, by the time a developer got their hands on the code, all the extra cruft and branching logic would be gone and they could start with exactly what they needed.

So that's what I did.

Here's a little snippet from the Pegasus codebase creator:

I know it looks simple, but each of these checkboxes controls a huge amount of logic—from the UI that is created all the way down to the data models and required packages. Some of these even have interdependencies.¹ And Pegasus—the codebase creator—takes on all this complexity so that the developers using Pegasus are shielded from it.

Maintenance of this project has been… interesting. But over time I've managed to come up with solutions for most of the big problems I've run into.

With that out of the way, let's peek under the hood.

The secret to dynamic code generation: cookiecutter

At its core, SaaS Pegasus is a cookiecutter project. Cookiecutter is an awesome little utility that lets you create projects from templates. In my case, Pegasus is the template, and the codebases it produces are the projects.

Cookiecutter has two main components:

1. A set of configuration variables. These are the equivalent of USE_BILLING = True in the example above.
2. A logic/templating engine that sits on top of those variables (written in Jinja2). This is the equivalent of all the if USE_BILLING logic, above.

The examples from the project editing UI above produce a cookiecutter configuration dictionary that looks like this:

cookiecutter = {
    "use_teams": "y",
    "use_teams_example": "n",
    "use_translations": "y",
    "use_wagtail": "n",
    "use_openai": "y",
}

Then, you can add Jinja2 markup to any file in your project to write logic that depends on these variables.

For example, in Pegasus, I want to include or exclude certain URL paths depending on whether a feature is turned on:

urlpatterns = [
    # some urls are always included
    path('admin/', admin.site.urls),
    # but many are turned on/off depending on your configuration
{% if cookiecutter.use_translations == 'y' %}
    path('i18n/', include('django.conf.urls.i18n')),
{% endif %}
{% if cookiecutter.use_teams == 'y' %}
    path('teams/', include('apps.teams.urls')),
{% endif %}
{% if cookiecutter.use_openai == 'y' %}
    path('openai/', include('apps.openai_example.urls')),
{% endif %}
    # and so on...
]

Each of these {% if cookiecutter.use_thing == 'y' %} statements runs against the generated cookiecutter dictionary and either keeps or leaves out that content from the project.

Now, each generated project will have the right set of URLs based on the chosen configuration. The same trick can be applied to data models, business logic, the UI, dependencies, and so on—and no extra logic or code will be included in the final output!

Handling larger changes with cookiecutter hooks

The syntax above works great for changes within a file, but sometimes you need to do larger operations, e.g. include or exclude entire files or directories. For these bigger pieces, cookiecutter provides hooks that run before/after project creation. Cookiecutter's hooks are Python scripts that have access to the generated project directory. So, for example, if you want to delete entire files or directories based on a variable you can do that like this:

using_translations = '{{ cookiecutter.use_translations }}' == 'y'
if not using_translations:
    # remove locale files and middleware if we aren't using translations
    remove(os.path.join(project_dir, 'locale'))
    remove(os.path.join('apps', 'web', 'locale_middleware.py'))

Jinja templating and hooks can get you quite far—almost everything I've had to configure in Pegasus can be done with one of these two patterns.

How do you template a template?

SaaS Pegasus generates Django projects—which include HTML templates with their own template language that looks very similar to Jinja2:

<nav>
{% if user.is_authenticated %}
  <a class="navbar-item" href="{% url 'pegasus_examples:examples_home' %}">
    Examples Gallery
  </a>
{% endif %}
</nav>

This template snippet adds a navigation link to the example gallery only if the user is authenticated. But what if we need to layer on some cookiecutter logic? What if we only want to include this code in the template if the project was built with examples enabled?

A first attempt might look something like this:

<nav>
{% if cookiecutter.use_examples == 'y' %} <!-- cookiecutter check -->
{% if user.is_authenticated %}            <!-- Django template check -->
  <a class="navbar-item" href="{% url 'pegasus_examples:examples_home' %}">
    Examples Gallery
  </a>
{% endif %}                               <!-- end Django template check -->
{% endif %}                               <!-- end cookiecutter check -->
</nav>

But there's a problem. When cookiecutter is building this project, Jinja2 can't tell the difference between the cookiecutter options ({% if cookiecutter.use_examples == 'y' %}) and the Django ones ({% if user.is_authenticated %}). It tries to evaluate the whole file, but chokes as soon as it runs into something it doesn't understand (in this case, the reference to user.is_authenticated).

In order to make it work we have to tell cookiecutter's Jinja syntax to ignore the similar-looking Django template syntax. We can do this with the {% raw %} tag—which stops evaluation on everything inside it. Here's the updated example:

<nav>
{% if cookiecutter.use_examples == 'y' %} <!-- cookiecutter check -->
{% raw %}                                 <!-- pause cookiecutter parsing -->
{% if user.is_authenticated %}            <!-- Django template check -->
  <a class="navbar-item" href="{% url 'pegasus_examples:examples_home' %}">
    Examples Gallery
  </a>
{% endif %}                               <!-- end Django template check -->
{% endraw %}                              <!-- unpause cookiecutter parsing -->
{% endif %}                               <!-- end cookiecutter check -->

Now cookiecutter will ignore the troublesome {% if user.is_authenticated %} statement, and everything works correctly.

These templates can get quite unwieldy when there's a lot of intermingled cookiecutter and Django template logic! For kicks, try to make sense of the example below, taken straight from Pegasus's source code.

<body{% endraw %}{% if use_multiple %}{% raw %}{% if pg_is_material_bootstrap %} class="g-sidenav-show  bg-gray-200"{% endif %}{% endraw %}{% endif %}{% if use_material %} class="g-sidenav-show  bg-gray-200"{% endif %}{% if using_htmx %}{% raw %} hx-headers='{"X-CSRFToken": "{{ csrf_token }}"}'{% endraw %}{% endif %}{% raw %}>

It ain't pretty but it works...

Testing code that makes code

Testing a cookiecutter project presents its own challenges. The first one being, simply: how do you test it?

Importantly, cookiecutter projects themselves are not valid code. Remember our urls.py file?

urlpatterns = [
    # some urls are always included
    path('admin/', admin.site.urls),
    # but many are turned on/off depending on your configuration
{% if cookiecutter.use_translations == 'y' %}
    path('i18n/', include('django.conf.urls.i18n')),
{% endif %}
    # and so on...
]

This file will crash a Python interpreter the moment a {% %} tag is reached. Yes, once you run cookiecutter you should (hopefully!) get valid code, but the cookiecutter project—Pegasus itself—is incomprehensible nonsense.

So I can't run a test suite on the project itself. Instead I have to run the test suite on a generated project. And tests become a two-stage test process:

1. Generate a project codebase from the Pegasus template.
2. Run the tests on the generated project.

This is straightforward enough to script, but raises a new question: What project(s) should be tested?

Attempting to get decent test coverage

There are currently 25 configurable variables you can use in Pegasus. This means there are about 2²⁵—or ~33 million—different ways to configure a Pegasus project. That's a lot of combinations!²

Most of the bug reports I get for Pegasus inevitably end up being tied to some unique combination of variables—e.g. "building without teams or subscriptions, but with API keys enabled causes profile picture uploads to fail". It's a lot to manage.

In the early days, I'd fix the bug, then add an item to my release checklist to make sure this particular combination of variables still worked the next time around. The release checklist started getting quite long!

Eventually, as the number of options (and users) grew, so did the number of combinations I had to test. And soon I found myself spending entire days just banging on different combinations of variables looking for bugs. I needed something more sustainable.

Enter: automation. With the help of a friend, I turned to Github Actions, and we managed to set up a series of jobs that:

1. Generate a list of configurations to test.
2. Generate new projects based on each configuration.
3. Install and run the test suite (and anything else) on each generated project.

Now every change to Pegasus gets run through ~25 hand-picked build configurations, each of which ensures that a particular combination of features still works properly.

And now whenever I find a new bug involving a particular combination of flags—I add a line to the config file to test that particular combination, and I can rest assured it won't happen again!

Under the hood this works via Github's matrix support. The first job of the workflow creates the matrix of configurations as a JSON output, and the next job uses it to setup the project and run CI on each one.³

Testing coverage problem (mostly) solved!

Automating away complexity with post-processing

So that's the underlying code and testing infrastructure. But, there are still things that are hard to do in cookiecutter.

Code formatting, for one, is a nightmare. All the cookiecutter Jinja syntax makes it very difficult to control whitespace in the generated output.

Also compiled files are a mess. If a cookiecutter variable impacts JavaScript code, managing how that ends up in a compiled JavaScript bundle is impossible to do by hand. Similarly, managing a compiled requirements.txt file based off a cookiecuttered requirements.in file is a huge hassle.

To solve these issues, I've introduced a custom post-processing step to the pipeline. After cookiecutter generates the project, but before the developer downloads it, I run a number of things server-side to put the finishing touches on the project. Formatting the code with black, running isort, building the final requirements.txt file, compiling and bundling the front end, and so on.

Doing it this way helps avoid having to manage crazy, complex cookiecutter logic on anything that can be automated.

Wrapping up: complexity all the way down

At the end of the day, building Pegasus is—like most software projects—an exercise in managing complexity.

My starting goal was to remove as much complexity as possible from the end-product of Pegasus: the generated codebases. The simpler the generated codebases are, the better Pegasus the product will be.

But—the more complexity I shield from the end-user, the more I take on myself, and the harder the project is to maintain. This eventually slows me down and hurts the product too—if in a less direct way.

So I also have to remove complexity for myself—as I've done with cookiecutter, automated testing, post-processing and so on.

You could call what I'm doing complexity-driven development (CDD) or complexity juggling. As some area of Pegasus gets too complex, I prioritize coming up with a way to make it easier—either for my end-users or for myself. Once things are streamlined, I move onto something else.

Pegasus still isn't perfect, but the rate of complexity growth has stabilized. There are still a lot of balls in the air, but so far I'm managing to keep them afloat.

---

Thanks to Michael Lynch and Rowena Luk for providing feedback on a draft of this.

If you want to find out when I publish new content you can subscribe below, or follow me on Twitter.

And if you want to take a massive shortcut to launch your next project, I hope you'll consider SaaS Pegasus. Hopefully this post has at least convinced you that it's something I work quite hard on.

Notes