Posts on Tom Phillips

Staging is a wasteful lie: the case for the mono-environment

Thu, 08 Jan 2026 06:00:00 +0000

It is common for software engineering teams to deploy new versions of their software to staging environments before deploying it to users in production. In this post I argue that non-production environments are inherently wasteful and describe an alternative: the mono-environment.

I was first introduced to the concept of a mono-environment in 2018 at a SaaS startup. We had a solid engineering team, but were frustrated by slow delivery. After examining all our options, we realised that the costs of a staging environment outweighed the benefits, so we got rid of it, doubled down on our effective practices, and focussed on production.

Since then I’ve only worked in mono-environments, mostly at fintech startups, and even a bank. I now think it should be the default approach used by developers, especially at startups trying to achieve product-market fit.

Some former colleagues published a one-page website on mono-environments, but otherwise there is little to read about the case for a mono-environment and how to work safely in one. If anything, “testing in production” is a euphemism for sloppy and unprofessional practices. This article sets out to correct that.

Why are staging environments wasteful?

Your users can only use and get value from production, therefore any effort that goes into delivering software to staging is inherently wasteful. For many developers staging is their primary deployment target, so staging also presents a misalignment of incentives between developers and users.

You could argue that users indirectly benefit from staging because it’s a tool for inspection, i.e. checking functionality and looking for bugs. But this is still problematic because you can’t inspect quality into the software development process.¹ By the time a change reaches staging, if the quality isn’t there, then it’s a choice between shipping it anyway, re-working it, or binning it. All of these are wasteful. Instead, the quality needs to be there from the start.

Using staging as a quality gate also reduces the throughput of changes to production, thereby lengthening user feedback cycles and delaying delivery of value to users.

Staging environments are also unreliable quality gates because they are unrepresentative of production in two key ways. Firstly, staging is usually under lower load than production. Secondly, the data in staging is often unrepresentative of data in production, so a change might have the desired behaviour in staging but fail in production when the application processes real-world data. In my experience, this is particularly problematic with AI systems. So for staging to be a reliable indicator, it has to be very similar to production, but in practice each step towards parity gets progressively harder and more expensive.

Long-lived non-production environments have extra infrastructure costs. Configuring smaller resources for staging adds complexity and further diverges staging from production, making it an even more unreliable indicator of change quality.

Your developers also have the cognitive overhead of working with staging. What other changes are in staging? Has someone released my change to production? Of course, there are ways to manage this, but it is additional effort that could be spent on something your users care about.

Let’s not forget the misery of restarting work you thought you had finished weeks ago, because it was finally deployed to production and someone found a bug.

All the above is made much worse when you have multiple non-production environments. I once had a client where one team had five environments for their application. Things didn’t go well for them.

Overall, staging gives a false sense of security and creates a poor developer experience. It encourages large batch sizes, lowers throughput to users, and delays feedback. In a startup, all of this presents an existential threat because it reduces the rate of learning from users and increases cash burn.

What if we only operate production?

If we get rid of staging and only operate a production environment – a mono-environment – then how can we do so safely? Specifically, we want to maintain quality, reduce risk, and minimise waste. We need to replace environmental isolation with logical isolation, which means moving safeguards into the code.

Practically, we can do this by taking existing software development practices like testing, feature flagging and continuous deployment, and doing them well to work in fast feedback cycles. Here are five practices I think are key for a successful mono-environment.

Practice 1: testing

Use automated tests to ensure your change has the desired behaviour, and doesn’t break existing behaviour, before deploying to production.

We need to be confident that a change has the expected behaviour. We can get that confidence with automated tests. At risk of alienating some of you: use test-driven development (TDD). It gives you rapid feedback, so you know you’re moving in the right direction. The test suite guards against regressions and is also a form of documentation.

TDD is a skill. A common mistake doing TDD is testing implementation instead of behaviour, which makes for brittle tests that fail en masse when you refactor code. Another mistake is writing non-deterministic (“flaky”) or slow tests, which give you unreliable, slow feedback.

It’s also important to write tests of the right granularity: a unit test checks the behaviour of a single module, an integration test checks multiple modules function together correctly, and a smoke test checks basic functionality for a user (e.g. logging in and seeing the home page). The right mix will depend on your specific requirements and application architecture; further discussion is beyond the scope of this post.

Practice 2: local environments

Use ephemeral local environments to boost confidence in a change and get feedback via demos.

Even with TDD, you might miss an edge case or simply assert the wrong behaviour in your tests, so it’s useful to be able to do manual tests. It’s also useful to be able to demo a change to a colleague or customer for feedback as early as possible during development.

Ephemeral local environments are useful for these situations. A local environment won’t tell you exactly how your change will behave in production, much like staging wouldn’t, but they are still a useful tool to increase your confidence in a change without all the downsides of a long-lived non-production environment. They’re great for debugging too.

Set up a script to launch your local environment with one command. Check it into your repository so everyone can use it. The extent to which you can set up a complete local environment will likely depend on your architecture. If you’re using containers and have a monolithic backend and React frontend, then it’s trivial to bring up an entire app with a tool like Docker Compose. In a distributed system, it’s likely to be harder, so you might only bring up the service of interest and use stubs for everything else.

Practice 3: feature flagging

Use feature flags to test new functionality in production for a subset of users.

Feature flags allow you to turn on or off functionality in production based on some criteria, like the user making the request, without a deployment:

if featureFlags.get("canViewNewFeature", user):
    newFeature()

This is useful in a mono-environment because you can develop a feature in private behind a feature flag and find out how it behaves in production. I typically configure a new flag so that only colleagues can see the functionality, then when I’m happy with the change, I’ll either turn it on for everyone or do a gradual roll-out.

Lots of companies offering feature flag software for reasonable prices. Pick one you like.

It’s tempting to feature flag every change, but I advise against it. In startups, iteration speed is key, and in a mono-environment, it’s easy and fast to revert a change or deploy another, so more often than not I tend not to bother with a flag since it gets me feedback from more users at lower cost (because no time is spent on flag management). But for incomplete features or uncertain changes, I’ll use one.

In your tests, mock the feature flag provider so that you can specify the flag state for any given test.

Practice 4: continuous deployment

Use continuous deployment to ship changes quickly and seamlessly.

Shipping software is a key activity in software development. Frictionless deployment means faster feedback and delivery of value.

Continuous integration is the practice of developers integrating their code changes into the main branch of the repository frequently (at least daily) and building and testing the software on every commit. Continuous delivery is the practice of being able to deploy a new version of the software at a moment’s notice. (So continuous integration is a prerequisite for continuous delivery.) In a mono-environment, we use continuous deployment, which is the practice of automatically deploying your software on every change.

For continuous deployment, you typically need two pipelines. The first build and test pipeline runs on every commit and builds the software and runs the unit and integration tests. On the main branch, after a successful build and test run, the second deployment pipeline deploys the software.

Both need to be fast and reliable. I find it hard to tolerate anything longer than a few minutes. It’s essential to keep the build and test pipeline green, otherwise your deployments are blocked, which delays feedback on new changes and fixes for issues.

I also like to run smoke tests against production after every deployment to check I haven’t broken basic functionality like logging in and seeing the expected home page. These tests should use a dedicated test user/tenant.

Overall, deployments should be business-as-usual and completely ordinary events.

Practice 5: monitor production

Use monitoring and observability tools to know what’s happening to your users when they use your software.

You need to know what’s happening in production because it’s what your users experience. It’s what actually matters. If they are getting errors or performance problems, then you need to fix it fast, or at least be able to proactively get in touch before they contact you. Think about problems like uncaught exceptions and unexpected 400/500 HTTP response codes in logs. If you have a small number of users, then you might fire alerts every time, otherwise only when errors cross thresholds. Also track application events, so you can understand what your users are doing.

The technical approach to take depends on your particular situation. If you have a monolithic architecture and low number of users then you might be fine with log-based monitoring. On the other hand if you have a distributed system under high load, you might need more sophisticated observability tooling. I can’t really comment on the latter, as I’ve not worked on a distributed system with it set up (which caused me many headaches manually tracing requests through the logs of different services), so listen to an expert like Charity Majors instead.

Summary

All the techniques so far are about incrementally increasing the confidence in your change by repeatedly running through feedback loops. During development, each practice enables lots of frequent and fast feedback cycles, thereby minimising risk and increasing confidence in delivering value for users.

Common objections

Compliance requires us to have a staging environment

Verify this for yourself. On many occasions I’ve been told that a piece of legislation or regulation requires something specific, then when I’ve looked it up myself I’ve found the requirements to be much more flexible. Understand the regulations, work out how you can meet them, and discuss it with the relevant experts. For example, you might be able to argue that automated testing, continuous deployments, and comprehensive monitoring are more effective controls than infrequent manual checks in a staging environment.

What about database migrations?

Database migrations make people nervous in a mono-environment, but it’s worth remembering that a successful migration in staging doesn’t mean it will work in production. For example, a migration might run in tens of milliseconds in staging, but take much longer on a larger production database and cause an outage. Production databases often contain old, unusual data not present in staging, which could cause unexpected surprises.

Work in a pair and write down a plan. Consider what could go wrong and work out how you would roll back. Look at the query plan. Ensure you know what to do to recover from a disaster. Migrations should also be part of your continuous deployment pipeline.

Often, you need to use the expand and contract pattern. For example, let’s say you want to rename a column:

Migration 1: add the new column (expand).
Deploy a new version of your app that writes to the new column, and reads from the new and old columns, e.g. coalesce(new_col, old_col).
Migration 2: backfill any null values in the new column from the old column.
Deploy a new version of your app that reads and writes only from the new column (contract).
Migration 3: drop the old column. Optionally, make the new column not null.

Incremental delivery goes hand in hand with a mono-environment

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

— Agile Manifesto principle #3

Big bang releases are high risk because you find out if you’re right or wrong late in the development process. In contrast, incremental delivery (small, frequent changes) enables early feedback before it is too late to change tack.

Elephant carpaccio, coined by Alistair Cockburn, is a useful tool for thinking about incremental delivery. Carpaccio is an Italian dish of meat sliced so thin you can almost see through it. (I’m a vegetarian, so I’ve never tried it.) In software carpaccio, developers take a user story – the elephant – and slice it so thin that they can deliver a slice each day or, even better, several slices each day. To get a feel for this, watch Douglas Squirrel talk about delivery of a colour picker using elephant carpaccio.

Elephant carpaccio, and incremental delivery, are a natural fit for a mono-environment, because a mono-environment enables seamless delivery of each slice into the hands of users, and creates a tight feedback loop.

Conclusion

Staging and other non-production environments give a false sense of security. Time spent on them, instead of delivering value to users, is a waste of time and money.

Instead of relying on environmental isolation, the mono-environment relies on logical isolation by moving safeguards into the code. It achieves safety through rigorous, automated software engineering practices and aligns all developer effort with the environment that generates value and revenue: production.

Counterintuitively, the speed of delivery in a mono-environment is safer than using a staging environment, because incremental delivery in a single environment leads to tighter feedback loops and more learning, and a higher chance of achieving product-market fit before cash runs out. A mono-environment creates a virtuous cycle of faster and more robust delivery.

Lastly, I want to note how enjoyable it is to work in a mono-environment. It’s not unusual for my current team of 6 to have 30+ production deployments (including infrastructure changes and database migrations) and multiple customer conversations in a single day. Shipping so frequently can be hard to imagine if you’ve not done it before. I love technology, but what I love even more is shipping software people find useful, and a mono-environment helps me do that.

Thanks to Mike Hancock for feedback on a draft of this post.

I first heard the phrase “you can’t inspect quality into a process” from Dave Farley on his Modern Software Engineering YouTube channel. I looked into its origin and apparently it is a quote from the statistician Harold F. Dodge. ↩︎

Staging is a wasteful lie: the case for the mono-environment by Tom Phillips is licensed under CC BY 4.0

AGI fantasy is a blocker to actual engineering

Fri, 14 Nov 2025 06:32:56 +0000

Reading Empire of AI by Karen Hao, I was struck by how people associated with OpenAI believe in AGI. They really do think someone, perhaps them, will build AGI, and that it will lead to either the flourishing or destruction of humanity.

Elon Musk founded OpenAI because he thought Demis Hassabis was an evil genius who would build AGI first:

…Musk would regularly characterise Hassabis as a supervillain who needed to be stopped. Musk would make unequivocally clear that OpenAI was the good to DeepMind’s evil. … “He literally made a video game where an evil genius tries to create AI to take over the world,” Musk shouted [at an OpenAI off-site], referring to Hassabis’s 2004 title Evil Genius, “and fucking people don’t see it. Fucking people don’t see it! And Larry [Page]? Larry thinks he controls Demis but he’s too busy fucking windsurfing to realize that Demis is gathering the power.”

OpenAI’s co-founder and chief scientist Ilya Sutskever regularly told audiences and employees to “feel the AGI”. At a company off-site in Yosemite in September 2022, employees gathered around a firepit:

In the pit, [Sutskever] had placed a wooden effigy that he’d commissioned from a local artist, and began a dramatic performance. This effigy, he explained represented a good, aligned AGI that OpenAI had built, only to discover it was actually lying and deceitful. OpenAI’s duty, he said, was to destroy it. … Sutskever doused the effigy in lighter fluid and lit on fire.

I think it’s remarkable that what was until recently sci-fi fantasy has become a mainstream view in Silicon Valley.

Hao writes that GPT-2 was a bet on the “pure language” hypothesis, that asserts that since we communicate through language, then AGI should emerge from training a model solely on language. This is contrast to the “grounding” hypothesis, that asserts an AGI needs to perceive the world. Successfully scaling GPT to GPT-2 convinced enough people at OpenAI that the pure language hypothesis was valid. They just needed more data, more model parameters, and more compute.

So the belief in AGI, plus the recent results from LLMs, necessitates scaling, and justifies building data centres that consume hundreds of litres of water a second, run on polluting gas generators because the grid can’t supply the power (and might use as much power as entire cities), driving up CO2 emissions from manufacture and operation of new hardware, and exploits and traumatises data workers to make sure ChatGPT doesn’t generate outputs like child sexual abuse material and hate speech or encourage users to self-harm. (The thirst for data is so great that they stopped curating training data and instead consume the internet, warts and all, and manage the model output using RLHF.)

And this is all fine, because they’re going to make AGI and the expected value (EV) of it will be huge! (Briefly, the argument goes that if there is a 0.001% chance of AGI delivering an extremely large amount of value, and 99.999% chance of much less or zero value, then the EV is still extremely large because (0.001% * very_large_value) + (99.999% * small_value) = very_large_value).

But AGI arguments based on EV are nonsensical because the values and probabilities are made up and unfalsifiable. They also ignore externalities like environmental damage, which in contrast to AGI, have known negative value and certain probability: costs borne by everyone else right now.

As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment).

LLMs-as-AGI fail on all three fronts. The computational profligacy of LLMs-as-AGI is dissatisfying, and the exploitation of data workers and the environment unacceptable. Instead, if we drop the AGI fantasy, we can evaluate LLMs and other generative models as solutions for specific problems, rather than all problems, with proper cost benefit analysis. For example, by using smaller purpose-built generative models, or even discriminative (non-generative) models. In other words, make trade-offs and actually do engineering.

Discuss on Hacker News.

AGI fantasy is a blocker to actual engineering by Tom Phillips is licensed under CC BY 4.0

Understanding Kotlin DSLs

Tue, 04 Mar 2025 10:02:35 +0000

I’m learning Kotlin for a new job and I was confused by type-safe builders. They’re used to create domain specific languages (DSLs) in Kotlin. Two common use cases are configuring routes in Ktor server or building HTML. I think they’re confusing because they combine a bunch of Kotlin concepts. Here’s how I broke it down to understand it.

Let’s start with a data class to describe a pet. Note the member properties are mutable and have default values. I’ll come back to why at the end.

data class Pet(var name: String? = null, var description: String? = null)

Next we need a builder function. It takes one argument, init, which is a function type with receiver. In other words, init takes no arguments, and operates in the context of a Pet instance, and returns Unit. The builder function returns an instance of Pet.

fun pet(init: Pet.() -> Unit): Pet {  
    val pet = Pet()  
    pet.init()  
    return pet  
}

You use it to create an instance of Pet like this:

var jessica = pet {  
    name = "Jessica"  
    description = "Black and white cat"  
}

There are two confusing things going on here.

First, there’s no () calling the pet function. Instead, there’s a trailing lambda. In Kotlin, when the last argument to a function is a function, then you can omit the parentheses and pass the function as a lambda expression in curly brackets.

Second, how do the assignments to name and description end up as member properties of jessica? Previously I said init is a function type with receiver, and inside these types of function, the receiving object can be accessed via this. But where’s the this? You don’t need it because Kotlin infers that name is actually this.name.

We can then nest another class inside Pet. Let’s add information about the Owner of the pet.

data class Owner(var name: String? = null)  
  
data class Pet(var name: String? = null, var description: String? = null, val owners: MutableList<Owner> = mutableListOf<Owner>()) {  
    fun owner(init: Owner.() -> Unit): Owner {  
        val owner = Owner()  
        owner.init()  
        owners.add(owner)  
        return owner  
    }  
}

We’ve added a member property called owners and a member function called owner. This is another builder, like pet, that creates an instance of Owner, calls init, then adds the owner to the pet’s list of owners, and returns the owner.

Put this altogether to get our DSL:

var jessica = pet {  
    name = "Jessica"  
    description = "Black and white moggy"  
    owner {  
        name = "Tom"  
    }  
    owner {  
        name = "Kelly"  
    }  
}

// println(jessica) returns:
// Pet(name=Jessica, description=Black and white moggy, owners=[Owner(name=Tom), Owner(name=Kelly)])

When I defined Pet I noted that the member properties must be mutable and have default values. You need defaults because the builder functions create an instance without passing any arguments to the constructor. Then because init sets the member properties, they must be mutable too. The exception is Pet.owners, because we don’t want to reassign the (mutable) list reference, just modify the contents of it.

Here’s the code in Kotlin Playground.

Understanding Kotlin DSLs by Tom Phillips is licensed under CC BY 4.0

Leaving iPhone after 16 years for Android

Sun, 02 Mar 2025 00:00:00 +0000

For 16 years I’ve used an iPhone. I’ve owned maybe 4-5 models and my current one is an iPhone 11. The battery is knackered and I started to think what I’d do if I needed to replace it. iPhones are getting bigger and more expensive. The cheapest model, the iPhone 16e, starts at £599. I don’t want to spend that much money on a phone.

Lately I’ve become uncomfortable with Apple’s control over what I can do on my phone. If it’s not on the App Store, then I can’t run it, despite having bought the hardware. I used to find genuinely useful apps, but I can’t remember the last time I downloaded something like that. Nowadays it’s full of low-quality apps designed to extract micro-transactions from users, including children. Apple makes a ton of money from their commission.

Apple has the ability to remove apps from the App Store, including ones already purchased and downloaded, so oppressive regimes can force Apple to remove apps that they don’t like. In the UK, Apple has disabled end-to-end encryption for iCloud data, so the government – or whoever manages to compromise Apple’s key management – can access the data of users in iCloud, including all device backups. This is hugely invasive. I can’t even switch to another provider because Apple locks iOS down.

Let’s not forget Apple Intelligence: I don’t want generative AI crap on my phone or computer.

Overall, I don’t trust American tech companies to do the right thing anymore. They operate in an authoritarian regime. I can only see the situation getting worse.

So a week ago I switched to a Pixel 8a running GrapheneOS.

GrapheneOS is a privacy and security focussed mobile operating system based on the Android Open Source Project. One of its features is sandboxed Google Play, so you can download things like banking apps that don’t work or are unreliable on other Android OS like /e/os and LineageOS.

I chose a Pixel 8a because it is a device recommended by GrapheneOS and comes with 7 years of support from launch (May 2024). Wirecutter also recommends it as their budget pick. It’s also well-priced. I paid £349. I’d have preferred a Fairphone over a Google device, but they’re more expensive and unsupported by GrapheneOS.

Installing GrapheneOS via WebUSB was easy (and really cool). Overall, I’ve found it intuitive and easy to use. Perhaps it’s slightly less polished than iOS, but it’s fine.

It was easy to find replacement for apps. I already use Fastmail, so I downloaded their app for my email, calendars and contacts. I downloaded DAVx5 to sync my Fastmail contacts to the Contacts app. For now I have some calendars shared with my partner on iCloud. I’m accessing those via Fastmail too.

I simply stopped using iMessage. Everyone uses WhatsApp or, better, Signal anyway.

I replaced iCloud Photos with Ente. It uses end-to-end encryption. I’ve not uploaded my whole library yet, but it’s working fine so far.

I swapped Overcast for AntennaPod. I have noticed the absence of volume normalization, but I can live without it.

I’ve used Things to manage todo lists since ~2008. But really I have simple needs. So I’m trying out a plaintext todo list in Obsidian, with end-to-end encrypted sync via Obsidian Sync, and simply putting date-sensitive items in my calendar.

The only niggle so far was some lag on the GrapheneOS camera app when pressing the shutter button. I downloaded the Google Camera app and disabled network access. It’s much faster.

I’ve not tried out device backups yet, but since my data is either in Fastmail, Ente, 1Password or Obsidian, I feel comfortable with the risk. I might try backing up to a USB stick.

I installed Obsidian and Tor via .apk files. You just download and install it! I got Organic Maps from the Accrescent store. I might try out Obtainium.

Overall, I’m really happy. It’s been much easier than I expected to switch. I’ve probably spent 3-4 hours on it in total. Go for it!

Leaving iPhone after 16 years for Android by Tom Phillips is licensed under CC BY 4.0

What do you learn from your MVP?

Fri, 14 Jun 2024 00:00:00 +0000

Minimum viable product (MVP) has become meaningless business jargon.

The problem with so-called MVPs is that no one learns anything from them. There is no hypothesis under test.

If your MVP doesn’t unambiguously validate or invalidate a hypothesis, then it’s impossible to know whether you are succeeding or failing.

What do you know about your customers? What problems do they have? How do they get value from your product? Will the answers to these questions change if you build the MVP? If they don’t change, then whatever you’re building is a waste of time and money, and it’s not an MVP.

In The Lean Startup (first published in 2011!), Eric Ries writes that an MVP completes one cycle of the build, measure, learn loop.

If we’re honest, most faux MVPs are just collective agreement on what is a barely acceptable use of everyone’s time. Something to keep people busy. A minimum tolerable output.

So next time someone says “it’s an MVP”, ask “what are we learning from it?”.

What do you learn from your MVP? by Tom Phillips is licensed under CC BY-SA 4.0

Will dbt adopt a proprietary licence? I think so

Tue, 02 Apr 2024 00:00:00 +0000

Data build tool (dbt) changed the way many organisations transform data and do analytics. I think more change is afoot: I predict dbt Labs will adopt a proprietary licence for dbt Core and the project will fork.

dbt Labs use the loose open core business model, where the primary functionality in dbt Core is covered under an open source licence (Apache 2.0) with extra proprietary software wrapping around it, namely dbt Cloud.

The open core creates a large funnel of potential customers for the proprietary product. The difficulty with the open core business model lies in deciding whether a feature goes in the open core or proprietary product. The company wants to add functionality to the proprietary product to drive conversion of users to customers, but this often conflicts with the interests of the users of the open core. Despite this conflict of interest, the open core depends on the success of the the proprietary product to fund its development.

In dbt’s case, I think dbt Cloud isn’t a compelling product. Despite using dbt Core for years I have never bought dbt Cloud. I don’t know of any organisations who use it; I know of one who stopped.

A recent job advert says dbt Labs have over 4,100 dbt Cloud customers. At $100/month that’s ~$5M/year. Maybe they have some big enterprise deals, but that revenue is probably dwarfed by payroll. LinkedIn lists 421 members associated with dbt Labs. A conservative estimate of 300 employees at $100k/year is $30M/year, which would be a loss of ~$25M/year. Not great for an eight year old company valued at $4.2B

What can dbt Labs do? They already cut headcount by 15% and switched to consumption-based pricing. Can they ship some compelling features? I’m not convinced. Glassdoor reviews (taken with a large pinch of salt) complain about leadership and the product roadmap. In an interview last December, their CEO Tristan Handy said:

[W]hat I want to do is increasingly make living in the dbt ecosystem feel like living in the Apple ecosystem, which is to say that the hard stuff [like CI, observability, data cataloging] just kind of vanishes into the background. And you forget that that was ever a problem and you spend all your time thinking about business value.

This is a nice idea but it will be hard to execute. He acknowledged that comparing dbt Labs with Apple is a stretch, but putting that aside, the revenue models are completely different: people pay Apple $1000 for an iPhone and then pay even more for services (to the EU’s chagrin). In contrast, entry into the dbt ecosystem is free and users spend money elsewhere (e.g. on a cloud data warehouse or managed data transfer services).

By Tristan’s own admission, and unlike Apple, dbt Labs’ products aren’t even best-in-class:

So we just launched this thing called dbt Explorer. And you look at dbt Explorer and you compare it to a fully featured data catalogue … and I mean, gosh, it’s like David and Goliath, like, our product is kind of a toy. But it does a lot of things, the basic things that you want, and it is automatically in the hands of everybody who is in the dbt ecosystem. And that’s really powerful. And so we want to continue to do that, we’re not going to try to build the most complex best-in-class orchestration tool or best-in-class observer. But when you have a huge user group, and you can say, hey, here’s a really useful thing, and it works with all these other things, then you get to kind of raise the bar on the set of tooling that everyone has access to.

I can’t see how this will lead to a sustainable open source business. Even if they do convert a fraction of their users to customers, these customers are then at risk of churning to a more sophisticated product in the future.

What might happen next? dbt Labs coined “analytics engineering” and if you’re analytics engineer practicing what dbt Labs preach, you probably have a lot of models. What surprises me is that 5% of the dbt install base has over 5000 models! These customers are sitting ducks, locked-in to dbt.

How can dbt Labs get money from those users? By pulling the same move as HashiCorp and switching new releases of dbt Core to the Business Source License (BuSL), which would block competitors offering products based on it.

dbt Labs then release new features to tame huge dbt projects now efficiency and cost optimisation are in vogue and perhaps they will start to see more customers. Meanwhile, they can claim to be committed to open source, despite the BuSL not being an OSI-approved licence.

Coincidentally, HashiCorp’s former chief revenue officer Brandon Sweeney is now dbt Labs’ chief operating officer.

If this does happen, a fork of dbt core is in the interests of all the vendors selling “dbt compatible” products, much like businesses who got together to create the OpenTofu fork of Terraform. I think it would be good for users too. I am surprised by how long dbt has dominated the transformation part of the data stack. More competition and diversity of thought in the data industry is good for everyone.

I think dbt Labs should have stuck with their original consultancy business Fishtown Analytics and made dbt a free software product, charging their clients for a proprietary distribution. But that business model probably wouldn’t go down well with their investors. Unfortunately I see dbt ending up being yet another example of the consequences of choosing the wrong open source business model.

Will dbt adopt a proprietary licence? I think so by Tom Phillips is licensed under CC BY 4.0