The Rise of Machine Learning Platforms

New technology inevitably suffers from what I tend to call the platform problem — people build platforms, rather than products. This happens when people, or companies, see a new emerging technology but don’t quite know what to do with it yet. The problem continues until the platforms are good enough, or widespread enough, that people automatically pick an existing one rather than reinventing the wheel. They start, in other words, to build products.

The problem is that people inevitably build platforms for longer than is really necessary. Every developer likes a green field project. Every developer wants to be the one to build the marketplace, or build the platform.

Early on building platforms is necessary, somebody has to sell the shovels, and irrespective of the hype over the last few years we're still early enough for building platforms make sense when it comes to machine learning and artificial intelligence. Because, despite the many applications that machine learning models are being put to, we still haven't quite got a handle on the infrastructure of building and deploying them.

Interestingly though, when it comes machine learning, we face two platform problems, not just one. It's not just that we don't have VisiCalc, we don't even have the Apple II to run it on in the first place.

The first platform problem we have is hardware related. The rush to scale our cloud infrastructure has propelled Nvidia's ascent to be the world's most valuable company, with the company's high end GPUs having played a vital part during the scrabble of the last couple of years.

However, scaling the infrastructure for AI has left a $600 billion revenue hole. That's annual revenue that has to be found to before the massive investment in silicon can turn a profit. There are more than a few hints here of the original dotcom boom, and the bust that followed, which left dark fibre that more two decades later we're still in the process of lighting up. So it remains to be seen whether we're overbuilding GPU in the cloud in the frantic race to market.

There's also the question of whether we're building the right hardware in the first place, whether the racks of Nvida GPU in our data centres are the right shovels. While training AI models will, by necessity, remain in the cloud, there are indications that inferencing is moving — almost with a sense of inevitably — towards the edge. The latest generation of edge hardware is more than capable of running models in real time on CPU, without the need for dedicated accelerator hardware, or racks of specialized GPU in the cloud.

But the move to the edge is also being partially driven by an evolution in model architectures. We're seeing both chained models — where existing light weight tinyML models are used as triggers for the more resource intensive SLMs or LLMs — alongside the use of full-scale LLMs running in the cloud to classify and label data to train a "traditional" tinyML model such as a MobileNetV2, which can be deployed to the edge and run on microcontroller-sized hardware inside a couple of hundred KB of RAM.

Which brings us to the second platform problem, the software one. To reach for a strained analogy. We're in still in the early UNIX era where for the most part people are carefully hand crafting their own tools — it's just that this generation of tooling is often written in Python rather than C.

But writing software like that is a huge bottleneck. Traditional software started out custom-developed within companies, for their own use; it only became a mass-produced commodity when demand grew past available supply, which is when most people's day-to-day tooling changed from programming to spreadsheets. With machine learning we've still to reach the Visicalc era.

The rise of prompt engineering as skill is an artefact of the platform problem we see today with the software we have wrapped around our models. The new LLMs are tools that we're not quite sure how to use, so they have been exposed directly to end users in the most raw form. The hallucinations we commonly see when using LLMs are inherent, but the reason we're seeing them is that the models are not properly constrained. The biggest platform problem we have when it comes to models, is that we don't have a universal abstraction of what a platform for machine learning model should do. But what it certainly shouldn't do is let end users hack directly on the model itself, that way just lies hallucinations and prompt injection attacks.

We are however starting to see the beginnings of these platforms emerge at least for "traditional" machine learning models, with startups like Edge Impulse, and enterprise focused offerings like Intel's Geti. But I think it's still somewhat arguable what Visicalc for machine learning might look like, and what it should be capable of abstracting away for the end user. Whether it's "traditional" machine learning models, or the new LLMs, it's not yet obvious what depth of knowledge a user should necessarily need to have to make use of them. How much abstraction tooling should provide.

The spreadsheet is a tool, but it's an extremely flexible one. Even in the hands of novice it can perform complicated tasks, its power is inherent in a combination of its abstraction of the underlying calculations and replication of those calculations. Its arrival opened up areas of domain knowledge to non-domain experts — double entry book keeping for instance — through the use of templates and abstraction. Users of spreadsheets are not experts in using spreadsheets necessarily. Instead they are domain experts using the spreadsheet as a tool. They know how to use the tool, and the tool abstracts away the domain knowledge needed to use the tool itself. We need the same for machine learning, for AI.

Right now there is a huge gap between the model zoos, and provided example code, built to do what are usually tinkertoy-level tasks and the ability to bring your own data. Until we have tooling and platforms that let us bring our own data, build our own models, and then use that model in production to do a task that solves a problem for us — without the domain knowledge around the model itself, but only that of the problem space we are working in — then we're not yet leveraging machine learning.

We need models and tooling that can be used day-to-day by engineers, not tooling that is aimed at researchers.