Dave, I Really Think I'm Entitled to an Answer to That Question (in Verilog)

Deep learning…check. Complex natural language interactions with a computer...check. 2001: A Space Odyssey reference...check. DAVE (Deriving Automatically Verilog from English), out of New York University, seems to check all the right boxes for a major advancement bordering on science fiction, but does it deliver as promised?

Specifications for digital systems are first written in natural language, then are translated into a Hardware Definition Language (HDL) with significant effort by engineers. DAVE seeks to take the difficulty out of defining hardware in FPGAs by automatically translating natural language directly into Verilog, a HDL.

DAVE is far from the first natural language programming effort, but they are taking a new approach by harnessing the well-known GPT-2 text generating model developed by OpenAI. In order to get Verilog results from the model, it had to be fine-tuned for the task. To this end, a dataset was created, consisting of tasks — English language descriptions of hardware specifications — and results—corresponding Verilog output. This dataset was used to train the DAVE model.

The researchers found that DAVE gave the correct response to validation tests in a very surprising 94.8% of all cases. The team at NYU was kind enough to set up a Google Colab notebook that will let anyone try the model out, so I did just that.

First, I ran an existing sample unchanged (I supply the task, DAVE returns the result):

TASK: Given 'a' and 'b', check if these are both true and return the result in 'c'.
RESULT:
assign c = a & b;

Nice. So far so good. Next, I made a few small changes that I would not expect to change the results:

TASK: For 'a' and 'b', check if these are both 1 and return the result in 'c'.
RESULT:
assign c = 1 & b;

Uh oh. Not looking so good anymore. After that I decided to try an extremely simple assignment statement, just to see how it can deal with language it may not have exactly seen before:

TASK: Set 'somevar' to 7.
RESULT:
assign somevar =!(d | f);

In the words of HAL 9000: “Just what do you think you're doing, Dave?”

By all appearances, DAVE will be added to the list of natural language programming tools that offered much promise, but were not able to deliver a workable real-world solution. To their credit, this is a very difficult problem to solve and the solution they developed will help move the needle of progress along. Exercises like this can be a reminder of exactly why we have special languages to program computers — we need concise and highly explicit means to program complex digital logic. Any natural language approach short of what we see on the USS Enterprise just forces us to learn yet another language that has to be formulated in specific ways and using a set of predefined terms to get results close to what we expect. And in that case, what exactly is the advantage over a high-level programming or HDL language?