Bounds are the Future

I have quickly come to have the opinion that the software testing processes we use currently will become the most prominent component of all software development as large language models and whatever they turn into in the coming years continue to gain prominence. I also think abstractions of these processes will be applicable to a wide variety of areas beyond software. This isn't a new idea, but I do think the context of it is changing very quickly.

For context:

The concept of encapsulation is not a barebones necessity to the creation of a program, but the concept of procedure is (unless some abstraction is declarative in nature and hiding the state machine from you). I think encapsulation is possibly the most important piece to building larger systems, not just in the realm of software, but anywhere abstractions about how a system transforms information can be applied. Humans, especially programmers, understand encapsulation very well — I would go as far as to say that encapsulation in our own minds is the main reason that humans can understand anything in the first place.

The best way I can abstract software testing to all these situations is: Consider some sort of encapsulated object, some system that has some sort of function, input/output relationship, transforms the state of something in a certain way. Let's say a combustion engine. It takes in fuel and transforms said fuel into kinetic energy and exhaust. That is the interface of the function of the engine, the basic role it fills, parameter and return types of engine.run.

The thing is there is so much more information necessary to describe the nature of this system than the types of its input and outputs. Power to fuel ratios, rpm, mass flow rates, vibrational characteristics, and so much more. These are the actual return values and side effects of the function, not just the statement of their type. All these characteristics can be quantified in some fashion such that they can be related to each other.

Now finally getting back to testing. We as the designers of these encapsulated objects should be able to describe the way in which the different characteristics of a system should behave. In other words, it is our job to define the logical bounds inside which a system should exist. Those bounds, so long as they are sufficiently defined, are really the same sort of information as a comprehensive range of software tests. They are the rules which you want the characteristics of the system to obey as the designer.

I need to clarify that I don't think we are going to start writing code to check if our car's engine is working correctly — more power to you if you'd like to do so. I think what is more likely is that we will universally describe all sorts of objects not particularly in the way in which we want them to work, but in a much more general sense of how we want their behavior to be bounded. This is the most exact way to convey the absolute truths of a system, and this sort of information is exactly what the tools of the future will thrive on.

The more I write this out, the more it seems painfully obvious. If you tell a computer a very specific set of goals with rules and tell it to not come back until it gets it right, it will do exactly that — it does not have a choice (sans cosmic ray bit flips). The thing that has changed in the past several years is that for the first time ever we now actually possess systems such as LLMs that can return valid solutions to complex problems.

I hope this wasn't just 'duh, that's obvious' all the way through and I am not trying to downplay the key role that software testing has already had for decades. This was one of the realizations over the past few months that really drove home how powerful and practical these tools are, at least to me. My opinion is that strictly bounded generative design is the future of not only software design, but almost all work that involves any sort of design process.

Edit: I had earlier removed a part talking about how a lot of people are really short sighted about all of this and just have not figured out how to use the tools yet. I did not want to be negative when I did not need to be, but... then I saw this video.

I am not trying to completely discredit Low Level Learning — I quite like several videos of his, I just think that he is very wrong in this situation.

This was made like a hit piece. Someone who knows software as well as LLL should understand that if you throw in a horrid description of what you want you will also probably get something just as horrid. He expects security to magically pop into the program without ever even mentioning it in the prompt. This is insane to me — you set the model up to fail and laughed at it when it did.

It's also pretty crazy to me that he cannot imagine a world in which someone is able to define an abstraction of what security is and apply this information consistently to make generated programs secure.

I am not trying to advocate for the absolute worship of our AI overlords — there still needs to exist a process of human thought. I am saying that some of you are really starting to look like grumpy old people who will never get with the times. That's the exact sort of person I never want to be, both professionally and personally.

.... Also I know I'm not the only one who thinks this!