sigfpe: Not trusting the compiler seems reasonable. But, where does
it stop? Do you trust the microcode in the processor? (I remember
about 10-15 years ago the pentium processors had a bug in floating
point arithmetic). Do we need formal verification of everything from
the source code down to the logic gates (i.e. the compiler and
microcode)? Then, what about the logic gates? We know they work from
theoretical and experimental physics and decades of processors doing
what we expect them to. But we cant verify mathematically that they
work.

I’m sure the questions have already been tackled by mathematicians and computer scientists. I’m curious what the answers are.

(Sorry for the late reply; I’ve been traveling).

I need to say “this absolutely definitely works”.

I’d rephrase that as I’d like to say this absolutely works. If there’s a finite number of proofs we can fit in one page, at one point we’re guaranteed to spill into the second page. I’d like to hope that I’ll never reach my personal limits in my lifetime, but that ought to be less and more less likely with every generation.

After all, the idea of a true but unprovable statement is not exactly new — even if people are still in denial about it. And that is a much more devasting thing than the “true but only provable by messy proof” situation of the 4 color theorem and things like it. What is the logical requirement that the truth be neat and human-understandible?

Now, intuition is of course a good thing. One point being missed by Strogatz is that a major purpose “Wolframian” experiments is exactly to build intuition by exploring the space of what is possible, instead of sticking to the easy ones. And by exploring the larger space, you get a sense of where known knowledge fits.

Actually I think it’s different. I don’t trust the compiler (and this isn’t just abstract theoretical doubt, it’s based on actual experience). Binary executables do need to be checked. This checking takes the form of testing and using the principle of philosophical induction: once I’ve tested it enough, I’ll decide code is good enough to release. But I’ll never say “this absolutely definitely works”. In mathematics I need to say “this absolutely definitely works”.

Kidding aside, this “problem” has been around since the Hilbert basis theorem and I worry about it not one bit. Even if you look at a computer proof as an oracle, it isn’t as if you only get one question. As Robbie said, knowledge accretes around a fact, and eventually comprehension dawns. I feel that the people who worry about this have never actually done any mathematical reseach and felt the process happen in their head.

I was always under the impression that people still don’t really understand why the Monstrous Moonshine conjectures are true despite Borcherds’s proof. His proof works but it doesn’t seem to satisfy people’s intutions. So I think think there’s nothing really new about proof without understanding. In fact, ever since mathematicians first figured out how to do algebraic manipulation there have been proofs without understanding.

A fascinating, and easy-to-read, account of the conflicts between mathematicians and computer scientists in the development of formal verification can be found in this book by Donald MacKenzie (a sociologist of technology at the University of Edinburgh, Scotland): “Mechanizing Proof: Computing, Risk, and Trust” (MIT Press, 2004).