How do you represent a pointer?

Recently, I’ve been working on a C compiler, and got to the point where I had to support pointers in C.

Most people think that a pointer is a number (with its length being your computer’s word size, normally 64 bits these days), but that doesn’t work. In the previous example above, I did pointer arithmetic on an int* start, with start++ (increment start by 1). However, since an int is 4 bytes, and since each increment prints the next number, this actually increments the pointer by 4 bytes. But if I had a char*, since char is 1 byte, incrementing a char increments the pointer by 1 byte.

On top of its size, a pointer also needs to store its type (since you can’t pass a pointer of a different type to a function that wants a specific pointer), but you also need to support casting.

A pointer can be cast to a number, or to another pointer type. If cast to a number, and then back to a pointer, what happens to its type information, or its size? Does it disappear? Some important questions to ponder.

Also, you’d want to have type-safety with pointers. It would be odd to take a pointer to an int and then change it to a pointer to another type, like an array. It makes sense to increment or decrement the pointer to an array, but not for a pointer to an int. If this is the case, then pointers should also hold their type.

On top of that, pointers can point to initialized or uninitialized memory. So that needs to be handled too – in C++, the end pointer of a collection always points to one past the end of a collection, which is generally uninitialized.

Also for optimization reasons, it’s also useful to know if pointers alias, or if a certain location in memory is only accessible by one pointer or multiple. In the is_a example above, if we know that the two pointers are different and cannot access the same location, then we can emit “return true” for the function. But that also requires keeping track of where it came from, which is called provenance. This information is important for optimizations, but also useful for error messages to the user – if the compiler knows where a pointer came from, and the user tries to use it incorrectly, it’ll have more information about how things went wrong.

Pointers are complicated. They’re also not just a number.

Unit Tests

The first way is the most obvious. Write out a test case and verify it makes sense.

We could write that 2 + 2 should be 4.

#[test]
fn unit() {
    assert_eq!(add(2, 2), 4);
}

But there are a few problems:

1. We have to know what inputs to provide the function and what output that would return before writing the test case.

Let’s say I misunderstand how addition works, and write this implementation:

fn add(a: u32, b: u32) -> u32 {
    a + b + 1
}

It’s pretty easy to also write my test program incorrectly:

#[test]
fn unit() {
    assert_eq!(add(2, 2), 5);
}

And then the unit test is worse than useless – it reassures me that I’m testing an input properly when I am not.

2. There are 4 billion choices for the first parameter to the function, and 4 billion more choices for the second. I can’t write all of these test cases by hand. Moreover, it’s hard to find interesting test cases that actually trigger a bug.

Snapshot Tests

Snapshot tests are like unit tests, but instead of having to write out the output, you verify that the output is correct. This is useful for cases where it’s a pain to write the output value, like for an AST in a compiler.

They make it easier to write tests, but they still share the pitfall with unit tests, where you still need to know what went wrong.

Mutation Tests

Mutation tests are a bit different from the previous ways of testing: they take your existing unit tests and change (mutate) the body of a function under test, and check if the tests still hold.

For example, for the add function, it might change the operator or the numbers to something else like so:

fn add(a: u32, b: u32) -> u32 {
    a - b
}

The test runner would then run all of your unit tests and note whether or not they fail. If they don’t all fail, then there’s some overlap between this new program and your program, and either the tests are not representative enough, or the function could be implemented incorrectly.

Either way, mutation tests are useful because even if you implement a function incorrectly, they have a chance of telling you that you’ve done something incorrectly.

But they do give false positives:

Imagine the add function is mutated to the following:

fn add(a: u32, b: u32) -> u32 {
    a * b
}

And the test case is:

#[test]
fn unit() {
    assert_eq!(add(2, 2), 4);
}

This test case passes with the unchanged add function and the newly changed one. Thus, this function will be flagged as potentially implemented incorrectly, even if it is correct.

Fuzzing

What happens if we want to try to generate some use cases? One way of doing this is by using some “interesting” cases and see what happens.

If we do the following:

add(u32::MAX, u32::MAX);

We’ll get an integer overflow, and the program will crash.

Fuzzing creates interesting test cases and then tests to see if the function(s) under test crash with some input. If it does, it tries to find the shortest input that does so.

However, it’s not foolproof:

Imagine we wrote add like so:

fn add(a: u32, b: u32) -> u32 {
    a.saturating_add(b)
}

This function should no longer crash, rendering our fuzz tests useless.

Fuzzing is good for testing lots of use cases, but we lose a nice property – we’re just trying to make the function crash, and no longer verifying its output.

Property Tests

What if we know some things about the function we’re trying to test?

If we add two numbers where both numbers are >= 0, we know that the number has to be at least as large as the maximum of both numbers.

We can formalize this:

#[test]
fn property() {
    let X: u32 = random_number();
    let Y: u32 = random_number();
    let result = add(X, Y);
    assert!(result >= X.max(Y));
}

We have now found some useful property to test, so instead of just testing for a crash, we also have some guarantee that the function is correct.

That being said, we could still fool the property by writing this:

fn add(a: u32, b: u32) -> u32 {
    a.max(b) + 1
}

Which would pass our previous property test.

the REPL

The REPL is a prompt that interprets results on the fly and returns them to the user. Almost all interpreted languages have one, and they call them a “REPL”, which is short for Read Evaluate Print Loop.

In lisp, it’s four functions:

(loop (print (eval (read))))

1. You read input from the user
2. You evaluate it
3. You print the result
4. You repeat infinitely

In lisp it’s easy to implement like this because it’s easy to evaluate any lisp program you write (lisp programs are interpreted the same way they’re typed, so evaluating is not that hard).

Simply putting together these four functions in this way allows for a new way of experimenting with programming – you can write code and get immediate feedback, instead of having to write code, compile it, and run it. Most people start off with learning interpreted languages, and these four functions are responsible for getting many people into programming, myself included.

The Universal Server

Joe Armstrong wrote about his favorite program, The Universal Server.

It’s simple:

universal_server() ->
    receive
       {become, F} ->
           F()
    end.

5 lines of code define a process that can take any arbitrary work and becomes that function.

He defines a factorial function, which calculates the factorial of an integer it receives:

factorial_server() ->
    receive
       {From, N} ->
           From ! factorial(N),
           factorial_server()
    end.


factorial(0) -> 1;
factorial(N) -> N * factorial(N-1).

And then you can write a function that spawns it and run it.

test() ->
    Pid = spawn(fun universal_server/0),
    Pid ! {become, fun factorial_server/0},
    Pid ! {self(), 50},
    receive
        X -> X
    end.

In 5 lines, this defines a universal protocol for servers – they receive input and return a message. Any server that follows that rule can be spawned by this universal server. Pretty cool. If you have a server running a version right now and you want it to run another version? Go write the code for a new server, and send it over.

A Metacircular evaluator

Another lisp program? A Metacircular evaluator is an evaluator that can evaluate programs written in itself.

This is from Structure and Interpretation of Computer Programs:

(define (eval exp env)
  (cond ((self-evaluating? exp) exp)
        ((variable? exp) (lookup-variable-value exp env))
        ((quoted? exp) (text-of-quotation exp))
        ((assignment? exp) (eval-assignment exp env))
        ((definition? exp) (eval-definition exp env))
        ((if? exp) (eval-if exp env))
        ((lambda? exp)
         (make-procedure (lambda-parameters exp)
                         (lambda-body exp)
                         env))
        ((begin? exp)
         (eval-sequence (begin-actions exp) env))
        ((cond? exp) (eval (cond->if exp) env))
        ((application? exp)
         (apply (eval (operator exp) env)
                (list-of-values (operands exp) env)))
        (else
         (error "Unknown expression type -- EVAL" exp))))

There’s more to the program, but this is the core of it.

All lisp programs can be defined by two things: their body, the actual program and an environment, which is the bindings required to make the language work. For example, if you have a program:

(print x)

You need some definition for two things, x and print. Normally you’d define x in the course of your program, but you probably wouldn’t define the print function of your language. Thus, you can divide this program into two pieces, its environment (what it needs to run) and the textual definition of the program itself.

{ print: "some implementation here" } ; The "environment"
(print x) ; The program

That’s all you need to evaluate your language, a set of variables, and the program itself. If you want to write a modern programming language, you need to support scopes:

let mut x = 10;
{
    x = 15;
}

All you need to support that is nested hashmaps:

{ x: 10, { x: 15 }}

If you’re in a nested scope, choose that value instead of the higher level value. A language is itself and its environment.

A Simple Regular Expression Engine

This comes from The Practice of Programming by Rob Pike and Brian Kernighan, a simple regular expression matcher.

/* match: search for regexp anywhere in text */
int match(char *regexp, char *text)
{
    if (regexp[0] == '^')
        return matchhere(regexp+1, text);
    do {    /* must look even if string is empty */
        if (matchhere(regexp, text))
            return 1;
    } while (*text++ != '\0');
    return 0;
}

/* matchhere: search for regexp at beginning of text */
int matchhere(char *regexp, char *text)
{
    if (regexp[0] == '\0')
        return 1;
    if (regexp[1] == '*')
        return matchstar(regexp[0], regexp+2, text);
    if (regexp[0] == '$' && regexp[1] == '\0')
        return *text == '\0';
    if (*text!='\0' && (regexp[0]=='.' || regexp[0]==*text))
        return matchhere(regexp+1, text+1);
    return 0;
}

/* matchstar: search for c*regexp at beginning of text */
int matchstar(int c, char *regexp, char *text)
{
    do {    /* a * matches zero or more instances */
        if (matchhere(regexp, text))
            return 1;
    } while (*text != '\0' && (*text++ == c || c == '.'));
    return 0;
}

The driver function, match, checks if a string matches a given regular expression. If the regex starts with ^, it must only match from the beginning, so it checks that with matchhere. If not, the regex could match from any position in the string, so it checks for the regexes’ match from every position in the string.

The helper function, matchhere, checks if the regex matches starting from the first position in the string. It handles * for repetition, and $ for ending.

matchstar finds zero or more matches of a regex in the string. This is done with a do while loop because there are 0 or more occurrences possible.

All in all, three functions that clearly explain what they’re doing and compose beautifully to tackle the problem at hand. Regexes are extremely useful in programming, and to see most of them boiled down to just these three functions makes this program beautiful, in my eyes.

The Rest?

There are many beautiful programs – I just chose a few from my memory but I’m sure you have plenty that spring to mind – if so, write about them!

Computability and Consciousness

There are a lot of problems that are hard for computers to solve but are easy for us. For example, driving in a closed course, walking, and handwriting detection all took many years for computers to handle tolerably (sometime within the last 20 years). Clearly we’re not like computers – it doesn’t take billions of operations in our mind to walk, see, or read or write.

Also computers can’t solve all problems. Computers have many light switches that can be used to signal two states (on or off). In terms of a computer, that means that a computer program can take in as input some finite number of 0s and 1s and return as a response some finite number of 0s and 1s. Any problem that fits within that realm is solvable. In more math terms, that means that computers can only solve problems that fit within the countably infinite set – any problem that lies on the number line from 0 to infinity. The set of countably infinite numbers is enumerable: if you add 1 to 0 you get 1, if you add 1 to 1, you get 2, and so on. You can keep going until you hit infinity.

But the actual number of questions that we can pose is more than that. If you ask the question of how many numbers there are between 0 and 1, and prove that a computer could solve them all, you would try to prove that the numbers between 0 and 1 are enumerable, and thus, countably infinite. But if you say there are 3 steps, 0, 0.5, 1, you can further subdivide those to 0, 0.25, 0.5, 0.75, 1. And so on and so forth infinitely. Thus we can’t actually enumerate from 0 to 1, because any smaller gap from 0 to the number proposed has a sea of infinity between it, even if the number was infinitely small. Thus we can’t even count the numbers between 0 and 1. Thus, computers cannot solve all problems that we care about, since there are an uncountably infinite number of them, and computers can only count to infinity.

Since we know that computers can’t solve all problems, the next question is to ask if computers can solve some other questions we care about, like consciousness. Thinking about evolution, if the Earth and all life on it could be modeled as some large state machine, where every organism would have some set of states it could call every tick of life and all life on earth would be a collection of these. If life really acted in this way, it would be possible to have life on earth be simulated by a computer. But there’s an assumption that we could model life with some state machine, which is something computable. But is what we do captured by a state machine? The actions that we perform could possibly be computable, but the motivations may not be. And if so, consciousness would be fundamentally different actions in a state machine.

A toy OS

The bridge between hardware and software is the operating system. The code that protects you from yourself. I started off by reading through Operating Systems: Three Easy Pieces, and implementing most of the projects throughout the book. The companion code and problems outline some basic programs to write, but you’re generally benchmarking your host OS to learn more about it. That wasn’t enough for me.

I moved onto reading Lion’s commentary on Unix since it gave commentary on an actual system (Unix V6), but I found the code hard to digest since it was using C from the 70s. Finally, I stumbled upon Octox which is an implementation of xv6-riscv in rust, and you can run the entire system with a cargo run command. I took this code, tweaked it a bunch, and learned a lot about operating systems in the process.

Operating systems have the tricky job of taking a scarce resource (the hardware of the computer) and doling them out to multiple users at the same time, while also not allowing the users to step on each other’s toes. The operating system also has to provide a stable interface that doesn’t change, since any user visible changes require changes in a lot of code that rely upon the OS, and I learned about providing a strict interface boundary. Data in, data out. Don’t break your promises. For a normal monolithic kernel, it gets very hard to keep this promise, because so much of the kernel is hidden from the user. For a microkernel, since most of the functionality of the OS is owned by various services in user and kernel space, it’s not as bad to break backwards compatibility, since that only affects the code that requires that particular service, rather than the whole world.

Boundaries are important for all of us.

A Compiler

I followed Crafting Interpreters to implement two interpreters, the first part in Rust and the second part in C. A compiler really is just another program – you give it a program, some knobs to tweak, and you get back an executable program.

Writing a compiler was useful in more areas than language design – I improved at parsing various data structures and stopped trying to hack a duct tape parser with regex, but started composing parsers off of smaller parts to make a better parser. My knowledge in interpreting code also helped for projects at work that required constructing languages that supported powerful features like macros, but I also learned that testing is invaluable to a program as complicated as a compiler.

You have to generate tests for any arbitrarily valid grammar and the compiler always needs to give the correct result. The way that most compiler writers do that these days? Fuzzing. I added it to my toolbox and started fuzzing my own projects to find bugs. I found a lot of them. Now I have a lot more bugs to fix.

Overall, the world became easier to parse.

A Critique of Verbal Behavior

• A Critique of Verbal Behavior

Psychologists have long wondered how language is acquired by humans. It seems to defy all of our traditional understanding – people who have been isolated at birth long enough seem not to “acquire” language in the complex ways that socialized humans do. However, children who grow up in otherwise linguistically lacking environments can still command language in a complex way, even if other people around them do not. Thus, language acquisition is probably not all acquired by the environment, or determined genetically since both factors intertwine to determine our control of language.

In the first half of the 1900s, Behaviorism became a popular way of explaining cognition scientifically – compared to previous movements which were untestable, like Psychoanalytics, which focuses on the unconscious self, Behaviorists sought to make more of psychology directly testable. The Behaviorists turned their eye to language acquisition, and a prominent figure at the time, Skinner, wrote the book “Verbal Behavior”, and a then young Chomsky wrote a rebuttal to it.

The crux of the rebuttal takes Skinner’s point that external stimuli are of “overwhelming importance” to language acquisition, since behaviorists believe that reinforcement is paramount in enforcing behaviors in humans, and counters it. Chomsky explains that the rules of grammar are far too complicated to be learned from simple interactions with people, and thus, children who utter sentences must have some innate knowledge about the rules of grammar and stimulus helps them learn new words and concepts but not the underlying grammar. Chomsky says “This task [language] is accomplished in an astonishingly short time, to a large extent independently of intelligence, and in a comparable way by all children…”.

He concludes by saying: “The fact that all normal children acquire essentially comparable grammars of great complexity with remarkable rapidity suggests that human beings are somehow specially designed to do this, with data-handling or ‘hypothesis-formulating’ ability of unknown character and complexity.”

While it’s not true that Skinner believed the language acquisition was purely via stimulus (and so Skinner and Chomsky agreed that their must be something natural to language acquisition), this paper was widely influential in psychology and computation, and sparked a whole set of debates on the computability of language and how we acquire language.

A Theory of Justice

• A Theory of Justice

A Theory of Justice is a political philosophy book by John Rawls from 1971, who famously talks about a thought experiment called “the original position” also referred to as the “veil of ignorance”.

Imagine that you’re a soul, and some arbiter gives you two choices for the society you could live in. Since you’re just a soul (thus you don’t know anything about your physical or mental capabilities or the structure of society), what rules should you employ to make society work out for you?

Rawls argues that with this starting point, the only way to create a society is to create one that is “fair to all”, since if you choose to privilege an arbitrary group in society, there is a chance you will not end up in that group, and thus, be disenfranchised. So, you’ll lean towards a fair society. Thus, Rawls says, the original position informs a set of principles that create a fair society for all, and one that everyone would agree upon.

The principles are:

1. Each person has a set of basic liberties, compatible with the rest of society.
2. Everyone has an equal opportunity to pursue any office or position
3. Socioeconomic inequalities are set to give the greatest benefit to the worst off.

A Theory of Justice does well in explaining a simple thought experiment and using it as the basis for some axioms to build society and then discuss how said society would function, and sparked a lot of discussion in a lot of different fields.

They even made a musical out of it.

The End of History

• The End of History

The End of History is a book by Francis Fukuyama in 1992, a political scientist who makes a bold claim: with the fall of the Soviet Union, “the end of history” has come, that Western liberal democracy is the final form of human government, and that it will remain that way, forever.

Fukuyama draws upon Hegel and Marx to explain that history is a progression of different eras with different dominant political frameworks: mercantilism, capitalism, communism, liberal democracy all came one after another and coincided with each other.

This is probably the most “position” writing on this list. I chose this book (it’s also a paper) because I completely disagree with it – in 2024, things look a lot less rosy than they did in 1992. Russia has re-emerged as a power, North Korea gained weapons of mass destruction, China is increasing pressure on Taiwan and Hong Kong, and the political situation in the middle east has completely crumbled after the war on terror.

Fukuyama also intertwines western liberal democracy with economics – he picks liberal democracy for the political side and capitalism for the economic side. While there has been some turn against liberal democracy and towards autarky, a lot of people changed their minds about the efficacy of capitalism after the Great Financial Crash. Capitalism is on top for now and so is liberal democracy, but will it be that way forever, as Fukuyama so bravely declares? I think not.

Even if I’m wrong, forever is a long time so I’ll just keep stalling until I’m dead or right.

Prospect Theory

• Prospect Theory

Two psychologists write a math paper and win a Nobel prize in economics. Want any more argument to study widely?

This paper written by Tversky and Kahneman starts out by discussing the way most economists thought of utility at the time, expected utility theory, discusses its shortfalls, and then outlines a new theory, prospect theory.

In economics, you choose a game for participants to play, and all actors have certain choices to make, and all choices grant a certain amount of utility that can be reduced to a single number. All agents in this game should make all choices rationally, given the amount of information they have available. Rational in this case means agents should maximize utility. This is of course mathematically optimal, but as the paper points out, we aren’t perfectly rational, and so this assumption is flawed.

Prospect theory notes experimentally that we view losses and gains differently, and are not always risk-neutral. If you’re given a choice between losing money and gaining money, people dislike losing money a lot more than gaining money. We tend to remember the bad times more than the good times. That makes sense – we’re all attracted like moths to a flame to bad news or gossip. Why shouldn’t it be the case for us as economic actors?

This paper was widely influential, and economics in particular has seen a boon in experimental research – every year we find more cognitive biases in the ways we interact in economic settings, and learn more about human psychology through these findings.

Sometimes challenging the mainstream has good prospects.

The Unreasonable Effectiveness of Mathematics in the Natural Sciences

• The Unreasonable Effectiveness of Mathematics in the Natural Sciences

In 1960, a mathematician named Eugene Wigner published this paper, where he notes that mathematics is extremely useful in modeling the real world.

In 2024, I think you could change the title of this piece to “The Unreasonable Effectiveness of Mathematics”, full stop. Even as a international relations major taking mainly economics and political science classes, I had a lot of math and statistics classes to take. You just can’t read papers in those fields without math. Every economics paper has either statistics or math in it. Almost all political science papers require a solid foundation in statistics to even read.

Mathematics has really won out over here, and it took over the natural sciences a long time ago.

It’s kind of funny to read this paper with Wigner giving such basic examples like gravity or using linear algebra in physics, when we live in a world now with AI models using linear algebra to create models that can respond in a chatbot like fashion.

Why Most Published Research Findings are False

• Why Most Published Research Findings are False

In 2005, a physician named John Ioannidis wrote this highly talked about paper that uses some back of the envelope numbers to come to the conclusion that at least 50% of papers (in medicine) might be false positives, and thus, false.

Variables in the study that have an effect return significant results, but there is some acceptance of false-positives, determined by the p-value the researcher chooses. Normally, a false-positive rate of 5%, or less than 0.05 is considered acceptable, so there’s a 5% chance that something that did not have an effect shows up as having an effect. So the p-value metric is supposed to make the upper bound of false-positives 5%.

Unfortunately, it’s not quite so rosy. Since journals tend to only publish significant results (since scientists read papers to learn what has an effect, instead of combing through articles to find out what had no effect). For an example, let’s say there is a researcher, A, who finds that some variable B, is significant in some experiment E. They rush to publish a result and are published. Meanwhile, 19 other researches try the same thing and find no effect, which follows since our p-value is 0.05. However, even though they were 19 chances to disprove the efficacy of changing variable B in the experiment, those weren’t published, only the false positive.

Whether or not “most” published research findings are false is up for debate, but Ioannidis points out that experiments in medicine use experiments with smaller sizes in clinical trials to find medicine that works for patients. But this is doomed, since the smaller the sample, the less power of the study and the more likely it is to find a false-positive, which would do more harm than good to society at large.

The paper gives 6 corollaries for ways to find research that’s more likely to be wrong:

1. The smaller the studies conducted in a scientific field, the less likely findings are to be true.
2. The smaller the effect size, the less likely findings are to be true.
3. The greater the number and lesser the selection of tested relationships in a scientific field, the less likely findings are to be true.
4. The greater the flexibility in designs, definitions, outcomes, and analytical modes, the less likely findings are to be true.
5. The greater the financial and other interests and prejudices in a scientific field, the less likely findings are to be true.
6. The hotter a field, the less likely findings are to be true.

This paper goes against the grain by questioning how useful currently published research is, and gives a reason for why the current replication crisis is ongoing. Researchers pursue fairly complicated models which give them a large degree of freedom to find significant results, regardless of if they’re significant.

C is not a Low-level Language

• C is not a Low-level Language

David Chisnall, a computer architect wrote this paper in 2018 to discuss why C is not a low-level language, and all the troubles that causes, motivated by the recent exploitations in instruction level parallelism, spectre and meltdown.

Low level languages map closely to the underlying hardware, in contrast to higher-level languages, which are more abstract and map moreso towards the way a programmer might think about a problem. C is considered a low-level language, which makes it a fit for writing performant systems, like operating systems, compilers, and databases. However, C is a language that was written for the PDP-11, a 16-bit computer from the 1960s. C defines a virtual machine where instructions execute one by one in serial order. This made sense in when C was written, since computers actually executed instructions one by one.

In 2024, that hasn’t been true for decades. Compilers are free to reorder code in order to speed up execution, so C code that you write might be executed in a completely different order than the one that you specify. Even worse? Many instructions are executed on different processors at the same time (the author notes that a modern intel processor has ~180 instructions in flight at the same time), and states that this quest for speed without embedding this into the language caused these exploitations and numerous more throughout the years.

As well, memory is no longer sequential (as the C virtual machine believes) and computers have layers of caches, many more registers, especially ones which can process the same data in parallel, and padding is required on data structures to avoid large performance penalties in data fetching. None of these features are built into C, because C predates all of these requirements, and it would be nigh impossible to add these restrictions into C, and thus, we should look towards newer languages to code in, since the future will likely be on more heterogeneous computing (CPUs, GPUs, FPGAs, mobile, cloud, desktop), and the hardware will have lots of extra functionality to tap into that C has a hard time using.

I like this paper because it challenges common gospel and gives reasons for why our mental model of the code we write is so important. Most programmers don’t really care about the virtual machine their code targets, but it has a lot of implications in practice. Code reordering might hoist a pointer dereference before a null check, thus crashing a program that should never crash. A small change in some code might cause a program to run 10x slower due to an increase in cache misses. There are many more problems that occur in practice, but neither the programmer nor the compiler writer really have a way around it, since even the processor can reorder instructions due to the need for speed.

We’re all doomed.

Email

I’m going to start with email since it has capabilities to create a community (with mailing lists). Email between person to person isn’t that different from sending a letter, but creating carbon copies allow it to create a community: you can send the same mail to many people and they can respond in a thread.

Needless to say, email became very popular, but people don’t use it as a social community much: some OSS software uses it for development (Linux, Postgres) and some folks use it to arrange meetings or discussions. Although using email is very convenient, many people hate it:

Donald Knuth famously said this about email in 1990:

I have been a happy man ever since January 1, 1990, when I no longer had an email address. I’d used email since about 1975, and it seems to me that 15 years of email is plenty for one lifetime.

Email is a wonderful thing for people whose role in life is to be on top of things. But not for me; my role is to be on the bottom of things. What I do takes long hours of studying and uninterruptible concentration. I try to learn certain areas of computer science exhaustively; then I try to digest that knowledge into a form that is accessible to people who don’t have time for such study.

On the other hand, I need to communicate with thousands of people all over the world as I write my books. I also want to be responsive to the people who read those books and have questions or comments. My goal is to do this communication efficiently, in batch mode — like, one day every six months. So if you want to write to me about any topic, please use good ol’ snail mail…

Many people dream of “inbox zero”, where they have no more emails to read or respond to. Email suffers the same problem almost everything that reduces friction does: it causes problems of abundance.

In the 60s and 70s in America, as telephones became more popular, some people racked up huge phone bills to talk to their friends and keep up with the gossip in their friend group. The telephone, a tool for increased communication, did the opposite: it isolated people from their community.

Email is a rudimentary social network: it wasn’t very popular until the 90s and people didn’t use its social capabilities much. Onto the next era.

Forums

With the World Wide Web becoming popular in the late 90s, people wanted to find similarly minded people to discuss things with: so they created specialized websites, called forums for this purpose. Someone (or a team of people) manage the site, and everyone discusses on or off-topic things. Moderation was loose and things were hectic, but they worked. Sometimes your favorite forum would disappear, or the mods would make it invite-only, but anybody was free to spin up their own new forum.

Forums had threads as a first-class feature: you could make a thread and users could comment on it to discuss. Some forums had upvotes and downvotes natively, expressing approval or disapproval of posts on the forum.

Reddit saw the use of forums and decided to create a meta-forum: a website where users could create a forum about any topic, moderate it however they wished, and so users wouldn’t have to remember all the forums they used in the past. Reddit would handle the discovery: they would aggregate every subreddit like google did for the web, and they had an interesting way of arranging content: highly upvoted posts would float to the top, so users could read “interesting” content quickly, keeping them engaged.

It was a smash hit. By the early 2010s, many communities moved to Reddit, and over 10 years later, Reddit is now looking to IPO.

But can you really turn a forum run by users into a profitable business?

Reddit’s business model is to own the website reddit.com, and own searching, but there’s no reason that anybody has to keep using the site. Reddit is the overlord of the fiefdom, but the vassals don’t get paid taxes.

As well, there’s been a turn against “engaging” content. Our attention spans are finite, and engaging content in surplus becomes trite.

The upvote has lost, and the karma police will collect their dues. I predict a messy IPO.

The resurgence of scarcity

For most normal economic goods, suppliers prefer scarcity to increase profit margins, but they prefer surplus to maximize profits. If one supplier can corner the market, there’s a monopoly price that introduces scarcity to strictly maximize profits. If there are enough suppliers, all suppliers tend towards surplus to maximize their payoffs (which is zero).

On the other side, users tend to prefer goods in surplus since it lowers the cost of their goods and increases quality and choice.

But the economics of surplus don’t make sense for everything.

When people left the communist bloc to see the western world, surplus was all they saw. Grocery stores were stocked to the brim where they stores would waste perfectly good food to entice customers with only the freshest food. There weren’t lines to order food or shortages.

But while we don’t starve, we have another problem: in America, the obesity rate is nearly 45%. In 1960, it was 13%. Our life expectancy hasn’t gotten much better either, from about 70 years to 76 years, with most of the gains coming from earlier on in life.

With attention too, people are rethinking their relationship with technology. Communities without internet have popped up, with eager urbanites moving into those rural areas. Others have decided to restrict the time that they use technology to unwind. Basically, self-imposed scarcity, with promising results, like improving mental health and increasing the meaning in their life.

Most of all, with the ever increasing tide of technology and capitalism, people are re-learned that the incentives of suppliers and consumers aren’t always aligned. Capitalized social media platforms like Facebook, Reddit, and Twitter need your attention to keep generating profits.

Why attention matters

As a disgruntled 20-something, I used social media as a crutch to stop myself from thinking the woes in my life. I wasn’t going anywhere, I was working increasingly long hours doing hard labor jobs that I knew I couldn’t maintain, being paid next to nothing to maintain my pitiful existence. Where I worked, I knew of people who died by alcoholism, suicide, and were maimed in the workplace. Every extra day I worked in these conditions, I increased my chances of being one of those people, long forgotten.

I talked to a coworker who had been working similar jobs for 20 years, being paid minimum wage just like myself. He couldn’t afford to raise his family and was in heavy debt. His health was terrible from the alcohol and drug consumption needed to keep up with the 80 hour work weeks. He had professed a love of social media as a tool to stop himself from thinking about his miserable experience. He told me all this to say that time and focus was the resource he misused the most in his life. He was a 40 something guy, but I was in my 20s. I had time. I took his lesson to heart and began to take care of myself and preserve my attention. I learned a desirable skill to leave the never-ending cycle of low skill, low paid work and started to turn my life around.

As Socrates said, “The unexamined life is not worth living”. I wish I learned that sooner.

The Unix File API

If you write a file in Unix, you just need a few system calls. open, read, write, seek, and close. That’s all you need. The caller doesn’t need to know about the perils of writing a file, or where the bytes go, they create a file, read and write data from it, and that’s all they need to do.

None of the implementation details bleed out to the caller: you don’t have to worry about what kind of device you’re writing to, whether it be an SSD, HDD, tape, or even a piece of paper. The operating system figures that out for you. If you read or write, the operating system moves the needle on tape, the pen on paper, or talks to the SSD’s block controller.

The interface itself doesn’t care about its backing store, since it assumes the caller doesn’t. If the caller notices that operations are slow, they might start looking at the storage media, but not before admiring how nice the interface is.

Writing to persistent storage is a gargantuan task: Linux file system drivers are millions of lines of dense code that works behind the scenes, and the user doesn’t have to know a thing about it.

Other operating systems decided to create system calls that were tied to the physical media. That meant that programmers would have to change their code if their storage media changed, which meant the code was brittle and hard to port. The Unix file API sidesteps all that. The user doesn’t have to change a thing no matter the backing media.

The API is also easy to optimize – if your storage media supports concurrent reading and writing, read and write can fire off as many workers as needed to make your request as fast as possible, or read out of order on disk as long as the caller has their requirements met, which is the bytes arrive in the order they are on disk.

A true paragon of good design.

UTF-8

In the 1990s, many people speaking many different languages started to use the internet. Many languages have more than 256 symbols, so they don’t fit in a byte, and many languages had created their own custom encodings that made browsing the web a struggle – you’d need to download dozens of encodings for all these different languages to just browse the internet.

That was until UTF-8 became dominant. Now, UTF-8 is the encoding of the web, but it wasn’t always that way – even 15 years ago the Japanese web had many pages using Shift-JIS.

The ingenuity of UTF-8 lies in its interoperability with ASCII with no overhead, and the ability to encode either 1, 2, 3, or 4-byte characters by only using 1, 5, 8, or 11 bits to signal the remaining length of each character. With 21 bits still remaining for a 4-byte character, UTF-8 can support 2^21 bytes, or a little over 2 million characters.

That should be enough for a while.

TCP

TCP was introduced 50 years ago, and it still holds up as good design: it provides one guarantee, like re-requesting lost packets and providing in-order transmission of packets between two programs. TCP also notifies the client of an error if packets couldn’t be received from the sender. These strengths, in exchange for less throughput and higher latency make it easy for people to implement higher level protocols on top of TCP, like HTTP. As well, its transmission control algorithms were invaluable in helping the internet scale to its current usage.

TCP shows the importance of abstraction in design – IP, which TCP is built on top of, may lose packets in transmission due to network congestion, load balancing, or other hardware problems in the physical or link layers. Packets could also be corrupted, or come in any order. TCP handles error handling and, with little overhead, turns those unwiedly packets into a set of ordered packets.

If you’re skeptical of the guarantees TCP provides, try writing an application without it, using just UDP – it’s hard. You have to build a robust in-order network stack at the system level so the application layer can consume bytes in-order, or write your application to tolerate out of order transmissions, dropped packets, and unbounded timeouts.

TCP takes care of that.

Message Passing

If Smalltalk won, this section would be called “Object-Oriented Languages”. Alas, Smalltalk did not win, and everyone thinks OOP means Java style interfaces and inheritance. Sigh. Erlang might be the only true Object Oriented language still around.

Message Passing is what it sounds like. You pass a message to an actor, who receives the message and acts accordingly based on the message. In Erlang, objects are processes that can only receive and send messages and cannot share any state except by message passing. This metaphor is very powerful up and down the stack.

A computer nowadays has many cores (my computer has 16), and they all have their own memory and have to read and write to both core-local and shared caches in order to receive and broadcast their state (the results of their computations). Rather than have each core have access to each other core (which would get messy quickly), cores can pass messages to each other, allowing them to coordinate (e.g. invalidate a value because it has become stale, or publish a new value since it’s been updated).

If I choose to read a webpage on my browser, I receive some HTML, CSS, and JS over the wire and my browser renders it. If I pass some extra parameters along with my request, like query parameters or a body, I can customize the response I get. I can also send some data to the computer, which it can respond to in kind by sending a POST request. I have no access to the internals of the server, and the server has no access to my computer’s internals. Message passing becomes the boundary. As long as both sides make backwards compatible changes (like still responding to the same messages the same way), either side can make any arbitrary changes.

When we converse with other people, we pass messages through spoken word to each other to understand each other’s internal state (emotions).We also use the written word to do the same thing. Without message passing, we would have to do brain surgery to understand each other’s thoughts, and even then, since we’re not translate physical signals from the brain into coherent thoughts, this wouldn’t work either.

When I try to design something, I always prefer stateless message passing. If two actors can get away with message passing, even if it makes things a little harder in the short-term, it’s almost always worth it in the long-term. The less I need to know, the better.

The resurgence of Microkernels

Linux developers got tired of always having to go into the kernel to make a change. If I want to improve a scheduler, I shouldn’t have to know about anything other than tasks. Fundamentally, a scheduler should see a list of tasks that want to run, some information about each task, pick one, and run it for some time before restarting. That’s it. I should be able to judge whether or not my change was good without thinking about the rest of the system. If I want to debug some code that runs in the kernel, I should have a way to do so that doesn’t involve peeking into the kernel. What was the solution? Creating user space equivalents of critical kernel space facilities, like the following:

• eBPF (extended Berkeley Packet Filter) was created to allow users to run sandboxed programs in the kernel without having to load a module or recompile the kernel. It became especially popular in the networking space, since observability and performance of networks are in short supply in linux.

• FUSE (File system in userspace) was created as an interface to allow non-privileged users to write and run a file system in userspace that calls into kernel space.

• Sched_ext is a kernel subsystem that allows writing schedulers in userspace.

Lots of users may want to customize parts of their OS, tweaking subsystems to optimize their use cases. Computing has become more heterogeneous, and users have more diverse needs than ever, since they use a plethora of devices like embedded devices, phones, tablets, laptops, desktops, cloud daily. It’s impossible for a single monolith to make all the right choices for every user. Instead, imagine a distribution of Linux that really cared about power consumption, so it decided to pick power efficient subsystems, or a hyperscaler distribution that optimized for virtualization. It’s hard to do with current linux, but with the changes coming in the future to break up the monolithic kernels into distinct services, maybe the microkernel future will be coming soon.

Document your thoughts

I don’t frequently comment my code, mainly from my years working in corporate. I brought that habit over to my personal projects, because I believe the code itself should be written in a way that intent is clear. What’s not clear, however, is why you chose to do something one way over any other way, and the trade-offs that entailed. For that, I’ll write an architecture file that explains some of the decisions, goals, and explicit non-goals. This document is invaluable when coming back to a project.

As well, tests are great for jogging the mind. If I write a library, I find myself wanting to run it to see what it did and where I left off. Tests let me do that – I can read the tests and see what features/bugs the library has. I’ll write some failing tests and a comment that explains the feature or bug briefly, and leave it failing. When I come back to it, I’ll try to fix it up.

On top of traditional testing, I’ve recently fallen in love with fuzzing my code. Fuzzing involves generating random structured data and then calling a function in your program, and trying to crash (or otherwise trigger a bad state). I recently wrote a bitcask database in Rust, and when I thought it was satisfactory, I wrote up a few unit tests and I started fuzzing my app. In less than a second it found 3 bugs, checking code paths that would be difficult to do by hand. These test cases are then easily transferable to your normal unit testing suite, which can act like a regression suite.

Managing State

The second part I find difficult about writing a project is managing state – If I start out with a small core and adding features to it in an unstructured way, I tend to end up with a jumbled mess with state everywhere. The frontend world took a nod from the functional world and decided to keep state in stores, where state and state management routines are housed, and the rest of the application has to call into the stores to get or set specific values. These values are also purely functional, which allows for time-travel debugging, where you can go back in time to a point where the app was at a certain state. Gary Bernhardt took this further and espoused a philosophy called “functional core, imperative shell”. Any part of the application that can be pure, or only rely on its arguments is easy to test and hard to get wrong. As much of as possible of the program should be written in this style. Any part of the program that is required to deal with state, or has side effects (network requests, Disk I/O) should be shunted around the functional core, and use integration tests. This is kind of like MVC, where the controllers and the model are the imperative shell, making network requests or doing file I/O, and the view shuffles data between both of them.

This breaks your application up into modules based on how much they interact with state. This is a big win for understanding your code after a break – the duties of each module is clearer, and testing duties are also clearer, since you don’t have to stub any part of the system out, just integration test + fuzz the boundaries of the code and fuzz + unit test the functional core.

Having fun

Normally I dislike structure, instead doing things that I enjoy without strictly adhering to rules. What I found interesting was that having just the right amount of structure increased the amount of enjoyment I got from coding and other projects. I didn’t have to spend as much time reimmersing myself in the project, which I found boring and a blocker to coming back to hard projects. It also made me a better programmer, since testing was easier and I could spot out troublesome patterns of code and became more efficient at testing. Also, I started tackling harder projects because I could build things up incrementally faster and not hit the cognitive wall where making a change was higher than the amount of enjoyment I would get from said change.

The right amount of structure is like the right amount of spice. It makes everything taste better.

Write every day