Vegard Stikbakke

Detecting file changes on macOS with kqueue

Tue, 24 Mar 2026 00:00:00 +0000

A while ago I wrote a small file watcher in Go for my own use with an accompanying blog post. I needed a tool that I could just plop in front of the command I was running as part of my iteration loop. I use it for recompiling C files when I modify them,

reload gcc main.c -o main && ./main

and for rebuilding and reloading my static site on file changes.

reload make

It has two modes.

If one or more explicit files are mentioned in the command, it will watch those files.
If it does not find any filenames, it will watch all files in the working directory.

The only thing reload needs to know is whether any file it is watching has changed. If a file has changed, it reruns the command.

It works great! But I copped out on the part that was the most unfamiliar to me, namely detecting file changes. I used the fsnotify package which is a nice cross-platform Go library for this. It supports macOS as well as Linux but since it's for my own use, I don't care about Linux support. More importantly, I wanted to understand what fsnotify did under the hood.

On macOS it uses the kqueue event notification interface. Let's take a look at how this works, writing some C code to test it, and then finally implement it in the reload Go program.

#`kqueue` data structures

The kqueue() function call creates a new kernel event queue (a kqueue) and returns a file descriptor. We register and wait for system events using the kevent() function call, which uses the kevent data structure. This has five fields that we need to care about.

ident The source of the event. In this case, it will be a file descriptor for the file we want to watch.
filter The kernel filter used to process the event.
flags Actions to perform on the event.
fflags Filter-specific flags.
udata Opaque user data identifier. We'll use this to store the filename for easy lookup later.

Well, which kernel filter do we use if we want to watch a file for changes? There are 9 possible filters, but the one we're looking for is EVFILT_VNODE.

EVFILT_VNODE   Takes a file descriptor as the identifier and the events to watch for in fflags,
               and returns when one or more of the requested events occurs on the descriptor.

It goes on to list 10 possible events that can be used in fflags, but the only flag we need is NOTE_WRITE.

NOTE_WRITE     A write occurred on the file referenced by the descriptor.

Finally, flags defines the actions to perform on the event. There are 10 possible flags, but we only need EV_ADD to register the event in the kqueue, and EV_CLEAR to reset the event state after delivery. Without EV_CLEAR, we'd get the first file change again and again - try it!

If there are multiple writes to this file before we retrieve the event from the kqueue, these will be collapsed into one event. However, if multiple files are written, these will be distinct events. So in practice, in reload, we'll likely want a window such that we don't rerun the command on every single event if they are close in time.

We now have all we need to initialize a kevent structure.

#Watching named files

Let's write a program to watch all files passed as arguments. We omit error handling for rare errors for brevity, but keep for stuff that might happen such as trying to open a file that doesn't exist.

First we open the files to watch with O_EVTONLY. The open man pages say the event-only mode is only intended for monitoring a file for changes, such as with kqueue. We'll need the following headers:

#include <sys/types.h>
#include <sys/event.h>
#include <sys/time.h>

We create an array of kevent structs representing the changes we care about, namely writes to files provided on the command-line. We use the EV_SET macro to initialize the kevent struct.

// Open all files and set up change events
int nfiles = argc - 1;
int *fds = malloc(nfiles * sizeof(int));
struct kevent *changes = malloc(nfiles * sizeof(struct kevent));
for (int i = 0; i < nfiles; i++) {
    fds[i] = open(argv[i + 1], O_EVTONLY);
    if (fds[i] == -1) {
        fprintf(stderr, "open(%s): \n", argv[i + 1]);
        exit(1);
    }

    EV_SET(
        &changes[i],
        fds[i],
        EVFILT_VNODE,
        EV_ADD | EV_CLEAR,
        NOTE_WRITE,
        0,
        (void *)argv[i + 1] // udata: opaque user data. Store filename here.
    );
}

We register the events with a call to kevent().

// Register all events at once
int kq = kqueue();
kevent(
    kq, // the queue
    changes, // array of kevent events to register
    nfiles, // length of array
    NULL, // struct to populate with event (not used here)
    0, // number of events to wait for
    NULL // timeout if waiting for event (irrelevant here)
);

We're now ready to start an event loop and listen to file changes. We use kevent() for this as well. After we get an event back, we can look at event.fflags to see which file event was emitted. In our case we're only listening for NOTE_WRITE so that should always be true.

struct kevent event;
while (1) {
    kevent(
        kq, // queue
        NULL, // array of events to register
        0, // no events to register
        &event, // struct to populate with event
        1, // number of events to wait for
        NULL // no timeout; wait forever
    );

    if (event.fflags & NOTE_WRITE) {
        const char *name = (const char *)event.udata;
        printf("[%s] written\n", name);
    }
}

The full code for this is in this GitHub Gist.

#Watching a directory

In the second mode of reload, we watch the current working directory for any file changes.

reload make

Let's first look at how to watch a directory. First we open the directory itself, and we watch it like we did above for a single file.

int fd = open(directory, O_EVTONLY);

This emits events for new files added to the directory and for file deletions. Such changes involve writing to the file on disk that represents the directory.

However, this does not emit events when there are changes to an existing file, so it's not sufficient for our use. We need to open all files within the directory and watch these for changes individually! This means that when a file is created, we need to add this file to be watched.

I wrote the reload program in Go, so we now switch over to looking at Go code.

#Implementing in Go

We need a reference to a kqueue, and we need to keep track of which file descriptors we open. We also want to be able to refer to files by path.

type watcher struct {
	kq      int            // kqueue file descriptor
	fds     map[string]int // path -> file descriptor mapping
	fdPaths map[int]string // file descriptor -> path mapping (reverse lookup)
}

Notice the CloseOnExec call when we create the kqueue below. When reload re-runs your command, Go's exec package uses the fork + exec pattern. A fork clones the parent's open file descriptors (fd) into the child process, including our kqueue fd and every file descriptor we have opened for watching. The child process doesn't need these, and leaking them can cause subtle problems. The watched files can't be fully deleted by the operating system since their reference count never hits zero, and the child holds a kqueue it never drains.

The O_CLOEXEC flag tells the kernel to automatically close these file descriptors when exec runs. We set it on the kqueue itself here, and on each watched file in Add below.

func newWatcher() (*watcher, error) {
	kq, err := unix.Kqueue()
	if err != nil {
		return nil, err
	}

	// Set close-on-exec flag so child processes don't inherit the kqueue fd
	unix.CloseOnExec(kq)
	return &watcher{
		kq:      kq,
		fds:     make(map[string]int),
		fdPaths: make(map[int]string),
	}, nil
}

When adding a file to be watched, we open the file and register for file writes on the kqueue.

func (w *watcher) Add(path string) error {
	// Skip if already watching
	if _, exists := w.fds[path]; exists {
		return nil
	}

	// Open file with event-only flag to only get events,
	// and close on exec, so that exec'ing the command to reload
	// does not copy the file descriptors.
	fd, err := unix.Open(path, unix.O_EVTONLY|unix.O_CLOEXEC, 0)
	if err != nil {
		return err
	}

	// Register file/directory
	w.fds[path] = fd
	w.fdPaths[fd] = path
	_, err = unix.Kevent(
		w.kq,
		// changes to look for
		[]unix.Kevent_t{{
			Ident:  uint64(fd),
			Filter: unix.EVFILT_VNODE,
			Flags:  unix.EV_ADD | unix.EV_CLEAR,
			Fflags: uint32(unix.NOTE_WRITE),
		}},
		nil, // events to populate: none here
		nil, // no timeout; not populating events anyway
	)

	if err != nil {
		unix.Close(fd)
		return err
	}

	return nil
}

To add all files in a directory, we walk the directory tree and call Add for each entry, be it a file or nested directory.

func (w *watcher) addRecursive(path string) error {
	info, err := os.Stat(path)
	if err != nil {
		return err
	}

	// If it's a file, watch it directly
	if !info.IsDir() {
		return w.Add(path)
	}

	// Walk the directory tree and add all directories and files
	return filepath.WalkDir(path, func(walkPath string, d os.DirEntry, err error) error {
		// non-nil when WalkDir fails to access walkPath
		if err != nil {
			return err
		}

		// Watch the directory for new files and deletions
		err = w.Add(walkPath)
		if err != nil {
			return err
		}
		return nil
	})
}

Finally, we set up the event loop where we listen on the kqueue until we get a file event. Once we do, we return back so the main program can reload the command.

// Wait until any file changes, and return the filename of the changed file
func (w *watcher) Wait() (string, error) {
	events := make([]unix.Kevent_t, 1)
	for {
		n, err := unix.Kevent(
			w.kq,   // the queue
			nil,    // changes
			events, // events to populate
			nil,    // no timeout
		)

		if err != nil {
			if err == unix.EINTR {
				continue
			}
			return "", err
		}

		if n > 0 {
			path, ok := w.fdPaths[int(events[0].Ident)]
			if ok {
				return path, nil
			}
		}
	}
}

As I mentioned, if a file is created in a directory, we need to add that file to be watched as well. We solve that by re-walking the directory if the file event comes from a directory and not a file.

// If the file event was from a directory, it may have
// created a new file. If so, we need to add it!
info, _ := os.Stat(path)
if info != nil && info.IsDir() {
	// Re-walk to pick up new files
	watcher.addRecursive(path)
}

Note that we don't remove a file if it was deleted. This means we are leaking file descriptors when files are deleted, and causing the OS to be unable to clean these up. But for my use case it's so rare that I both delete a file and need to keep the program running for a long time that we don't cover this here.

#Final thoughts

This was fun! I did not know about kqueue before, but I learned that it's easy to work with, and can be used for many inter process communication use cases on macOS.

Another way to solve the file watching problem would be the naive way, that is, to poll for changes. Since kqueue requires an open fd per watched file, kqueue won't scale to very large directory trees. The third way this could be done on macOS is to use FSEvents, which does not have the problem of fd exhaustion.

I would be very happy if you checked out the reload code on GitHub, tried it out on your Mac, or reached out to me with any thoughts or experiences doing something similar.

Finally, if you're interested in reading more about kqueue, you can read this post about implementing a bidirectional streaming server using kqueue in C, and the original paper.

Recently

Wed, 04 Feb 2026 00:00:00 +0000

I want to write more! I like the format of Tom MacWright's Recently posts. So here's what I've been doing and thinking about recently, for the first month of 2026.

#Family

My wife and I just had our third child at the start of this year - our third son! The older boys are 5 and 3, and they are very proud to have a baby brother. I have to say, it's so lovely to have a baby! And family life with three kids has not been so much different from two so far.

#Reading

I reread The Path to Power, the first Lyndon Johnson book by Robert Caro. It's an absolutely incredible book. The combination of LBJ's life being incredibly dramatic, and him working incredibly hard; unprecedented levels of research by Caro and his wife; and Caro's stellar writing. Chapters are thematic and have excellent narrative arcs, making for really great reading.

#Programming

The biggest shift in programming in the past year is the rise of agentic coding. Having an agent (a program using an LLM) write code for me is incredibly fun! I made a spelling game for my oldest in an hour that has given us hours of fun.

My oldest has almost learned to read. This morning I made a macOS game for him: It shows a random <5 letter word and you have to speak the word out loud. It was a great success! pic.twitter.com/xFbamJhJ33
— Vegard Stikbakke (@vegardstikbakke) January 6, 2026

For the past weeks I've switched from Claude Code by Anthropic to using the pi agent, which Armin Ronacher wrote nicely about here. Pros compared to Claude Code:

It's minimal, in a good way.
It doesn't ask for permission at all (yes you can do that with Claude Code, too).
It's open source, extensible, and sessions can easily be shared.

#Watching

My wife and I got completely hooked on The Pitt, a medical drama series. We binged season 1 as fast as we could. It's a medical drama series which I was sure I was too high brow to like, but it was so fun. It did win five Emmys last year! One season spans one shift at the ICU, so it's real-time, which is a very neat format.

In the evenings when my wife is sleeping and I'm soothing the baby, I've been watching Veep. It's laugh out loud funny. Great characters, great dialogue. Turns out Jesse Armstrong of Succession fame is one of the writers. Succession is possibly the best and funniest TV series I've seen, so I'm not surprised!

#Work

Next week is my last at Dune, where I've worked for five years. When I joined I was the fourth person in the company, we're now more than 60. We raised a Series A, a Series B. I worked with so many incredibly kind and smart coworkers. It's been very fun, and I've learned a ton. But now it's time to try something new!

How I rewrote my site using Claude Code

Wed, 31 Dec 2025 00:00:00 +0000

This site is now built using a static site builder that Claude Code wrote for me. Iterating on the site is so much fun now!

If you're a software engineer and not currently using AI for coding, I hope this post can entice you into giving it a try. You might be underestimating how good it is!

#Previous version

The site used to be built using Hugo which is a popular, feature complete static site builder. I had found a template I liked and modified it, and it produced the HTML I needed for my Markdown content. But I still found it very complicated to make the changes I wanted. Writing my own static site builder was definitely an option, but I had considered it too cumbersome.

#Why use AI

Meanwhile, it's the end of 2025 and AI tooling for software engineers did so many leaps forward this year that it's hard to fathom.¹ I'm now regularly using agents at work to write code. What's an agent, you ask? An LLM agent runs tools in a loop to achieve a goal (via Simon Willison). I use agents for:

Writing new application code
Writing tests
Modifying existing code
Debugging
Finding terminal commands (so instead of googling)

Writing a simple static site builder for this small blog is a perfect task for a coding agent, as they are especially good at starting from scratch. I could get a new site done a lot quicker than by writing it all by myself. I could even do it in a few minutes while having some downtime at work.

Let's look at how we built it.

#First Claude session

I wanted to have a program that was able to build the HTML for my existing blog posts. Claude did this in the initial session which spanned at most 10 minutes. I wrote 153 words in total across 6 messages:

Initial prompt
3 sets of clarifying questions before it started to write code
2 more follow up messages from me after it was done writing the code

To start, I created a new directory and copied over my existing blog posts. I opened claude and gave it this prompt:

I want to create a static site from scratch. The code to build it needs to be in Go. I want it to be very minimal: The site builder needs to be able to take md blog posts and embed them. I have posts in the content repo. Their slugs need to be preserved.

I wanted to use Go because it's simple and I know it well, the standard library has great HTML templating support, and the developer tooling is excellent. Claude replied:

⏺ I'll help you create a minimal static site generator in Go for your blog posts. Let me first explore the current state of your repository to understand what exists.

⏺ Explore(Explore repository structure) Haiku 3.5
  ⎿  Done (6 tool uses · 12.8k tokens · 28s)

By pressing Shift + Tab twice I had set Claude in Plan Mode which will first create an implementation plan. The plan is a markdown file persisted to disk. See this blog post by Armin Ronacher for a good investigation of how Plan Mode works. It will usually ask clarifying questions until it decides the plan is clear.

⏺ Now I need to clarify some details about your requirements:

⏺ User answered Claude's questions:
  ⎿  · What should the generated site structure include? → Just a bio blurb,
      and a header which has links "Vegard Stikbakke" (links back to the
     bio, "Posts", and "Books"
     · How should post slugs be determined? → From frontmatter field
     · Where should the generated HTML files be output? → public/ directory
     · Do you want any additional features? → Draft post filtering

It asked a few more questions, 8 in total.

⏺ Perfect! Now I'll create the final implementation plan:

⏺ Updated plan
  ⎿  /plan to preview

⏺ User approved Claude's plan
  ⎿  Plan saved to: ~/.claude/plans/hashed-snuggling-codd.md · /plan to edit

Once I confirmed the implementation plan, it started to execute on the plan, reading it back from disk. The complete plan:

     Static Site Generator in Go - Implementation Plan

     Overview

     Build a minimal static site generator in Go that converts markdown blog
      posts to HTML, preserving slugs from frontmatter and filtering drafts.

     Requirements Summary

     - Homepage: Bio from content/about.md
     - Navigation header: "Vegard Stikbakke" (home) | "Posts" | "Books"
     - Posts listing: All blog posts (excluding drafts)
     - Books listing: Placeholder data with metadata (title, author, date
     read, rating, summary)
     - Individual post pages: /blog/{slug}/index.html
     - Output directory: public/
     - Filter posts with draft: true
     - Preserve slugs from frontmatter slug field

     File Structure

     personal-site-static/
     ├── main.go              # Entry point, orchestrates build
     ├── types.go             # Data structures (Post, Page, Book, Site)
     ├── parser.go            # Markdown parsing and frontmatter extraction
     ├── templates.go         # HTML templates
     ├── generator.go         # HTML generation logic
     ├── go.mod               # Go module definition
     ├── content/             # Input markdown files (existing)
     └── public/              # Output directory (generated)

     Implementation Steps

     1. Create types.go

     Define core data structures:
     type Frontmatter struct {
         Title  string `yaml:"title"`
         Slug   string `yaml:"slug"`
         Date   string `yaml:"date"`
         Draft  bool   `yaml:"draft"`
     }

     type Post struct {
         Title       string
         Slug        string
         Date        time.Time
         DateString  string
         Draft       bool
         HTMLContent string
     }

     type Page struct {
         Title       string
         HTMLContent string
     }

     type Book struct {
         Title      string
         Author     string
         DateRead   string
         Rating     int
         Summary    string
     }

     type Site struct {
         Posts      []Post
         AboutPage  Page
         Books      []Book
     }

     2. Create parser.go

     Implement markdown and frontmatter parsing:
     - extractFrontmatter(): Split YAML frontmatter from content
     - parsePost(): Parse individual blog post files
     - loadPosts(): Load all posts from content/blog/
     - loadPage(): Load static pages (about.md)
     - parseDate(): Handle multiple date formats (quoted, unquoted, ISO
     8601)
     - filterPublished(): Remove drafts from post list
     - sortPostsByDate(): Sort posts newest first

     Dependencies:
     - github.com/yuin/goldmark - Markdown to HTML conversion
     - gopkg.in/yaml.v3 - YAML frontmatter parsing

     3. Create templates.go

     Define HTML templates using html/template:
     - Base layout with navigation header
     - Homepage template (bio content)
     - Posts listing template
     - Books listing template
     - Individual post template
     - Minimal inline CSS for clean styling

     Navigation structure (on all pages):
     <nav>
       <a href="/">Vegard Stikbakke</a>
       <a href="/posts/">Posts</a>
       <a href="/books/">Books</a>
     </nav>

     4. Create generator.go

     Implement HTML generation functions:
     - generateHomepage(): Create /index.html with about.md content
     - generatePostsListing(): Create /posts/index.html with all published
     posts
     - generateBooksListing(): Create /books/index.html with placeholder
     book data
     - generateIndividualPosts(): Create /blog/{slug}/index.html for each
     post
     - renderToFile(): Helper to render template and write to file

     5. Create main.go

     Wire everything together:
     1. Load posts from content/blog/
     2. Filter out drafts
     3. Sort posts by date (newest first)
     4. Load about.md page
     5. Create placeholder book data (3-5 sample books)
     6. Clean and recreate public/ directory
     7. Generate all HTML files
     8. Print success message

     6. Initialize Go module

     go mod init github.com/vegarsti/personal-site
     go get github.com/yuin/goldmark
     go get gopkg.in/yaml.v3

     7. Build and test

     go build -o ssg
     ./ssg

     Key Technical Decisions

     1. Markdown library: goldmark - CommonMark compliant, pure Go,
     well-maintained
     2. Template system: html/template - Built-in, type-safe
     3. Date handling: Support multiple formats with flexible parsing
     4. URL structure: /blog/{slug}/index.html for clean URLs
     5. Styling: Inline CSS in templates for simplicity
     6. Draft filtering: Only filter blog posts, not static pages

     Edge Cases Handled

     1. Missing frontmatter fields: Error if title or slug missing, optional
      date
     2. Date formats: Support quoted strings, unquoted dates, ISO 8601
     3. Posts without dates: Still displayed but sorted to end
     4. Draft filtering: Only applies to blog posts in posts listing
     5. about.md has draft: true: Still used for homepage (not filtered)

     Placeholder Book Data

     Generate 3-5 sample books with structure:
     Book{
         Title: "Example Book",
         Author: "Author Name",
         DateRead: "2024-01",
         Rating: 5,
         Summary: "Brief summary...",
     }

     Output Structure

     public/
     ├── index.html                    # Homepage with bio
     ├── posts/
     │   └── index.html                # Posts listing
     ├── books/
     │   └── index.html                # Books listing
     └── blog/
         ├── how-do-pipes-work-sigpipe/
         │   └── index.html
         ├── csprimer/
         │   └── index.html
         └── [other slugs]/
             └── index.html

     Critical Files to Create

     1. /Users/vegard/dev/personal-site-static/main.go
     2. /Users/vegard/dev/personal-site-static/types.go
     3. /Users/vegard/dev/personal-site-static/parser.go
     4. /Users/vegard/dev/personal-site-static/generator.go
     5. /Users/vegard/dev/personal-site-static/templates.go
     6. /Users/vegard/dev/personal-site-static/go.mod

     Success Criteria

     - ✓ Single go build command produces working binary
     - ✓ Running ./ssg generates complete site in public/
     - ✓ All published posts appear in posts listing
     - ✓ Drafts are excluded from posts listing
     - ✓ Individual post pages preserve slugs from frontmatter
     - ✓ Navigation header works on all pages
     - ✓ Homepage displays about.md content
     - ✓ Books page shows placeholder book data
     - ✓ Clean, minimal styling
     - ✓ Total code under 500 lines

It then started to write code:

⏺ Great! Now let me implement the static site generator. I'll start by creating a todo list to track progress:

A typical problem for LLMs in long conversations has been to lose track of the goal, and Claude uses TODO lists to good effect.

It then went on without intervention from me for several minutes, writing its planned code and trying to build the program. It failed to build several times but was able to recover. This is the magic of agents: It can use a tool, such as code compilation, to autonomously figure out why it's not working and fix it. This is a whole different world than last year when I was copy/pasting between ChatGPT and my editor.

⏺ Bash(go build -o ssg && ./ssg)
  ⎿  Error: Exit code 1
     # github.com/vegarsti/personal-site
     ./parser.go:60:24: filepath.Base undefined (type string has no field or
     method Base)

Eventually it finished:

⏺ Perfect! I've successfully built your minimal static site generator in Go. Here's what was created:

  Generated Files

  - types.go - Data structures (Post, Page, Book, Site)
  - parser.go - Markdown parsing and frontmatter extraction
  - templates.go - HTML templates with minimal CSS
  - generator.go - HTML generation functions
  - main.go - Main orchestration logic
  - go.mod - Go module with dependencies

  Features Implemented

  ✓ Homepage - Displays bio from content/about.md
  ✓ Posts listing - Shows all published blog posts at /posts/
  ✓ Books page - Displays placeholder book data with ratings at /books/
  ✓ Individual posts - Each post at /blog/{slug}/
  ✓ Navigation - "Vegard Stikbakke" | "Posts" | "Books" on all pages
  ✓ Draft filtering - Excludes posts with draft: true (18 published posts)
  ✓ Slug preservation - Uses slug from frontmatter (falls back to filename)
  ✓ Clean styling - Minimal, readable CSS

  Usage

  # Build the generator
  go build -o ssg

  # Generate the site
  ./ssg

  The site is generated in the public/ directory with this structure:
  public/
  ├── index.html                    # Homepage with bio
  ├── posts/index.html              # Posts listing
  ├── books/index.html              # Books with placeholder data
  └── blog/{slug}/index.html        # Individual posts

  The generator successfully processed all 39 markdown files, filtering out 21 drafts and generating 18 published posts plus the placeholder books page.

I opened the generated HTML files in a browser but was unable to navigate the site correctly, so I asked a follow-up question:

Can you successfully navigate the site?

⏺ Let me check if the navigation links are properly set up by examining the generated HTML:

⏺ Bash(grep -A 3 "<nav>" public/index.html)
  ⎿      <nav>
             <a href="/">Vegard Stikbakke</a>
             <a href="/posts/">Posts</a>
             <a href="/books/">Books</a>

⏺ Bash(grep -A 3 "<nav>" public/posts/index.html)
  ⎿      <nav>
             <a href="/">Vegard Stikbakke</a>
             <a href="/posts/">Posts</a>
             <a href="/books/">Books</a>

⏺ Bash(grep -A 3 "<nav>" public/blog/how-do-pipes-work-sigpipe/index.html)
  ⎿      <nav>
             <a href="/">Vegard Stikbakke</a>
             <a href="/posts/">Posts</a>
             <a href="/books/">Books</a>

⏺ Bash(grep 'href="/blog/' public/posts/index.html | head -5)
  ⎿          <a href="/blog/books-2024/">Books I read in 2024</a>
             <a href="/blog/file-watcher/">A file watcher</a>
             <a href="/blog/extract-table/">Extract Table API</a>
     … +2 lines (ctrl+o to expand)

⏺ Bash(ls public/blog/books-2024/ && ls public/blog/file-watcher/)
  ⎿  index.html
     index.html

⏺ The navigation structure looks correct! However, there's one important caveat:

  The site uses absolute paths (starting with /), which means:

  ❌ Won't work if you just open public/index.html directly in a browser (using file:// protocol)
  ✅ Will work when served by a web server

  You can test it by running a local server:

⏺ Bash(cd public && python3 -m http.server 8000 &)
  ⎿  Running in the background (↓ to manage)

⏺ Task Output(non-blocking) b69f192
  ⎿  (No content)

⏺ Bash(sleep 1 && curl -s http://localhost:8000/ | grep -A 4 "<nav>")
  ⎿      <nav>
             <a href="/">Vegard Stikbakke</a>
             <a href="/posts/">Posts</a>
     … +2 lines (ctrl+o to expand)

⏺ Bash(curl -s http://localhost:8000/posts/ | grep -o '<h1>.*</h1>')
  ⎿  <h1>Posts</h1>

⏺ Bash(curl -s http://localhost:8000/blog/how-do-pipes-work-sigpipe/ | grep -o
      '<h1>.*</h1>')
  ⎿  <h1>How do Unix pipes work?</h1>

⏺ Bash(curl -s http://localhost:8000/books/ | grep -o '<h1>.*</h1>')
  ⎿  <h1>Books</h1>

⏺ Bash(curl -s http://localhost:8000/posts/ | grep -o 'href="/blog/[^"]*"' |
      head -3)
  ⎿  href="/blog/books-2024/"
     href="/blog/file-watcher/"
     href="/blog/extract-table/"

⏺ Bash(curl -s http://localhost:8000/blog/books-2024/ | grep '<h1>')
  ⎿  <h1>Books I read in 2024</h1>

⏺ Kill Shell(Kill shell: b69f192)
  ⎿  Shell b69f192 killed
  ⎿  Interrupted · What should Claude do instead?

How to run the local server was useful information, so I told it to add this to a README.

Actually, can you add this to the README? how to serve the files locally

The README didn't exist, so it wrote that.

⏺ Write(README.md)
  ⎿  Wrote 74 lines to README.md
     # Static Site Generator
     A minimal static site generator written in Go that converts markdown
     blog posts to HTML.
     ## Features
     - Converts markdown posts to HTML with clean URLs
     - Preserves slugs from frontmatter
     - Filters out draft posts
     - Minimal, readable styling
     … +64 lines (ctrl+o to expand)

⏺ Done! I've created a README.md with clear instructions on how to serve the files locally.

  Yes, the site navigation works perfectly! I tested it by:
  1. Starting a local Python web server on port 8000
  2. Accessing all pages (homepage, posts, books, individual posts)
  3. Verifying all navigation links work correctly

  The site has full navigation between:
  - Homepage (/) → Bio from about.md
  - Posts (/posts/) → Listing of 18 published posts
  - Books (/books/) → Placeholder book data with ratings
  - Individual posts (/blog/{slug}/) → Each post page

  All pages have the same navigation header: "Vegard Stikbakke" | "Posts" | "Books"

  The only caveat is that the site must be served via a web server (like the Python command in the README) because it uses absolute paths. It won't work if you just double-click the HTML files to open them directly in a browser.

If you're interested, you can see the full session here.

#Adding more functionality

One really useful capability for the agent to have is being able to browse websites in a headless browser, so I installed Puppeteer as described here. In one session I asked it to tweak the styling:

I want the blog post page to look more like https://macwright.com/writing. It should have two columns: Blog post title and date. The link to the blog post should be regular text color but when hovered should have an underline.

After writing code, it confirmed it worked by using Puppeteer:

⏺ Excellent! The build succeeded. Let me serve it and take a screenshot to verify the new layout.

⏺ Bash(cd public && python3 -m http.server 8000 &)
  ⎿  Running in the background (↓ to manage)

⏺ puppeteer - puppeteer_navigate (MCP)(url: "http://localhost:8000/blog/")
  ⎿  Navigated to http://localhost:8000/blog/

⏺ puppeteer - puppeteer_screenshot (MCP)(name: "posts-page", width: 1200,
                                        height: 800)
  ⎿  Screenshot 'posts-page' taken at 1200x800
  ⎿  [Image]

⏺ Great! The layout is working. Let me test the hover effect to verify the underline appears:

⏺ puppeteer - puppeteer_hover (MCP)(selector: ".post-title a")
  ⎿  Hovered .post-title a

⏺ puppeteer - puppeteer_screenshot (MCP)(name: "posts-page-hover", width: 1200,
                                        height: 800)
  ⎿  Screenshot 'posts-page-hover' taken at 1200x800
  ⎿  [Image]

⏺ Perfect! The hover effect is working - you can see the underline on "Books I read in 2024" in the hover screenshot. Let me clean up the background server.

Here's the full session.

Here are links to other session you may find interesting:

Happy prompting!

Andrej Karpathy's tweet Dec 26, Armin Ronacher: A Year of Vibes, Mario Zechner: What I learned building an opinionated and minimal coding agent, Peter Steinberger: Shipping at Inference-Speed ↩︎

Books I read in 2025

Fri, 26 Dec 2025 00:00:00 +0000

I read 16 books this year. The full list is at the bottom, and on Goodreads. These are the books I recommend. Mostly nonfiction, one fiction.

#Breakneck: China's Quest to Engineer the Future by Dan Wang

Dan Wang is a China expert whose book came out this year.

The main idea in Breakneck is that the US is a lawyerly society while China is an engineering state, and at the same time the two peoples are similar in temperament. Each chapter focuses on a specific domain - high tech, the one-child policy, Zero COVID. China applies engineering to social problems as well. The one child policy chapter was particularly harrowing.

It's mind boggling how much rail is built, how many highrises. A main takeaway for me is that building big public projects produces civic pride.

A fast and engaging read, highly recommended! And if you need more, Tyler Cowen's blurb on the book says

The best recent book on China, on China and America, and, arguably, the best book of the year flat out.

#All the Pretty Horses by Cormac McCarthy

I love Cormac McCarthy! His writing style is minimal and with incredible depictions of landscapes and personalities. This was the book that launched him into the mainstream. You should also read Blood Meridian.

It's set in the late 1940s and follows a pair of teenage boys who go to Mexico to work as cowboys. An incredible and tragic coming of age book about love, fairness, and the end of the old West.

#The Scaling Era: An Oral History of AI, 2019-2025 by Dwarkesh Patel, Gavin Leech

Dwarkesh, still in his early twenties, came out of nowhere to create an excellent podcast that I really enjoy. He does hour-long interviews with public intellectuals, executives, academics, engineers. He's done many interviews focused on AI, and this book is a compilation of what leaders in the field think.

It's a great read if you're familiar with recent AI developments. It's surprisingly well-edited and fun to read.

#The Soul of a New Machine by Tracy Kidder

Bryan Cantrill's favorite book and won a Pulitzer Prize, do you need more convincing?

A book about the male urge to build something great. It follows Data General Corporation's skunkworks project as they design (and debug!) a new computer in the 1980s. The team is led by the enigmatic Tom West. He hires mostly college graduates, who of course end up working night and day for years.

I read this in 2019 but enjoyed it even more the second time round. I recommend it to any engineer who would recognize themselves in it.

#Show Stopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft by G. Pascal Zachary

Like The Soul of a New Machine, this follows a big technology project: The building and shipping of Windows NT at Microsoft.

As project leader Dave Cutler says, the creation of Windows NT may be the last time anyone ever assembles a team to build a completely new computer operating system, and this book gives a good account of the personalities, the stresses, and the working environment involved in making it happen.

Recommended, but don't pick this over The Soul of a New Machine.

#Hackers: Heroes of the Computer Revolution by Steven Levy

A collection of biographies of groups of hackers from the 1950s through the 1980s. Mainframe hackers, hardware hackers, game hackers.

I enjoyed this a lot. Although at almost 500 pages you need to be interested in this stuff.

#Steve Jobs & The NeXT Big Thing by Randall E. Stross

10 years after starting apple and having started the personal computer revolution, Jobs resigned after a clash of visions with the new CEO. This is when he starts NeXT. NeXT flops commercially, but is bought by Apple after 13 years. Jobs, having now returned to Apple, has an incredible second stint at Apple. This is of course when they invented the iPod and iPhone.

Crucially, this book is written before Jobs' final arc. This book is just about NeXT and how thoroughly Jobs messed up, and it's very interesting to read. A lot of hubris, overengineering, and failing to consider the tradeoffs necessary to produce a good business. It's a very different book than the biography of Jobs by Walter Isaacson, which is a lot more favorable to Jobs.

The operating system developed at NeXT, NeXTSTEP, became the foundation of what's now macOS and iOS, so today we have a lot to thank them for!

#Full list

All the Pretty Horses (1992) by Cormac McCarthy
Writing for Developers: Blogs that get read (2025) by Piotr Sarna, Cynthia Dunlop
The Crossing (1994) by Cormac McCarthy
The Soul of a New Machine (1981) by Tracy Kidder (reread)
Show Stopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft (1994) by G. Pascal Zachary (reread)
A Philosophy of Software Design (2019) by John Osterhout (reread)
Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers (2019) by Andy Greenberg
Steve Jobs & The NeXT Big Thing (1993) by Randall E. Stross
Conspiracy: Peter Thiel, Hulk Hogan, Gawker, and the Anatomy of Intrigue (2018) by Ryan Holiday
Hackers: Heroes of the Computer Revolution (1984) by Steven Levy
The Staff Engineer's Path: A Guide for Individual Contributors Navigating Growth and Change (2022) by Tanya O. Reilly
UNIX: A History and a Memoir (2019) by Brian W. Kernighan (reread)
The Art of Readable Code: Simple and Practical Techniques for Writing Better Code (2010) by Dustin Boswell (reread)
Breakneck: China's Quest to Engineer the Future (2025) by Dan Wang
The Scaling Era: An Oral History of AI, 2019-2025 (2025) by Dwarkesh Patel, Gavin Leech
Too Big to Fail (2009) by Andrew Ross Sorkin

Books I read in 2024

Sun, 12 Jan 2025 00:00:00 +0000

I finished 9 books this year, in total 2006 pages. That is on the lower end of reading since I started tracking, not that anyone's counting! ▁▃█▅▃▄▃▁

As usual, recommendations in bold. Below I'll briefly say something about each book.

Tidy First? A Personal Exercise in Empirical Software Design (2024) by Kent Beck
The Death of Ivan Ilyich (1886) by Leo Tolstoy
Blood Meridian (1985) by Cormac McCarthy (reread)
The Spy Who Came In From the Cold (1963) by John le Carré
As I Lay Dying (1930) by William Faulkner
The Undoing Project: A Friendship That Changed Our Minds (2016) by Michael Lewis
The Old Man and the Sea (1952) by Ernest Hemingway
Code Health Guardian: The Old-New Role of a Human Programmer in the AI Era (2024) by Artie Shevchenko
21 Church Fathers (2000) by Peter Halldorf

There were a handful of books I spent a meaningful amount of time reading but didn't complete.

The Making of the Atomic Bomb (1986) by Richard Rhodes
Softwar: An Intimate Portrait of Larry Ellison and Oracle (2003) by Matthew Symonds
Rust in Action (2021) by Tim McNamara
Hyperion (1989) by Dan Simmons
Writing for Developers: Blogs that Get Read by Piotr Sarna and Cynthia Dunlop (2024)

#Fiction

I read more fiction books than non-fiction books this year! 🎉

The Death of Ivan Ilyich (1886) by Leo Tolstoy. A gripping novella about death, family, career, facades, hypocrisy, and the meaning of life. You should read it!
Blood Meridian (1985) by Cormac McCarthy. So beautiful! McCarthy's prose is just incredible. See (Probably not) All of the similes in Blood Meridian. And so dark! It's basically about violence and evil. Set in the mid-1800s, follows the Kid who joins a scalp-hunting expedition led by the enigmatic Judge. The Judge is rendered so well that I'm not sure I would watch a movie adaptation if there ever is one.
The Spy Who Came In From the Cold (1963) by John le Carré. I wanted to try something by the renowned spy novel master, and was recommended this one. It was okay 🤷‍♂️
As I Lay Dying (1930) by William Faulkner. Set in the South. The mother of a family is dying and each chapter is narrated by one of the grieving family members. It was interesting and engaging, and probably a classic for good reason. There's often talk of Faulkerian writing, but I didn't see anything super particular in his style. The book hasn't been much on my mind since I read it, but worth reading anyway.
The Old Man and the Sea (1952) by Ernest Hemingway. I read this in two sittings! It's short, exciting and touching.
Hyperion (1989) by Dan Simmons. As with so many science fiction books, I loved the exposition. But the book is too lengthy and too dialogue heavy, so I abandoned it after 200 pages.

#Non-fiction

Tidy First? A Personal Exercise in Empirical Software Design (2024) by Kent Beck. A small book (~100 pages) by Kent Beck of agile and TDD fame. Very pragmatic and enjoyable book on how to think about the balance between adding new features to a codebase and improving the code.
The Undoing Project: A Friendship That Changed Our Minds (2016) by Michael Lewis. Michael Lewis needs no introduction. This is about Daniel Kahneman and Amos Tversky, the Israeli psychologists who founded behavioral economics. Super engaging as always with Lewis, and interesting subject.
Code Health Guardian: The Old-New Role of a Human Programmer in the AI Era (2024) by Artie Shevchenko. Very 2024 book title, but the book is less about AI and more some good recommendations about maintaining a codebase.
21 Church Fathers (2000) by Peter Halldorf. Great primer on the most influential people in the Church in the first millennium.
The Making of the Atomic Bomb (1986) by Richard Rhodes. At almost 900 pages, this is an undertaking! Rhodes is a historian but also a journalist, which shines through because this is a fun read. It covers the people and discoveries that led to nuclear fission and the atomic bomb. Not surprisingly it won the Pulitzer and is praised in general.
Softwar: An Intimate Portrait of Larry Ellison and Oracle (2003) by Matthew Symonds. Ellison is founder of Oracle and the third-wealthiest person in the world. An interesting read! The book is very detailed. It feels like it covers every happening in Oracle since the founding until the publishing of the book. Thus I didn't end up finishing it, only about 250 out of 720 pages in. What's fun about the book is that Ellison required being able to add his own footnotes to it. Symonds had the last word on the text in the book itself, Ellison on the footnoes. These - surprisingly to me - made him come across as both humble and sympathetic.
Rust in Action (2021) by Tim McNamara. Writing a lot of Rust at work, which is both fun and frustrating. Haven't used this enough to say anything sensible, though!
Writing for Developers: Blogs that Get Read by Piotr Sarna, Cynthia Dunlop (2024). Very practical book about writing blogs. Was released late December, so haven't finished it yet. If you want to read it, you should join Phil Eaton's book club on it.

#Source code for the graphic above

~ # cat book_stats.csv | awk -F ',' 'NR>1 {print $3}' | spark
▁▃█▅▃▄▃▁

where spark is Zach Holman's https://zachholman.com/spark/.

A file watcher

Sun, 25 Aug 2024 00:00:00 +0000

I wrote reload, my own file watcher!

The last year and a half I've been making my way through CSPrimer. It consists of hundreds of computer science problems, so working through these means I'm writing hundreds of small programs. Like most programming, it's a very iterative process where I tweak my program and rerun it, typically switching from my editor to run a command.

gcc main.c -o main && ./main

It would be nice if this just reran automatically whenever I needed it to, so I didn't have to switch between my editor and the terminal!

Of course, there exist tools like this. They're sometimes called file watchers or hot reloaders. I tried a few existing tools such as entr and wach but they weren't just right.

So I figured I'd write my own.

After a few hours the result was reload. It's one main.go file with less than 200 lines. It does exactly what I need, nothing more.

#Requirements

I wanted to be able to just put reload in front of the command I'd usually run. To do that, what reload needed to do was

figure out which file(s) to watch
figure out which command to run
run the command
listen to file changes
reload the command on file changes

And it also needed to

run command pipelines like gcc main.c -o main && ./main sequentially
cancel the command if it's still running when there's a file change
deduplicate file events -- so that a change to main.c and the main binary file don't trigger two subsequent reloads

There needs to be some concurrent code here, but Go excels at this with channels and goroutines.

In the rest of the post, let's look at how reload

Figures out which files to watch
Detects file changes
Runs the command

#Figuring out which files to watch

Some of these tools have you explicitly list the files that should trigger a reload. For my use case, my command usually has the filename already. Why not just grab that?

So reload looks at each part of the command, and checks if it's the filename of an existing file. If it is, we add it to the list of files to watch. If we didn't find any file, we fall back to watching the working directory.

Kinda neat!

// Find files in command that we will watch
toWatch := make([]string, 0)
for _, part := range input {
	// Check if there's a file to watch
	info, err := os.Stat(part)
	if os.IsNotExist(err) {
		continue
	}
	check(err)
	if !slices.Contains(toWatch, info.Name()) {
		toWatch = append(toWatch, info.Name())
	}
}

// Fall back to watching the working directory
if len(toWatch) == 0 {
	wd, err := os.Getwd()
	check(err)
	toWatch = append(toWatch, wd)
}

#Detecting file changes

The most unfamiliar thing to me here was watching for file changes.

I used the fsnotify package to handle file notifications. It supports all the usual platforms like macOS, Windows, and Linux. On macOS it uses the kqueue notification interface, whereas on Linux it uses inotify.

Here's the code for that.

watcher, err := fsnotify.NewWatcher()
check(err)
defer watcher.Close()

for _, file := range toWatch {
	err = watcher.Add(file)
	check(err)
}

fileChanges := make(chan string)
wg.Add(1)
go func() {
	defer wg.Done()
	lastChange := time.Now()
	dedupWindow := 100 * time.Millisecond
	for event := range watcher.Events {
		if event.Has(fsnotify.Write) {
			// Treat multiple events at same time as one
			if time.Since(lastChange) < dedupWindow {
				continue
			}
			lastChange = time.Now()
			fileChanges <- event.Name
		}
	}
}()

And we listen on the fileChanges channel in the code to run the command, which we will see now.

#Running commands

To create a command that we can run, and cancel, we use exec.CommandContext from Go's standard library. We can specify its standard in, standard out, and standard error. In our case, we want standard out and standard error to be the operating system's stdout and stderr, since we want the command to behave as if we ran it directly in the terminal.

sh -c '<command>' allows us to pass the full provided command. All use of pipes and operators like && and || will just work, as it's handed over to the shell.

// In `main()`
// ...

// First run the command
runCommand(ctx, command, fileChanges)

// Then rerun it on file changes
for name := range fileChanges {
	fmt.Fprintf(os.Stderr, "--- Changed: %s\n", name)
	fmt.Fprintf(os.Stderr, "--- Running: %s\n", command)
	runCommand(ctx, command, fileChanges)
}

// ...

func runCommand(ctx context.Context, command string, fileChanges chan string) {
	// Create child context so we can cancel this command
	// without cancelling the entire program
	commandCtx, commandCancel := context.WithCancel(ctx)
	defer commandCancel()

	// Cancel and rerun the command if the file changes
	go func() {
		name, ok := <-fileChanges
		// The channel was closed, shut down
		if !ok {
			return
		}
		commandCancel()
		fileChanges <- name
	}()

	// Run the command using `sh -c <command>` to allow for
	// shell syntax such as pipes and boolean operators
	cmd := exec.CommandContext(commandCtx, "sh", []string{"-c", command}...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	_ = cmd.Run() // It's fine if the command fails!
}

I hope you'll try it out!

Extract Table API

Sat, 17 Aug 2024 00:00:00 +0000

How great would it be to pull a table from an image or PDF and instantly turn it into structured text?

I built this! Check out extract-table.com.

Uploading an image produces a result like in the screenshot above, hosted at result.extract-table.com.

Please try it!

The site uses api.extract-table.com which can also be used directly.

$ curl https://api.extract-table.com -X \
    POST -H "Content-Type: image/png" \
    -H "Accept: text/csv" \
    --data-binary @invoice.png
Items,Type,,Quantity,Price,Total
Test lonut,Rate,Plan,2,$11.00/$242.00,$253.00
bulk import item2,MRC,,1,$2.20,$2.20
bulk import item2 desc,,,,,
bulk import item4,NRC,,2,$4.40,$8.80
bulk import item4 desc,,,,,
bulk import item7,NRC,,2,$0.00,$0.00
bulk import item7 desc,,,,,
asdasd,NRC,,2,$0.00,$0.00
asdasd,,,,,
mine,MRC,,2,$121.00,$242.00
2bulk import item1,MRC,,1,$33.00,$33.00
bulk import item 1 desc,,,,,

The API is an AWS Lambda function which uses Amazon Textract, AWS's OCR service that gives you all words with their bounding box. It even has a mode for detecting tables, which is 10x more expensive.

I actually built this in 2021 and posted about it on Hacker News. Since then the API has been used 26 thousand times. I originally used the table detection algorithm, but that became too expensive to host for free.

I've since written my own table detection algorithm, which was used to produce the result above. But explaining that algorithm is for another post!

Browse the code.

Blogs I like

Wed, 31 Jul 2024 00:00:00 +0000

Books I read in 2023

Sat, 06 Jan 2024 00:00:00 +0000

I read 11 books this year, in total 4488 pages.

I would recommend all of the books in bold. That turned out to be almost all of them. Below the list I'll group the books and talk a bit about each book.

How the World Really Works: A Scientist's Guide to Our Past, Present and Future (2022) by Vaclav Smil
A Promised Land (2020) by Barack Obama
Working in Public: The Making and Maintenance of Open Source Software (2020) by Nadia Eghbal
Crafting Interpreters (2021) by Robert Nystrom
Skunk Works: A Personal Memoir of My Years at Lockheed (1994) by Ben R. Rich & Leo Janos
The Idea Factory: Bell Labs and the Great Age of American Innovation (2012) by Jon Gertner
Dealers of Lightning: Xerox PARC and the Dawn of the Computer Age (2000) by Michael A. Hiltzik
A Mind At Play (2017) by Jimmy Soni & Rob Goodman
Of Mice and Men (1937) by John Steinbeck
American Pastoral (1997) by Philip Roth
Poor Charlie's Almanack: The Essential Wit and Wisdom of Charles T. Munger (2005) by Charles T. Munger

#History of technology

Skunk Works: A Personal Memoir of My Years at Lockheed (1994) by Ben R. Rich & Leo Janos. This was sadly underwhelming, I ended up slogging through it. It has a really high ranking on Goodreads (4.4), and several people have said it's one of their favorite books. I found it to be long winded and lacking a good narrative throughout. It would have benefited from better editing. Ben Rich was the director of Lockheed from 1975 to 1991, and Lockheed developed stealth airplanes that were crucial in the Cold War. So it was still interesting to learn about Lockheed and the stealth airplanes they developed. Crucially they did this with a capable and nimble team unimpeded by bureacracy, and I think this is what makes it popular. A lot of this thinking has clearly influenced a lot of tech companies.
The Idea Factory: Bell Labs and the Great Age of American Innovation (2012) by Jon Gertner. This is a must read if you're into the history of tech! Super interesting, very well narrated, about the research lab that all big companies dream of emulating. Bell Telephone was a monopoly, the only (?) phone company in the US, which made a lot of money. Luckily they used a lot of the profits to fund basic research at Bell Labs. So much of the technology that the modern world relies on was invented at Bell Labs. To list some out of a long list:
- The transistor
- Laser
- Quality control theory
- Information theory
- Computer music (in 1957!)
- Computer art (1962!)
- The fast Fourier transform (FFT)
- Unix
Dealers of Lightning: Xerox PARC and the Dawn of the Computer Age (2000) by Michael A. Hiltzik. Not as gripping as The Idea Factory, but still enjoyable. Xerox failed to commercialize their innovations, but PARC invented so many things we take for granted in computers, and many alumni went on to be crucial at Apple and Microsoft.
- the graphical user interface (GUI)
- WYSIWYG text editing
- object-oriented programming (OOP)
- Ethernet

#Programming

Crafting Interpreters (2021) by Robert Nystrom. Highly recommended if you're a programmer! It walks through implementing a programming language using two different approaches and languages. Good prose with fun illustrations and analogies, which makes it extra fun to read. I followed along by typing in all the code. This is a long book (865 pages) so that was time consuming, but really educational.

#Biographies

A Promised Land (2020) by Barack Obama. I honestly don't remember much of this. It covers his campaign and his first term as President.
A Mind At Play (2017) by Jimmy Soni & Rob Goodman. Great read. Biography about Claude Shannon, the mathematician who came up with information theory while at Bell Labs. Brilliant and very strange guy.

#Other non fiction

How the World Really Works: A Scientist's Guide to Our Past, Present and Future (2022) by Vaclav Smil. You should really read this if you want to know about climate change and what is actually required to transition away from carbons. It's very dense, but very interesting, and just ~250 pages. Chock full of numbers and sentences like "In two centuries, the human labor to produce a kilogram of American wheat was reduced from 10 minutes to less than two seconds." Smil has a holistic systems thinking view of the world which seems obviously correct. Again, read this.

Reading Vaclav Smil's new book How the World Really Works. Great book. Very effective at conveying how thoroughly dependent we are on fossil fuels.
— Vegard Stikbakke (@vegardstikbakke) February 4, 2023

Working in Public: The Making and Maintenance of Open Source Software (2020) by Nadia Eghbal. Recommended reading if you're a programmer. About open source and the various models of open source projects that exist.
Poor Charlie's Almanack: The Essential Wit and Wisdom of Charles T. Munger (2005) by Charles T. Munger. This was just re-released by Stripe Press. They keep publishing bangers. Essentially a collection of speeches by Charlie Munger, the lesser known partner of Warren Buffett. Munger is well known in certain circles for his focus on mental models. He has a lot of interesting takes on "the psychology of misjudgment", essentially biases, which he claims he's used to good effect over the years. And his (Berkshire Hathaway's) record speaks for itself.

"If the facts don't hang together on a latticework of theory, you don't have them in a usable form. You've got to have mental models in your head."

#Fiction

Two books by prominent American novelists. I really enjoyed both!

Of Mice and Men (1937) by John Steinbeck. I've had this in my bookshelf for a few years. Short and heart warming and sad.
American Pastoral (1997) by Philip Roth. Very funny and interesting. I'd been wanting to read something by Philip Roth, who is regarded as one of the great American writers of the 20th century.

Just posted my annual roundup of the books I read last year.

Some tech history, some biographies, some fiction, and a programming book.

Don't read Skunk Works.

Do read Vaclav Smil, you'll be more informed on climate change. pic.twitter.com/6ymMzMSkwV
— Vegard Stikbakke (@vegardstikbakke) January 6, 2024

Books I read in 2022

Thu, 12 Jan 2023 00:00:00 +0000

I read 14 books this year, spanning 5878 pages. Not too bad with two kids of 0 and 2 years.

I'm trying something new this year, so instead of just listing out the books I'm also going to group them and say something about them. As always, those in bold were my favourites.

Master of the Senate (2002) by Robert Caro
The Making of Prince of Persia: Journals 1985 - 1993 (2011) Jordan Mechner
The Passage of Power (2012) by Robert Caro
The Founders: The Story of Paypal (2022) by Jimmy Soni
Working Backwards: Insights, Stories, and Secrets from Inside Amazon (2021) by Bryar and Carr
The Big Score: The Billion Dollar Story of Silicon Valley (1985) by Michael Malone
Kotlin in Action (2016) by Jemerov and Isakova
Moneyball (2003) by Michael Lewis
The Mythical Man Month (1975) by Fred Brooks (reread)
The Big Short (2010) by Michael Lewis
The Last Lion: Winston Spencer Churchill: Visions of Glory, 1874-1932 (1983) by William Manchester
How Query Engines Work (2022) Andy Grove
Never Let Me Go (2005) by Kazuo Ishiguro
The Premonition: A Pandemic Story (2021) by Michael Lewis

#Biographies

Master of the Senate (2002) by Robert Caro
The Passage of Power (2012) by Robert Caro
The Last Lion: Winston Spencer Churchill: Visions of Glory, 1874-1932 (1983) by William Manchester

My first and only big reads this year were book 3 and 4 of Robert Caro's Years of Lyndon Johnson. I just can't recommend this series enough! Caro has done an unbelievable amount of research, and his writing is very engaging.

Master of the Senate (spanning 1167 pages!) was somehow simultaneously a description of how the Senate works and a page turner. It's an excellent drama on how Johnson ascended to the role of Senate majority leader, and uses that power to incredible gain, passing among others a landmark civil rights bill. Learned a lot about how legislation in the US works, too.

In The Passage of Power, Johnson joins JFK as his vice president, Kennedy is assassinated, and Johnson takes power as president (this is his life-long dream!). It ends a few months after he came to power, in which time he had accomplished a lot already. My favorite thing to come out of this was getting a more nuanced view of JFK. He was not as likable as I thought: Far more cynical. But also incredibly courageous.

I also read another biography about a politician: The Last Lion 1 about Churchill. What a remarkable man. Listened to this on 2x on audiobook while painting the house, I think I would have enjoyed it more if I'd sat down and read it. But wow, Churchill was impressive. For example, he spent multiple stints as a war correspondent. During the Boer War in South Africa he was captured as a prisoner of war and makes an escape that's straight out of a movie. And he was hilarious. I laughed out loud many times. I'm hoping to read The Last Lion book 2 this year.

#Silicon Valley history

As always, I read some Silicon Valley history. This is probably the biggest theme going through my reading in previous years (2018, 2019, 2020, 2021).

The Big Score: The Billion Dollar Story of Silicon Valley (1985) by Michael Malone. Written in 1985, so it's one of the books written during the birth of Silicon Valley as we know it. It's based on the author's interviews from his time as a tech reporter. Iconoclastic founders have been a staple of Silicon Valley since the beginning!
The Founders: The Story of Paypal (2022) by Jimmy Soni. Highly enjoyable. The PayPal mafia is well known, so it was fun to read the origin story. Particularly interesting topics: The merger with X.com (Elon Musk's site) and Musk subsequently proposing a rewrite of the entire PayPal codebase; Max Levchin (now CEO of Affirm) fighting fraud.

Kotlin in Action (2016) by Jemerov and Isakova. I started writing Kotlin at work, and reading this was a boon to my productivity.
The Mythical Man Month: Essays on Software Engineering (1975) by Fred Brooks (reread). A foundational text in software engineering management. The list of ideas presented is long, and if you're a software engineer, you're bound to have heard some. E.g. Adding manpower to a late software project makes it later. Highly recommended for anyone in software.
How Query Engines Work (2022) Andy Grove. I've become interested in databases recently. This was a quick read about, well, how query engines work, in particular query planning. Grove (not that Andy Grove) started the DataFusion query engine project that is now part of Apache Arrow. The book has a fairly low level of detail, which is good for an overview, but the book felt a bit incomplete. Turns out the book is still in progress and available online, so I would recommend reading there.

#Michael Lewis books

I rewatched both The Big Short and Moneyball this year, and was inspired to read the books behind the movies. I've read other things by Lewis before, and his writing is interesting and easy to read.

Moneyball (2003) by Michael Lewis
The Big Short (2010) by Michael Lewis
The Premonition: A Pandemic Story (2021) by Michael Lewis. About the Covid pandemic, and one big takeaway from that is how badly CDC was set up to handle it. Bad incentives all around.

#Other non fiction

The Making of Prince of Persia: Journals 1985 - 1993 (2011) Jordan Mechner
Working Backwards: Insights, Stories, and Secrets from Inside Amazon (2021) by Bryar and Carr. Amazon's six pager memos are well known by now, and has inspired a culture of design docs in software companies. This book describes that and other interesting Amazon management things.

#Fiction

I only finished one novel in 2022. I also started Kafka's The Trial, but it was too frustrating!

Never Let Me Go (2005) by Kazuo Ishiguro. Thought provoking and sad.

Got any recommendations? Hit me up!

Posted an overview of the 14 books I read in 2022, and some words about my favourites! Take a look to find out how reading about the Senate can be interesting, and a previous time Musk joined a company and suggested a rewrite of the codebase https://t.co/enOFO2LRuU
— Vegard Stikbakke (@vegardstikbakke) January 12, 2023

What's coming in Go 1.18 (for LWN.net)

Tue, 15 Feb 2022 00:00:00 +0000

I wrote a paid blog post for LWN.net on a freelance basis: What's coming in Go 1.18

My open source contributions

Mon, 10 Jan 2022 00:00:00 +0000

I've worked full time as a software engineer for around two and a half years, since May 2019. Almost all of the code I've written has been in closed-source repos, but occasionally I've ran into something somewhere else that I was able to fix. I compiled a list of the pull requests (PRs) I've submitted that have been accepted upstream. There are 15 PRs in total, and most of them are typos in documentation, but I have a few code fixes as well.

#Python

A bug fix, bpo-38932, to a test library in the Python standard library, which was included in Python 3.9.

https://github.com/python/cpython/pull/17409

Two fixes to typeshed, which is where external type annotations for the Python standard library live.

#Others

alexbrainman/odbc is an ODBC driver in Go. It failed to compile on ARM-based Macs due to the Homebrew path for these being different. https://github.com/alexbrainman/odbc/pull/168
Documentation fix to FastAPI, which is an excellent API framework in Python. https://github.com/tiangolo/fastapi/pull/1015

#Typos

I'm really good at spotting typos, so I've fixed a number of these.

Books I read in 2021

Sun, 02 Jan 2022 00:00:00 +0000

Below is a list of the 12 books I read this year, spanning 4244 pages (according to Goodreads). This is less than 2018 (16), 2019 (40), and 2020 (23), but considering this is my first full year with a toddler, I'm pleased!

Those in bold were my favourites.

Hard Drive: Bill Gates and the Making of the Microsoft Empire by James Wallace and Jim Erickson
The Dark Forest by Liu Cixin
How the Internet Happened by Brian McCullough
Staff Engineer: Leadership Beyond the Management Track by Will Larson
The Art of Readable Code by Dustin Boswell and Trevor Foucher
Amazon Unbound by Brad Stone
Writing an Interpreter in Go by Thorsten Ball (reread)
The Man from the Future: The Visionary Life of John von Neumann by Ananyo Bhattacharya
Means of Ascent by Robert Caro
Foundation by Isaac Asimov (reread)
Foundation and Empire by Isaac Asimov (reread)
Second Foundation by Isaac Asimov (reread)

And here are books I started but didn't or haven't finished.

Selfish Reasons to Have More Kids by Bryan Caplan
Game Theory: A Nontechnical Introduction by Morton Davis
The Box by Marc Levinson
The Innovator's Dilemma by Clayton Christensen
The Rise of Christianity by Rodney Stark
Working Backwards: Insights, Stories, and Secrets from Inside Amazon by Colin Bryar and Bill Carr
Mindstorms by Seymour Papert
The Contrarian: Peter Thiel and Silicon Valley's Pursuit of Power by Max Chafkin

2021 book thread
— Vegard Stikbakke (@vegardstikbakke) January 17, 2021

til.vegardstikbakke.com

Tue, 10 Aug 2021 00:00:00 +0000

Simon Willison and others (Josh Branchaud, David Winterbottom) have started posting their Today I Learned snippets, or TILs for short.

Occasionally over the past couple years, I've noted down a useful command, fact or code snippet in a folder. Instead of just keeping these in a private Git repo, I decided to do the same:

I created a site on Netlify with the subdomain - which was super easy, by the way!
I copied the design and scripts that Winterbottom uses - here's my Git repo.
I migrated my existing notes to the new format.

And all of a sudden I had a site of pre-populated snippets!

Head over to til.vegardstikbakke.com to check it out!

Waiting for dependencies in tests

Fri, 28 May 2021 00:00:00 +0000

As part of my development workflow, I usually run some integration tests. An integration test means that code interacts with some outside dependency. This could be a temporary Postgres server in a Docker container, or it could be an HTTP server. These dependencies usually take some time to start. In particular, enough time to start that our tests would fail if we ran them immediately after starting our dependencies.

To make this concrete, let's use docker-compose to start a Postgres server in a Docker container. Our docker-compose.yml becomes

version: "3.8"
services:
  postgres:
    image: postgres:13-alpine
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: admin
      POSTGRES_USER: admin

and we can start it by doing docker-compose up.

We also want a Makefile to perform the whole test, including the setup. The structure of our Makefile is

start Docker containers
wait until dependencies are ready
execute tests
shut down Docker containers

The lazy way to do step 2 is to wait a set amount of time before executing tests, typically by running the shell command sleep N. To be sure that the dependencies are indeed ready when we start our tests, we will want to set N to a safe upper bound on the depency's startup time. If we run locally, we probably have the Docker containers cached, so docker-compose up might not take long. However, if we run our CI in Github Actions, the containers will not be cached, and so the wait time is longer and more unpredictable. This means that if we set the time to a comfortable upper bound so we sleep long enough in CI, we risk wasting time each time we run our tests locally.

But there are better solutions!

#`until`

The until shell command will run a command until it succeeds. Like a while loop but with the predicate switched. If we find a shell command that returns exit code 0 (indicating success) if the dependencies are ready, and not 0 if they are not ready, we can put this inside an until loop.

It turns out that the Postgres client psql comes with a command called pg_isready, which will do just that: It pings a Postgres server and returns a non-zero exit code if it's not ready.

The complete Makefile becomes

all:
	make test

test:
	docker-compose up -V -d postgres
	until pg_isready --host localhost --port 5432; do \
		sleep 0.1; \
	done
	echo "my test here"
	docker-compose down

Try it yourself by cloning this Github repo and running make!

#Bonus: HTTP server

If we want to wait for an HTTP server to be responsive, we can instead run this curl command:

curl --output /dev/null --silent --head --fail http://localhost:${PORT}

which becomes the following in a Makefile:

until $$(curl --output /dev/null --silent --head --fail "http://localhost:8080"); do \
	sleep 0.1; \
done

Books I read in 2020

Mon, 04 Jan 2021 00:00:00 +0000

Below is a list of the 23 books I read this year (8151 pages). Those in bold were my favourites. See also what I read in 2018 and 2019.

Flow by Mihaly Csikszentmihalyi
Super Pumped by Mike Isaac
Educated by Tara Westover
In the Company of Giants by Rama Dev Jager and Ortiz Rafael
Uncanny Valley by Anna Wiener
Inside the Mind of Xi Jinping by François Bougon
Founders at Work by Jessica Livingston
Writing an Interpreter in Go by Thorsten Ball
Writing a Compiler in Go by Thorsten Ball
Debugging by David J. Agans
Facebook by Steven Levy
A Philosophy of Software Design (reread) by John Osterhout
The Three-Body Problem by Cixin Liu
The Art of Doing Science and Engineering by Richard Hamming
Netflixed by Gina Keating
The J Primer by Eric Iverson
Ten Essays on FizzBuzz by Joel Grus
The Infinite Machine by Camila Russo
The Go Programming Language by Alan A. A. Donovan and Brian Kernighan
What Every Web Developer Should Know About HTTP by K. Scott Allen
The Perfect Store: Inside eBay by Adam Cohen
Designing Data-Intensive Applications by Martin Kleppmann
The Path to Power by Robert Caro

Some interesting people's 2020 books:

2020 book thread
— Vegard Stikbakke (@vegardstikbakke) January 20, 2020

How do Unix pipes work?

Sat, 21 Mar 2020 00:00:00 +0000

Pipes are cool! We saw how handy they are in a previous blog post. Let's look at a typical way to use the pipe operator. We have some output, and we want to look at the first lines of the output. Let's download The Brothers Karamazov by Fyodor Dostoevsky, a fairly long novel.

> wc -l karamazov.txt
  36484 karamazov.txt

If we cat this file, it will be printed to the terminal.

> cat brothers_karamazov.txt
... many lines of text!
***FINIS***

It takes a noticeable amount of time to finish. Now let's just look at the first two lines by piping it into head.

> cat karamazov.txt | head -n 2
The Project Gutenberg EBook of The Brothers Karamazov by Fyodor
Dostoyevsky

Now it's done in an instant. It seems that the cat operation terminates when head is done! Of course, head -n 2 only needs two lines of input to output what it's supposed to output. But how does cat know to stop when head is finished?

In this blog post, we'll learn a bit about how pipes work, and write a small cat clone in Python and Go.

Processes in a pipeline are started simultaneously. We can confirm this by running

> sleep 100 | head

in one terminal window, and taking a look at running processes with ps in another.

> ps
  PID TTY           TIME CMD
52892 ttys007    0:00.00 sleep 100
52893 ttys007    0:00.00 head

From the Unix manual page on pipes, man(7) pipe, we learn that if we pipe process1 into process2, the second process will wait (block) until it receives input. Furthermore, if process2 is finished, it closes its end of the pipe. This will cause a SIGPIPE signal to be generated for process1.

#Writing a `cat` clone in Python

Let's try to write a simple cat clone that mimics this behavior. Instead of reading from file, let's read from standard input. That means we should be able to pipe into it. In Python, standard input is found in sys.stdin.

# cat.py
import sys

for line in sys.stdin:
    print(line, end='')

This behaves the same as the regular cat command if we pipe into it.

cat karamazov | python cat.py
... many lines of text!
***FINIS*

But does it also stop when we pipe it into head?

> cat karamazov | python cat.py | head -n 2
The Project Gutenberg EBook of The Brothers Karamazov by Fyodor
Dostoyevsky
Traceback (most recent call last):
  File "cat.py", line 4, in <module>
    print(line, end='')
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

We see that we get an error BrokenPipeError. But if we look closely at the output here, we see that it's printed twice. We can try catching the error with a try/except.

# cat.py
import sys

for line in sys.stdin:
    try:
        print(line, end='')
    except BrokenPipeError:
        pass

Now we only get the second part of the error output from before.

> cat karamazov.txt | python cat.py  | head -n 2
The Project Gutenberg EBook of The Brothers Karamazov by Fyodor
Dostoyevsky
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

Since we know that a terminated process to the right of a pipe sends a SIGPIPE when it's done, maybe this means that this signal isn't handled properly?

Indeed, a quick search leads us to a section of the Python documentation, namely Note on SIGPIPE. It gives a recipe for handling it. Our program now becomes

# cat.py
import sys
import os

for line in sys.stdin:
    try:
        print(line, end='')
    except BrokenPipeError:
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())

and it works as we want, by getting rid of the pesky error message.

> cat karamazov.txt | python cat.py  | head -n 2
The Project Gutenberg EBook of The Brothers Karamazov by Fyodor
Dostoyevsky

But an explanation of how this works would be nice. The recipe has a comment saying that

Python flushes standard streams on exit; redirect remaining output to devnull to avoid another BrokenPipeError at shutdown.

What we do here, apparently, is redirect the remaining error output to /dev/null. /dev/null refers to [https://en.wikipedia.org/wiki/Null_device](the null device), which accepts output, but discards it all. In other words, to not print something, we can redirect output there.

The first line in the except BrokenPipeError block opens /dev/null. The second line calls os.dup2, which takes two file descriptors. It "duplicate[s] file descriptor fd to fd2, closing the latter first if necessary." So in the second line we duplicate /dev/null into standard output, and possibly close standard output first.

But I don't really understand this. It turns out that another way of silencing the output is to close standard error.

import sys

for line in sys.stdin:
    try:
        print(line, end='')
    except BrokenPipeError:
        sys.stderr.close()

#Writing a `cat` clone in Go

I want to learn some Go, so let's try writing the same program in it. In Go, we can apparently simply copy from standard in to standard out. The first program becomes a tidy one liner.

// cat.go
package main

import (
    "os"
    "io"
)

func main() {
    io.Copy(os.Stdout, os.Stdin)
}

When we run this, it also complains about SIGPIPE:

> cat karamazov.txt | go run cat.go | head -n 2
The Project Gutenberg EBook of The Brothers Karamazov by Fyodor
Dostoyevsky
signal: broken pipe

Turns out ignoring the SIGPIPE signal also is one line.

// cat.go
package main

import (
    "os"
    "io"
    "os/signal"
    "syscall"
)

func main() {
    signal.Ignore(syscall.SIGPIPE)
    io.Copy(os.Stdout, os.Stdin)
}

I hope you learned something reading this!

This post got some attention on Hacker News, and many valid concerns were raised. You should read the comments there!

If you liked this post, you might like my other post Problem solving with Unix commands.

I fixed a bug in Python!

Mon, 09 Mar 2020 00:00:00 +0000

This fall, my colleague and I discovered a bug in the unittest module in Python. In January, my PR was merged onto the cpython master!

Books I read in 2019

Sun, 05 Jan 2020 00:00:00 +0000

This year I read 40 books, surpassing my goal of 30 (see my year in books on Goodreads). I'm pleasantly surprised at how much reading it's possible to get in, just by reading when commuting and in various idle windows during the afternoon. Last year I read 5k pages over 16 books, whereas this year I read 12k (12137). Below is a list of the books I read. Those in bold are especially recommended, either because they were particularly enjoying or interesting.

Show Stopper by G. Pascal Zachary
Stubborn Attachments by Tyler Cowen
Sapiens by Yuval Noah Harari
Creative Selection by Ken Kocienda
The Great Stagnation by Tyler Cowen
We Are The Nerds by Christine Lagorio-Chafkin
High Growth Handbook by Elad Gil
Shoe Dog by Phil Knight
Hackers and Painters by Paul Graham
AI Superpowers by Kai-Fu Lee
How Asia Works by Joe Studwell
Jeg vil dø - et essay om dødsønske og dødshjelp, mennesket og dets samfunn by Andreas and Randi Espegren Masvie
The Soul of a New Machine by Tracy Kidder
The Crying of Lot 49 by Thomas Pynchon
In the Beginning...Was the Command Line by Neal Stephenson
Working by Robert A. Caro
Big Business by Tyler Cowen
Japan: The Story of a Nation by Edwin Reischauer
12 Rules for Life by Jordan Peterson
The Effective Engineer by Edmond Lau
Range by David Epstein
An Elegant Puzzle by Will Larson
Gambling with Other People's Money by Russ Roberts
Brave New World by Aldous Huxley
Fuglane (The Birds) by Tarjei Vesaas
Good to Great by Jim Collins
The Power Broker by Robert A. Caro
Finite and Infinite Games by James P. Carse
Crime and Punishment by Fyodor Dostoyevsky
Accidental Empires by Robert Cringely
How I Sold My Business by Alvin Alexander
Secrets of Sand Hill Road by Scott Kupor
Resilient Management by Lara Hogan
High Output Management by Andrew S. Grove
Thinking in Systems by Donella H. Meadows
Resume Speed by Lawrence Block
The Man Who Solved the Market by Gregory Zuckerman
The Dream Machine by M. Mitchell Waldrop
UNIX: A History and a Memoir by Brian Kernighan
Start Small, Stay Small: A Developer's Guide to Launching a Startup by Rob Walling

Op-ed in Norway’s leading business newspaper about university admissions

Wed, 24 Jul 2019 00:00:00 +0000

I wrote an op-ed in DN, the biggest Norwegian financial daily newspaper about how entrance to Norwegian universities is broken and extremely expensive for the Norwegian state: https://www.dn.no/innlegg/utdannelse/samordna-opptak/medisin/forbedring-av-vitnemal-gir-meningslos-slosing/2-1-643154

Problem solving with Unix commands

Wed, 13 Feb 2019 00:00:00 +0000

I am starting to realize that the Unix command-line toolbox can fix absolutely any problem related to text wrangling. Let me tell you about a problem I had, and how I used some Unix command-line utilities to solve it.

#The problem

I'm working on research for my master thesis. As with many statisticians, I am running a lot of simulations. I first simulate some data according to some numerical seed (to ensure reproducibility), and then use an algorithm to estimate something based on that data. For each simulation run, I create some files, typically like so:

dataset-directory/0001_data.csv
dataset-directory/0001_A.csv

Sometimes a run fails. This doesn't really matter in this case: For any failed simulation, I can just do another one. For the 0001 data, I had a successful run with algorithm A. Therefore I want to use the 0001 data on algorithm B as well. But I need to keep track of which runs of A have failed.

After running algorithm A on a lot of data, I end up with a large list of files like

dataset-directory/0001_data.csv
dataset-directory/0001_A.csv
dataset-directory/0002_data.csv
dataset-directory/0002_A.csv
dataset-directory/0003_data.csv
dataset-directory/0003_A.csv
dataset-directory/0004_data.csv
dataset-directory/0005_data.csv
dataset-directory/0005_A.csv
dataset-directory/0006_data.csv
dataset-directory/0006_A.csv
dataset-directory/0007_data.csv
dataset-directory/0007_A.csv
dataset-directory/0008_data.csv
dataset-directory/0009_data.csv
dataset-directory/0009_A.csv
...
dataset-directory/0499_A.csv

The astute observer will note that the file for algorithm A on data 0004 and 0008 are missing. How can I get a list of all the numbers for which A didn't succeed?

I certainly could go over manually, but that would be error prone, and incredibly boring. It's much better to write a program to do it!

#The solution

To be obtuse: Those that didn't succeed are the numbers from 0001 to 0500 except those that succeeded. And one handy command to get a list of numbers is seq:

(If only one number is given, it is implied that the sequence starts with 1. seq 2 10 would do what you think it would, as well.)

Now, if we can get a list of all the successful runs, we should be able to get what we want by cross-checking the list of successful runs with a seq command which prints all possible numbers!

Most command-line utilities do one pretty specific thing. For example, with cut you can get the characters on specific locations on each line

$> cat text
Lorem ipsum
dolor sit amet
$> cat text | cut -c 2-5
orem
olor

Notice here the use of the so-called pipe operator |. Like I said, most utilities do one specific thing, and it does that thing well. The neat thing is that these can be combined. By using these pipes, the output from the command to the left of the pipe is directed to the command to the right. Note that these commands treat the input as a stream of lines, which is often really handy.

We can get a list of the successful file names by piping the list of files into a grep command, which is a command which can use regular expressions. Since all files start with an equal length of 4 digits, we can match these to the regular expression \d\d\d\d, matching 4 digits in a row, and add the file ending for the A algorithm to the regular expression. To get the list of files with one line for each file, we can simply do ls. (Although ls doesn't give each file its own line when calling it separately, it turns out that piping the output from ls will.)

$> ls dataset-directory | grep '\d\d\d\d_A.csv'
0009_A.csv
0001_A.csv
0002_A.csv
0005_A.csv
0007_A.csv
0003_A.csv
0006_A.csv
...

For some reason, these show up in a scrambled order after using grep. We can use sort to fix that. And we are only interested in the numbers, so we can use cut -c 1-4 to extract the number parts.

$> ls dataset-directory | grep '\d\d\d\d_A.csv' | sort | cut -c 1-4
0001
0002
0003
0005
0006
0007
0009
...
0499

These numbers aren't exactly the same as the numbers from the seq command, since these are zero-padded. Therefore we write a quick Python script to parse them as integers.

# parse.py
import sys

for line in sys.stdin:
    i = int(line)
    print(i)

Now, piping into this script will give us the numbers that we want:

$> ls dataset-directory | grep '\d\d\d\d_A.csv' | cut -c 1-4 | python3 parse.py
1
2
3
5
6
7
9
...
499

We're getting there! Now we have to figure out how to cross-check these lists of numbers. Luckily, there exists a command called comm, which checks for common characters in two input streams. To get the input of a sequence of commands such as the one above, we can evaluate it and redirect it, which we do by wrapping it in <(...).

$> comm <(ls dataset-directory | grep '\d\d\d\d_A.csv' | cut -c 1-4 | python3 parse.py) <(seq 500)
        1
        2
        3
    4
        5
        6
        7
    8
        9
    10
...
    500

This output is a bit disorienting. If we read the manual of comm (by doing man comm), we see that comm "produces three text columns as output: lines only in file1; lines only in file2; and lines in both files." To suppress column 1 -- which is empty, since no numbers are only from the file list -- call comm with the flag -1. And since we are not interested in the numbers which are in both streams, we suppress with the -3 flag as well.

$> comm -1 -3 <(ls dataset-directory | grep '\d\d\d\d_A.csv' | cut -c 1-4 | python3 parse.py) <(seq 500)
4
8
...
500

And we're done!

Update: This post generated some interesting discussion on Hacker News. There are many ways to solve this problem, and the way I did it is probably not the best. Be sure to check it out for tips on how to improve. I was also asked to post this on dev.to, which I did!

If you liked this post, you might like my other post How do Unix pipes work?.

Books I read in 2018

Mon, 21 Jan 2019 00:00:00 +0000

As a kid I read a lot of books. Not so the last couple of years. In 2018 I managed to get thoroughly into reading again. There is just so much to learn, and the more I read, the more I want to read. If you're interested, this is my Goodreads account.

In 2018, I apparently read 5267 pages total over 16 books (see My Year in Books on Goodreads). I'm pretty satisfied with that. The themes of the books seem to be mainly business, startups, tech and biographies.

Here's a list of the books I read. Those in bold are especially recommended.

Why We Sleep by Matthew Walker
The Lean Startup by Eric Ries
The Everything Store by Brad Stone
The Hard Thing About Hard Things by Ben Horowitz
The Mythical Man-Month by Frederik P. Brooks Jr.
Liar's Poker by Michael Lewis
Zero to One by Peter Thiel
In the Plex by Steven Levy
Fresh Off the Boat by Eddie Huang
Steve Jobs by Walter Isaacson
The New New Thing by Michael Lewis
The Manager's Path by Camille Fournier
Bad Blood by John Carreyrou
A Philosophy of Software Design by John Ousterhout
Masters of Doom by David Kushner
Stoner by John Williams (only fiction book this year)

I got a Kindle for Christmas, and I'm loving it so far. Looking forward to reading more this year!

Enforcing keyword arguments in Python 3

Tue, 27 Nov 2018 00:00:00 +0000

Today I learned that in Python 3, one can enforce the use of keyword arguments in functions!

Consider a function for doing safe division. Several things can go wrong when doing division directly, and therefore we might want to use a function which could handle possible errors. One possible error is dividing by zero. This is an undefined operation, and it will raise an error. Another is that in Python, division always returns a floating point number, and Python does not support arbitrarily large floating point numbers. Thus one might get an overflow error. (See here.) We might want to introduce two flags for handling these types of errors in our division function, ignore_overflow and ignore_zero_division. By default, these should be False.

def safe_division(number, divisor, ignore_overflow=False, ignore_zero_division=False):
    try:
        return number / divisor
    except OverflowError:
        if ignore_overflow:
            return 0
        else:
            raise
    except ZeroDivisionError:
        if ignore_zero_division:
            return float('inf')
        else:
            raise

Calling this straightforwardly like safe_division(1, 0, True, False) is not very readable. We would have to check the actual function body to see what this function call would actually do. We can call it like

safe_division(1, 0, ignore_overflow=False, ignore_zero_division=True)

or even

safe_division(1, 0, ignore_zero_division=True, ignore_overflow=False)

(notice how we swapped the order here). However, specifying the keywords is only optional behavior for those calling the function.

A solution for this is to enforce using keyword arguments. This is possible to do in Python 2 by specifying a keyword argument dictionary, typically called **kwargs.

However, in Python 3, there is a simple way to enforce it! By adding a * in the function arguments, we force all succeeding arguments to be named. E.g. by having the first argument be *, we force all arguments to be named.

def safe_division(*, number, divisor, ignore_overflow, ignore_zero_division):
    try:
        return number / divisor
    except OverflowError:
        if ignore_overflow:
            return 0
        else:
            raise
    except ZeroDivisionError:
        if ignore_zero_division:
            return float('inf')
        else:
            raise

If we now call safe_division(1, 0, True, False), or even safe_division(1, 0) we would get an error:

>>> safe_division(1, 2, True, False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: safe_division() takes 0 positional arguments but 4 were given

We now have to use it like this:

>>> safe_division(number=10**1000, divisor=3**-100,
ignore_overflow=True, ignore_zero_division=False)
0

This makes writing defensive Python easier!

Vegard Stikbakke

Detecting file changes on macOS with kqueue

#kqueue data structures

#Watching named files

#Watching a directory

#Implementing in Go

#Final thoughts

Recently

#Family

#Reading

#Programming

#Watching

#Work

How I rewrote my site using Claude Code

#Previous version

#Why use AI

#First Claude session

#Adding more functionality

Books I read in 2025

#Breakneck: China's Quest to Engineer the Future by Dan Wang

#All the Pretty Horses by Cormac McCarthy

#The Scaling Era: An Oral History of AI, 2019-2025 by Dwarkesh Patel, Gavin Leech

#The Soul of a New Machine by Tracy Kidder

#Show Stopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft by G. Pascal Zachary

#Hackers: Heroes of the Computer Revolution by Steven Levy

#Steve Jobs & The NeXT Big Thing by Randall E. Stross

#Full list

Books I read in 2024

#Fiction

#Non-fiction

#Source code for the graphic above

A file watcher

#Requirements

#Figuring out which files to watch

#Detecting file changes

#Running commands

Extract Table API

Blogs I like

Books I read in 2023

#History of technology

#Programming

#Biographies

#Other non fiction

#Fiction

Books I read in 2022

#Biographies

#Silicon Valley history

#Tech related

#Michael Lewis books

#Other non fiction

#Fiction

What's coming in Go 1.18 (for LWN.net)

My open source contributions

#Python

#Others

#Typos

Books I read in 2021

til.vegardstikbakke.com

Waiting for dependencies in tests

#until

#Bonus: HTTP server

Books I read in 2020

How do Unix pipes work?

#Writing a cat clone in Python

#Writing a cat clone in Go

I fixed a bug in Python!

Books I read in 2019

Op-ed in Norway’s leading business newspaper about university admissions

Problem solving with Unix commands

#The problem

#The solution

Books I read in 2018

Enforcing keyword arguments in Python 3

#`kqueue` data structures

#`until`

#Writing a `cat` clone in Python

#Writing a `cat` clone in Go