williamcotton.com

Introducing Web Pipe

July 23rd, 2025

Web Pipe is a DSL for writing web applications. At this point it's a toy, an experiment in language and runtime design. It's more of a declarative configuration language than anything else, based on JSON data flowing between steps in a pipeline. It is definitely not a general purpose language. It is very opinionated out of the box, yet highly extensible. It's fast and has a very low memory profile because it is written in C and designed to do nothing but take in HTTP requests and return HTTP responses.

It looks like this:

GET /hello
  |> jq: `{ world: ":)" }`

Each step in the pipeline is executed by dedicated middleware. Middleware are shared objects that are dynamically loaded when the application launches. Generally speaking, each step in the middleware takes in a Jansson JSON object, does some work, and returns another JSON object. In the above case this middleware is jq, itself a DSL for building and transforming JSON objects. It's embeddable and able to be precompiled.

The first step in a pipeline is passed a request object with common HTTP request data like params and query objects.

GET /lua/:id/example
  |> lua: `
    local id = request.params.id
    local name = request.query.name
    return {
      message = "Hello from Lua!",
      id = id,
      name = name
    }
  `

// /lua/1/example?name=Alice

{
  "id": "1",
  "message": "Hello from Lua!",
  "name": "Alice"
}

It has what else you would expect a request object to have, cookies, headers, et al.

This particular endpoint uses Lua, another embeddable scripting language. It's important to note that the JSON passing through the pipeline is not serialized and deserialized between steps. Lua has data structures that closely mirror JSON so it's easy to implement recursive conversion functions.

Each middleware shared object must expose at the very least a specific C function interface. The Lua middleware is no exception.

json_t *middleware_execute(json_t *input, void *arena, arena_alloc_func alloc_func,
  arena_free_func free_func, const char *lua_code, json_t *middleware_config,
  char **contentType, json_t *variables);

You can browse the Lua middleware implementation yourself, but in brief the middleware execution function takes in a JSON object, a per-request memory arena, a couple of allocator functions, some Lua code and returns another JSON object.

Jansson has custom memory allocation which pairs well with a per-request memory arena. At the beginning of an incoming HTTP request a memory arena is created and Jansson is configured to use this arena for all JSON objects created during the lifetime of the request. This applies to JSON objects created in the middleware as well. And if the middleware chooses to use the memory arena and allocator functions it can make use of the per-request arena and hand off the deallocation completely to the runtime. It's much more performant and easier than dealing with malloc and free. Blech.

And of course what would a web application be without a database?

config pg {
  host: $WP_PG_HOST || "localhost"
  port: $WP_PG_PORT || "5432"
  database: $WP_PG_DATABASE || "wp-test"
  user: $WP_PG_USER || "postgres"
  password: $WP_PG_PASSWORD || "postgres"
  ssl: false
}

pg pageQuery = `SELECT * FROM pages WHERE id = $1`

GET /page/:id
  |> jq: `{ sqlParams: [.params.id | tostring] }`
  |> pg: pageQuery
  |> jq: `{ page: .data.pageQuery.rows[0] }`

We've now got a pipeline with multiple steps. It takes in a request, plucks out the id, passes it as SQL parameters to the Postgres middleware, and then converts that response into the JSON object that is ultimately returned. The Postgres middleware is expecting a JSON object with a sqlParams array.

We can also see an example of how the Postgres middleware is configured. Web Pipe supports dotenv files with environment variables taking precedence.

In addition to simple variable declarations for things like SQL queries an entire pipeline can be reused.

pipeline getPageById =
  |> jq: `{ sqlParams: [.id] }`
  |> pg: `SELECT * FROM pages WHERE id = $1`
  |> jq: `{ page: .data.rows[0] }`

GET /page/:id
  |> jq: `{ id: (.params.id | tostring) }`
  |> pipeline: getPageById

Our Postgres middleware interacts with the runtime in another manner by exposing another specific C function that is also dynamically loaded at application launch.

json_t *execute_sql(const char *sql, json_t *params, void *arena, arena_alloc_func alloc_func);

This middleware function registers with a global database provider. Another middleware, say Lua, can check the registry and see that there is a database provider and set a global function for executing SQL queries.

GET /pages
  |> lua: `
    local limit = request.query.limit or 5
    local result = executeSql("SELECT * FROM pages LIMIT $1", { limit })

    return {
      data = result
    }
  `

Errors are handled by middleware returning a standardized errors array in the JSON object response. For example, when there's a database error the Postgres middleware adds an errors array with an object that contains a type and any other arbitrary data related to that particular error. The rest of the pipeline is short circuited to either a JSON response or a result block.

GET /nonexistant-table
  |> pg: `SELECT * FROM nonexistent_table`
  |> result
    ok(200):
      |> jq: `{success: true, data: .data}`
    sqlError(500):
      |> jq: `{
        error: "Database error",
        sqlstate: .errors[0].sqlstate,
        message: .errors[0].message,
        query: .errors[0].query
      }`
    default(500):
      |> jq: `{error: "Internal server error"}`

In the case of a database error the Postgres middleware sets the type to sqlError. This type is matched against definitions in the result block. Each pathway contains nested pipelines. These pipelines can even include other result blocks. It's pipelines all the way down.

Getting stuff out of a database is all fine and dandy, but what about putting stuff into a database?

POST /pages
  |> validate: `{
    title: string(3..30),
    content: string(10..1000)
  }`
  |> jq: `{
    sqlParams: [.body.title, .body.content]
  }`
  |> pg: `INSERT INTO pages (title, content) VALUES ($1, $2) RETURNING *`
  |> jq: `{
    data: .data.rows[0]
  }`
  |> result
    ok(201):
      |> jq: `{
        data: .data.rows[0]
      }`
    validationError(400):
      |> jq: `{
        message: "Validation failed",
        errors: .errors
      }`
    default(500):
      |> jq: `{
        message: "Internal server error",
        errors: .errors
      }`

Here we introduce a new middleware that validates the body of a POST request. This body can either be JSON or form data. If the data doesn't pass the validation step it short circuits to the result block and returns the validation errors in the response.