williamcotton.com

Introducing Web Pipe

July 23rd, 2025

Web Pipe is a DSL for writing web applications. At this point it's a toy, an experiment in language and runtime design. It's more of a declarative configuration language than anything else, based on JSON data flowing between steps in a pipeline. It is definitely not a general purpose language.

It looks like this:

GET /hello/:world
  |> jq: `{ world: .params.world }`
  |> handlebars: `<p>hello, {{world}}</p>`

describe "hello, world"
  it "calls the route"
    when calling GET /hello/world
    then status is 200
    and output equals `<p>hello, world</p>`

Each step in the pipeline is executed by dedicated middleware. Middleware are shared objects that are dynamically loaded when the application launches. Generally speaking, each step in the middleware takes in a JSON object, does some work, and returns another JSON object. In the above case the pipeline has middleware for jq, itself a DSL for building and transforming JSON objects, and handlebars, a templating language.

The first step in a pipeline is passed a request object with common HTTP request data like params and query objects.

GET /lua/:id/example
  |> lua: `
    local id = request.params.id
    local name = request.query.name
    return {
      message = "Hello from Lua!",
      id = id,
      name = name
    }
  `

describe "lua"
  it "calls the route"
    when calling GET /lua/123/example?name=example
    then status is 200
    and output equals `{
      "message": "Hello from Lua!",
      "id": "123",
      "name": "example"
    }`

It has what else you would expect a request object to have, cookies, headers, et al.

This particular endpoint uses Lua, another embeddable scripting language. It's important to note that the JSON passing through the pipeline is not serialized and deserialized between steps. Lua has data structures that closely mirror JSON so it's easy to implement recursive conversion functions.

And of course what would a web application be without a database?

config pg {
  host: $WP_PG_HOST || "localhost"
  port: $WP_PG_PORT || "5432"
  database: $WP_PG_DATABASE || "wp-test"
  user: $WP_PG_USER || "postgres"
  password: $WP_PG_PASSWORD || "postgres"
  ssl: false
}

pg pageQuery = `SELECT * FROM pages WHERE id = $1`

GET /page/:id
  |> jq: `{ sqlParams: [.params.id | tostring] }`
  |> pg: pageQuery
  |> jq: `{ page: .data.pageQuery.rows[0] }`

We've now got a pipeline with multiple steps. It takes in a request, plucks out the id, passes it as SQL parameters to the Postgres middleware, and then converts that response into the JSON object that is ultimately returned. The Postgres middleware is expecting a JSON object with a sqlParams array.

We can also see an example of how the Postgres middleware is configured. Web Pipe supports dotenv files with environment variables taking precedence.

In addition to simple variable declarations for things like SQL queries an entire pipeline can be reused.

pipeline getPageById =
  |> jq: `{ sqlParams: [.id] }`
  |> pg: `SELECT * FROM pages WHERE id = $1`
  |> jq: `{ page: .data.rows[0] }`

GET /page/:id
  |> jq: `{ id: (.params.id | tostring) }`
  |> pipeline: getPageById

describe "getPageById pipeline"
  with mock pg returning `{
    "data": {
      "rows": [
        { "id": "123", "name": "example" }
      ]
    }
  }`

  it "calls the pipeline"
    when executing pipeline getPageById
    with input `{ "id": "123" }`
    then output equals `{
      "page": { "id": "123", "name": "example" }
    }`

There's also some functions injected in the global Lua runtime:

GET /pages
  |> lua: `
    local limit = request.query.limit or 5
    local result = executeSql("SELECT * FROM pages LIMIT $1", { limit })

    return {
      data = result
    }
  `

Errors are handled by middleware returning a standardized errors array in the JSON object response. For example, when there's a database error the Postgres middleware adds an errors array with an object that contains a type and any other arbitrary data related to that particular error. The rest of the pipeline is short circuited to either a JSON response or a result block.

GET /nonexistant-table
  |> pg: `SELECT * FROM nonexistent_table`
  |> result
    ok(200):
      |> jq: `{success: true, data: .data}`
    sqlError(500):
      |> jq: `{
        error: "Database error",
        sqlstate: .errors[0].sqlstate,
        message: .errors[0].message,
        query: .errors[0].query
      }`
    default(500):
      |> jq: `{error: "Internal server error"}`

In the case of a database error the Postgres middleware sets the type to sqlError. This type is matched against definitions in the result block. Each pathway contains nested pipelines. These pipelines can even include other result blocks. It's pipelines all the way down.

Getting stuff out of a database is easy enough, but what about putting stuff into a database?

POST /pages
  |> validate: `{
    title: string(3..30),
    content: string(10..1000)
  }`
  |> jq: `{
    sqlParams: [.body.title, .body.content]
  }`
  |> pg: `INSERT INTO pages (title, content) VALUES ($1, $2) RETURNING *`
  |> jq: `{
    data: .data.rows[0]
  }`
  |> result
    ok(201):
      |> jq: `{
        data: .data.rows[0]
      }`
    validationError(400):
      |> jq: `{
        message: "Validation failed",
        errors: .errors
      }`
    default(500):
      |> jq: `{
        message: "Internal server error",
        errors: .errors
      }`

Here we introduce a new middleware that validates the body of a POST request. This body can either be JSON or form data. If the data doesn't pass the validation step it short circuits to the result block and returns the validation errors in the response.