williamcotton.com

GraphQL DataLoader Pattern in Custom DSL

January 4th, 2026

I'm not sure if you've spent much time working with GraphQL but one of the trickier issues to solve out of the box is the N+1 problem. The N+1 problem arises when there are nested queries and a request is made per entity. Imagine a team with hundreds of employees. Retrieving teams and their employees necessitates first a query to retrieve the list of teams with IDs and then to make individual queries for each employee on a team. This is not very efficient!

The DataLoader pattern mitigates the issue. It's pretty basic in operation. When a GraphQL query comes in with a nested response, say asking for a list of all teams and their related employees, first a query is made for teams, then a DataLoader keyed by team IDs is called with a list of keys that represent those teams. The DataLoader then makes a single query with that list of keys, formats the response, and continues the GraphQL execution.

It's not too dissimilar to how N+1 problems are solved with other tools. For example, the ActiveRecord pattern typically introduces an "includes" method that follows the same general approach of taking a list of IDs and making a single additional query.

While the relationships between data are rather easy to express in ActiveRecord, it can be rather verbose and somewhat tedious to describe them in general-purpose languages like TypeScript.

This is the perfect opportunity for the introduction of a domain-specific language (DSL) for building the relationships between graphs of data. The DSL we'll be looking at is called Web Pipe.

Web Pipe does a lot more than just define GraphQL schemas and execute queries. You can read a basic introduction here. For now we'll just focus on this single topic.

First, we need to configure our GraphQL API endpoint and define our schema.

config graphql {
  endpoint: "/graphql"
}

graphqlSchema = `
  type Employee {
    id: ID!
    teamId: ID!
    name: String!
    email: String!
  }

  type Team {
    id: ID!
    name: String!
    employees: [Employee!]!
  }

  type Query {
    teams: [Team!]!
    team(id: ID!): Team!
    employees(teamId: ID!): [Employee!]!
    employee(id: ID!): Employee!
  }
`

Then we need to define the resolvers for the base-level queries and mutations.

query teams =
  |> pg: `
    SELECT id, name FROM teams
  `
  |> jq: `.data.rows`

query team =
  |> pg([.id]): `
    SELECT id, name FROM teams WHERE id = $1
  `
  |> jq: `.data.rows[0]`

query employees =
  |> pg([.teamId]): `
    SELECT id, name, email FROM employees WHERE team_id = $1
  `
  |> jq: `.data.rows`

query employee =
  |> pg([.id]): `
    SELECT id, name, email FROM employees WHERE id = $1
  `
  |> jq: `.data.rows[0]`

At this point we can call the GraphQL endpoint and see some results.

curl -X POST http://localhost:7770/graphql \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "{ teams { id name } }"
  }' | jq

{
  "data": {
    "teams": [
      {
        "id": 1,
        "name": "design"
      },
      {
        "id": 2,
        "name": "product"
      },
      {
        "id": 3,
        "name": "engineering"
      }
    ]
  }
}

In order to make this more useful and to tackle our N+1 problem, we can use a loader pipeline to fetch the employees for each team in a single swoop.

resolver Team.employees =
  |> loader(.parent.id): EmployeesByTeamLoader

pipeline EmployeesByTeamLoader =
  |> pg([(.keys | map(tostring) | join(","))]): `
    SELECT id, name, email, team_id
    FROM employees
    WHERE team_id::text = ANY(string_to_array($1, ','))
    ORDER BY team_id, id
  `
  |> jq: `
    reduce .data.rows[] as $row (
      {};
      .[$row.team_id | tostring] += [$row]
    )
  `

The resolver is what is called for each team. Behind the scenes, these calls are batched and the loader pipeline is invoked once with the full list of keys. With Web Pipe these keys are automatically provided to the loader pipeline in JSON format which can then be used to construct the SQL query in the EmployeesByTeamLoader pipeline. The following JQ expression is used to map the flat list of rows into the Data Loader object structure expected by Web Pipe.

With this in place it now allows us to call the GraphQL endpoint and see the employees for each team in an efficient manner.

curl -X POST http://localhost:7770/graphql \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "{ teams { id name employees { id name email } } }"
  }' | jq

{
  "data": {
    "teams": [
      {
        "id": 1,
        "name": "design",
        "employees": [
          {
            "id": 1,
            "name": "Alice",
            "email": "alice@example.com"
          },
          {
            "id": 2,
            "name": "Bob",
            "email": "bob@example.com"
          }
        ]
      },
      ...
    ]
  }
}