Optimising PostgreSQL queries with an open dataset

Benjamin Geer
2026-03-10

Français : Optimiser des requêtes PostgreSQL avec un jeu de données ouvert

Table of Contents

Overview #

This post is a brief illustration of some database optimisation techniques, using the BRÉF (Base Révisée des Élu·es de France / Revised database of representatives elected in France), which contains data about representatives elected in France from 1948 to 2020. The BRÉF is based on French public data, in which a team of researchers has corrected many errors and inconsistencies. It’s published as a PostgreSQL database that consists of several tables and offers many opportunities for joins and optimisations.

Here I’ll be using this database to develop and optimise a SQL query with several joins, to get information about mayors and the places where they were elected. I’ll build up the query from simpler parts, which I’ll optimise as I go along. For this exercise, I’ll assume that we can’t change the database schema, and that we’re willing to pay some cost, in storage space and update performance, to improve query performance. Therefore I’ll focus on what we can do by creating indexes. To identify and test potential optimisations, we’ll have two sources of information:

PostgreSQL’s query plans, which we can get by using its EXPLAIN command.
Performance tests run with my sqlstopwatch tool, which runs queries many times and reports the average response time of each query.

Optimising a simple query #

In the terminology used in the BRÉF,

a mandate is an elected position (member of parliament, senator, etc.), while a function is an executive role that an individual holds during a mandate. For example, a person could have a mandate as a city councillor, and at the same time have the function of mayor on the council.

Hence there is a Mandate table and a Function table, and we’ll start with those.

Let’s start by finding all the functions of type MAIRE (mayor). We’ll use EXPLAIN ANALYZE to have Postgres run the query, then tell us the query plan (EXPLAIN) along with how long it took (ANALYZE).

1
2
3
4


EXPLAIN ANALYZE
SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE'

1
2
3
4
5
6
7


QUERY PLAN
Seq Scan on "Function"  (cost=0.00..5596.96 rows=113829 width=4)
  (actual time=0.010..17.567 rows=114192 loops=1)
  Filter: (("FunctionType")::text = 'MAIRE'::text)
  Rows Removed by Filter: 158845
Planning Time: 0.217 ms
Execution Time: 20.136 ms

The query plan is a tree, and each node has an estimated cost in arbitrary units. Note that ’the cost of an upper-level node includes the cost of all its child nodes’. Seq Scan means that the database scans the whole Function table looking for matching strings. To avoid this, let’s create an index on the FunctionType column and run EXPLAIN ANALYZE with the same query again. (In this exercise we’ll just use B-Tree indexes, which the CREATE INDEX command uses by default.)

1
2
3


CREATE INDEX index_function ON "Function" (
    "FunctionType"
)

The result is a different query plan:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


QUERY PLAN
Bitmap Heap Scan on "Function"  (cost=1286.60..4893.46 rows=113829 width=4)
  (actual time=3.764..11.449 rows=114192 loops=1)
  Recheck Cond: (("FunctionType")::text = 'MAIRE'::text)
  Heap Blocks: exact=822
  ->  Bitmap Index Scan on index_function
      (cost=0.00..1258.14 rows=113829 width=0)
      (actual time=3.589..3.589 rows=114192 loops=1)
        Index Cond: (("FunctionType")::text = 'MAIRE'::text)
Planning Time: 0.314 ms
Execution Time: 13.551 ms

Postgres has done a bitmap scan, which Postgres developer Tom Lane explained in this mailing list post.

A bitmap scan fetches all the tuple-pointers from the index in one go, sorts them using an in-memory “bitmap” data structure, and then visits the table tuples in physical tuple-location order…. If the bitmap gets too large we convert it to “lossy” style, in which we only remember which pages contain matching tuples instead of remembering each tuple individually. When that happens, the table-visiting phase has to examine each tuple on the page and recheck the scan condition to see which tuples to return.

This is more efficient than the full table scan, but we’re still paying the cost of fetching the tuples (rows) from the table and rechecking the scan condition. Since we’re only reading the MandateId column, we can put it in the index as well, to create a covering index for this query:

1
2
3
4
5
6


CREATE INDEX index_function ON "Function" (
    "FunctionType"
)
INCLUDE (
    "MandateId"
)

With that, we get a different query plan:

1
2
3
4
5
6
7
8


QUERY PLAN
Index Only Scan using index_function on "Function"
  (cost=0.42..4490.35 rows=115003 width=4)
  (actual time=0.027..7.354 rows=114192 loops=1)
  Index Cond: ("FunctionType" = 'MAIRE'::text)
  Heap Fetches: 0
Planning Time: 0.184 ms
Execution Time: 9.025 ms

Now Postgres is able to do an index-only scan, because all the columns we need are in the index. It looks like an improvement, but let’s run a performance test to check. Let’s test several queries that search for various functions, including mayor. We’ll run the test first with no index, then with each of the two indexes above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE'
FETCH FIRST 750 ROWS ONLY

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE DELEGUE'
FETCH FIRST 750 ROWS ONLY

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'PRESIDENT DU CONSEIL REGIONAL'
FETCH FIRST 750 ROWS ONLY

SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'PREMIER ADJOINT AU MAIRE'
FETCH FIRST 750 ROWS ONLY

Clearly, having an index is much better than not having one. Compared to the index that just contains FunctionType, the index that includes MandateId (and thus enables an index-only scan) doesn’t make a big difference for some values of FunctionType, but it’s more than twice as fast for MAIRE.

Sorting #

We’d like to get the results sorted by function start date:

1
2
3
4
5


EXPLAIN ANALYZE
SELECT "MandateId"
FROM "Function"
WHERE "FunctionType" = 'MAIRE'
ORDER BY "FunctionStartDate"

With no index, the database says this query takes about twice as long as it did without the sort:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


QUERY PLAN
Sort  (cost=15354.29..15644.28 rows=115995 width=8)
      (actual time=32.564..37.109 rows=114192 loops=1)
  Sort Key: "FunctionStartDate"
  Sort Method: external merge  Disk: 2016kB
  ->  Seq Scan on "Function"
      (cost=0.00..5596.96 rows=115995 width=8)
      (actual time=0.016..17.797 rows=114192 loops=1)
        Filter: (("FunctionType")::text = 'MAIRE'::text)
        Rows Removed by Filter: 158845
Planning Time: 0.398 ms
Execution Time: 40.132 ms

Interestingly, if we use the index from the previous section with this query, it doesn’t change the query plan this time. Postgres has decided that our index isn’t likely to provide any benefit for this query. To make the index worth using here, let’s add the FunctionStartDate column to it, so the entries for each function type are sorted by start date in the index:

1
2
3
4
5
6
7


CREATE INDEX index_function ON "Function" (
    "FunctionType",
    "FunctionStartDate"
)
INCLUDE (
    "MandateId"
)

1
2
3
4
5
6
7
8


QUERY PLAN
Index Only Scan using index_function on "Function"
  (cost=0.42..4743.89 rows=115995 width=8)
  (actual time=0.052..9.744 rows=114192 loops=1)
  Index Cond: ("FunctionType" = 'MAIRE'::text)
  Heap Fetches: 0
Planning Time: 0.432 ms
Execution Time: 12.275 ms

Now once again we have an index-only scan, which looks more efficient. Let’s run a performance test comparing the previous index with this one. We’ll use the queries from the previous section, with ORDER BY "FunctionStartDate" added to each one.

It seems that the more rows to be sorted, the greater the advantage of using an index that’s already sorted in the order that the ORDER BY clause requests. The advantage is very significant for MAIRE (114,192 rows) and PREMIER ADJOINT AU MAIRE (37,511 rows), but less significant in the other cases, which return far fewer rows.

Adding joins #

Now that we have MandateId, we can get foreign keys from the corresponding row of the Mandate table, and do joins to get more information about the person who was elected and the place where the election occurred. Let’s start with the municipality (commune). The Area table contains data about different sorts of administrative divisions, including municipalities. Before we can find out more about the municipality, we need its ID from the Mandate table. So let’s start with that and just look at the query plan, without running the query:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


EXPLAIN
SELECT
    func."FunctionStartDate",
    m."AreaId"
FROM 
    "Function" func,
    "Mandate" m
WHERE
    func."FunctionType" = 'MAIRE'
    AND func."MandateId" = m."MandateId"
ORDER BY func."FunctionStartDate"
FETCH FIRST 750 ROWS ONLY

1
2
3
4
5
6
7
8
9


QUERY PLAN
Limit  (cost=0.85..968.89 rows=750 width=36)
  ->  Nested Loop  (cost=0.85..147087.20 rows=113957 width=36)
        ->  Index Only Scan using index_function on "Function" func
            (cost=0.42..4664.34 rows=113957 width=8)
              Index Cond: ("FunctionType" = 'MAIRE'::text)
        ->  Index Scan using "pk_Mandat" on "Mandate" m
            (cost=0.43..1.25 rows=1 width=36)
              Index Cond: ("MandateId" = func."MandateId")

The database is still doing an Index Only Scan for the Function table, but it’s doing an Index Scan for the Mandate table, using that table’s primary key index. This means that it traverses the index until it finds the matching MandateId, then fetches the corresponding row from the Mandate table to get the AreaId. We can add an index containing AreaId to get an index-only scan:

1

CREATE INDEX index_mandate ON "Mandate" ("MandateId") INCLUDE ("AreaId")

Now the query plan looks like this:

1
2
3
4
5
6
7
8
9


QUERY PLAN
Limit  (cost=0.85..505.54 rows=750 width=10)
  ->  Nested Loop  (cost=0.85..76310.71 rows=113401 width=10)
        ->  Index Only Scan using index_function on "Function" func
            (cost=0.42..4643.26 rows=113401 width=8)
              Index Cond: ("FunctionType" = 'MAIRE'::text)
        ->  Index Only Scan using index_mandate on "Mandate" m
            (cost=0.43..0.63 rows=1 width=10)
              Index Cond: ("MandateId" = func."MandateId")

It looks like the cost has gone down. Let’s run a performance test, once again with different values for FunctionType, to compare the primary key index with the covering index. In both cases we’ll keep our best-performing index on Function.

The covering index doesn’t make a huge difference, but it still seems to be worth using.

Now that we have the AreaId of the municipality, we can get its name and its official geographical code (which is distinct from its postcode). We’d also like to get the name of the person who was elected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


SELECT
    func."FunctionStartDate",
    area."AreaCode" as code_commune,
    area."AreaName" as commune,
    indiv."FirstName",
    indiv."LastName"
FROM "Function" func
JOIN "Mandate" m ON func."MandateId" = m."MandateId"
JOIN "Area" area ON m."AreaId" = area."AreaId"
JOIN "Individual" indiv ON m."IndividualId" = indiv."IndividualId"
WHERE func."FunctionType" = 'MAIRE'
ORDER BY func."FunctionStartDate"
FETCH FIRST 100 ROWS ONLY

Since we have several joins, I’m using the JOIN keyword to make the query a bit easier to read. (This doesn’t have any effect on the query plan.) And since the query returns more columns now, I’m selecting fewer rows so the performance test results aren’t dominated by the time it takes to return the query results.

For the join between Mandate and Individual, we’ll add IndividualId to our index on Mandate:

1
2
3


CREATE INDEX index_mandate ON "Mandate" ("MandateId") INCLUDE (
    "AreaId", "IndividualId"
)

Then we’ll add covering indexes on Area and Individual for our query:

1
2
3
4
5


CREATE INDEX index_area ON "Area" ("AreaId") INCLUDE ("AreaCode", "AreaName");

CREATE INDEX index_indiv ON "Individual" ("IndividualId") INCLUDE (
    "FirstName", "LastName"
);

With this, we get the following query plan:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


QUERY PLAN
Limit  (cost=1.57..132.35 rows=100 width=38)
  ->  Nested Loop  (cost=1.57..148131.29 rows=113274 width=38)
        ->  Nested Loop  (cost=1.15..96170.50 rows=113274 width=34)
              ->  Nested Loop  (cost=0.85..82875.64 rows=113274 width=22)
                    ->  Index Only Scan using index_function on "Function" func
                        (cost=0.42..4636.71 rows=113274 width=8)
                          Index Cond: ("FunctionType" = 'MAIRE'::text)
                    ->  Index Only Scan using index_mandate on "Mandate" m
                        (cost=0.43..0.69 rows=1 width=22)
                          Index Cond: ("MandateId" = func."MandateId")
              ->  Memoize  (cost=0.30..0.32 rows=1 width=24)
                    Cache Key: m."AreaId"
                    Cache Mode: logical
                    ->  Index Only Scan using index_area on "Area" area
                        (cost=0.29..0.31 rows=1 width=24)
                          Index Cond: ("AreaId" = (m."AreaId")::text)
        ->  Index Only Scan using index_indiv on "Individual" indiv
            (cost=0.42..0.46 rows=1 width=28)
              Index Cond: ("IndividualId" = (m."IndividualId")::text)

All the data comes from the indexes; the tables aren’t involved at all. If we don’t create those last two indexes, we get Index Scan rather than Index Only Scan on Area and Individual. Let’s compare the performance of this query with and without those two indexes.

There’s very little difference, but let’s keep the indexes.

Let’s now compare the performance of this last query with no indexes (apart from the primary key indexes that were created by default) and with all the indexes we’ve made:

This query is 14 to 24 times faster with the indexes than without them.

More joins #

Let’s try to get the name of the department that the municipality belongs to. We can add a join on the Inclusion table to find the Area that represents the department:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


EXPLAIN
SELECT
    func."FunctionStartDate",
    area1."AreaCode" as code_commune,
    area1."AreaName" as commune,
    area2."AreaName" as departement,
    indiv."FirstName",
    indiv."LastName"
FROM "Function" func
JOIN "Mandate" m ON func."MandateId" = m."MandateId"
JOIN "Area" area1 ON m."AreaId" = area1."AreaId"
JOIN "Inclusion" incl on m."AreaId" = incl."IncludedAreaId"
JOIN "Area" area2 on incl."IncludingAreaId" = area2."AreaId"
JOIN "Individual" indiv ON m."IndividualId" = indiv."IndividualId"
WHERE func."FunctionType" = 'MAIRE'
AND area2."AreaType" = 'Département'
ORDER BY func."FunctionStartDate"
FETCH FIRST 100 ROWS ONLY

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


QUERY PLAN
Limit  (cost=35292.57..35304.24 rows=100 width=51)
  ->  Gather Merge  (cost=35292.57..35351.14 rows=502 width=51)
        Workers Planned: 2
        ->  Sort  (cost=34292.55..34293.18 rows=251 width=51)
              Sort Key: func."FunctionStartDate"
              ->  Nested Loop  (cost=30010.19..34282.96 rows=251 width=51)
                    ->  Parallel Hash Join  (cost=30009.76..34167.82 rows=251 width=47)
                          Hash Cond: (func."MandateId" = m."MandateId")
                          ->  Parallel Index Only Scan using index_function on "Function" func
                              (cost=0.42..3980.29 rows=47235 width=8)
                                Index Cond: ("FunctionType" = 'MAIRE'::text)
                          ->  Parallel Hash  (cost=29971.32..29971.32 rows=3042 width=47)
                                ->  Hash Join  (cost=2914.37..29971.32 rows=3042 width=47)
                                      Hash Cond: ((m."AreaId")::text = (area1."AreaId")::text)
                                      ->  Parallel Seq Scan on "Mandate" m
                                          (cost=0.00..24881.12 rows=572112 width=22)
                                      ->  Hash  (cost=2911.27..2911.27 rows=248 width=43)
                                            ->  Nested Loop
                                                (cost=967.16..2911.27 rows=248 width=43)
                                                  ->  Hash Join
                                                      (cost=966.88..2832.29 rows=248 width=19)
                                                        Hash Cond:
                                                        ((incl."IncludingAreaId")::text =
                                                         (area2."AreaId")::text)
                                                        ->  Seq Scan on "Inclusion" incl
                                                            (cost=0.00..1607.70
                                                             rows=98170 width=15)
                                                        ->  Hash
                                                            (cost=965.40..965.40
                                                             rows=118 width=19)
                                                              ->  Seq Scan on "Area" area2
                                                                  (cost=0.00..965.40
                                                                   rows=118 width=19)
                                                                    Filter:
                                                                    (("AreaType")::text =
                                                                    'Département'::text)
                                                  ->  Index Only Scan using index_area
                                                      on "Area" area1
                                                      (cost=0.29..0.32 rows=1 width=24)
                                                        Index Cond:
                                                        ("AreaId" =
                                                         (incl."IncludedAreaId")::text)
                    ->  Index Only Scan using index_indiv on "Individual" indiv
                        (cost=0.42..0.46 rows=1 width=28)
                          Index Cond: ("IndividualId" = (m."IndividualId")::text)

What has happened here? Postgres has decided that a hash join would be more efficient than adding another nested loop. Instead of using our index on the Mandate table, it plans to read about 500,000 rows into a hash table in memory. It uses the index sorted by date on Function, but since a hash table isn’t sorted, it then has to sort the results again. If we run EXPLAIN ANALYZE, we can see that the query takes about 600 ms to run; just getting the department’s name has made our query hundreds of times slower.

We have several options:

Give the query planner a better option than the hash join.
Optimise the hash join by selecting fewer columns (which isn’t really an option in this case) or fewer rows (we could, for example, add a WHERE condition like func."FunctionStartDate" < DATE '1975-01-01').
Do the join in the application. In this case, since there are only 101 departments in France, and they change very rarely, the application could read them on startup. Each municipality’s geographical code (which we already have from the previous query) contains the code of its department, so it would be easy for the application to look up the department in memory.

Let’s try the first option. We can add this index:

1
2
3


CREATE INDEX index_inclusion ON "Inclusion" ("IncludedAreaId") INCLUDE (
    "IncludingAreaId"
);

Now the query plan is back to nested loops:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


QUERY PLAN
Limit  (cost=1002.32..10605.10 rows=100 width=51)
  ->  Nested Loop  (cost=1002.32..58907.08 rows=603 width=51)
        ->  Gather Merge  (cost=1001.89..58630.47 rows=603 width=47)
              Workers Planned: 2
              ->  Nested Loop  (cost=1.87..57560.84 rows=251 width=47)
                    ->  Nested Loop  (cost=1.58..57476.99 rows=272 width=41)
                          ->  Nested Loop  (cost=1.28..53851.27 rows=107699 width=37)
                                ->  Nested Loop  (cost=0.85..36596.43 rows=47235 width=22)
                                      ->  Parallel Index Only Scan using index_function
                                          on "Function" func
                                          (cost=0.42..3980.29 rows=47235 width=8)
                                            Index Cond: ("FunctionType" = 'MAIRE'::text)
                                      ->  Index Only Scan using index_mandate on "Mandate" m
                                          (cost=0.43..0.69 rows=1 width=22)
                                            Index Cond: ("MandateId" = func."MandateId")
                                ->  Memoize  (cost=0.43..0.46 rows=2 width=15)
                                      Cache Key: m."AreaId"
                                      Cache Mode: logical
                                      ->  Index Only Scan using index_inclusion
                                          on "Inclusion" incl
                                          (cost=0.42..0.45 rows=2 width=15)
                                            Index Cond: ("IncludedAreaId" = (m."AreaId")::text)
                          ->  Memoize  (cost=0.30..0.34 rows=1 width=19)
                                Cache Key: incl."IncludingAreaId"
                                Cache Mode: logical
                                ->  Index Scan using "pk_Territoire" on "Area" area2
                                    (cost=0.29..0.33 rows=1 width=19)
                                      Index Cond:
                                      (("AreaId")::text = (incl."IncludingAreaId")::text)
                                      Filter: (("AreaType")::text = 'Département'::text)
                    ->  Index Only Scan using index_area on "Area" area1
                        (cost=0.29..0.31 rows=1 width=24)
                          Index Cond: ("AreaId" = (m."AreaId")::text)
        ->  Index Only Scan using index_indiv on "Individual" indiv
            (cost=0.42..0.46 rows=1 width=28)
              Index Cond: ("IndividualId" = (m."IndividualId")::text)

The cost is still rather high, though, and EXPLAIN ANALYZE says the query takes about 12 ms. A performance test shows that it can be 10-20 times slower than our previous query:

It would be more efficient to keep the previous query, and do the join for the departments in the application.

Conclusion #

This exercise has demonstrated a few important points about database optimisation:

Each time we change a query, we have to check the query plan again, and run a performance test, to see what effect the change has on performance.
Different queries may need different indexes.
An index-only scan can provide the best query performance in some cases, but we have to test it to find out.
Sometimes the best optimisation is to request less data.

You can find all the scripts and configuration I used for this exercise in this git repository.

To learn more about this subject, I recommend Markus Winand’s website Use the Index, Luke!, and his book SQL Performance Explained.

Categories:

Projects

Topics: