Before You Scale It, EXPLAIN It

Vansh Koul — Tue, 10 Nov 2026 12:00:00 GMT

Why should we care about slow queries?

The idea is simple: get as much as possible out of your existing database instances before adding another replica, sharding your data, partitioning tables, and whatnot.

I know these things sound exciting and might even look good on your resume, but they also bring along a fair bit of complexity. More moving parts, more things to maintain, and more things that can go wrong.

Before we go down that road, shouldn’t we first make sure we’re getting the most out of what we already have?

A database has limited resources. CPU, memory, disk I/O, connections, all of it. Every query consumes some of these resources. If we’re making the database scan a million rows when it only needs a few hundred, we’re essentially making it do unnecessary work.

And this adds up. Inefficient queries can occupy connections for longer, consume more CPU and I/O, and reduce the number of requests the database can handle efficiently.

The idea is to make the database do less unnecessary work so it has more room to do the work that actually matters.

What causes slow queries?

A few usual suspects:

Poorly written queries: Fetching more data than necessary, inefficient joins, unnecessary operations, and so on.
Missing or improper indexes: The database ends up scanning far more rows than it needs to because the available indexes don’t align with the query.
Skewed data: Sometimes the data distribution doesn’t play nicely with the query optimizer, causing it to choose a suboptimal execution plan.
Resource contention: Lock contention, high concurrency, CPU saturation, and disk I/O bottlenecks.
Application-level inefficiencies: Sometimes the query itself isn’t particularly bad. It’s just being executed thousands of times because of how the application is written.

How I usually approach this

My investigation generally follows four steps:

Find the expensive queries. Figure out which query patterns are consuming the most resources. Execution time alone doesn’t tell the whole story; frequency and cumulative execution time matter too.
Figure out what’s going wrong. Look at the execution plan, understand what the database is doing, and identify what could be improved.
Find where the query lives in the codebase. A query doesn’t magically appear in the database. Trace it back to the service or API that’s generating it.
Fix it and see if it actually helped. Make the change, deploy it, and compare the results. No point celebrating an optimization that only looks good on paper.

Finding the expensive queries

First things first, check your logs.

Since our databases run on AWS RDS, I usually start with CloudWatch. Head over to Database Insights / Performance Insights and look at the top SQL queries by wait activity.

There are three metrics I usually pay attention to:

Calls/sec: How frequently is this query being executed?
Average latency: How much time does each execution take on average?
Rows read: How many rows does the database have to read to execute the query?

Don’t look at these metrics in isolation. It’s mostly the combination of all three that tells you where the actual problem is.

For example, a query taking 200 ms might not look particularly bad. But if it’s being executed 100 times per second, that’s a lot of database work adding up.

Similarly, a query might execute only a handful of times but take several seconds and read hundreds of thousands of rows. That’s worth investigating too.

The idea is to identify queries that are either consuming a disproportionate amount of database resources or contributing significantly to application latency.

Once you have a shortlist, you know where to start digging instead of randomly opening SQL queries and hoping to find something interesting.

Now, what’s wrong with the query?

So you’ve identified an expensive query. What’s next?

Let’s ask MySQL itself. Put two words before your SELECT query:

EXPLAIN ANALYZE
SELECT *
FROM orders
WHERE customer_id = 101;

One important thing: EXPLAIN ANALYZE actually executes the query. It doesn’t just show what MySQL plans to do. So be mindful when running it, especially against production databases. A query that takes a few seconds can still put unnecessary load on your database. Avoid blindly running expensive queries during peak traffic.

How do we read the output?

Let’s take a simple example.

Suppose our orders table contains a few million records, and we run the query above to fetch all orders belonging to a particular customer.

Imagine MySQL gives us something like this:

-> Filter: (orders.customer_id = 101)
   (actual time=0.120..85.400 rows=120 loops=1)
    -> Table scan on orders
       (actual time=0.100..78.200 rows=1000000 loops=1)

Let’s break this down.

Table scan on orders: MySQL is scanning the table instead of using an index to locate the relevant rows.
rows=1000000: The scan produced approximately one million rows for the filter operation.
Filter: MySQL applies the customer_id condition to those rows.
rows=120: Only 120 rows match our condition.
actual time: This shows the observed execution time for the operation. In this example, the scan takes around 78 ms, while the filter operation takes around 85 ms.

We’re scanning a million rows just to retrieve 120. That’s a lot of unnecessary work.

A suitable index on customer_id could help:

CREATE INDEX idx_orders_customer_id
ON orders(customer_id);

Run EXPLAIN ANALYZE again and compare the execution plans.

You should now see an index lookup instead of a full table scan, assuming the optimizer considers that index beneficial.

That’s a much better starting point.

Also, don’t just look at the rows value. Pay attention to loops too. An operation returning a few rows might be executed thousands of times as part of a nested-loop join. That’s where things can get interesting.

A small but important detail: rows × loops

One thing I always pay attention to in EXPLAIN ANALYZE is the combination of actual rows and loops.

In MySQL’s tree-format output, actual rows and execution time are reported per loop. So if an operation returns 1 row but executes 50,000 times, that’s potentially 50,000 rows being processed across those loops.

For example:

-> Index lookup on orders using idx_customer_id
   (actual time=0.020..0.025 rows=1 loops=50000)

At first glance, returning just one row looks harmless. But look at the number of loops.

We’re performing that lookup 50,000 times!

That’s approximately 50,000 rows returned across all those executions. Even though each individual lookup is cheap, the cumulative cost can become significant.

This is particularly important when investigating nested-loop joins, where an operation on the inner table can execute once for every row produced by the outer table.

So don’t just look at how many rows an operation returns. Look at how many times it’s executed too.

A useful rule of thumb: inspect actual rows × loops to understand the total number of rows produced by an operation. Remember, this isn’t necessarily the number of physical disk reads. InnoDB might already have the relevant pages in its buffer pool.

What about joins?

Let’s take a real example.

We have two tables:

package: One row per package, with columns like franchise_id and created_on.
package_tracking: Tracking information for each package, with columns like estimated_delivery_date, payment_due_date and estimated_return_date.

A lot of our queries fetch package details using those tracking dates. One of them looks like this: get all packages of a franchise with an estimated delivery date in the next 30 days.

SELECT
    p.id,
    p.tracking_number,
    pt.estimated_delivery_date
FROM package p
JOIN package_tracking pt
    ON pt.package_id = p.id
WHERE p.franchise_id = 42
  AND pt.estimated_delivery_date
      BETWEEN CURDATE() AND CURDATE() + INTERVAL 30 DAY;

Looks harmless. There’s an index on package.franchise_id and an index on package_tracking.package_id. Let’s put EXPLAIN ANALYZE in front of it and see what MySQL gives us. Something like this:

-> Nested loop inner join
   (actual time=0.9..2140 rows=1800 loops=1)
    -> Index lookup on p using idx_package_franchise_id (franchise_id=42)
       (actual time=0.5..310 rows=250000 loops=1)
    -> Filter: (pt.estimated_delivery_date between ...)
       (actual time=0.007..0.007 rows=0.0072 loops=250000)
        -> Index lookup on pt using idx_package_tracking_package_id (package_id=p.id)
           (actual time=0.006..0.006 rows=1 loops=250000)

There are a few things we want to inspect here.

1. Join order: MySQL starts from package and then goes to package_tracking. The optimizer decides this order based on its estimates and the available indexes, not on the order we wrote the tables in.

2. Access method: Both tables use index lookups. No table scans. On the surface, everything looks fine.

3. Rows and loops: This is where it falls apart. The lookup on package returns 250,000 rows, every package this franchise has ever had. Then the lookup on package_tracking runs 250,000 times (loops=250000), once for each of those packages. Only after fetching the tracking row does MySQL apply the date filter, and almost everything gets thrown away. rows=0.0072 per loop × 250,000 loops ≈ 1,800 rows.

We’re doing 250,000 lookups to return 1,800 rows.

Understanding nested-loop joins

This is how a nested-loop join works:

Fetch M rows from table A.
For every row fetched from A, look for matching rows in table B.
Repeat until all M rows have been processed.

The cost depends mostly on M. Here, A is package, and M is every package of the franchise, because the only filter MySQL can apply to package is franchise_id. The date filter lives in the other table, so it can only be applied after the join.

You might think: why not start from package_tracking instead, using an index on estimated_delivery_date? MySQL can do that, but then it has the opposite problem. It fetches packages due in the next 30 days for every franchise, then looks up each package just to discard the ones from other franchises. Either way, one of the two filters gets applied too late.

So there are two levers:

Reduce the number of rows fetched from the first table.
Make sure the matching rows in the second table can be found efficiently.

The second one is already taken care of. Each lookup into package_tracking uses an index and is cheap. The problem is purely that we do it 250,000 times.

Reducing the rows from the first table

The question I asked was: is there anything on the package table that tells me a package can’t possibly be delivered in the next 30 days?

There was. A package created years ago isn’t going to have an estimated delivery date next month. In our case, it was safe to say that any package with an upcoming delivery date was created within the last year.

So I added a filter on created_on:

SELECT
    p.id,
    p.tracking_number,
    pt.estimated_delivery_date
FROM package p
JOIN package_tracking pt
    ON pt.package_id = p.id
WHERE p.franchise_id = 42
  AND p.created_on >= CURDATE() - INTERVAL 1 YEAR
  AND pt.estimated_delivery_date
      BETWEEN CURDATE() AND CURDATE() + INTERVAL 30 DAY;

With a composite index to support it:

CREATE INDEX idx_package_franchise_created
ON package(franchise_id, created_on);

Now MySQL can narrow down the packages before it starts the join:

-> Nested loop inner join
   (actual time=0.8..410 rows=1800 loops=1)
    -> Index range scan on p using idx_package_franchise_created
       over (franchise_id = 42 AND created_on >= ...)
       (actual time=0.4..60 rows=45000 loops=1)
    -> Filter: (pt.estimated_delivery_date between ...)
       (actual time=0.007..0.007 rows=0.04 loops=45000)
        -> Index lookup on pt using idx_package_tracking_package_id (package_id=p.id)
           (actual time=0.006..0.006 rows=1 loops=45000)

Same 1,800 rows, but 45,000 lookups instead of 250,000.

The filter looks redundant. Logically, the date condition on package_tracking already decides which rows come back. But MySQL can’t apply that condition until after the join. The created_on condition can be applied before it, using an index. The idea is to filter out unnecessary rows as early as possible rather than carrying them through the join.

One big caveat: this filter is only correct because of a business rule. If a package can ever be created more than a year before its delivery date, this query silently drops it. No error, just missing data. So make sure the assumption actually holds, and leave a comment next to the query explaining why the filter is there. Otherwise someone will either remove it as “useless” or keep it after the rule changes.

Also, don’t blindly add filters everywhere. MySQL’s optimizer can already push down and derive some conditions on its own. Check whether the extra condition actually changes the execution plan and reduces the work being done.

That’s why EXPLAIN ANALYZE is so useful. It helps us move beyond guessing and understand what MySQL is actually doing.

Sometimes, the problem isn’t the query

The created_on trick worked because we had a column on package that loosely tracked the dates we cared about.

But what if you don’t have one? Or what if the query is on payment_due_date, and some payments are due years after the package was created?

You’ve optimized the query, added appropriate indexes, investigated the join order, and checked the execution plan. And yet, the query is still doing way too much work.

Maybe it’s time to look beyond the query itself.

Look at what the query actually needs: packages of one franchise with a date in some range. The franchise is on one table, and the date is on another. No index can span two tables, so no single index can answer this query. MySQL will always have to over-fetch on one side and filter on the other.

That’s a schema problem, not a query problem.

The fix is to get both columns into the same table. For example, add franchise_id to package_tracking:

ALTER TABLE package_tracking
ADD COLUMN franchise_id INT NOT NULL;

CREATE INDEX idx_tracking_franchise_edd
ON package_tracking(franchise_id, estimated_delivery_date);

Now the query can filter on package_tracking alone:

SELECT
    p.id,
    p.tracking_number,
    pt.estimated_delivery_date
FROM package_tracking pt
JOIN package p
    ON p.id = pt.package_id
WHERE pt.franchise_id = 42
  AND pt.estimated_delivery_date
      BETWEEN CURDATE() AND CURDATE() + INTERVAL 30 DAY;

And the plan becomes:

-> Nested loop inner join
   (actual time=0.3..25 rows=1800 loops=1)
    -> Index range scan on pt using idx_tracking_franchise_edd
       over (franchise_id = 42 AND ... <= estimated_delivery_date <= ...)
       (actual time=0.2..4 rows=1800 loops=1)
    -> Single-row index lookup on p using PRIMARY (id=pt.package_id)
       (actual time=0.010..0.010 rows=1 loops=1800)

MySQL reads exactly the 1,800 tracking rows it needs, and does 1,800 primary key lookups. No business-rule assumptions, and it works the same way for payment_due_date and estimated_return_date with their own composite indexes.

Of course, this isn’t free:

We’re duplicating data. franchise_id now lives in two places, and both need to stay in sync. Luckily, a package rarely changes franchise, so this is a fairly cheap duplication. Duplicating something that changes often would be a very different story.
It needs a migration. Add the column, backfill it in batches for existing rows, and update every code path that inserts tracking rows. On a large table, adding a column and building indexes needs some care too.
Every index has a write cost. Three composite indexes, one per date column, means every insert and update on package_tracking has a bit more work to do.

Whether it’s worth it depends on how often these queries run and how much they cost. If date-based queries like these are all over your application, it probably is.

This is just one example. Other common ones:

A frequently accessed attribute might be buried inside a large JSON column.
A list of IDs might be stored as a comma-separated string, which no index can help with.
A dashboard might aggregate millions of rows on every page load, when a summary table would do.

Schema changes introduce migration complexity, consistency concerns and additional maintenance. But there are cases where no amount of query rewriting will compensate for a data model that doesn’t align with the application’s access patterns.

That’s when it’s worth stepping back and asking whether the problem is really the query, or the way we’ve modelled the data in the first place.

Now, let’s find where the query lives in our codebase

We’ve identified the expensive query and have a fair idea of what’s happening inside the database. Now comes the next question: where is this query being triggered from?

A database query doesn’t magically appear. Somewhere in our application, there’s code responsible for executing it.

This is where I’d start looking:

Find the source: Search for the query in your codebase. If it’s generated dynamically or through an ORM, use the query logs or tracing information to identify the service and code path responsible for it.
Check how frequently it’s executed: Is it being called once per API request, inside a loop, or multiple times for the same data?
Look for unnecessary database calls: Are we fetching data that we don’t need? Could multiple queries be combined into one? Are we running into an N+1 query problem?
Check pagination and filtering: Are we fetching thousands of records when the application only needs a handful? Are we doing unnecessary processing in the application that the database could handle more efficiently?

For example, imagine an API that fetches 100 orders and then executes another query for each order to fetch its customer details.

That’s potentially 101 database queries for a single API request!

Depending on the use case, we might be able to fetch the required information using a single JOIN instead.

The exact approach will vary depending on your application architecture, ORM and codebase. The important thing is to understand not just how expensive a query is, but also why the application is executing it in the first place.

Sometimes the best database optimization isn’t changing the SQL at all. It’s fixing the application code that’s generating it.

Fix it, deploy it and verify

We’ve identified the expensive query, understood its execution plan, figured out what needs to change, and traced it back to our application.

Now it’s time to fix it.

Make the required changes, whether that’s adding an index, rewriting the query, changing the application logic, or modifying the schema.

But we’re not done yet.

Always verify the improvement after deploying the change.

Go back to the same metrics you used to identify the problem:

Average latency: Did the query become faster?
Rows read: Are we scanning fewer rows?
Calls/sec: Has the query frequency changed?
Database load: Are we consuming fewer resources?
Application latency: Did the API response time actually improve?

Run EXPLAIN ANALYZE again and compare the execution plans wherever appropriate.

Also, keep an eye on the side effects. Adding indexes can improve read performance but introduce additional overhead for writes. Changing the schema can introduce consistency and migration concerns.

Don’t optimize one query at the expense of the entire workload.

And that’s pretty much it. Database optimization is an iterative process. Find the expensive queries, understand why they’re expensive, fix the underlying problem, and verify that things actually improved.

No amount of clever SQL or fancy infrastructure matters if the database isn’t doing its job efficiently.

A few final thoughts

Scaling is sometimes necessary. But before adding replicas, sharding tables or throwing more resources at the database, it’s worth spending some time understanding what’s actually slowing it down.

I’ve seen cases where a few changes to a query or an index were enough to make a noticeable difference. No new infrastructure, no complicated setup. Just fixing what was already there.

And that’s really the point of this whole exercise. Before making the database bigger, make sure it isn’t doing unnecessary work.

Happy optimizing!

koul1sh