Intermittent Delays Processing Inventory Updates

Incident Report for Consignly

Postmortem

Postmortem: Intermittent Delays Processing Inventory Updates

At Consignly, we hold ourselves to a high standard for reliability and performance, and we sincerely apologise for the disruption this issue may have caused to your operations.

Summary

On 2026-06-15 9:10pm UTC (2026-06-16 9:10am NZT / 7:10am AEST), we identified an issue causing some inventory-related operations to take longer than expected. In some cases, these operations timed out or appeared delayed to users.

The issue was caused by database contention during an inventory totals recalculation process. Under certain conditions, related operations were waiting behind one another instead of completing promptly.

We deployed a targeted optimisation and updated the recalculation process to reduce this locking and blocking behaviour. The original contention issue has been resolved, and normal inventory operations are now completing reliably.

Following deployment, we identified a secondary impact: although the updated process substantially reduces contention, it can take longer to complete. This has no material impact on normal day-to-day operations, but can extend the duration of large bulk status updates.

As a temporary measure, bulk status updates are limited to batches of 500 consignments. This reduces the period during which related inventory activity may need to wait while each batch is processed.

Impact

Some users experienced delays when performing inventory-related actions, including operations that update stock quantities or recalculate inventory totals.

The original issue occurred when several related inventory updates were processed close together and database operations began waiting on one another.

The deployed fix resolved that contention. However, we subsequently found that the revised process can take longer when repeated across a large bulk operation.

Bulk status updates are processed transactionally so that, if an operation fails, it can be safely rolled back to a consistent state. During this processing window, related inventory activity may need to wait.

For smaller or normal operations, this is not materially noticeable. For larger bulk updates, the longer processing window increased the possibility of affecting concurrent picking, receiving, and other inventory activity.

What Happened

We observed that some inventory update operations were taking significantly longer than normal. Initial investigation showed that the affected operations were not primarily limited by CPU, memory, or general database capacity.

Instead, the database was spending most of the time waiting for related operations to complete.

Further investigation identified a specific inventory recalculation process as the main contributor. During periods of concurrent activity, this process could cause other inventory updates to queue behind it.

How We Investigated

We reviewed database performance telemetry, query history, execution behaviour, and live blocking information while the issue was occurring.

This allowed us to narrow the problem from a general performance issue to a specific contention pattern within the inventory recalculation flow.

The key finding was that the affected process was not consistently slow because of overall system load. It became slow when concurrent inventory updates attempted to use the same underlying data access path.

After deploying the initial optimisation, we continued monitoring the process. This follow-up monitoring showed that the new implementation successfully reduced the original locking and blocking, but could take longer to complete during large bulk operations.

Root Cause

The original root cause was an inefficient database access pattern within an inventory totals recalculation process.

The process was calculating the correct totals, but the database did not have an optimal way to locate and update the required records. Under concurrent load, this increased the likelihood of blocking and caused multiple inventory operations to wait behind one another.

The revised process uses a more controlled and predictable update flow. This resolves the original contention pattern, but can require additional processing time.

For normal activity, that additional time is not material. It becomes more noticeable when the process is repeated across a large number of consignments in a single bulk operation.

Resolution

We made two targeted changes.

First, we added a more specific database optimisation so the recalculation process can locate the required records more directly.

Second, we changed the recalculation routine to use a more predictable update flow, reducing the locking and blocking behaviour seen during concurrent operations.

These changes resolved the original issue and restored reliable performance for normal inventory activity.

To address the secondary impact on large bulk updates, we introduced a temporary limit of 500 consignments per bulk status update.

Processing large updates in smaller batches shortens each transactional window and reduces the amount of time that related inventory operations may need to wait.

Organisations needing to update more than 500 consignments can continue to do so by processing multiple batches.

Prevention and Follow-Up

We are continuing to monitor the affected inventory processes and are investigating further opportunities to improve the performance of the updated recalculation flow.

At present, we do not have a confirmed timeframe for an additional performance improvement. Until that work is complete and validated, bulk status updates may need to remain limited to batches of 500 consignments.

We have also improved our diagnostic capability for this class of issue. During the investigation, we introduced more targeted visibility into database blocking, allowing us to identify which operations are waiting and what they are waiting on more quickly.

Current Status

The original locking and blocking issue has been resolved.

Normal inventory operations are completing reliably. A temporary limit of 500 consignments per bulk status update remains in place to minimise disruption to concurrent picking, receiving, and other inventory activity.

We will provide a further update when additional performance improvements have been developed and validated.

Posted Jun 17, 2026 - 23:33 UTC

Resolved

All pick and receive operations continue to perform as expected. An explanation of the incident will be available soon.
Posted Jun 17, 2026 - 22:15 UTC

Update

The fix applied for the earlier performance issue remains stable. We have identified a related impact when bulk updating very large numbers of consignments, so a temporary limit has been applied to restrict bulk status updates to batches of 500 consignments at a time.

We will continue to monitor performance and provide a further update when this temporary limit is lifted.
Posted Jun 17, 2026 - 00:56 UTC

Update

We are continuing to monitor for any further issues.
Posted Jun 16, 2026 - 04:06 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 16, 2026 - 02:19 UTC

Update

We are currently testing a fix for the issue.
Posted Jun 16, 2026 - 01:44 UTC

Identified

We are aware of reports that Consignly is responding slower than normal to pick, receive, and allocation requests.
Our engineering team are investigating and have identified the issue and are working on a resolution.
We will provide an update ASAP as testing continues.
Posted Jun 16, 2026 - 01:08 UTC
This incident affected: Consignly Web Portal, Warehouse App (Android App), and Developer tools and services (Consignly API).