Optimizing Large-Scale Data Processing: Lessons from Real-World Scenarios

UpdatedFebruary 16, 2025

•4 min read•View as Markdown

Optimizing Large-Scale Data Processing: Lessons from Real-World Scenarios

Patrick

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Stop Wasting Queries: How to Cancel MySQL Requests When the Client Disconnects

Introduction In a high-traffic web application, many users may be clicking around at once. Imagine a user who requests a long list of items, then quickly navigates away before the server has finished fetching them from the database. By default, that ...

Apr 30, 20255 min read37

Get errors from NestJS and inject to Angular forms

In every new app, we need to know how we should handle errors from the backend.After creating a lot of frontends and backends I decided to share my experience with forms and validation errors. Probably if you are reading this article you are also thi...

Sep 10, 20234 min read80

[Angular] Custom icon component

Hi, sometimes we don't want to implement huge library to use one small component. In a simple example this is Material from Angular. When we decide to use only Material library then the problem doesn't exist, but when we want to use bootstrap or tail...

Oct 2, 20222 min read199

Simple encryption and decryption with secret in TypeScript

In this article I will show you how to easier using built-in nodejs module crypto. For this code we need SECRET_KEY and SECRET_IV environments. You can write everything in this variables but if you change values after encrypted data, you decryption ...

Jun 25, 20211 min read370

Simple encryption and decryption with secret in TypeScript

TS_DEV

6 posts

Optimizing Data Processing – Practical Insights

Recently, I had the opportunity to work on a project where processing very large and coherent data sets became crucial. These don’t have to be strictly production data – what really matters is that they are as close to real-world scenarios as possible and cover a wide range of situations your application might encounter in its target environment. I’d like to share some lessons and observations from my own experience.

1. Testing on large and coherent data

Limitations of unit tests
While unit tests help detect errors in specific functions or modules, they don’t fully reflect scenarios where multiple processes run simultaneously and access the same data set.

Scale vs. parallel operations
It’s only when you have large, coherent data sets capable of handling concurrent queries that you can observe:

read/write collisions,
delays that accumulate during processing,
how locking mechanisms in databases actually behave under load.

2. MySQL partitioning and query performance

Partitioned table structures
If you’re using MySQL, consider partitioning—especially if you have a very large number of rows. Properly designed partitions can significantly speed up SELECT and UPDATE queries, particularly when those queries match partition keys.

Query parameterization
An example might be reducing query time from around 12 seconds to about 1 second by adapting the WHERE clauses and introducing dedicated indexes.
Also keep in mind simultaneous write operations or potential replication, which can impose additional performance constraints.

3. Parallel access to cache (e.g., Redis)

Send multiple requests at once
If you use a cache (e.g., Redis) to store various relationships or linked entities, consider retrieving multiple keys in parallel. Instead of fetching data one by one:

Collect all the keys (e.g., customer, related object, status) needed for a given record.
Send multiple requests in a single batch, minimizing network overhead.

Leverage parallelism in your application logic
Similarly—if you’re processing a batch of multiple records at once, you can try updating them concurrently, provided there are no dependencies among them. This way:

The load on the database and cache can be better distributed.
The total processing time is reduced.

4. Memory usage and scaling in k8s

Node.js Inspector
When your application starts to “bloat” in terms of memory usage, Node.js Inspector can help you diagnose which parts are consuming excessive RAM. You can trace how different modules operate and quickly spot memory leaks.

Manual GC invocation
In certain scenarios (e.g., after processing a large batch of data), you can invoke the Garbage Collector manually:

  if (global.gc) {
    global.gc();
  } else {
    console.warn('Manual GC is not exposed. Run Node with --expose-gc');
  }

While not always recommended, it can sometimes speed up memory cleanup in controlled situations.

Scaling in k8s
Remember that in Kubernetes, Deployments are what get scaled (i.e., the number of replicas), which translates to the number of Pods (containers) running in the cluster. If your service is stateless and uses fewer resources, you can more easily increase the number of instances to handle higher loads. Meanwhile, for databases or caches, you often need to adjust configurations and resource allocations to keep the overall system balanced and performant.

In closing

Speaking from the perspective of someone who has personally dealt with the complexities of massively parallel operations:

Use data sets that are as close as possible to production – they don’t need to be the actual production data, but they should be coherent, span the full range of possible states, and be large enough to reveal bottlenecks.

Unit tests are great for catching bugs in isolated logic, but they won’t capture issues arising from concurrent reads and writes.

Parallel fetching and updates in both cache and databases can drastically speed up processing, provided it aligns with your business logic.

Monitor resource usage – leverage Node.js Inspector, dedicated APM (Application Performance Monitoring) tools, and remember that manual GC calls might help in certain scenarios.

Scaling in k8s involves increasing Deployment replicas—keep an eye on whether your database, cache, and network are prepared for higher traffic.

I hope these insights help you avoid some pitfalls and optimize your applications more efficiently. Good luck with your future projects!

Command Palette