The Mystery of the Failing System: A Software Engineering Investigation

January 31, 2025 Pritam MondalLeave a comment

You know how detective stories—whether in police dramas or medical series like House or The Good Doctor—always have that moment where everything seems fine… until suddenly, it’s not?

Software engineering is no different. If you love detective stories, debugging systems might just be your kind of thrill.

Let me tell you a real story.

Chapter 1: The First Signs of Trouble

A few days ago, a critical system we built started failing. Users reported that some operations weren’t working. No clear error messages. No immediate patterns. Just a lot of complaints.

It felt like being on a plane in clear skies and suddenly hearing the engines sputter—alarming, yet no obvious clue why.

We kicked off our investigation.

Were the triggers firing correctly?
Was Lambda executing as expected?
Was the engine service running properly?
Was SNS failing to send messages?
Was the API erroring out somewhere?

Everything seemed to be working. No outright failures. But something was definitely wrong.

To make things more confusing, this was happening in both dev and QA environments.

If the issue started in QA, why was it now happening in dev too?
And why weren’t we seeing any obvious failures in the logs?

We checked everything we could access, but we didn’t have direct access to the QA job queue. However, we did have access to dev’s queue, and the failure was present there too.

This ruled out a one-off deployment bug. It wasn’t just a bad rollout affecting one environment.

Something deeper was wrong.

The system was broken in multiple places, yet there was no smoking gun.

Chapter 2: The Clue in the Logs

After hours of combing through logs, we finally found something:

Lambda was failing to access an SSM parameter due to insufficient permissions.

Strange. The IAM policy showed that Lambda had access. Why was it still failing?

Digging deeper, we noticed something odd:

The permission was assigned at the parent level for the SSM parameter.
But the error was occurring on a specific child parameter.

That’s when it hit us—encryption keys.

Each environment has its own encryption keys. Lambda had permission to read from SSM, but it couldn’t decrypt the child parameter because it was encrypted with a different key from another environment.

This explained why it was failing in QA—but why did dev suddenly start failing as well?

That didn’t make sense.

In QA, the problem was clear: Lambda was trying to access a parameter encrypted with a key from a different environment, which it didn’t have access to.

But in dev, this shouldn’t even happen. Unlike QA, dev doesn’t have multiple environments, so the KMS encryption issue shouldn’t exist there at all.

That meant something else was wrong in dev.

We updated the key permissions, redeployed Lambda, and tested the fix.

Lambda was now working fine in both dev and QA.

But both environments were still broken.

Chapter 3: The Hidden Adversary

With Lambda fixed, dev should have started working again.

But it was still broken.

At this point, a rollback was tempting, but we had a major release coming up, and reverting now would cause unnecessary panic and delays. Tension was rising—time was ticking away.

We checked the logs again. Nothing new. Everything looked fine.

So we changed strategies. Instead of blindly searching, we actively simulated the issue.

We created a debugging tunnel.
Attached a debugger to the running API instance.
Stopped the engine on the server.
Ran the engine locally to see what happened.

The API was working fine.

But then we noticed something strange—jobs were getting pulled from the queue but were not hitting our local engine instance.

That meant something else was consuming the jobs before they reached our local instance.

This led us to suspect there might be a hidden engine instance running somewhere.

We checked the server.

And there it was.

A rogue process—an older instance of the engine, still running on the server.

Someone had manually started it—probably for testing—and forgotten to shut it down.

It wasn’t part of our deployment system. It wasn’t updated. And it wasn’t visible in the usual monitoring dashboards.

Since our system supports multiple instances only when they share the same deployment, this rogue instance was causing conflicts.

We killed the process.

Dev was now fully operational.

But QA? Still down.

Chapter 4: The Missing Piece

If the rogue process was the problem in dev, was the same thing happening in QA?

We checked for hidden instances in QA. No rogue processes. No extra engines running.

So if there was no hidden instance, why was QA still failing?

That’s when we realized:

QA doesn’t just have the engine service.

It also has the processor service.

And in QA, the processor service runs across multiple swimlanes—each engine server can have multiple swimlane instances, and the processor service runs on all those instances.

A quick check confirmed our suspicion:

The processor service was stopped on Swimlane 1.

And the reason test jobs were failing?

It wasn’t a coincidence—the testers had explicitly configured all test jobs to run only on Swimlane 1.

So even though other swimlanes were running fine, every single test job was failing, making it look like the entire QA environment was broken.

Why was it stopped?

It turned out, another developer had noticed that jobs weren’t running in QA.

To debug the issue, they had stopped the processor service to run it locally—but never restarted it.

This was a classic case of a mistaken diagnosis.

The real issue was in Lambda.
But since the developer was looking in the wrong place, they thought the processor service was the problem.
They stopped the processor service, but never restarted it.

And since Lambda had been fixed, but the processor was still down, QA remained broken.

We restarted the processor service on Swimlane 1.

Chapter 5: The Final Fix

To summarize, here’s what we had to do to restore everything:

Fixed Lambda’s encryption key issue.
Killed the rogue engine process in dev.
Restarted the processor service on Swimlane 1 in QA.

The system came back to life.

And with that, the mystery was solved.

The Fear and Dilemma

October 25, 2021August 1, 2022 Pritam MondalLeave a comment

The fear of messing up, that engulfs my psyche 
Being aggressive is not so much, a counter to it.
Every move I play, every word I say,
I know, will be judged, 
based on my roleplay.

If we are but a character in our own novels,
how capable are we to become the lead?
preferred by our choice.
Do we bend the story, kill the king?
set the Damsel in distress,
Only to have her rescued and marry her in a palace?

What do we do that makes it ours?
what should we do to keep it fair?

If Venus had come to earth, and we bowed before her,
would’ve been simple, not much unfair.
For I wouldn’t have to play always by the book,
For I wouldn’t have to seek new ways to look cool.

Things are changing and I’m ageing as well,
I know not much mistakes, I can afford before farewell. 
Judgement will be passed and nothing can be done,
I may be conscientious but am I really beloved.
How do I actualize, in Maslow’s pyramid?
Do I follow the Buddha and skip the in-between?

There’s been a hollow in my heart, for far too long now,
in the Quest to fill the void, I need to go on an adventure.
It is this that I have trouble with,
why do I have to be alone, in my own story?

This is a war that must be fought
For every inch of her precious soul, a pacifist will die alone.
A nice one will give up, a bad one will hurt,
How do I be one, who wins in the end,
How do I write it so it ends up being fair?

The damsel, I had made her in a poetic regret,
For she didn’t wish it to be rescued by me,
For she didn’t have to be in danger to begin.
How do I rescue her when she’s not in danger?
How do I make her fall in love?
When there’s little I can offer.

Leave it to fate or should I build it bit slow,
Or should I give up on my own novel,
As I’m not heroic enough of a character? 
Decisions that make us Kings, lead to the bloody boulevard
How can I be so heartless to soak her in blood?

Cultural transgression and relics of patriarchy

March 26, 2019March 27, 2019 Pritam MondalLeave a comment

Cultural norms revolve around our society’s progression. This patriarchal culture and mindset will be changed over time and it’ll happen so spontaneously; while Love is a constant, binding people together since eternity, the cultural norms around it have evolved ever since, Marriage being a mode of conduct for social acknowledgement of ownership to it becoming notion of lovers being united forever. But in all cases a true motive of marriage remains that is to announce to the society that this woman is taken. While there are some tribes where they mark the man with symbols as well but the underlying idea of it hovers around marking a good commodity with a sold out tag. However the society has progressed much from the days of King Oedipus when it was socially acceptable if not celebrated to marry a deceased king’s wife like an Inheritable property similar to the throne. Social acceptance around two people loving each other has a long line of history an marriage being just one aspect of it. If love be the foundation of marriage then it can still be seen in some parts of the world Love is considered as least valued factor while arranging a marriage.In a progressive society also, there could be extreme measures taken to ensure none can take ownership of the commodity called Wife after a man’s death as not a long time ago this country itself was suffering from Satidah. And the freakish thing about it was at that time it was socially justified and was morally acceptable as well, logics were inferred that if you take an oath to be a partner in life and death and life after death then its only romantic for you to die when your partner dies. Now this is quite unthinkable right now but back in the day if some woman thought it so then she would have been criticised by the society. This gives us the insight that Our moraliy and ethics should not be static or firm beliefs they should be challenged whenever our perception changes or grows. The mediaeval customs are mostly perished but still the relics of patriarchy remains. One can argue a person should only go with a partner who believes in same ethics as he or she does but that leaves out a question for us, what if this acceptance of a dominant symbolism is just as wrong as accepting Satidah in their respective timelines? What if some certain man a century ago was so ahead oh his time that he accepted marrying a widow without any second thought but the society around them cursed them for it. The society has no place in their decision making but it can sure smash a newer thought to dust and thus unnecessarily slow down the path of progression. For all I could say we need to accept the other person in our lives as a complete different person altogether, in that we respect their individuality and unique personality. I can fall in love with a woman for all the reasons there is and for all the reasons there isn’t but what if suddenly she says, she won’t be displaying relics of patriarchy by wearing vermillion on her neatly parted hair that her mother told her to maintain since adolescence? Shall I forget all the love I have for that woman upon hearing this and declare it a deadend for our relationship? Or should I make her believe that for sake of social acceptance and my inner cowardice of not being able to fight the social norms for better, she should wear it even after she found a profound reason not to do that? Or should I use the old trick of forcing her to do it for Love or should I use my ego and peer support to logically or forcibly brainwash her mind in a process of questionable enlightenment so that she would submit to acceptance of it herself and would feel guilty of thinking as such? This lives me with one diring question of what would I have done if I was born two centuries ago in the age of Satidah

Pritam Utpalodhi

Why Supreme Court Struck down Section 497 Adultery law?

September 30, 2018 Pritam MondalLeave a comment

Supreme court has struck down another archaic law on 27th September that is IPC Section 497 the Adultery law. The 158 years old law stated a woman has absolutely no say in the case of adultery, she can not even be punished for adultery, she is just a commodity, she has no say in it. Such medieval mentality is because of the time when this law was created 158 years ago, women were seen as properties of their husbands.
So if your wife involves in adultery with a person then that person will be punished. A typical case of Adultery means a conflict between Two Men about the Ownership of a property called ‘Woman‘. Furthermore, the law states, A Man can not involve himself in a relationship with another Man’s wife.

This again emphasizes the fact women were treated as properties of men and also there is another important thing be noted here, Read attentively the line where it says, a Man can not be in relationship with another man’s wife, that means a Man actually Can involve in relationship with other unmarried women, and there will be no legal issues.
This is the reason why so many marital rapes happen in the subcontinent and in most marriages in typical orthodox families, the woman has to suffer a lot of ordeals of what one being ‘not-being-loved’, with this kind of archaic laws active and where marriage is rather treated as a service agreement with non-negotiable terms for lifetime, people of this subcontinent often forget Marriage is based on mutual understanding and above all Love.

Now that this medieval law is withdrawn, this is one step forward towards a better and more mature society. In a good society, Marriage should be meant after Love only. In a healthy society think of marriage as an act of officially committing or publicly acknowledging being in a relationship with someone. But in reality the Indian marriages are often like a formal contract that you sign for Life, like owning a slave in medieval ages. So now, if a married man or woman stops loving his wife or husband at some point or if he or she fell in Love with someone else or in a typical conservative Indian marriage where She is never Loved by her husband, but she finds someone who loves her, in these cases the marriage should be considered as Failed marriage and that the official relationship is to be considered as already over.

Up until now what was happening here is that the husband files a case against the other man and that other man by law is declared a Criminal, and in all this the wife has no say except she could face severe Mal-treatment from her Sasural. The remedy Supreme Court suggested is, when an incident of adultery takes place then instead of putting a case on that other Man and tagging him as Criminal the married couple should just divorce and move on with their new life. Cause when you do not even love your officially acknowledged wife or husband anymore then that relationship is but just a contract only, and the Other Man or the Other Woman should not be tagged as Criminal for loving someone with conscience. The statement “adultery is legal” is not appropriate, it can mean many things, but what Supreme Court emphasized is on Decriminalizing adultery meaning, loving someone outside of marriage can be anything but isn’t an act of Crime by any means, and so the parties involved are not to be referred as Criminals.

One should remember adult relationships are conscientious, when it is conscientious from both sides then it isn’t rape or any crime like that. An adult man or woman is no one’s property thus none can forbid them from loving anyone. Now the question of morality comes whether you should do it or not, now decriminalizing adultery doesn’t mean its mandatory, if someone truly loves his or her life partner then there’s no logic in involving in extra marital affairs. The law simply means this Conscientious act of Love is not to be considered as a crime, this may very well mean the official marriage is over but this is no act of Crime.

Actually in a way this will enforce the responsibilities of a relationship, cause in southern Asia in Indian subcontinent specially marriage is a bond often taken for granted. Marriage here is like owning someone for life, the husbands here more often than not usually don’t care much about their wives after marriage, treat them as luxury commodity. Now that this law is changed, the men and women of the subcontinent finally will have to understand, that they need to put actual effort in Maintaining the relationship from both sides and have to take care of each other’s problems and needs.

Now there are several divorce laws and relationship laws some of them are even based on Mīmāṃsā, Manusmiti, Muslim laws, some of them makes the man liable to pay for his divorced wife’s living. Many women exploit that but now that the case of adultery is struck down, now in a case of a woman who is leaving a man for another man I think new laws will be made and the Man can defend himself financially.

I think this is a liberal step, western countries long have this liberty. Loving someone other than your wife or husband is not a crime but what that means is that the current marriage is over. We are likely to witness an increase in divorce rates but that’s not necessarily a bad thing for the society, faulty marriages will be over. Divorces will be like breakups and marriages be like commitments and that is how it should be.

Humanity is Lost

August 23, 2016August 23, 2016 Pritam Mondal1 Comment

Well, today I was travelling in train and as you probably know in India many hawkers and beggars could be seen in your train journey as free or improvised entertainment services, and sometimes the hawkers do actually come in handy but the Beggars Never .

Today, I encountered such dramatic experience that I think has had gone too far. I was sitting (not quite sitting you know the word ‘Improvise’ ) in general compartment.

When at Bolpur (Santiniketan) Station some kids got in. One girl probably 7 years old holding an infant child in one hand and some musical pebbles at other with the company of her another brother who is no more than 5 years in this planet.

They started their act (or not) as such, the infant child was crying helplessly, I looked into its eyes, it was heart-breaking sadness, and his elder sister was consoling him saying things like “Don’t cry baby, mamma is just there you see.. Don’t cry mamma is there..” (and pointing fingers accordingly ) with no absolute sign of their Mamma or Whomsoever she is.

The passengers can easily understand that who ever their Mother is, Is Either Dead or worse happened to her. And that leaves these kids totally insecure.

The act started growling, the young brother brought a toy with him. A fluffy rubbery plastic bird shaped toy with a hit sensor-ed dancing light mechanism in it.

He was hitting that toy on passenger seats and the light was twinkling and that made that infant child smile for a while. Then he put the head of that toy inside his mouth, chewing on it.

The sister took the toy from his brother and gave it to the infant and he also put it in his mouth the head of the bird was working like a nipple.

All this going on and the sister was repeatedly saying “mamma is there..mamma is here..” The brother was holding an empty stainless steel glass at other hand often biting and stretching the edges with hands and was giving creepy smiles frequently.

I was watching or rather observing this with rapt attention with both my hands resting on my lap. When Suddenly, the boy held the glass in front of me, actually touching my hands with it.

It was the time when you put some coins in that glass overwhelmed by emotions thinking about those unfortunate kids and how they’ll survive this cruel world!

I looked into his eyes and said No, I could’ve given them one rupee or two but that would have been gone in vain. Because Whosoever has sent them into this business actually wants this to happen.

The child started crying again, and nothing could stop it. I asked the sister to take him home . She didn’t answer.

They don’t even know counting to hundred, yet they were collecting money trading on human emotions risking their lives, their future, everything .

They can get mid-day meal in any government primary school, if they are orphan then enormous clubs are there in Bolpur who would happily help them.

At the age when you are supposed to play childish games with friends, some people have thrown them in the gutter of the society. And instead learning counting to hundred they learned even more mature element of life – Lying .

Some people gave them some coins and they went to the next section repeating the same act recursively .