The ERP Hacker

Son of Son of God Module

June 1st, 2011

Recently, I was asked to delve into a third-party application used by our tech support team to keep track of end user requests. Several years earlier, I had hacked into this old, classic-ASP web application and set up a round-robin system so that the tickets would be distributed evenly among the techs instead of being assigned by request type or priority. This ugly hack had worked for about a year, but when the new version of the software was installed, it blew away all my changes.

Now that I had a little free time, I was asked to look at it again. This time I vowed to encapsulate all the logic in a separate .Net assembly so that if any future upgrades erased my changes, it would be a simple matter to plug them back in.

Unfortunately, what I had in mind took for granted that the ASP code would be somewhat de-coupled in its architecture. That wasn’t actually the case. The block of code that assigned the tech had dependencies all over the place. To make matters worse, it didn’t appear to be working correctly, anyway. Currently, it assigned every ticket to the same person, who then had the responsibility to reassign the tickets to his collegues.

I decided that I would forget about how it was working now, and simply focus on getting it to work the way we wanted. I created a web service that would look at two tables, one of which had a list of techs and a boolean field indicating whether or not they were currently in the queue. The second held one row containing the ID of the next tech in the rotation. When one called the web service, it returned the ID of the next tech, then changed the value in the “current tech” table to prepare for the next call.

I probably should have simply written another ASP script to do this, because as it turned out, I had to write such a script anyway.

Wanting to minimize changes to the existing code, I replaced a call to the critical subroutine with one in a separate ASP file that called the web service to get the next tech ID. Since the application wanted a name, not the ID, I then had to use that ID to get the name from the existing system. I also had to filter out the requests that were being made to Human Resources rather than tech support and treat them differently.

Calling the web service from server-side was tricky. I ended up using something akin to AJAX, but without the A, the J, or the X. After some trial and error, I was able to make the web service calls with consistency.

I got the code working fairly well, but when I set up a test harness to see how it would look to the end user, I was astonished to find that every ticket was still being assigned to the same tech as before (let’s call him the “static tech”). I broke the processes down as much as possible, verifying that every step was completing correctly. I ran a sniffer on the HTTP traffic and saw that the correct value was being returned by the web service.

While I was puzzling this out, I noticed something strange. The page loaded with the static tech selected in a drop-down, but when I looked at the HTML source I saw that the expected tech was actually the default. There was javascript running after the fact and changing the value on the page.

I changed the javascript, and fixed the problem…but this was only for the branch of execution that was running because I was registered as a tech. End users would see something different. Therefore, I changed my permissions so that when I ran the test it would work in a way consistent with the end users’ experiences.

It should have worked with no problem, so naturally it didn’t work at all. Once again, every ticket was being assigned to the “static tech”.

It made no sense at all. I pondered the problem for a moment, remembering that if something seemed to make no sense, it was in fact my invalid assumptions and not reality that was causing the confusion. So, I looked at the facts.

First, in end user mode, the application did not give the user the opportunity to change the tech assignment. The tech value was in a hidden field instead of a drop-down.

Second, I had disabled the javascript that had caused the problem before, so that couldn’t be the issue. In fact, this was a moot point, because in this mode the value was hidden and thus there was no reason for javascript to fiddle with it.

I concluded that, as unlikely as it may seem, there was another process interceding between the assignment of the tech and the rendering of the next screen. I just had to find it.

I went through the code line by line, jumping from one ASP file to another as I tried to chase down the distributed spaghetti logic. Finally, I came to what I was able to determine was a call to a COM object…in fact, the javascript I had disabled made a call to this same object.

The purpose of this call, according to the documentation, was to override whatever decisions had been made by the earlier logic and substitute whatever value was produced by this mysterious, black-box process.

To sum up, the EXPECTED behavior of this system was as follows: First, find the appropriate person to assign the ticket to based on request type and priority. Second, forget all of that and replace this value with whatever is returned by the COM object. Finally, just in case either of the first two didn’t take, replace whatever is on the screen with yet another call to the same COM object via a short javascript block.

Oh, but it gets better.

I went into the file system and tracked down the COM object in question. To my astonishment, I found that it was a .Net assembly!

So…classic ASP was being used as a massive, convoluted shell over a few .Net assemblies being used as middleware. I could accept that. The company that developed this software was probably moving gradually from 15-year-old technology to 10-year-old technology. Nothing wrong with that.

But then I looked at the assembly in Reflector.

The object that was being called was a classic God Module, with hundreds of methods and properties. It did everything, from making ticket assignments to tracking hardware to washing and folding shirts. I tracked down the method that I needed to look at, and found something like this:

string GetTechnician()
{
return juniorGodModule.GetTechnician();
}

So I went to the definition of juniorGodModule…and found something virtually identical to the God Module. Once again, there were hundreds of methods and properties. Once again, it seemed to do everything.

But, at least, it was encapsulated.

Within juniorGodModule, I found this method:

string GetTechnician()
{
return juniorJuniorGodModule.GetTechnician();
}

Fortunately for my sanity, the definition of juniorJuniorGodModule was hidden. Unfortunately, the logic behind the static tech assignment would remain a mystery.

As a last-ditch effort to understand what was going on, I ran a SQL trace to find out where the pseudo-COM object was getting the returned value. Here is what I found:

SELECT name from Techs where ID = 19

So. For some reason, known only to the God Module and its little nested junior God Modules, the correct tech for any situation was always #19. Don’t ask me why. It wasn’t the first value on the list under any sorting scheme I could fathom. As far as I could tell, it was a completely random selection, without the randomness of being…you know…DIFFERENT each time.

I settled on an ugly, ugly solution to the problem. After going to the trouble to encapsulate my logic in a web service, I was now forced to change two ASP files in addition to adding a third. But, at least it is working. For now.

I haven’t mentioned the name of the system in question, for obvious reasons. However, this is the type of problem you can expect when venturing into the bowels of any ERP system. Enter at your own peril.

Entry Filed under: Uncategorized