PDA

View Full Version : 120 fucking hours


sarettah
10-15-2009, 12:09 AM
and no crashes or errors yet :okthumb:

I have a set of programs I wrote for a client (I would call it a system but since the client did not buy the system I quoted and elected instead to go a program at a time it is not a system as such yet) that have been put in place since April of this year.

From the beginning there have been issues. Lots of table crashes, processes going off into never never land all sorts of shit.

I did what I could to shore up the code to make sure that it was not the problem. I wrote in table locks that I should not have had to. I wrote a control system to keep processes from clashing, again something that I usually would not have to worry about. I even wrote a little maintenance system for the databases that are involved. It goes out once an hour, does table checks on every table and does repairs if it needs to.

From April till last week I had not seen one 24 hour period that did not have some issue. Even with the measures I took there were still lots of crashes, lots of unexplained errors, lots of MY TIME down the drain just keeping things up and running.

From the onset of the issues I told both my client and the host that there were server issues. I asked the host to run diagnostics (multiple times) I argued with them, I fought with them, I kissed their ass, whatever I could to try to get them to do a real evaluation. No good, they kept coming back that all is fine and it must be my code.

Now programmers are a type of people, as a general rule, that you can get to doubt themselves, and I is a programmer, so I bit the bullet and just continued working on the issues writing around what I had to write around, changing code that I knew was good, just trying to keep the programs up and running. I even started doubting my abilities as a programmer and started.

Most of this I could not charge the client for because I do not consider something done until it is up and running pretty flawlessly and we just never got to that point in the process.

Last Thursday, there was a pretty major server issue. PHP was seqfaulted from what I could tell but some programs would run and some wouldn't. Very strange. We had had multiple seqfault isasues with php and the techs had recompiled php for me at least 3 or 4 times.

I got the host's support on the phone and went through some checks with them, again indicating that I thought we had a server issue. Amazingly, this time the tech came to the same conclusion and decided that they needed to change out the box.

On Friday, right around noon they put us into a new box, just moved the hard drives over.

Since Friday at noon there have been no table crashes (usually 4 or 5 a day) and no actual errors until tonight when I had a process go off into never never land, but even that was unlike the errors that I had been seeing because the mysql thread was still processing and the process was running ok, it was just taking longer then I thought it should to finish. So, today I actually had to spend 5 minutes killing a process and then running the code in the browser so I could make sure it was all cool.

So, it was, after all, a server issue (Bad RAM most probably). This cost my client some money (I did bill about 18 hours of maintenance for stuff I could prove was on the shoulders of the host) and it cost me somewhere above 150 hours of my time in what should have been unnecessary coding and maintenance time keeping things up and running and writing code that should not have been needed to be written.

Well, I am not here to bitch about my client or the host in particular. I think I would have gotten the same kind of treatment from about any host out there. Almost all server techs are very quick to blame the programmers code without digging too much into the server, same with tech support in all areas of life I guess.

I will ask that any host that deals with me, PLEASE PLEASE PLEASE, if I tell you there is a server issue then there probably is a server issue. If I am saying it, I am saying it after I have checked my code, after I have tested every possible thing I can think of, because I really do not want to have to hit up support.

I will not blame your equipment unnecessarilly, please do not blame my code unnecessarilly.

In this case I end up eating about 150 hours. :(

RawAlex
10-15-2009, 10:50 AM
Sadly, PHP is pretty good at tossing out random segfault errors because of various incompatibilities that seem to exist between PHP and other common products on a server. So memory errors / motherboard issues would not be the first on the list, especially if the problem can be repeated over and over again.

When you cannot get satisfaction from the host, get a third party to come in and inspect the server. Tests can be run remotely that will tell them much about what is going on.

sarettah
10-15-2009, 11:58 AM
Sadly, PHP is pretty good at tossing out random segfault errors because of various incompatibilities that seem to exist between PHP and other common products on a server. So memory errors / motherboard issues would not be the first on the list, especially if the problem can be repeated over and over again.

When you cannot get satisfaction from the host, get a third party to come in and inspect the server. Tests can be run remotely that will tell them much about what is going on.

Hey Alex.

One of the things that was telling me that it was hardware and not the program was that I could not get it to replicate.

I will keep that in mind about the third party. One problem in this configuration is that I am NOT the client, I am merely representing them to tech support for a select set of programs, not the entire server.

So I end up in the middle kind of.

Thanx for the input.

TheEnforcer
10-15-2009, 02:29 PM
Yikes.. sounds like a pain in the ass.. hope you can get it sorted out...

sarettah
10-16-2009, 12:24 AM
Yikes.. sounds like a pain in the ass.. hope you can get it sorted out...

Nothing to sort out. New box fixed all the issues. Client is out some money. I am out a bunch of time.

Cost of doing business.