Thread not waking up?

[al_paso]

Here is the issue happening on of our production servers:

There is a class that runs as a Thread (extends Thread, impements run() etc. etc.)
Basically it reads from the database a bunch of values and populates an ArrayList.
The ArrayList var is defined as private static and has public getter methods that return the arraylist. These methods are not static.

Her's how the run method is implemented:
Note: Code is pseudo code.

public void run(){
int retryno = 0;
while(retryno < 20){
try{
readDBAndGetaData();
if(retryno>0){
retryno = 0;
}
Thread.sleep(1 hour);
}
catch(SQLException sqle){
logger.log("Exception in pool, incrementing retry no to "+ ++retryno, e);
}
catch(InterruptedException ie)
logger.log("Interrupted...");
}
}
}


The private static ArrayList var gets populated in readDBAndGetaData().
The problem is that the ArrayList var does not get the updated data sometimes.

Could the problem be:
a) The getter methods not being static
b) Long sleep time - 1 hour
c) combination of both
d) anything else that I don't know

Any help on this would be appreciated.
Meanwhile I have started looking at the Timer and TimerTask classes to rework the functionality.
Not that it matters but this running on jserv, Apache on Solaris 2.8
JDK 1.3

Thanks

[dlorde]

Well, the data might not be updated if more than 20 SQLExceptions are thrown. I don't know for sure, because you give no details about readDBAndGetaData(), and you don't mention whether you get any log messages. As you only posted pseudocode, we can only guess whether the actual code is implemented sensibly, but judging from the pseudocode, it probably isn't...

However, at a guess, I'd say the cause is unlikely to be a, b, or c, and is probably d.

G.I.G.O...
Anon.

[al_paso]

Thanks for the response..

Well, the data might not be updated if more than 20 SQLExceptions are thrown.
I am sure 20 exceptions are not thrown (it would have appeared in the logs), in fact not even one is thrown.

Okay..it is not a sql problem I know that for sure.

The run method is pretty much what it is in reality except the part where it says Thread.sleep(1 hour).
Having said that what do you see wrong in the run() method?

Also I was doing some research on copying objects and realised that in the readDBAndGetaData().
local variables have been declared that are actually populated and then "copied" over to the private static class vars!!..

private_static_class_var_list = local_var_list;

Shouldn't the .clone() method be used here?
Could that be the issue?

This was written by a fellow co-worker earlier and I am trying to debug this issue now.
Keep in mind the private static class vars are returned from the public getter methods.

Thanks

[cjard]

are you sure that the database is providing updated info? transactional systems can return the same data over and over while they are in transaction. there could be a data caching issue going on with the readDB method..

oh, and why are you using statics?

[al_paso]

Yes, the data is updated fine..I am +ve about that.

The reason for having the member List var as static is that the object is basically a pool. (it's called the Bid pool) and I want to have one copy of the data being shared and accessed always.


I just found that there is an issue of the db connection timing out that is causing the Thread to die. So I guess its the db connection..which is the culprit. I had to put catch (Exception e) as well as catch(Error e) blocks to get a meaningful output onto the server logs. I am not sure why the connection is timing out though..I will continue to investigate..
Let me know in case you have ideas...
Thanks for the help cjard

[al_paso]

If anybody is interested, I am having the same discussion in the sun forum(talk about being desperate..) and it is getting interesting!!

http://forums.java.sun.com/thread.jsp?forum=31&thread=499635&start=0&range=15#2360858

[cjard]

Originally posted by al_paso
Yes, the data is updated fine..I am +ve about that.

The reason for having the member List var as static is that the object is basically a pool. (it's called the Bid pool) and I want to have one copy of the data being shared and accessed always.


I just found that there is an issue of the db connection timing out that is causing the Thread to die. So I guess its the db connection..which is the culprit. I had to put catch (Exception e) as well as catch(Error e) blocks to get a meaningful output onto the server logs. I am not sure why the connection is timing out though..I will continue to investigate..
Let me know in case you have ideas...
Thanks for the help cjard

connections are designed to time out, otherwise the whole internet would be chock full of the buggers :)

rewrite your code to be more tolerant. use a db connection pool.. every hour, retrieve a connection from the pool, use it, and put it back. if you do not do this, you MAY exhaust reources on the server machine once you get past N open connections (concurrency issue) if the server is unix - a unix system may open only N file descriptors (including sockets) - the actual value of N depends on what else your machine is doing.
Windows will probably suffer a similar limitation, but end of the day, it's bad programming practice to retain a hold on resources for extended periods of time, and hardly ever use them

[al_paso]

use a db connection pool.. every hour, retrieve a connection from the pool, use it, and put it back.

That is precisely what I am doing. There is a connection pool and the application "checks out" connections from there.
I have not seen the Thread die after I put in those Exception blocks but does that mean it is not encountering the exceptions any more? Seems inconsistent behavior to me but I will keep investigating.
Thanks for all your help cjard.:)

[cjard]

erm.. now, your program is:

implementing a db connection pool
or
using a third party conenction pooling module?


-
if youre using a third party one, notify the author, as his code isnt correctly reaping the pool for dead connections and removing them. I used a very simple pooler for a project a long time ago, named maybe PoolMan, JPoolMan or JDBCPool, i cant remember, and it worked really well, including dead-connection-avoidance

note that 3rd party pools usually have a setting to control the reap interval, and many of them test a connection before they give it to you, to check it is still alive..

if you have implemented your own, then check that these steps are present and correct

[al_paso]

Using a 3rd party connection pool from BitMechanic....

So far it has worked beautifully. But thanks for the tips. I will check them out.
One thing that I am trying right now is getting a Context outside of the pool by specifying the tns name directly. That way, hopefully the connection will be reaped after a long time.
So far so good...let's see how it goes...

[cjard]

actually, that might be the pool i used...

hangon a sec

yes, it was.. however, my pool wasnt so inactive...

are you sure youve set the pool reaping options correctly?

[al_paso]

are you sure youve set the pool reaping options correctly?
Well, yeah....I am providing the max checkout time to be 10 secs.

Here's is the pseudo code:



static ConnectionPoolManager _dbpool = null;
_dbpool = new ConnectionPoolManager( maxCheckOutSecs);
_dbpool.addAlias(_alias,
"oracle.jdbc.driver.OracleDriver",
"jdbc: oracle: oci8:@" + _tnsname,
_uname,
_pwd,
_maxConns,
900, //idletime 15 mts
10// max.checkout time 10 secs
);


The parameters are loaded from the servletConfig within a initialization servlet and retrieved via config.getInitParameter().

[cjard]

and your application is using the pool correctly, yes? it is not getting a connection from the pool and holding onto it for an hour?

[al_paso]

it is not getting a connection from the pool and holding onto it for an hour?

What do you mean by "holding on"? Since the max checkout secs time has been specified as 10 secs the connection will time-out after that, unless there is a bug in the driver code.
To answer your question....yes my application is using the pool correctly and not "holding on" to the connnection.

Before the Thread "goes to sleep" for an hour, the connection context is explicitly close()-ed.
BTW looks like providing a connection context outside of the application pool seems to have worked. I have not seen the Thread dying since Sunday night when I made the change on the server :)

I am thinking now as to what could have happened..

The query definately takes a while to execute ~28 secs
Maybe the connection pooling was getting goofed up when the Thread was using a connection from the general application pool in the context of the servlet container (some wierd dynamics involved there, not sure what). By specifying a seperate connection context outside ofthe pool provided a new unexhausted pool of connections to the Thread.

I would love to hear your views on this as well. :)

[cjard]

it's somewhere in the documentation that you should set the connection timeout period to les than the longest running query, because the pool manager then starts killing connections that are attached to running queries...

[al_paso]

Isn't the connection time-out same as max checkout time? If yes, then yes, the time out of 10 secs is less than the longest query.

I could have increased the timeout but we tend/try to stick to this timeout value to force ourselves during development to optimize all queries to run below 10 secs.