Tuesday, the day I killed the database server …

Duplicate queries of duplicate queries that never finished continued until they consumed all resources on the database server.

Until I shut down the cluster.

(And saved the database server and all the other applications whose databases also live on that server).

How’d I do it?

In a running cluster, using Weblogic 9.2, the underpinnings of Bb Vista 8.0, I changed the JMS server settings from “WebCTManagedNode A (migratable)” to “WebCTManagedNode A”.

Weblogic told me the configuration change was activated and no restart was necessary.

Exactly like on our development and test environments.

Only this time people were doing tasks which were reliant upon the JMS servers, and the application kept asking the database for a response on their behalf … I think.

Chime in those of you who might understand the Weblogic/Vista underlying architecture.

Another for my list of “Things to Never do Again.”

Advertisements

One response to “Tuesday, the day I killed the database server …

  1. Laura,
    This will 100% break a happy cluster.
    Basically, Weblogic “migrates” migratable targets around to do the JMS trickery that drives so many people crazy.
    In actual fact the JMS server doesn’t move – the migratable targets do – and so setting a server to have a non-migratable migratable target will cause the whole box and dice to, well, fall apart.
    Feel free to email me direct if you want to discuss in more detail 🙂
    ben

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s