laura gekeler

This site will be taken down Feb. 28th

I am archiving the site over at wordpress, which involves re-uploading and re-linking all assets (yuck) but I wanted you to know we are approaching the end of an era. I began this blog in May of 2007, all about my travails, tests, and triumphs with Blackboard Vista. I covered the gamut from backend server issues to faculty and student support.

There is another blog I contribute to called Learning Technologies @ND which I hope to post at more frequently now that I'm shutting this one down. I'm also interested in pulling together a wiki which will be about Notre Dame's evaluation process for our next learning management system, and that potentially will be a place many of you may want to visit and even participate on.

You will continue to find me active in cyberspace by googling Laura Gekeler.

Peace Out.

Skipping a year only (we hope): Purdue's TLT Conference

This is a bummer for the midwest. Purdue has held a free Teaching and Learning with Technology conference for 12 years, attracting larger and larger crowds of higher ed and K-12 folk.

This year, and abruptly, budget-cuts due to state budget shortfalls have chopped it. Sniff.

Here's hoping for next year in West Lafayette!

Laura's Lapse

I see my postings over the past 6 months have dwindled. Dwindled to around 2 posts per month. I am guest posting over at Learning Technology @ Notre Dame , so it's not like I'm 'off the cloud' (that's like 'off the air' in cyberspeak).

 

Must. Revive. Vision. Must. Do. More. Research.

 

Topics you'd like to see?

Desire to Live DOES!

This patent war is Ovah. It is finished.

The Chronicle says it here.

Ray Hendersen of Blackboard comments here.

Ray, if you're reading this... SomeBODY had to  "come to the view that it was best for us to bring the matter with Desire2Learn to a close." And long time coming too!

Your Bb Vista cluster config.xml file

Tip: Keep a 'last known good' config.xml file on your admin node.

With even a small cluster, anytime a change to a setting in the Weblogic console causes a bad write-out to this file (like almost every other time!), all you have to do to recover is copy your last known good config.xml file over the faulty one, create an empty file named REFRESH on each node (including admin) and rename all of the ../WebCTDomain/server/ directories  (all nodes, including admin) so that a new /server directory gets created.

It shouldn't happen. It sounds goofy. But it's a lifesaver when you need it!

*In some cases the Vista_WLSstore database table also needs to be renamed so that it gets recreated on application startup.

Weblogic 9.2 and Bb Vista cluster node migration 101

There've been a couple recent incidents at Notre Dame requiring our Bb Vista cluster to be restarted. In follow-up root cause analysis, the culprit has been dropped multicast healthcheck packets.

The admin, for reasons I haven't discovered yet, has not received the communication, and believing the node to be unavailable, has begun a migration process which failed. But what exactly is it attempting to migrate? Where is the failure?

I do know WHEN the failure happens, with catastrophic consequence. It happens when a configured number of JMS health checks are not received by the admin. The setting is configurable in your Weblogic Console here (not that I'd mess with these!):

Left Panel > Environment > Servers > Click on Configuration tab, Tuning sub-tab and hit the Advanced portion at the bottom of the screen to reveal all. We're interested in the Period Length and the Idle Period Until Timeout

The default settings are 1 health check every 60000 milliseconds (1 per minute) with a period interval of 4 before migration starts. In our case, packets drop a few times a week but rarely 4 in a row, so the migration process doesn't begin (then fail and hose the cluster) more than once or twice a year. Too many times a year for my tastes.

And you can see the evidence in your weblogic.log on the admin node. Check your logs for the word 'connect' . You'll get the connected AND disconnected events that happen WITHOUT attempted migrations. Are there any? If so, should you be worried?

(I dunno the answer to this yet. I don't think I have to worry unless the migration attempt happens, as in the the BEA-1410 73 event, aka, the dreaded JMS node migration attempt.)

In testing I've done, however, I haven't been simulating dropped healthchecks, instead I've been purposely shutting down the JMS node first, out of sequence to see what happens to the rest of the cluster. And that has proven helpful in my understanding of the process...

...because the JMS node does migrate correctly. In fact, I've been part way through a quiz on a server I'm shutting down, and my session even fails over. So far, I haven't been able to make any kind of migration I know about fail.

Banner ICGORLDI extract causes stuck thread

Since registration for spring opens Nov. 16th and we're integrated through the Luminis Message Broker with the Registrar's system, it is vital that spring courses and sections exist in Bb Vista prior to the flood of events open registration causes.

So, Notre Dame's standard procedure is:

	1. Registrar: Add the term to ACTIVE_TERM on GORICCR (event generation begins) Don Steinke. 2. Registrar (Don again): Run GURIROL for the STUDENT and FACULTY roles (these are the only ones that depend on the active term for assignment) Don Steinke –doesn’t have a term param, so does whatever persons are active at the time. CMS ADmin and Luminis Admin do these steps once the RFC appears in our ‘to-do’ list and the Registrar has signaled readiness: 3. Luminis: Stop LMB from processing events (events begin to queue) 4. CMS: Extract ICGORLDI from Banner for term 5. Luminis: Import ICGORLDI to Luminis 6. CMS: Import ICGORLDI to Concourse 7. Luminis: Restart LMB processing events (queue first) 8. CMS: Reconnect Concourse to LMB

 

The ICGORLDI_XXXXXXXX.xml this time was 76 mg in size. I used my standard perl scripts to chop and filter smaller file sizes, figuring I could get away with importing 25 mg at a time. It turned out that almost 50 mg of the xml extract were <person> tags, so I ran those 2 files first with standard results. Each one took about an hour and a half. Wish may or may not be an improved time now that our JVM startup option for MaxPermSize has been altered from 192m to 256m.

But, the real weirdness happened on the 3rd import I kicked off at 3:30pm, leaving the office an hour later. Around 6pm the siapi import command finished, but a STUCK thread message began to be logged, meanwhile unaware though I was the import was still being processed.

All night long, while the logging which I still normally keep at DEBUG for our systemintegration stuff chewed up hard drive space... 24 gig by the time Operations called me at 6:30am the next morning (must call them to set monitoring threshold higher... there was only 600 mg left on the volume by that point!).

At 6:30am I freed up 27 gig. By 8 at the office 10gig had been consumed by logging again.

So, you would think the node would have to be restarted in order to stabilize, right? And because it's the JMS node, to avoid a failed JMS migration, the cluster should be restarted.

This was not the case.

I thought it would be, but I wanted to wait until all of the xml was committed to the database, which seemed to still be ongoing based on the webct.log.

I grabbed the sourcedID from the webct.log for the currently importing LC context change on a cross listed section; grepped the original xml for how far down the million line file the system was now working and recognized that only 10,000 lines of the xml remained to be processed. I confirmed that in the UI by getting a count of how many sections with titles "Cross Listed Section Group" and sourcedIDs ending in the term code I was importing, and figured I could manage the logging until the process finished and then restart the cluster. Meanwhile, the Linux utility "top" was displaying two processes eating up 45% of memory, but no pegging of the cpu and very low I/O wait. ps -ef confirmed that there was no 3rd process, like siapi, running.

After the <group> tags in an ICGORLDI xml file, the <membership> tags are next, these are both child section memberships in a parent section as well as person memberships, or enrollments, both teacher and student, in a section.

I can not say with certainty when this long running STUCK thread got 'unstuck' since I had changed the logging level on the fly in the Weblogic console for framework.ejb to FATAL. That way I didn't have to watch hard drive space so closely. I suspect as soon as the last of the xml import was committed to the database it changed.

And the node continues on. And the cluster continues on.

All's well that ends well.

Note to self: Test JMS node failover in Test on our current 8.0.3 when Node A is shutdown first. I really could use this tool in my toolkit for future incidents.

Participate in Ed Garay's LMS Poll...

Ed (@garay on Twitter) came back from #Educause09 and put up this poll on twtpoll, but even if you don't have a twitter account, you can participate and review results here:

He asks one question: "What do you sense individual HigherEd schools will od in terms of their LMS system offering(s) two years from now?

 

What do you think?

 

Oh, and here's an interesting Educause09 search looking for Blackboard references from the conference.

Another 'fool' touting reasons why GOOG would do well to buy Blackboard

James Altucher did that thing Nov. 1st for the New York Post where he discourages us from sending our kids to college, based on potential earnings vs debt. Read it here.

Today he's blogging on the Wall Street Journal blogs advising us how to invest that would-be-college money IN education stocks. And lo and behold again we hear Google would do well to buy Blackboard [BBBB]. Another pick Altucher has, is investing in a degree as a diesel mechanic OR investing in the company that offers diesel mechanic training. See how this works? Investing in yourself and your earning potential in:  "How to make education pay off" 

But how do you invest in your happiness and satisfaction?

Desire2Live ...has (Lived that is).

Michael Feldstein has the word on his blog, e-Literate. Blackboard lost. Blackboard owes Desire2Learn BIG money. And Blackboard plans to try again.

These two entries in the past month will bring you up to speed.

Bb Owes Desire2Learn over Three Million Dollars.

The Cost of the Blackboard Patent Suit (and Who Pays it)

I hope BBBB stock owners put some pressure on Blackboard that this suing as a business strategy is seriously knocking down their profit. Maybe their voices, rather than the ongoing chorus of imploring higher ed clients, will be heeded.