CallManager not sending CDR to Billing Server – Troubleshooting Cisco CDR Repository Manager

May 15, 2010

A few days ago, one of our customers raised an issue.
The customer reported that they haven’t been receiving any CDR from the CallManager Cluster.
The customer has a remote Billing Server and the files were being sent to the server by FTP.

I mapped out three possible issues that could have suddenly caused this:

– issues with the CallManager Cluster (cluster consists of 3 nodes)
– issues with communication between the CallManager Cluster and the Billing Server (maybe even a firewall)
– issues with the Billing Server (permissions, home folder, wrong path e.t.c)

So I asked the customer the usual questions like “when was the last time you got any CDRs via FTP”, “has there been any maintenance”.

The customer reported that they stopped receiving CDRs on the Billing Server right after a Call Manager upgrade. The customer was recently upgraded from CallManager v6 to v7.

A quick check revealed some things:

– CDR was being generated by the server. This was good to know at least
– Cisco CAR Scheduler wasn’t running on the Publisher, so I started this immediately
– The two other node (Subscribers) were running Cisco CAR Scheduler
– Cisco CDR Agent was running on all the nodes
– Cisco CDR Repository Manager was running on all the nodes

At first glance, this didn’t look right. Some of the services running on the Subscribers were meant to run on the Publisher only.
In order to understand better, I turned to the Cisco Voice Engineer’s best friend –> CallManager Tracee (I actually hate it sometimes :-)!!!!)
The information in the traces didn’t show any attempts to send the CDR files via FTP

I did a bit of research and found a great link with some very useful info that confirmed what I was thinking:

Cisco CDR Repository Manager should be running on the First Node ONLY (the Publisher)
Cisco CAR Scheduler should be running on the First Node ONLY (the Publisher)
Cisco CDR Agent should be running on all the call processing nodes

The link Troubleshooting CUCM 5.x / 6.x CDR Analysis and Reporting “Data Missing” issues did a good job in explaining a few things.

Other important info included CLI commands that would let you know the following:

– whether the CDR Agent is moving files from the Subs to the Pub
– if the CDR files have been processed by the Publisher
– whether the Repository Manager was unable to transfer the files to the Billing Server

Another link I stumbled across, though it didn’t have much info that related to the issue was Troubleshooting CallManager 6.x 7.x CDR / CMR & how to fix error 10021. I gave it a quick glance and made a note of a few things.

I’ve just pulled a new set of CCM traces from the Cluster, and this time I can see the files being sent by FTP; see below:

INFO  [Thread-8] cdrrep.FtpManager (FtpManager.java:78) – FtpManager constructor 0: Establish server connection to host [x.x.x.x] successfully!
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.91MB (59.00%), Used = 2.72MB (41.00%)]
INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 2
INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 3
INFO  [Thread-8] cdrrep.FtpManager (FtpManager.java:227) – FTP success: cdr_201005041150_1807 to x.x.x.x
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:352) – CDRREP after sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.66MB (55.30%), Used = 2.96MB (44.70%)]
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.57MB (53.85%), Used = 3.06MB (46.15%)]

One last thing I did was to go into CDR Management (CallManager Serviceability –> Tools –> CDR Management) and verify that the info there was correct.

So in all, it was quite interesting for me because I haven’t come across many CDR issues in my Cisco Life.

I hope someone finds this useful 🙂

Advertisements

Been ages….

February 16, 2010

Hey Folks!!

It’s been ages since I last posted.
I appreciate the folks that keep stopping by and even sending me e-mails and leaving comments!

I’ve been quite busy at work and I’ve literally not had much time to blog…..or even tweet  🙂
I’ve been working some crazy hours, so blogging has been on the back-burner.

Recent developments have prompted me to “upgrade” my lab and bring up to scratch.
One of the major accounts I work on have had a few of their clusters migrated to CallManager v7; I’ve also got three friends studying for the CCIE Voice v3 lab, so I’m trying to get in the study-flow of CUCM 7.

So far, I’ve setup my ESXi and installed CUCM 7.
I’ve had issues installing UCCX 7, but I’m hoping it should be sorted before the week runs out.

I’ll try to blog about the stuff I play with from time to time.

Right now, it’s 2:11am and I’ve just finished completed the pilot phase of a project I’m working on, so my eyes are getting really heavy 🙂

Be back sooooooooooooooooon!!!


THE JOY OF A FIRST IPT CALL

March 9, 2009

 

Do you remember the first IPT call you made in your lab or at work??

I’ve got a three year old daughter. You’ll be amazed how much one can learn from these little ones.
Apparently, she’s been watching me configure my “mini-voice-lab” and making calls between HQ and Branch 1.
One of my HQ phones is a 7940 and the BR1 phone is an IP Blue soft-phone running on my laptop.

Now, either she knew what the number to dial, or she actually knew the redial button…….but all I heard was “Daddy!!!!! It is WORKING!!!!!!!”

Apparently, she made a call between the HQ and BR1 IP Phones and she could hear her voice coming out of my laptop……EUREKA!!!!!!!
I couldn’t get her away from the phone after that. I think the most fascinating thing for her was that she could hear herself through my laptop speaker. LOL!!  She kept saying “I can hear my voice”.

Now that feeling took me back to 2002…..the first time EVER I configured VoIP!!!!
The truth was I had no clue what I was doing back then.
I was a zealous intern working at a company called United Telesys (now called Emperion West Africa).

I was working with Sly, a senior engineer tasked with setting up voip for two branches of a bank (I think it was either Intercontinental Bank or Diamond Bank). One branch was located in Trans-Amadi, while the other branch was located in Olu-Obasanjo, both in Port Harcourt, Nigeria.

I remember we simulated the bank’s WLAN and had two 1750’s on each end of the wireless connection.
One 1750 had an FXO, while the other had an FXS………..I can remember the feeling too well……..I was so overwhelmed when the phones rang.

I’m sure every IPT/UC Engineer has had this 1st Call feeling before……..please don’t forget it cos it’s a memory to cherish 😉


MWI, or should I say Phantom MWI’s

February 20, 2009

We’ve just deployed some Unified Messaging (UM) servers with Primary and Failover pairs, so we’re currently migrating users from Unity Voicemail only servers (VMO) to Unified Messaging servers (UM).
The current VMO servers (3 of them) currently serve 3 clusters, consisting of about 40 sites, in a multi-tenant deployment, hence we’re running this as a phased migration.

While the migration itself is going on without stress, we started getting MWI issues on different site right after cutting them over. This had me and my colleague pulling our hair out in some occasions cos users were raising tickets like: “Can’t retrieve my messages“, “No messages but light won’t go off” “I’ve  got voicemails in my e-mail box, but the lights on my phone isn’t coming on“…….the list is endless!!!

After some troubleshooting, we found out that users weren’t deleting their messages on the VMO servers prior to migration (even though they had all been advised to listen to messages and delete them on the eve of the cutover).
Because we are cutting-over the sites in phases, we can’t take the VMO servers out of service, at least no yet cos they were still being used. Also, the project lead doesn’t want to delete the subscribers off the VMO, just in case we needed to regress.

My colleague and I came up with a simple solution with three steps:

– we launched Message Store Manager within the Unity Tools Depot and ran a script to move unread messages from inbox into deleted items or Unity Dumpster for subscribers that had been cutover from VMO to UM

– Secondly, we used Bulk Edit tool to uncheck USE MWI for Message Notification (under the messages tab)

– Lastly, we resync’ed the MWI on the new UM servers (under UTIM).

That was the last we saw of the Phanton MWIs  😉