A few days ago, one of our customers raised an issue.
The customer reported that they haven’t been receiving any CDR from the CallManager Cluster.
The customer has a remote Billing Server and the files were being sent to the server by FTP.
I mapped out three possible issues that could have suddenly caused this:
- issues with the CallManager Cluster (cluster consists of 3 nodes)
- issues with communication between the CallManager Cluster and the Billing Server (maybe even a firewall)
- issues with the Billing Server (permissions, home folder, wrong path e.t.c)
So I asked the customer the usual questions like “when was the last time you got any CDRs via FTP”, “has there been any maintenance”.
The customer reported that they stopped receiving CDRs on the Billing Server right after a Call Manager upgrade. The customer was recently upgraded from CallManager v6 to v7.
A quick check revealed some things:
- CDR was being generated by the server. This was good to know at least
- Cisco CAR Scheduler wasn’t running on the Publisher, so I started this immediately
- The two other node (Subscribers) were running Cisco CAR Scheduler
- Cisco CDR Agent was running on all the nodes
- Cisco CDR Repository Manager was running on all the nodes
At first glance, this didn’t look right. Some of the services running on the Subscribers were meant to run on the Publisher only.
In order to understand better, I turned to the Cisco Voice Engineer’s best friend –> CallManager Tracee (I actually hate it sometimes :-)!!!!)
The information in the traces didn’t show any attempts to send the CDR files via FTP
I did a bit of research and found a great link with some very useful info that confirmed what I was thinking:
Cisco CDR Repository Manager should be running on the First Node ONLY (the Publisher)
Cisco CAR Scheduler should be running on the First Node ONLY (the Publisher)
Cisco CDR Agent should be running on all the call processing nodes
The link Troubleshooting CUCM 5.x / 6.x CDR Analysis and Reporting “Data Missing” issues did a good job in explaining a few things.
Other important info included CLI commands that would let you know the following:
- whether the CDR Agent is moving files from the Subs to the Pub
- if the CDR files have been processed by the Publisher
- whether the Repository Manager was unable to transfer the files to the Billing Server
Another link I stumbled across, though it didn’t have much info that related to the issue was Troubleshooting CallManager 6.x 7.x CDR / CMR & how to fix error 10021. I gave it a quick glance and made a note of a few things.
I’ve just pulled a new set of CCM traces from the Cluster, and this time I can see the files being sent by FTP; see below:
INFO [Thread-8] cdrrep.FtpManager (FtpManager.java:78) – FtpManager constructor 0: Establish server connection to host [x.x.x.x] successfully!
INFO [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.91MB (59.00%), Used = 2.72MB (41.00%)]
INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 2
INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 3
INFO [Thread-8] cdrrep.FtpManager (FtpManager.java:227) – FTP success: cdr_201005041150_1807 to x.x.x.x
INFO [Thread-8] cdrrep.CDRSender (CDRSender.java:352) – CDRREP after sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.66MB (55.30%), Used = 2.96MB (44.70%)]
INFO [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.57MB (53.85%), Used = 3.06MB (46.15%)]
One last thing I did was to go into CDR Management (CallManager Serviceability –> Tools –> CDR Management) and verify that the info there was correct.
So in all, it was quite interesting for me because I haven’t come across many CDR issues in my Cisco Life.
I hope someone finds this useful