CallManager not sending CDR to Billing Server – Troubleshooting Cisco CDR Repository Manager

May 15, 2010

A few days ago, one of our customers raised an issue.
The customer reported that they haven’t been receiving any CDR from the CallManager Cluster.
The customer has a remote Billing Server and the files were being sent to the server by FTP.

I mapped out three possible issues that could have suddenly caused this:

– issues with the CallManager Cluster (cluster consists of 3 nodes)
– issues with communication between the CallManager Cluster and the Billing Server (maybe even a firewall)
– issues with the Billing Server (permissions, home folder, wrong path e.t.c)

So I asked the customer the usual questions like “when was the last time you got any CDRs via FTP”, “has there been any maintenance”.

The customer reported that they stopped receiving CDRs on the Billing Server right after a Call Manager upgrade. The customer was recently upgraded from CallManager v6 to v7.

A quick check revealed some things:

– CDR was being generated by the server. This was good to know at least
– Cisco CAR Scheduler wasn’t running on the Publisher, so I started this immediately
– The two other node (Subscribers) were running Cisco CAR Scheduler
– Cisco CDR Agent was running on all the nodes
– Cisco CDR Repository Manager was running on all the nodes

At first glance, this didn’t look right. Some of the services running on the Subscribers were meant to run on the Publisher only.
In order to understand better, I turned to the Cisco Voice Engineer’s best friend –> CallManager Tracee (I actually hate it sometimes :-)!!!!)
The information in the traces didn’t show any attempts to send the CDR files via FTP

I did a bit of research and found a great link with some very useful info that confirmed what I was thinking:

Cisco CDR Repository Manager should be running on the First Node ONLY (the Publisher)
Cisco CAR Scheduler should be running on the First Node ONLY (the Publisher)
Cisco CDR Agent should be running on all the call processing nodes

The link Troubleshooting CUCM 5.x / 6.x CDR Analysis and Reporting “Data Missing” issues did a good job in explaining a few things.

Other important info included CLI commands that would let you know the following:

– whether the CDR Agent is moving files from the Subs to the Pub
– if the CDR files have been processed by the Publisher
– whether the Repository Manager was unable to transfer the files to the Billing Server

Another link I stumbled across, though it didn’t have much info that related to the issue was Troubleshooting CallManager 6.x 7.x CDR / CMR & how to fix error 10021. I gave it a quick glance and made a note of a few things.

I’ve just pulled a new set of CCM traces from the Cluster, and this time I can see the files being sent by FTP; see below:

INFO  [Thread-8] cdrrep.FtpManager (FtpManager.java:78) – FtpManager constructor 0: Establish server connection to host [x.x.x.x] successfully!
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.91MB (59.00%), Used = 2.72MB (41.00%)]
INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 2
INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:105) – CDRSender wakes up for destination 3
INFO  [Thread-8] cdrrep.FtpManager (FtpManager.java:227) – FTP success: cdr_201005041150_1807 to x.x.x.x
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:352) – CDRREP after sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.66MB (55.30%), Used = 2.96MB (44.70%)]
INFO  [Thread-8] cdrrep.CDRSender (CDRSender.java:345) – CDRREP before sending the file: JVM status: [Max = 246.56MB, Total = 6.62MB, Free = 3.57MB (53.85%), Used = 3.06MB (46.15%)]

One last thing I did was to go into CDR Management (CallManager Serviceability –> Tools –> CDR Management) and verify that the info there was correct.

So in all, it was quite interesting for me because I haven’t come across many CDR issues in my Cisco Life.

I hope someone finds this useful 🙂

Advertisements

Been ages….

February 16, 2010

Hey Folks!!

It’s been ages since I last posted.
I appreciate the folks that keep stopping by and even sending me e-mails and leaving comments!

I’ve been quite busy at work and I’ve literally not had much time to blog…..or even tweet  🙂
I’ve been working some crazy hours, so blogging has been on the back-burner.

Recent developments have prompted me to “upgrade” my lab and bring up to scratch.
One of the major accounts I work on have had a few of their clusters migrated to CallManager v7; I’ve also got three friends studying for the CCIE Voice v3 lab, so I’m trying to get in the study-flow of CUCM 7.

So far, I’ve setup my ESXi and installed CUCM 7.
I’ve had issues installing UCCX 7, but I’m hoping it should be sorted before the week runs out.

I’ll try to blog about the stuff I play with from time to time.

Right now, it’s 2:11am and I’ve just finished completed the pilot phase of a project I’m working on, so my eyes are getting really heavy 🙂

Be back sooooooooooooooooon!!!


RE: Got my Number

June 29, 2009

 I’ve been getting e-mails to write an update on my quest to obtain the CCIE number.

Apologies it has taken over 3 weeks, but I got plunged right into work the very next day I got back to the UK.

 After passing my CCIE, I felt like Daniel, who in the Bible was thrown in a lions’ den and miraculously came out alive.

I’ve had my eye on the CCIE for a while (right after I wrote my CCNA in 2001), however I wasn’t motivated enough to go the R&S way, though that was my main experience.

 I completed my CCVP in November 2008 and went straight on to write the CCIE Voice Written exam in December 2008.

I started the CCIE Voice Lab pursuit right after passing my written, however it was around the festive period, hence I couldn’t get much done in December.

 I kicked off in January trying to learn the technologies.

I spent a huge amount of time watching videos and reading some Cisco docs.

While I don’t recommend reading ALL of the Cisco docs (front-to-back) on the lab kit, I would suggest spending quality time to master certain areas. I remember getting a good grasp on IOS CoR using the Call Manager SRND (though I believe I used CCM v7 SRND).

 Another thing I remember doing was learning how to access Cisco docs using the “new” Doc CD (http://www.cisco.com/web/psa/products/index.html) that replaced the UniverCD.

 I started out the CCIE journey with 2 good friends, but I soon found that I was alone after they both reviewed their commitments and concluded that they would wait for the Version 3 exams.

After my friends dropped out, I started to spend less time doing online labs cos we used to work through labs together.

This gave me more time to study the technologies and I think it paid off.

 Along the way, I met James Key of Blindhog. He became a friend and mentor.

We would setup WebEx sessions on the fly and end up spending hours on the phone going through problems.

There were some areas I had absolutely no clue about which James was able to explain thoroughly.

 Another person I met through a CCIE friend was Dharm. In my own eyes, I was ready for CCIE…….well, until I met Dharm.

He gave me access to his kit remotely and also allowed me come over to his place to use the kit.

 I met Dharm about 5 weeks to my exam. I knew lots of stuff, but I had no strategy to pass the exam.

He helped breakdown lots of other concepts and I realised I was going to fail despite studying and labbing like a MAD MAN for 4 months.

In my opinion, I think this guy should have a training school 🙂

As I’m writing this, we’re waiting for the result of his own CCIE Voice exam…….I’m sure he’s going to pass!!!!

 I flew to San Jose to face the greatest challenge of my life up until now, and came back a VICTOR!!!!

 In all, I had about 5 months of solid preparation after my written. Yes I know it sounds crazy……CCIE-In-5-Months 😉

To be honest, I couldn’t tell people I had only prepared for 5 months……they would have thought I was crazy!

 One thing I’m still unable to quantify is how much work experience helped me. I believe experience from work really helped me with areas like Call Routing, Gatekeeper and Unity.

 If you’re married, with or without kids, you need to appreciate the fact that the CCIE pursuit would in many ways stand between you and your family. It’s very important to have your family’s support and consent before embarking on such a great journey.

 All I can say is that I’ve got the MOST supportive wife in the world!!!!!!!!!

 Wishing you all the best……God Bless!!!

 

Skybaba

CCIE Voice


To the Lion’s den and back!!!!

May 30, 2009

 
I sat my first CCIE Voice lab attempt on Tuesday, 26th May 2009.
To God be the Glory, I passed my exam!!!

The last few days have been hectic.
I’m currently in the US, and I’ve got a busy schedule.

I plan to write a comprehensive report on my prep and lab experience.

Right now, I guess I’m still basking in the euphoria of being part of the first 1000 people (in the world) that hold a CCIE in VOICE!

😉


THE JOY OF A FIRST IPT CALL

March 9, 2009

 

Do you remember the first IPT call you made in your lab or at work??

I’ve got a three year old daughter. You’ll be amazed how much one can learn from these little ones.
Apparently, she’s been watching me configure my “mini-voice-lab” and making calls between HQ and Branch 1.
One of my HQ phones is a 7940 and the BR1 phone is an IP Blue soft-phone running on my laptop.

Now, either she knew what the number to dial, or she actually knew the redial button…….but all I heard was “Daddy!!!!! It is WORKING!!!!!!!”

Apparently, she made a call between the HQ and BR1 IP Phones and she could hear her voice coming out of my laptop……EUREKA!!!!!!!
I couldn’t get her away from the phone after that. I think the most fascinating thing for her was that she could hear herself through my laptop speaker. LOL!!  She kept saying “I can hear my voice”.

Now that feeling took me back to 2002…..the first time EVER I configured VoIP!!!!
The truth was I had no clue what I was doing back then.
I was a zealous intern working at a company called United Telesys (now called Emperion West Africa).

I was working with Sly, a senior engineer tasked with setting up voip for two branches of a bank (I think it was either Intercontinental Bank or Diamond Bank). One branch was located in Trans-Amadi, while the other branch was located in Olu-Obasanjo, both in Port Harcourt, Nigeria.

I remember we simulated the bank’s WLAN and had two 1750’s on each end of the wireless connection.
One 1750 had an FXO, while the other had an FXS………..I can remember the feeling too well……..I was so overwhelmed when the phones rang.

I’m sure every IPT/UC Engineer has had this 1st Call feeling before……..please don’t forget it cos it’s a memory to cherish 😉


IMPLEMENTING GATEKEEPERS WITH HOP-OFF TECH-PREFIX

March 1, 2009

In the course of my CCIE studies, I’ve found the “Gatekeeper Address Resolution on ARQ” chart extremely useful.
Having a good understanding of the call-flow displayed the chart is key to understanding the debug gatekeeper main 10 and how the gatekeeper logically looks through configured parameters to decide how a call will be routed.

The chart (shown below) should be available in the CCIE Voice Lab exam. It can be found on Page 10-40 (pg 350 figure 10-14) in the CCM 4.1 SRND, which we’ve been told would be available in the lab.

gk-address-resolution-for-arq

 

gk-address-resolution-on-arq_1

 

Now back to the Hop-off Tech Prefix.
While there are many ways to route calls on a gatekeeper, we may need to use the Hop-off Tech-prefix configuration scenario in the lab if we’re specifically asked to do that, or if we’re instructed not to use the following:

 – Tech prefix
 – Default gw-type-prefix
 – E.164 Aliases (including Alias Static commands for CCM numbers)
 – Zone Prefix

Looking at the chart, we’ll need to match Step 1. Though it’s typically used to route calls to Remote Zones via a Location Request (LRQ), the gatekeeper can also generate an ACF at this stage if the Hop-off points to a Local Zone.

There isn’t much configuration needed to get this working; however, for your configuration to “stay working“, you would need to “hardcode” the signalling port used by your GK-Controlled Trunk (on Call Manager) to always use port 1720 as its Gateway Signalling Port.
This is quite important because CCM dynamically assigns port numbers for gateway signalling and for RAS.
These ports are set to 1720 and 1719 respectively on the gatekeeper by default.

If you don’t do this and the proctor resets your Trunks before grading (or maybe he/she even reboots the CCM box), then you’re screwed big-time 🙂

You can verify the Signalling and RAS ports CCM registered with on the gatekeeper with the show gatekeeper endpoints command:

 
GK-RTR#sh gatekeeper endpoints
                    GATEKEEPER ENDPOINT REGISTRATION
                    ================================
CallSignalAddr  Port  RASSignalAddr   Port  Zone Name         Type    Flags
————— —– ————— —– ———         —-    —–
10.10.50.103    2789  10.10.50.103    4318  gk-zone1          VOIP-GW
    H323-ID: CCM_GKCT_1
    Voice Capacity Max.=  Avail.=  Current.= 0
172.16.30.254   1720  172.16.30.254   57634 gk-zone1          VOIP-GW
    H323-ID: CME_Trunk
    Voice Capacity Max.=  Avail.=  Current.= 0
Total number of active registrations = 2

 
As shown above, CME registers with 1720 by default, but CCM registers with 2789.
If CCM_GKCT trunk is reset, the port number would change to something totally different.
To configure CCM to use the port 1720 (signalling port) for a particular gatekeeper-controlled trunk through the CCM Service Parameters:

Navigate to the CCM Service Parameters and do a search for 1720. You’ll find the section under Clusterwide Parameters (Device – H323).
Change the name from None to the name of your trunk. In my case, my trunk is called CCM_GKCT.
Once this is done, you’ll notice that the port number for your CCM trunk would now be 1720 after you reset it.

 
ccm_srvc_param_gkct_1720

 

We would now need to type in 3 commands to cater for the prefixes in HQ, BR1 and BR2.
gatekeeper
 zone local gk-zone1 myccielab.com 172.16.40.254
 gw-type-prefix 2* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720
 gw-type-prefix 3* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720
 gw-type-prefix 4* hopoff gk-zone1 gw ipaddr 172.16.30.254 1720
 no shutdown

 

Depending on whether we’ve been instructed to provide redundancy, we may need to include the secondary call-processing engine in the command for HQ and BR1 prefixes.
I’ve only got one call-manager running in my home lab, but here’s how you’ll configure it with two call-managers (assuming your second call-manager IP Address is 10.10.50.104)

 
 gw-type-prefix 2* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720 gw ipaddr 10.10.50.104 1720
 gw-type-prefix 3* hopoff gk-zone1 gw ipaddr 10.10.50.103 1720 gw ipaddr 10.10.50.104 1720

 

You can verify your configuration by using the show gatekeeper gw-type-prefix command (or by making calls ;-))

 

GK-RTR(config-gk)#do sh gatekeeper gw-type-prefix                         
GATEWAY TYPE PREFIX TABLE
=========================
Prefix: 2*    (Hopoff zone gk-zone1)
  Statically-configured gateways (not necessarily currently registered):
    10.10.50.103:1720
  Zone gk-zone1 master gateway list:
    10.10.50.103:1720 CCM_GKCT_1

Prefix: 3*    (Hopoff zone gk-zone1)
  Statically-configured gateways (not necessarily currently registered):
    10.10.50.103:1720
  Zone gk-zone1 master gateway list:
    10.10.50.103:1720 CCM_GKCT_1

Prefix: 4*    (Hopoff zone gk-zone1)
  Statically-configured gateways (not necessarily currently registered):
    172.16.30.254:1720
  Zone gk-zone1 master gateway list:
    172.16.30.254:1720 CME_Trunk

 

PS: I’ve assumed your CME side has the following configuration:

dial-peer voice 1000 voip
 destination-pattern [23]…$
 session target ras
 incoming called-number .

 

Now, that’s a long post!!!!!


H.323 GATEKEEPERS

February 26, 2009

H.323 GATEKEEPERS

Gatekeepers are interesting devices that can be used for a variety of purposes within our Dial-Plans.
Some mandatory gatekeeper functions include Address Translation, Bandwidth Control and Zone Management.
One thing to note is that gatekeepers do NOTget involved with Codec Negotiation and any form of Digit Manipulation.

According to IP Expert instructor, Mark Snow “most students I’ve taught have the biggest problems with H323 Gatekeepers and……..“.

This statement is so true, considering that there are thousands of Call Manager deployments out there that don’t use gatekeepers.
I’ve been quite fortunate to work on an account that simply cannot function without gatekeepers because of the sheer amount of intra-cluster and inter-cluster communications spanning the entire globe.

I plan to dedicate a couple of blog-posts to this monstrous component of any large-scale dial-plan deployment.