1

Topic: Microphones remaining offline after QHost reboot

Hi All,

I have a strange problem in the UK with a CM system that is not polling PCU microphones at start-up. The Qhost reports all microphones as being offline. However the microphones themselves believe they are online (steady network LED).

The only way to bring the microphones online is to carry out a page from each microphone or cycle the power to the microphone and/or CAB8i or disable/re-enable the switch port for each CAB8i.

All of these force an event and after which the CM recognises the PCU.

System has 37 microphones, 34 PCU1's and 3 PCU2's

CM version 4.01b (not confirmed), rest of project here 4.01b (this project design is not mine, but i'm investigating problem)

Microphone versions are

PCU1 - 3.20
PCU2 - 2.10

After restarting the servers i'm getting this message every so often

ca485srv CABNW1   1   216   No RX activity
ca485srv CABNW1   1   216   CABS configured in ratc for CA485

When I generate a page i get this message (typical)

ca485srv CABNW1   1   216   Found addr 7 on MAC blah blah blah
ca485srv CABNW1   1   216   Ignoring type 0x4f sent by offline 7
ca485srv CABNW1   1   216   Sending who's there instead
ca485srv CABNW1   1   216   PCU1,3.20[7] ACTIVE

Last edited by Big Gaz (2009-08-26 15:44:00)

2

Re: Microphones remaining offline after QHost reboot

I can now confirm the version of CM is V4.02b

I have done some more testing this evening and reduced the amount of network traffic, by isolating all bar a handful of Cab8o's and 1 Cab8i with 6 microphones attached. But still I have the same problem. When the Qhost is restarted only a couple of the microphones comeback online within further intervention. And those microphones reporting as being offline, locally show a steady network LED. Therefore in the microphones eye it believes it is online, but the Qhost say's otherwise.

Any suggestions greatly received

3

Re: Microphones remaining offline after QHost reboot

Hi Garry,

One thought I had; with the Servers Started have you tried a 'sin' command from a shell with no arguments? This will give you a list of all the running processes on the system,it will be interesting to see what state QNX reports the CA485 Server,it should read 'RECV blocked' I believe if all was well. As you are seeing no RX ACTIVITY from a 'sin ar' command there appears to be no 485 Heartbeat(or bad poll) from the CA485,when you execute a Page code from a PCU Device and the Q-Host shows the given PCU as online is there an ACK command visible for that address in Verbose mode? 
This maybe an Issue with the serial TX Mac address,what are the Hi,Mid and Lo values?
It would also be interesting to see if a PCU device behaves as expected when connected to a Com Port.

These two lines are Interesting:
ca485srv CABNW1   1   216   Ignoring type 0x4f sent by offline 7
ca485srv CABNW1   1   216   Sending who's there instead

Joe/Ivor have you(s) seen this before?

I don't think where looking at a cabling Issue as everything works fine when a page is executed and there are no missed polls when the PCU is active or is there?
Just to recap on how this issue came about what was the sequence of events in the system to get to the state it is in now?

All energy flows according to the whims of the Great Magnet. What a fool I was to defy him.

4

Re: Microphones remaining offline after QHost reboot

The problem came to light after Qhost crashes were taking place and whilst investigating said crashes, microphone not coming back online after a reboot or start servers was seen.

The Qhost crashing has been resolved (hopefully) after a faulty PCU was discovered and replaced.  The log do longer shows corrupt data messages and no crashes have been seen for over a month.

It may be possible to hook up one microphone to the qhost com port, should there be a free port. Does the Qhost have two com ports as standard, as one is being used for Cobbler link in this instance. 

As regards the other requests I will give them a whirl tonight. I may need some more detail on what exactly you are asking me to carry out. But I'm sure this can be discussed later.

Last edited by Big Gaz (2009-08-27 15:36:06)

5

Re: Microphones remaining offline after QHost reboot

Garry, the Q-Host has 2 Com Ports.(you will need a 485/232 converter).

With doing the 'sin' command I am trying to see what the CA485 Process is doing,if it is showing 'RECV blocked' then it is up to date and is waiting for some work, if it is in the 'READY' state then it is either running or isn't due to the processor running another process. So I'm wondering if there are other processes consuming all the CPU cycles due to higher Process priorities.

All energy flows according to the whims of the Great Magnet. What a fool I was to defy him.

6

Re: Microphones remaining offline after QHost reboot

I don't have an RS232 to RS485 converter handy and obtaining one this late in the day is going to be a problem. So the PCU directly connected to Qhost will have to wait.

As regards 'missing poll' events, these are not seen in the log file under normal operation.

I can run the 'SIN' command as requested and report back my findings.

As regards the serial TX mac address Hi, Mid, Lo value, where am I looking for this data?

Is there anything else you would like me to do whilst i'm there?

7

Re: Microphones remaining offline after QHost reboot

The TX Serial MAC Values reside within the CM Cab blocks, do a Ctrl&Q to display the sub device blocks then go into the Dynavar device and do another Ctrl&Q now you will see the Cab Peek Value table,there you will find the TX Mac values.
If you can swap the primary Q-Host with the in-active secondary Host and test it with the system.

All energy flows according to the whims of the Great Magnet. What a fool I was to defy him.

8

Re: Microphones remaining offline after QHost reboot

No problems on the DynaVar data.

What's the simplest way to transfer the Qhost config over?

Save CM config on Qhost1.... copy via network to MM1.... replace Qhost1 with Qhost2.... power up.... and copy config from MM1 to Qhost2

or is there a better way to transfer config between Qhost 1 and 2.... if I set up the IP addressing correctly on Qhost2, will Qhost 2 get updated automatically or is this done at midnight or via some other daily scheduled event?

9

Re: Microphones remaining offline after QHost reboot

Big Gaz wrote:

No problems on the DynaVar data.

Just out of curiosity, how did you verify this data?

Big Gaz wrote:

What's the simplest way to transfer the Qhost config over?

The easiest way is to use the 'save' and 'restore' file options in the Setup/Files menu.  If the two Q-Hosts are already configured to be redundant, the secondary Q-Host should already have been kept 'in sync'... just stop the servers on the primary and start the servers on the secondary.

James wrote:

Joe/Ivor have you(s) seen this before?

Yes, but only the UK!  ...just kidding.  When we saw this in the past, I think it was traced down to the 485 driver on the CAB behaving oddly... rebooting the CAB seemed to solve it.  Checking/upgrading the firmware version in the CABs may help.

Thanks,

Joe

10

Re: Microphones remaining offline after QHost reboot

Hi Joe,

No problems on the dynavar data, sorry for the bad english!, this statement simply means i'm going to collect the data tonight and forward it on in the morning.

For whatever reason, the 2nd Qhost has not been set up properly or has a bespoke set-up (still awaiting reply from installer). The IP addressing on the 2nd Qhost is quite different to that of the rest of the system and in my opinion is invisible to Qhost1.... certainly can't ping it from MM1.

example: main system is 10.15.215.xxx (MM1, MM2, QHost1 etc..), Qhost 2 config has IP addresses in the range 10.10.10.xxx

I suspect the redundant operation of the Qhosts never got completed, but I could be wrong.

Joe, how did you trace the faulty CAB, what process did you use to track down the suspect CAB?

11

Re: Microphones remaining offline after QHost reboot

Just been thinking about the suspect CAB issue....

Last night I turned off all network ports connected CAB8i's and CAB8o's to MM1/MM2, other than one racks worth of CAB8o's and CAB8i's.

And the Qhost still failed to recover these PCU microphones.

So i doubt we have a CAB issue.

I can and will isolate the rest of the CAB8's, leaving just 1 CAB8i and the CN DSP cards in MM1 and MM2 on the network. Then repeat the test.

12

Re: Microphones remaining offline after QHost reboot

We have fixed it!

Whilst recording the DynaVar data for Peavey UK, the DynaVar MM device flushed and restarted the RS485 network. After which all microphones came back as expected.

A number of restarts failed to repeat the problems of before.

We can only conclude that some event caused the DynaVar device to fail, but not completely.

13

Re: Microphones remaining offline after QHost reboot

Excellent, great work!

FYI, going by memory the CM CAB device has the dynavar device configured so that opening the device window resets all the data.  The CM should eventually reset that data.  In our installs, we usually disable the functionality that clears the data for ease of troubleshooting.  Let me know if you want more specific information on the settings and I'll look it up.

Thanks,

Joe

14

Re: Microphones remaining offline after QHost reboot

Any additional information would be gratefully received

Thanks

Garry

15

Re: Microphones remaining offline after QHost reboot

OK... no problem.  Looking into it, it's a lot simpler than I remembered...

1) Place a new CAB device in the MWare file (example - Devices/ControlMatrix/CAB/CAB Network 1/CAB 8i)
2) Open the CAB GUI and then double-click 'ControlMatrix RS-485 Settings'.
3) In Edit Mode, Select the 'Reset 485 port for ControlMatrix'.  Right-click, select 'Properties'
4) In the 'Control Object Properties' dialogue, click the 'Control' tab.
5) Deselect "Active press and release action when window is opened and closed".

This will prevent the Dynavar settings from clearing out (and loosing coms with the paging stations) when this window is opened and/or closed!

Now, you'll need to repeat the steps in one more place...

1) In the CAB's GUI, press CTL+Q to view the primitives and wiring.
2) Double Click the Dyna-Var(tm) block.
3) Repeat above steps 3-5 in this window.

Hope this helps!!

Thanks,

Joe

16

Re: Microphones remaining offline after QHost reboot

Thanks Joe,

Much appreciated

17

Re: Microphones remaining offline after QHost reboot

No problem!

Thanks,

Joe