Page 1 of 1

Many intermittent failures with serial packet

Posted: Mon Mar 04, 2019 12:05 pm
by ChrisA
Reading encoders values fails intermittently.

I am using the ROS roboclaw_node Python node which makes calls to the python roboclaw driver. Specifically, it calls roboclaw.ReadEncM2() and roboclaw.ReadEncM1() 10 times per second. The Roboclaw is connected to a Raspberry Pi3b with a USB cable

I noticed that many times encoder values of "0" where being detected so I looked carefully at the code and see that the encoder value was initialized to zero then roboclaw.ReadEncMx() is called. If the function fails the value remains zero.

I changed this. with a "hack" Now I init the value to "1234567" and if after roboclaw.ReadEncM1() returns the value is 1234567 I log a warning and do not publish the bad data.

By experiment, I found that roboclaw.ReadEncMx() fails after big changes in roboclaw.SpeedM1M2(). roboclaw.SpeedM1M2 is called 20 times per second and works fine if every call passes the same speed values but when I make big changes (such as reversing the motors) I see a string of failed roboclaw.ReadEncMx() When it gets to failing, I see from 1 to 5 failed reads per second for a few seconds and then some tens of seconds of not failing

The diagnostic reads also fail and return error statuses.

But eventually, it gets worse...

After a while, the roboclaw become unresponsive to commands but recovers after I close the serial port, wait and then re-open it. Some time two cycles or close /open are required. (A power cycle always fixes the problem but mostly close-wait-open the port fixes it.)

EDIT: I've just determined that serial reads can hang "forever". This pretty much puts the roboclaw off-line

QUESTION: Is the roboclaw not able to process packet serial commands at the fill "wire speed" of the USB cable?
Could I be overloading the processor inside the Roboclaw. Have I uncovered a firmware bug?
Even if I write 10 bytes of data 20times per second that is only about 2Kbit per second

My next step is to write the simplest possible Python script that causes this same kind of failure.

Is there a software-reset command that I missed.


(BTW, roboclaw_node.py makes calls to the driver library but does not check status. I'll fix that and eventually a pull request.

Re: Many intermittent failures with serial packet

Posted: Mon Mar 04, 2019 4:48 pm
by ChrisA
I have more information:

The Roboclaw is hanging on writes.

I changed the last function in roboclaw_driver.py to this:

Code: Select all

def Open(comport, rate):
    global port
    port = serial.Serial(comport,
                         baudrate=rate,
                         timeout=0.1,
                         interCharTimeout=0.01,
                         write_timeout=0.1)
    return
The only change was to add "write_timeout=0.1" And now I can see the place where the hang happened. If I remove the write_timeout the write blocks "forever" with the timeout set it raises a timeout exception. In either case, the result is the same. I can no longer send commands to the Roboclaw until it is reset.

Code: Select all

  File "/home/chris/projects/ros_ws_armhf/src/roboclaw_ros/roboclaw_node/nodes/roboclaw_driver/roboclaw_driver.py", line 126, in _sendcommand
    port.write(chr(address))
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 579, in write
    raise writeTimeoutError
serial.serialutil.SerialTimeoutException: Write timeout

Re: Many intermittent failures with serial packet

Posted: Tue Mar 05, 2019 9:32 am
by Basicmicro Support
A write timeout will only happen if there is a problem with your communciations port. It is not a software problem.

If you are using a USB adapter that is the probably source of the problem. DC motors produce alot of noise and can cause glitches in the USB connection. If you are using a USB to TTL serial adapter that could also get glitched by noise(though less likely) from the motor controller. In either case I would look at getting a USB isolator adapter. I got one a couple weeks ago and it completely elliminated USB dropouts from motor noise even on my largest motors.

Here is the one I am using: https://www.amazon.com/YaeCCC-Isolator- ... ooghydr-20

If you are getting write timeouts with a hardware serial port you have a driver or hardware problem on your computer. We arent using hardware flow control so there should never ever be a write timeout using a real serial port.

Re: Many intermittent failures with serial packet

Posted: Tue Mar 12, 2019 3:06 am
by ChrisA
I am connecting the USB port on the roboclaw to the USB port on the Raspberry Pi3 using a short USB cable. My guess is that it should be reliable because USB uses differential signaling.

Would it be more reliable to use the 5 volt serial pins on the roboclaw? But that means level converting the 3.3 volt serial on the Pi. I figured USB would be best.

That electrical USB isolator seems to be maybe for prevention of ground loops? I will have to read more about it.

I'll try also re-routing the wires

Re: Many intermittent failures with serial packet

Posted: Tue Mar 12, 2019 1:02 pm
by Basicmicro Support
The Roboclaw is 5v tolerant but is actually a 3.3v device. It can be directly connected to the RPi I/O pins. Yes, the TTL serial will be more noise resistance than the USB.

USB is differential but very low voltage and very high freqneucy. Any noise can kill the signal. On linux it usually recovers but can cause delays and would need to be handled in your code correctly. If the USB disconnects and recovers your code would need to have timeouts to retry etc. The time it takes to recover/reconnect is also unknown. The much slower and higher voltage TTL signals can handle a lot more noise and even when noise causes a bit error it is caught by the CRC16 checksum and can be detected and reset within 10ms.

Re: Many intermittent failures with serial packet

Posted: Mon Apr 22, 2019 3:48 am
by mathewp
What's the best way to verify this root cause explanation? Is the USB problem/reset visible to a Linux host in way that I can correlate to the failures? I'm wondering because I am having similar problems and I chose the Roboclaw for the combination of USB and encoder support. My robot uses two other USB devices and a camera yet only the Roboclaw exhibits issues despite similar proximity to motors etc. I'm keen to understand more details so I can see if I can avoid the problem.

Re: Many intermittent failures with serial packet

Posted: Tue Apr 23, 2019 10:03 am
by Basicmicro Support
I assume you are talking about detect USB virtual serial port glitches? If the problem is a USB Serial Port glitch due to noise, the serial port handle(eg _port in the Roboclaw python library) will throw an exception if you try to use any serial port commands. The simple solution would then be to catch the exception. You may or may not be able to just close and reopen the com port.