USB hotplugging

[jadonk]

USB hotplug does not work consistently on the 3.8 kernel

[DanLipsitt]

I've confirmed this on A5C running stock Angstrom v2012.12, Ubuntu 12.04 (3.8.13-bone20), and Linaro (3.8.13.0-1-linaro-am335x) using at least three different external hub chipsets.

If the hub is attached before boot, things are fine. When the hub is plugged in after boot, I get intermittent error messages as soon as something is plugged into the hub. The hub will then disable itself until reboot. For example, the light on an optical mouse will not come on. The usb port stays down until the board is reset (not just rebooted) The error messages on the console look like this:

[  353.882129] usb 1-1: device descriptor read/64, error -110
[  364.503183] usb 1-1: device not accepting address 6, error -110
[  375.020757] usb 1-1: device not accepting address 7, error -110
[  375.027955] hub 1-0:1.0: unable to enumerate USB device on port 1

I also see this sometimes:

[  205.727396] CAUTION: musb: Babble Interrupt Occurred

[RobertCNelson]

Please don't test with bone20, that was from back in June 2013, and does NOT show the current status of the kernel.

Please see this image for a more updated kernel..
http://elinux.org/BeagleBoardUbuntu#eMMC:_BeagleBone_Black

[DanLipsitt]

OK, I compiled bone28 from git with your build scripts, @RobertCNelson (without changing the kernel config), and I'm still getting the same errors.

[coreyfro]

I can confirm this problem with bone28. I have also built https://github.com/RobertCNelson/linux-dev versions 3.8 and 3.12. Most recently, I have built https://github.com/beagleboard/kernel/tree/3.12. I am having hot plug issues with all versions. I see that https://github.com/beagleboard/kernel/tree/3.13 is now available but I have not yet tried it.

All kernels are built with their respective, default, config file with no modification.

I will add that this appears to happen consistently, 100% of the time, when hot-plugging hubs and wireless adapters plugged in to hubs which were plugged in at boot, but never (that I have observed) with memory devices (thumb drives or USB/SD adapters) hot-plugged in to the BBB directly or the HUB. This has led me to speculate some failure or inconsistency in how power is negotiated, since hubs and wifi-adapters have some power requirements and memory devices may not, but this is pure, unqualified speculation.

All of these systems work fine, 100% of the time, when these devices are cold-plugged. If the BBB's are booted with the hubs and wireless adapters already plugged in, they work without failure.

If there is more I can give you, please let me know. I have one of these filesystems frozen for further testing and can boot it up on request.

Here are three systems, all running the same filesystem image and kernel version, each going through the same process of hot-plugging after boot. This may seem excessive, but I just want to assure you that the complete set of hardware is different for each.

coreyfro@high-xoltage:~/bbb$ ssh ubuntu@10.0.0.38
Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.12.0-rc7-00080-g94c591f armv7l)

 * Documentation:  https://help.ubuntu.com/
Last login: Sat Jan  1 00:01:48 2000

ubuntu@ubuntu-armhf:~$ sudo tailf /var/log/kern.log
Jan  1 00:00:12 ubuntu-armhf kernel: [   13.308413] usb0: HOST MAC c8:a0:30:aa:62:80
Jan  1 00:00:12 ubuntu-armhf kernel: [   13.419546] g_multi gadget: Mass Storage Function, version: 2009/09/11
Jan  1 00:00:12 ubuntu-armhf kernel: [   13.419584] g_multi gadget: Number of LUNs=1
Jan  1 00:00:12 ubuntu-armhf kernel: [   13.419616]  lun0: LUN: removable file: /dev/mmcblk0p1
Jan  1 00:00:12 ubuntu-armhf kernel: [   13.419762] g_multi gadget: Multifunction Composite Gadget
Jan  1 00:00:12 ubuntu-armhf kernel: [   13.419781] g_multi gadget: g_multi ready
Jan  1 00:00:13 ubuntu-armhf kernel: [   13.983710] g_multi gadget: high-speed config #2: Multifunction with CDC ECM
Jan  1 00:58:08 ubuntu-armhf kernel: [ 3489.032524] CAUTION: musb: Babble Interrupt Occurred
Jan  1 00:58:13 ubuntu-armhf kernel: [ 3494.206736] usb 2-1: reset high-speed USB device number 2 using musb-hdrc
Jan  1 00:58:28 ubuntu-armhf kernel: [ 3509.318719] usb 2-1: device descriptor read/64, error -110

Jan  1 00:58:44 ubuntu-armhf kernel: [ 3524.538727] usb 2-1: device descriptor read/64, error -110
Jan  1 00:58:44 ubuntu-armhf kernel: [ 3524.758711] usb 2-1: reset high-speed USB device number 2 using musb-hdrc

ssh ubuntu@10.0.0.2
Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.12.0-rc7-00080-g94c591f armv7l)

[...]

Jan  1 00:59:09 ubuntu-armhf kernel: [ 3550.200247] CAUTION: musb: Babble Interrupt Occurred
Jan  1 00:59:14 ubuntu-armhf kernel: [ 3555.374668] usb 2-1: reset high-speed USB device number 2 using musb-hdrc
Jan  1 00:59:30 ubuntu-armhf kernel: [ 3570.486660] usb 2-1: device descriptor read/64, error -110

coreyfro@high-xoltage:~/bbb$ ssh ubuntu@10.0.0.34
Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.12.0-rc7-00080-g94c591f armv7l)

[...]

Jan  1 01:00:14 ubuntu-armhf kernel: [ 3615.174827] CAUTION: musb: Babble Interrupt Occurred
Jan  1 01:00:19 ubuntu-armhf kernel: [ 3620.350426] usb 2-1: reset high-speed USB device number 2 using musb-hdrc
Jan  1 01:00:34 ubuntu-armhf kernel: [ 3635.462424] usb 2-1: device descriptor read/64, error -110

[coreyfro]

I have been working with the https://github.com/beagleboard/kernel/tree/3.13 kernel and I am still having hot plug trouble. Unfortunately, although it seems that there are some hotplug situations where 3.13 is improved over 3.12 and the previous kernel, the situation we've been requiring hot plugging for, the failure with hotplugging persists with 100% recreateability.

Specifically, we will have a BBB and a hub installed in an inaccessible enclosure, and the ports to the hub will be accessible externally.

When we hot plug in WLAN adapters in to the hub, we cause the fault that I described above.

However, if both the hub and the WLAN adapter are NOT plugged in, initially, 90% of the time, we can plug the hub, with the adapter in one of its ports, and everything will be fine.

It is odd that we cannot hot plug just the adapter but we can hot plug the combination of the adapter and hub if they are assembled together.

Once this fault happens, the USB subsystem will remain in an unstable state. For instance, once the fault happens, we cannot hotplug the hub/WLAN adapter combination.

USB Thumbdrives do not seem to cause the hotplugging fault.

We have performed these tests with a wide range of WLAN Adapters and USB Hubs. No singular combination works any better than any other.

All hubs are self powered, and we have used bench power supplies to make sure that they are given precisely 5.0 volts, just to rule out faulty or out of spec power adapters.

Just to remove any ambiguity, here is our exact test setup.

Ubuntu 12.04.2 LTS
Linux localhost 3.12.0-00080-g260a04b #1 SMP Sun Nov 10 22:51:44 PST 2013 armv7l armv7l armv7l GNU/Linux

Bus 002 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 13fe:5200 Kingston Technology Company Inc.
Bus 002 Device 004: ID 0586:341e ZyXEL Communications Corp.

And because I don't know what else to include:

[RobertCNelson]

BTW, just for reference, what sha tag are you using with the 3.13 branch, as I still haven't officially pushed anything after branching it from 3.12 so they should be identical.

Thanks for the in depth testing, one little trick I used on the older musb ip used on the original beagleboard when using usb 2.0 hubs. Always leave at-least one device plugged in at all times, such as a small cheap flash drive. For some reason the musb driver & usb hub always seemed to work better in this situation when then hotplugging other devices.

[RobertCNelson]

BTW, just noticed in your picture, have you tried powering with the 5volt barrel connector vs the usb-slave port, you might be running out of power from the hot plug surge..

[coreyfro]

That's a great point. I've been itching to test with our actual power solution but we hadn't terminated all our 5V switching PSU's. I just requested that we expedite that so I can wire up my test rig with better power. We have a number of PSU's to test with and I wired each of them up. I tested them, one at a time, with one beagle bone, and the babble problem hasn't happened. This is a great sign and I hope to have the entire rig wired up by COB today.

I also used the following fix on one image, and the problem stopped, but I haven't tested this broadly, either.
https://groups.google.com/forum/#!topic/beagleboard/C6gMT2_FfiM

Since we have a better power solution, I am hoping we don't need to do the software fix, but it's good to have two effective solutions.

[coreyfro]

Turns out I spoke too soon. I inadvertently left a flashdrive in the hub (they're so damn small, these days!) and, indeed, as soon as I took it out of the equation, the babble problem returned, even with clean power.

I tested the above soft fix (https://groups.google.com/forum/#!topic/beagleboard/C6gMT2_FfiM) without the flash drive, and, sure enough, babble errors returned.

Damn it, I feel like a gnoob.

So far, the only fix seems to be to leave a flash drive in the hub. I think maybe I need to have a peer review my work. I'm going to write a report with my tests and findings and if my peers agree, I'll post it here. It's obviously time I collected my thoughts.

[coreyfro]

Is this a related issue?

http://o.cs.uvic.ca:20810/perl/cid.pl?cid=1374a430f81a67c5c594c3f3c84c58845ed7caec

[coreyfro]

Just wanted to give a status update on the progress of my particular build, and also ask a not so simple question.

Because we will have a number of devices plugged in to our hub, on boot, the hot-plugging issue is being treated as resolved by us, but I am very hesitant to give up on this because I am afraid of how this problem might otherwise manifest.

I have begun chipping away at options in my kernel config, pruning features we may not require as well as deselecting many modules we will never use. Many of the features I have removed are power saving features, and I did so in the hopes that these features might be related to the issue.

I am also trying the new 3.12.1 Linux-Dev repo.

I have a fairly lean kernel now, and even the most up-to-date kernel, but the USB babble issue persists.

I am resigned to be satisfied with this, but I have a nagging feeling that won't go away.

So I have a question to ask those familiar with the issue. Does anyone know how this babble/hotplug problem may otherwise affect the stability of a system with the populated hub work around?

How do I better aide the kernel community in resolving this issue?

I'd like to add that our company hopes to sell 100 devices A WEEK, populated with BBB's. That's $18,000 a month to CircuitCo.

CircuitCo, are you listening?

We are investigating other options. CircuitCo and TI aren't the only horses in our stable.

[cantona]

The following codes can workaround that...

#!/bin/sh

echo "on" > /sys/bus/usb/devices/usb1/power/control

while [ 1 ]; do
cat /dev/bus/usb/001/001 > /dev/null
sleep 5
cat /dev/bus/usb/001/002 > /dev/null
sleep 5
done

[DanLipsitt]

@cantona, won't your code throw away random usb traffic?

[cantona]

@DanLipsitt don't know? But I didn't notice any problem with it.

BTW, the USB hot plug issue doesn't happen while power up bone with mini USB cable. You guys same?

[madscientist42]

Heh... I'm about to work with Robert's 3.13 patch sets to see if it's going to work right for me on my project. I'm seeing USB hotplugging on the Debian build on the Bone just fine with the 3.13 code.

We'll see what comes of it.