ASMCA – Page 8 – Beyond The Data

kernel: xen_netfront: GSO size must not be zero

Env:

Oracle VM Server 3.3.1

Host: hostbase0
VM: hostvm0

Problem:
User cannot access from outside cluster via ssh.
ssh will hang here:

  [suse@suse ~]$ssh hostvm0 -l root  -vvv
  ...
  ...
  debug2: kex_parse_kexinit: reserved 0
  debug2: mac_setup: found hmac-md5
  debug1: kex: server-&gt;client aes128-ctr hmac-md5 none
  debug2: mac_setup: found hmac-md5
  debug1: kex: client-&gt;server aes128-ctr hmac-md5 none
  debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024&lt;1024&lt;8192) sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
  debug2: dh_gen_key: priv key bits set: 138/256
  debug2: bits set: 482/1024
  debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY   &lt;-- stops here
  ^C

[suse@suse ~]$ssh hostvm0 -l root -vvv

...

debug2: kex_parse_kexinit: reserved 0

debug2: mac_setup: found hmac-md5

debug1: kex: server->client aes128-ctr hmac-md5 none

debug2: mac_setup: found hmac-md5

debug1: kex: client->server aes128-ctr hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug2: dh_gen_key: priv key bits set: 138/256

debug2: bits set: 482/1024

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY <-- stops here

  #ssh host06  -vvv
  ...
  ...
  debug2: kex_parse_kexinit: first_kex_follows 0

  debug2: kex_parse_kexinit: reserved 0
  debug2: mac_setup: found hmac-md5
  debug1: kex: server-&gt;client aes128-ctr hmac-md5 none
  debug2: mac_setup: found hmac-md5
  debug1: kex: client-&gt;server aes128-ctr hmac-md5 none
  debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024&lt;1024&lt;8192) sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP         &lt;--stops here.
  debug3: Wrote 24 bytes for a total of 1005

#ssh host06 -vvv

...

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: mac_setup: found hmac-md5

debug1: kex: server->client aes128-ctr hmac-md5 none

debug2: mac_setup: found hmac-md5

debug1: kex: client->server aes128-ctr hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP <--stops here.

debug3: Wrote 24 bytes for a total of 1005

from ifconfig output, I can see a lot of err/invalid packages:

 RX packets:1321 errors:21 dropped:1073 overruns:0 frame:0

1	RX packets:1321 errors:21 dropped:1073 overruns:0 frame:0

from sys log: can see a lot of err mesg on xen_netfront.

        Nov  5 21:32:18 hostvm0 kernel: xen_netfront: GSO size must not be zero
        Nov  5 21:32:20 hostvm0 kernel: xen_netfront: GSO size must not be zero
        Nov  5 21:32:51 hostvm0 kernel: xen_netfront: GSO size must not be zero
        Nov  5 21:32:57 hostvm0 kernel: xen_netfront: GSO size must not be zero
        Nov  5 21:32:57 hostvm0 kernel: xen_netfront: GSO size must not be zero

Nov 5 21:32:18 hostvm0 kernel: xen_netfront: GSO size must not be zero

Nov 5 21:32:20 hostvm0 kernel: xen_netfront: GSO size must not be zero

Nov 5 21:32:51 hostvm0 kernel: xen_netfront: GSO size must not be zero

Nov 5 21:32:57 hostvm0 kernel: xen_netfront: GSO size must not be zero

Fix:

It’s an oracle bug which marked in Metalink ID:1929676.1

Upgrade Linux kernel on Host to latest:

current kernel: kernel-uek-3.8.13-26.4.2.el6uek.x86_64

# yum update -y kernel-uek
...
...
  Installing : kernel-uek-firmware-3.8.13-44.1.4.el6uek.noarch              1/2
  Installing : kernel-uek-3.8.13-44.1.4.el6uek.x86_64                       2/2

# yum update -y kernel-uek

...

Installing : kernel-uek-firmware-3.8.13-44.1.4.el6uek.noarch 1/2

Installing : kernel-uek-3.8.13-44.1.4.el6uek.x86_64 2/2

bond0: received packets cannot be forwarded while LRO is enabled

Env:

Oracle ODA Appliance, which based on ORACLE VM Server set.

Dom0/Host: odahost1
Dom1/VM: odahostdb1

Problem:

use can not ssh onto Dom1/VM from outside ODA. ping is ok.

SSH will hang here:

#ssh root@odahostdb1
...
...
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Connection closed by 10.191.6.55

#ssh root@odahostdb1

...

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

Connection closed by 10.191.6.55

From Dom0 /var/log/message:

Nov 21 21:40:49 odahost1 last message repeated 2 times
Nov 21 21:41:50 odahost1 last message repeated 7 times
Nov 21 21:44:40 odahost1 kernel: bond0: received packets cannot be forwarded while LRO is enabled
Nov 21 21:46:48 odahost1 last message repeated 16 times
Nov 21 21:48:48 odahost1 last message repeated 8 times

Nov 21 21:40:49 odahost1 last message repeated 2 times

Nov 21 21:41:50 odahost1 last message repeated 7 times

Nov 21 21:44:40 odahost1 kernel: bond0: received packets cannot be forwarded while LRO is enabled

Nov 21 21:46:48 odahost1 last message repeated 16 times

Nov 21 21:48:48 odahost1 last message repeated 8 times

This problem is due to LRO of eth0 not disabled.
Although bond0 LRO is disabled by default, eth0 which bond0 is based on is not.

[root@odahost1 ~]# ethtool -k bond0 | grep large-receive-offload
large-receive-offload: off
[root@odahost1 ~]# ethtool -k eth0 | grep large-receive-offload
large-receive-offload: on

[root@odahost1 ~]# ethtool -k bond0 | grep large-receive-offload

large-receive-offload: off

[root@odahost1 ~]# ethtool -k eth0 | grep large-receive-offload

large-receive-offload: on

Fix:
just disable LRO on eth0

[root@odahost1 network-scripts]# ethtool -K eth0 lro off

1	[root@odahost1 network-scripts]# ethtool -K eth0 lro off

restart network service.

To make the changes permenant:

If Dom0/Host is EL5:
add following line to /etc/rc.local

  /sbin/ethtool -K eth0 lro off

1	/sbin/ethtool -K eth0 lro off

If Dom0/Host is EL6+:
add following line to ifcfg-eth0:

  ETHTOOL_OPTS="-K ${DEVICE} lro off"

1	ETHTOOL_OPTS="-K ${DEVICE} lro off"

the problem on EL5/6 different fix is on initscripts package version.

EL6 with initscripts 9.03+ accepts this ETHTOOL_OPTS, but EL5 with initscripts 8.45 does not.

Refer:

Xen LRO is not disabled automatically with a bonded bridged interface with Intel X5xx 10 Gb Ethernet Adapter

Poor TCP Performance on VMs; dmesg:”received packets cannot be forwarded while LRO is enabled”

EL7 yum repo failed on libprotobuf-c.so.0

EL7 with EPEL repo.

libgadu-1.12.0-1.el7.centos

           Requires: libprotobuf-c.so.0()(64bit)
           Removing: protobuf-c-0.15-8.el7.centos.x86_64 (installed)
               libprotobuf-c.so.0()(64bit)
           Updated By: protobuf-c-1.0.1-1.el7.x86_64 (epel)
              ~libprotobuf-c.so.1()(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

libgadu-1.12.0-1.el7.centos

Requires: libprotobuf-c.so.0()(64bit)

Removing: protobuf-c-0.15-8.el7.centos.x86_64 (installed)

libprotobuf-c.so.0()(64bit)

Updated By: protobuf-c-1.0.1-1.el7.x86_64 (epel)

~libprotobuf-c.so.1()(64bit)

You could try using --skip-broken to work around the problem

You could try running: rpm -Va --nofiles --nodigest

Well libgadu was based on libprotobuf-c version 0, but EL7 choose new version 1.

[Fix]:

1. install new libprotobuf-c in force.

[root@t430z Downloads]# rpm -Uvh protobuf-c-*
error: Failed dependencies:
        libprotobuf-c.so.0()(64bit) is needed by (installed) libgadu-1.12.0-1.el7.centos.x86_64
[root@t430z Downloads]# rpm -Uvh protobuf-c-* --nodeps
Preparing...                          ################################# [100%]
Updating / installing...
   1:protobuf-c-1.0.1-1.el7           ################################# [ 25%]
   2:protobuf-c-devel-1.0.1-1.el7     ################################# [ 50%]
Cleaning up / removing...
   3:protobuf-c-devel-0.15-8.el7.cento################################# [ 75%]
   4:protobuf-c-0.15-8.el7.centos     ################################# [100%]

[root@t430z Downloads]# rpm -Uvh protobuf-c-*

error: Failed dependencies:

libprotobuf-c.so.0()(64bit) is needed by (installed) libgadu-1.12.0-1.el7.centos.x86_64

[root@t430z Downloads]# rpm -Uvh protobuf-c-* --nodeps

Preparing... ################################# [100%]

Updating / installing...

1:protobuf-c-1.0.1-1.el7 ################################# [ 25%]

2:protobuf-c-devel-1.0.1-1.el7 ################################# [ 50%]

Cleaning up / removing...

3:protobuf-c-devel-0.15-8.el7.cento################################# [ 75%]

4:protobuf-c-0.15-8.el7.centos ################################# [100%]

2. download latest libgadu src rpm from koji:

libgadu-1.12.0-4.fc22.src.rpm

It’s based on protobuf-c version 1, and was prepared for fc22, instead of fc21.

3. re- compile new rpm for EL7

update libgadu.

[root@t430z workspace]# rpm -Uvh libgadu-1.12.0-4.el7.centos.x86_64.rpm libgadu-devel-1.12.0-4.el7.centos.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:libgadu-1.12.0-4.el7.centos      ################################# [ 25%]
   2:libgadu-devel-1.12.0-4.el7.centos################################# [ 50%]
Cleaning up / removing...
   3:libgadu-devel-1.12.0-1.el7.centos################################# [ 75%]
   4:libgadu-1.12.0-1.el7.centos      ################################# [100%]

[root@t430z workspace]# rpm -Uvh libgadu-1.12.0-4.el7.centos.x86_64.rpm libgadu-devel-1.12.0-4.el7.centos.x86_64.rpm

Preparing... ################################# [100%]

Updating / installing...

1:libgadu-1.12.0-4.el7.centos ################################# [ 25%]

2:libgadu-devel-1.12.0-4.el7.centos################################# [ 50%]

Cleaning up / removing...

3:libgadu-devel-1.12.0-1.el7.centos################################# [ 75%]

4:libgadu-1.12.0-1.el7.centos ################################# [100%]

4. yum update now works .

Running fcitx on EL7 xfce

I thought it should be an easy task, while not…

My Env:

locale: en_US.UTF-8
kernel: 3.10.0-123.4.2.el7.x86_64

I removed ibus related rpms except “ibus-libs”, as it’s required by gnome/gtk.

then build all fcitx packages:
From:

fcitx-4.2.8.4-1.fc20.src.rpm
fcitx-configtool-0.4.7-2.fc20.src.rpm

1 2	fcitx-4.2.8.4-1.fc20.src.rpm fcitx-configtool-0.4.7-2.fc20.src.rpm

To:

fcitx-libs-4.2.8.4-1.el7.centos.x86_64
fcitx-gtk3-4.2.8.4-1.el7.centos.x86_64
fcitx-gtk2-4.2.8.4-1.el7.centos.x86_64
fcitx-table-4.2.8.4-1.el7.centos.x86_64
fcitx-configtool-0.4.7-2.el7.centos.x86_64
fcitx-data-4.2.8.4-1.el7.centos.noarch
fcitx-pinyin-4.2.8.4-1.el7.centos.x86_64
fcitx-devel-4.2.8.4-1.el7.centos.x86_64
fcitx-4.2.8.4-1.el7.centos.x86_64
fcitx-table-chinese-4.2.8.4-1.el7.centos.noarch

fcitx-libs-4.2.8.4-1.el7.centos.x86_64

fcitx-gtk3-4.2.8.4-1.el7.centos.x86_64

fcitx-gtk2-4.2.8.4-1.el7.centos.x86_64

fcitx-table-4.2.8.4-1.el7.centos.x86_64

fcitx-configtool-0.4.7-2.el7.centos.x86_64

fcitx-data-4.2.8.4-1.el7.centos.noarch

fcitx-pinyin-4.2.8.4-1.el7.centos.x86_64

fcitx-devel-4.2.8.4-1.el7.centos.x86_64

fcitx-4.2.8.4-1.el7.centos.x86_64

fcitx-table-chinese-4.2.8.4-1.el7.centos.noarch

switch im doesnt work.

I have to rebuild package: imsettings and im-chooser. as the origin one doesnt enable support for XFCE.

1. fetch following packages: (as f20 doesnt provide the latest im-chooser package, I use el7 version.)

im-chooser-1.6.4-4.el7.src.rpm
imsettings-1.6.7-1.fc20.src.rpm

1 2	im-chooser-1.6.4-4.el7.src.rpm imsettings-1.6.7-1.fc20.src.rpm

2. modify spec to support el7. (enable xfce supports)

[root@t430z tmp]# diff im-chooser.spec /home/suse/workspace/im-chooser.spec
8a9,10
&gt; %global _with_xfce 1
&gt;

[root@t430z tmp]# diff im-chooser.spec /home/suse/workspace/im-chooser.spec

8a9,10

> %global _with_xfce 1

[root@t430z tmp]# diff imsettings.spec /home/suse/workspace/imsettings.spec
12d11
&lt; %if !0%{?rhel}
14d12
&lt; %endif
118d115
&lt; %if !0%{?rhel}
189d185
&lt; %endif
218,220c214
&lt; %if 0%{?rhel}
&lt; rm -f $RPM_BUILD_ROOT%{_libdir}/imsettings/libimsettings-{lxde,xfce,mate-gsettings}.so
&lt; %endif
---
&gt; #rm -f $RPM_BUILD_ROOT%{_libdir}/imsettings/libimsettings-{lxde,xfce,mate-gsettings}.so
293d286
&lt; %if !0%{?rhel}
309d301
&lt; %endif

[root@t430z tmp]# diff imsettings.spec /home/suse/workspace/imsettings.spec

12d11

< %if !0%{?rhel}

14d12

< %endif

118d115

< %if !0%{?rhel}

189d185

< %endif

218,220c214

< %if 0%{?rhel}

< rm -f $RPM_BUILD_ROOT%{_libdir}/imsettings/libimsettings-{lxde,xfce,mate-gsettings}.so

< %endif

---

> #rm -f $RPM_BUILD_ROOT%{_libdir}/imsettings/libimsettings-{lxde,xfce,mate-gsettings}.so

293d286

< %if !0%{?rhel}

309d301

< %endif

3. im-chooser build has deps on imsettings-devel:
build imsettings first and inst devel pkg :

[root@t430z workspace]# rpm -Uvh /root/rpmbuild/RPMS/x86_64/imsettings-devel-1.6.7-1.el7.centos.x86_64.rpm --nodeps
Preparing... ################################# [100%]
Updating / installing...
1:imsettings-devel-1.6.7-1.el7.cent################################# [100%]

[root@t430z workspace]# rpm -Uvh /root/rpmbuild/RPMS/x86_64/imsettings-devel-1.6.7-1.el7.centos.x86_64.rpm --nodeps

Preparing... ################################# [100%]

Updating / installing...

1:imsettings-devel-1.6.7-1.el7.cent################################# [100%]

4. after all packages built, install/update:

[root@t430z x86_64]# rpm -Uvh imsettings-xfce-1.6.7-1.el7.centos.x86_64.rpm imsettings-1.6.7-1.el7.centos.x86_64.rpm imsettings-gsettings-1.6.7-1.el7.centos.x86_64.rpm imsettings-libs-1.6.7-1.el7.centos.x86_64.rpm im-chooser-xfce-1.6.4-4.el7.centos.x86_64.rpm im-chooser-common-1.6.4-4.el7.centos.x86_64.rpm im-chooser-1.6.4-4.el7.centos.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:imsettings-libs-1.6.7-1.el7.cento################################# [ 8%]
2:im-chooser-xfce-1.6.4-4.el7.cento################################# [ 17%]
3:im-chooser-common-1.6.4-4.el7.cen################################# [ 25%]
4:imsettings-1.6.7-1.el7.centos ################################# [ 33%]
5:imsettings-xfce-1.6.7-1.el7.cento################################# [ 42%]
6:imsettings-gsettings-1.6.7-1.el7.################################# [ 50%]
7:im-chooser-1.6.4-4.el7.centos ################################# [ 58%]
Cleaning up / removing...
8:im-chooser-1.6.4-4.el7 ################################# [ 67%]
9:im-chooser-common-1.6.4-4.el7 ################################# [ 75%]
10:imsettings-gsettings-1.6.3-9.el7 ################################# [ 83%]
11:imsettings-1.6.3-9.el7 ################################# [ 92%]
12:imsettings-libs-1.6.3-9.el7 ################################# [100%]

[root@t430z x86_64]# rpm -Uvh imsettings-xfce-1.6.7-1.el7.centos.x86_64.rpm imsettings-1.6.7-1.el7.centos.x86_64.rpm imsettings-gsettings-1.6.7-1.el7.centos.x86_64.rpm imsettings-libs-1.6.7-1.el7.centos.x86_64.rpm im-chooser-xfce-1.6.4-4.el7.centos.x86_64.rpm im-chooser-common-1.6.4-4.el7.centos.x86_64.rpm im-chooser-1.6.4-4.el7.centos.x86_64.rpm

Preparing... ################################# [100%]

Updating / installing...

1:imsettings-libs-1.6.7-1.el7.cento################################# [ 8%]

2:im-chooser-xfce-1.6.4-4.el7.cento################################# [ 17%]

3:im-chooser-common-1.6.4-4.el7.cen################################# [ 25%]

4:imsettings-1.6.7-1.el7.centos ################################# [ 33%]

5:imsettings-xfce-1.6.7-1.el7.cento################################# [ 42%]

6:imsettings-gsettings-1.6.7-1.el7.################################# [ 50%]

7:im-chooser-1.6.4-4.el7.centos ################################# [ 58%]

Cleaning up / removing...

8:im-chooser-1.6.4-4.el7 ################################# [ 67%]

9:im-chooser-common-1.6.4-4.el7 ################################# [ 75%]

10:imsettings-gsettings-1.6.3-9.el7 ################################# [ 83%]

11:imsettings-1.6.3-9.el7 ################################# [ 92%]

12:imsettings-libs-1.6.3-9.el7 ################################# [100%]

5. change immodules.cache as Im using en locale.

[root@t430z tmp]# cat /usr/lib64/gtk-2.0/2.10.0/immodules.cache
# GTK+ Input Method Modules file
# Automatically generated file, do not edit
# Created by gtk-query-immodules-2.0-64 from gtk+-2.24.22
#
# ModulesPath = /root/.gtk-2.0/2.10.0/x86_64-redhat-linux-gnu/immodules:/root/.gtk-2.0/2.10.0/immodules:/root/.gtk-2.0/x86_64-redhat-linux-gnu/immodules:/root/.gtk-2.0/immodules:/usr/lib64/gtk-2.0/2.10.0/x86_64-redhat-linux-gnu/immodules:/usr/lib64/gtk-2.0/2.10.0/immodules:/usr/lib64/gtk-2.0/x86_64-redhat-linux-gnu/immodules:/usr/lib64/gtk-2.0/immodules
#
"/usr/lib64/gtk-2.0/2.10.0/immodules/im-fcitx.so"
"fcitx" "Fcitx (Flexible Input Method Framework)" "fcitx" "/usr/share/locale" "en:ja:ko:zh"

[root@t430z tmp]# cat /usr/lib64/gtk-2.0/2.10.0/immodules.cache

# GTK+ Input Method Modules file

# Automatically generated file, do not edit

# Created by gtk-query-immodules-2.0-64 from gtk+-2.24.22

# ModulesPath = /root/.gtk-2.0/2.10.0/x86_64-redhat-linux-gnu/immodules:/root/.gtk-2.0/2.10.0/immodules:/root/.gtk-2.0/x86_64-redhat-linux-gnu/immodules:/root/.gtk-2.0/immodules:/usr/lib64/gtk-2.0/2.10.0/x86_64-redhat-linux-gnu/immodules:/usr/lib64/gtk-2.0/2.10.0/immodules:/usr/lib64/gtk-2.0/x86_64-redhat-linux-gnu/immodules:/usr/lib64/gtk-2.0/immodules

"/usr/lib64/gtk-2.0/2.10.0/immodules/im-fcitx.so"

"fcitx" "Fcitx (Flexible Input Method Framework)" "fcitx" "/usr/share/locale" "en:ja:ko:zh"

6. reboot, and use imsettings-switch, im-chooser to choose fcitx. reboot.

If you hit issues, you may try w/r here:
Note for GNOME Later than 3.6

update thunderbird xpi `Quote Colors`

There was a very nice plugin for tb, named “quote-colors”:
https://addons.mozilla.org/en-us/thunderbird/addon/quote-colors/

but currently it doesnt support a higher version > tb 3.0

simply it can be done as following:

1. Download xpi:

mv quote_colors-0.3-tb+sm.xpi  quote_colors-0.3-tb+sm.xpi.zip
unzip quote_colors-0.3-tb+sm.xpi.zip

1 2	mv quote_colors-0.3-tb+sm.xpi quote_colors-0.3-tb+sm.xpi.zip unzip quote_colors-0.3-tb+sm.xpi.zip

2. vi install.rdf

[suse@suse /tmp/quotecolors]$diff install.rdf.old install.rdf
16c16
&lt;         &lt;em:maxVersion&gt;3.0.*&lt;/em:maxVersion&gt;
---
&gt;         &lt;em:maxVersion&gt;33.0.*&lt;/em:maxVersion&gt;

[suse@suse /tmp/quotecolors]$diff install.rdf.old install.rdf

16c16

< <em:maxVersion>3.0.*</em:maxVersion>

---

> <em:maxVersion>33.0.*</em:maxVersion>

3. update and pack zip

zip -u quote_colors-0.3-tb+sm.xpi.zip
cp quote_colors-0.3-tb+sm.xpi.zip  quote_colors-0.3-tb+sm.xpi

1 2	zip -u quote_colors-0.3-tb+sm.xpi.zip cp quote_colors-0.3-tb+sm.xpi.zip quote_colors-0.3-tb+sm.xpi

4. install the new xpi.