Troubleshooting Network Issues¶

Step 1: Check Network Addresses (on tsuru)¶

After ssh’ing to tsuru, run:

ip address

Interfaces 2, 4, and 5 should be UP.

If DOWN, run Ron’s setup scripts to configure the ip addresses for the network interfaces:

bash ~rp/setup.sh
ip address

We want to make sure the camera is initialize when testing the network connections.

First, log in to tsuru (vnc is helpful here to run Xi-CAM).

If the Controls GUI does not show in Xi-CAM Acquire after selecting fastccd device, check that you have $PASSWD_MONGO and $USER_MONGO set.

If initalizing the camera (via the Initialize button) fails, check the service logs:

journalctl -f -u fastccd_support_ioc

This may show something like socket.timeout=timed out.

Re-check the network interface addresses (Step 1).

Retry the initialization of the camera in Xi-CAM.

We will then want to test the connection between tsuru and the camera interface node.

CIN IP: 10.0.5.207

On tsuru, run:

sudo ping -f 10.0.5.207

After a few seconds, CTRL-C and check packet loss (20% is usual).

Next, we will want to look at the connection from the fastccdATCA1 server to the fabric switch.

ssh to fastccdatca1.dchp.lbl.gov and repeat the ping flood from step 3.

This may also show packet loss of around 20%.

is this relevant??? – recording for note purposes, modify later

On fastcameraatca1 and tsuru, check the net.core.rmem_default and net.core.rmem_max values:

systemctl net.core.rmem_default
systemctl net.core.rmem_max

Max should match default. If not, modify net.core.rmem_default to match the net.core.rmem_max in the following files:

Then, reboot:

sudo shutdown -r 0

After rebooting, reconnect and repeat steps 3 and 4.