Troubleshooting network issues with simple commands is not a very difficult topic once there is a little toolbox to pull from. Needless to say networking topics can be very complicated and any specialist on the field will tell you how cumbersome it can really get. However with a few set of commands one can get some important information that will help when dealing with network issues or in doubt.
This article isn’t the panacea but will give some context to use each command, so any user, specially novice ones can start digging when encountering alledged network ‘problems’. As a reminder be careful where you do use these commands. Using some of these at work can trigger some suspicious looks on you and you may even cause trouble, so don’t do things if in doubt.
If you find the articles in Adminbyaccident.com useful to you, please consider making a donation.
Use this link to get $200 credit at DigitalOcean and support Adminbyaccident.com costs.
Get $100 credit for free at Vultr using this link and support Adminbyaccident.com costs.
Mind Vultr supports FreeBSD on their VPS offer.
To start of the mechanics of this article we’ll use the following example.
Context 0. As a user on a UNIX device you want to know the ip address of your own box.
Command. ifconfig
The ifconfig command allows us to not only get information relative to the network interfaces but also to configure it. For the context 0 we will issue the ‘ifconfig’ command at the command prompt and explain a few bits of the result.
[albert@Barcelona ~]$ ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,
VLAN_HWFILTER>
ether 50:65:f3:1f:7b:3c
inet 192.168.54.101 netmask 0xffffff00 broadcast 192.168.54.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[albert@Barcelona ~]$
The ‘em0’ section is relative to the only physical network interface available on this device. Servers will typically have two or more, so one can see names such as em0, em1, em2, etc.
There are other parameters such as if the interface is ‘UP’ like this one, or ‘DOWN’. There is also the support of being assigned an ip by a DHCP server with the ‘BROADCAST’ parameter, the availability of the interface with the ‘RUNNING’ one and some others such as a list of the available options.
The information we really want is the ‘inet’ address on the ‘em0’ interface, which is 192.168.54.101 for our case. We can forget about the rest for now.
The ‘lo0’ interface is always available and it’s the system loopback NIC. Programs can send information between them using that loopback interface without going out of the current box.
Context 1. I know my device’s ip but I want to know if I can reach the internet.
Command. ping
The ‘ping’ command works by using the ICMP protocol of the IP stack. It is used for troubleshooting issues on networks. When using it against an ip we should get a response, the packets that are sent will be replied back. If lost, the line is not available or there’s an issue because some packets are dropped.
We can ping google.com first.
[albert@Barcelona ~]$ ping google.com
PING google.com (172.217.16.238): 56 data bytes
64 bytes from 172.217.16.238: icmp_seq=0 ttl=51 time=32.211 ms
64 bytes from 172.217.16.238: icmp_seq=1 ttl=51 time=32.242 ms
64 bytes from 172.217.16.238: icmp_seq=2 ttl=51 time=31.931 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 31.931/32.128/32.242/0.140 ms
[albert@Barcelona ~]$
As we can see google.com is resolved at the ip address 172.217.16.238. The number of packets can be identified after the ‘icmp_seq=’ column, starting by number zero. The ‘time’ parameter will tell us the latency involved. Typically the closer we are, the lesser the time. The ‘ttl’ parameter is the ‘Time To Live’ which holds the timespan of the packet. If the declared time is overcome the packet will be dropped or the data revalidated again.
A connectivity loss due to a disconnected cable will probably look like this:
[albert@Barcelona ~]$ ping google.com
ping: cannot resolve google.com: Host name lookup failure
[albert@Barcelona ~]$
But a working but faulty connection that sometimes works can look like this:
[albert@Barcelona ~]$ ping google.com
PING google.com (172.217.17.14): 56 data bytes
64 bytes from 172.217.17.14: icmp_seq=0 ttl=51 time=33.422 ms
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
64 bytes from 172.217.17.14: icmp_seq=4 ttl=51 time=33.999 ms
64 bytes from 172.217.17.14: icmp_seq=5 ttl=51 time=33.142 ms
^C
--- google.com ping statistics ---
6 packets transmitted, 3 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 33.142/33.521/33.999/0.357 ms
[albert@Barcelona ~]$
A 50% packet loss comment is quite self explanatory, isn’t it?.
Context 2. Some colleague has internet but you haven’t. You check your cables and everything is connected. What’s wrong?
You can switch words like ‘colleague’ for ‘server’ and you totally changed context from a ‘user-friendly’ one to a panicking sysadmin.
Command. ping, ifconfig
Yet again ping can be useful but we need to know if we are even arriving to the gateway. At home the gateway will be your router, but at the office it will probably be a switch device allowing your device to send and receive packets and having an ip assigned to it (like if it gave it life itself). How to spot the gateway? Easy.
Launch ifconfig like so.
[albert@Barcelona ~]$ ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
ether 50:65:f3:1f:7b:3c
inet 192.168.54.101 netmask 0xffffff00 broadcast 192.168.54.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[albert@Barcelona ~]$
Notice the numbers at the right of the ‘broadcast’ parameter. This is the range this LAN (network) has been given. From 192.168.54.1 up to 192.168.54.255. This is a /24 wide network. And our device has been given an address within that range. In our example the address is 192.168.54.101.
But we want to reach the internet yet we can’t. Let’s check if we get to the gateway. But where does the gateway sit? Typically network administrators assign the first ip of the broadcast range to the device routing the packets. Let’s then ping 192.168.54.1 and see what happens.
[albert@Barcelona ~]$ ping 192.168.54.1
PING 192.168.54.1 (192.168.54.1): 56 data bytes
64 bytes from 192.168.54.1: icmp_seq=0 ttl=64 time=0.442 ms
64 bytes from 192.168.54.1: icmp_seq=1 ttl=64 time=0.515 ms
^C
--- 192.168.54.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.442/0.479/0.515/0.036 ms
[albert@Barcelona ~]$
We can reach the gateway, so our device has connectivity to a part of the network. The problem is not on our device but somewhere down the line. If you contact support this will help them. But if you are a sysadmin you can tell this exact same to your network administrator.
Context 3. I can reach the gateway but I can’t reach the internet. I want to know at what point I do get stuck.
Command. traceroute
The ‘traceroute’ command will display the route packets are taking. So in the case you want to check what steps is your computer making until it gets to a specific resource, you can use this command. Let’s try reaching google.com first.
[albert@Barcelona ~]$ traceroute google.com
traceroute to google.com (216.58.201.174), 64 hops max, 40 byte packets
1 Home Router (192.168.54.1) 1.238 ms 1.228 ms 1.150 ms
2 81.32.24.156 (81.32.24.156) 29.833 ms 22.981 ms 23.767 ms
3 10.255.209.113 (10.255.209.113) 22.611 ms 22.635 ms 22.764 ms
4 10.34.201.137 (10.34.201.137) 23.650 ms 23.676 ms 24.170 ms
5 10.34.201.133 (10.34.201.133) 23.413 ms 23.292 ms 24.022 ms
6 81.52.188.113 (81.52.188.113) 23.550 ms 23.386 ms 23.679 ms
7 193.251.242.4 (193.251.242.4) 23.355 ms 24.359 ms 24.501 ms
8 193.251.132.198 (193.251.132.198) 33.056 ms
193.251.252.95 (193.251.252.95) 31.989 ms
193.251.128.104 (193.251.128.104) 34.295 ms
9 193.251.132.51 (193.251.132.51) 31.304 ms
193.251.133.56 (193.251.133.56) 33.106 ms
193.251.243.147 (193.251.243.147) 32.669 ms
10 72.14.195.132 (72.14.195.132) 33.434 ms 31.639 ms
google-19.gw.opentransit.net (193.251.255.26) 31.851 ms
11 108.170.253.225 (108.170.253.225) 31.975 ms 31.808 ms 31.978 ms
12 108.170.233.247 (108.170.233.247) 32.396 ms
72.14.233.125 (72.14.233.125) 33.800 ms
108.170.233.247 (108.170.233.247) 32.730 ms
13 google.com (216.58.201.174) 32.298 ms 32.849 ms 32.898 ms
[albert@Barcelona ~]$
After 13 hops the packet reaches its final destination google.com. First the packet reaches the gateway, in this example a home router placed at 192.168.54.1. Then it goes out through the home router’s public interface on address 81.32.24.156. It then goes into an exchange where a NAT is configured. From then it starts jumping through several network devices until it gets to its final destination.
In an office environment with a public interface issue the result can easily look similar to this:
[albert@Barcelona ~]$ traceroute google.com
traceroute to google.com (216.58.201.174), 64 hops max, 40 byte packets
1 10.24.130.1 (10.24.130.1) 1.238 ms 1.228 ms 1.150 ms
2 10.10.2.1 (10.10.2.1) 2.833 ms 2.981 ms 2.767 ms
3 10.2.10.5 (10.2.10.5) 2.611 ms 2.635 ms 2.764 ms
4 * * *
5 * * *
6 * * *
^C
[albert@Barcelona ~]$
Anytime you see those asterisks * there’s been a reset and a new hop is attempted. And as you can see the gateway here is a Cisco device with ip address 10.24.130.1. The packet has reached the gateway and it’s been routed to the next step, where another network device has trated it. The ip address 10.10.2.1 can be a second switch that has also routed the packet to the address 10.2.10.5 where the firewall has stopped it. This is highly hypothetical since your environment’s network topology can be very different. However giving this information to your support clerk or your network administrator can help him out figuring out your pc’s / server’s issues.
In this above example a new or modified rule in the firewall may be blocking internet access to all the users sitting at the VLAN 10.24.130.1/24.
Context 4. You know where you are blocked but you want to know what device 10.2.10.5 is related to.
Command. nslookup.
The nslookup command is relative to a program that makes queries to domain name servers. In a big corporation environment is quite easy that your company has some type of Domain Controller (such as a box equipped with Microsoft’s Active Directory). Usually companies add entries to every single device so IT personnel doesn’t have to remember every ip on their network nor they have to use huge and impractical tables on spreadsheets. So in theory you could query what device name has the IT team give to the device sitting at ip 10.2.10.5. The result may be unintelligible for you, but they know.
[albert@Barcelona ~]$ nslookup 165.227.245.121
121.245.227.165.in-addr.arpa name = adminbyaccident.com.
[albert@Barcelona ~]$
As you can see solving the ip 165.227.245.121 we get this is adminbyaccident.com’s address.
Context 5. You are a contractor sending important logs to a customer, yet they don’t receive any bits from the mentioned box. You want to show them the box is sending out packets and the problem is network related.
Command. tcpdump.
You’ll love tcpdump. It does what it says, so it dumps data from a network connection. Let’s make one dump. On one terminal we’ll launch a ping stream of packets to simulate the flow of packets for our context. On a different terminal we will use tcpdump to verify the stream is flowing.
Terminal 1.
[albert@Barcelona ~]$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=54 time=32.289 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=32.629 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=32.234 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=32.528 ms
Terminal 2
root@Barcelona:~ # tcpdump -i em0 dst 1.1.1.1 >> /var/log/albert.collect
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C3 packets captured
26 packets received by filter
0 packets dropped by kernel
root@Barcelona:~ #
When we cancel the tcpdump and print the content collected at /var/log/albert.collect we can see:
root@Barcelona:~ # cat /var/log/albert.collect
21:38:00.047156 IP 192.168.54.101 > one.one.one.one: ICMP echo request, id 58894, seq 5, length 64
21:38:01.107128 IP 192.168.54.101 > one.one.one.one: ICMP echo request, id 58894, seq 6, length 64
21:38:02.111168 IP 192.168.54.101 > one.one.one.one: ICMP echo request, id 58894, seq 7, length 64
root@Barcelona:~ #
This is the proof the box we are on is sending packets to the 1.1.1.1 ip adress. If there were no packets there wouldn’t be any captured and thus the file empty.
Context 6. You are on the other side, you are the customer and the service provider states he is sending logs to you but to no avail since you aren’t getting any. How to show them.
Command. tcpdump.
Yet again, tcpdump can help us out. This time instead of setting the ‘dst’ flag for the destination parameter, we’ll use the ‘src’ one. Let’s imagine we should be getting logs from ip 81.32.24.156.
root@Barcelona:~ # tcpdump -i em0 src 81.32.24.156 >> /var/log/albert.collect
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C0 packets captured
73 packets received by filter
0 packets dropped by kernel
root@Barcelona:~ #
For a lapse of approx 5 seconds we’ve launched tcpdump and the interface has had 73 packets through it, however none was from the source we where expecting from, ip 81.32.24.156. Let’s check it with cat.
root@Barcelona:~ # cat /var/log/albert.collect
root@Barcelona:~ #
The file is empty.
A similar way to produce the same result:
root@Barcelona:~ # tcpdump -w "/var/log/albert.collect" -i em0 src 81.32.24.156
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C0 packets captured
3 packets received by filter
0 packets dropped by kernel
root@Barcelona:~ #
Context 7. Is this box really in physical contact with the switch it is supposed to be conected to?
Command. arp.
The ‘arp’ command corresponds to the ARP protocol. And using it we can get the latest MAC addresses our box has been encountered with, hence other boxes.
[albert@Barcelona ~]$ arp -a
Fedora (192.168.73.109) at 08:00:17:b9:88:74 on em0 expires in 1148 seconds [ethernet]
Router (192.168.54.1) at 6a:df:96:49:3e:fe on em0 expires in 939 seconds [ethernet]
? (192.168.54.101) at 50:15:f6:1f:5b:3d on em0 permanent [ethernet]
elk (192.168.73.113) at 08:00:17:f9:2d:09 on em0 expires in 988 seconds [ethernet]
[albert@Barcelona ~]$
As we can see there’s been a few boxes, such as a Fedora VM sitting on 192.168.73.109, the firewall and router device located at 192.168.54.1, as well as an unknown device marked with an interrogant and with ip 192.168.54.101 and finally an ELK stack VM at 192.168.73.113.
On a Windows box the output is similar-:
PS C:\Users> arp -a
Interface: 192.168.1.100 --- 0x14
Internet Address Physical Address Type
192.168.54.1 6a-df-96-49-3e-fe dynamic
192.168.54.101 50-15-f6-1f-5b-3d dynamic
192.168.54.255 ff-ff-ff-ff-ff-ff static
224.0.0.2 01-00-5e-00-00-02 static
224.0.0.22 01-00-5e-00-00-16 static
224.0.0.251 01-00-5e-00-00-fb static
224.0.0.252 01-00-5e-00-00-fc static
239.255.255.250 01-00-5e-7f-ff-fa static
255.255.255.255 ff-ff-ff-ff-ff-ff static
PS C:\Users>
In case there are no other devices found on the arp table there’s a very good chance the physical cable isn’t present or broken. Try a few pings to other devices on your network, such as the gateway and/or your colleagues computer. After that some entries should appear on the arp table. If any something is broken and it’s probably something physical.
Context 8. Who is actually connected to this server of mine?
Command. netstat.
The netstat command has been very helpful to me in order to check actual connections on a box, such as a web server. This command as the man page states shows network status and statistics.
To check who is currently connected one may try the next command:
[Get@This ~]$ netstat -4
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 localhost.9000 localhost.58143 TIME_WAIT
tcp4 0 36 128.48.114.12.ssh 95.160.134.16.49452 ESTABLISHED
tcp4 0 0 localhost.9000 localhost.33542 TIME_WAIT
tcp4 0 1144 128.48.114.12.ssh 222.186.30.145.32171 ESTABLISHED
udp4 0 0 localhost.ntp *.*
udp4 0 0 10.19.0.5.ntp *.*
udp4 0 0 128.48.114.12.ntp *.*
[Get@This ~]$
The server is sitting on the ip 128.48.114.12 and has two ssh connections. One legitimate and the other one from a nasty and persistant attacker. Fail2ban will take care of the latter.
Context 9. Is that port on that box really open?
Command. netcat.
The netcat command does one thousand, good, interesting things. It is called a networking swiss army knife for a reason.
Let’s try if port 80 is open on this very website.
albert@Barcelona:~ # nc -v www.adminbyaccident.com 80
Connection to www.adminbyaccident.com 80 port [tcp/http] succeeded!
^C
albert@Barcelona:~ #
And indeed it’s open. However port 10000 is closed.
albert@Barcelona:~ # nc -v www.adminbyaccident.com 10000
nc: connect to www.adminbyaccident.com port 10000 (tcp) failed: Operation timed out
albert@Barcelona:~ #
In fact once the connection had been stablish on port 80, the web server would accept commands. However there are a few security barriers in place so it doesn’t answer unless you use a browser.
albert@Barcelona:~ # nc -v www.adminbyaccident.com 80
Connection to www.adminbyaccident.com 80 port [tcp/http] succeeded!
GET / HTTP/1.1
host: www.adminbyaccident.comHTTP/1.1 400 Bad Request
Date: Thu, 20 Feb 2020 20:12:33 GMT
Server: Apache/2.4.41 (FreeBSD) OpenSSL/1.1.1d-freebsd
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
^C
albert@Barcelona:~ #
Context 10. What ip does this other box have? What host name does this ip have?
Command. host.
The host command does one thing and it performs it very conveniently. It queries DNS servers, so whenever you ask for an ip it will tell you the hostname of the box you are asking for or reverse. You may be asking for a hostname or an A record and it will tell you what ip does that have. Simple. Nice. Good.
We first ask for a hostname, or domain name, better said.
albert@Barcelona:~ # host adminbyaccident.com
adminbyaccident.com has address 165.227.245.121
adminbyaccident.com mail is handled by 1 redirect.ovh.net.
albert@Barcelona:~ #
Asking for an ip will resolve the hostname or any DNS record associated to it.
albert@Barcelona:~ # host 165.227.245.121
121.245.227.165.in-addr.arpa domain name pointer adminbyaccident.com.
albert@Barcelona:~ #
So this is it for this simple but I hope a bit useful article on a few useful network commands.
If you find the articles in Adminbyaccident.com useful to you, please consider making a donation.
Use this link to get $200 credit at DigitalOcean and support Adminbyaccident.com costs.
Get $100 credit for free at Vultr using this link and support Adminbyaccident.com costs.
Mind Vultr supports FreeBSD on their VPS offer.