[aclug-L] Fw: Multiple Networks On One Wire

["Dustin" <dustin@xxxxxxxxxxxxxx>]

Found this in some google searching I was doing and thought some people
might find it of interest:

Networking on Steroids with Linux

By Moshe Bar, 24.6.1999

Moshe Bar is an Israeli system administrator and OS researcher, who started
learning UNIX on a PDP-11 with AT&T Unix Release 6 back in 1981. He holds a
M.Sc in computer science and is finishing his Ph.D. thesis. Visit Moshe's
website or email him.

Have you ever wondered how those people in charge of networking in your
company
make incredible things happen like having two networks on the same wire? Or
having multiple web servers respond to the same url? This article shows some
of
the secrets of the trade.

One of the most powerful weapons for networking tricks is ARP/RARP. The
Address
Resolution Protocol is a distant cousin of TCP/IP and is in fact implemented
with it in most Unixes in kernel space (in the PC world also knows as
protected
mode). Its function is to translate logical IP addresses into Ethernet
hardware
(the network card) addresses, which are unique for every adapter. RARP
(Reverse
ARP), however, is usually implemented as a user-land protocol suite.

Every time you access a Byte.com web page, your IP packets travel through a
complex maze of hubs switches routers, hosts and networks. All of these
systems
must fulfill the real-world requirements of high availability, redundancy
and
speed. To make it work, the networking guys have to resort to fool around
with
IP addresses and MAC addresses (Ethernet Adapter addresses) and fooling
client's ARP request into believing their temporary version of network
topology. How can that be achieved?

The ARP Protocol

ARP is useful for locating the MAC address or Ethernet address corresponding
to
a certain IP address. IP addresses for IP version 4 (the current Internet)
are
32bit long like 192.168.1.199. MAC addresses are 48bit long (the first 24bit
indicate the vendor, HP MACs always start with 00:60:b0). ARP is the primary
method in a TCP/IP network to establish IP connections to new, previously
uncontacted hosts. ARP is defined as a low-level protocol that hides the
underlying network physical addressing, permitting one to assign an
arbitrary
IP address to every machine. Think of ARP as part of the physical network
system, and not as part of the internet protocol.

If a machine knows its own MAC address but doesn't know its IP address (such
as
diskless clients, network computers and PC without name service configured)
it
can use RARP. DHCP, the Dynamic Host Configuration Protocol as described in
RFC
983, widely used in corporate LANs to auto-configure workstations during
boot-up, has to use RARP.

As was mentioned earlier RARP usually is a user-space implementation in
Unixes
because it makes higher degree of configurability possible. RARP reads the
file
/etc/ethers to correlate MAC addresses to IP addresses. This file has to be
maintained by the SysAdmin.

ARP failures usually have devastating consequences for the entire network
segment (potentially thousands of computers). The symptoms usually are slow
or
intermittent network connections and occasional broadcast storms. As a
result
NFS performance slows down to a crawl and just opening a small file over the
network gives the impression of wading through molasses.

RARP failures just affect one machine and don't have consequences for other
hosts in the network. The workstation will fail to boot properly and the
display usually shows some message like this:

No response for MAC address 8:0:20:01:b:03

Proxy ARP and Published ARP

ARP maintains its cache of IP:MAC mappings on each machine. On a Linux
system,
display it out using arp -a

[root@mosheb /root]# arp -a

saturn.toptier.co.il (192.168.100.7) at 00:10:4B:41:5C:3A [ether] on eth0

? (212.28.208.9) at 00:08:C7:9F:45:06 [ether] on eth0

? (212.28.208.1) at 00:10:4B:41:56:28 [ether] on eth0

A Solaris system gives you far more detail:

# arp -a

Net to Media Table

Device   IP Address              Mask     Flags  Phys Addr

------ -------------------- --------------- ----- ---------------

hme0   194.88.76.178       255.255.255.255      00:80:5f:ad:63:16

hme0   mailhost.uk     255.255.255.255      00:a0:24:e7:d3:4b

hme0   e0-2501-13.uk       255.255.255.255       00:00:0c:8e:ee:68

hme0   194.88.76.147       255.255.255.255      00:90:27:78:73:18

hme0   194.88.76.165       255.255.255.255      00:a0:cc:52:6a:86

hme0   mosheb              255.255.255.255 SP   08:00:20:a7:96:ab

hme0   194.88.76.89        255.255.255.255      00:10:4b:78:ec:cf

hme0   194.88.76.100       255.255.255.255      00:40:05:33:30:89

hme0   194.88.76.4         255.255.255.255      00:40:90:d0:0b:73

hme0 BASE-ADDRESS.MCAST.NET 240.0.0.0     SM   01:00:5e:00:00:00

Completed entries represent queries that returned at least one reply. Those
marked "U" or "UNRESOLVED" are from queries that were never answered,
because
the IP address in not on the local network or because the host owning it was
down or not listening.

Entries marked "S", static, or "PERM", permanent, will never be removed from
the cache, typically there are inserted on behalf of devices that have
limited
TCP/IP implementations (like printers) and don't answer ARP requests. To
create
a permanent ARP entry try as root:

mosheb# arp -s 192.168.1.120 00:60:40:b1:a:83

A published entry is one for which the local host acts as a proxy server.
When
the local host sees an ARP request come by for that IP address, it answers
with
the MAC address in the cache, not its own MAC address. Publishing ARP
entries
allows one machine to answer ARP requests for machines that can't hear the
broadcasts because they are on the other side of a router, or because they
are
not listening. Published entries are marked with a "P" or "PUBLISH" in the
ARP
cache report. Have a look at the ARP man-pages to more info on ARP.

Using permanent entries prevents the local machine from sending ARP
requests;
creating a published entry allows the local machine to answer ARP requests
coming from another machine, on behalf of hosts that can't handle their own
ARP
traffic.

ARP entries age in the cache. An entry that is not completed within 5
minutes
is removed and entries not used in 20 minutes are flushed.

Proxy ARP allows you to designate one machine to answer ARP requests for
others
(hence the term proxy ARP), providing the MAC address of the router, for
instance, to PPP clients connection over phone lines. The router knows how
to
handle forwarding to the correct destination. You could also use published
ARP
entries, but that would fill slots in the ARP cache, making this solution
minimally scalable. Publishing ARP entries is adequate for a handful or
remote
machines; to support hundreds of them you will be better of with a proxy ARP
server so that the machine's ARP cache doesn't become full of infrequently
used
entries.

Multi-homed hosts and MAC addresses

If you are connecting multiple network interfaces to a switch or hub that
wants
to see a unique MAC address on each port, you will need to modify the
default
Linux configuration. To get a list of all MACs in the system, do a:

mosheb# ifconfig -a

[root@mosheb /root]# ifconfig -a

eth0     Link encap:Ethernet  HWaddr 00:60:97:25:F5:5F

         inet addr:192.168.100.36 Bcast:192.168.127.255  Mask:255.255.128.0

         UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

         RX packets:8109350 errors:0 dropped:0 overruns:0 frame:0

         TX packets:3417583 errors:0 dropped:0 overruns:0 carrier:123

         collisions:9534 txqueuelen:100

         Interrupt:11 Base address:0x6100

eth0:0   Link encap:Ethernet  HWaddr 00:60:97:25:F5:5F

         inet addr:212.29.208.20  Bcast:212.29.208.255  Mask:255.255.255.192

         UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

         Interrupt:11 Base address:0x6100

lo       Link encap:Local Loopback

         inet addr:127.0.0.1  Mask:255.0.0.0

         UP LOOPBACK RUNNING  MTU:3924  Metric:1

         RX packets:105751 errors:0 dropped:0 overruns:0 frame:0

         TX packets:105751 errors:0

You will notice that the same Ethernet address shows up for every interface,
creating an N:1 mapping for that machine that ARP manages just fine on
multiple
networks. When you connect those interfaces to the same network, for
example,
multiple ports of a switch - to gain improved input bandwidth to the server
using 2 IP addresses on dedicated switch segments - the switch is likely to
be
confused by seeing the same MAC address show up on multiple ports.

The easiest way to fix this is by assigning each interface its own MAC
address.
The "ether" paramater to ifconfig allows you to set the MAC address:

mosheb# ifconfig eth0 mosheb-eth0 ether 8:0:ae:83:1:2

mosheb# ifconfig eth0 mosheb-eth0 ether 9:0:ae:83:1:2

mosheb# ifconfig eth0 mosheb-eth0 ether 10:0:ae:83:1:2

Be careful not to set the last bit in the first value in the MAC address, as
this bit is reserved for indicating multicast addresses.

Dealing with the Router

As long as you have just single hosts and switches everything remains easy
and
simple. The trouble starts when you have a middle-man, ie the router. ARP is
the glue that guides the packet from the sender to the router without
cheating
on the value of the real, destination IP address. That's what happens:

My local machine mosheb(IP 194.91.1.20) has to send a packet to node
"sofaraway", at IP 194.90.1.5. After checking the IP routing table, mosheb
decides that the packet has to go to router "baynet" at IP 194.90.1.1.

Mosheb might send an ARP request first for baynet's MAC address,
corresponding
to its IP address determined from the routing tables.

Using the router's MAC address, mosheb constructs the outbound packet with
baynet's MAC address in the Ethernet destination field and sofaraway's IP
address in the destination IP field. The packet then goes from mosheb to the
router (baynet), where it gets another interface based on the destination IP
address. The router may also have to ARP for the next hop's MAC address on
the
next network.

You will have noticed  that keeping an ARP cache significantly reduces
network
latency.

One  frequent error in ARP configuration is the situation where the IP
address
of a machine is actually the broadcast address and it will answer ARP
requests
for all machines. This has to be kept in mind especially when shifting form
0-based broadcast networks to 1-based broadcast networks.

Multiple IP networks on a single wire

Let's get to the magic now. As we saw before, proxy ARP allows you to
distribute an IP network over several physical segments. The advantages of
doing so is that you can use the same IP address space instead of dividing
it
into many subclasses. Now, the beauty of ARP magic is that you can do the
reverse as well, i.e. make several logical IP networks over the same
physical
segment.

The reasons for wanting to do so could be that you would want to change the
IP
numbering of your network and therefore need to run both schemes in parallel
for some time. Or maybe you want to merge two separate networks together
(with
all the mergers and acquisitions going on, this happens very often).
Company
LANs more and more adopt Ethernet switches and the creation of virtual LANs
often requires maintaining several IP networks on a series of ports switched
like a single segment. Instead of changing the IP numbers of your machines,
ARP
gives you the possibility to get around IP routing issues by cleverly
defining
routing and ARP settings.

To merge several IP networks with the routing solution, you use a zero-cost
route that indicates the target network is directly connected to the local
machine. Let's assume the first IP network is 192.9.200.0, and the "new"
network is 201.2.14.0. To add a route to the 201-network, you'd insert the
following line in appropriate place in your start-up scripts (as of Kernel
2.0.34, this feature was planned, however not yet implemented):

mosheb# route add network 201.2.14.0 `hostname` metric 0

Zero-cost routes (ie routes with a metric of zero) tell the TCP/IP stack
that
the local machine will send an ARP request for each machine on the target
network, since that network is accessible over the local eth0 interface.
Normally, a host will send an ARP request using the IP address of the
designated router for the network but in this case the router is the machine
itself so it does the ARP for the destination IP address. Strange-looking
ARP
requests for IP addresses that shouldn't be on a particular network segment
may
be caused by a misplaced zero-cost route.

In another case proxy ARP might help us again: while merging two IP networks
together, you have some "old" machines that have no need to reach other
systems
on the same network, but have to use the single router that has an IP
address
on the "new" IP network. That is, router 201.2.14.100 has to handle packets
from hosts 192.9.200.x, and you don't want to create zero-cost routes on
every
one of those machines. Using proxy ARP or a published ARP entry on one of
the
201.2.14.x machines, create a "fake" IP address for the router on the
192.9.200.0 network, say, 192.9.200.100. Then insert an ARP entry using the
MAC
address of the router, so that ARP request for IP address 192.9.200.10 are
answered with the Ethernet address of your router, and packets will be
delivered to the router from hosts on the 192.9.200.x network. Make sure
those
"old" machines put 192.9.200.100 in their default route at boot, or
otherwise
manually set up a route through the "fake" IP address. Note that this
address
won't appear in RIP packets or other routing information, so you'll have to
use
a route command to point you old machines to the router of multiple
personalities. So you will have to use a route command to point your old
machines at the router of multiple personalities.

Virtual Interfaces

When you're renumbering a network or trying to make a machine impersonate
multiple personalities, you need to create virtual interfaces on top of a
single physical network interface. In Linux (as in most other Unixes) this
capability is built into the ifconfig command and IP implementation.

Virtual interfaces use notations of eth0:1, eth0:2 and so on to number the
virtual inteface created on top of the eth0 device.

To turn on the virtual interface, add new lines to your boot script or type
by
hand:

mosheb#  ifconfig eth0:1 inet 192.168.1.135 netmask 255.255.255.0 up

If you use multiple IP networks on the same physical segment, you may end up
confusing the routing table. A typical problem is a message like "packet
from
unknown router" or "network unreachable". In our previous example, any
machine
on the 192.9.200.0 network may receive a packet from a router using IP
address
201.2.14.100, but become confused when it finds that it has no direct
connection to network 201.2.14.0. Adding a second IP address on the local
interface, or a zero-cost router will solve the problem.

By adding the second and consecutive IP addresses to an interface, you cause
it
to answer ARP requests for all of those addresses. Linux maintains a list of
IP
addresses per interface and matches incoming ARP requests accordingly. The
same
MAC addresses will appear in ARP caches tied to several IP addresses if you
use
the virtual interfaces. This could confuse network management discovery
tools.

Like the virtual memory or file system cache of the Linux kernel, ideally
you
never have to deal with it or know it is there, but when it breatks it makes
your life misery as a SysAdmin. If you know where to investigate and
understand
the general principles, you won't have to stumble in the dark.

Moshe Bar



References

Internetworking with TCP/IP, 3rd Edition, Volume I and II, by Douglas E.
Comer,
Prentice Hall, ISBN 0-13-216987-8, pages 231 & ff.

Solaris Networking Guide, by David Feinstein, Bar-Ilan University


-- This is the discussion@xxxxxxxxx list.  To unsubscribe,
visit http://tmp2.complete.org/cgi-bin/listargate-aclug.cgi