April 2006


These days I’ve implemeted a Linux bridge that does traffic accounting and IP restrictions per mac adress and decided to share my experience. In my setup on eth0 is my gateway and on eth1 is switch with servers connected to it. First we need to bridge eth0 and eth1, you’ll need brctl command, to install it on Gentoo Linux run:


emerge bridge-utils

This commands bridge eth0 and eth1 interfaces, DO NOT run them if you’re connected to the box through eth0 or eth1. You may want to change brname to something more meaningfull:


/sbin/brctl addbr brname
/sbin/brctl setageing brname 86400
/sbin/brctl addif brname eth0
/sbin/brctl addif brname eth1
/sbin/ifconfig brname up

Now as you connect some computers on eth0 and eth1 they should see each other.

Now to limit IP addresses per mac address you will need ebtables /you should enable them in kernel configuration too/, to install it on Gentoo I use:


emerge ebtables

First lets create file /etc/ipmac.conf that will contain mac addresses and IP adresses assigned to it. The file is in format:


00:01:03:D6:63:EF 87.120.40.143

You can assign multiple IPs to one mac address with multiple lines with the same mac. Here is the script that implement this rules for me, remember in my setup servers are on eth1 and their gateway is on eth0:


#!/bin/bash

EBTABLES=/sbin/ebtables

echo Loading Rules

$EBTABLES -F

cat /etc/ipmac.conf | while read -r mac ip; do

echo "Allowing $ip from $mac"
$EBTABLES -A FORWARD -p IPv4 -i eth1 -o eth0 -s $mac --ip-src $ip -j ACCEPT
$EBTABLES -A FORWARD -p IPv4 -i eth0 -o eth1 -d $mac --ip-dst $ip -j ACCEPT
$EBTABLES -A FORWARD -p ARP -i eth1 -o eth0 --arp-opcode 2 --arp-ip-src $ip --arp-mac-src $mac -j ACCEPT
$EBTABLES -A FORWARD -p ARP -i eth0 -o eth1 --arp-opcode 2 --arp-ip-dst $ip --arp-mac-dst $mac -j ACCEPT

done;

$EBTABLES -A FORWARD -p ARP -i eth1 -o eth0 --arp-opcode ! 2 -j ACCEPT
$EBTABLES -A FORWARD -p ARP -i eth0 -o eth1 --arp-opcode ! 2 -j ACCEPT

$EBTABLES -A FORWARD -i eth1 -o eth0 -j DROP
$EBTABLES -A FORWARD -i eth0 -o eth1 -j DROP

save the script and execute it. Now your servers should be able to only assign IP addresses from /etc/ipmac.conf.

Now if you want to have MRTG graphs for every IP and every mac address you will need to have php and mrtg installed, on Gentoo just type:


emerge php mrtg

You will need some web server also to see serv the graphs, in my case it is apache, which home directory is /var/www/localhost/htdocs. You’ll need to create directory for mrtg:


mkdir /var/www/localhost/htdocs/mrtg

Now save this script somewhere and mark it executable:


#!/usr/bin/php
< ?

$data=`/sbin/ebtables -L --Lc --Lmac2`;

$ip=array();
$mac=array();

preg_match_all('#-p IPv4 -s ([0-9a-f\:]+) -i eth1 -o eth0 --ip-src ([0-9\.]+) -j ACCEPT , pcnt = \d+ -- bcnt = (\d+)#', $data, $m);
for ($i=0; $i<count($m[0]); $i++){
if (!isset($ip[$m[2][$i]]))
$ip[$m[2][$i]]=array('in' => 0, 'out' => 0);
if (!isset($mac[$m[1][$i]]))
$mac[$m[1][$i]]=array('in' => 0, 'out' => 0);
$ip[$m[2][$i]]['out']+=1*$m[3][$i];
$mac[$m[1][$i]]['out']+=1*$m[3][$i];
}

preg_match_all('#-p IPv4 -d ([0-9a-f\:]+) -i eth0 -o eth1 --ip-dst ([0-9\.]+) -j ACCEPT , pcnt = \d+ -- bcnt = (\d+)#', $data, $m);
for ($i=0; $i<count ($m[0]); $i++){
if (!isset($ip[$m[2][$i]]))
$ip[$m[2][$i]]=array('in' => 0, 'out' => 0);
if (!isset($mac[$m[1][$i]]))
$mac[$m[1][$i]]=array('in' => 0, 'out' => 0);
$ip[$m[2][$i]]['in']+=1*$m[3][$i];
$mac[$m[1][$i]]['in']+=1*$m[3][$i];
}

$cfg=fopen('/etc/mrtg/tr.cfg', 'w');
fwrite($cfg, "EnableIPv6: no
WorkDir: /var/www/localhost/htdocs/mrtg
Options[_]: bits,growright
");

foreach ($ip as $k => $v){
$fn="/var/www/localhost/htdocs/mrtg/data/ip-$k";
$f=fopen($fn, "w");
$v['in']=round($v['in']);
$v['out']=round($v['out']);
fwrite($f, "{$v[in]}\n{$v[out]}\n");
fclose($f);
fwrite($cfg, "

Target[localhost_ip$k]: `cat $fn`
SetEnv[localhost_ip$k]: MRTG_INT_IP=\"$k\" MRTG_INT_DESCR=\"$k\"
MaxBytes[localhost_ip$k]: 1250000
Title[localhost_ip$k]: ip$k -- btcgate
PageTop[localhost_ip$k]: <h1>ip$k -- btcgate</h1>
<table>
<tr><td>System:</td> <td>btcgate in \"Cherni vrah 66, Sofia 1407\"</td></tr>
<tr><td>Maintainer:</td> <td>anton@titov.net</td></tr>
<tr><td>Description:</td><td>ip $k</td></tr>
<tr><td>ifType:</td> <td>ethernetCsmacd (6)</td></tr>
<tr><td>ifName:</td> <td></td></tr>
<tr><td>Max Speed:</td> <td>10.0 Mbits/s</td></tr>
</table>

");
}

foreach ($mac as $k => $v){
$fn="/var/www/localhost/htdocs/mrtg/data/mac-$k";
$f=fopen($fn, "w");
$v['in']=round($v['in']);
$v['out']=round($v['out']);
fwrite($f, "{$v[in]}\n{$v[out]}\n");
fclose($f);
fwrite($cfg, "

Target[localhost_mac$k]: `cat $fn`
SetEnv[localhost_mac$k]: MRTG_INT_IP=\"$k\" MRTG_INT_DESCR=\"mac $k\"
MaxBytes[localhost_mac$k]: 1250000
Title[localhost_mac$k]: mac$k -- btcgate
PageTop[localhost_mac$k]: <h1>mac$k -- btcgate</h1>
<table>
<tr><td>System:</td> <td>btcgate in \"Cherni vrah 66, Sofia 1407\"</td></tr>
<tr><td>Maintainer:</td> <td>anton@titov.net</td></tr>
<tr><td>Description:</td><td>mac $k</td></tr>
<tr><td>ifType:</td> <td>ethernetCsmacd (6)</td></tr>
<tr><td>ifName:</td> <td></td></tr>
<tr><td>Max Speed:</td> <td>10.0 Mbits/s</td></tr>
</table>

");
}

fclose($cfg);
`/usr/bin/mrtg /etc/mrtg/tr.cfg`;
`/usr/bin/indexmaker --output=/var/www/localhost/htdocs/mrtg/index.html --title="traffic report" --sort=name --enumerate /etc/mrtg/traffic.cfg /etc/mrtg/tr.cfg`;

execute this script once. You should see many MRTG errors. Execute it once more, you should see less errors. Once more and errors should vanish. Setup a cronjob to execute this script every 5 minutes. Point your browser to http://ip.of.the.bridge/mrtg and you should see the graphs.

“ip.of.the.bridge” is actually a tricky part, in my case I have third interface - eth2 which have local 10.x.x.x address and use it to view the traffic. You can also assign ip address to the bridge interface.

After few unsuccesfull google searches about changing linux software raid sync speed (while waiting for idle CentOS to sync 3×200Gb SATA raid1) I decided to run:


ffind /proc/ -name \*raid\*

which lead me to files /proc/sys/dev/raid/speed_limit_max an /proc/sys/dev/raid/speed_limit_min, which are quite self-explanatory and now I’m happily syncing with 40Mb/s.