Various questions

Various questions

[António Godinho]

Hya,

	I'm kinda new at iptables so I've been using scripts that I've
downloaded from the internet and latelly using shorewall.
	Trying to learn more and improve what I have I've several
questions about iptables. 
	I'm running a pentium 166 with 32M ram has my home gateway. Eth1
it's my dhcp connection to the cable modem/internet and eth0 my
connection to internal switch 10.10.10.0/24 with 4 computers inside.
	I've made a script to catch all rules that shorewall sends to
iptables and I'll start from that on. I'll post the rules at the end of
the email.
	
	I think that each redundante rule I add will decrease the
performance of my gateway, since I've an old computer and not enought
money to upgrade it...

	1. Like you can see on the list of rules below I've lots of
rules loging and after droping invalid ( internal IP's ) comming from
the internet:

iptables -A rfc1918 -s 172.16.0.0/12 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 172.16.0.0/12 -j logdrop
	
	Is it possible to DROP all those IP's in one rule? ( I don't
need to log them since they are invalid anyway )


	2. I've this in one of the rules:

iptables -A rfc1918 -m conntrack --ctorigdst 255.255.255.255 -j RETURN

	RETURN isn't a user defined chain, and I don't see any info
about this on the how-to's, can someone explain me what that does?


	3. This is defining priority ports? Like shaping the traffic?

iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --dport ssh -j TOS
--set-tos 16
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --dport ssh -j TOS
--set-tos 16
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --sport ssh -j TOS
--set-tos 16
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --sport ssh -j TOS
--set-tos 16

	3. Since all my traffic from intranet to gateway loc2fw chain
it's from trusted ip's ( it's from me or any member of my famely ),
that's one of the things I can remove, anyone has any other advice or
improvent to the listo f rules I've bellow?



	Sorry for this really long email, thanx,

		António Godinho


###		Rules
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
iptables -F
iptables -X
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -p udp --dport 53 -j ACCEPT
iptables -A INPUT -p ! icmp -m state --state INVALID -j DROP
iptables -A OUTPUT -p udp --dport 53 -j ACCEPT
iptables -A OUTPUT -p ! icmp -m state --state INVALID -j DROP
iptables -A FORWARD -p udp --dport 53 -j ACCEPT
iptables -A FORWARD -p ! icmp -m state --state INVALID -j DROP
iptables -N newnotsyn
iptables -A newnotsyn -j LOG --log-level info --log-prefix
FIREWALL:newnotsyn:DROP:
iptables -A newnotsyn -j DROP
iptables -N icmpdef
iptables -N common
iptables -N reject
iptables -N dynamic
iptables -N eth1_fwd
iptables -A eth1_fwd -j dynamic
iptables -N eth1_in
iptables -A eth1_in -j dynamic
iptables -N eth0_fwd
iptables -A eth0_fwd -j dynamic
iptables -N eth0_in
iptables -A eth0_in -j dynamic
iptables -A reject -p tcp -j REJECT --reject-with tcp-reset
iptables -A reject -p udp -j REJECT
iptables -A common -p icmp -j icmpdef
iptables -A common -p udp --dport 135 -j reject
iptables -A common -p udp --dport 137:139 -j reject
iptables -A common -p udp --dport 445 -j reject
iptables -A common -p tcp --dport 139 -j reject
iptables -A common -p tcp --dport 445 -j reject
iptables -A common -p tcp --dport 135 -j reject
iptables -A common -p udp --dport 1900 -j DROP
iptables -A common -d 255.255.255.255 -j DROP
iptables -A common -d 224.0.0.0/4 -j DROP
iptables -A common -p tcp --dport 113 -j reject
iptables -A common -p udp --sport 53 -mstate --state NEW -j DROP
iptables -A common -d 213.22.99.255 -j DROP
iptables -A common -d 10.10.10.255 -j DROP
iptables -A eth1_in -p udp --dport 67:68 -j ACCEPT
iptables -A OUTPUT -o eth1 -p udp --dport 67:68 -j ACCEPT
iptables -N rfc1918
iptables -N logdrop
iptables -A logdrop -j LOG --log-level info --log-prefix
FIREWALL:logdrop:DROP:
iptables -A logdrop -j DROP
iptables -A rfc1918 -s 255.255.255.255 -j RETURN
iptables -A rfc1918 -m conntrack --ctorigdst 255.255.255.255 -j RETURN
iptables -A rfc1918 -s 169.254.0.0/16 -j DROP
iptables -A rfc1918 -m conntrack --ctorigdst 169.254.0.0/16 -j DROP
iptables -A rfc1918 -s 172.16.0.0/12 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 172.16.0.0/12 -j logdrop
iptables -A rfc1918 -s 192.0.2.0/24 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 192.0.2.0/24 -j logdrop
iptables -A rfc1918 -s 192.168.0.0/16 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 192.168.0.0/16 -j logdrop
iptables -A rfc1918 -s 0.0.0.0/7 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 0.0.0.0/7 -j logdrop
iptables -A rfc1918 -s 2.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 2.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 5.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 5.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 7.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 7.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 10.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 10.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 23.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 23.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 27.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 27.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 31.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 31.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 36.0.0.0/7 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 36.0.0.0/7 -j logdrop
iptables -A rfc1918 -s 39.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 39.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 41.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 41.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 42.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 42.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 49.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 49.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 50.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 50.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 58.0.0.0/7 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 58.0.0.0/7 -j logdrop
iptables -A rfc1918 -s 60.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 60.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 70.0.0.0/7 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 70.0.0.0/7 -j logdrop
iptables -A rfc1918 -s 72.0.0.0/5 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 72.0.0.0/5 -j logdrop
iptables -A rfc1918 -s 83.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 83.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 84.0.0.0/6 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 84.0.0.0/6 -j logdrop
iptables -A rfc1918 -s 88.0.0.0/5 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 88.0.0.0/5 -j logdrop
iptables -A rfc1918 -s 96.0.0.0/3 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 96.0.0.0/3 -j logdrop
iptables -A rfc1918 -s 127.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 127.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 197.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 197.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 198.18.0.0/15 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 198.18.0.0/15 -j logdrop
iptables -A rfc1918 -s 201.0.0.0/8 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 201.0.0.0/8 -j logdrop
iptables -A rfc1918 -s 240.0.0.0/4 -j logdrop
iptables -A rfc1918 -m conntrack --ctorigdst 240.0.0.0/4 -j logdrop
iptables -A eth1_fwd -m state --state NEW -j rfc1918
iptables -A eth1_in -m state --state NEW -j rfc1918
iptables -N fw2net
iptables -A fw2net -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A fw2net -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A fw2net -p tcp -m state --state NEW --dport 53 -j ACCEPT
iptables -A fw2net -p udp -m state --state NEW --dport 53 -j ACCEPT
iptables -N loc2net
iptables -A loc2net -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A loc2net -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A loc2net -p udp -m state --state NEW --dport 53 -j ACCEPT
iptables -N loc2fw
iptables -A loc2fw -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A loc2fw -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A loc2fw -p tcp -m state --state NEW --dport 53 -j ACCEPT
iptables -A loc2fw -p udp -m state --state NEW --dport 53 -j ACCEPT
iptables -A loc2fw -p tcp -m state --state NEW --dport 22 -j ACCEPT
iptables -N net2fw
iptables -A net2fw -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A net2fw -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A net2fw -p tcp -m state --state NEW --dport 22 -j ACCEPT
iptables -A loc2fw -p icmp --icmp-type 8 -j ACCEPT
iptables -A net2fw -p icmp --icmp-type 8 -j ACCEPT
iptables -N fw2loc
iptables -A fw2loc -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A fw2loc -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A fw2loc -p icmp --icmp-type 8 -j ACCEPT
iptables -A fw2net -p icmp --icmp-type 8 -j ACCEPT
iptables -A loc2fw -p tcp -m state --state NEW --dport 80 -j ACCEPT
iptables -A net2fw -p tcp -m state --state NEW --dport 80 -j ACCEPT
iptables -A loc2fw -p tcp -m state --state NEW --dport 21 -j ACCEPT
iptables -A loc2fw -p tcp -m state --state NEW --dport 5900:5910 -j
ACCEPT
iptables -A net2fw -p tcp -m state --state NEW --dport 21 -j ACCEPT
iptables -N net2all
iptables -A net2all -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A net2all -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A net2all -j common
iptables -A net2all -j LOG --log-level info --log-prefix
FIREWALL:net2all:DROP:
iptables -A net2all -j DROP
iptables -N all2all
iptables -A all2all -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A all2all -m state --state NEW -p tcp ! --syn -j newnotsyn
iptables -A all2all -j common
iptables -A all2all -j LOG --log-level info --log-prefix
FIREWALL:all2all:REJECT:
iptables -A all2all -j reject
iptables -A fw2net -j ACCEPT
iptables -A fw2loc -j all2all
iptables -A net2fw -j net2all
iptables -A loc2fw -j all2all
iptables -A loc2net -j ACCEPT
iptables -t mangle -N pretos
iptables -t mangle -N outtos
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --dport ssh -j TOS
--set-tos 16
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --dport ssh -j TOS
--set-tos 16
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --sport ssh -j TOS
--set-tos 16
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --sport ssh -j TOS
--set-tos 16
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --dport ftp -j TOS
--set-tos 16
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --dport ftp -j TOS
--set-tos 16
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --sport ftp -j TOS
--set-tos 16
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --sport ftp -j TOS
--set-tos 16
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --sport ftp-data -j TOS
--set-tos 8
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --sport ftp-data -j TOS
--set-tos 8
iptables -t mangle -A outtos -p tcp -d 0.0.0.0/0 --dport ftp-data -j TOS
--set-tos 8
iptables -t mangle -A pretos -p tcp -d 0.0.0.0/0 --dport ftp-data -j TOS
--set-tos 8
iptables -t mangle -A PREROUTING -j pretos
iptables -t mangle -A OUTPUT -j outtos
iptables -A OUTPUT -o eth1 -d 0.0.0.0/0 -j fw2net
iptables -A eth1_in -s 0.0.0.0/0 -j net2fw
iptables -A eth1_fwd -o eth0 -d 0.0.0.0/0 -j net2all
iptables -A OUTPUT -o eth0 -d 0.0.0.0/0 -j fw2loc
iptables -A eth0_in -s 0.0.0.0/0 -j loc2fw
iptables -A eth0_fwd -o eth1 -d 0.0.0.0/0 -j loc2net
iptables -A FORWARD -i eth1 -j eth1_fwd
iptables -A INPUT -i eth1 -j eth1_in
iptables -A FORWARD -i eth0 -j eth0_fwd
iptables -A INPUT -i eth0 -j eth0_in
iptables -A INPUT -j common
iptables -A INPUT -j LOG --log-level info --log-prefix
FIREWALL:INPUT:REJECT:
iptables -A INPUT -j reject
iptables -A OUTPUT -j common
iptables -A OUTPUT -j LOG --log-level info --log-prefix
FIREWALL:OUTPUT:REJECT:
iptables -A OUTPUT -j reject
iptables -A FORWARD -j common
iptables -A FORWARD -j LOG --log-level info --log-prefix
FIREWALL:FORWARD:REJECT:
iptables -A FORWARD -j reject
iptables -D INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -D INPUT -p udp --dport 53 -j ACCEPT
iptables -D OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -D OUTPUT -p udp --dport 53 -j ACCEPT
iptables -D FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -D FORWARD -p udp --dport 53 -j ACCEPT

RE: Various questions

[Daniel Chemko]

	
>	Is it possible to DROP all those IP's in one rule? ( I don't
>need to log them since they are invalid anyway )


Ideally this is done for you with /proc/sys/net/ipv4/conf/*/rp_filter == 1 or 2. Basically it says that if a packet enters the interface with an ip address that is not on the subnet associated with it, then just drop the packet. That said, I don't see how these rules would seriously determent performance that much.

>	RETURN isn't a user defined chain, and I don't see any info
>about this on the how-to's, can someone explain me what that does?

RETURN means return to the chain that called it, so

iptables -A INPUT -j MYCHAIN 
iptables -A MYCHAIN -j RETURN (Returns back to the INPUT chain)


>	3. This is defining priority ports? Like shaping the traffic?

yes

>	3. Since all my traffic from intranet to gateway loc2fw chain
>it's from trusted ip's ( it's from me or any member of my famely ),
>that's one of the things I can remove, anyone has any other advice or
>improvent to the listo f rules I've bellow?

If you are memory bound, these will not help you much. If your memory is really the contention, then try lowering the timeouts for things like established or unreplied connections, etc.. these things are what takes up the memory.

RE: Various questions

[António Godinho]

>>	Is it possible to DROP all those IP's in one rule? ( I don't
>>need to log them since they are invalid anyway )
>
>
>Ideally this is done for you with /proc/sys/net/ipv4/conf/*/rp_filter
== 1 >or 2. Basically it says that if a packet enters the interface with
an ip >address that is not on the subnet associated with it, then just
drop the >packet. That said, I don't see how these rules would seriously
determent >performance that much.

Well I've made an script to enable this on a specific run level and I've
added this:
echo "2" > /proc/sys/net/ipv4/conf/all/rp_filter

It was on the packet filter how-to:)


>	3. This is defining priority ports? Like shaping the traffic?
>
>yes
>
>>	3. Since all my traffic from intranet to gateway loc2fw chain
>>it's from trusted ip's ( it's from me or any member of my famely ),
>>that's one of the things I can remove, anyone has any other advice or
>>improvent to the listo f rules I've bellow?
>
>If you are memory bound, these will not help you much. If your memory
is >really the contention, then try lowering the timeouts for things
like >established or unreplied connections, etc.. these things are what
takes up >the memory.

I've seen a comment on some forum that iptables "eats" lots of memory,
so on old computers the scripts should be optimized. Anyway scripts
should be optimized on any computer I guess. Thanx for your advice I'll
dig on that.


	António Godinho

RE: Various questions

[Daniel Chemko]

A lot of the memory 'eating' in Netfilter is from storing runtime information about active connections when you are NATing connections, which I imagine you are.

To reduce the memory glut in Netfilter, I would say use less connections :-) or else limit the time that connections stay in the system. Mind you, if you shrink them too much, you will get dropped connections during normal operations.

I would say that 32MB is very limited. You may want to scalp any and everything else on the machine. You can save up a decent chunk of mem by just shutting down crap. For a firewall machine, this is pretty obvious.


Maybe I am not the best to describe this. I have a memory leak somewhere in my setup that leads to a reboot every month, and I have 256 MB.

Re: Various questions

[Mogens Valentin]

Daniel Chemko wrote:
> 
> 
> >       Is it possible to DROP all those IP's in one rule? ( I don't
> >need to log them since they are invalid anyway )
> 
> Ideally this is done for you with /proc/sys/net/ipv4/conf/*/rp_filter == 1 or 2.

Do note that the rp_filter code will check only for zero (0) or
non-zero.
It changed sometime in the kernel 2.2 tree. Up until that point we could
use 1,2, or 3 to achieve ingress, egress or both. Nowadays it'll do
ingress and egress when using any non-zero value.

-- 
Kind regards / venlig hilsen,
Mogens Valentin, Mr Dev

IT Networking, Security, Server Setup
www.danbbs.dk/~monz   mrdev@danbbs.dk
Phone +45 32 525 878  Cell 51 227 668

Various Questions

[Carl Cook]

On Fri 07 January 2011 08:14:17 Hubert Kario wrote:
> I'd suggest at least 
> mkfs.btrfs -m raid1 -d raid0 /dev/sdc /dev/sdd
> if you really want raid0

I don't fully understand -m or -d.  Why would this make a truer raid0 that with no options?


Is it necessary to use fdisk on new drives in creating a BTRFS multi-drive array?  Or is this all that's needed:
# mkfs.btrfs /dev/sdb /dev/sdc
# btrfs filesystem show

Is this related to 'subvolumes'?  The FAQ implies that a subvolume is like a directory, but also like a partition.  What's the rationale for being able to create a subvolume under a subvolume, as Hubert says so he can "use the shadow_copy module for samba to publish the snapshots  to windows clients."  I don't have any windows clients, but what difference does his structure make?

I know that if using SATA+LVM, turn off the writeback cache on the drive, as it doesn't do cash flushing, and ensure NCQ is on.  But does this also apply to a BTRFS array?  If so, is this done in rc.local with 
hdparm -I /dev/sdb
hdparm -I /dev/sdc


How do you know what options to rsync are on by default?  I can't find this anywhere.  For example, it seems to me that --perms -ogE  --hard-links and --delete-excluded should be on by default, for a true sync?

If using the  --numeric-ids switch for rsync, do you just have to manually make sure the IDs and usernames are the same on source and destination machines?

For files that fail to transfer, wouldn't it be wise to use  --partial-dir=DIR to at least recover part of lost files?

The rsync man page says that rsync uses ssh by default, but is that the case?  I think -e may be related to engaging ssh, but don't understand the explanation.

So for my system where there is a backup server, I guess I run the rsync daemon on the backup server which presents a port, then when the other systems decide it's time for a backup (cron) they:
- stop mysql, dump the database somewhere, start mysql;
- connect to the backup server's rsync port and dump their data to (hopefully) some specific place there.
Right?

Re: Various Questions

[C Anthony Risinger]

On Fri, Jan 7, 2011 at 11:15 AM, Carl Cook <CACook@quantum-sci.com> wro=
te:
>
> On Fri 07 January 2011 08:14:17 Hubert Kario wrote:
>> I'd suggest at least
>> mkfs.btrfs -m raid1 -d raid0 /dev/sdc /dev/sdd
>> if you really want raid0
>
> I don't fully understand -m or -d. =A0Why would this make a truer rai=
d0 that with no options?

this will give you RAID0 for your data, but RAID1 for your metadata,
making it less likely that the FS itself gets corrupted, even though
you will lose some data in crash cases, if i understand correctly.

> Is it necessary to use fdisk on new drives in creating a BTRFS multi-=
drive array? =A0Or is this all that's needed:
> # mkfs.btrfs /dev/sdb /dev/sdc
> # btrfs filesystem show

depends on whether you need /boot partitions or other partitions.
what you have works fine though.

> Is this related to 'subvolumes'? =A0The FAQ implies that a subvolume =
is like a directory, but also like a partition. =A0What's the rationale=
 for being able to create a subvolume under a subvolume, as Hubert says=
 so he can "use the shadow_copy module for samba to publish the snapsho=
ts =A0to windows clients." =A0I don't have any windows clients, but wha=
t difference does his structure make?

just his preference to put it there... the snapshot of a snapshot can
go anywhere.  it doesn't have to reside under it's "parent", the
parent was just used as a base, it's not bound to it in any way AFAIK.

> How do you know what options to rsync are on by default? =A0I can't f=
ind this anywhere. =A0For example, it seems to me that --perms -ogE =A0=
--hard-links and --delete-excluded should be on by default, for a true =
sync?

the links and command Freddie Cash posted are a really good base to wor=
k from.

> So for my system where there is a backup server, I guess I run the rs=
ync daemon on the backup server which presents a port, then when the ot=
her systems decide it's time for a backup (cron) they:
> - stop mysql, dump the database somewhere, start mysql;
> - connect to the backup server's rsync port and dump their data to (h=
opefully) some specific place there.
> Right?

you don't have to stop mysql, you just need to "freeze" any new,
incoming writes, and flush (ie. let finish) whatever is happening
right now.  this ensures mysql is _internally_ consistent on the disk.

see comment by Lloyd Standish here:

http://dev.mysql.com/doc/refman/5.1/en/backup-methods.html

C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Various Questions

[Freddie Cash]

On Fri, Jan 7, 2011 at 9:15 AM, Carl Cook <CACook@quantum-sci.com> wrot=
e:
> How do you know what options to rsync are on by default? =C2=A0I can'=
t find this anywhere. =C2=A0For example, it seems to me that --perms -o=
gE =C2=A0--hard-links and --delete-excluded should be on by default, fo=
r a true sync?

Who cares which ones are on by default?  List the ones you want to use
on the command-line, everytime.  That way, if the defaults change,
your setup won't.

> If using the =C2=A0--numeric-ids switch for rsync, do you just have t=
o manually make sure the IDs and usernames are the same on source and d=
estination machines?

You use the --numeric-ids switch so that it *doesn't* matter if the
IDs/usernames are the same.  It just sends the ID number on the wire.
Sure, if you do an ls on the backup box, the username will appear to
be messed up.  But if you compare the user ID assigned to the file,
and the user ID to the backed up etc/passwd file, they are correct.
Then, if you ever need to restore the HTPC from backups, the
etc/passwd file is transferred over, the user IDs are transferred
over, and when you do an ls on the HTPC, everything matches up
correctly.

> For files that fail to transfer, wouldn't it be wise to use =C2=A0--p=
artial-dir=3DDIR to at least recover part of lost files?

Or, just run rsync again, if the connection is dropped.

> The rsync man page says that rsync uses ssh by default, but is that t=
he case? =C2=A0I think -e may be related to engaging ssh, but don't und=
erstand the explanation.

Does it matter what the default is, if you specify exactly how you
want it to work on the command-line?

> So for my system where there is a backup server, I guess I run the rs=
ync daemon on the backup server which presents a port, then when the ot=
her systems decide it's time for a backup (cron) they:
> - stop mysql, dump the database somewhere, start mysql;
> - connect to the backup server's rsync port and dump their data to (h=
opefully) some specific place there.
> Right?

That's one way (push backups).  It works ok for small numbers of
systems being backed up.  But get above a handful of machines, and it
gets very hard to time everything so that you don't hammer the disks
on the backup server.

Pull backups (backups server does everything) works better, in my
experience.  Then you just script things up once, run 1 script, worry
about 1 schedule, and everything is stored on the backups server.  No
need to run rsync daemons everywhere, just run the rsync client, using
-e ssh, and let it do everything.

If you need it to run a script on the remote machine first, that's
easy enough to do:
  - ssh to remote system, run script to stop DBs, dump DBs, snapshot
=46S, whatever
  - then run rsync
  - ssh to remote system run script to start DBs, delete snapshot, what=
ever

You're starting to over-think things.  Keep it simple, don't worry
about defaults, specify everything you want to do, and do it all from
the backups box.

--=20
=46reddie Cash
fjwcash@gmail.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Various Questions

[Carl Cook]

Wow, this rsync and backup system is pretty amazing.  I've always just tarred each directory manually, but now find I can RELIABLY automate backups, and have SOLID versioning to boot.  Thanks to everyone who advised, especially Freddie and Anthony.

I am still waiting for hardware for my backup server, but have been preparing.  On the backup server I'll be doing pull backups for everything except my phone (which is connected intermittently).  I'm going to set up a cron script on the backup server to pull backups once a week (as opposed to once/mo which I've done for 12 years).  I am at a loss how to to lock the database on the HTPC while exporting the dump, as per Lloyd Standish, but will study it.  (Freddie gave a nice script, but it doesn't seem to lock/flush first)  Also don't know how to email results/success/fail on completion, as I've not a very good coder.

But here is my proposed cron:
btrfs subvolume snapshot hex:///home /media/backups/snapshots/hex-{DATE}
rsync --archive --hard-links --delete-during --delete-excluded --inplace --numeric-ids -e ssh --exclude-from=/media/backups/exclude-hex hex:///home /media/backups/hex
btrfs subvolume snapshot droog:///home /media/backups/snapshots/droog-{DATE}
rsync --archive --hard-links --delete-during --delete-excluded --inplace --numeric-ids -e ssh --exclude-from=/media/backups/exclude-droog droog:///home /media/backups/droog

My root filesystems are ext4, so I guess they cannot be snapshotted before backup.  My home directories are/will be BTRFS though.


On Fri 07 January 2011 08:14:17 Hubert Kario wrote:
>> I'd suggest at least 
>> mkfs.btrfs -m raid1 -d raid0 /dev/sdc /dev/sdd
>> if you really want raid0
>
> I don't fully understand -m or -d.  Why would this make a truer raid0 that with no options?

I am beginning to suspect that this is the -default- behavior, as described in the wiki:
"# Create a filesystem across four drives (metadata mirrored, data striped)"

Should I turn off the writeback cache on each drive when running BTRFS?

Re: Various Questions

[Carl Cook]

In addition to the questions below, if anyone has a chance could you advise on why my destination drive has more data  than the source after this command:
# rsync --hard-links --delete --inplace --archive --numeric-ids /media/disk/* /home
sending incremental file list
sent 658660 bytes  received 2433 bytes  1322186.00 bytes/sec
total size is 1355368091626  speedup is 2050192.77

# df /media/disk
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md2             1868468340 1315408384 553059956  71% /media/disk
# df /home
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdb             3907029168 1325491836 2581537332  34% /home




On Fri 07 January 2011 10:55:43 Carl Cook wrote:
> 
> Wow, this rsync and backup system is pretty amazing.  I've always just tarred each directory manually, but now find I can RELIABLY automate backups, and have SOLID versioning to boot.  Thanks to everyone who advised, especially Freddie and Anthony.
> 
> I am still waiting for hardware for my backup server, but have been preparing.  On the backup server I'll be doing pull backups for everything except my phone (which is connected intermittently).  I'm going to set up a cron script on the backup server to pull backups once a week (as opposed to once/mo which I've done for 12 years).  I am at a loss how to to lock the database on the HTPC while exporting the dump, as per Lloyd Standish, but will study it.  (Freddie gave a nice script, but it doesn't seem to lock/flush first)  Also don't know how to email results/success/fail on completion, as I've not a very good coder.
> 
> But here is my proposed cron:
> btrfs subvolume snapshot hex:///home /media/backups/snapshots/hex-{DATE}
> rsync --archive --hard-links --delete-during --delete-excluded --inplace --numeric-ids -e ssh --exclude-from=/media/backups/exclude-hex hex:///home /media/backups/hex
> btrfs subvolume snapshot droog:///home /media/backups/snapshots/droog-{DATE}
> rsync --archive --hard-links --delete-during --delete-excluded --inplace --numeric-ids -e ssh --exclude-from=/media/backups/exclude-droog droog:///home /media/backups/droog
> 
> My root filesystems are ext4, so I guess they cannot be snapshotted before backup.  My home directories are/will be BTRFS though.
> 
> 
> On Fri 07 January 2011 08:14:17 Hubert Kario wrote:
> >> I'd suggest at least 
> >> mkfs.btrfs -m raid1 -d raid0 /dev/sdc /dev/sdd
> >> if you really want raid0
> >
> > I don't fully understand -m or -d.  Why would this make a truer raid0 that with no options?
> 
> I am beginning to suspect that this is the -default- behavior, as described in the wiki:
> "# Create a filesystem across four drives (metadata mirrored, data striped)"
> 
> Should I turn off the writeback cache on each drive when running BTRFS?
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Various Questions

[Ian! D. Allen]

On Sat, Jan 08, 2011 at 05:25:19AM -0800, Carl Cook wrote:
> In addition to the questions below, if anyone has a chance could you
> advise on why my destination drive has more data than the source after
> this command:
> # rsync --hard-links --delete --inplace --archive --numeric-ids /media/disk/* /home
> sending incremental file list
> sent 658660 bytes  received 2433 bytes  1322186.00 bytes/sec
> total size is 1355368091626  speedup is 2050192.77
> 
> # df /media/disk
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/md2             1868468340 1315408384 553059956  71% /media/disk
> # df /home
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdb             3907029168 1325491836 2581537332  34% /home

This has little to do with btrfs; it happens with many file systems due
to file system infrastructure details such as directory sizes, sparse
file handling, file fragmentation, etc.

For example: If you have a directory with a huge number of file names
in it, the actual directory disk space used will be large and will not
be reclaimed when you delete all the file names from the directory.
You would have to remove the directory itself and recreate it to reclaim
that space.  Also, using rsync without --sparse (which can't work with
--inplace), sparse files on the source may get expanded to take real
disk blocks on the destination.

Unless you use "dd" to copy a partition exactly, including all the file
system infrastructure details, any copy you make will be subject to the
vagaries of how the file system decides to lay out the data.

-- 
| Ian! D. Allen  -  idallen@idallen.ca  -  Ottawa, Ontario, Canada
| Home Page: http://idallen.com/   Contact Improv: http://contactimprov.ca/
| College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/
| Defend digital freedom:  http://eff.org/  and have fun:  http://fools.ca/

Re: Various Questions

[Freddie Cash]

On Sat, Jan 8, 2011 at 5:25 AM, Carl Cook <CACook@quantum-sci.com> wrot=
e:
>
> In addition to the questions below, if anyone has a chance could you =
advise on why my destination drive has more data =C2=A0than the source =
after this command:
> # rsync --hard-links --delete --inplace --archive --numeric-ids /medi=
a/disk/* /home
> sending incremental file list

What happens if you delete /home, then run the command again, but
without the *?  You generally don't use wildcards for the source or
destination when using rsync.  You just tell it which directory to
start in.

If you do an "ls /home" and "ls /media/disk" are they different?

--=20
=46reddie Cash
fjwcash@gmail.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Various Questions

[Carl Cook]

I'd rather not do the copy again unless necessary, as it took a day.

Directories look identical, but who knows?  I'm going to try and figure out how to do a file-by-file crc check, for peace of mind.


On Sat 08 January 2011 17:26:25 Freddie Cash wrote:
> On Sat, Jan 8, 2011 at 5:25 AM, Carl Cook <CACook@quantum-sci.com> wrote:
> >
> > In addition to the questions below, if anyone has a chance could you advise on why my destination drive has more data  than the source after this command:
> > # rsync --hard-links --delete --inplace --archive --numeric-ids /media/disk/* /home
> > sending incremental file list
> 
> What happens if you delete /home, then run the command again, but
> without the *?  You generally don't use wildcards for the source or
> destination when using rsync.  You just tell it which directory to
> start in.
> 
> If you do an "ls /home" and "ls /media/disk" are they different?
> 
> 

Re: Various Questions

[Fajar A. Nugraha]

On Sun, Jan 9, 2011 at 8:16 PM, Carl Cook <CACook@quantum-sci.com> wrot=
e:
>
> I'd rather not do the copy again unless necessary, as it took a day.
>
> Directories look identical, but who knows? =A0I'm going to try and fi=
gure out how to do a file-by-file crc check, for peace of mind.

try "du --apparent-size -slh"
It should rule out any differences caused by sparse files and hardlinks=
=2E

>
>
> On Sat 08 January 2011 17:26:25 Freddie Cash wrote:
>> On Sat, Jan 8, 2011 at 5:25 AM, Carl Cook <CACook@quantum-sci.com> w=
rote:
>> >
>> > In addition to the questions below, if anyone has a chance could y=
ou advise on why my destination drive has more data =A0than the source =
after this command:
>> > # rsync --hard-links --delete --inplace --archive --numeric-ids /m=
edia/disk/* /home

Are you SURE you don't get the command mixed up? The last argument to
rsync should be the destination. Your command looks like you're
copying things to /home.

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Various Questions

[Alan Chandler]

On 09/01/11 13:37, Fajar A. Nugraha wrote:
> On Sun, Jan 9, 2011 at 8:16 PM, Carl Cook<CACook@quantum-sci.com>  wrote:
>    
>> I'd rather not do the copy again unless necessary, as it took a day.
>>
>> Directories look identical, but who knows?  I'm going to try and figure out how to do a file-by-file crc check, for peace of mind.
>>      
> try "du --apparent-size -slh"
> It should rule out any differences caused by sparse files and hardlinks.
>
>    
>>
>> On Sat 08 January 2011 17:26:25 Freddie Cash wrote:
>>      
>>> On Sat, Jan 8, 2011 at 5:25 AM, Carl Cook<CACook@quantum-sci.com>  wrote:
>>>        
>>>> In addition to the questions below, if anyone has a chance could you advise on why my destination drive has more data  than the source after this command:
>>>> # rsync --hard-links --delete --inplace --archive --numeric-ids /media/disk/* /home
>>>>          
> Are you SURE you don't get the command mixed up? The last argument to
> rsync should be the destination. Your command looks like you're
> copying things to /home.
>    

What is also important is that use of * - it means all the . files at 
the top level are NOT being copied

rsync is clever enough to notice if you have the / at the end of the 
source to know whether you want the directory to be put into the 
destination or the contents of the directory.  The / at the end of the 
source means copy the contents.

This could be (I am not sure of the exact scope of --delete) the reason 
why the destination has more data than the source.  If --delete is not 
deleting /home/.* files (if there any there).

-- 
Alan Chandler
http://www.chandlerfamily.org.uk