Ulogd - mysql addresses are in network-byte order

Ulogd - mysql addresses are in network-byte order

[marty]

This is NOT a bug,  but I believe it needs consideration for change.
So lets call it a feature request to stay friendly.

ulogd.c:733 assigning `ip.saddr(?)' as source for MYSQL(ip.saddr)
ulogd.c:733 assigning `ip.daddr(?)' as source for MYSQL(ip.daddr)

On a little-endian architecture these values are incompatable with
the native math functions and totally unsuitable for making
comparisons in mysql.

eg:
if (( ip.saddr > nnnnnnnnn ) AND ( ip.saddr < mmmmmmmm)) ...
This simply will not work on a little endian machine.

It is impractical to do a byte order conversion using a bunch
of the high level routines within mysql, and it may not be
timely to do it later using a scripting language.

IMHO I believe it is appropriate for these values to be in
host-byte order before they are ever assigned to mysql.
This would then match the byte order of any machine.
If there are compelling reasons to use network byte order,
I suggest this be a configurable option, not the default.

Thanks for a great piece of software,

Marty B.

Re: Ulogd - mysql addresses are in network-byte order

[Pablo Neira Ayuso]

On Sat, Dec 31, 2011 at 11:36:36AM -0500, marty wrote:
> This is NOT a bug,  but I believe it needs consideration for change.
> So lets call it a feature request to stay friendly.
> 
> ulogd.c:733 assigning `ip.saddr(?)' as source for MYSQL(ip.saddr)
> ulogd.c:733 assigning `ip.daddr(?)' as source for MYSQL(ip.daddr)
> 
> On a little-endian architecture these values are incompatable with
> the native math functions and totally unsuitable for making
> comparisons in mysql.
> 
> eg:
> if (( ip.saddr > nnnnnnnnn ) AND ( ip.saddr < mmmmmmmm)) ...
> This simply will not work on a little endian machine.
> 
> It is impractical to do a byte order conversion using a bunch
> of the high level routines within mysql, and it may not be
> timely to do it later using a scripting language.
> 
> IMHO I believe it is appropriate for these values to be in
> host-byte order before they are ever assigned to mysql.
> This would then match the byte order of any machine.
> If there are compelling reasons to use network byte order,
> I suggest this be a configurable option, not the default.
> 
> Thanks for a great piece of software,

Thanks for the report.

Would you be brave enough to send me a patch to address this?

Re: Ulogd - mysql addresses are in network-byte order

[marty]

On 12/31/2011 12:27 PM, Pablo Neira Ayuso wrote:
> On Sat, Dec 31, 2011 at 11:36:36AM -0500, marty wrote:
>> This is NOT a bug,  but I believe it needs consideration for change.
>> So lets call it a feature request to stay friendly.
>>
>> ulogd.c:733 assigning `ip.saddr(?)' as source for MYSQL(ip.saddr)
>> ulogd.c:733 assigning `ip.daddr(?)' as source for MYSQL(ip.daddr)
>>
>> On a little-endian architecture these values are incompatable with
>> the native math functions and totally unsuitable for making
>> comparisons in mysql.
>>
>> eg:
>> if (( ip.saddr>  nnnnnnnnn ) AND ( ip.saddr<  mmmmmmmm)) ...
>> This simply will not work on a little endian machine.
>>
>> It is impractical to do a byte order conversion using a bunch
>> of the high level routines within mysql, and it may not be
>> timely to do it later using a scripting language.
>>
>> IMHO I believe it is appropriate for these values to be in
>> host-byte order before they are ever assigned to mysql.
>> This would then match the byte order of any machine.
>> If there are compelling reasons to use network byte order,
>> I suggest this be a configurable option, not the default.
>>
>> Thanks for a great piece of software,
>
> Thanks for the report.
>
> Would you be brave enough to send me a patch to address this?
>

Sure, but would you be brave enough to accept my patch?
Seriously, before I start, please get consensus on the issue of
configuration options for network byte order addresses.
That means I might need to work on more lib code, to be complete.
I wouldn't want to break anything people needed and if that
option is not necessary I can have a tested patch in a couple days.
Let me know what is best.

Thanks,

Marty B

Re: Ulogd - mysql addresses are in network-byte order

[Pablo Neira Ayuso]

On Sat, Dec 31, 2011 at 01:28:42PM -0500, marty wrote:
> On 12/31/2011 12:27 PM, Pablo Neira Ayuso wrote:
> >On Sat, Dec 31, 2011 at 11:36:36AM -0500, marty wrote:
> >>This is NOT a bug,  but I believe it needs consideration for change.
> >>So lets call it a feature request to stay friendly.
> >>
> >>ulogd.c:733 assigning `ip.saddr(?)' as source for MYSQL(ip.saddr)
> >>ulogd.c:733 assigning `ip.daddr(?)' as source for MYSQL(ip.daddr)
> >>
> >>On a little-endian architecture these values are incompatable with
> >>the native math functions and totally unsuitable for making
> >>comparisons in mysql.
> >>
> >>eg:
> >>if (( ip.saddr>  nnnnnnnnn ) AND ( ip.saddr<  mmmmmmmm)) ...
> >>This simply will not work on a little endian machine.
> >>
> >>It is impractical to do a byte order conversion using a bunch
> >>of the high level routines within mysql, and it may not be
> >>timely to do it later using a scripting language.
> >>
> >>IMHO I believe it is appropriate for these values to be in
> >>host-byte order before they are ever assigned to mysql.
> >>This would then match the byte order of any machine.
> >>If there are compelling reasons to use network byte order,
> >>I suggest this be a configurable option, not the default.
> >>
> >>Thanks for a great piece of software,
> >
> >Thanks for the report.
> >
> >Would you be brave enough to send me a patch to address this?
> >
> 
> Sure, but would you be brave enough to accept my patch?
> Seriously, before I start, please get consensus on the issue of
> configuration options for network byte order addresses.
> That means I might need to work on more lib code, to be complete.
> I wouldn't want to break anything people needed and if that
> option is not necessary I can have a tested patch in a couple days.
> Let me know what is best.

Never mind. Post the patches. We'll review them.

Re: Ulogd - mysql addresses are in network-byte order

[marty]

Here, for your review is a patch to
address the issue that I reported.

--- orig.ulogd_raw2packet_BASE.c	2011-12-08 11:55:09.000000000 -0500
+++ ulogd_raw2packet_BASE.c	2012-01-01 23:40:14.000000000 -0500
@@ -717,8 +717,8 @@
  		return ULOGD_IRET_OK;
  	len -= iph->ihl * 4;

-	okey_set_u32(&ret[KEY_IP_SADDR], iph->saddr);
-	okey_set_u32(&ret[KEY_IP_DADDR], iph->daddr);
+	okey_set_u32(&ret[KEY_IP_SADDR], ntohl(iph->saddr));
+	okey_set_u32(&ret[KEY_IP_DADDR], ntohl(iph->daddr));
  	okey_set_u8(&ret[KEY_IP_PROTOCOL], iph->protocol);
  	okey_set_u8(&ret[KEY_IP_TOS], iph->tos);
  	okey_set_u8(&ret[KEY_IP_TTL], iph->ttl);


Marty B.

Re: Ulogd - mysql addresses are in network-byte order

[Pablo Neira Ayuso]

On Mon, Jan 02, 2012 at 12:03:46AM -0500, marty wrote:
> 
> Here, for your review is a patch to
> address the issue that I reported.
> 
> --- orig.ulogd_raw2packet_BASE.c	2011-12-08 11:55:09.000000000 -0500
> +++ ulogd_raw2packet_BASE.c	2012-01-01 23:40:14.000000000 -0500
> @@ -717,8 +717,8 @@
>  		return ULOGD_IRET_OK;
>  	len -= iph->ihl * 4;
> 
> -	okey_set_u32(&ret[KEY_IP_SADDR], iph->saddr);
> -	okey_set_u32(&ret[KEY_IP_DADDR], iph->daddr);
> +	okey_set_u32(&ret[KEY_IP_SADDR], ntohl(iph->saddr));
> +	okey_set_u32(&ret[KEY_IP_DADDR], ntohl(iph->daddr));
>  	okey_set_u8(&ret[KEY_IP_PROTOCOL], iph->protocol);
>  	okey_set_u8(&ret[KEY_IP_TOS], iph->tos);
>  	okey_set_u8(&ret[KEY_IP_TTL], iph->ttl);

Many other plugins rely on the address in network byte order.

Can you fix this in the mysql plugin by adding some configurable
option?

more suggestions for your next patches, please:

* include short description before the patch.
* they have to apply with patch -p1 < file.patch

you can generate this with git diff HEAD (there are more advanced ways
to do this in git though).

thanks.

Re: Ulogd - mysql addresses are in network-byte order

[marty]

On 01/02/2012 01:08 PM, Pablo Neira Ayuso wrote:
> On Mon, Jan 02, 2012 at 12:03:46AM -0500, marty wrote:
>>
>> Here, for your review is a patch to
>> address the issue that I reported.
>>
>> --- orig.ulogd_raw2packet_BASE.c	2011-12-08 11:55:09.000000000 -0500
>> +++ ulogd_raw2packet_BASE.c	2012-01-01 23:40:14.000000000 -0500
>> @@ -717,8 +717,8 @@
>>   		return ULOGD_IRET_OK;
>>   	len -= iph->ihl * 4;
>>
>> -	okey_set_u32(&ret[KEY_IP_SADDR], iph->saddr);
>> -	okey_set_u32(&ret[KEY_IP_DADDR], iph->daddr);
>> +	okey_set_u32(&ret[KEY_IP_SADDR], ntohl(iph->saddr));
>> +	okey_set_u32(&ret[KEY_IP_DADDR], ntohl(iph->daddr));
>>   	okey_set_u8(&ret[KEY_IP_PROTOCOL], iph->protocol);
>>   	okey_set_u8(&ret[KEY_IP_TOS], iph->tos);
>>   	okey_set_u8(&ret[KEY_IP_TTL], iph->ttl);
>
> Many other plugins rely on the address in network byte order.
>
> Can you fix this in the mysql plugin by adding some configurable
> option?

Seems like I already offered config options previously, no...
And as I recall I gave fair warning, which you ignored.
Simply put the host arch should determine IP format. That is a given.
That is easily changed to network format where/when required.

Personally I would write the BASE code as I have,
and let the option for network byte order be available
as a config option, as I suggested from the start.

The mysql-plugin does not have the keys available directly.
It pretty much passes data blindly.
I am hesitant to change this because it works nice.

> more suggestions for your next patches, please:
>
> * include short description before the patch.
> * they have to apply with patch -p1<  file.patch

Agreed, I should submit patches with more info.

> you can generate this with git diff HEAD (there are more advanced ways
> to do this in git though)

Yes, I can do lots of things; even stand on my head.
But I fixed the issue as requested and if that is not sufficient I am 
terribly sorry, but I am a 60+ yr old man with a big schedule.
The code works fine for my purposes and you are not inspiring me.

Marty B.