Configuration file monitoring - reporting content changes

Configuration file monitoring - reporting content changes

[Burn Alting]

All,

I am interested in any Linux based capability that will monitor
identified files and report on actual changes to the monitored file. I
know there are methods of recording that the file has been changed (e.g.
aide and/or monitor writes via auditd), but I want to know what has
changed ... basically something that would provide a 'diff' like output.

Now there are tools like Samhain that will record the content changes of
a file that is <= 92000 bytes in size, but I am interested in a more
lightweight solution ... perhaps a simple inotify(7) based utility that
perhaps maintains a copy of the file(s) in core (in compressed format)
and based on inotify() returns checks for changes and reports (somehow
yet to be defined) the before/after changes.

Is there anything 'out there' that list members are aware of?

If not, would the following utility be of interest? On startup, load the
monitored file(s) (saving a compressed copy in memory). Then, using
inotify, monitor for changes and if so, emit some kind of record
defining the change and change the compressed in-memory copy. If so, is
our mailing list and the contributed portion of auditd an appropriate
repository for such a tool.

Naturally, such a tool would be supported by appropriate auditd
monitoring that will take care of changing file attributes etc and file
writes. That is, auditd tells me who and the utility tells me what.


Regards
Burn

Re: Configuration file monitoring - reporting content changes

[Burn Alting]

Mon, 2015-07-20 at 21:09 +0900, Tetsuo Handa wrote:
> Burn Alting wrote:
> > All,
> > 
> > I am interested in any Linux based capability that will monitor
> > identified files and report on actual changes to the monitored file. I
> > know there are methods of recording that the file has been changed (e.g.
> > aide and/or monitor writes via auditd), but I want to know what has
> > changed ... basically something that would provide a 'diff' like output.
> > 
> > Now there are tools like Samhain that will record the content changes of
> > a file that is <= 92000 bytes in size, but I am interested in a more
> > lightweight solution ... perhaps a simple inotify(7) based utility that
> > perhaps maintains a copy of the file(s) in core (in compressed format)
> > and based on inotify() returns checks for changes and reports (somehow
> > yet to be defined) the before/after changes.
> > 
> > Is there anything 'out there' that list members are aware of?
> 
> If you don't need to protect "copy of the file(s)" from malicious attacks,
> I think cvs / svn / git etc. would be sufficient.
> 
> If you need to protect "copy of the file(s)" from malicious attacks,
> you would need to use access control mechanisms (e.g. SELinux).
Tetsuo,

I appreciate what you say, but I am looking at this problem from the
point of view of
a. There is no central management of configuration files, which means
git/puppet is not a viable solution.
b. The entity that  will change the file is authorized to make the
change. That is, I just want to know what has changed. I am assuming
there are sufficient access control mechanisms to 'protect the file'.

Regards

Re: Configuration file monitoring - reporting content changes

[Smith, Gary R]

Hello Burn,

Have you considered iwatch (no, not the Apple wrist gadget). It monitors
files and can alert on a large set file conditions. Check out this man
page at: http://manpages.ubuntu.com/manpages/utopic/man1/iwatch.1.html

Best regards,

Gary Smith

On 7/20/15 4:56 AM, Burn Alting wrote:
> All,
>
> I am interested in any Linux based capability that will monitor
> identified files and report on actual changes to the monitored file. I
> know there are methods of recording that the file has been changed (e.g.
> aide and/or monitor writes via auditd), but I want to know what has
> changed ... basically something that would provide a 'diff' like output.
>
> Now there are tools like Samhain that will record the content changes of
> a file that is <= 92000 bytes in size, but I am interested in a more
> lightweight solution ... perhaps a simple inotify(7) based utility that
> perhaps maintains a copy of the file(s) in core (in compressed format)
> and based on inotify() returns checks for changes and reports (somehow
> yet to be defined) the before/after changes.
>
> Is there anything 'out there' that list members are aware of?
>
> If not, would the following utility be of interest? On startup, load the
> monitored file(s) (saving a compressed copy in memory). Then, using
> inotify, monitor for changes and if so, emit some kind of record
> defining the change and change the compressed in-memory copy. If so, is
> our mailing list and the contributed portion of auditd an appropriate
> repository for such a tool.
>
> Naturally, such a tool would be supported by appropriate auditd
> monitoring that will take care of changing file attributes etc and file
> writes. That is, auditd tells me who and the utility tells me what.
>
>
> Regards
> Burn
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>

Re: Configuration file monitoring - reporting content changes

[Burn Alting]

Gary,

Thanks. Although quite comprehensive about raising an alert when a large
variety action occur to a file, it still doesn't give me the core
requirement of reporting what content has changed. At best, one could
use the 'execute a command option' to, say do a diff on certain actions,
but you would also need to set up capability to replicate the file in
the first place, and, hope no one deletes the replicated file which
would defeat the purpose of the watch. Too many moving parts that can be
attacked.

To me, something like iwatch is a useful capability that can fire off an
alert as soon as the action occurs. At this point a Security Operations
Centre could take action. Although, I would expect them to wait for the
auditd events that not only record the action, but also the high level
command involved (PROCTITLE) and the user (auid) and what privileges
(*uid, *gid) they had.

Again, I am looking for a light weight, self contained tool.

Regards

On Mon, 2015-07-20 at 17:53 +0000, Smith, Gary R wrote:
> Hello Burn,
> 
> Have you considered iwatch (no, not the Apple wrist gadget). It monitors
> files and can alert on a large set file conditions. Check out this man
> page at: http://manpages.ubuntu.com/manpages/utopic/man1/iwatch.1.html
> 
> Best regards,
> 
> Gary Smith
> 
> On 7/20/15 4:56 AM, Burn Alting wrote:
> > All,
> >
> > I am interested in any Linux based capability that will monitor
> > identified files and report on actual changes to the monitored file. I
> > know there are methods of recording that the file has been changed (e.g.
> > aide and/or monitor writes via auditd), but I want to know what has
> > changed ... basically something that would provide a 'diff' like output.
> >
> > Now there are tools like Samhain that will record the content changes of
> > a file that is <= 92000 bytes in size, but I am interested in a more
> > lightweight solution ... perhaps a simple inotify(7) based utility that
> > perhaps maintains a copy of the file(s) in core (in compressed format)
> > and based on inotify() returns checks for changes and reports (somehow
> > yet to be defined) the before/after changes.
> >
> > Is there anything 'out there' that list members are aware of?
> >
> > If not, would the following utility be of interest? On startup, load the
> > monitored file(s) (saving a compressed copy in memory). Then, using
> > inotify, monitor for changes and if so, emit some kind of record
> > defining the change and change the compressed in-memory copy. If so, is
> > our mailing list and the contributed portion of auditd an appropriate
> > repository for such a tool.
> >
> > Naturally, such a tool would be supported by appropriate auditd
> > monitoring that will take care of changing file attributes etc and file
> > writes. That is, auditd tells me who and the utility tells me what.
> >
> >
> > Regards
> > Burn
> >
> > --
> > Linux-audit mailing list
> > Linux-audit@redhat.com
> > https://www.redhat.com/mailman/listinfo/linux-audit
> >
> 

Re: Configuration file monitoring - reporting content changes

[Steve Grubb]

On Monday, July 20, 2015 09:53:47 PM Burn Alting wrote:
> I am interested in any Linux based capability that will monitor
> identified files and report on actual changes to the monitored file.

I know of nothing that does this. But as long as the list of files is limited, 
it doesn't sound like a hard program to write.

> I know there are methods of recording that the file has been changed (e.g.
> aide and/or monitor writes via auditd), but I want to know what has
> changed ... basically something that would provide a 'diff' like output.
> 
> Now there are tools like Samhain that will record the content changes of
> a file that is <= 92000 bytes in size, but I am interested in a more
> lightweight solution ... perhaps a simple inotify(7) based utility that
> perhaps maintains a copy of the file(s) in core (in compressed format)
> and based on inotify() returns checks for changes and reports (somehow
> yet to be defined) the before/after changes.

It would have to be after the changes since inotify would tell you something 
happened.

> Is there anything 'out there' that list members are aware of?
> 
> If not, would the following utility be of interest?

I am certain there are people that are interested in this even if no one is 
speaking up on it.


> On startup, load the monitored file(s) (saving a compressed copy in memory).
> Then, using inotify, monitor for changes and if so, emit some kind of record
> defining the change and change the compressed in-memory copy. If so, is
> our mailing list and the contributed portion of auditd an appropriate
> repository for such a tool.

That's an interesting question. 

> Naturally, such a tool would be supported by appropriate auditd
> monitoring that will take care of changing file attributes etc and file
> writes. That is, auditd tells me who and the utility tells me what.

Correlating the changes might be interesting. There can be a long time between 
opening a file and closing it. The inotify might trigger on the changes during 
flushing to disk. Or what if the file was mmap'ed? I don't know if that would be 
caught. But there's only 1 way to find out. :-)

Like I said, I think its a straight forward program to write. No one's 
specifically asked for this. But we tap dance around the subject by patching 
programs to record what is being changed (shadow-utils). So, there is a 
precedence that this is needed. But Common Criteria makes it only for trusted 
databases. One file you would exempt, I presume, is /etc/shadow and 
/etc/gshadow.

Any one else with an opinion?

-Steve

RE: EXT :Re: Configuration file monitoring - reporting content changes

[Boyce, Kevin P (AS)]

Not to hijack your thread here, but as long as there might be development in the area of auditing changes to file with aide (sounds like maybe it would be an aide plugin), I would suggest it would also be very nice to know the names of files being copied/burned to removable media.  I don't know how one would accomplish this though.

Kevin

-----Original Message-----
From: linux-audit-bounces@redhat.com [mailto:linux-audit-bounces@redhat.com] On Behalf Of Steve Grubb
Sent: Monday, July 20, 2015 7:08 PM
To: linux-audit@redhat.com; burn@swtf.dyndns.org
Subject: EXT :Re: Configuration file monitoring - reporting content changes

On Monday, July 20, 2015 09:53:47 PM Burn Alting wrote:
> I am interested in any Linux based capability that will monitor 
> identified files and report on actual changes to the monitored file.

I know of nothing that does this. But as long as the list of files is limited, it doesn't sound like a hard program to write.

> I know there are methods of recording that the file has been changed (e.g.
> aide and/or monitor writes via auditd), but I want to know what has 
> changed ... basically something that would provide a 'diff' like output.
> 
> Now there are tools like Samhain that will record the content changes 
> of a file that is <= 92000 bytes in size, but I am interested in a 
> more lightweight solution ... perhaps a simple inotify(7) based 
> utility that perhaps maintains a copy of the file(s) in core (in 
> compressed format) and based on inotify() returns checks for changes 
> and reports (somehow yet to be defined) the before/after changes.

It would have to be after the changes since inotify would tell you something happened.

> Is there anything 'out there' that list members are aware of?
> 
> If not, would the following utility be of interest?

I am certain there are people that are interested in this even if no one is speaking up on it.


> On startup, load the monitored file(s) (saving a compressed copy in memory).
> Then, using inotify, monitor for changes and if so, emit some kind of record
> defining the change and change the compressed in-memory copy. If so, is
> our mailing list and the contributed portion of auditd an appropriate
> repository for such a tool.

That's an interesting question. 

> Naturally, such a tool would be supported by appropriate auditd
> monitoring that will take care of changing file attributes etc and file
> writes. That is, auditd tells me who and the utility tells me what.

Correlating the changes might be interesting. There can be a long time between 
opening a file and closing it. The inotify might trigger on the changes during 
flushing to disk. Or what if the file was mmap'ed? I don't know if that would be 
caught. But there's only 1 way to find out. :-)

Like I said, I think its a straight forward program to write. No one's 
specifically asked for this. But we tap dance around the subject by patching 
programs to record what is being changed (shadow-utils). So, there is a 
precedence that this is needed. But Common Criteria makes it only for trusted 
databases. One file you would exempt, I presume, is /etc/shadow and 
/etc/gshadow.

Any one else with an opinion?

-Steve

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

RE: EXT :Re: Configuration file monitoring - reporting content changes

[Burn Alting]

Kevin,

I believe there may be some work in this space occurring in the not too
distant future.

Regards

On Tue, 2015-07-21 at 13:47 +0000, Boyce, Kevin P (AS) wrote:
> Not to hijack your thread here, but as long as there might be development in the area of auditing changes to file with aide (sounds like maybe it would be an aide plugin), I would suggest it would also be very nice to know the names of files being copied/burned to removable media.  I don't know how one would accomplish this though.
> 
> Kevin
> 
> -----Original Message-----
> From: linux-audit-bounces@redhat.com [mailto:linux-audit-bounces@redhat.com] On Behalf Of Steve Grubb
> Sent: Monday, July 20, 2015 7:08 PM
> To: linux-audit@redhat.com; burn@swtf.dyndns.org
> Subject: EXT :Re: Configuration file monitoring - reporting content changes
> 
> On Monday, July 20, 2015 09:53:47 PM Burn Alting wrote:
> > I am interested in any Linux based capability that will monitor 
> > identified files and report on actual changes to the monitored file.
> 
> I know of nothing that does this. But as long as the list of files is limited, it doesn't sound like a hard program to write.
> 
> > I know there are methods of recording that the file has been changed (e.g.
> > aide and/or monitor writes via auditd), but I want to know what has 
> > changed ... basically something that would provide a 'diff' like output.
> > 
> > Now there are tools like Samhain that will record the content changes 
> > of a file that is <= 92000 bytes in size, but I am interested in a 
> > more lightweight solution ... perhaps a simple inotify(7) based 
> > utility that perhaps maintains a copy of the file(s) in core (in 
> > compressed format) and based on inotify() returns checks for changes 
> > and reports (somehow yet to be defined) the before/after changes.
> 
> It would have to be after the changes since inotify would tell you something happened.
> 
> > Is there anything 'out there' that list members are aware of?
> > 
> > If not, would the following utility be of interest?
> 
> I am certain there are people that are interested in this even if no one is speaking up on it.
> 
> 
> > On startup, load the monitored file(s) (saving a compressed copy in memory).
> > Then, using inotify, monitor for changes and if so, emit some kind of record
> > defining the change and change the compressed in-memory copy. If so, is
> > our mailing list and the contributed portion of auditd an appropriate
> > repository for such a tool.
> 
> That's an interesting question. 
> 
> > Naturally, such a tool would be supported by appropriate auditd
> > monitoring that will take care of changing file attributes etc and file
> > writes. That is, auditd tells me who and the utility tells me what.
> 
> Correlating the changes might be interesting. There can be a long time between 
> opening a file and closing it. The inotify might trigger on the changes during 
> flushing to disk. Or what if the file was mmap'ed? I don't know if that would be 
> caught. But there's only 1 way to find out. :-)
> 
> Like I said, I think its a straight forward program to write. No one's 
> specifically asked for this. But we tap dance around the subject by patching 
> programs to record what is being changed (shadow-utils). So, there is a 
> precedence that this is needed. But Common Criteria makes it only for trusted 
> databases. One file you would exempt, I presume, is /etc/shadow and 
> /etc/gshadow.
> 
> Any one else with an opinion?
> 
> -Steve
> 
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit

Re: Configuration file monitoring - reporting content changes

[John Dennis]

On 07/20/2015 07:08 PM, Steve Grubb wrote:
> On Monday, July 20, 2015 09:53:47 PM Burn Alting wrote:
>> I am interested in any Linux based capability that will monitor
>> identified files and report on actual changes to the monitored file.
>
> I know of nothing that does this. But as long as the list of files is limited,
> it doesn't sound like a hard program to write.

> Any one else with an opinion?

Yes :-) I'm not so sure it's an easy program to write and be robust in a 
variety of scenarios. I know because I wrote such a program once. The 
basic problem is most people think in terms of monitoring a file by name 
(e.g. it's pathname). But inotify operates on inodes, not filenames. If 
that file is subject to any variety of log rotation strategies or 
modifications by a configuration manager whereby the file is renamed or 
moved to a different directory then any program using inotify to monitor 
the file needs to become reasonably sophisticated and be able to track 
those changes. It is entirely possible for two processes to have opened 
the same file by name but have them be 2 different files (e.g. after 
opening the file path is modified but the process still has the original 
inode open, now a 2nd process opens the same filename but gets a 
different inode). Conflating inodes with filenames can lead to 
unexpected results and if the purpose is some sort of security 
monitoring it will be important these issues are accounted for.

Some of this is discussed in these documents which accompany the lwatch 
(Log Watch) program I wrote:

https://jdennis.fedorapeople.org/lwatch/html/InotifyOverview.html
https://jdennis.fedorapeople.org/lwatch/html/LogWatchOverview.html


-- 
John

Re: Configuration file monitoring - reporting content changes

[Steve Grubb]

On Tuesday, July 21, 2015 10:38:31 AM John Dennis wrote:
> On 07/20/2015 07:08 PM, Steve Grubb wrote:
> > On Monday, July 20, 2015 09:53:47 PM Burn Alting wrote:
> >> I am interested in any Linux based capability that will monitor
> >> identified files and report on actual changes to the monitored file.
> > 
> > I know of nothing that does this. But as long as the list of files is
> > limited, it doesn't sound like a hard program to write.
> > 
> > Any one else with an opinion?
> 
> Yes :-) I'm not so sure it's an easy program to write and be robust in a
> variety of scenarios. I know because I wrote such a program once. The
> basic problem is most people think in terms of monitoring a file by name
> (e.g. it's pathname). But inotify operates on inodes, not filenames. If
> that file is subject to any variety of log rotation strategies or
> modifications by a configuration manager whereby the file is renamed or
> moved to a different directory then any program using inotify to monitor
> the file needs to become reasonably sophisticated and be able to track
> those changes. It is entirely possible for two processes to have opened
> the same file by name but have them be 2 different files (e.g. after
> opening the file path is modified but the process still has the original
> inode open, now a 2nd process opens the same filename but gets a
> different inode). Conflating inodes with filenames can lead to
> unexpected results and if the purpose is some sort of security
> monitoring it will be important these issues are accounted for.

I recently was doing some experimenting with the fanotify API. In my mind, I 
think its likely to be better. But it has limitations such as mmap'ed file may 
not generate a modify event. So, if I were going to do it, I'd start there. 
But you do raise a whole lot of good points. My guess is this would watch 
config files which logrotate wouldn't apply. But yes, editors do open a temp 
copy and then do a rename. In my experimenting, I didn't bother to see how 
fanotify handle renames. (You would think its a modify event.)

-Steve


> Some of this is discussed in these documents which accompany the lwatch
> (Log Watch) program I wrote:
> 
> https://jdennis.fedorapeople.org/lwatch/html/InotifyOverview.html
> https://jdennis.fedorapeople.org/lwatch/html/LogWatchOverview.html

Re: Configuration file monitoring - reporting content changes

[Burn Alting]

On Tue, 2015-07-21 at 13:59 -0400, Steve Grubb wrote:
> On Tuesday, July 21, 2015 10:38:31 AM John Dennis wrote:
> > On 07/20/2015 07:08 PM, Steve Grubb wrote:
> > > On Monday, July 20, 2015 09:53:47 PM Burn Alting wrote:
> > >> I am interested in any Linux based capability that will monitor
> > >> identified files and report on actual changes to the monitored file.
> > > 
> > > I know of nothing that does this. But as long as the list of files is
> > > limited, it doesn't sound like a hard program to write.
> > > 
> > > Any one else with an opinion?
> > 
> > Yes :-) I'm not so sure it's an easy program to write and be robust in a
> > variety of scenarios. I know because I wrote such a program once. The
> > basic problem is most people think in terms of monitoring a file by name
> > (e.g. it's pathname). But inotify operates on inodes, not filenames. If
> > that file is subject to any variety of log rotation strategies or
> > modifications by a configuration manager whereby the file is renamed or
> > moved to a different directory then any program using inotify to monitor
> > the file needs to become reasonably sophisticated and be able to track
> > those changes. It is entirely possible for two processes to have opened
> > the same file by name but have them be 2 different files (e.g. after
> > opening the file path is modified but the process still has the original
> > inode open, now a 2nd process opens the same filename but gets a
> > different inode). Conflating inodes with filenames can lead to
> > unexpected results and if the purpose is some sort of security
> > monitoring it will be important these issues are accounted for.
> 
> I recently was doing some experimenting with the fanotify API. In my mind, I 
> think its likely to be better. But it has limitations such as mmap'ed file may 
> not generate a modify event. So, if I were going to do it, I'd start there. 
> But you do raise a whole lot of good points. My guess is this would watch 
> config files which logrotate wouldn't apply. But yes, editors do open a temp 
> copy and then do a rename. In my experimenting, I didn't bother to see how 
> fanotify handle renames. (You would think its a modify event.)
> 
> -Steve

I suppose one could follow tail(1)'s lead and monitor via a combination
of inotify() supplemented by file, path and inode monitoring as well.
Perhaps not elegant, but it may do the job.

Burn