Issue while testing new fvaddr feature

Issue while testing new fvaddr feature

[Vinicius Petrucci]

Hello,

I am experimenting with the new fixed address (fvaddr) feature of
DAMON to monitor memory regions originated from user-level mmap/malloc
calls.
I used "strace" to intercept the mmap calls and derive "start" and
"end" attributes for monitoring using the "masim" program
(https://github.com/sjp38/masim) as an example that creates two memory
allocations and keeps accessing them for a long time.
Then, I used "perf record" to get the data, but perf doesn't seem to
be recording anything. Nothing is shown after issuing "perf script".

Could you please help me spot any mistakes with my tests (if any)
using the Bash script below? Could also please let me know whether or
not that is reproducible in your system?

I am using kernel 6.0.2 and I checked that "fvaddr" is supported via
"damo features".
All steps in the script below ("test-fvaddr-damon.sh") runs without
any errors. It is just that nothing appears in the perf data.

-------- test-fvaddr-damon.sh --------------

strace -e mmap ./masim/masim hint_test_inf 2> strace.out &


sleep 2


echo 1 > /sys/kernel/mm/damon/admin/kdamonds/nr_kdamonds
echo 1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/nr_contexts

echo fvaddr > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/operations

echo 1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/nr_targets

echo $(pidof masim) >
/sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/pid_target

echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state



start1=`cat strace.out | tail -2 | head -1 | cut -f2 -d=`

start2=`cat strace.out | tail -2 | tail -1 | cut -f2 -d=`



# convert to decimal

start1=$((${start1}))

start2=$((${start2}))



size1=`cat strace.out | tail -2 | cut -d, -f2 | head -1`

size2=`cat strace.out | tail -2 | cut -d, -f2 | tail -1`



end1=$(($start1+$size1-1))

end2=$(($start1+$size1-1))



echo $start1 $end1

echo $start2 $end2



echo 2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/nr_regions



echo $start1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/0/start

echo $end1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/0/end



echo $start2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/1/start

echo $end2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/1/end



perf record -e damon:damon_aggregated &



sleep 15



kill $(pidof perf)



echo off > /sys/kernel/mm/damon/admin/kdamonds/0/state



perf script



killall strace

----------- hint_test_inf -------------

#regions

# name, length

a, 1073741824 # 1 GiB

b, 1073741824 # 1 GiB



# phase 1

# name of phase

phase 1

# time in ms

9999999999999

# access patterns

# name of region, randomness, stride, probability

a, 0, 64, 99

b, 0, 64, 1



# phase 2

phase 2

# time in ms

10000

# access patterns

# name of region, randomness, stride, probability

a, 0, 64, 1

b, 0, 64, 99


Thank you!

Re: Issue while testing new fvaddr feature

[SeongJae Park]

Hi Vinicius,

On Tue, 18 Oct 2022 21:49:22 -0500 Vinicius Petrucci <vpetrucci@gmail.com> wrote:

> Hello,
> 
> I am experimenting with the new fixed address (fvaddr) feature of
> DAMON to monitor memory regions originated from user-level mmap/malloc
> calls.
> I used "strace" to intercept the mmap calls and derive "start" and
> "end" attributes for monitoring using the "masim" program
> (https://github.com/sjp38/masim) as an example that creates two memory
> allocations and keeps accessing them for a long time.
> Then, I used "perf record" to get the data, but perf doesn't seem to
> be recording anything. Nothing is shown after issuing "perf script".
> 
> Could you please help me spot any mistakes with my tests (if any)
> using the Bash script below? Could also please let me know whether or
> not that is reproducible in your system?
> 
> I am using kernel 6.0.2 and I checked that "fvaddr" is supported via
> "damo features".
> All steps in the script below ("test-fvaddr-damon.sh") runs without
> any errors. It is just that nothing appears in the perf data.

Thank you for reaching out to us with this kind question!

> 
> -------- test-fvaddr-damon.sh --------------
> 
> strace -e mmap ./masim/masim hint_test_inf 2> strace.out &
> 
> 
> sleep 2
> 
> 
> echo 1 > /sys/kernel/mm/damon/admin/kdamonds/nr_kdamonds
> echo 1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/nr_contexts
> 
> echo fvaddr > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/operations
> 
> echo 1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/nr_targets
> 
> echo $(pidof masim) >
> /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/pid_target
> 
> echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state
> 
> 
> 
> start1=`cat strace.out | tail -2 | head -1 | cut -f2 -d=`
> 
> start2=`cat strace.out | tail -2 | tail -1 | cut -f2 -d=`
> 
> 
> 
> # convert to decimal
> 
> start1=$((${start1}))
> 
> start2=$((${start2}))
> 
> 
> 
> size1=`cat strace.out | tail -2 | cut -d, -f2 | head -1`
> 
> size2=`cat strace.out | tail -2 | cut -d, -f2 | tail -1`
> 
> 
> 
> end1=$(($start1+$size1-1))
> 
> end2=$(($start1+$size1-1))
> 
> 
> 
> echo $start1 $end1
> 
> echo $start2 $end2
> 
> 
> 
> echo 2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/nr_regions
> 
> 
> 
> echo $start1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/0/start
> 
> echo $end1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/0/end
> 
> 
> 
> echo $start2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/1/start
> 
> echo $end2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/1/end
> 
> 
> 
> perf record -e damon:damon_aggregated &

kdamond doesn't read the values of the sysfs files after turned on by default.
You can force it to read the values again by writing a special keyword,
`commit`, to the `state` file of the kdamond[1].

Therefore, in this case, DAMON would run with no region, and hence it gets
nothing to report.  I think that should be why you show nothing from the `perf
script` command.

So, could you please set the start/end values before turning the kdamond on, or
write 'commit' to the 'state' file here, and check if it works?

If there is anything I missed or wrong, please let me know.


[1] https://www.kernel.org/doc/html/v6.0/admin-guide/mm/damon/usage.html#kdamonds-n


Thanks,
SJ

> 
> 
> 
> sleep 15
> 
> 
> 
> kill $(pidof perf)
> 
> 
> 
> echo off > /sys/kernel/mm/damon/admin/kdamonds/0/state
> 
> 
> 
> perf script
> 
> 
> 
> killall strace
> 
> ----------- hint_test_inf -------------
> 
> #regions
> 
> # name, length
> 
> a, 1073741824 # 1 GiB
> 
> b, 1073741824 # 1 GiB
> 
> 
> 
> # phase 1
> 
> # name of phase
> 
> phase 1
> 
> # time in ms
> 
> 9999999999999
> 
> # access patterns
> 
> # name of region, randomness, stride, probability
> 
> a, 0, 64, 99
> 
> b, 0, 64, 1
> 
> 
> 
> # phase 2
> 
> phase 2
> 
> # time in ms
> 
> 10000
> 
> # access patterns
> 
> # name of region, randomness, stride, probability
> 
> a, 0, 64, 1
> 
> b, 0, 64, 99
> 
> 
> Thank you!

Re: Issue while testing new fvaddr feature

[Vinicius Petrucci]

Hi SeongJae!

I appreciate your feedback on this. Your explanation makes sense and I
didn't notice the "commit" feature. Sorry!

Nevertheless, I am still not able to get any perf data, and now I am
getting some errors. Not sure what I am doing wrong now. I tried both
approaches.

1) I tried doing “on” only after setting the start/end address, and got:

# echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state

-bash: echo: write error: Invalid argument

2) When doing “on” just after setting pid_target, then setting
start/end addr regions (as originally), then added a new echo “commit”
to the state variable, the kdamond automatically goes to “off” state
and cannot be set to "on" anymore. As follows:

# echo commit > /sys/kernel/mm/damon/admin/kdamonds/0/state

# cat /sys/kernel/mm/damon/admin/kdamonds/0/state

off

# echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state

-bash: echo: write error: Invalid argument

Please let me know if you have any further directions on how to better
understand/debug this.

THanks again!

Re: Issue while testing new fvaddr feature

[SeongJae Park]

Hi Vinicius,

On Wed, 19 Oct 2022 18:58:15 -0500 Vinicius Petrucci <vpetrucci@gmail.com> wrote:

> Hi SeongJae!
> 
> I appreciate your feedback on this. Your explanation makes sense and I
> didn't notice the "commit" feature. Sorry!

No problem, I'm glad to be able to help you.

BTW, we usually reply to emails in interleaved style[1].  It would be better if
you could also use the style from next time.

[1] https://en.wikipedia.org/wiki/Posting_style

> 
> Nevertheless, I am still not able to get any perf data, and now I am
> getting some errors. Not sure what I am doing wrong now. I tried both
> approaches.
> 
> 1) I tried doing “on” only after setting the start/end address, and got:
> 
> # echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state
> 
> -bash: echo: write error: Invalid argument

I modified your script in the way and I was able to reproduce your issue.  I
also got below great debugging-important information thanks to your script:

    $ sudo bash test.sh
    140437117468672 140438191214591
    140436041629696 140438191214591
    test.sh: line 40: echo: write error: Invalid argument
    $

So, that means the script was trying to 140437117468672-140438191214591 of the
masim's virtual address space range as the first monitoring region, and
140436041629696-140438191214591 of the range as the second monitoring region.

As you can show, the two ranges are overlapping.  DAMON sysfs interface assumes
the regions will not overlap, and in order.  That is, your script enters
invalid regions, and therefore DAMON returns the error.  I admit that the
documentation is not kindly explanining this, sorry.  I will try to elaborate
the document later.

To show if this is the case, I slightly modified your test script to monitor
only the first region as below.

    diff --git a/test.sh b/test.sh
    index c4e0145feebe..b2869eefc25e 100644
    --- a/test.sh
    +++ b/test.sh
    @@ -18,25 +18,18 @@ start2=`cat strace.out | tail -2 | tail -1 | cut -f2 -d=`
    
     # convert to decimal
     start1=$((${start1}))
    -start2=$((${start2}))
    
     size1=`cat strace.out | tail -2 | cut -d, -f2 | head -1`
    -size2=`cat strace.out | tail -2 | cut -d, -f2 | tail -1`
    
     end1=$(($start1+$size1-1))
    -end2=$(($start1+$size1-1))
    
     echo $start1 $end1
    -echo $start2 $end2
    
    -echo 2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/nr_regions
    +echo 1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/nr_regions
    
     echo $start1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/0/start
     echo $end1 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/0/end
    
    -echo $start2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/1/start
    -echo $end2 > /sys/kernel/mm/damon/admin/kdamonds/0/contexts/0/targets/0/regions/1/end
    -
     echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state
    
     perf record -e damon:damon_aggregated &


After this change, 'perf script' shows some data.

    $ sudo perf script | head -n 15
           kdamond.0 99584 [008] 30042.929871: damon:damon_aggregated: target_id=0 nr_regions=4 139805163782144-139805292630016: 1 0
           kdamond.0 99584 [008] 30042.929876: damon:damon_aggregated: target_id=0 nr_regions=4 139805292630016-139805378531328: 1 0
           kdamond.0 99584 [008] 30042.929877: damon:damon_aggregated: target_id=0 nr_regions=4 139805378531328-139805722128384: 1 0
           kdamond.0 99584 [008] 30042.929877: damon:damon_aggregated: target_id=0 nr_regions=4 139805722128384-139806237528064: 1 0
           kdamond.0 99584 [008] 30043.036517: damon:damon_aggregated: target_id=0 nr_regions=7 139805163782144-139805202436096: 0 0
           kdamond.0 99584 [008] 30043.036520: damon:damon_aggregated: target_id=0 nr_regions=7 139805202436096-139805301219328: 0 0
           kdamond.0 99584 [008] 30043.036521: damon:damon_aggregated: target_id=0 nr_regions=7 139805301219328-139805378531328: 0 0
           kdamond.0 99584 [008] 30043.036522: damon:damon_aggregated: target_id=0 nr_regions=7 139805378531328-139805447249920: 0 0
           kdamond.0 99584 [008] 30043.036522: damon:damon_aggregated: target_id=0 nr_regions=7 139805447249920-139805722128384: 0 0
           kdamond.0 99584 [008] 30043.036523: damon:damon_aggregated: target_id=0 nr_regions=7 139805722128384-139806082908160: 1 1
           kdamond.0 99584 [008] 30043.036524: damon:damon_aggregated: target_id=0 nr_regions=7 139806082908160-139806237528064: 0 0
           kdamond.0 99584 [008] 30043.138314: damon:damon_aggregated: target_id=0 nr_regions=11 139805163782144-139805202436096: 1 0
           kdamond.0 99584 [008] 30043.138317: damon:damon_aggregated: target_id=0 nr_regions=11 139805202436096-139805281460224: 1 0
           kdamond.0 99584 [008] 30043.138318: damon:damon_aggregated: target_id=0 nr_regions=11 139805281460224-139805301219328: 0 1
           kdamond.0 99584 [008] 30043.138319: damon:damon_aggregated: target_id=0 nr_regions=11 139805301219328-139805370798080: 1 0

> 
> 2) When doing “on” just after setting pid_target, then setting
> start/end addr regions (as originally), then added a new echo “commit”
> to the state variable, the kdamond automatically goes to “off” state
> and cannot be set to "on" anymore. As follows:
> 
> # echo commit > /sys/kernel/mm/damon/admin/kdamonds/0/state
> 
> # cat /sys/kernel/mm/damon/admin/kdamonds/0/state
> 
> off
> 
> # echo on > /sys/kernel/mm/damon/admin/kdamonds/0/state
> 
> -bash: echo: write error: Invalid argument

When you write 'commit' to 'state' file, DAMON reads the region directories and
find that the input is invalid, so refuse to continue running.

DAMON does not reset the file inputs, though.  So when you write 'on' to
'state' file again, DAMON again reads the invalid region inputs and the
situation becomes same to the scenario 1 (set region first and write 'on').

> 
> Please let me know if you have any further directions on how to better
> understand/debug this.

Hope my answer explains the situation.  If there is anything I missed or wrong,
or if you have any more question, please feel free to let me know.

Again, thank you for your question.


Thanks,
SJ

> 
> THanks again!

Re: Issue while testing new fvaddr feature

[Vinicius Petrucci]

Hi,

> So, that means the script was trying to 140437117468672-140438191214591 of the
> masim's virtual address space range as the first monitoring region, and
> 140436041629696-140438191214591 of the range as the second monitoring region.
>

Oh. Jeez. Now I see a dumb oversight:

end2=$(($start1+$size1-1))

should be:

end2=$(($start2+$size2-1))

I should have tried one region at a time and debugged each region more
thoroughly... my bad!

> As you can show, the two ranges are overlapping.  DAMON sysfs interface assumes
> the regions will not overlap, and in order.  That is, your script enters
> invalid regions, and therefore DAMON returns the error.

With your kind help and explanation, everything makes total sense now.

Thanks!