[PATCH net v4 0/2] mptcp: pm: fix stale ADD_ADDR entry on id 0 removal

[PATCH net v4 0/2] mptcp: pm: fix stale ADD_ADDR entry on id 0 removal

[Kalpan Jani]

Hi Tao, Matt,

Thank you Tao for identifying the port mismatch in v3 and providing a
deterministic reproducer — both are incorporated in this series.

The problem with v3: mptcp_remove_anno_list_by_saddr() compares
addresses with use_port=true. Signal endpoints added without an
explicit port are stored in the anno_list with port 0, but msk_local
carries the connection's actual local port. So the v3 lookup always
missed the stale entry, making the patch a no-op. Confirmed with a
temporary printk showing msk_local.port=5000 and announced=0 on v3,
and announced=1 after clearing the port.

This series:

Patch 1 fixes mptcp_nl_remove_id_zero_address() to tear down the
pending ADD_ADDR symmetrically with the non-zero id path: copy
msk_local, clear the port, call mptcp_remove_anno_list_by_saddr(),
and decrement add_addr_signaled if announced.

Patch 2 adds the regression test based on Tao's reproducer. Verified:
  - unpatched kernel: WARNING: net/mptcp/pm fires, test reports FAIL
  - patched kernel:   no WARNING, test reports PASS

Tested with mptcp/mptcp-upstream-virtme-docker auto-normal. All
existing selftests pass (packetdrill_add_addr failure is pre-existing
and unrelated to this series).

Kalpan Jani (2):
  mptcp: pm: drop pending ADD_ADDR when removing id 0 endpoint
  selftests: mptcp: add regression test for stale ADD_ADDR on id 0
    removal

 net/mptcp/pm_kernel.c                         |  8 ++
 .../net/mptcp/mptcp_id0_stale_anno.sh         | 85 +++++++++++++++++++
 2 files changed, 93 insertions(+)
 create mode 100755 tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh

-- 
2.43.0

[PATCH net v4 1/2] mptcp: pm: drop pending ADD_ADDR when removing id 0 endpoint

[Kalpan Jani]

syzkaller hit the WARN_ON_ONCE() in mptcp_pm_alloc_anno_list() with
the in-kernel path manager.

When a signal endpoint is removed, the pending ADD_ADDR has to be
cancelled: its retransmit timer stopped and the anno_list entry
unlinked and freed. For a non-zero id endpoint this is done via
mptcp_nl_remove_subflow_and_signal_addr() ->
mptcp_pm_remove_anno_addr() -> mptcp_remove_anno_list_by_saddr().

The id 0 removal path, mptcp_nl_remove_id_zero_address(), does not do
this: it only queues a RM_ADDR and marks the id available again, but
leaves any pending anno_list entry and its armed retransmit timer
alive.

So when the id 0 endpoint is removed and re-added while its previously
sent ADD_ADDR is still awaiting the echo, the stale entry survives.
The kernel PM reselects id 0, reaches mptcp_pm_alloc_anno_list() a
second time, finds the stale entry and hits the WARN.

Make the id 0 removal path symmetric with the non-zero one: drop the
pending ADD_ADDR before queuing the RM_ADDR, and decrement
add_addr_signaled if the address had been announced. This closes the
race at its source, so the WARN_ON_ONCE() stays a valid assertion.

Signal endpoints added without an explicit port are stored in the
anno_list with port 0. mptcp_remove_anno_list_by_saddr() compares
addresses with use_port=true, and msk_local carries the connection's
actual local port, so passing msk_local directly misses the entry.
Clear the port before the lookup to match the stored key.

Fixes: 740d798e8767 ("mptcp: remove id 0 address")
Reported-by: syzbot+55c2a5c871441261ed14@syzkaller.appspotmail.com
Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/620
Suggested-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Kalpan Jani <kalpan.jani@mpiricsoftware.com>
---
 net/mptcp/pm_kernel.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/net/mptcp/pm_kernel.c b/net/mptcp/pm_kernel.c
index fc818b63752e..4ab1a339e3e9 100644
--- a/net/mptcp/pm_kernel.c
+++ b/net/mptcp/pm_kernel.c
@@ -1126,6 +1126,8 @@ static int mptcp_nl_remove_id_zero_address(struct net *net,
 	while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
 		struct sock *sk = (struct sock *)msk;
 		struct mptcp_addr_info msk_local;
+		struct mptcp_addr_info anno_addr;
+		bool announced;
 
 		if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
 			goto next;
@@ -1135,7 +1137,13 @@ static int mptcp_nl_remove_id_zero_address(struct net *net,
 			goto next;
 
 		lock_sock(sk);
+		/* Drop a possibly pending ADD_ADDR for this address. */
+		anno_addr = msk_local;
+		anno_addr.port = 0;
+		announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
 		spin_lock_bh(&msk->pm.lock);
+		if (announced)
+			msk->pm.add_addr_signaled--;
 		mptcp_pm_remove_addr(msk, &list);
 		mptcp_pm_rm_subflow(msk, &list);
 		__mark_subflow_endp_available(msk, 0);
-- 
2.43.0

[PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

[Kalpan Jani]

Add a kselftest that reproduces the bug fixed in the previous patch:
removing the id 0 endpoint while a pending ADD_ADDR echo is outstanding
left a stale anno_list entry alive, and a subsequent PM reselection
re-announced that address and tripped the WARN_ON_ONCE() in
mptcp_pm_alloc_anno_list().

The sequence is deterministic:
  1. Establish a fully-established MPTCP connection kept alive by a
     bidirectional /dev/zero stream.
  2. Signal 10.0.2.1 so the peer joins: the second subflow keeps the
     connection alive across the id 0 removal.
  3. Signal the MPC address 10.0.1.1: an anno_list entry is created.
  4. Delete id 0 (10.0.1.1): on an unfixed kernel the stale entry
     survives.
  5. Signal 10.0.3.1 to force a PM reselection: on an unfixed kernel
     this hits the stale entry and fires the WARN.

The test counts WARNING: net/mptcp/pm lines in dmesg before and after
the sequence and fails if new ones appear.

Co-developed-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Tested-by: Tao Cui <cuitao@kylinos.cn>
Signed-off-by: Kalpan Jani <kalpan.jani@mpiricsoftware.com>
---
 .../net/mptcp/mptcp_id0_stale_anno.sh         | 85 +++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100755 tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh

diff --git a/tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh b/tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh
new file mode 100755
index 000000000000..96c0d2256064
--- /dev/null
+++ b/tools/testing/selftests/net/mptcp/mptcp_id0_stale_anno.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Regression test for stale ADD_ADDR anno_list entry on id 0 removal
+
+. "$(dirname "${0}")/mptcp_lib.sh"
+
+ret=0
+ns1=""
+ns2=""
+err=$(mktemp)
+timeout_poll=30
+port=50000
+
+cleanup()
+{
+	rm -f "${err}"
+	mptcp_lib_ns_exit "${ns1}" "${ns2}"
+}
+
+mptcp_lib_check_mptcp
+mptcp_lib_check_tools ip
+
+trap cleanup EXIT
+
+mptcp_lib_ns_init ns1 ns2
+
+ip link add ns1eth1 netns "${ns1}" type veth peer name ns2eth1 netns "${ns2}"
+ip -net "${ns1}" link set lo up
+ip -net "${ns2}" link set lo up
+ip -net "${ns1}" link set ns1eth1 up
+ip -net "${ns2}" link set ns2eth1 up
+ip -net "${ns1}" addr add 10.0.1.1/24 dev ns1eth1
+ip -net "${ns1}" addr add 10.0.2.1/24 dev ns1eth1
+ip -net "${ns1}" addr add 10.0.3.1/24 dev ns1eth1
+ip -net "${ns2}" addr add 10.0.1.2/24 dev ns2eth1
+ip -net "${ns2}" addr add 10.0.2.2/24 dev ns2eth1
+ip -net "${ns2}" addr add 10.0.3.2/24 dev ns2eth1
+
+mptcp_lib_pm_nl_set_limits "${ns1}" 8 8
+mptcp_lib_pm_nl_set_limits "${ns2}" 8 8
+
+ip netns exec "${ns1}" ./mptcp_connect -t "${timeout_poll}" -l -p "${port}" \
+	0.0.0.0 < /dev/zero > /dev/null 2>"${err}" &
+spid=$!
+mptcp_lib_wait_local_port_listen "${ns1}" "${port}"
+ip netns exec "${ns2}" ./mptcp_connect -t "${timeout_poll}" -p "${port}" \
+	10.0.1.1 < /dev/zero > /dev/null 2>"${err}" &
+cpid=$!
+
+sleep 2
+
+warn_before=$(dmesg | grep -c "WARNING: net/mptcp/pm")
+
+# 1. signal 10.0.2.1: peer joins, second subflow keeps connection alive
+mptcp_lib_pm_nl_add_endpoint "${ns1}" 10.0.2.1 flags signal
+sleep 2
+
+# 2. signal MPC address 10.0.1.1: anno_list entry created for id 0
+mptcp_lib_pm_nl_add_endpoint "${ns1}" 10.0.1.1 flags signal
+sleep 1
+
+# 3. remove id 0: stale entry survives on unfixed kernels
+mptcp_lib_pm_nl_del_endpoint "${ns1}" 0 10.0.1.1
+sleep 1
+
+# 4. force PM reselection: hits stale entry on unfixed kernels
+mptcp_lib_pm_nl_add_endpoint "${ns1}" 10.0.3.1 flags signal
+sleep 2
+
+warn_after=$(dmesg | grep -c "WARNING: net/mptcp/pm")
+
+kill "${cpid}" "${spid}" 2>/dev/null
+wait "${cpid}" 2>/dev/null
+wait "${spid}" 2>/dev/null
+
+if [ "${warn_after}" -gt "${warn_before}" ]; then
+	mptcp_lib_result_fail "stale ADD_ADDR warning triggered on id 0 removal"
+	ret=1
+else
+	mptcp_lib_result_pass "no stale ADD_ADDR warning on id 0 removal"
+fi
+
+mptcp_lib_result_print_all_tap
+
+exit ${ret}
-- 
2.43.0

Re: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

[Matthieu Baerts]

Hi Kalpan,

On 22/06/2026 11:28, Kalpan Jani wrote:
> Add a kselftest that reproduces the bug fixed in the previous patch:
> removing the id 0 endpoint while a pending ADD_ADDR echo is outstanding
> left a stale anno_list entry alive, and a subsequent PM reselection
> re-announced that address and tripped the WARN_ON_ONCE() in
> mptcp_pm_alloc_anno_list().
> 
> The sequence is deterministic:
>   1. Establish a fully-established MPTCP connection kept alive by a
>      bidirectional /dev/zero stream.
>   2. Signal 10.0.2.1 so the peer joins: the second subflow keeps the
>      connection alive across the id 0 removal.
>   3. Signal the MPC address 10.0.1.1: an anno_list entry is created.
>   4. Delete id 0 (10.0.1.1): on an unfixed kernel the stale entry
>      survives.
>   5. Signal 10.0.3.1 to force a PM reselection: on an unfixed kernel
>      this hits the stale entry and fires the WARN.

Thank you for having added a reproducer, but in general:

- We try to avoid new "main" tests for maintainability reasons.

- If possible, please prefer packetdrill [1] for the reproducers: that's
easier to read (shorter, focussed on the essential to reproducer an
issue), faster to execute, and easier to maintain. Please try to use it,
and do not hesitate to ask for help if needed. Mastering packetdrill is
really helpful in this area of the kernel.

- If not, please check if adding a subtest to an existing selftest is
not possible, e.g. mptcp_join.sh

- We try to avoid using "sleep": it is either waiting for a too long
time, or not enough depending on the kernel config and test environment.
Prefer waiting for an action instead, e.g. checking if a connection has
been established, the number of subflows, etc.

[1] https://github.com/multipath-tcp/packetdrill/

> The test counts WARNING: net/mptcp/pm lines in dmesg before and after
> the sequence and fails if new ones appear.

That's not needed, the executor (i.e. CIs) will monitor that.

A few more requests about this series, please:

- base your patches on top of the export branch (or "for-review" or
"t/upstream" branches), but not the "export-net" one: the CI will always
try to apply them on top of the "export" branch, and that's easier for
me when I have to resolve conflicts. (Note that the "export" branch
contains both "net-next" and "net", plus all patches applied in our tree.)

- ideally, avoid renaming the patches, or at least the cover-letter,
between versions, so it is easier for reviewers and bots to follow. Here
for example, you switched to one patch to multiple ones: it would be
easier if the cover-letter has same title as your previous single patch.
If renaming is required, please add a note in the patch/cover-letter.
Note that b4 automatically adds a link to the previous version, that's
also helpful.

- use "mptcp-ne(x)t" prefixes instead of "ne(x)t" when you send patches
only to the MPTCP ML (which is what we prefer).

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

[kernel test robot]

Hi Kalpan,

kernel test robot noticed the following build errors:

[auto build test ERROR on net/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Kalpan-Jani/mptcp-pm-drop-pending-ADD_ADDR-when-removing-id-0-endpoint/20260623-033836
base:   net/main
patch link:    https://lore.kernel.org/r/20260622092838.1267134-3-kalpan.jani%40mpiricsoftware.com
patch subject: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal
config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260623/202606231019.rN367btY-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260623/202606231019.rN367btY-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202606231019.rN367btY-lkp@intel.com/

All errors (new ones prefixed by >>):

   net/mptcp/pm_kernel.c: In function 'mptcp_nl_remove_id_zero_address':
>> net/mptcp/pm_kernel.c:1154:29: error: implicit declaration of function 'mptcp_remove_anno_list_by_saddr' [-Wimplicit-function-declaration]
    1154 |                 announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
         |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


vim +/mptcp_remove_anno_list_by_saddr +1154 net/mptcp/pm_kernel.c

  1127	
  1128	static int mptcp_nl_remove_id_zero_address(struct net *net,
  1129						   struct mptcp_addr_info *addr)
  1130	{
  1131		struct mptcp_rm_list list = { .nr = 0 };
  1132		long s_slot = 0, s_num = 0;
  1133		struct mptcp_sock *msk;
  1134	
  1135		list.ids[list.nr++] = 0;
  1136	
  1137		while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
  1138			struct sock *sk = (struct sock *)msk;
  1139			struct mptcp_addr_info msk_local;
  1140			struct mptcp_addr_info anno_addr;
  1141			bool announced;
  1142	
  1143			if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
  1144				goto next;
  1145	
  1146			mptcp_local_address((struct sock_common *)msk, &msk_local);
  1147			if (!mptcp_addresses_equal(&msk_local, addr, addr->port))
  1148				goto next;
  1149	
  1150			lock_sock(sk);
  1151			/* Drop a possibly pending ADD_ADDR for this address. */
  1152			anno_addr = msk_local;
  1153			anno_addr.port = 0;
> 1154			announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
  1155			spin_lock_bh(&msk->pm.lock);
  1156			if (announced)
  1157				msk->pm.add_addr_signaled--;
  1158			mptcp_pm_remove_addr(msk, &list);
  1159			mptcp_pm_rm_subflow(msk, &list);
  1160			__mark_subflow_endp_available(msk, 0);
  1161			spin_unlock_bh(&msk->pm.lock);
  1162			release_sock(sk);
  1163	
  1164	next:
  1165			sock_put(sk);
  1166			cond_resched();
  1167		}
  1168	
  1169		return 0;
  1170	}
  1171	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal

[kernel test robot]

Hi Kalpan,

kernel test robot noticed the following build errors:

[auto build test ERROR on net/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Kalpan-Jani/mptcp-pm-drop-pending-ADD_ADDR-when-removing-id-0-endpoint/20260623-033836
base:   net/main
patch link:    https://lore.kernel.org/r/20260622092838.1267134-3-kalpan.jani%40mpiricsoftware.com
patch subject: [PATCH net v4 2/2] selftests: mptcp: add regression test for stale ADD_ADDR on id 0 removal
config: x86_64-kexec (https://download.01.org/0day-ci/archive/20260623/202606231240.AR0wvSEl-lkp@intel.com/config)
compiler: clang version 22.1.8 (https://github.com/llvm/llvm-project ca7933e47d3a3451d81e72ac174dcb5aa28b59d1)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260623/202606231240.AR0wvSEl-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202606231240.AR0wvSEl-lkp@intel.com/

All errors (new ones prefixed by >>):

>> net/mptcp/pm_kernel.c:1154:15: error: call to undeclared function 'mptcp_remove_anno_list_by_saddr'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1154 |                 announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
         |                             ^
   1 error generated.


vim +/mptcp_remove_anno_list_by_saddr +1154 net/mptcp/pm_kernel.c

  1127	
  1128	static int mptcp_nl_remove_id_zero_address(struct net *net,
  1129						   struct mptcp_addr_info *addr)
  1130	{
  1131		struct mptcp_rm_list list = { .nr = 0 };
  1132		long s_slot = 0, s_num = 0;
  1133		struct mptcp_sock *msk;
  1134	
  1135		list.ids[list.nr++] = 0;
  1136	
  1137		while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
  1138			struct sock *sk = (struct sock *)msk;
  1139			struct mptcp_addr_info msk_local;
  1140			struct mptcp_addr_info anno_addr;
  1141			bool announced;
  1142	
  1143			if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
  1144				goto next;
  1145	
  1146			mptcp_local_address((struct sock_common *)msk, &msk_local);
  1147			if (!mptcp_addresses_equal(&msk_local, addr, addr->port))
  1148				goto next;
  1149	
  1150			lock_sock(sk);
  1151			/* Drop a possibly pending ADD_ADDR for this address. */
  1152			anno_addr = msk_local;
  1153			anno_addr.port = 0;
> 1154			announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
  1155			spin_lock_bh(&msk->pm.lock);
  1156			if (announced)
  1157				msk->pm.add_addr_signaled--;
  1158			mptcp_pm_remove_addr(msk, &list);
  1159			mptcp_pm_rm_subflow(msk, &list);
  1160			__mark_subflow_endp_available(msk, 0);
  1161			spin_unlock_bh(&msk->pm.lock);
  1162			release_sock(sk);
  1163	
  1164	next:
  1165			sock_put(sk);
  1166			cond_resched();
  1167		}
  1168	
  1169		return 0;
  1170	}
  1171	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH net v4 0/2] mptcp: pm: fix stale ADD_ADDR entry on id 0 removal

[MPTCP CI]

Hi Kalpan,

Thank you for your modifications, that's great!

But sadly, our CI spotted some issues with it when trying to build it.

You can find more details there:

  https://github.com/multipath-tcp/mptcp_net-next/actions/runs/27944189360

Status: failure
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/279ca4d41984
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1114633

Feel free to reply to this email if you cannot access logs, if you need
some support to fix the error, if this doesn't seem to be caused by your
modifications or if the error is a false positive one.

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)

Re: [PATCH net v4 0/2] mptcp: pm: fix stale ADD_ADDR entry on id 0 removal

[MPTCP CI]

Hi Kalpan,

Thank you for your modifications, that's great!

Our CI did some validations and here is its report:

- KVM Validation: normal (except selftest_mptcp_join): Script error! ❓
- KVM Validation: normal (only selftest_mptcp_join): Script error! ❓
- KVM Validation: debug (except selftest_mptcp_join): Script error! ❓
- KVM Validation: debug (only selftest_mptcp_join): Script error! ❓
- KVM Validation: btf-normal (only bpftest_all): Script error! ❓
- KVM Validation: btf-debug (only bpftest_all): Script error! ❓
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/27944189431

Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/279ca4d41984
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1114633


If there are some issues, you can reproduce them using the same environment as
the one used by the CI thanks to a docker image, e.g.:

    $ cd [kernel source code]
    $ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
        --pull always mptcp/mptcp-upstream-virtme-docker:latest \
        auto-normal

For more details:

    https://github.com/multipath-tcp/mptcp-upstream-virtme-docker


Please note that despite all the efforts that have been already done to have a
stable tests suite when executed on a public CI like here, it is possible some
reported issues are not due to your modifications. Still, do not hesitate to
help us improve that ;-)

Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)

[PATCH net v4 1/2] mptcp: pm: drop pending ADD_ADDR when removing id 0 endpoint

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Hi Kalpan,

The CI build failure is just a leftover rename: patch 1 calls
mptcp_remove_anno_list_by_saddr(), which was renamed to
mptcp_pm_announced_remove() in 7d4dacc8ccca and no longer exists in the
current tree. One-liner:

  -	announced = mptcp_remove_anno_list_by_saddr(msk, &anno_addr);
  +	announced = mptcp_pm_announced_remove(msk, &anno_addr);

Cheers,
Tao