* [Qemu-devel] [PATCH V3 1/3] qemu-iotests: add unix socket help program
2013-09-02 9:18 [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for fd passing via SCM rights Wenchao Xia
@ 2013-09-02 9:18 ` Wenchao Xia
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 2/3] qemu-iotests: add infrastructure of fd passing via SCM Wenchao Xia
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Wenchao Xia @ 2013-09-02 9:18 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, armbru, lcapitulino, stefanha, Wenchao Xia
This program can do a sendmsg call to transfer fd with unix
socket, which is not supported in python2.
The built binary will not be deleted in clean, but it is a
existing issue in ./tests, which should be solved in another
patch.
Signed-off-by: Wenchao Xia <xiawenc@linux.vnet.ibm.com>
---
configure | 2 +-
tests/Makefile | 3 +-
tests/qemu-iotests/socket_scm_helper.c | 135 ++++++++++++++++++++++++++++++++
3 files changed, 138 insertions(+), 2 deletions(-)
create mode 100644 tests/qemu-iotests/socket_scm_helper.c
diff --git a/configure b/configure
index 0a55c20..5080c38 100755
--- a/configure
+++ b/configure
@@ -4544,7 +4544,7 @@ if [ "$dtc_internal" = "yes" ]; then
fi
# build tree in object directory in case the source is not in the current directory
-DIRS="tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/qapi-schema tests/tcg/xtensa"
+DIRS="tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/qapi-schema tests/tcg/xtensa tests/qemu-iotests"
DIRS="$DIRS pc-bios/optionrom pc-bios/spapr-rtas pc-bios/s390-ccw"
DIRS="$DIRS roms/seabios roms/vgabios"
DIRS="$DIRS qapi-generated"
diff --git a/tests/Makefile b/tests/Makefile
index baba9e9..c285805 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -172,6 +172,7 @@ tests/boot-order-test$(EXESUF): tests/boot-order-test.o $(libqos-obj-y)
tests/tmp105-test$(EXESUF): tests/tmp105-test.o $(libqos-omap-obj-y)
tests/i440fx-test$(EXESUF): tests/i440fx-test.o $(libqos-pc-obj-y)
tests/fw_cfg-test$(EXESUF): tests/fw_cfg-test.o $(libqos-pc-obj-y)
+tests/qemu-iotests/socket_scm_helper$(EXESUF): tests/qemu-iotests/socket_scm_helper.o
# QTest rules
@@ -250,7 +251,7 @@ check-report.html: check-report.xml
# Other tests
.PHONY: check-tests/qemu-iotests-quick.sh
-check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF)
+check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) tests/qemu-iotests/socket_scm_helper$(EXESUF)
$<
.PHONY: check-tests/test-qapi.py
diff --git a/tests/qemu-iotests/socket_scm_helper.c b/tests/qemu-iotests/socket_scm_helper.c
new file mode 100644
index 0000000..0e2b285
--- /dev/null
+++ b/tests/qemu-iotests/socket_scm_helper.c
@@ -0,0 +1,135 @@
+/*
+ * SCM_RIGHTS with unix socket help program for test
+ *
+ * Copyright IBM, Inc. 2013
+ *
+ * Authors:
+ * Wenchao Xia <xiawenc@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <sys/socket.h>
+#include <sys/un.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+
+/* #define SOCKET_SCM_DEBUG */
+
+/*
+ * @fd and @fd_to_send will not be checked for validation in this function,
+ * a blank will be sent as iov data to notify qemu.
+ */
+static int send_fd(int fd, int fd_to_send)
+{
+ struct msghdr msg;
+ struct iovec iov[1];
+ int ret;
+ char control[CMSG_SPACE(sizeof(int))];
+ struct cmsghdr *cmsg;
+
+ memset(&msg, 0, sizeof(msg));
+ memset(control, 0, sizeof(control));
+
+ /* Send a blank to notify qemu */
+ iov[0].iov_base = (void *)" ";
+ iov[0].iov_len = 1;
+
+ msg.msg_iov = iov;
+ msg.msg_iovlen = 1;
+
+ msg.msg_control = control;
+ msg.msg_controllen = sizeof(control);
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+
+ cmsg->cmsg_len = CMSG_LEN(sizeof(int));
+ cmsg->cmsg_level = SOL_SOCKET;
+ cmsg->cmsg_type = SCM_RIGHTS;
+ memcpy(CMSG_DATA(cmsg), &fd, sizeof(int));
+
+ do {
+ ret = sendmsg(fd, &msg, 0);
+ } while (ret < 0 && errno == EINTR);
+
+ if (ret < 0) {
+ fprintf(stderr, "Failed to send msg, reason: %s\n", strerror(errno));
+ }
+
+ return ret;
+}
+
+/* Convert string to fd number. */
+static int get_fd_num(const char *fd_str)
+{
+ int sock;
+ char *err;
+
+ errno = 0;
+ sock = strtol(fd_str, &err, 10);
+ if (errno) {
+ fprintf(stderr, "Failed in strtol for socket fd, reason: %s\n",
+ strerror(errno));
+ return -1;
+ }
+ if (!*fd_str || *err || sock < 0) {
+ fprintf(stderr, "bad numerical value for socket fd '%s'\n", fd_str);
+ return -1;
+ }
+
+ return sock;
+}
+
+/*
+ * To make things simple, the caller needs to specify:
+ * 1. socket fd.
+ * 2. path of the file to be sent.
+ */
+int main(int argc, char **argv, char **envp)
+{
+ int sock, fd, ret;
+
+#ifdef SOCKET_SCM_DEBUG
+ int i;
+ for (i = 0; i < argc; i++) {
+ fprintf(stderr, "Parameter %d: %s\n", i, argv[i]);
+ }
+#endif
+
+ if (argc != 3) {
+ fprintf(stderr,
+ "Usage: %s < socket-fd > < file-path >\n",
+ argv[0]);
+ return EXIT_FAILURE;
+ }
+
+
+ sock = get_fd_num(argv[1]);
+ if (sock < 0) {
+ return EXIT_FAILURE;
+ }
+
+ /* Now only open a file in readonly mode for test purpose. If more precise
+ control is needed, use python script in file operation, which is
+ supposed to fork and exec this program. */
+ fd = open(argv[2], O_RDONLY);
+ if (fd < 0) {
+ fprintf(stderr, "Failed to open file '%s'\n", argv[2]);
+ return EXIT_FAILURE;
+ }
+
+ ret = send_fd(sock, fd);
+ if (ret < 0) {
+ close(fd);
+ return EXIT_FAILURE;
+ }
+
+ close(fd);
+ return EXIT_SUCCESS;
+}
--
1.7.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* [Qemu-devel] [PATCH V3 2/3] qemu-iotests: add infrastructure of fd passing via SCM
2013-09-02 9:18 [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for fd passing via SCM rights Wenchao Xia
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 1/3] qemu-iotests: add unix socket help program Wenchao Xia
@ 2013-09-02 9:18 ` Wenchao Xia
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 3/3] qemu-iotests: add tests for runtime fd passing via SCM rights Wenchao Xia
2013-09-02 14:56 ` [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for " Stefan Hajnoczi
3 siblings, 0 replies; 7+ messages in thread
From: Wenchao Xia @ 2013-09-02 9:18 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, armbru, lcapitulino, stefanha, Wenchao Xia
This patch make use of the compiled scm helper program to transfer
fd via unix socket at runtime.
Signed-off-by: Wenchao Xia <xiawenc@linux.vnet.ibm.com>
---
QMP/qmp.py | 6 ++++++
tests/qemu-iotests/check | 1 +
tests/qemu-iotests/iotests.py | 23 +++++++++++++++++++++++
3 files changed, 30 insertions(+), 0 deletions(-)
diff --git a/QMP/qmp.py b/QMP/qmp.py
index c551df1..074f09a 100644
--- a/QMP/qmp.py
+++ b/QMP/qmp.py
@@ -188,3 +188,9 @@ class QEMUMonitorProtocol:
def settimeout(self, timeout):
self.__sock.settimeout(timeout)
+
+ def get_sock_fd(self):
+ return self.__sock.fileno()
+
+ def is_scm_available(self):
+ return self.__sock.family == socket.AF_UNIX
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 74628ae..2f7f78e 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -164,6 +164,7 @@ QEMU_IO -- $QEMU_IO
IMGFMT -- $FULL_IMGFMT_DETAILS
IMGPROTO -- $FULL_IMGPROTO_DETAILS
PLATFORM -- $FULL_HOST_DETAILS
+SOCKET_SCM_HELPER -- $SOCKET_SCM_HELPER
EOF
#MKFS_OPTIONS -- $FULL_MKFS_OPTIONS
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 33ad0ec..d79cc56 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -38,6 +38,8 @@ imgfmt = os.environ.get('IMGFMT', 'raw')
imgproto = os.environ.get('IMGPROTO', 'file')
test_dir = os.environ.get('TEST_DIR', '/var/tmp')
+socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
+
def qemu_img(*args):
'''Run qemu-img and return the exit code'''
devnull = open('/dev/null', 'r+')
@@ -80,6 +82,12 @@ class VM(object):
'-display', 'none', '-vga', 'none']
self._num_drives = 0
+ #This can be used to add unused monitor
+ def add_monitor_telnet(self, ip, port):
+ args = 'tcp:%s:%d,server,nowait,telnet' % (ip, port)
+ self._args.append('-monitor')
+ self._args.append(args)
+
def add_drive(self, path, opts=''):
'''Add a virtio-blk drive to the VM'''
options = ['if=virtio',
@@ -112,6 +120,21 @@ class VM(object):
self._args.append(','.join(options))
return self
+ def send_fd_scm(self, fd_file_path):
+ #In iotest.py, the qmp should always use unix socket.
+ assert self._qmp.is_scm_available()
+ bin = socket_scm_helper
+ if os.path.exists(bin) == False:
+ print "Scm help program does not present, path '%s'." % bin
+ return -1
+ fd_param = ["%s" % bin,
+ "%d" % self._qmp.get_sock_fd(),
+ "%s" % fd_file_path]
+ devnull = open('/dev/null', 'rb')
+ p = subprocess.Popen(fd_param, stdin=devnull, stdout=sys.stdout,
+ stderr=sys.stderr)
+ return p.wait()
+
def launch(self):
'''Launch the VM and establish a QMP connection'''
devnull = open('/dev/null', 'rb')
--
1.7.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* [Qemu-devel] [PATCH V3 3/3] qemu-iotests: add tests for runtime fd passing via SCM rights
2013-09-02 9:18 [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for fd passing via SCM rights Wenchao Xia
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 1/3] qemu-iotests: add unix socket help program Wenchao Xia
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 2/3] qemu-iotests: add infrastructure of fd passing via SCM Wenchao Xia
@ 2013-09-02 9:18 ` Wenchao Xia
2013-09-02 14:51 ` Stefan Hajnoczi
2013-09-02 14:56 ` [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for " Stefan Hajnoczi
3 siblings, 1 reply; 7+ messages in thread
From: Wenchao Xia @ 2013-09-02 9:18 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, armbru, lcapitulino, stefanha, Wenchao Xia
This case will test whether the monitor can receive fd at runtime.
To verify better, additional monitor is created to see if qemu
can handler two monitor instances correctly.
Signed-off-by: Wenchao Xia <xiawenc@linux.vnet.ibm.com>
---
tests/qemu-iotests/045 | 36 +++++++++++++++++++++++++++++++++++-
tests/qemu-iotests/045.out | 4 ++--
2 files changed, 37 insertions(+), 3 deletions(-)
diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index 2b6f1af..4381d8a 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -1,6 +1,6 @@
#!/usr/bin/env python
#
-# Tests for fdsets.
+# Tests for fdsets and getfd.
#
# Copyright (C) 2012 IBM Corp.
#
@@ -125,5 +125,39 @@ class TestFdSets(iotests.QMPTestCase):
'No file descriptor supplied via SCM_RIGHTS')
self.vm.shutdown()
+#Add fd at runtime, there are two ways: monitor related or fdset related
+class TestSCMFd(iotests.QMPTestCase):
+ def setUp(self):
+ self.vm = iotests.VM()
+ qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
+ #Add a unused monitor, to verify it works fine when two minitor present
+ self.vm.add_monitor_telnet("0",4445)
+ self.vm.launch()
+
+ def tearDown(self):
+ self.vm.shutdown()
+ os.remove(image0)
+
+ def _send_fd_by_SCM(self):
+ ret = self.vm.send_fd_scm(image0)
+ self.assertEqual(ret, 0, 'Failed to send fd with UNIX SCM')
+
+ def test_add_fd(self):
+ self._send_fd_by_SCM()
+ result = self.vm.qmp('add-fd', fdset_id=2, opaque='image0:r')
+ self.assert_qmp(result, 'return/fdset-id', 2)
+
+ def test_getfd(self):
+ self._send_fd_by_SCM()
+ result = self.vm.qmp('getfd', fdname='image0:r')
+ self.assert_qmp(result, 'return', {})
+
+ def test_closefd(self):
+ self._send_fd_by_SCM()
+ result = self.vm.qmp('getfd', fdname='image0:r')
+ self.assert_qmp(result, 'return', {})
+ result = self.vm.qmp('closefd', fdname='image0:r')
+ self.assert_qmp(result, 'return', {})
+
if __name__ == '__main__':
iotests.main(supported_fmts=['raw'])
diff --git a/tests/qemu-iotests/045.out b/tests/qemu-iotests/045.out
index 3f8a935..dae404e 100644
--- a/tests/qemu-iotests/045.out
+++ b/tests/qemu-iotests/045.out
@@ -1,5 +1,5 @@
-......
+.........
----------------------------------------------------------------------
-Ran 6 tests
+Ran 9 tests
OK
--
1.7.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [Qemu-devel] [PATCH V3 3/3] qemu-iotests: add tests for runtime fd passing via SCM rights
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 3/3] qemu-iotests: add tests for runtime fd passing via SCM rights Wenchao Xia
@ 2013-09-02 14:51 ` Stefan Hajnoczi
2013-09-03 2:09 ` Wenchao Xia
0 siblings, 1 reply; 7+ messages in thread
From: Stefan Hajnoczi @ 2013-09-02 14:51 UTC (permalink / raw)
To: Wenchao Xia; +Cc: kwolf, lcapitulino, qemu-devel, stefanha, armbru
On Mon, Sep 02, 2013 at 05:18:13PM +0800, Wenchao Xia wrote:
> @@ -125,5 +125,39 @@ class TestFdSets(iotests.QMPTestCase):
> 'No file descriptor supplied via SCM_RIGHTS')
> self.vm.shutdown()
>
> +#Add fd at runtime, there are two ways: monitor related or fdset related
> +class TestSCMFd(iotests.QMPTestCase):
> + def setUp(self):
> + self.vm = iotests.VM()
> + qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
> + #Add a unused monitor, to verify it works fine when two minitor present
> + self.vm.add_monitor_telnet("0",4445)
Sorry to be picky: please use the same whitespace style as the existing
code in these patches.
# Comments have a space after the hash
function(args, have, space)
> + self.vm.launch()
> +
> + def tearDown(self):
> + self.vm.shutdown()
> + os.remove(image0)
> +
> + def _send_fd_by_SCM(self):
> + ret = self.vm.send_fd_scm(image0)
> + self.assertEqual(ret, 0, 'Failed to send fd with UNIX SCM')
> +
> + def test_add_fd(self):
> + self._send_fd_by_SCM()
> + result = self.vm.qmp('add-fd', fdset_id=2, opaque='image0:r')
> + self.assert_qmp(result, 'return/fdset-id', 2)
> +
> + def test_getfd(self):
> + self._send_fd_by_SCM()
> + result = self.vm.qmp('getfd', fdname='image0:r')
> + self.assert_qmp(result, 'return', {})
> +
> + def test_closefd(self):
> + self._send_fd_by_SCM()
> + result = self.vm.qmp('getfd', fdname='image0:r')
> + self.assert_qmp(result, 'return', {})
> + result = self.vm.qmp('closefd', fdname='image0:r')
> + self.assert_qmp(result, 'return', {})
It would be good to also check the error cases like the existing tests
do (e.g. getfd fdname=asdf -> error).
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [Qemu-devel] [PATCH V3 3/3] qemu-iotests: add tests for runtime fd passing via SCM rights
2013-09-02 14:51 ` Stefan Hajnoczi
@ 2013-09-03 2:09 ` Wenchao Xia
0 siblings, 0 replies; 7+ messages in thread
From: Wenchao Xia @ 2013-09-03 2:09 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, lcapitulino, qemu-devel, stefanha, armbru
于 2013-9-2 22:51, Stefan Hajnoczi 写道:
> On Mon, Sep 02, 2013 at 05:18:13PM +0800, Wenchao Xia wrote:
>> @@ -125,5 +125,39 @@ class TestFdSets(iotests.QMPTestCase):
>> 'No file descriptor supplied via SCM_RIGHTS')
>> self.vm.shutdown()
>>
>> +#Add fd at runtime, there are two ways: monitor related or fdset related
>> +class TestSCMFd(iotests.QMPTestCase):
>> + def setUp(self):
>> + self.vm = iotests.VM()
>> + qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
>> + #Add a unused monitor, to verify it works fine when two minitor present
>> + self.vm.add_monitor_telnet("0",4445)
>
> Sorry to be picky: please use the same whitespace style as the existing
> code in these patches.
>
> # Comments have a space after the hash
> function(args, have, space)
>
OK, will check all the comments and fix.
>> + self.vm.launch()
>> +
>> + def tearDown(self):
>> + self.vm.shutdown()
>> + os.remove(image0)
>> +
>> + def _send_fd_by_SCM(self):
>> + ret = self.vm.send_fd_scm(image0)
>> + self.assertEqual(ret, 0, 'Failed to send fd with UNIX SCM')
>> +
>> + def test_add_fd(self):
>> + self._send_fd_by_SCM()
>> + result = self.vm.qmp('add-fd', fdset_id=2, opaque='image0:r')
>> + self.assert_qmp(result, 'return/fdset-id', 2)
>> +
>> + def test_getfd(self):
>> + self._send_fd_by_SCM()
>> + result = self.vm.qmp('getfd', fdname='image0:r')
>> + self.assert_qmp(result, 'return', {})
>> +
>> + def test_closefd(self):
>> + self._send_fd_by_SCM()
>> + result = self.vm.qmp('getfd', fdname='image0:r')
>> + self.assert_qmp(result, 'return', {})
>> + result = self.vm.qmp('closefd', fdname='image0:r')
>> + self.assert_qmp(result, 'return', {})
>
> It would be good to also check the error cases like the existing tests
> do (e.g. getfd fdname=asdf -> error).
>
OK. I'll wait a few days and respin with fix.
--
Best Regards
Wenchao Xia
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for fd passing via SCM rights
2013-09-02 9:18 [Qemu-devel] [PATCH V3 0/3] qemu-iotests: add test for fd passing via SCM rights Wenchao Xia
` (2 preceding siblings ...)
2013-09-02 9:18 ` [Qemu-devel] [PATCH V3 3/3] qemu-iotests: add tests for runtime fd passing via SCM rights Wenchao Xia
@ 2013-09-02 14:56 ` Stefan Hajnoczi
3 siblings, 0 replies; 7+ messages in thread
From: Stefan Hajnoczi @ 2013-09-02 14:56 UTC (permalink / raw)
To: Wenchao Xia; +Cc: kwolf, lcapitulino, qemu-devel, stefanha, armbru
On Mon, Sep 02, 2013 at 05:18:10PM +0800, Wenchao Xia wrote:
> This series add test case for fd passing with unix socket at runtime. Since
> getfd and closefd interface will interact with monitor's data, so it will
> help to do regression test for monitor patches. Since python2 do not support
> sendmsg(), so a C helper program is added to do the job.
>
> v2:
> 1: add missing $ in the makefile rule.
>
> v3:
> Address Eric's comments:
> 1: typo fix, remove "." in the end of error message, strick
> check argc as "!=", use EXIT_SUCCESS and EXIT_FAILURE as exit
> values, strict error check for strtol() call.
> Address Luiz's comments:
> 1: change the helper program parameter as "bin < socket-fd > < file-path >",
> the program open the file itself now, data parameter is removed and blank
> is always used as iov data, better usage tip message, folder the string parsing
> code into a function.
> 2: related change for helper program parameter change.
> 3: related change for helper program parameter change.
> Other:
> 1: remove "LINK" rule in makefile, remove fd checking code inside send_fd()
> since it is already checked before calling, add '' around %s for path and
> number string in error message.
> 2: renamed fd_bin to bin in send_fd_scm() to tip better, add '' around %s
> for path in error message.
>
> Wenchao Xia (3):
> 1 qemu-iotests: add unix socket help program
> 2 qemu-iotests: add infrastructure of fd passing via SCM
> 3 qemu-iotests: add tests for runtime fd passing via SCM rights
>
> QMP/qmp.py | 6 ++
> configure | 2 +-
> tests/Makefile | 3 +-
> tests/qemu-iotests/045 | 36 ++++++++-
> tests/qemu-iotests/045.out | 4 +-
> tests/qemu-iotests/check | 1 +
> tests/qemu-iotests/iotests.py | 23 ++++++
> tests/qemu-iotests/socket_scm_helper.c | 135 ++++++++++++++++++++++++++++++++
> 8 files changed, 205 insertions(+), 5 deletions(-)
> create mode 100644 tests/qemu-iotests/socket_scm_helper.c
Looks useful. I have left comments about whitespace cleanups and
requested a few additional tests for addfd/getfd/closefd errors.
^ permalink raw reply [flat|nested] 7+ messages in thread