From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from picard.linux.it (picard.linux.it [213.254.12.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C533DC4332F for ; Sun, 16 Oct 2022 12:59:20 +0000 (UTC) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id E40743CAFC3 for ; Sun, 16 Oct 2022 14:59:18 +0200 (CEST) Received: from in-3.smtp.seeweb.it (in-3.smtp.seeweb.it [217.194.8.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by picard.linux.it (Postfix) with ESMTPS id 3D66D3CAFCE for ; Sun, 16 Oct 2022 14:58:45 +0200 (CEST) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by in-3.smtp.seeweb.it (Postfix) with ESMTPS id 2EB541A005E9 for ; Sun, 16 Oct 2022 14:58:43 +0200 (CEST) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29G9CmS3035709 for ; Sun, 16 Oct 2022 12:58:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=p4BOMfp9fDG2SmH9S72lP5BjA1P54MZ0/79pFvaJuz4=; b=m5vWgsq4Jd1Csu5SA9uncdhgX1/eDNcXGAav77O4DKbf/PQDVRTlo5KKLpGqutT9EbYN fZQDWLO6nfl/cvAUpE8JX+SD1ifbwY76emYzBT3sjZbQ6LvdpQjcEu3PLS5FVeZIsdmp DV50Xk2ScELQ3IEwBRQeXOnX06JEsQtnKXBcdkOxTHXerxaS6LhfLLQhb7uwPx5W5nIl xdAV6Tiyj7KUvVNZ9nQJh2/luY0oPy78MtL3s2BFHKTheKRHxcZvsj5X9RPUqdH5Ta3I tnDQoTKYOoUctSNkBFuNLANq0EC4An9jYhnBlJ2oR7tU7bU4WKOwmozTrrzdREucGL7E vQ== Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k86ufmcnv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sun, 16 Oct 2022 12:58:41 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29GCpB2P010001 for ; Sun, 16 Oct 2022 12:58:38 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma04ams.nl.ibm.com with ESMTP id 3k7mg91kk2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sun, 16 Oct 2022 12:58:38 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29GCwZTS48955710 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 16 Oct 2022 12:58:35 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CED3211C04C; Sun, 16 Oct 2022 12:58:35 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BD34811C04A; Sun, 16 Oct 2022 12:58:32 +0000 (GMT) Received: from tarunpc.ibmuc.com (unknown [9.43.63.52]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Sun, 16 Oct 2022 12:58:32 +0000 (GMT) From: Tarun Sahu To: ltp@lists.linux.it Date: Sun, 16 Oct 2022 18:27:04 +0530 Message-Id: <20221016125731.249078-3-tsahu@linux.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221016125731.249078-1-tsahu@linux.ibm.com> References: <20221016125731.249078-1-tsahu@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: D-CJ5Qtts6_bvR3nRm0jUaJu1PH0_t3y X-Proofpoint-ORIG-GUID: D-CJ5Qtts6_bvR3nRm0jUaJu1PH0_t3y X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-16_09,2022-10-14_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 phishscore=0 bulkscore=0 lowpriorityscore=0 impostorscore=0 suspectscore=0 spamscore=0 malwarescore=0 priorityscore=1501 adultscore=0 mlxscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210160077 X-Virus-Scanned: clamav-milter 0.102.4 at in-3.smtp.seeweb.it X-Virus-Status: Clean Subject: [LTP] [PATCH 02/29] Hugetlb: Migrating libhugetlbfs chunk-overcommit X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.ibm.com, sbhat@linux.ibm.com, vaibhav@linux.ibm.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ltp-bounces+ltp=archiver.kernel.org@lists.linux.it Sender: "ltp" Migrating the libhugetlbfs/testcases/chunk-overcommit.c test Test Description: Some kernel versions after hugepage demand allocation was added used a dubious heuristic to check if there was enough hugepage space available for a given mapping. The number of not-already-instantiated pages in the mapping was compared against the total hugepage free pool. It was very easy to confuse this heuristic into overcommitting by allocating hugepage memory in chunks, each less than the total available pool size but together more than available. This would generally lead to OOM SIGKILLs of one process or another when it tried to instantiate pages beyond the available pool. Signed-off-by: Tarun Sahu --- runtest/hugetlb | 1 + testcases/kernel/mem/.gitignore | 1 + .../kernel/mem/hugetlb/hugemmap/hugemmap08.c | 173 ++++++++++++++++++ 3 files changed, 175 insertions(+) create mode 100644 testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c diff --git a/runtest/hugetlb b/runtest/hugetlb index f7ff81cb3..664f18827 100644 --- a/runtest/hugetlb +++ b/runtest/hugetlb @@ -4,6 +4,7 @@ hugemmap04 hugemmap04 hugemmap05 hugemmap05 hugemmap06 hugemmap06 hugemmap07 hugemmap07 +hugemmap08 hugemmap08 hugemmap05_1 hugemmap05 -m hugemmap05_2 hugemmap05 -s hugemmap05_3 hugemmap05 -s -m diff --git a/testcases/kernel/mem/.gitignore b/testcases/kernel/mem/.gitignore index df5256ec8..003ce422b 100644 --- a/testcases/kernel/mem/.gitignore +++ b/testcases/kernel/mem/.gitignore @@ -5,6 +5,7 @@ /hugetlb/hugemmap/hugemmap05 /hugetlb/hugemmap/hugemmap06 /hugetlb/hugemmap/hugemmap07 +/hugetlb/hugemmap/hugemmap08 /hugetlb/hugeshmat/hugeshmat01 /hugetlb/hugeshmat/hugeshmat02 /hugetlb/hugeshmat/hugeshmat03 diff --git a/testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c b/testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c new file mode 100644 index 000000000..63a731e09 --- /dev/null +++ b/testcases/kernel/mem/hugetlb/hugemmap/hugemmap08.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: LGPL-2.1-or-later +/* + * Copyright (C) 2005-2006 David Gibson & Adam Litke, IBM Corporation. + * + * Test Name: Chunk Overcommit + * + * Test Description: Some kernel versions after hugepage demand allocation was + * added used a dubious heuristic to check if there was enough hugepage space + * available for a given mapping. The number of not-already-instantiated pages + * in the mapping was compared against the total hugepage free pool. It was + * very easy to confuse this heuristic into overcommitting by allocating + * hugepage memory in chunks, each less than the total available pool size but + * together more than available. This would generally lead to OOM SIGKILLs of + * one process or another when it tried to instantiate pages beyond the + * available pool. + * + * HISTORY + * Written by David Gibson & Adam Litke + * + */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include + +#include "hugetlb.h" + +#define PROC_OVERCOMMIT "/proc/sys/vm/nr_overcommit_hugepages" +#define WITH_OVERCOMMIT 0 +#define WITHOUT_OVERCOMMIT 1 + +static char *verbose; +static char hfile[MAXPATHLEN]; +static int fd = -1; +static long hpage_size; + +static void test_chunk_overcommit(void) +{ + unsigned long totpages, chunk1, chunk2; + void *p, *q; + pid_t child; + int status; + + totpages = SAFE_READ_MEMINFO("HugePages_Free:"); + + fd = SAFE_OPEN(hfile, O_RDWR | O_CREAT, 0600); + SAFE_UNLINK(hfile); + + chunk1 = (totpages / 2) + 1; + chunk2 = totpages - chunk1 + 1; + + if (verbose) + tst_res(TINFO, "overcommit: %ld hugepages available: " + "chunk1=%ld chunk2=%ld", totpages, chunk1, chunk2); + + p = SAFE_MMAP(NULL, chunk1*hpage_size, PROT_READ|PROT_WRITE, MAP_SHARED, + fd, 0); + + /* Can't use SAFE_MMAP here, as test needs to process the output of mmap */ + q = mmap(NULL, chunk2*hpage_size, PROT_READ|PROT_WRITE, MAP_SHARED, + fd, chunk1*hpage_size); + if (q == MAP_FAILED) { + if (errno != ENOMEM) { + tst_res(TFAIL | TERRNO, "mmap() chunk2"); + goto fail; + } else { + tst_res(TPASS, "Successful without overcommit pages"); + goto pass; + } + } + + if (verbose) + tst_res(TINFO, "Looks like we've overcommitted, testing..."); + /* Looks like we're overcommited, but we need to confirm that + * this is bad. We touch it all in a child process because an + * overcommit will generally lead to a SIGKILL which we can't + * handle, of course. + */ + child = SAFE_FORK(); + + if (child == 0) { + memset(p, 0, chunk1*hpage_size); + memset(q, 0, chunk2*hpage_size); + exit(0); + } + + SAFE_WAITPID(child, &status, 0); + + if (WIFSIGNALED(status)) { + tst_res(TFAIL, "Killed by signal \"%s\" due to overcommit", + strsignal(WTERMSIG(status))); + goto fail; + } + + tst_res(TPASS, "Successful with overcommit pages"); + +pass: + SAFE_MUNMAP(p, chunk1*hpage_size); + if (q && q != MAP_FAILED) + SAFE_MUNMAP(q, chunk2*hpage_size); + SAFE_CLOSE(fd); + return; +fail: + tst_brk(TBROK, "Once failed, No point in continuing the test"); +} + +static void run_test(unsigned int test_type) +{ + long saved_oc_hugepages; + + SAFE_FILE_LINES_SCANF(PROC_OVERCOMMIT, "%ld", &saved_oc_hugepages); + switch (test_type) { + case WITH_OVERCOMMIT: + if (saved_oc_hugepages > 0) + SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 0); + break; + case WITHOUT_OVERCOMMIT: + if (saved_oc_hugepages < 0) + tst_brk(TCONF, "Kernel appears to lack dynamic hugetlb pool " + "support"); + else if (saved_oc_hugepages == 0) + SAFE_FILE_PRINTF(PROC_OVERCOMMIT, "%d", 2); + break; + default: + tst_brk(TCONF, "Not a proper test type"); + break; + } + test_chunk_overcommit(); +} + +static void setup(void) +{ + if (tst_hugepages < 3) + tst_brk(TCONF, "Not enough hugepages for testing."); + + if (!Hopt) + Hopt = tst_get_tmpdir(); + SAFE_MOUNT("none", Hopt, "hugetlbfs", 0, NULL); + + snprintf(hfile, sizeof(hfile), "%s/ltp_huetlbfile%d", Hopt, getpid()); + hpage_size = SAFE_READ_MEMINFO("Hugepagesize:")*1024; +} + +static void cleanup(void) +{ + if (fd >= 0) + SAFE_CLOSE(fd); + umount2(Hopt, MNT_DETACH); +} + +static struct tst_test test = { + .needs_root = 1, + .needs_tmpdir = 1, + .forks_child = 1, + .options = (struct tst_option[]) { + {"v", &verbose, "Turns on verbose mode"}, + {"H:", &Hopt, "Location of hugetlbfs, i.e. -H /var/hugetlbfs"}, + {"s:", &nr_opt, "Set the number of the been allocated hugepages"}, + {} + }, + .tcnt = 2, + .setup = setup, + .cleanup = cleanup, + .test = run_test, + .hugepages = {5, TST_REQUEST}, +}; + -- 2.31.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp