* RE: [PATCH 11/16] byteorder: provide a linux/byteorder.h with {be, le}_to_cpu() and cpu_to_{be, le}() macros
From: David Laight @ 2014-05-28 8:45 UTC (permalink / raw)
To: 'Cody P Schafer', LKML, Linux PPC; +Cc: Sukadev Bhattiprolu
In-Reply-To: <1401236684-10579-12-git-send-email-dev@codyps.com>
RnJvbTogQ29keSBQIFNjaGFmZXINCj4gUmF0aGVyIG1hbnVhbGx5IHNwZWNpZnlpbmcgdGhlIHNp
emUgb2YgdGhlIGludGVnZXIgdG8gYmUgY29udmVydGVkLCBrZXkNCj4gb2ZmIG9mIHRoZSB0eXBl
IHNpemUuIFJlZHVjZXMgZHVwbGljYXRlIHNpemUgaW5mbyBhbmQgdGhlIG9jY3VyYW5jZSBvZg0K
PiBjZXJ0YWluIHR5cGVzIG9mIGJ1Z3MgKHVzaW5nIHRoZSB3cm9uZyBzaXplZCBjb252ZXJzaW9u
KS4NCi4uLg0KPiArI2RlZmluZSBiZV90b19jcHUodikgXA0KPiArCV9fYnVpbHRpbl9jaG9vc2Vf
ZXhwcihzaXplb2YodikgPT0gc2l6ZW9mKHVpbnQ4X3QpICwgdiwJXA0KPiArCV9fYnVpbHRpbl9j
aG9vc2VfZXhwcihzaXplb2YodikgPT0gc2l6ZW9mKHVpbnQxNl90KSwgYmUxNl90b19jcHUodiks
IFwNCj4gKwlfX2J1aWx0aW5fY2hvb3NlX2V4cHIoc2l6ZW9mKHYpID09IHNpemVvZih1aW50MzJf
dCksIGJlMzJfdG9fY3B1KHYpLCBcDQo+ICsJX19idWlsdGluX2Nob29zZV9leHByKHNpemVvZih2
KSA9PSBzaXplb2YodWludDY0X3QpLCBiZTY0X3RvX2NwdSh2KSwgXA0KPiArCQkodm9pZCkwKSkp
KQ0KLi4uDQoNCkknbSBub3QgYXQgYWxsIHN1cmUgdGhhdCB1c2luZyB0aGUgJ3NpemUnIG9mIHRo
ZSBjb25zdGFudCB3aWxsIHJlZHVjZQ0KdGhlIG51bWJlciBvZiBidWdzIC0gaXQganVzdCBpbnRy
b2R1Y2VzIGEgd2hvbGUgbmV3IGNhdGVnb3J5IG9mIGJ1Z3MuDQoNClVzaW5nIHRoZSBzaXplIG9m
IHRoZSBkZXN0aW5hdGlvbiBtaWdodCBoZWxwLCBidXQgdGhhdCBtYWtlcyB0aGUgY29kZQ0KdWds
eS4NCg0KR2V0dGluZyBvbmUgb2YgdGhlIHN0YXRpYyBhbmFseXNlcnMgdG8gZmluZCB0aGUgb2J2
aW91cyBlcnJvcnMgaXMNCnByb2JhYmx5IG1vcmUgYXBwcm9wcmlhdGUuDQoNCglEYXZpZA0KDQo=
^ permalink raw reply
* Kernel 3.15: Boot problems with a PA6T board
From: Christian Zigotzky @ 2014-05-28 8:53 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev
In-Reply-To: <1401251015.5468.5.camel@concordia>
Hi Michael,
Thank you for your answer and thank you for your help. :-)
On 28.05.2014 06:23, Michael Ellerman wrote:
> On Wed, 2014-05-28 at 01:08 +0200, Christian Zigotzky wrote:
>> Hi Michael,
>>
>> Thanks a lot for your answer.
>>
>> ...
>>
>> 18a1a7a1d862ae0794a0179473d08a414dd49234 <- It doesn't boot. Error messages: Oops: Machine check, sig: 7 [#1] CPU: 1 PID: 1 Comm: swapper/0 not tainted
>> d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b <- It boots. :-)
>> 0f5a869600141a0d5575e3190af01a050c081b07 <- It boots. :-)
>> c7e64b9ce04aa2e3fad7396d92b5cb92056d16ac <- It boots. :-)
>> d3e144532703fe2454b56eddb56f30d2d620187b <- It boots. :-)
>>
>> I think the machine check is the problem.
> Yes I think it is. Do you get any more info, or just that one line?
>
> So I think the latest working commit we have is d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b.
>
> I'm going to guess that cd427485357c0c4b99f69719251baacf25946e11 is BAD. Can
> you please confirm or deny that?
It's not BAD. It boots.
Rgds,
Christian
>
> Assuming cd42748 is bad, you should do a git bisect between it and 18a1a7a.
> That should be a fairly quick bisect. That would be:
>
> $ git bisect start
> $ git bisect good d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b
> $ git bisect bad cd427485357c0c4b99f69719251baacf25946e11
>
> If cd42748 is *good*, then you'll need to do a bigger bisect from d8ff9cd to
> 18a1a7a.
>
>
> cheers
>
>
>
^ permalink raw reply
* Re: [PATCH 1/2] powerpc/mm: Check paca psize is up to date for huge mappings
From: Aneesh Kumar K.V @ 2014-05-28 9:27 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev; +Cc: aneesh.kumar, Paul Mackerras
In-Reply-To: <1401265278-29307-1-git-send-email-mpe@ellerman.id.au>
Michael Ellerman <mpe@ellerman.id.au> writes:
> We have a bug in our hugepage handling which exhibits as an infinite
> loop of hash faults. If the fault is being taken in the kernel it will
> typically trigger the softlockup detector, or the RCU stall detector.
>
> The bug is as follows:
>
> 1. mmap(0xa0000000, ..., MAP_FIXED | MAP_HUGE_TLB | MAP_ANONYMOUS ..)
> 2. Slice code converts the slice psize to 16M.
> 3. The code on lines 539-540 of slice.c in slice_get_unmapped_area()
> synchronises the mm->context with the paca->context. So the paca slice
> mask is updated to include the 16M slice.
> 3. Either:
> * mmap() fails because there are no huge pages available.
> * mmap() succeeds and the mapping is then munmapped.
> In both cases the slice psize remains at 16M in both the paca & mm.
> 4. mmap(0xa0000000, ..., MAP_FIXED | MAP_ANONYMOUS ..)
> 5. The slice psize is converted back to 64K. Because of the check on line 539
> of slice.c we DO NOT update the paca->context. The paca slice mask is now
> out of sync with the mm slice mask.
> 6. User/kernel accesses 0xa0000000.
> 7. The SLB miss handler slb_allocate_realmode() **uses the paca slice mask**
> to create an SLB entry and inserts it in the SLB.
> 18. With the 16M SLB entry in place the hardware does a hash lookup, no entry
> is found so a data access exception is generated.
> 19. The data access handler calls do_page_fault() -> handle_mm_fault().
> 10. __handle_mm_fault() creates a THP mapping with do_huge_pmd_anonymous_page().
> 11. The hardware retries the access, there is still nothing in the hash table
> so once again a data access exception is generated.
> 12. hash_page() calls into __hash_page_thp() and inserts a mapping in the
> hash. Although the THP mapping maps 16M the hashing is done using 64K
> as the segment page size.
> 13. hash_page() returns immediately after calling __hash_page_thp(), skipping
> over the code at line 1125. Resulting in the mismatch between the
> paca->context and mm->context not being detected.
> 14. The hardware retries the access, the hash it generates using the 16M
> SLB entry does NOT match the hash we inserted.
> 15. We take another data access and go into __hash_page_thp().
> 16. We see a valid entry in the hpte_slot_array and so we call updatepp()
> which succeeds.
> 17. Goto 14.
>
> We could fix this in two ways. The first would be to remove or modify
> the check on line 539 of slice.c.
>
> The second option is to cause the check of paca psize in hash_page() on
> line 1125 to also be done for THP pages.
>
> We prefer the latter, because the check & update of the paca psize is
> not done until we know it's necessary. It's also done only on the
> current cpu, so we don't need to IPI all other cpus.
>
> Without further rearranging the code, the simplest fix is to pull out
> the code that checks paca psize and call it in two places. Firstly for
> THP/hugetlb, and secondly for other mappings as before.
>
> Thanks to Dave Jones for trinity, which originally found this bug.
>
> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> ---
> arch/powerpc/mm/hash_utils_64.c | 31 ++++++++++++++++++++-----------
> 1 file changed, 20 insertions(+), 11 deletions(-)
>
> diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
> index d766d6e..6650699 100644
> --- a/arch/powerpc/mm/hash_utils_64.c
> +++ b/arch/powerpc/mm/hash_utils_64.c
> @@ -960,6 +960,22 @@ void hash_failure_debug(unsigned long ea, unsigned long access,
> trap, vsid, ssize, psize, lpsize, pte);
> }
>
> +static void check_paca_psize(unsigned long ea, struct mm_struct *mm,
> + int psize, bool user_region)
> +{
> + if (user_region) {
> + if (psize != get_paca_psize(ea)) {
> + get_paca()->context = mm->context;
> + slb_flush_and_rebolt();
> + }
> + } else if (get_paca()->vmalloc_sllp !=
> + mmu_psize_defs[mmu_vmalloc_psize].sllp) {
> + get_paca()->vmalloc_sllp =
> + mmu_psize_defs[mmu_vmalloc_psize].sllp;
> + slb_vmalloc_update();
> + }
> +}
> +
> /* Result code is:
> * 0 - handled
> * 1 - normal page fault
> @@ -1081,6 +1097,8 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
> WARN_ON(1);
> }
> #endif
> + check_paca_psize(ea, mm, psize, user_region);
> +
> goto bail;
> }
>
> @@ -1121,17 +1139,8 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
> #endif
> }
> }
> - if (user_region) {
> - if (psize != get_paca_psize(ea)) {
> - get_paca()->context = mm->context;
> - slb_flush_and_rebolt();
> - }
> - } else if (get_paca()->vmalloc_sllp !=
> - mmu_psize_defs[mmu_vmalloc_psize].sllp) {
> - get_paca()->vmalloc_sllp =
> - mmu_psize_defs[mmu_vmalloc_psize].sllp;
> - slb_vmalloc_update();
> - }
> +
> + check_paca_psize(ea, mm, psize, user_region);
> #endif /* CONFIG_PPC_64K_PAGES */
>
> #ifdef CONFIG_PPC_HAS_HASH_64K
> --
> 1.9.1
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
^ permalink raw reply
* Re: [PATCH 2/2] selftests/powerpc: Test the THP bug we fixed in the previous commit
From: Aneesh Kumar K.V @ 2014-05-28 9:27 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev; +Cc: aneesh.kumar, Paul Mackerras
In-Reply-To: <1401265278-29307-2-git-send-email-mpe@ellerman.id.au>
Michael Ellerman <mpe@ellerman.id.au> writes:
> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> ---
> tools/testing/selftests/powerpc/Makefile | 2 +-
> tools/testing/selftests/powerpc/mm/Makefile | 18 ++++++
> .../selftests/powerpc/mm/hugetlb_vs_thp_test.c | 72 ++++++++++++++++++++++
> 3 files changed, 91 insertions(+), 1 deletion(-)
> create mode 100644 tools/testing/selftests/powerpc/mm/Makefile
> create mode 100644 tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c
>
> diff --git a/tools/testing/selftests/powerpc/Makefile b/tools/testing/selftests/powerpc/Makefile
> index 316194f..b3dbe9e 100644
> --- a/tools/testing/selftests/powerpc/Makefile
> +++ b/tools/testing/selftests/powerpc/Makefile
> @@ -13,7 +13,7 @@ CFLAGS := -Wall -O2 -flto -Wall -Werror -DGIT_VERSION='"$(GIT_VERSION)"' -I$(CUR
>
> export CC CFLAGS
>
> -TARGETS = pmu copyloops
> +TARGETS = pmu copyloops mm
>
> endif
>
> diff --git a/tools/testing/selftests/powerpc/mm/Makefile b/tools/testing/selftests/powerpc/mm/Makefile
> new file mode 100644
> index 0000000..357ccbd
> --- /dev/null
> +++ b/tools/testing/selftests/powerpc/mm/Makefile
> @@ -0,0 +1,18 @@
> +noarg:
> + $(MAKE) -C ../
> +
> +PROGS := hugetlb_vs_thp_test
> +
> +all: $(PROGS)
> +
> +$(PROGS): ../harness.c
> +
> +run_tests: all
> + @-for PROG in $(PROGS); do \
> + ./$$PROG; \
> + done;
> +
> +clean:
> + rm -f $(PROGS)
> +
> +.PHONY: all run_tests clean
> diff --git a/tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c b/tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c
> new file mode 100644
> index 0000000..3d8e5b0
> --- /dev/null
> +++ b/tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c
> @@ -0,0 +1,72 @@
> +#include <stdio.h>
> +#include <sys/mman.h>
> +#include <unistd.h>
> +
> +#include "utils.h"
> +
> +/* This must match the huge page & THP size */
> +#define SIZE (16 * 1024 * 1024)
> +
> +static int test_body(void)
> +{
> + void *addr;
> + char *p;
> +
> + addr = (void *)0xa0000000;
> +
> + p = mmap(addr, SIZE, PROT_READ | PROT_WRITE,
> + MAP_HUGETLB | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
> + if (p != MAP_FAILED) {
> + /*
> + * Typically the mmap will fail because no huge pages are
> + * allocated on the system. But if there are huge pages
> + * allocated the mmap will succeed. That's fine too, we just
> + * munmap here before continuing.
> + */
> + munmap(addr, SIZE);
> + }
> +
> + p = mmap(addr, SIZE, PROT_READ | PROT_WRITE,
> + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
> + if (p == MAP_FAILED) {
> + printf("Mapping failed @ %p\n", addr);
> + perror("mmap");
> + return 1;
> + }
> +
> + /*
> + * Either a user or kernel access is sufficient to trigger the bug.
> + * A kernel access is easier to spot & debug, as it will trigger the
> + * softlockup or RCU stall detectors, and when the system is kicked
> + * into xmon we get a backtrace in the kernel.
> + *
> + * A good option is:
> + * getcwd(p, SIZE);
> + *
> + * For the purposes of this testcase it's preferable to spin in
> + * userspace, so the harness can kill us if we get stuck. That way we
> + * see a test failure rather than a dead system.
> + */
> + *p = 0xf;
> +
> + munmap(addr, SIZE);
> +
> + return 0;
> +}
> +
> +static int test_main(void)
> +{
> + int i;
> +
> + /* 10,000 because it's a "bunch", and completes reasonably quickly */
> + for (i = 0; i < 10000; i++)
> + if (test_body())
> + return 1;
> +
> + return 0;
> +}
> +
> +int main(void)
> +{
> + return test_harness(test_main, "hugetlb_vs_thp");
> +}
> --
> 1.9.1
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alexander Graf @ 2014-05-28 11:37 UTC (permalink / raw)
To: Alex Williamson; +Cc: aik, Gavin Shan, kvm-ppc, qiudayu, linuxppc-dev
In-Reply-To: <1401238674.3289.679.camel@ul30vt.home>
On 28.05.14 02:57, Alex Williamson wrote:
> On Wed, 2014-05-28 at 02:44 +0200, Alexander Graf wrote:
>> On 28.05.14 02:39, Alex Williamson wrote:
>>> On Wed, 2014-05-28 at 00:49 +0200, Alexander Graf wrote:
>>>> On 27.05.14 20:15, Alex Williamson wrote:
>>>>> On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
>>>>>> The patch adds new IOCTL commands for sPAPR VFIO container device
>>>>>> to support EEH functionality for PCI devices, which have been passed
>>>>>> through from host to somebody else via VFIO.
>>>>>>
>>>>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>>>>> ---
>>>>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
>>>>>> drivers/vfio/pci/Makefile | 1 +
>>>>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
>>>>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
>>>>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
>>>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
>>>>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
>>>>>> 7 files changed, 308 insertions(+), 7 deletions(-)
>>>>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
>>>> [...]
>>>>
>>>>>> +
>>>>>> + return ret;
>>>>>> +}
>>>>>> +
>>>>>> static long tce_iommu_ioctl(void *iommu_data,
>>>>>> unsigned int cmd, unsigned long arg)
>>>>>> {
>>>>>> @@ -283,6 +363,11 @@ static long tce_iommu_ioctl(void *iommu_data,
>>>>>> tce_iommu_disable(container);
>>>>>> mutex_unlock(&container->lock);
>>>>>> return 0;
>>>>>> + case VFIO_EEH_PE_SET_OPTION:
>>>>>> + case VFIO_EEH_PE_GET_STATE:
>>>>>> + case VFIO_EEH_PE_RESET:
>>>>>> + case VFIO_EEH_PE_CONFIGURE:
>>>>>> + return tce_iommu_eeh_ioctl(iommu_data, cmd, arg);
>>>>> This is where it would have really made sense to have a single
>>>>> VFIO_EEH_OP ioctl with a data structure passed to indicate the sub-op.
>>>>> AlexG, are you really attached to splitting these out into separate
>>>>> ioctls?
>>>> I don't see the problem. We need to forward 4 ioctls to a separate piece
>>>> of code, so we forward 4 ioctls to a separate piece of code :). Putting
>>>> them into one ioctl just moves the switch() into another function.
>>> And uses an extra 3 ioctl numbers and gives us extra things to update if
>>> we ever need to add more ioctls, etc. ioctl numbers are an address
>>> space, how much address space do we really want to give to EEH? It's
>>> not a big difference, but I don't think it's completely even either.
>>> Thanks,
>> Yes, that's the point. I by far prefer to have you push back on anyone
>> who introduces useless ioctls rather than have a separate EEH number
>> space that people can just throw anything in they like ;).
> Well, I appreciate that, but having them as separate ioctls doesn't
> really prevent that either. Any one of these 4 could be set to take a
> sub-option to extend and contort the EEH interface. The only way to
> prevent that would be to avoid the argsz+flags hack that make the ioctl
> extendable. Thanks,
Sure, that's what patch review is about. I'm really more concerned about
whose court the number space is in - you or Gavin. If we're talking
about top level ioctls you will care a lot more.
But I'm not religious about this. You're the VFIO maintainer, so it's
your call. I just personally cringe when I see an ioctl that gets an
"opcode" and a "parameter" argument where the "parameter" argument is a
union with one struct for each opcode.
Alex
^ permalink raw reply
* Kernel 3.15: Boot problems with a PA6T board
From: Christian Zigotzky @ 2014-05-28 11:25 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev
In-Reply-To: <5385A3F4.8070004@xenosoft.de>
On 28.05.2014 10:53, Christian Zigotzky wrote:
> Hi Michael,
>
> Thank you for your answer and thank you for your help. :-)
>
> On 28.05.2014 06:23, Michael Ellerman wrote:
>> On Wed, 2014-05-28 at 01:08 +0200, Christian Zigotzky wrote:
>>> Hi Michael,
>>>
>>> Thanks a lot for your answer.
>>>
>>> ...
>>>
>>> 18a1a7a1d862ae0794a0179473d08a414dd49234 <- It doesn't boot. Error
>>> messages: Oops: Machine check, sig: 7 [#1] CPU: 1 PID: 1 Comm:
>>> swapper/0 not tainted
>>> d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b <- It boots. :-)
>>> 0f5a869600141a0d5575e3190af01a050c081b07 <- It boots. :-)
>>> c7e64b9ce04aa2e3fad7396d92b5cb92056d16ac <- It boots. :-)
>>> d3e144532703fe2454b56eddb56f30d2d620187b <- It boots. :-)
>>>
>>> I think the machine check is the problem.
>> Yes I think it is. Do you get any more info, or just that one line?
>>
>> So I think the latest working commit we have is
>> d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b.
>>
>> I'm going to guess that cd427485357c0c4b99f69719251baacf25946e11 is
>> BAD. Can
>> you please confirm or deny that?
> It's not BAD. It boots.
>
> Rgds,
>
> Christian
>>
>> Assuming cd42748 is bad, you should do a git bisect between it and
>> 18a1a7a.
>> That should be a fairly quick bisect. That would be:
>>
>> $ git bisect start
>> $ git bisect good d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b
>> $ git bisect bad cd427485357c0c4b99f69719251baacf25946e11
>>
>> If cd42748 is *good*, then you'll need to do a bigger bisect from
>> d8ff9cd to
>> 18a1a7a.
OK :-)
-> git clone
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux2-git
-> git bisect start
-> git bisect good d8ff9cdf68fd119d491f3de90e1a612afc2f3b2b
-> git bisect bad 18a1a7a1d862ae0794a0179473d08a414dd49234
Output:
Bisecting: 5900 revisions left to test after this (roughly 13 steps)
[cb1595563880a81dab6eab9a5ecb4520d2e76077] Merge tag 'tty-3.15-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
Unfortunately it doesn't boot. :-(
>>
>>
>> cheers
>>
>>
>>
>
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alexander Graf @ 2014-05-28 11:41 UTC (permalink / raw)
To: Gavin Shan, Alex Williamson; +Cc: aik, qiudayu, linuxppc-dev, kvm-ppc
In-Reply-To: <20140528005532.GA5528@shangw>
On 28.05.14 02:55, Gavin Shan wrote:
> On Tue, May 27, 2014 at 12:15:27PM -0600, Alex Williamson wrote:
>> On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
>>> The patch adds new IOCTL commands for sPAPR VFIO container device
>>> to support EEH functionality for PCI devices, which have been passed
>>> through from host to somebody else via VFIO.
>>>
>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>> ---
>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
>>> drivers/vfio/pci/Makefile | 1 +
>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
>>> 7 files changed, 308 insertions(+), 7 deletions(-)
>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
[...]
>>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>>> index cb9023d..c5fac36 100644
>>> --- a/include/uapi/linux/vfio.h
>>> +++ b/include/uapi/linux/vfio.h
>>> @@ -455,6 +455,72 @@ struct vfio_iommu_spapr_tce_info {
>>>
>>> #define VFIO_IOMMU_SPAPR_TCE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>>
>>> +/*
>>> + * EEH functionality can be enabled or disabled on one specific device.
>>> + * Also, the DMA or IO frozen state can be removed from the frozen PE
>>> + * if required.
>>> + */
>>> +struct vfio_eeh_pe_set_option {
>>> + __u32 argsz;
>>> + __u32 flags;
>>> + __u32 option;
>>> +#define VFIO_EEH_PE_SET_OPT_DISABLE 0 /* Disable EEH */
>>> +#define VFIO_EEH_PE_SET_OPT_ENABLE 1 /* Enable EEH */
>>> +#define VFIO_EEH_PE_SET_OPT_IO 2 /* Enable IO */
>>> +#define VFIO_EEH_PE_SET_OPT_DMA 3 /* Enable DMA */
>> This is more of a "command" than an "option" isn't it? Each of these
>> probably needs a more significant description.
>>
> Yeah, it would be regarded as "opcode" and I'll add more description about
> them in next revision.
Please just call them commands.
>
>>> +};
>>> +
>>> +#define VFIO_EEH_PE_SET_OPTION _IO(VFIO_TYPE, VFIO_BASE + 21)
>>> +
>>> +/*
>>> + * Each EEH PE should have unique address to be identified. PE's
>>> + * sharing mode is also useful information as well.
>>> + */
>>> +#define VFIO_EEH_PE_GET_ADDRESS 0 /* Get address */
>>> +#define VFIO_EEH_PE_GET_MODE 1 /* Query mode */
>>> +#define VFIO_EEH_PE_MODE_NONE 0 /* Not a PE */
>>> +#define VFIO_EEH_PE_MODE_NOT_SHARED 1 /* Exclusive */
>>> +#define VFIO_EEH_PE_MODE_SHARED 2 /* Shared mode */
>>> +
>>> +/*
>>> + * EEH PE might have been frozen because of PCI errors. Also, it might
>>> + * be experiencing reset for error revoery. The following command helps
>>> + * to get the state.
>>> + */
>>> +struct vfio_eeh_pe_get_state {
>>> + __u32 argsz;
>>> + __u32 flags;
>>> + __u32 state;
>>> +};
>> Should state be a union to better describe the value returned? What
>> exactly is the address and why does the user need to know it? Does this
>> need user input or could we just return the address and mode regardless?
>>
> Ok. I think you want enum (not union) for state. I'll have macros for the
> state in next revision as I did that for other cases.
>
> Those macros defined for "address" just for ABI stuff as Alex.G mentioned.
> There isn't corresponding ioctl command for host to get address any more
> because QEMU (user) will have to figure it out by himself. The "address"
> here means PE address and user has to figure it out according to PE
> segmentation.
Why would the user ever need the address?
Alex
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Gavin Shan @ 2014-05-28 12:49 UTC (permalink / raw)
To: Alexander Graf
Cc: aik, Gavin Shan, kvm-ppc, Alex Williamson, qiudayu, linuxppc-dev
In-Reply-To: <5385CB6F.50300@suse.de>
On Wed, May 28, 2014 at 01:41:35PM +0200, Alexander Graf wrote:
>
>On 28.05.14 02:55, Gavin Shan wrote:
>>On Tue, May 27, 2014 at 12:15:27PM -0600, Alex Williamson wrote:
>>>On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
>>>>The patch adds new IOCTL commands for sPAPR VFIO container device
>>>>to support EEH functionality for PCI devices, which have been passed
>>>>through from host to somebody else via VFIO.
>>>>
>>>>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>>>---
>>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
>>>> drivers/vfio/pci/Makefile | 1 +
>>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
>>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
>>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
>>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
>>>> 7 files changed, 308 insertions(+), 7 deletions(-)
>>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
>
>[...]
>
>>>>diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>>>>index cb9023d..c5fac36 100644
>>>>--- a/include/uapi/linux/vfio.h
>>>>+++ b/include/uapi/linux/vfio.h
>>>>@@ -455,6 +455,72 @@ struct vfio_iommu_spapr_tce_info {
>>>> #define VFIO_IOMMU_SPAPR_TCE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>>>+/*
>>>>+ * EEH functionality can be enabled or disabled on one specific device.
>>>>+ * Also, the DMA or IO frozen state can be removed from the frozen PE
>>>>+ * if required.
>>>>+ */
>>>>+struct vfio_eeh_pe_set_option {
>>>>+ __u32 argsz;
>>>>+ __u32 flags;
>>>>+ __u32 option;
>>>>+#define VFIO_EEH_PE_SET_OPT_DISABLE 0 /* Disable EEH */
>>>>+#define VFIO_EEH_PE_SET_OPT_ENABLE 1 /* Enable EEH */
>>>>+#define VFIO_EEH_PE_SET_OPT_IO 2 /* Enable IO */
>>>>+#define VFIO_EEH_PE_SET_OPT_DMA 3 /* Enable DMA */
>>>This is more of a "command" than an "option" isn't it? Each of these
>>>probably needs a more significant description.
>>>
>>Yeah, it would be regarded as "opcode" and I'll add more description about
>>them in next revision.
>
>Please just call them commands.
>
Ok. I guess you want me to change the macro names like this ?
#define VFIO_EEH_CMD_DISABLE 0 /* Disable EEH functionality */
#define VFIO_EEH_CMD_ENABLE 1 /* Enable EEH functionality */
#define VFIO_EEH_CMD_ENABLE_IO 2 /* Enable IO for frozen PE */
#define VFIO_EEH_CMD_ENABLE_DMA 3 /* Enable DMA for frozen PE */
>>
>>>>+};
>>>>+
>>>>+#define VFIO_EEH_PE_SET_OPTION _IO(VFIO_TYPE, VFIO_BASE + 21)
>>>>+
>>>>+/*
>>>>+ * Each EEH PE should have unique address to be identified. PE's
>>>>+ * sharing mode is also useful information as well.
>>>>+ */
>>>>+#define VFIO_EEH_PE_GET_ADDRESS 0 /* Get address */
>>>>+#define VFIO_EEH_PE_GET_MODE 1 /* Query mode */
>>>>+#define VFIO_EEH_PE_MODE_NONE 0 /* Not a PE */
>>>>+#define VFIO_EEH_PE_MODE_NOT_SHARED 1 /* Exclusive */
>>>>+#define VFIO_EEH_PE_MODE_SHARED 2 /* Shared mode */
>>>>+
>>>>+/*
>>>>+ * EEH PE might have been frozen because of PCI errors. Also, it might
>>>>+ * be experiencing reset for error revoery. The following command helps
>>>>+ * to get the state.
>>>>+ */
>>>>+struct vfio_eeh_pe_get_state {
>>>>+ __u32 argsz;
>>>>+ __u32 flags;
>>>>+ __u32 state;
>>>>+};
>>>Should state be a union to better describe the value returned? What
>>>exactly is the address and why does the user need to know it? Does this
>>>need user input or could we just return the address and mode regardless?
>>>
>>Ok. I think you want enum (not union) for state. I'll have macros for the
>>state in next revision as I did that for other cases.
>>
>>Those macros defined for "address" just for ABI stuff as Alex.G mentioned.
>>There isn't corresponding ioctl command for host to get address any more
>>because QEMU (user) will have to figure it out by himself. The "address"
>>here means PE address and user has to figure it out according to PE
>>segmentation.
>
>Why would the user ever need the address?
>
I will remove those "address" related macros in next revision because it's
user-level bussiness, not related to host kernel any more.
If the user is QEMU + guest, we need the address to identify the PE though PHB
BUID could be used as same purpose to get PHB, which is one-to-one mapping with
IOMMU group on sPAPR platform. However, once the PE address is built and returned
to guest, guest will use the PE address as input parameter in subsequent RTAS
calls.
If the user is some kind of application who just uses the ioctl() without supporting
RTAS calls. We don't need care about PE address.
Thanks,
Gavin
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alexander Graf @ 2014-05-28 13:12 UTC (permalink / raw)
To: Gavin Shan; +Cc: aik, kvm-ppc, Alex Williamson, qiudayu, linuxppc-dev
In-Reply-To: <20140528124947.GA19346@shangw>
On 28.05.14 14:49, Gavin Shan wrote:
> On Wed, May 28, 2014 at 01:41:35PM +0200, Alexander Graf wrote:
>> On 28.05.14 02:55, Gavin Shan wrote:
>>> On Tue, May 27, 2014 at 12:15:27PM -0600, Alex Williamson wrote:
>>>> On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
>>>>> The patch adds new IOCTL commands for sPAPR VFIO container device
>>>>> to support EEH functionality for PCI devices, which have been passed
>>>>> through from host to somebody else via VFIO.
>>>>>
>>>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>>>> ---
>>>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
>>>>> drivers/vfio/pci/Makefile | 1 +
>>>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
>>>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
>>>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
>>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
>>>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
>>>>> 7 files changed, 308 insertions(+), 7 deletions(-)
>>>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
>> [...]
>>
>>>>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>>>>> index cb9023d..c5fac36 100644
>>>>> --- a/include/uapi/linux/vfio.h
>>>>> +++ b/include/uapi/linux/vfio.h
>>>>> @@ -455,6 +455,72 @@ struct vfio_iommu_spapr_tce_info {
>>>>> #define VFIO_IOMMU_SPAPR_TCE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>>>> +/*
>>>>> + * EEH functionality can be enabled or disabled on one specific device.
>>>>> + * Also, the DMA or IO frozen state can be removed from the frozen PE
>>>>> + * if required.
>>>>> + */
>>>>> +struct vfio_eeh_pe_set_option {
>>>>> + __u32 argsz;
>>>>> + __u32 flags;
>>>>> + __u32 option;
>>>>> +#define VFIO_EEH_PE_SET_OPT_DISABLE 0 /* Disable EEH */
>>>>> +#define VFIO_EEH_PE_SET_OPT_ENABLE 1 /* Enable EEH */
>>>>> +#define VFIO_EEH_PE_SET_OPT_IO 2 /* Enable IO */
>>>>> +#define VFIO_EEH_PE_SET_OPT_DMA 3 /* Enable DMA */
>>>> This is more of a "command" than an "option" isn't it? Each of these
>>>> probably needs a more significant description.
>>>>
>>> Yeah, it would be regarded as "opcode" and I'll add more description about
>>> them in next revision.
>> Please just call them commands.
>>
> Ok. I guess you want me to change the macro names like this ?
>
> #define VFIO_EEH_CMD_DISABLE 0 /* Disable EEH functionality */
> #define VFIO_EEH_CMD_ENABLE 1 /* Enable EEH functionality */
> #define VFIO_EEH_CMD_ENABLE_IO 2 /* Enable IO for frozen PE */
> #define VFIO_EEH_CMD_ENABLE_DMA 3 /* Enable DMA for frozen PE */
Yes, the ioctl name too.
>
>>>>> +};
>>>>> +
>>>>> +#define VFIO_EEH_PE_SET_OPTION _IO(VFIO_TYPE, VFIO_BASE + 21)
>>>>> +
>>>>> +/*
>>>>> + * Each EEH PE should have unique address to be identified. PE's
>>>>> + * sharing mode is also useful information as well.
>>>>> + */
>>>>> +#define VFIO_EEH_PE_GET_ADDRESS 0 /* Get address */
>>>>> +#define VFIO_EEH_PE_GET_MODE 1 /* Query mode */
>>>>> +#define VFIO_EEH_PE_MODE_NONE 0 /* Not a PE */
>>>>> +#define VFIO_EEH_PE_MODE_NOT_SHARED 1 /* Exclusive */
>>>>> +#define VFIO_EEH_PE_MODE_SHARED 2 /* Shared mode */
>>>>> +
>>>>> +/*
>>>>> + * EEH PE might have been frozen because of PCI errors. Also, it might
>>>>> + * be experiencing reset for error revoery. The following command helps
>>>>> + * to get the state.
>>>>> + */
>>>>> +struct vfio_eeh_pe_get_state {
>>>>> + __u32 argsz;
>>>>> + __u32 flags;
>>>>> + __u32 state;
>>>>> +};
>>>> Should state be a union to better describe the value returned? What
>>>> exactly is the address and why does the user need to know it? Does this
>>>> need user input or could we just return the address and mode regardless?
>>>>
>>> Ok. I think you want enum (not union) for state. I'll have macros for the
>>> state in next revision as I did that for other cases.
>>>
>>> Those macros defined for "address" just for ABI stuff as Alex.G mentioned.
>>> There isn't corresponding ioctl command for host to get address any more
>>> because QEMU (user) will have to figure it out by himself. The "address"
>>> here means PE address and user has to figure it out according to PE
>>> segmentation.
>> Why would the user ever need the address?
>>
> I will remove those "address" related macros in next revision because it's
> user-level bussiness, not related to host kernel any more.
Ok, so the next question is whether there will be any state outside of
GET_MODE left in the future.
Alex
> If the user is QEMU + guest, we need the address to identify the PE though PHB
> BUID could be used as same purpose to get PHB, which is one-to-one mapping with
> IOMMU group on sPAPR platform. However, once the PE address is built and returned
> to guest, guest will use the PE address as input parameter in subsequent RTAS
> calls.
>
> If the user is some kind of application who just uses the ioctl() without supporting
> RTAS calls. We don't need care about PE address.
>
> Thanks,
> Gavin
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] powerpc, kexec: Fix "Processor X is stuck" issue during kexec from ST mode
From: Vivek Goyal @ 2014-05-28 13:31 UTC (permalink / raw)
To: Srivatsa S. Bhat
Cc: ego, kexec, linux-kernel, suzuki, ebiederm, mahesh, paulus,
linuxppc-dev
In-Reply-To: <20140527105511.24309.74320.stgit@srivatsabhat.in.ibm.com>
On Tue, May 27, 2014 at 04:25:34PM +0530, Srivatsa S. Bhat wrote:
> If we try to perform a kexec when the machine is in ST (Single-Threaded) mode
> (ppc64_cpu --smt=off), the kexec operation doesn't succeed properly, and we
> get the following messages during boot:
>
> [ 0.089866] POWER8 performance monitor hardware support registered
> [ 0.089985] power8-pmu: PMAO restore workaround active.
> [ 5.095419] Processor 1 is stuck.
> [ 10.097933] Processor 2 is stuck.
> [ 15.100480] Processor 3 is stuck.
> [ 20.102982] Processor 4 is stuck.
> [ 25.105489] Processor 5 is stuck.
> [ 30.108005] Processor 6 is stuck.
> [ 35.110518] Processor 7 is stuck.
> [ 40.113369] Processor 9 is stuck.
> [ 45.115879] Processor 10 is stuck.
> [ 50.118389] Processor 11 is stuck.
> [ 55.120904] Processor 12 is stuck.
> [ 60.123425] Processor 13 is stuck.
> [ 65.125970] Processor 14 is stuck.
> [ 70.128495] Processor 15 is stuck.
> [ 75.131316] Processor 17 is stuck.
>
> Note that only the sibling threads are stuck, while the primary threads (0, 8,
> 16 etc) boot just fine. Looking closer at the previous step of kexec, we observe
> that kexec tries to wakeup (bring online) the sibling threads of all the cores,
> before performing kexec:
>
> [ 9464.131231] Starting new kernel
> [ 9464.148507] kexec: Waking offline cpu 1.
> [ 9464.148552] kexec: Waking offline cpu 2.
> [ 9464.148600] kexec: Waking offline cpu 3.
> [ 9464.148636] kexec: Waking offline cpu 4.
> [ 9464.148671] kexec: Waking offline cpu 5.
> [ 9464.148708] kexec: Waking offline cpu 6.
> [ 9464.148743] kexec: Waking offline cpu 7.
> [ 9464.148779] kexec: Waking offline cpu 9.
> [ 9464.148815] kexec: Waking offline cpu 10.
> [ 9464.148851] kexec: Waking offline cpu 11.
> [ 9464.148887] kexec: Waking offline cpu 12.
> [ 9464.148922] kexec: Waking offline cpu 13.
> [ 9464.148958] kexec: Waking offline cpu 14.
> [ 9464.148994] kexec: Waking offline cpu 15.
> [ 9464.149030] kexec: Waking offline cpu 17.
>
> Instrumenting this piece of code revealed that the cpu_up() operation actually
> fails with -EBUSY. Thus, only the primary threads of all the cores are online
> during kexec, and hence this is a sure-shot receipe for disaster, as explained
> in commit e8e5c2155b (powerpc/kexec: Fix orphaned offline CPUs across kexec),
> as well as in the comment above wake_offline_cpus().
>
> It turns out that cpu_up() was returning -EBUSY because the variable
> 'cpu_hotplug_disabled' was set to 1; and this disabling of CPU hotplug was done
> by migrate_to_reboot_cpu() inside kernel_kexec().
>
> Now, migrate_to_reboot_cpu() was originally written with the assumption that
> any further code will not need to perform CPU hotplug, since we are anyway in
> the reboot path. However, kexec is clearly not such a case, since we depend on
> onlining CPUs, atleast on powerpc.
>
> So re-enable cpu-hotplug after returning from migrate_to_reboot_cpu() in the
> kexec path, to fix this regression in kexec on powerpc.
>
> Also, wrap the cpu_up() in powerpc kexec code within a WARN_ON(), so that we
> can catch such issues more easily in the future.
>
> Fixes: c97102ba963 (kexec: migrate to reboot cpu)
> Cc: stable@vger.kernel.org
> Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
> ---
>
> arch/powerpc/kernel/machine_kexec_64.c | 2 +-
> kernel/kexec.c | 8 ++++++++
> 2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/machine_kexec_64.c b/arch/powerpc/kernel/machine_kexec_64.c
> index 59d229a..879b3aa 100644
> --- a/arch/powerpc/kernel/machine_kexec_64.c
> +++ b/arch/powerpc/kernel/machine_kexec_64.c
> @@ -237,7 +237,7 @@ static void wake_offline_cpus(void)
> if (!cpu_online(cpu)) {
> printk(KERN_INFO "kexec: Waking offline cpu %d.\n",
> cpu);
> - cpu_up(cpu);
> + WARN_ON(cpu_up(cpu));
> }
> }
> }
> diff --git a/kernel/kexec.c b/kernel/kexec.c
> index c8380ad..28c5706 100644
> --- a/kernel/kexec.c
> +++ b/kernel/kexec.c
> @@ -1683,6 +1683,14 @@ int kernel_kexec(void)
> kexec_in_progress = true;
> kernel_restart_prepare(NULL);
> migrate_to_reboot_cpu();
> +
> + /*
> + * migrate_to_reboot_cpu() disables CPU hotplug assuming that
> + * no further code needs to use CPU hotplug (which is true in
> + * the reboot case). However, the kexec path depends on using
> + * CPU hotplug again; so re-enable it here.
> + */
> + cpu_hotplug_enable();
> printk(KERN_EMERG "Starting new kernel\n");
> machine_shutdown();
After migrate_to_reboot_cpu(), we are calling machine_shutdown() which
calls disable_nonboot_cpus() and which in turn calls _cpu_down().
So it is kind of odd that we first migrate to boot cpu, and then disable
all non-boot cpus and after that powerpc goes ahead and onlines all
cpus.
I think this is not a good idea. For whatever reason if powerpc has to
online all cpus, then it should happne earlier and not in machine_kexec().
In fact I think generic code expects that all non-boot cpus are disabled
so that generic code can use all the RAM as it wants to. Now if powerpc
breaks that assumption, it will lead to various kind of issues.
So I think we need to go back and see if we can find a way where we
don't have to online all cpus in first kernel. And second kernel needs
to have a way to detect it and online things.
Thanks
Vivek
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alex Williamson @ 2014-05-28 16:17 UTC (permalink / raw)
To: Alexander Graf; +Cc: aik, Gavin Shan, kvm-ppc, qiudayu, linuxppc-dev
In-Reply-To: <5385CA86.3010700@suse.de>
On Wed, 2014-05-28 at 13:37 +0200, Alexander Graf wrote:
> On 28.05.14 02:57, Alex Williamson wrote:
> > On Wed, 2014-05-28 at 02:44 +0200, Alexander Graf wrote:
> >> On 28.05.14 02:39, Alex Williamson wrote:
> >>> On Wed, 2014-05-28 at 00:49 +0200, Alexander Graf wrote:
> >>>> On 27.05.14 20:15, Alex Williamson wrote:
> >>>>> On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
> >>>>>> The patch adds new IOCTL commands for sPAPR VFIO container device
> >>>>>> to support EEH functionality for PCI devices, which have been passed
> >>>>>> through from host to somebody else via VFIO.
> >>>>>>
> >>>>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> >>>>>> ---
> >>>>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
> >>>>>> drivers/vfio/pci/Makefile | 1 +
> >>>>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
> >>>>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
> >>>>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
> >>>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
> >>>>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
> >>>>>> 7 files changed, 308 insertions(+), 7 deletions(-)
> >>>>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
> >>>> [...]
> >>>>
> >>>>>> +
> >>>>>> + return ret;
> >>>>>> +}
> >>>>>> +
> >>>>>> static long tce_iommu_ioctl(void *iommu_data,
> >>>>>> unsigned int cmd, unsigned long arg)
> >>>>>> {
> >>>>>> @@ -283,6 +363,11 @@ static long tce_iommu_ioctl(void *iommu_data,
> >>>>>> tce_iommu_disable(container);
> >>>>>> mutex_unlock(&container->lock);
> >>>>>> return 0;
> >>>>>> + case VFIO_EEH_PE_SET_OPTION:
> >>>>>> + case VFIO_EEH_PE_GET_STATE:
> >>>>>> + case VFIO_EEH_PE_RESET:
> >>>>>> + case VFIO_EEH_PE_CONFIGURE:
> >>>>>> + return tce_iommu_eeh_ioctl(iommu_data, cmd, arg);
> >>>>> This is where it would have really made sense to have a single
> >>>>> VFIO_EEH_OP ioctl with a data structure passed to indicate the sub-op.
> >>>>> AlexG, are you really attached to splitting these out into separate
> >>>>> ioctls?
> >>>> I don't see the problem. We need to forward 4 ioctls to a separate piece
> >>>> of code, so we forward 4 ioctls to a separate piece of code :). Putting
> >>>> them into one ioctl just moves the switch() into another function.
> >>> And uses an extra 3 ioctl numbers and gives us extra things to update if
> >>> we ever need to add more ioctls, etc. ioctl numbers are an address
> >>> space, how much address space do we really want to give to EEH? It's
> >>> not a big difference, but I don't think it's completely even either.
> >>> Thanks,
> >> Yes, that's the point. I by far prefer to have you push back on anyone
> >> who introduces useless ioctls rather than have a separate EEH number
> >> space that people can just throw anything in they like ;).
> > Well, I appreciate that, but having them as separate ioctls doesn't
> > really prevent that either. Any one of these 4 could be set to take a
> > sub-option to extend and contort the EEH interface. The only way to
> > prevent that would be to avoid the argsz+flags hack that make the ioctl
> > extendable. Thanks,
>
> Sure, that's what patch review is about. I'm really more concerned about
> whose court the number space is in - you or Gavin. If we're talking
> about top level ioctls you will care a lot more.
>
> But I'm not religious about this. You're the VFIO maintainer, so it's
> your call. I just personally cringe when I see an ioctl that gets an
> "opcode" and a "parameter" argument where the "parameter" argument is a
> union with one struct for each opcode.
Well, what would it look like...
struct vfio_eeh_pe_op {
__u32 argsz;
__u32 flags;
__u32 op;
};
Couldn't every single one of these be a separate "op"? Are there any
cases where we can't use the ioctl return value?
VFIO_EEH_PE_DISABLE
VFIO_EEH_PE_ENABLE
VFIO_EEH_PE_UNFREEZE_IO
VFIO_EEH_PE_UNFREEZE_DMA
VFIO_EEH_PE_GET_MODE
VFIO_EEH_PE_RESET_DEACTIVATE
VFIO_EEH_PE_RESET_HOT
VFIO_EEH_PE_RESET_FUNDAMENTAL
VFIO_EEH_PE_CONFIGURE
It doesn't look that bad to me, what am I missing? Thanks,
Alex
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alex Williamson @ 2014-05-28 16:32 UTC (permalink / raw)
To: Gavin Shan; +Cc: aik, agraf, kvm-ppc, qiudayu, linuxppc-dev
In-Reply-To: <20140528005532.GA5528@shangw>
On Wed, 2014-05-28 at 10:55 +1000, Gavin Shan wrote:
> On Tue, May 27, 2014 at 12:15:27PM -0600, Alex Williamson wrote:
> >On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
> >> The patch adds new IOCTL commands for sPAPR VFIO container device
> >> to support EEH functionality for PCI devices, which have been passed
> >> through from host to somebody else via VFIO.
> >>
> >> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> >> ---
> >> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
> >> drivers/vfio/pci/Makefile | 1 +
> >> drivers/vfio/pci/vfio_pci.c | 20 +++++---
> >> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
> >> drivers/vfio/pci/vfio_pci_private.h | 5 ++
> >> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
> >> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
> >> 7 files changed, 308 insertions(+), 7 deletions(-)
> >> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
> >>
> >> diff --git a/Documentation/vfio.txt b/Documentation/vfio.txt
> >> index b9ca023..d890fed 100644
> >> --- a/Documentation/vfio.txt
> >> +++ b/Documentation/vfio.txt
> >> @@ -305,7 +305,15 @@ faster, the map/unmap handling has been implemented in real mode which provides
> >> an excellent performance which has limitations such as inability to do
> >> locked pages accounting in real time.
> >>
> >> -So 3 additional ioctls have been added:
> >> +4) According to sPAPR specification, A Partitionable Endpoint (PE) is an I/O
> >> +subtree that can be treated as a unit for the purposes of partitioning and
> >> +error recovery. A PE may be a single or multi-function IOA (IO Adapter), a
> >> +function of a multi-function IOA, or multiple IOAs (possibly including switch
> >> +and bridge structures above the multiple IOAs). PPC64 guests detect PCI errors
> >> +and recover from them via EEH RTAS services, which works on the basis of
> >> +additional ioctl commands.
> >> +
> >> +So 7 additional ioctls have been added:
> >>
> >> VFIO_IOMMU_SPAPR_TCE_GET_INFO - returns the size and the start
> >> of the DMA window on the PCI bus.
> >> @@ -316,6 +324,17 @@ So 3 additional ioctls have been added:
> >>
> >> VFIO_IOMMU_DISABLE - disables the container.
> >>
> >> + VFIO_EEH_PE_SET_OPTION - enables or disables EEH functionality on the
> >> + specified device. Also, it can be used to remove IO or DMA
> >> + stopped state on the frozen PE.
> >> +
> >> + VFIO_EEH_PE_GET_STATE - retrieve PE's state: frozen or normal state.
> >> +
> >> + VFIO_EEH_PE_RESET - do PE reset, which is one of the major steps for
> >> + error recovering.
> >> +
> >> + VFIO_EEH_PE_CONFIGURE - configure the PCI bridges after PE reset. It's
> >> + one of the major steps for error recoverying.
> >>
> >> The code flow from the example above should be slightly changed:
> >>
> >> @@ -346,6 +365,77 @@ The code flow from the example above should be slightly changed:
> >> ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map);
> >> .....
> >>
> >> +Based on the initial example we have, the following piece of code could be
> >> +reference for EEH setup and error handling:
> >> +
> >> + struct vfio_eeh_pe_set_option option = { .argsz = sizeof(option) };
> >> + struct vfio_eeh_pe_get_state state = { .argsz = sizeof(state) };
> >> + struct vfio_eeh_pe_reset reset = { .argsz = sizeof(reset) };
> >> + struct vfio_eeh_pe_configure configure = { .argsz = sizeof(configure) };
> >> +
> >> + ....
> >> +
> >> + /* Get a file descriptor for the device */
> >> + device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, "0000:06:0d.0");
> >> +
> >> + /* Enable the EEH functionality on the device */
> >> + option.option = VFIO_EEH_PE_SET_OPT_ENABLE;
> >> + ioctl(container, VFIO_EEH_PE_SET_OPTION, &option);
> >> +
> >> + /* You're suggested to create additional data struct to represent
> >> + * PE, and put child devices belonging to same IOMMU group to the
> >> + * PE instance for later reference.
> >> + */
> >> +
> >> + /* Check the PE's state and make sure it's in functional state */
> >> + ioctl(container, VFIO_EEH_PE_GET_STATE, &state);
> >> +
> >> + /* Save device's state. pci_save_state() would be good enough
> >> + * as an example.
> >> + */
> >> +
> >> + /* Test and setup the device */
> >> + ioctl(device, VFIO_DEVICE_GET_INFO, &device_info);
> >> +
> >> + ....
> >> +
> >> + /* When 0xFF's returned from reading PCI config space or IO BARs
> >> + * of the PCI device. Check the PE state to see if that has been
> >> + * frozen.
> >> + */
> >> + ioctl(container, VFIO_EEH_PE_GET_STATE, &state);
> >> +
> >> + /* Waiting for pending PCI transactions to be completed and don't
> >> + * produce any more PCI traffic from/to the affected PE until
> >> + * recovery is finished.
> >> + */
> >> +
> >> + /* Enable IO for the affected PE and collect logs. Usually, the
> >> + * standard part of PCI config space, AER registers are dumped
> >> + * as logs for further analysis.
> >> + */
> >> + option.option = VFIO_EEH_PE_SET_OPT_IO;
> >> + ioctl(container, VFIO_EEH_PE_SET_OPTION, &option);
> >> +
> >> + /* Issue PE reset */
> >> + reset.option = VFIO_EEH_PE_RESET_HOT;
> >> + ioctl(container, VFIO_EEH_PE_RESET, &reset);
> >> + reset.option = VFIO_EEH_PE_RESET_DEACTIVATE;
> >> + ioctl(container, VFIO_EEH_PE_RESET, &reset);
> >> +
> >> + /* Configure the PCI bridges for the affected PE */
> >> + ioctl(container, VFIO_EEH_PE_CONFIGURE, &configure);
> >> +
> >> + /* Restored state we saved at initialization time. pci_restore_state()
> >> + * is good enough as an example.
> >> + */
> >> +
> >> + /* Hopefully, error is recovered successfully. Now, you can resume to
> >> + * start PCI traffic to/from the affected PE.
> >> + */
> >> +
> >> + ....
> >> +
> >> -------------------------------------------------------------------------------
> >>
> >> [1] VFIO was originally an acronym for "Virtual Function I/O" in its
> >> diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
> >> index 1310792..faad885 100644
> >> --- a/drivers/vfio/pci/Makefile
> >> +++ b/drivers/vfio/pci/Makefile
> >> @@ -1,4 +1,5 @@
> >>
> >> vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
> >> +vfio-pci-y += vfio_pci_eeh.o
> >
> >Why do we build this w/o CONFIG_EEH? Can't we define the functions as
> >static inline in the header in that case?
> >
>
> Ok. Will do in next revision.
>
> >> obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
> >> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
> >> index 7ba0424..7c8d26a 100644
> >> --- a/drivers/vfio/pci/vfio_pci.c
> >> +++ b/drivers/vfio/pci/vfio_pci.c
> >> @@ -156,8 +156,10 @@ static void vfio_pci_release(void *device_data)
> >> {
> >> struct vfio_pci_device *vdev = device_data;
> >>
> >> - if (atomic_dec_and_test(&vdev->refcnt))
> >> + if (atomic_dec_and_test(&vdev->refcnt)) {
> >> + vfio_pci_eeh_release(vdev->pdev);
> >> vfio_pci_disable(vdev);
> >> + }
> >>
> >> module_put(THIS_MODULE);
> >> }
> >> @@ -165,19 +167,25 @@ static void vfio_pci_release(void *device_data)
> >> static int vfio_pci_open(void *device_data)
> >> {
> >> struct vfio_pci_device *vdev = device_data;
> >> + int ret;
> >>
> >> if (!try_module_get(THIS_MODULE))
> >> return -ENODEV;
> >>
> >> if (atomic_inc_return(&vdev->refcnt) == 1) {
> >> - int ret = vfio_pci_enable(vdev);
> >> - if (ret) {
> >> - module_put(THIS_MODULE);
> >> - return ret;
> >> - }
> >> + ret = vfio_pci_enable(vdev);
> >> + if (ret)
> >> + goto error;
> >> +
> >> + ret = vfio_pci_eeh_open(vdev->pdev);
> >> + if (ret)
> >> + goto error;
> >> }
> >>
> >> return 0;
> >> +error:
> >> + module_put(THIS_MODULE);
> >> + return ret;
> >> }
> >>
> >> static int vfio_pci_get_irq_count(struct vfio_pci_device *vdev, int irq_type)
> >> diff --git a/drivers/vfio/pci/vfio_pci_eeh.c b/drivers/vfio/pci/vfio_pci_eeh.c
> >> new file mode 100644
> >> index 0000000..9c25207
> >> --- /dev/null
> >> +++ b/drivers/vfio/pci/vfio_pci_eeh.c
> >> @@ -0,0 +1,46 @@
> >> +/*
> >> + * EEH functionality support for VFIO PCI devices.
> >> + *
> >> + * Copyright Gavin Shan, IBM Corporation 2014.
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License version 2 as
> >> + * published by the Free Software Foundation.
> >> + */
> >> +
> >> +#include <linux/device.h>
> >> +#include <linux/eventfd.h>
> >> +#include <linux/file.h>
> >> +#include <linux/interrupt.h>
> >> +#include <linux/iommu.h>
> >> +#include <linux/module.h>
> >> +#include <linux/mutex.h>
> >> +#include <linux/notifier.h>
> >> +#include <linux/pci.h>
> >> +#include <linux/pm_runtime.h>
> >> +#include <linux/slab.h>
> >> +#include <linux/types.h>
> >> +#include <linux/uaccess.h>
> >> +#include <linux/vfio.h>
> >
> >Cleanup the includes
> >
>
> Ok. Will do in next revision.
>
> >> +#ifdef CONFIG_EEH
> >> +#include <asm/eeh.h>
> >> +#endif
> >
> >This shouldn't even be compiles w/o CONFIG_EEH
> >
>
> Yeah. Will fix in next revision as you suggested.
>
> >> +
> >> +#include "vfio_pci_private.h"
> >> +
> >> +int vfio_pci_eeh_open(struct pci_dev *pdev)
> >> +{
> >> + int ret = 0;
> >> +
> >> +#ifdef CONFIG_EEH
> >> + ret = eeh_dev_open(pdev);
> >> +#endif
> >> + return ret;
> >> +}
> >> +
> >> +void vfio_pci_eeh_release(struct pci_dev *pdev)
> >> +{
> >> +#ifdef CONFIG_EEH
> >> + eeh_dev_release(pdev);
> >> +#endif
> >> +}
> >> diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h
> >> index 9c6d5d0..c3cbe40 100644
> >> --- a/drivers/vfio/pci/vfio_pci_private.h
> >> +++ b/drivers/vfio/pci/vfio_pci_private.h
> >> @@ -90,4 +90,9 @@ extern void vfio_pci_virqfd_exit(void);
> >>
> >> extern int vfio_config_init(struct vfio_pci_device *vdev);
> >> extern void vfio_config_free(struct vfio_pci_device *vdev);
> >> +
> >> +/* EEH stub */
> >> +extern int vfio_pci_eeh_open(struct pci_dev *pdev);
> >> +extern void vfio_pci_eeh_release(struct pci_dev *pdev);
> >
> >The #ifdef with static inlines should be here.
> >
>
> thanks for the detailed comments.
>
> >> +
> >> #endif /* VFIO_PCI_PRIVATE_H */
> >> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> >> index a84788b..666691b 100644
> >> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> >> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> >> @@ -21,6 +21,9 @@
> >> #include <linux/vfio.h>
> >> #include <asm/iommu.h>
> >> #include <asm/tce.h>
> >> +#ifdef CONFIG_EEH
> >> +#include <asm/eeh.h>
> >> +#endif
> >
> >Now we're infecting another file with EEH, can't we put it in a central
> >location for vfio?
>
> How about something like this:
>
> - Create drivers/vfio/vfio_eeh.c and it will be built when having CONFIG_EEH
> - Declare all EEH functions in include/linux/vfio.h (it's the only central
> header file I can find except you have a better one).
>
> #ifdef CONFIG_EEH
> extern int vfio_pci_eeh_open(struct pci_dev *pdev);
> extern void vfio_pci_eeh_release(struct pci_dev *pdev);
> extern long vfio_iommu_spapr_eeh_ioctl(struct iommu_table *tbl,
> unsigned int cmd, unsigned long arg);
> #else
> static int vfio_pci_eeh_open(struct pci_dev *pdev)
> {
> return 0;
> }
>
> static void vfio_pci_eeh_release(struct pci_dev *pdev)
> {
> }
>
> static inline vfio_iommu_spapr_eeh_ioctl(struct iommu_table *tbl,
> unsigned int cmd, unsigned long arg);
> {
> return -ENOTTY;
> }
> #endif
Seems like an improvement
> >>
> >> #define DRIVER_VERSION "0.1"
> >> #define DRIVER_AUTHOR "aik@ozlabs.ru"
> >> @@ -147,6 +150,83 @@ static void tce_iommu_release(void *iommu_data)
> >> kfree(container);
> >> }
> >>
> >> +static long tce_iommu_eeh_ioctl(void *iommu_data,
> >> + unsigned int cmd, unsigned long arg)
> >> +{
> >> + struct tce_container *container = iommu_data;
> >> + unsigned long minsz;
> >> + int ret = 0;
> >> +
> >> +#ifdef CONFIG_EEH
> >> + switch (cmd) {
> >> + case VFIO_EEH_PE_SET_OPTION: {
> >> + struct vfio_eeh_pe_set_option option;
> >> +
> >> + minsz = offsetofend(struct vfio_eeh_pe_set_option, option);
> >> + if (copy_from_user(&option, (void __user *)arg, minsz))
> >> + return -EFAULT;
> >> + if (option.argsz < minsz)
> >> + return -EINVAL;
> >> +
> >> + ret = eeh_pe_set_option(eeh_iommu_table_to_pe(container->tbl),
> >> + option.option);
> >> + break;
> >> + }
> >> + case VFIO_EEH_PE_GET_STATE: {
> >> + struct vfio_eeh_pe_get_state state;
> >> +
> >> + minsz = offsetofend(struct vfio_eeh_pe_get_state, state);
> >> + if (copy_from_user(&state, (void __user *)arg, minsz))
> >> + return -EFAULT;
> >> + if (state.argsz < minsz)
> >> + return -EINVAL;
> >> +
> >> + ret = eeh_pe_get_state(eeh_iommu_table_to_pe(container->tbl));
> >> + if (ret >= 0) {
> >> + state.state = ret;
> >> + if (copy_to_user((void __user *)arg, &state, minsz))
> >> + return -EFAULT;
> >> + ret = 0;
> >> + }
> >
> >This looks like one of those cases where you should just use the ioctl
> >return value.
> >
>
> Ok. I'll use ioctl return value to carry output information. Note it might
> have "0" as the output information.
I think that's fine.
> >> + break;
> >> + }
> >> + case VFIO_EEH_PE_RESET: {
> >> + struct vfio_eeh_pe_reset reset;
> >> +
> >> + minsz = offsetofend(struct vfio_eeh_pe_reset, option);
> >> + if (copy_from_user(&reset, (void __user *)arg, minsz))
> >> + return -EFAULT;
> >> + if (reset.argsz < minsz)
> >> + return -EINVAL;
> >> +
> >> + ret = eeh_pe_reset(eeh_iommu_table_to_pe(container->tbl),
> >> + reset.option);
> >> + break;
> >> + }
> >> + case VFIO_EEH_PE_CONFIGURE: {
> >> + struct vfio_eeh_pe_configure configure;
> >> +
> >> + minsz = offsetofend(struct vfio_eeh_pe_configure, flags);
> >> + if (copy_from_user(&configure, (void __user *)arg, minsz))
> >> + return -EFAULT;
> >> + if (configure.argsz < minsz)
> >> + return -EINVAL;
> >> +
> >> + ret = eeh_pe_configure(eeh_iommu_table_to_pe(container->tbl));
> >> + break;
> >> + }
> >> + default:
> >> + ret = -EINVAL;
> >> + pr_debug("%s: Cannot handle command %d\n",
> >> + __func__, cmd);
> >> + }
> >> +#else
> >> + ret = -ENOENT;
> >> +#endif
> >
> >Hmm, more like a BUG in the default case the way it's coded here (not
> >even sure it's worth the default entry). The #else case should probably
> >be -ENOTTY like other unimplemented ioctls.
> >
>
> Yeah, we don't need default case here and I'll remove it in next revision.
> Also, "-ENOTTY" will be used in future for unimplemented functions.
>
> >> +
> >> + return ret;
> >> +}
> >> +
> >> static long tce_iommu_ioctl(void *iommu_data,
> >> unsigned int cmd, unsigned long arg)
> >> {
> >> @@ -283,6 +363,11 @@ static long tce_iommu_ioctl(void *iommu_data,
> >> tce_iommu_disable(container);
> >> mutex_unlock(&container->lock);
> >> return 0;
> >> + case VFIO_EEH_PE_SET_OPTION:
> >> + case VFIO_EEH_PE_GET_STATE:
> >> + case VFIO_EEH_PE_RESET:
> >> + case VFIO_EEH_PE_CONFIGURE:
> >> + return tce_iommu_eeh_ioctl(iommu_data, cmd, arg);
> >
> >This is where it would have really made sense to have a single
> >VFIO_EEH_OP ioctl with a data structure passed to indicate the sub-op.
> >AlexG, are you really attached to splitting these out into separate
> >ioctls?
> >
>
> I really don't know. Alex.G want separate ioctl commands, but you
> suggested to combine them. Could you guys please just tell me which
> one (separate vs combined) I need to have in next revision? :-)
>
> >> }
> >>
> >> return -ENOTTY;
> >> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> >> index cb9023d..c5fac36 100644
> >> --- a/include/uapi/linux/vfio.h
> >> +++ b/include/uapi/linux/vfio.h
> >> @@ -455,6 +455,72 @@ struct vfio_iommu_spapr_tce_info {
> >>
> >> #define VFIO_IOMMU_SPAPR_TCE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
> >>
> >> +/*
> >> + * EEH functionality can be enabled or disabled on one specific device.
> >> + * Also, the DMA or IO frozen state can be removed from the frozen PE
> >> + * if required.
> >> + */
> >> +struct vfio_eeh_pe_set_option {
> >> + __u32 argsz;
> >> + __u32 flags;
> >> + __u32 option;
> >> +#define VFIO_EEH_PE_SET_OPT_DISABLE 0 /* Disable EEH */
> >> +#define VFIO_EEH_PE_SET_OPT_ENABLE 1 /* Enable EEH */
> >> +#define VFIO_EEH_PE_SET_OPT_IO 2 /* Enable IO */
> >> +#define VFIO_EEH_PE_SET_OPT_DMA 3 /* Enable DMA */
> >
> >This is more of a "command" than an "option" isn't it? Each of these
> >probably needs a more significant description.
> >
>
> Yeah, it would be regarded as "opcode" and I'll add more description about
> them in next revision.
>
> >> +};
> >> +
> >> +#define VFIO_EEH_PE_SET_OPTION _IO(VFIO_TYPE, VFIO_BASE + 21)
> >> +
> >> +/*
> >> + * Each EEH PE should have unique address to be identified. PE's
> >> + * sharing mode is also useful information as well.
> >> + */
> >> +#define VFIO_EEH_PE_GET_ADDRESS 0 /* Get address */
> >> +#define VFIO_EEH_PE_GET_MODE 1 /* Query mode */
> >> +#define VFIO_EEH_PE_MODE_NONE 0 /* Not a PE */
> >> +#define VFIO_EEH_PE_MODE_NOT_SHARED 1 /* Exclusive */
> >> +#define VFIO_EEH_PE_MODE_SHARED 2 /* Shared mode */
> >> +
> >> +/*
> >> + * EEH PE might have been frozen because of PCI errors. Also, it might
> >> + * be experiencing reset for error revoery. The following command helps
> >> + * to get the state.
> >> + */
> >> +struct vfio_eeh_pe_get_state {
> >> + __u32 argsz;
> >> + __u32 flags;
> >> + __u32 state;
> >> +};
> >
> >Should state be a union to better describe the value returned? What
> >exactly is the address and why does the user need to know it? Does this
> >need user input or could we just return the address and mode regardless?
> >
>
> Ok. I think you want enum (not union) for state. I'll have macros for the
> state in next revision as I did that for other cases.
>
> Those macros defined for "address" just for ABI stuff as Alex.G mentioned.
> There isn't corresponding ioctl command for host to get address any more
> because QEMU (user) will have to figure it out by himself. The "address"
> here means PE address and user has to figure it out according to PE
> segmentation.
>
> >> +
> >> +#define VFIO_EEH_PE_GET_STATE _IO(VFIO_TYPE, VFIO_BASE + 22)
> >> +
> >> +/*
> >> + * Reset is the major step to recover problematic PE. The following
> >> + * command helps on that.
> >> + */
> >> +struct vfio_eeh_pe_reset {
> >> + __u32 argsz;
> >> + __u32 flags;
> >> + __u32 option;
> >> +#define VFIO_EEH_PE_RESET_DEACTIVATE 0 /* Deactivate reset */
> >> +#define VFIO_EEH_PE_RESET_HOT 1 /* Hot reset */
> >> +#define VFIO_EEH_PE_RESET_FUNDAMENTAL 3 /* Fundamental reset */
> >
> >How does a user know which of these to use?
> >
>
> I think Ben already helped explaining it for a lot in the subsequent
> replies. Thanks to Ben :-)
>
> >> +};
> >> +
> >> +#define VFIO_EEH_PE_RESET _IO(VFIO_TYPE, VFIO_BASE + 23)
> >> +
> >> +/*
> >> + * One of the steps for recovery after PE reset is to configure the
> >> + * PCI bridges affected by the PE reset.
> >> + */
> >> +struct vfio_eeh_pe_configure {
> >> + __u32 argsz;
> >> + __u32 flags;
> >> +};
> >
> >Parameters are probably not necessary here.
> >
>
> Yep. I'll remove it in next revision. Thanks for your comments, Alex.
>
> >> +
> >> +#define VFIO_EEH_PE_CONFIGURE _IO(VFIO_TYPE, VFIO_BASE + 24)
> >> +
> >> /* ***************************************************************** */
> >>
> >> #endif /* _UAPIVFIO_H */
> >
> >How does a user learn that a device supports EEH? Do they just start
> >making ioctl calls and getting a failure? Shouldn't we make use of one
> >of the flag bits on the device or add a capability on the container for
> >the user to query?
> >
>
> User needs to make some ioctl calls to make sure EEH can be supported on
> one specific device:
>
> struct vfio_eeh_pe_set_option set_option;
>
> /* User have to make sure the device isn't a bridge and there
> * has one PE for it, which means that case of VFIO_EEH_PE_MODE_NONE.
> */
>
> set_option.argsz = sizeof(set_option);
> set_option.option = VFIO_EEH_PE_SET_OPT_ENABLE;
> ret = ioctl(container, VFIO_EEH_PE_SET_OPTION, &set_option);
> if (ret < 0) {
> /* EEH can't supported */
> }
No, the user should be able to query whether it's available with
VFIO_CHECK_EXTENSION.
I think I'm also still looking for an explanation of how the user learns
about the error that occurred, not just that it occurred, but what it
was. The description of the recovery process indicates a lot retrieval
step, but there's no interface here for the user to get logs. Is that a
TBD? Thanks,
Alex
^ permalink raw reply
* Re: [PATCH] tty/hvc/hvc_console: Fix wakeup of HVC thread on hvc_kick()
From: Greg Kroah-Hartman @ 2014-05-28 20:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev list, Jiri Slaby, Linux Kernel list
In-Reply-To: <1401235290.8262.9.camel@pasglop>
On Wed, May 28, 2014 at 10:01:30AM +1000, Benjamin Herrenschmidt wrote:
> On Fri, 2014-05-23 at 19:41 +1000, Benjamin Herrenschmidt wrote:
> > Some backends call hvc_kick() to wakeup the HVC thread from its
> > slumber upon incoming characters. This however doesn't work
> > properly because it uses msleep_interruptible() which is mostly
> > immune to wake_up_process(). It will basically go back to sleep
> > until the timeout is expired (only signals can really wake it).
> >
> > Replace it with a simple shedule_timeout_interruptible() instead,
> > which may wakeup earlier every now and then but we really don't
> > care in this case.
>
> Nobody commented ? :-)
>
> Greg, do you want to take this in the tty tree or can I stick it in
> powerpc ?
I'll take it, thanks.
greg k-h
^ permalink raw reply
* Re: NUMA topology question wrt. d4edc5b6
From: Nishanth Aravamudan @ 2014-05-28 20:37 UTC (permalink / raw)
To: Srivatsa S. Bhat
Cc: Dave Hansen, Srikar Dronamraju,
linuxppc-dev@lists.ozlabs.org list, Linux MM, Aneesh Kumar K.V,
nfont, Cody P Schafer, Anton Blanchard
In-Reply-To: <537E6285.3050000@linux.vnet.ibm.com>
On 23.05.2014 [02:18:05 +0530], Srivatsa S. Bhat wrote:
>
> [ Adding a few more CC's ]
>
> On 05/22/2014 01:34 AM, Nishanth Aravamudan wrote:
> > Hi Srivatsa,
> >
> > After d4edc5b6 ("powerpc: Fix the setup of CPU-to-Node mappings during
> > CPU online"), cpu_to_node() looks like:
> >
> > static inline int cpu_to_node(int cpu)
> > {
> > int nid;
> >
> > nid = numa_cpu_lookup_table[cpu];
> >
> > /*
> > * During early boot, the numa-cpu lookup table might not have been
> > * setup for all CPUs yet. In such cases, default to node 0.
> > */
> > return (nid < 0) ? 0 : nid;
> > }
> >
> > However, I'm curious if this is correct in all cases. I have seen
> > several LPARs that do not have any CPUs on node 0. In fact, because node
> > 0 is statically set online in the initialization of the N_ONLINE
> > nodemask, 0 is always present to Linux, whether it is present on the
> > system. I'm not sure what the best thing to do here is, but I'm curious
> > if you have any ideas? I would like to remove the static initialization
> > of node 0, as it's confusing to users to see an empty node (particularly
> > when it's completely separate in the numbering from other nodes), but
> > we trip a panic (refer to:
> > http://www.spinics.net/lists/linux-mm/msg73321.html).
> >
>
> Ah, I see. I didn't have any particular reason to default it to zero.
> I just did that because the existing code before this patch did the same
> thing. (numa_cpu_lookup_table[] is a global array, so it will be initialized
> with zeros. So if we access it before populating it via numa_setup_cpu(),
> it would return 0. So I retained that behaviour with the above conditional).
Ok, that seems reasonable to me (keeping the behavior the same as it was
before).
> Will something like the below [totally untested] patch solve the boot-panic?
> I understand that as of today first_online_node will still pick 0 since
> N_ONLINE is initialized statically, but with your proposed change to that
> init code, I guess the following patch should avoid the boot panic.
>
> [ But note that first_online_node is hard-coded to 0, if MAX_NUMNODES is = 1.
> So we'll have to fix that if powerpc can have a single node system whose node
> is numbered something other than 0. Can that happen as well? ]
I think all single-node systems are only Node 0, but I'm not 100% on
that.
> And regarding your question about what is the best way to fix this
> whole Linux MM's assumption about node0, I'm not really sure.. since I
> am not really aware of the extent to which the MM subsystem is
> intertwined with this assumption and what it would take to cure that
> :-(
Well, at this point, it might be fine to just leave it alone, as it
seems to be more trouble than it's worth -- and really the only
confusion is on those LPARs where there really isn't a Node 0. I'll take
another look later this week.
Thanks,
Nish
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Benjamin Herrenschmidt @ 2014-05-28 21:58 UTC (permalink / raw)
To: Gavin Shan
Cc: aik, Alexander Graf, kvm-ppc, Alex Williamson, qiudayu,
linuxppc-dev
In-Reply-To: <20140528124947.GA19346@shangw>
On Wed, 2014-05-28 at 22:49 +1000, Gavin Shan wrote:
>
> I will remove those "address" related macros in next revision because it's
> user-level bussiness, not related to host kernel any more.
>
> If the user is QEMU + guest, we need the address to identify the PE though PHB
> BUID could be used as same purpose to get PHB, which is one-to-one mapping with
> IOMMU group on sPAPR platform. However, once the PE address is built and returned
> to guest, guest will use the PE address as input parameter in subsequent RTAS
> calls.
>
> If the user is some kind of application who just uses the ioctl() without supporting
> RTAS calls. We don't need care about PE address.
I am a bit reluctant with that PE==PHB equation we seem to be introducing.
This isn't the case in HW. It's possible that this is how we handle VFIO *today*
in qemu but it doesn't have to be does it ?
It also paints us into a corner if we want to start implementing some kind of
emulated EEH for selected emulated devices and/or virtio.
Cheers,
Ben.
^ permalink raw reply
* Re: [PATCH 11/16] byteorder: provide a linux/byteorder.h with {be, le}_to_cpu() and cpu_to_{be, le}() macros
From: Cody P Schafer @ 2014-05-28 22:05 UTC (permalink / raw)
To: David Laight; +Cc: Sukadev Bhattiprolu, Linux PPC, LKML
In-Reply-To: <063D6719AE5E284EB5DD2968C1650D6D1724EEAF@AcuExch.aculab.com>
On Wed, May 28, 2014 at 3:45 AM, David Laight <David.Laight@aculab.com> wrote:
> From: Cody P Schafer
>> Rather manually specifying the size of the integer to be converted, key
>> off of the type size. Reduces duplicate size info and the occurance of
>> certain types of bugs (using the wrong sized conversion).
> ...
>> +#define be_to_cpu(v) \
>> + __builtin_choose_expr(sizeof(v) == sizeof(uint8_t) , v, \
>> + __builtin_choose_expr(sizeof(v) == sizeof(uint16_t), be16_to_cpu(v), \
>> + __builtin_choose_expr(sizeof(v) == sizeof(uint32_t), be32_to_cpu(v), \
>> + __builtin_choose_expr(sizeof(v) == sizeof(uint64_t), be64_to_cpu(v), \
>> + (void)0))))
> ...
>
> I'm not at all sure that using the 'size' of the constant will reduce
> the number of bugs - it just introduces a whole new category of bugs.
Certainly, if you mis-size the argument (and thus have missized one of
the variables containing the be value, probably a bug anyhow), there
will be problems.
I put this interface together because of an actual bug I wrote into
the initial code of the hv_24x7 driver (resized a struct member
without adjusting the be*_to_cpu() sizing).
Having this "auto sizing" macro means I can avoid encoding the size of
a struct field in multiple places.
^ permalink raw reply
* Re: [PATCH 11/16] byteorder: provide a linux/byteorder.h with {be,le}_to_cpu() and cpu_to_{be,le}() macros
From: Cody P Schafer @ 2014-05-28 22:07 UTC (permalink / raw)
To: Joe Perches; +Cc: Sukadev Bhattiprolu, Linux PPC, LKML
In-Reply-To: <1401237869.20587.12.camel@joe-AO725>
On Tue, May 27, 2014 at 7:44 PM, Joe Perches <joe@perches.com> wrote:
> On Tue, 2014-05-27 at 17:22 -0700, Cody P Schafer wrote:
>> Rather manually specifying the size of the integer to be converted, key
>> off of the type size. Reduces duplicate size info and the occurance of
>> certain types of bugs (using the wrong sized conversion).
> []
>> diff --git a/include/linux/byteorder.h b/include/linux/byteorder.h
> []
>> @@ -0,0 +1,34 @@
>> +#ifndef LINUX_BYTEORDER_H_
>> +#define LINUX_BYTEORDER_H_
>> +
>> +#include <asm/byteorder.h>
>> +
>> +#define be_to_cpu(v) \
>> + __builtin_choose_expr(sizeof(v) =3D=3D sizeof(uint8_t) , v, \
>> + __builtin_choose_expr(sizeof(v) =3D=3D sizeof(uint16_t), be16_to_c=
pu(v), \
>> + __builtin_choose_expr(sizeof(v) =3D=3D sizeof(uint32_t), be32_to_c=
pu(v), \
>> + __builtin_choose_expr(sizeof(v) =3D=3D sizeof(uint64_t), be64_to_c=
pu(v), \
>> + (void)0))))
>
> probably better to use BUILD_BUG instead of these 0 returns
>
They aren't 0 returns.
$ echo "int main(void) { int x =3D (void)0; return x; }" | gcc -x c -
<stdin>: In function =E2=80=98main=E2=80=99:
<stdin>:1:26: error: void value not ignored as it ought to be
^ permalink raw reply
* Re: [PATCH 11/16] byteorder: provide a linux/byteorder.h with {be, le}_to_cpu() and cpu_to_{be, le}() macros
From: Cody P Schafer @ 2014-05-28 22:11 UTC (permalink / raw)
To: David Laight; +Cc: Sukadev Bhattiprolu, Linux PPC, LKML
In-Reply-To: <CAPoQQ-17OHotfbczJvEoq=vAzu_+6-sV3M1ZAhDaQ45wB3NB4A@mail.gmail.com>
On Wed, May 28, 2014 at 5:05 PM, Cody P Schafer <dev@codyps.com> wrote:
> On Wed, May 28, 2014 at 3:45 AM, David Laight <David.Laight@aculab.com> wrote:
>> From: Cody P Schafer
>>> Rather manually specifying the size of the integer to be converted, key
>>> off of the type size. Reduces duplicate size info and the occurance of
>>> certain types of bugs (using the wrong sized conversion).
>> ...
>>> +#define be_to_cpu(v) \
>>> + __builtin_choose_expr(sizeof(v) == sizeof(uint8_t) , v, \
>>> + __builtin_choose_expr(sizeof(v) == sizeof(uint16_t), be16_to_cpu(v), \
>>> + __builtin_choose_expr(sizeof(v) == sizeof(uint32_t), be32_to_cpu(v), \
>>> + __builtin_choose_expr(sizeof(v) == sizeof(uint64_t), be64_to_cpu(v), \
>>> + (void)0))))
>> ...
>>
>> I'm not at all sure that using the 'size' of the constant will reduce
>> the number of bugs - it just introduces a whole new category of bugs.
>
> Certainly, if you mis-size the argument (and thus have missized one of
> the variables containing the be value, probably a bug anyhow), there
> will be problems.
>
> I put this interface together because of an actual bug I wrote into
> the initial code of the hv_24x7 driver (resized a struct member
> without adjusting the be*_to_cpu() sizing).
> Having this "auto sizing" macro means I can avoid encoding the size of
> a struct field in multiple places.
To clarify, the point I'm making here is that this simply cuts out 1
more place we can screw up endianness conversion sizing.
^ permalink raw reply
* [PATCH] powerpc/perf: Never program book3s PMCs with values >= 0x80000000
From: Anton Blanchard @ 2014-05-28 22:15 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, mpe; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]
We are seeing a lot of PMU warnings on POWER8:
Can't find PMC that caused IRQ
Looking closer, the active PMC is 0 at this point and we took a PMU
exception on the transition from negative to 0. Some versions of POWER8
have an issue where they edge detect and not level detect PMC overflows.
A number of places program the PMC with (0x80000000 - period_left),
where period_left can be negative. We can either fix all of these or
just ensure that period_left is always >= 1.
This patch takes the second option.
Cc: <stable@vger.kernel.org>
Signed-off-by: Anton Blanchard <anton@samba.org>
---
Attached is a graph that shows how this bug also causes problems in the
initial period calculation for samples. Once fixed we converge on
the correct period very quickly.
Index: b/arch/powerpc/perf/core-book3s.c
===================================================================
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -996,7 +996,22 @@ static void power_pmu_read(struct perf_e
} while (local64_cmpxchg(&event->hw.prev_count, prev, val) != prev);
local64_add(delta, &event->count);
- local64_sub(delta, &event->hw.period_left);
+
+ /*
+ * A number of places program the PMC with (0x80000000 - period_left).
+ * We never want period_left to be less than 1 because we will program
+ * the PMC with a value >= 0x800000000 and an edge detected PMC will
+ * roll around to 0 before taking an exception. We have seen this
+ * on POWER8.
+ *
+ * To fix this, clamp the minimum value of period_left to 1.
+ */
+ do {
+ prev = local64_read(&event->hw.period_left);
+ val = prev - delta;
+ if (val < 1)
+ val = 1;
+ } while (local64_cmpxchg(&event->hw.period_left, prev, val) != prev);
}
/*
[-- Attachment #2: perf_record_sampling_period.png --]
[-- Type: image/png, Size: 30902 bytes --]
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alexander Graf @ 2014-05-28 22:40 UTC (permalink / raw)
To: Alex Williamson; +Cc: aik, Gavin Shan, kvm-ppc, qiudayu, linuxppc-dev
In-Reply-To: <1401293853.2412.36.camel@ul30vt.home>
On 28.05.14 18:17, Alex Williamson wrote:
> On Wed, 2014-05-28 at 13:37 +0200, Alexander Graf wrote:
>> On 28.05.14 02:57, Alex Williamson wrote:
>>> On Wed, 2014-05-28 at 02:44 +0200, Alexander Graf wrote:
>>>> On 28.05.14 02:39, Alex Williamson wrote:
>>>>> On Wed, 2014-05-28 at 00:49 +0200, Alexander Graf wrote:
>>>>>> On 27.05.14 20:15, Alex Williamson wrote:
>>>>>>> On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
>>>>>>>> The patch adds new IOCTL commands for sPAPR VFIO container device
>>>>>>>> to support EEH functionality for PCI devices, which have been passed
>>>>>>>> through from host to somebody else via VFIO.
>>>>>>>>
>>>>>>>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>>>>>>> ---
>>>>>>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
>>>>>>>> drivers/vfio/pci/Makefile | 1 +
>>>>>>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
>>>>>>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
>>>>>>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
>>>>>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
>>>>>>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
>>>>>>>> 7 files changed, 308 insertions(+), 7 deletions(-)
>>>>>>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
>>>>>> [...]
>>>>>>
>>>>>>>> +
>>>>>>>> + return ret;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> static long tce_iommu_ioctl(void *iommu_data,
>>>>>>>> unsigned int cmd, unsigned long arg)
>>>>>>>> {
>>>>>>>> @@ -283,6 +363,11 @@ static long tce_iommu_ioctl(void *iommu_data,
>>>>>>>> tce_iommu_disable(container);
>>>>>>>> mutex_unlock(&container->lock);
>>>>>>>> return 0;
>>>>>>>> + case VFIO_EEH_PE_SET_OPTION:
>>>>>>>> + case VFIO_EEH_PE_GET_STATE:
>>>>>>>> + case VFIO_EEH_PE_RESET:
>>>>>>>> + case VFIO_EEH_PE_CONFIGURE:
>>>>>>>> + return tce_iommu_eeh_ioctl(iommu_data, cmd, arg);
>>>>>>> This is where it would have really made sense to have a single
>>>>>>> VFIO_EEH_OP ioctl with a data structure passed to indicate the sub-op.
>>>>>>> AlexG, are you really attached to splitting these out into separate
>>>>>>> ioctls?
>>>>>> I don't see the problem. We need to forward 4 ioctls to a separate piece
>>>>>> of code, so we forward 4 ioctls to a separate piece of code :). Putting
>>>>>> them into one ioctl just moves the switch() into another function.
>>>>> And uses an extra 3 ioctl numbers and gives us extra things to update if
>>>>> we ever need to add more ioctls, etc. ioctl numbers are an address
>>>>> space, how much address space do we really want to give to EEH? It's
>>>>> not a big difference, but I don't think it's completely even either.
>>>>> Thanks,
>>>> Yes, that's the point. I by far prefer to have you push back on anyone
>>>> who introduces useless ioctls rather than have a separate EEH number
>>>> space that people can just throw anything in they like ;).
>>> Well, I appreciate that, but having them as separate ioctls doesn't
>>> really prevent that either. Any one of these 4 could be set to take a
>>> sub-option to extend and contort the EEH interface. The only way to
>>> prevent that would be to avoid the argsz+flags hack that make the ioctl
>>> extendable. Thanks,
>> Sure, that's what patch review is about. I'm really more concerned about
>> whose court the number space is in - you or Gavin. If we're talking
>> about top level ioctls you will care a lot more.
>>
>> But I'm not religious about this. You're the VFIO maintainer, so it's
>> your call. I just personally cringe when I see an ioctl that gets an
>> "opcode" and a "parameter" argument where the "parameter" argument is a
>> union with one struct for each opcode.
> Well, what would it look like...
>
> struct vfio_eeh_pe_op {
> __u32 argsz;
> __u32 flags;
> __u32 op;
> };
>
> Couldn't every single one of these be a separate "op"? Are there any
> cases where we can't use the ioctl return value?
>
> VFIO_EEH_PE_DISABLE
> VFIO_EEH_PE_ENABLE
> VFIO_EEH_PE_UNFREEZE_IO
> VFIO_EEH_PE_UNFREEZE_DMA
> VFIO_EEH_PE_GET_MODE
> VFIO_EEH_PE_RESET_DEACTIVATE
> VFIO_EEH_PE_RESET_HOT
> VFIO_EEH_PE_RESET_FUNDAMENTAL
> VFIO_EEH_PE_CONFIGURE
>
> It doesn't look that bad to me, what am I missing? Thanks,
Yup, that looks well to me as well :)
Alex
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Alexander Graf @ 2014-05-28 22:46 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Gavin Shan
Cc: aik, qiudayu, Alex Williamson, linuxppc-dev, kvm-ppc
In-Reply-To: <1401314311.8262.26.camel@pasglop>
On 28.05.14 23:58, Benjamin Herrenschmidt wrote:
> On Wed, 2014-05-28 at 22:49 +1000, Gavin Shan wrote:
>> I will remove those "address" related macros in next revision because it's
>> user-level bussiness, not related to host kernel any more.
>>
>> If the user is QEMU + guest, we need the address to identify the PE though PHB
>> BUID could be used as same purpose to get PHB, which is one-to-one mapping with
>> IOMMU group on sPAPR platform. However, once the PE address is built and returned
>> to guest, guest will use the PE address as input parameter in subsequent RTAS
>> calls.
>>
>> If the user is some kind of application who just uses the ioctl() without supporting
>> RTAS calls. We don't need care about PE address.
> I am a bit reluctant with that PE==PHB equation we seem to be introducing.
>
> This isn't the case in HW. It's possible that this is how we handle VFIO *today*
> in qemu but it doesn't have to be does it ?
Right, but that's pure QEMU internals. From the VFIO kernel interface's
point of view, a VFIO group is a PE context, as that's what gets IOMMU
controlled together too.
> It also paints us into a corner if we want to start implementing some kind of
> emulated EEH for selected emulated devices and/or virtio.
I don't think so :). In QEMU the PHB emulation would have to notify the
"container" (IOMMU emulation layer -> PE) that a PE operation happened.
It's that emulation code's responsibility to broadcast operations across
its own emulated operations (recover config space access, reconfigure
BARs, etc) and the VFIO PE operations.
So from a kernel interface point of view, I think leaving out any
address information is the right way to go. Whether we managed to get
all QEMU internal interfaces modeled correctly yet has to be seen on the
next patch set revision :).
Alex
^ permalink raw reply
* Re: [PATCH 11/16] byteorder: provide a linux/byteorder.h with {be, le}_to_cpu() and cpu_to_{be, le}() macros
From: Joe Perches @ 2014-05-28 23:00 UTC (permalink / raw)
To: Cody P Schafer; +Cc: Sukadev Bhattiprolu, David Laight, Linux PPC, LKML
In-Reply-To: <CAPoQQ-11vW3Q0=LcS-myNZ6hXeYBeu==_prwHJXHvLfkUSmZHQ@mail.gmail.com>
On Wed, 2014-05-28 at 17:11 -0500, Cody P Schafer wrote:
> On Wed, May 28, 2014 at 5:05 PM, Cody P Schafer <dev@codyps.com> wrote:
> > On Wed, May 28, 2014 at 3:45 AM, David Laight <David.Laight@aculab.com> wrote:
> >> From: Cody P Schafer
> >>> Rather manually specifying the size of the integer to be converted, key
> >>> off of the type size. Reduces duplicate size info and the occurance of
> >>> certain types of bugs (using the wrong sized conversion).
> >> ...
> >>> +#define be_to_cpu(v) \
> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint8_t) , v, \
> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint16_t), be16_to_cpu(v), \
> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint32_t), be32_to_cpu(v), \
> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint64_t), be64_to_cpu(v), \
> >>> + (void)0))))
> >> ...
> >>
> >> I'm not at all sure that using the 'size' of the constant will reduce
> >> the number of bugs - it just introduces a whole new category of bugs.
> >
> > Certainly, if you mis-size the argument (and thus have missized one of
> > the variables containing the be value, probably a bug anyhow), there
> > will be problems.
> >
> > I put this interface together because of an actual bug I wrote into
> > the initial code of the hv_24x7 driver (resized a struct member
> > without adjusting the be*_to_cpu() sizing).
> > Having this "auto sizing" macro means I can avoid encoding the size of
> > a struct field in multiple places.
>
> To clarify, the point I'm making here is that this simply cuts out 1
> more place we can screw up endianness conversion sizing.
It does screw up other types when you do things like:
u8 foo = some_function();
cpu_to_be(foo + 1);
the return value is sizeof(int) not u8
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Gavin Shan @ 2014-05-28 23:13 UTC (permalink / raw)
To: Alexander Graf
Cc: aik, Gavin Shan, kvm-ppc, Alex Williamson, qiudayu, linuxppc-dev
In-Reply-To: <5385E0C3.8020000@suse.de>
On Wed, May 28, 2014 at 03:12:35PM +0200, Alexander Graf wrote:
>
>On 28.05.14 14:49, Gavin Shan wrote:
>>On Wed, May 28, 2014 at 01:41:35PM +0200, Alexander Graf wrote:
>>>On 28.05.14 02:55, Gavin Shan wrote:
>>>>On Tue, May 27, 2014 at 12:15:27PM -0600, Alex Williamson wrote:
>>>>>On Tue, 2014-05-27 at 18:40 +1000, Gavin Shan wrote:
>>>>>>The patch adds new IOCTL commands for sPAPR VFIO container device
>>>>>>to support EEH functionality for PCI devices, which have been passed
>>>>>>through from host to somebody else via VFIO.
>>>>>>
>>>>>>Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>>>>>>---
>>>>>> Documentation/vfio.txt | 92 ++++++++++++++++++++++++++++++++++++-
>>>>>> drivers/vfio/pci/Makefile | 1 +
>>>>>> drivers/vfio/pci/vfio_pci.c | 20 +++++---
>>>>>> drivers/vfio/pci/vfio_pci_eeh.c | 46 +++++++++++++++++++
>>>>>> drivers/vfio/pci/vfio_pci_private.h | 5 ++
>>>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 85 ++++++++++++++++++++++++++++++++++
>>>>>> include/uapi/linux/vfio.h | 66 ++++++++++++++++++++++++++
>>>>>> 7 files changed, 308 insertions(+), 7 deletions(-)
>>>>>> create mode 100644 drivers/vfio/pci/vfio_pci_eeh.c
>>>[...]
>>>
>>>>>>diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>>>>>>index cb9023d..c5fac36 100644
>>>>>>--- a/include/uapi/linux/vfio.h
>>>>>>+++ b/include/uapi/linux/vfio.h
>>>>>>@@ -455,6 +455,72 @@ struct vfio_iommu_spapr_tce_info {
>>>>>> #define VFIO_IOMMU_SPAPR_TCE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>>>>>+/*
>>>>>>+ * EEH functionality can be enabled or disabled on one specific device.
>>>>>>+ * Also, the DMA or IO frozen state can be removed from the frozen PE
>>>>>>+ * if required.
>>>>>>+ */
>>>>>>+struct vfio_eeh_pe_set_option {
>>>>>>+ __u32 argsz;
>>>>>>+ __u32 flags;
>>>>>>+ __u32 option;
>>>>>>+#define VFIO_EEH_PE_SET_OPT_DISABLE 0 /* Disable EEH */
>>>>>>+#define VFIO_EEH_PE_SET_OPT_ENABLE 1 /* Enable EEH */
>>>>>>+#define VFIO_EEH_PE_SET_OPT_IO 2 /* Enable IO */
>>>>>>+#define VFIO_EEH_PE_SET_OPT_DMA 3 /* Enable DMA */
>>>>>This is more of a "command" than an "option" isn't it? Each of these
>>>>>probably needs a more significant description.
>>>>>
>>>>Yeah, it would be regarded as "opcode" and I'll add more description about
>>>>them in next revision.
>>>Please just call them commands.
>>>
>>Ok. I guess you want me to change the macro names like this ?
>>
>>#define VFIO_EEH_CMD_DISABLE 0 /* Disable EEH functionality */
>>#define VFIO_EEH_CMD_ENABLE 1 /* Enable EEH functionality */
>>#define VFIO_EEH_CMD_ENABLE_IO 2 /* Enable IO for frozen PE */
>>#define VFIO_EEH_CMD_ENABLE_DMA 3 /* Enable DMA for frozen PE */
>
>Yes, the ioctl name too.
>
Ok. Thanks. I will also to rename those "option" / "command" related macros
to "VFIO_EEH_CMD_xxxx" in next revision.
>>
>>>>>>+};
>>>>>>+
>>>>>>+#define VFIO_EEH_PE_SET_OPTION _IO(VFIO_TYPE, VFIO_BASE + 21)
>>>>>>+
>>>>>>+/*
>>>>>>+ * Each EEH PE should have unique address to be identified. PE's
>>>>>>+ * sharing mode is also useful information as well.
>>>>>>+ */
>>>>>>+#define VFIO_EEH_PE_GET_ADDRESS 0 /* Get address */
>>>>>>+#define VFIO_EEH_PE_GET_MODE 1 /* Query mode */
>>>>>>+#define VFIO_EEH_PE_MODE_NONE 0 /* Not a PE */
>>>>>>+#define VFIO_EEH_PE_MODE_NOT_SHARED 1 /* Exclusive */
>>>>>>+#define VFIO_EEH_PE_MODE_SHARED 2 /* Shared mode */
>>>>>>+
>>>>>>+/*
>>>>>>+ * EEH PE might have been frozen because of PCI errors. Also, it might
>>>>>>+ * be experiencing reset for error revoery. The following command helps
>>>>>>+ * to get the state.
>>>>>>+ */
>>>>>>+struct vfio_eeh_pe_get_state {
>>>>>>+ __u32 argsz;
>>>>>>+ __u32 flags;
>>>>>>+ __u32 state;
>>>>>>+};
>>>>>Should state be a union to better describe the value returned? What
>>>>>exactly is the address and why does the user need to know it? Does this
>>>>>need user input or could we just return the address and mode regardless?
>>>>>
>>>>Ok. I think you want enum (not union) for state. I'll have macros for the
>>>>state in next revision as I did that for other cases.
>>>>
>>>>Those macros defined for "address" just for ABI stuff as Alex.G mentioned.
>>>>There isn't corresponding ioctl command for host to get address any more
>>>>because QEMU (user) will have to figure it out by himself. The "address"
>>>>here means PE address and user has to figure it out according to PE
>>>>segmentation.
>>>Why would the user ever need the address?
>>>
>>I will remove those "address" related macros in next revision because it's
>>user-level bussiness, not related to host kernel any more.
>
>Ok, so the next question is whether there will be any state outside
>of GET_MODE left in the future.
>
That's also user-level business and those macros should be removed as well :-)
Thanks,
Gavin
^ permalink raw reply
* Re: [PATCH v7 3/3] drivers/vfio: EEH support for VFIO PCI device
From: Benjamin Herrenschmidt @ 2014-05-28 23:18 UTC (permalink / raw)
To: Alexander Graf
Cc: aik, Gavin Shan, kvm-ppc, Alex Williamson, qiudayu, linuxppc-dev
In-Reply-To: <53866747.50703@suse.de>
On Thu, 2014-05-29 at 00:46 +0200, Alexander Graf wrote:
> I don't think so :). In QEMU the PHB emulation would have to notify the
> "container" (IOMMU emulation layer -> PE) that a PE operation happened.
> It's that emulation code's responsibility to broadcast operations across
> its own emulated operations (recover config space access, reconfigure
> BARs, etc) and the VFIO PE operations.
>
> So from a kernel interface point of view, I think leaving out any
> address information is the right way to go. Whether we managed to get
> all QEMU internal interfaces modeled correctly yet has to be seen on the
> next patch set revision :).
For a kernel interface I absolutely agree, all we care about is the group fd.
>From a qemu internal, I suppose it makes things easier to operate at the PHB
level for now and we can revisit later if the need arises.
Cheers,
Ben.
^ permalink raw reply
* Re: [PATCH 11/16] byteorder: provide a linux/byteorder.h with {be, le}_to_cpu() and cpu_to_{be, le}() macros
From: Cody P Schafer @ 2014-05-28 23:26 UTC (permalink / raw)
To: Joe Perches; +Cc: Sukadev Bhattiprolu, David Laight, Linux PPC, LKML
In-Reply-To: <1401318025.19335.15.camel@joe-AO725>
On Wed, May 28, 2014 at 6:00 PM, Joe Perches <joe@perches.com> wrote:
> On Wed, 2014-05-28 at 17:11 -0500, Cody P Schafer wrote:
>> On Wed, May 28, 2014 at 5:05 PM, Cody P Schafer <dev@codyps.com> wrote:
>> > On Wed, May 28, 2014 at 3:45 AM, David Laight <David.Laight@aculab.com> wrote:
>> >> From: Cody P Schafer
>> >>> Rather manually specifying the size of the integer to be converted, key
>> >>> off of the type size. Reduces duplicate size info and the occurance of
>> >>> certain types of bugs (using the wrong sized conversion).
>> >> ...
>> >>> +#define be_to_cpu(v) \
>> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint8_t) , v, \
>> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint16_t), be16_to_cpu(v), \
>> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint32_t), be32_to_cpu(v), \
>> >>> + __builtin_choose_expr(sizeof(v) == sizeof(uint64_t), be64_to_cpu(v), \
>> >>> + (void)0))))
>> >> ...
>> >>
>> >> I'm not at all sure that using the 'size' of the constant will reduce
>> >> the number of bugs - it just introduces a whole new category of bugs.
>> >
>> > Certainly, if you mis-size the argument (and thus have missized one of
>> > the variables containing the be value, probably a bug anyhow), there
>> > will be problems.
>> >
>> > I put this interface together because of an actual bug I wrote into
>> > the initial code of the hv_24x7 driver (resized a struct member
>> > without adjusting the be*_to_cpu() sizing).
>> > Having this "auto sizing" macro means I can avoid encoding the size of
>> > a struct field in multiple places.
>>
>> To clarify, the point I'm making here is that this simply cuts out 1
>> more place we can screw up endianness conversion sizing.
>
> It does screw up other types when you do things like:
>
> u8 foo = some_function();
>
> cpu_to_be(foo + 1);
>
> the return value is sizeof(int) not u8
Yep, that is a very good argument against the cpu_to_{be,le}()
variants. It might make sense to remove them and just have the
{be,le}_to_cpu() ones.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox