* Re: KCIDB database [not found] ` <DB9PR08MB68103176F5638C352F410BCBC4402@DB9PR08MB6810.eurprd08.prod.outlook.com> @ 2024-10-18 13:32 ` Nikolai Kondrashov [not found] ` <DB9PR08MB68104499218B100804567075C4402@DB9PR08MB6810.eurprd08.prod.outlook.com> 0 siblings, 1 reply; 5+ messages in thread From: Nikolai Kondrashov @ 2024-10-18 13:32 UTC (permalink / raw) To: Manoj Kumar; +Cc: kernelci@lists.linux.dev, Denys Fedoryshchenko Hi Manoj, > I looked at the below link to get installation steps for the kcidb-submit command. > > https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> > > This instruction is for Python version V3.9 but our backend is using the > Python 2.7.16 at the moment. > > Do you have instruction for python 2.7.16 to install this package KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python 2) is invoking the kcidb-submit command-line tool in a new process instead of using the KCIDB library directly. You will need to install Python 3 in your deployment, I assume. Perhaps Denys, or someone else on the KernelCI maillist (both CC'd) could tell you how we did that. Nick On 10/18/24 4:21 PM, Manoj Kumar wrote: > Hi Nikolai, > > > > I am planning to push test build data to public KCIDB . I am seeing a failure > because of the kcidb-submit command in a thread at backend. > > > > The class KcidbSubmit expect the command kcidb-submit installed on the backend > system. > > /class KcidbSubmit(object):/ > > / def __init__(self, kcidb_options):/ > > / kcidb_path = kcidb_options.get("kcidb_path", "")/ > > / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/ > > > > I looked at the below link to get installation steps for the kcidb-submit command. > > https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> > > > > This instruction is for Python version V3.9 but our backend is using the > Python 2.7.16 at the moment. > > Do you have instruction for python 2.7.16 to install this package > > > > Br, > > Manoj > > > > *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> > *Date: *Tuesday, 8 October 2024 at 11:46 > *To: *Manoj Kumar <ManojKumar08@arm.com> > *Subject: *Re: KCIDB database > > Hi Manoj, > > On 10/8/24 1:26 PM, Manoj Kumar wrote: >> One additional question regarding the credentials file. >> >> Attached credentials files was shared with me named “.kernelci-production-ci- >> arm.json” . >> >> The attached file contains multiple parameters , If I understood correctly all >> the contents of this file should be assigned to this variable https:// >> github.com/kernelci/kernelci-backend/ >> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >> github.com/kernelci/kernelci-backend/ >> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> > > Not the contents, but a path to the file itself. > >> Also , the email ID used by internal kernelCI deployment is different . They >> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>> >> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>”. > > That doesn't matter, the email in the JSON credentials is simply a user > (service account) ID and is specific to Google Cloud. It identifies a > particular submitter to the system. > > You don't really need to look into the credentials file at all. > > Nick > > On 10/8/24 1:26 PM, Manoj Kumar wrote: >> Hi Nick, >> >> >> >> Thanks for your prompt responses :thumbsup >> >> >> >> One additional question regarding the credentials file. >> >> Attached credentials files was shared with me named “.kernelci-production-ci- >> arm.json” . >> >> The attached file contains multiple parameters , If I understood correctly all >> the contents of this file should be assigned to this variable https:// >> github.com/kernelci/kernelci-backend/ >> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >> github.com/kernelci/kernelci-backend/ >> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >> >> >> >> Also , the email ID used by internal kernelCI deployment is different . They >> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>> >> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>”. >> >> >> >> Br, >> >> Manoj >> >> >> >> >> >> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >> *Date: *Tuesday, 8 October 2024 at 05:36 >> *To: *Manoj Kumar <ManojKumar08@arm.com> >> *Subject: *Re: KCIDB database >> >> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>> I do have some questions if you can help me find answer. >>> >>> * Do I need to use the private key from the credential json for this >>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>> <https://github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >> >> Yes, that needs to be pointing to the credentials JSON file. >> >>> * As the storage server is internal , I guess we need to open access to >>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>> files so the kcidb server can access them . Can you confirm this , please >> >> Yes, KCIDB right now will try to download every 256th URL you submit, that is >> under 5MB. The artifact caching system is in trial mode still. However, what's >> more important, *people* will try to open the URLs you submit. >> >> Opening the access to the whole server might be problematic for you guys, as >> you want to keep some results secret, IIRC. Maybe you could copy them to some >> other web server and translate the URLs as you submit them? >> >> Nick >> >> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>> Hi Nick, >>> >>> Nice to meet you too mate, I am going through the code base of kcidb.py to >>> push results. >>> >>> I will test pushing results to playground with the earlier credentials as you >>> mentioned. >>> >>> >>> >>> Also , I will be configuring internal kernelci deployment to push results to >>> public KCIDB for builds and tests. >>> >>> >>> >>> I do have some questions if you can help me find answer. >>> >>> * Do I need to use the private key from the credential json for this >>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>> <https://github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>> * As the storage server is internal , I guess we need to open access to >>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>> files so the kcidb server can access them . Can you confirm this , please >>> >>> >>> >>> Br, >>> >>> Manoj >>> >>> >>> >>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>> *Date: *Monday, 7 October 2024 at 12:15 >>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>> *Subject: *Re: KCIDB database >>> >>> Hi Manoj, >>> >>> Nice to meet you! >>> >>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM! >>> >>> The access parameters are out of date, although they will work for >>> submissions. Here's more up-to-date info: >>> >>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/ > kcidb/submitter_guide/> <https://docs.kernelci.org/ >> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>> kcidb/submitter_guide/> >>> >>> The credentials I gave ARM before should still work. I'd recommend restarting >>> submissions to the "playground" database ("playground_kcidb_new" topic) so you >>> can experiment freely without worrying about bothering anyone with invalid >>> data. Then when you're ready, switch to production ("kcidb_new"). >>> >>> Don't hesitate to reach out, if you have any questions. I'll add you as the >>> ARM's current contact here: >>> >>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/ > kcidb/issues/328> <https://github.com/kernelci/ >> kcidb/issues/328> <https://github.com/kernelci/ >>> kcidb/issues/328> >>> >>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite >>> you to our channel, so we can have more interactive Q&A, if needed. >>> >>> Nick >>> >>> On 10/7/24 12:37 PM, Manoj Kumar wrote: >>>> Hi Nikolai, >>>> >>>> I wanted to say hello to you and start the initial thread regarding pushing >>>> the test results to KernelCI KCIDB. >>>> I will be enabling the changes to push kernel test and build results to KCIDB >>>> from ARM testing . I was told by Cristian and Mark that you are the POC from >>>> kernelCI KCIDB side. >>>> >>>> The access parameters shared with me are below , can you confirm they are >>>> still valid and not expired / updated . >>>> >>>> >>>> >>>> >>>> >>>> The access parameters are: >>>> >>>> Google Cloud project name: kernelci-production >>>> Playground dataset name: playground_kernelci04 >>>> Playground submission queue topic: playground_kernelci_new >>>> Production dataset name: kernelci04 >>>> Production submission queue topic: kernelci_new >>>> >>>> >>>> >>>> Regards, >>>> >>>> Manoj >>>> >>>> >>>> >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy the >>> information in any medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <DB9PR08MB68104499218B100804567075C4402@DB9PR08MB6810.eurprd08.prod.outlook.com>]
* Re: KCIDB database [not found] ` <DB9PR08MB68104499218B100804567075C4402@DB9PR08MB6810.eurprd08.prod.outlook.com> @ 2024-10-18 15:05 ` Nikolai Kondrashov [not found] ` <PAXPR08MB6813E7233F5ACB43D4E7FC81C44E2@PAXPR08MB6813.eurprd08.prod.outlook.com> [not found] ` <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com> 0 siblings, 2 replies; 5+ messages in thread From: Nikolai Kondrashov @ 2024-10-18 15:05 UTC (permalink / raw) To: Manoj Kumar; +Cc: kernelci@lists.linux.dev, Denys Fedoryshchenko > It is not that straight forward to update python version to python3. Of course, and you cannot really do that, as KernelCI Legacy needs Python 2. I was rather suggesting you install Python 3 alongside Python 2, if possible for your target. > This upgrade will be disruptive for our testing and deployment unless the > KCIDB results push can be done with python2.7 which we will prefer at this > point of time. If this is hard, one (radical?) way to do this could be running Python 3 on a different machine (in a container?) and executing kcidb-submit there, via e.g. ssh (docker run?). But perhaps others have other ideas. Nick On 10/18/24 5:04 PM, Manoj Kumar wrote: > We used ansible playbooks to deploy backend server https://github.com/ > kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci- > backend-config> > > Which create virtual environment based on python2.7 > > https://github.com/kernelci/kernelci-backend-config/ > commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/ > kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071> > > > > It is not that straight forward to update python version to python3. > > This upgrade will be disruptive for our testing and deployment unless the > KCIDB results push can be done with python2.7 which we will prefer at this > point of time. > > > > br, > > Manoj > > > > *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> > *Date: *Friday, 18 October 2024 at 14:32 > *To: *Manoj Kumar <ManojKumar08@arm.com> > *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys > Fedoryshchenko <denys.f@collabora.com> > *Subject: *Re: KCIDB database > > Hi Manoj, > >> I looked at the below link to get installation steps for the kcidb-submit > command. >> >> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> >> >> This instruction is for Python version V3.9 but our backend is using the >> Python 2.7.16 at the moment. >> >> Do you have instruction for python 2.7.16 to install this package > > KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python > 2) is invoking the kcidb-submit command-line tool in a new process instead of > using the KCIDB library directly. > > You will need to install Python 3 in your deployment, I assume. Perhaps Denys, > or someone else on the KernelCI maillist (both CC'd) could tell you how we did > that. > > Nick > > On 10/18/24 4:21 PM, Manoj Kumar wrote: >> Hi Nikolai, >> >> >> >> I am planning to push test build data to public KCIDB . I am seeing a failure >> because of the kcidb-submit command in a thread at backend. >> >> >> >> The class KcidbSubmit expect the command kcidb-submit installed on the backend >> system. >> >> /class KcidbSubmit(object):/ >> >> / def __init__(self, kcidb_options):/ >> >> / kcidb_path = kcidb_options.get("kcidb_path", "")/ >> >> / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/ >> >> >> >> I looked at the below link to get installation steps for the kcidb-submit command. >> >> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> >> >> >> >> This instruction is for Python version V3.9 but our backend is using the >> Python 2.7.16 at the moment. >> >> Do you have instruction for python 2.7.16 to install this package >> >> >> >> Br, >> >> Manoj >> >> >> >> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >> *Date: *Tuesday, 8 October 2024 at 11:46 >> *To: *Manoj Kumar <ManojKumar08@arm.com> >> *Subject: *Re: KCIDB database >> >> Hi Manoj, >> >> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>> One additional question regarding the credentials file. >>> >>> Attached credentials files was shared with me named “.kernelci-production-ci- >>> arm.json” . >>> >>> The attached file contains multiple parameters , If I understood correctly all >>> the contents of this file should be assigned to this variable https:// >>> github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>> github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >> >> Not the contents, but a path to the file itself. >> >>> Also , the email ID used by internal kernelCI deployment is different . They >>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>> >>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>”. >> >> That doesn't matter, the email in the JSON credentials is simply a user >> (service account) ID and is specific to Google Cloud. It identifies a >> particular submitter to the system. >> >> You don't really need to look into the credentials file at all. >> >> Nick >> >> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>> Hi Nick, >>> >>> >>> >>> Thanks for your prompt responses :thumbsup >>> >>> >>> >>> One additional question regarding the credentials file. >>> >>> Attached credentials files was shared with me named “.kernelci-production-ci- >>> arm.json” . >>> >>> The attached file contains multiple parameters , If I understood correctly all >>> the contents of this file should be assigned to this variable https:// >>> github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>> github.com/kernelci/kernelci-backend/ >>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>> >>> >>> >>> Also , the email ID used by internal kernelCI deployment is different . They >>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>> >>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>”. >>> >>> >>> >>> Br, >>> >>> Manoj >>> >>> >>> >>> >>> >>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>> *Date: *Tuesday, 8 October 2024 at 05:36 >>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>> *Subject: *Re: KCIDB database >>> >>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>> I do have some questions if you can help me find answer. >>>> >>>> * Do I need to use the private key from the credential json for this >>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>> <https://github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>> >>> Yes, that needs to be pointing to the credentials JSON file. >>> >>>> * As the storage server is internal , I guess we need to open access to >>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>> files so the kcidb server can access them . Can you confirm this , please >>> >>> Yes, KCIDB right now will try to download every 256th URL you submit, that is >>> under 5MB. The artifact caching system is in trial mode still. However, what's >>> more important, *people* will try to open the URLs you submit. >>> >>> Opening the access to the whole server might be problematic for you guys, as >>> you want to keep some results secret, IIRC. Maybe you could copy them to some >>> other web server and translate the URLs as you submit them? >>> >>> Nick >>> >>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>> Hi Nick, >>>> >>>> Nice to meet you too mate, I am going through the code base of kcidb.py to >>>> push results. >>>> >>>> I will test pushing results to playground with the earlier credentials as you >>>> mentioned. >>>> >>>> >>>> >>>> Also , I will be configuring internal kernelci deployment to push results to >>>> public KCIDB for builds and tests. >>>> >>>> >>>> >>>> I do have some questions if you can help me find answer. >>>> >>>> * Do I need to use the private key from the credential json for this >>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>> <https://github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> * As the storage server is internal , I guess we need to open access to >>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>> files so the kcidb server can access them . Can you confirm this , please >>>> >>>> >>>> >>>> Br, >>>> >>>> Manoj >>>> >>>> >>>> >>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>> *Date: *Monday, 7 October 2024 at 12:15 >>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>> *Subject: *Re: KCIDB database >>>> >>>> Hi Manoj, >>>> >>>> Nice to meet you! >>>> >>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM! >>>> >>>> The access parameters are out of date, although they will work for >>>> submissions. Here's more up-to-date info: >>>> >>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/ > kcidb/submitter_guide/> <https://docs.kernelci.org/ >> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>> kcidb/submitter_guide/> >>>> >>>> The credentials I gave ARM before should still work. I'd recommend restarting >>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you >>>> can experiment freely without worrying about bothering anyone with invalid >>>> data. Then when you're ready, switch to production ("kcidb_new"). >>>> >>>> Don't hesitate to reach out, if you have any questions. I'll add you as the >>>> ARM's current contact here: >>>> >>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/ > kcidb/issues/328> <https://github.com/kernelci/ >> kcidb/issues/328> <https://github.com/kernelci/ >>> kcidb/issues/328> <https://github.com/kernelci/ >>>> kcidb/issues/328> >>>> >>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite >>>> you to our channel, so we can have more interactive Q&A, if needed. >>>> >>>> Nick >>>> >>>> On 10/7/24 12:37 PM, Manoj Kumar wrote: >>>>> Hi Nikolai, >>>>> >>>>> I wanted to say hello to you and start the initial thread regarding pushing >>>>> the test results to KernelCI KCIDB. >>>>> I will be enabling the changes to push kernel test and build results to KCIDB >>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from >>>>> kernelCI KCIDB side. >>>>> >>>>> The access parameters shared with me are below , can you confirm they are >>>>> still valid and not expired / updated . >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> The access parameters are: >>>>> >>>>> Google Cloud project name: kernelci-production >>>>> Playground dataset name: playground_kernelci04 >>>>> Playground submission queue topic: playground_kernelci_new >>>>> Production dataset name: kernelci04 >>>>> Production submission queue topic: kernelci_new >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Manoj >>>>> >>>>> >>>>> >>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>> confidential and may also be privileged. If you are not the intended >>>>> recipient, please notify the sender immediately and do not disclose the >>>>> contents to any other person, use it for any purpose, or store or copy the >>>>> information in any medium. Thank you. >>>> >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy the >>> information in any medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <PAXPR08MB6813E7233F5ACB43D4E7FC81C44E2@PAXPR08MB6813.eurprd08.prod.outlook.com>]
* Re: KCIDB database [not found] ` <PAXPR08MB6813E7233F5ACB43D4E7FC81C44E2@PAXPR08MB6813.eurprd08.prod.outlook.com> @ 2024-10-24 15:56 ` Nikolai Kondrashov 0 siblings, 0 replies; 5+ messages in thread From: Nikolai Kondrashov @ 2024-10-24 15:56 UTC (permalink / raw) To: Manoj Kumar; +Cc: kernelci@lists.linux.dev, Denys Fedoryshchenko On 10/24/24 6:13 PM, Manoj Kumar wrote: > Any suggestion for the right approach with your past experiences with KCIDB I really never had to do that, but after checking with your distro's docs, and finding that having both Python 2 and Python 3 installed together is impossible (if that's indeed true), my next step would be to try with Docker, using the official Python 3 container images as the base. Nick On 10/24/24 6:13 PM, Manoj Kumar wrote: > Any suggestion for the right approach with your past experiences with KCIDB > > > > Br, > > Manoj > > > > *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> > *Date: *Friday, 18 October 2024 at 16:05 > *To: *Manoj Kumar <ManojKumar08@arm.com> > *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys > Fedoryshchenko <denys.f@collabora.com> > *Subject: *Re: KCIDB database > >> It is not that straight forward to update python version to python3. > > Of course, and you cannot really do that, as KernelCI Legacy needs Python 2. > I was rather suggesting you install Python 3 alongside Python 2, if possible > for your target. > >> This upgrade will be disruptive for our testing and deployment unless the >> KCIDB results push can be done with python2.7 which we will prefer at this >> point of time. > > If this is hard, one (radical?) way to do this could be running Python 3 on a > different machine (in a container?) and executing kcidb-submit there, via e.g. > ssh (docker run?). > > But perhaps others have other ideas. > > Nick > > On 10/18/24 5:04 PM, Manoj Kumar wrote: >> We used ansible playbooks to deploy backend server https://github.com/ <https://github.com/> >> kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci- >> backend-config> >> >> Which create virtual environment based on python2.7 >> >> https://github.com/kernelci/kernelci-backend-config/ <https://github.com/ > kernelci/kernelci-backend-config/> >> commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/ >> kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071> >> >> >> >> It is not that straight forward to update python version to python3. >> >> This upgrade will be disruptive for our testing and deployment unless the >> KCIDB results push can be done with python2.7 which we will prefer at this >> point of time. >> >> >> >> br, >> >> Manoj >> >> >> >> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >> *Date: *Friday, 18 October 2024 at 14:32 >> *To: *Manoj Kumar <ManojKumar08@arm.com> >> *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys >> Fedoryshchenko <denys.f@collabora.com> >> *Subject: *Re: KCIDB database >> >> Hi Manoj, >> >>> I looked at the below link to get installation steps for the kcidb-submit >> command. >>> >>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> <https://docs.kernelci.org/ >>> kcidb/installation/> >>> >>> This instruction is for Python version V3.9 but our backend is using the >>> Python 2.7.16 at the moment. >>> >>> Do you have instruction for python 2.7.16 to install this package >> >> KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python >> 2) is invoking the kcidb-submit command-line tool in a new process instead of >> using the KCIDB library directly. >> >> You will need to install Python 3 in your deployment, I assume. Perhaps Denys, >> or someone else on the KernelCI maillist (both CC'd) could tell you how we did >> that. >> >> Nick >> >> On 10/18/24 4:21 PM, Manoj Kumar wrote: >>> Hi Nikolai, >>> >>> >>> >>> I am planning to push test build data to public KCIDB . I am seeing a failure >>> because of the kcidb-submit command in a thread at backend. >>> >>> >>> >>> The class KcidbSubmit expect the command kcidb-submit installed on the backend >>> system. >>> >>> /class KcidbSubmit(object):/ >>> >>> / def __init__(self, kcidb_options):/ >>> >>> / kcidb_path = kcidb_options.get("kcidb_path", "")/ >>> >>> / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/ >>> >>> >>> >>> I looked at the below link to get installation steps for the kcidb-submit command. >>> >>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> <https://docs.kernelci.org/ >>> kcidb/installation/> >>> >>> >>> >>> This instruction is for Python version V3.9 but our backend is using the >>> Python 2.7.16 at the moment. >>> >>> Do you have instruction for python 2.7.16 to install this package >>> >>> >>> >>> Br, >>> >>> Manoj >>> >>> >>> >>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>> *Date: *Tuesday, 8 October 2024 at 11:46 >>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>> *Subject: *Re: KCIDB database >>> >>> Hi Manoj, >>> >>> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>>> One additional question regarding the credentials file. >>>> >>>> Attached credentials files was shared with me named “.kernelci-production-ci- >>>> arm.json” . >>>> >>>> The attached file contains multiple parameters , If I understood correctly all >>>> the contents of this file should be assigned to this variable https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>> >>> Not the contents, but a path to the file itself. >>> >>>> Also , the email ID used by internal kernelCI deployment is different . They >>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>> >>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”. >>> >>> That doesn't matter, the email in the JSON credentials is simply a user >>> (service account) ID and is specific to Google Cloud. It identifies a >>> particular submitter to the system. >>> >>> You don't really need to look into the credentials file at all. >>> >>> Nick >>> >>> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>>> Hi Nick, >>>> >>>> >>>> >>>> Thanks for your prompt responses :thumbsup >>>> >>>> >>>> >>>> One additional question regarding the credentials file. >>>> >>>> Attached credentials files was shared with me named “.kernelci-production-ci- >>>> arm.json” . >>>> >>>> The attached file contains multiple parameters , If I understood correctly all >>>> the contents of this file should be assigned to this variable https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> >>>> >>>> >>>> Also , the email ID used by internal kernelCI deployment is different . They >>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>> >>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”. >>>> >>>> >>>> >>>> Br, >>>> >>>> Manoj >>>> >>>> >>>> >>>> >>>> >>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>> *Date: *Tuesday, 8 October 2024 at 05:36 >>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>> *Subject: *Re: KCIDB database >>>> >>>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>>> I do have some questions if you can help me find answer. >>>>> >>>>> * Do I need to use the private key from the credential json for this >>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> <https://github.com/kernelci/ >>>> kernelci-backend/> >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>>> <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> >>>> Yes, that needs to be pointing to the credentials JSON file. >>>> >>>>> * As the storage server is internal , I guess we need to open access to >>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>> >>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>>> files so the kcidb server can access them . Can you confirm this , please >>>> >>>> Yes, KCIDB right now will try to download every 256th URL you submit, that is >>>> under 5MB. The artifact caching system is in trial mode still. However, what's >>>> more important, *people* will try to open the URLs you submit. >>>> >>>> Opening the access to the whole server might be problematic for you guys, as >>>> you want to keep some results secret, IIRC. Maybe you could copy them to some >>>> other web server and translate the URLs as you submit them? >>>> >>>> Nick >>>> >>>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>>> Hi Nick, >>>>> >>>>> Nice to meet you too mate, I am going through the code base of kcidb.py to >>>>> push results. >>>>> >>>>> I will test pushing results to playground with the earlier credentials as you >>>>> mentioned. >>>>> >>>>> >>>>> >>>>> Also , I will be configuring internal kernelci deployment to push results to >>>>> public KCIDB for builds and tests. >>>>> >>>>> >>>>> >>>>> I do have some questions if you can help me find answer. >>>>> >>>>> * Do I need to use the private key from the credential json for this >>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> <https://github.com/kernelci/ >>>> kernelci-backend/> >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>>> <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>>> * As the storage server is internal , I guess we need to open access to >>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>> >>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>>> files so the kcidb server can access them . Can you confirm this , please >>>>> >>>>> >>>>> >>>>> Br, >>>>> >>>>> Manoj >>>>> >>>>> >>>>> >>>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>>> *Date: *Monday, 7 October 2024 at 12:15 >>>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>>> *Subject: *Re: KCIDB database >>>>> >>>>> Hi Manoj, >>>>> >>>>> Nice to meet you! >>>>> >>>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM! >>>>> >>>>> The access parameters are out of date, although they will work for >>>>> submissions. Here's more up-to-date info: >>>>> >>>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/ > kcidb/submitter_guide/> <https://docs.kernelci.org/ >> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>>> kcidb/submitter_guide/> >>>>> >>>>> The credentials I gave ARM before should still work. I'd recommend restarting >>>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you >>>>> can experiment freely without worrying about bothering anyone with invalid >>>>> data. Then when you're ready, switch to production ("kcidb_new"). >>>>> >>>>> Don't hesitate to reach out, if you have any questions. I'll add you as the >>>>> ARM's current contact here: >>>>> >>>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/ > kcidb/issues/328> <https://github.com/kernelci/ >> kcidb/issues/328> <https://github.com/kernelci/ >>> kcidb/issues/328> <https://github.com/kernelci/ >>>> kcidb/issues/328> <https://github.com/kernelci/ >>>>> kcidb/issues/328> >>>>> >>>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite >>>>> you to our channel, so we can have more interactive Q&A, if needed. >>>>> >>>>> Nick >>>>> >>>>> On 10/7/24 12:37 PM, Manoj Kumar wrote: >>>>>> Hi Nikolai, >>>>>> >>>>>> I wanted to say hello to you and start the initial thread regarding pushing >>>>>> the test results to KernelCI KCIDB. >>>>>> I will be enabling the changes to push kernel test and build results to KCIDB >>>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from >>>>>> kernelCI KCIDB side. >>>>>> >>>>>> The access parameters shared with me are below , can you confirm they are >>>>>> still valid and not expired / updated . >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> The access parameters are: >>>>>> >>>>>> Google Cloud project name: kernelci-production >>>>>> Playground dataset name: playground_kernelci04 >>>>>> Playground submission queue topic: playground_kernelci_new >>>>>> Production dataset name: kernelci04 >>>>>> Production submission queue topic: kernelci_new >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Manoj >>>>>> >>>>>> >>>>>> >>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>>> confidential and may also be privileged. If you are not the intended >>>>>> recipient, please notify the sender immediately and do not disclose the >>>>>> contents to any other person, use it for any purpose, or store or copy the >>>>>> information in any medium. Thank you. >>>>> >>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>> confidential and may also be privileged. If you are not the intended >>>>> recipient, please notify the sender immediately and do not disclose the >>>>> contents to any other person, use it for any purpose, or store or copy the >>>>> information in any medium. Thank you. >>>> >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy the >>> information in any medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com>]
* Re: KCIDB database [not found] ` <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com> @ 2024-11-08 13:15 ` Nikolai Kondrashov [not found] ` <DB9PR08MB68105069590B2A3A065DEBC2C45D2@DB9PR08MB6810.eurprd08.prod.outlook.com> 1 sibling, 0 replies; 5+ messages in thread From: Nikolai Kondrashov @ 2024-11-08 13:15 UTC (permalink / raw) To: Manoj Kumar; +Cc: kernelci@lists.linux.dev, Denys Fedoryshchenko Hi Manoj, Glad to hear you managed to make it work! It looks like KernelCI legacy is sending schema v1 indeed: https://github.com/kernelci/kernelci-backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L149 Gosh, that's old. However, it's working fine in production. I suspect this is because production is using an older version of KCIDB, and in newer versions support for old schemas is broken. I'll work on a fix, which should be available today. Meanwhile, could you please respond with a complete JSON you're sending, as an attachment, so I could make sure I reproduce this correctly. Thank you! Nick On 11/8/24 11:59 AM, Manoj Kumar wrote: > Hi Nikolai, > > > > I managed to create virtual environment python3.9 on kernelci backend and > worked my way up to push a build message to the kcidb database. > > The backend code is generating data structure with schema version “1”. > > Below is the content of the message after adding a debug print. > > "builds": [ > > { > > "origin": "arm", > > "config_name": "x86_64_defconfig", > > "architecture": "x86_64", > > "start_time": "2024-10-16T11:30:41.592000+00:00", > > "revision_origin_id": "2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2", > > "log_url": "", > > "duration": "1224.9012200832367", > > "origin_id": "kernelci.org:670fa3e1e5905849ba40dabe", > > "description": "v6.12-rc3-44-g2f87d0916ce0", > > "misc": { > > "kernel_image_size": "" > > }, > > "valid": "true", > > "compiler": "gcc (Debian 12.2.0-14) 12.2.0", > > "revision_origin": "arm", > > "config_url": "", > > "output_files": [] > > } > > ], > > *"version": "1"* > > > > > > In response to the above , I got a validation failure from the KCIDB server > which is expecting structure/ schema version to be high > > ValidationError: '1' is not of type 'object' > > Failed validating 'type' in schema['properties']['version']: > > {'type': 'object', > > 'properties': {'major': {'type': 'integer', > > 'const': 4, > > 'description': 'Major number of the schema ' > > 'version.\n' > > '\n' > > 'Increases represent ' > > 'backward-incompatible ' > > 'changes. E.g. deleting or ' > > 'renaming a property, ' > > 'changing a property type, ' > > 'restricting values, making a ' > > 'property required, or adding ' > > 'a new required property.'}, > > > > My question to you , what is the expected schema version on the Kernelci > legacy backend code so that the message will be accepted by KCIDB database > > > > > > Br, > > Manoj > > > > *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> > *Date: *Friday, 18 October 2024 at 16:05 > *To: *Manoj Kumar <ManojKumar08@arm.com> > *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys > Fedoryshchenko <denys.f@collabora.com> > *Subject: *Re: KCIDB database > >> It is not that straight forward to update python version to python3. > > Of course, and you cannot really do that, as KernelCI Legacy needs Python 2. > I was rather suggesting you install Python 3 alongside Python 2, if possible > for your target. > >> This upgrade will be disruptive for our testing and deployment unless the >> KCIDB results push can be done with python2.7 which we will prefer at this >> point of time. > > If this is hard, one (radical?) way to do this could be running Python 3 on a > different machine (in a container?) and executing kcidb-submit there, via e.g. > ssh (docker run?). > > But perhaps others have other ideas. > > Nick > > On 10/18/24 5:04 PM, Manoj Kumar wrote: >> We used ansible playbooks to deploy backend server https://github.com/ <https://github.com/> >> kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci- >> backend-config> >> >> Which create virtual environment based on python2.7 >> >> https://github.com/kernelci/kernelci-backend-config/ <https://github.com/ > kernelci/kernelci-backend-config/> >> commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/ >> kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071> >> >> >> >> It is not that straight forward to update python version to python3. >> >> This upgrade will be disruptive for our testing and deployment unless the >> KCIDB results push can be done with python2.7 which we will prefer at this >> point of time. >> >> >> >> br, >> >> Manoj >> >> >> >> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >> *Date: *Friday, 18 October 2024 at 14:32 >> *To: *Manoj Kumar <ManojKumar08@arm.com> >> *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys >> Fedoryshchenko <denys.f@collabora.com> >> *Subject: *Re: KCIDB database >> >> Hi Manoj, >> >>> I looked at the below link to get installation steps for the kcidb-submit >> command. >>> >>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> <https://docs.kernelci.org/ >>> kcidb/installation/> >>> >>> This instruction is for Python version V3.9 but our backend is using the >>> Python 2.7.16 at the moment. >>> >>> Do you have instruction for python 2.7.16 to install this package >> >> KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python >> 2) is invoking the kcidb-submit command-line tool in a new process instead of >> using the KCIDB library directly. >> >> You will need to install Python 3 in your deployment, I assume. Perhaps Denys, >> or someone else on the KernelCI maillist (both CC'd) could tell you how we did >> that. >> >> Nick >> >> On 10/18/24 4:21 PM, Manoj Kumar wrote: >>> Hi Nikolai, >>> >>> >>> >>> I am planning to push test build data to public KCIDB . I am seeing a failure >>> because of the kcidb-submit command in a thread at backend. >>> >>> >>> >>> The class KcidbSubmit expect the command kcidb-submit installed on the backend >>> system. >>> >>> /class KcidbSubmit(object):/ >>> >>> / def __init__(self, kcidb_options):/ >>> >>> / kcidb_path = kcidb_options.get("kcidb_path", "")/ >>> >>> / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/ >>> >>> >>> >>> I looked at the below link to get installation steps for the kcidb-submit command. >>> >>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> <https://docs.kernelci.org/ >>> kcidb/installation/> >>> >>> >>> >>> This instruction is for Python version V3.9 but our backend is using the >>> Python 2.7.16 at the moment. >>> >>> Do you have instruction for python 2.7.16 to install this package >>> >>> >>> >>> Br, >>> >>> Manoj >>> >>> >>> >>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>> *Date: *Tuesday, 8 October 2024 at 11:46 >>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>> *Subject: *Re: KCIDB database >>> >>> Hi Manoj, >>> >>> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>>> One additional question regarding the credentials file. >>>> >>>> Attached credentials files was shared with me named “.kernelci-production-ci- >>>> arm.json” . >>>> >>>> The attached file contains multiple parameters , If I understood correctly all >>>> the contents of this file should be assigned to this variable https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>> >>> Not the contents, but a path to the file itself. >>> >>>> Also , the email ID used by internal kernelCI deployment is different . They >>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>> >>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”. >>> >>> That doesn't matter, the email in the JSON credentials is simply a user >>> (service account) ID and is specific to Google Cloud. It identifies a >>> particular submitter to the system. >>> >>> You don't really need to look into the credentials file at all. >>> >>> Nick >>> >>> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>>> Hi Nick, >>>> >>>> >>>> >>>> Thanks for your prompt responses :thumbsup >>>> >>>> >>>> >>>> One additional question regarding the credentials file. >>>> >>>> Attached credentials files was shared with me named “.kernelci-production-ci- >>>> arm.json” . >>>> >>>> The attached file contains multiple parameters , If I understood correctly all >>>> the contents of this file should be assigned to this variable https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> >>>> >>>> >>>> Also , the email ID used by internal kernelCI deployment is different . They >>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>> >>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”. >>>> >>>> >>>> >>>> Br, >>>> >>>> Manoj >>>> >>>> >>>> >>>> >>>> >>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>> *Date: *Tuesday, 8 October 2024 at 05:36 >>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>> *Subject: *Re: KCIDB database >>>> >>>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>>> I do have some questions if you can help me find answer. >>>>> >>>>> * Do I need to use the private key from the credential json for this >>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> <https://github.com/kernelci/ >>>> kernelci-backend/> >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>>> <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> >>>> Yes, that needs to be pointing to the credentials JSON file. >>>> >>>>> * As the storage server is internal , I guess we need to open access to >>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>> >>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>>> files so the kcidb server can access them . Can you confirm this , please >>>> >>>> Yes, KCIDB right now will try to download every 256th URL you submit, that is >>>> under 5MB. The artifact caching system is in trial mode still. However, what's >>>> more important, *people* will try to open the URLs you submit. >>>> >>>> Opening the access to the whole server might be problematic for you guys, as >>>> you want to keep some results secret, IIRC. Maybe you could copy them to some >>>> other web server and translate the URLs as you submit them? >>>> >>>> Nick >>>> >>>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>>> Hi Nick, >>>>> >>>>> Nice to meet you too mate, I am going through the code base of kcidb.py to >>>>> push results. >>>>> >>>>> I will test pushing results to playground with the earlier credentials as you >>>>> mentioned. >>>>> >>>>> >>>>> >>>>> Also , I will be configuring internal kernelci deployment to push results to >>>>> public KCIDB for builds and tests. >>>>> >>>>> >>>>> >>>>> I do have some questions if you can help me find answer. >>>>> >>>>> * Do I need to use the private key from the credential json for this >>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> <https://github.com/kernelci/ >>>> kernelci-backend/> >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>>> <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>>> * As the storage server is internal , I guess we need to open access to >>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>> >>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>>> files so the kcidb server can access them . Can you confirm this , please >>>>> >>>>> >>>>> >>>>> Br, >>>>> >>>>> Manoj >>>>> >>>>> >>>>> >>>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>>> *Date: *Monday, 7 October 2024 at 12:15 >>>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>>> *Subject: *Re: KCIDB database >>>>> >>>>> Hi Manoj, >>>>> >>>>> Nice to meet you! >>>>> >>>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM! >>>>> >>>>> The access parameters are out of date, although they will work for >>>>> submissions. Here's more up-to-date info: >>>>> >>>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/ > kcidb/submitter_guide/> <https://docs.kernelci.org/ >> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>>> kcidb/submitter_guide/> >>>>> >>>>> The credentials I gave ARM before should still work. I'd recommend restarting >>>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you >>>>> can experiment freely without worrying about bothering anyone with invalid >>>>> data. Then when you're ready, switch to production ("kcidb_new"). >>>>> >>>>> Don't hesitate to reach out, if you have any questions. I'll add you as the >>>>> ARM's current contact here: >>>>> >>>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/ > kcidb/issues/328> <https://github.com/kernelci/ >> kcidb/issues/328> <https://github.com/kernelci/ >>> kcidb/issues/328> <https://github.com/kernelci/ >>>> kcidb/issues/328> <https://github.com/kernelci/ >>>>> kcidb/issues/328> >>>>> >>>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite >>>>> you to our channel, so we can have more interactive Q&A, if needed. >>>>> >>>>> Nick >>>>> >>>>> On 10/7/24 12:37 PM, Manoj Kumar wrote: >>>>>> Hi Nikolai, >>>>>> >>>>>> I wanted to say hello to you and start the initial thread regarding pushing >>>>>> the test results to KernelCI KCIDB. >>>>>> I will be enabling the changes to push kernel test and build results to KCIDB >>>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from >>>>>> kernelCI KCIDB side. >>>>>> >>>>>> The access parameters shared with me are below , can you confirm they are >>>>>> still valid and not expired / updated . >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> The access parameters are: >>>>>> >>>>>> Google Cloud project name: kernelci-production >>>>>> Playground dataset name: playground_kernelci04 >>>>>> Playground submission queue topic: playground_kernelci_new >>>>>> Production dataset name: kernelci04 >>>>>> Production submission queue topic: kernelci_new >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Manoj >>>>>> >>>>>> >>>>>> >>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>>> confidential and may also be privileged. If you are not the intended >>>>>> recipient, please notify the sender immediately and do not disclose the >>>>>> contents to any other person, use it for any purpose, or store or copy the >>>>>> information in any medium. Thank you. >>>>> >>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>> confidential and may also be privileged. If you are not the intended >>>>> recipient, please notify the sender immediately and do not disclose the >>>>> contents to any other person, use it for any purpose, or store or copy the >>>>> information in any medium. Thank you. >>>> >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy the >>> information in any medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <DB9PR08MB68105069590B2A3A065DEBC2C45D2@DB9PR08MB6810.eurprd08.prod.outlook.com>]
* Re: KCIDB database [not found] ` <DB9PR08MB68105069590B2A3A065DEBC2C45D2@DB9PR08MB6810.eurprd08.prod.outlook.com> @ 2024-11-08 14:51 ` Nikolai Kondrashov 0 siblings, 0 replies; 5+ messages in thread From: Nikolai Kondrashov @ 2024-11-08 14:51 UTC (permalink / raw) To: Manoj Kumar; +Cc: kernelci@lists.linux.dev, Denys Fedoryshchenko Hi Manoj, OK, the bug is now fixed: https://github.com/kernelci/kcidb-io/pull/91 Please update your local copy of kcidb-io. > Below is the code snippet for test message at kernelCI backend. Thank you. I can see a few problems with that data: * The `duration` is a string, and it should be a number * The `log_url` is a `none` string, and should either be a URL, or missing from the data. * Same for the `config_url`. Whenever you want to test your data sample, you can simply pipe it to `kcidb-validate` and it will validate it for you, without needing to go through the whole submission process. Nick On 11/8/24 4:01 PM, Manoj Kumar wrote: > Hi Nick, > > I created a test data structure in the file https://github.com/kernelci/ > kernelci-backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ > kcidb.py#L149 <https://github.com/kernelci/kernelci-backend/ > blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L149> > > > > The same structure is converted to Json and being pushed to upstream as shown > below. > > Below is the code snippet for test message at kernelCI backend. > > > > kcidb_build = { > > 'revision_origin': 'arm', > > 'revision_origin_id': '2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2', > > 'origin': 'arm', > > 'origin_id': 'kernelci.org:670fa3e1e5905849ba40dabe', > > 'start_time': '2024-10-16T11:30:41.592000+00:00', > > 'description': 'v6.12-rc3-44-g2f87d0916ce0', > > 'duration': '1224.9012200832367', > > 'architecture': 'x86_64', > > 'compiler': 'gcc (Debian 12.2.0-14) 12.2.0', > > 'log_url': 'none', > > 'config_name': 'x86_64_defconfig', > > 'config_url': 'none', > > 'output_files': [], > > 'valid': True, > > 'misc': { > > 'kernel_image_size': '', > > }, > > } > > kcidb_data = { > > 'version': '1', > > } > > kcidb_data['builds'] = [kcidb_build] > > > > json_data = json.dumps(kcidb_data, indent=2) > > utils.LOG.info(json_data) > > kcidb_submit.write(json_data) > > > > br, > > Manoj > > *From: *Manoj Kumar <ManojKumar08@arm.com> > *Date: *Friday, 8 November 2024 at 09:59 > *To: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> > *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys > Fedoryshchenko <denys.f@collabora.com> > *Subject: *Re: KCIDB database > > Hi Nikolai, > > > > I managed to create virtual environment python3.9 on kernelci backend and > worked my way up to push a build message to the kcidb database. > > The backend code is generating data structure with schema version “1”. > > Below is the content of the message after adding a debug print. > > "builds": [ > > { > > "origin": "arm", > > "config_name": "x86_64_defconfig", > > "architecture": "x86_64", > > "start_time": "2024-10-16T11:30:41.592000+00:00", > > "revision_origin_id": "2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2", > > "log_url": "", > > "duration": "1224.9012200832367", > > "origin_id": "kernelci.org:670fa3e1e5905849ba40dabe", > > "description": "v6.12-rc3-44-g2f87d0916ce0", > > "misc": { > > "kernel_image_size": "" > > }, > > "valid": "true", > > "compiler": "gcc (Debian 12.2.0-14) 12.2.0", > > "revision_origin": "arm", > > "config_url": "", > > "output_files": [] > > } > > ], > > *"version": "1"* > > > > > > In response to the above , I got a validation failure from the KCIDB server > which is expecting structure/ schema version to be high > > ValidationError: '1' is not of type 'object' > > Failed validating 'type' in schema['properties']['version']: > > {'type': 'object', > > 'properties': {'major': {'type': 'integer', > > 'const': 4, > > 'description': 'Major number of the schema ' > > 'version.\n' > > '\n' > > 'Increases represent ' > > 'backward-incompatible ' > > 'changes. E.g. deleting or ' > > 'renaming a property, ' > > 'changing a property type, ' > > 'restricting values, making a ' > > 'property required, or adding ' > > 'a new required property.'}, > > > > My question to you , what is the expected schema version on the Kernelci > legacy backend code so that the message will be accepted by KCIDB database > > > > > > Br, > > Manoj > > > > *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> > *Date: *Friday, 18 October 2024 at 16:05 > *To: *Manoj Kumar <ManojKumar08@arm.com> > *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys > Fedoryshchenko <denys.f@collabora.com> > *Subject: *Re: KCIDB database > >> It is not that straight forward to update python version to python3. > > Of course, and you cannot really do that, as KernelCI Legacy needs Python 2. > I was rather suggesting you install Python 3 alongside Python 2, if possible > for your target. > >> This upgrade will be disruptive for our testing and deployment unless the >> KCIDB results push can be done with python2.7 which we will prefer at this >> point of time. > > If this is hard, one (radical?) way to do this could be running Python 3 on a > different machine (in a container?) and executing kcidb-submit there, via e.g. > ssh (docker run?). > > But perhaps others have other ideas. > > Nick > > On 10/18/24 5:04 PM, Manoj Kumar wrote: >> We used ansible playbooks to deploy backend server https://github.com/ <https://github.com/> >> kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci- >> backend-config> >> >> Which create virtual environment based on python2.7 >> >> https://github.com/kernelci/kernelci-backend-config/ <https://github.com/ > kernelci/kernelci-backend-config/> >> commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/ >> kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071> >> >> >> >> It is not that straight forward to update python version to python3. >> >> This upgrade will be disruptive for our testing and deployment unless the >> KCIDB results push can be done with python2.7 which we will prefer at this >> point of time. >> >> >> >> br, >> >> Manoj >> >> >> >> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >> *Date: *Friday, 18 October 2024 at 14:32 >> *To: *Manoj Kumar <ManojKumar08@arm.com> >> *Cc: *kernelci@lists.linux.dev <kernelci@lists.linux.dev>, Denys >> Fedoryshchenko <denys.f@collabora.com> >> *Subject: *Re: KCIDB database >> >> Hi Manoj, >> >>> I looked at the below link to get installation steps for the kcidb-submit >> command. >>> >>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> <https://docs.kernelci.org/ >>> kcidb/installation/> >>> >>> This instruction is for Python version V3.9 but our backend is using the >>> Python 2.7.16 at the moment. >>> >>> Do you have instruction for python 2.7.16 to install this package >> >> KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python >> 2) is invoking the kcidb-submit command-line tool in a new process instead of >> using the KCIDB library directly. >> >> You will need to install Python 3 in your deployment, I assume. Perhaps Denys, >> or someone else on the KernelCI maillist (both CC'd) could tell you how we did >> that. >> >> Nick >> >> On 10/18/24 4:21 PM, Manoj Kumar wrote: >>> Hi Nikolai, >>> >>> >>> >>> I am planning to push test build data to public KCIDB . I am seeing a failure >>> because of the kcidb-submit command in a thread at backend. >>> >>> >>> >>> The class KcidbSubmit expect the command kcidb-submit installed on the backend >>> system. >>> >>> /class KcidbSubmit(object):/ >>> >>> / def __init__(self, kcidb_options):/ >>> >>> / kcidb_path = kcidb_options.get("kcidb_path", "")/ >>> >>> / self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/ >>> >>> >>> >>> I looked at the below link to get installation steps for the kcidb-submit command. >>> >>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/ > kcidb/installation/> <https://docs.kernelci.org/ >> kcidb/installation/> <https://docs.kernelci.org/ >>> kcidb/installation/> >>> >>> >>> >>> This instruction is for Python version V3.9 but our backend is using the >>> Python 2.7.16 at the moment. >>> >>> Do you have instruction for python 2.7.16 to install this package >>> >>> >>> >>> Br, >>> >>> Manoj >>> >>> >>> >>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>> *Date: *Tuesday, 8 October 2024 at 11:46 >>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>> *Subject: *Re: KCIDB database >>> >>> Hi Manoj, >>> >>> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>>> One additional question regarding the credentials file. >>>> >>>> Attached credentials files was shared with me named “.kernelci-production-ci- >>>> arm.json” . >>>> >>>> The attached file contains multiple parameters , If I understood correctly all >>>> the contents of this file should be assigned to this variable https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>> >>> Not the contents, but a path to the file itself. >>> >>>> Also , the email ID used by internal kernelCI deployment is different . They >>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>> >>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”. >>> >>> That doesn't matter, the email in the JSON credentials is simply a user >>> (service account) ID and is specific to Google Cloud. It identifies a >>> particular submitter to the system. >>> >>> You don't really need to look into the credentials file at all. >>> >>> Nick >>> >>> On 10/8/24 1:26 PM, Manoj Kumar wrote: >>>> Hi Nick, >>>> >>>> >>>> >>>> Thanks for your prompt responses :thumbsup >>>> >>>> >>>> >>>> One additional question regarding the credentials file. >>>> >>>> Attached credentials files was shared with me named “.kernelci-production-ci- >>>> arm.json” . >>>> >>>> The attached file contains multiple parameters , If I understood correctly all >>>> the contents of this file should be assigned to this variable https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https:// >>>> github.com/kernelci/kernelci-backend/ >>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> >>>> >>>> >>>> Also , the email ID used by internal kernelCI deployment is different . They >>>> will be “bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com <mailto:bot@kernelci-dev.oss.arm.com>>>> >>>> and bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com <mailto:bot@kernelci.oss.arm.com>>>>”. >>>> >>>> >>>> >>>> Br, >>>> >>>> Manoj >>>> >>>> >>>> >>>> >>>> >>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>> *Date: *Tuesday, 8 October 2024 at 05:36 >>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>> *Subject: *Re: KCIDB database >>>> >>>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>>> I do have some questions if you can help me find answer. >>>>> >>>>> * Do I need to use the private key from the credential json for this >>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> <https://github.com/kernelci/ >>>> kernelci-backend/> >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>>> <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>> >>>> Yes, that needs to be pointing to the credentials JSON file. >>>> >>>>> * As the storage server is internal , I guess we need to open access to >>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>> >>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>>> files so the kcidb server can access them . Can you confirm this , please >>>> >>>> Yes, KCIDB right now will try to download every 256th URL you submit, that is >>>> under 5MB. The artifact caching system is in trial mode still. However, what's >>>> more important, *people* will try to open the URLs you submit. >>>> >>>> Opening the access to the whole server might be problematic for you guys, as >>>> you want to keep some results secret, IIRC. Maybe you could copy them to some >>>> other web server and translate the URLs as you submit them? >>>> >>>> Nick >>>> >>>> On 10/7/24 7:49 PM, Manoj Kumar wrote: >>>>> Hi Nick, >>>>> >>>>> Nice to meet you too mate, I am going through the code base of kcidb.py to >>>>> push results. >>>>> >>>>> I will test pushing results to playground with the earlier credentials as you >>>>> mentioned. >>>>> >>>>> >>>>> >>>>> Also , I will be configuring internal kernelci deployment to push results to >>>>> public KCIDB for builds and tests. >>>>> >>>>> >>>>> >>>>> I do have some questions if you can help me find answer. >>>>> >>>>> * Do I need to use the private key from the credential json for this >>>>> parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/ > kernelci-backend/> <https://github.com/kernelci/ >> kernelci-backend/> <https://github.com/kernelci/ >>> kernelci-backend/> <https://github.com/kernelci/ >>>> kernelci-backend/> >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 >>>>> <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38> >>>>> * As the storage server is internal , I guess we need to open access to >>>>> storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci-> > <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->> >> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>> >>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>> >>>>> backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/ >>>>> kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/ >>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173> >>>>> files so the kcidb server can access them . Can you confirm this , please >>>>> >>>>> >>>>> >>>>> Br, >>>>> >>>>> Manoj >>>>> >>>>> >>>>> >>>>> *From: *Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> >>>>> *Date: *Monday, 7 October 2024 at 12:15 >>>>> *To: *Manoj Kumar <ManojKumar08@arm.com> >>>>> *Subject: *Re: KCIDB database >>>>> >>>>> Hi Manoj, >>>>> >>>>> Nice to meet you! >>>>> >>>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM! >>>>> >>>>> The access parameters are out of date, although they will work for >>>>> submissions. Here's more up-to-date info: >>>>> >>>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/ > kcidb/submitter_guide/> <https://docs.kernelci.org/ >> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>> kcidb/submitter_guide/> <https://docs.kernelci.org/ >>>>> kcidb/submitter_guide/> >>>>> >>>>> The credentials I gave ARM before should still work. I'd recommend restarting >>>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you >>>>> can experiment freely without worrying about bothering anyone with invalid >>>>> data. Then when you're ready, switch to production ("kcidb_new"). >>>>> >>>>> Don't hesitate to reach out, if you have any questions. I'll add you as the >>>>> ARM's current contact here: >>>>> >>>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/ > kcidb/issues/328> <https://github.com/kernelci/ >> kcidb/issues/328> <https://github.com/kernelci/ >>> kcidb/issues/328> <https://github.com/kernelci/ >>>> kcidb/issues/328> <https://github.com/kernelci/ >>>>> kcidb/issues/328> >>>>> >>>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite >>>>> you to our channel, so we can have more interactive Q&A, if needed. >>>>> >>>>> Nick >>>>> >>>>> On 10/7/24 12:37 PM, Manoj Kumar wrote: >>>>>> Hi Nikolai, >>>>>> >>>>>> I wanted to say hello to you and start the initial thread regarding pushing >>>>>> the test results to KernelCI KCIDB. >>>>>> I will be enabling the changes to push kernel test and build results to KCIDB >>>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from >>>>>> kernelCI KCIDB side. >>>>>> >>>>>> The access parameters shared with me are below , can you confirm they are >>>>>> still valid and not expired / updated . >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> The access parameters are: >>>>>> >>>>>> Google Cloud project name: kernelci-production >>>>>> Playground dataset name: playground_kernelci04 >>>>>> Playground submission queue topic: playground_kernelci_new >>>>>> Production dataset name: kernelci04 >>>>>> Production submission queue topic: kernelci_new >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Manoj >>>>>> >>>>>> >>>>>> >>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>>> confidential and may also be privileged. If you are not the intended >>>>>> recipient, please notify the sender immediately and do not disclose the >>>>>> contents to any other person, use it for any purpose, or store or copy the >>>>>> information in any medium. Thank you. >>>>> >>>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>>> confidential and may also be privileged. If you are not the intended >>>>> recipient, please notify the sender immediately and do not disclose the >>>>> contents to any other person, use it for any purpose, or store or copy the >>>>> information in any medium. Thank you. >>>> >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy the >>> information in any medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-11-08 14:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <547ddf99-1b75-c463-e6c7-32389c461468@redhat.com>
[not found] ` <AS8PR08MB6407FD1D5E33918F5DA6032A9E772@AS8PR08MB6407.eurprd08.prod.outlook.com>
[not found] ` <PAXPR08MB6813993A4C70BDE277129A76C47D2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <d9f9111e-9890-4795-9f8a-a6946a8fcbb9@redhat.com>
[not found] ` <PAXPR08MB68136B5FD6BFD45B2BC1BE2DC47D2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <9ab1e3c8-9c7f-4657-815a-13a68e0b87f7@redhat.com>
[not found] ` <PAXPR08MB68139AC2C5DFA28EC9A4CCC6C47E2@PAXPR08MB6813.eurprd08.prod.outlook.com>
[not found] ` <2626f2ce-adb0-4aad-bbfc-c19f634a4d30@redhat.com>
[not found] ` <DB9PR08MB68103176F5638C352F410BCBC4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-10-18 13:32 ` KCIDB database Nikolai Kondrashov
[not found] ` <DB9PR08MB68104499218B100804567075C4402@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-10-18 15:05 ` Nikolai Kondrashov
[not found] ` <PAXPR08MB6813E7233F5ACB43D4E7FC81C44E2@PAXPR08MB6813.eurprd08.prod.outlook.com>
2024-10-24 15:56 ` Nikolai Kondrashov
[not found] ` <DB9PR08MB68109AF608CFB742F567968DC4532@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-11-08 13:15 ` Nikolai Kondrashov
[not found] ` <DB9PR08MB68105069590B2A3A065DEBC2C45D2@DB9PR08MB6810.eurprd08.prod.outlook.com>
2024-11-08 14:51 ` Nikolai Kondrashov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox