Re: Contribute to Kernel-CI with a new Lab

["Santiago.Esteban via info" <santiago.esteban@microchip.com>]

[-- Attachment #1: Type: text/plain, Size: 2081 bytes --]

On 12/11/20 9:49, Jan Lübbe wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> Hi Santi,
>
> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>> Hi KernelCI,
>>
>> A few months back I contacted you about adding a new lab to KernelCI
>> infrastructure. It took us longer than I wished, but now we finally
>> have been able to connect our farm to an internal KernelCI deployment
>> (using docker).
>>
>> As I explained before, our farm holds 4 boards with different SoCs
>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>> support.
>>
>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>> tests. We use "kci_data" tool to publish test results on KernelCI.
> That's very interesting. :) Do you have published the interface code
> somewhere? I've been trying to find some time to do that myself, but it
> seems you've beaten me to it. ;)
>
> Regards
> Jan

Hi Jan,

No, I haven't publish it, till now. I've never though it would be useful 
to anybody but me  ;)

I have a python script that is tailored (too much) to our systems. It 
performs some actions that depend on our infrastructure and how I've 
implemented the labgrid tests (for example, I grab the results from a 
pytest  json report). At the end, it creates a "tmeta_<board>.json" file 
(equivalent to the "bmeta.json") that is later published with "kci_data" 
tool.

It could be used as an inspiration to make a more generic 
"kci_labgrid_test" tool if there are interest, but, all kernelci 
important stuff, still needs to be validated ;)

I have attached it to this email the script and (more important) an 
example of the output it produces.

BR,

Santi

>
>> We would like to work with you to be able to publish these results.
>> Will it be possible to get a token for the staging database? I'm sure
>> that there are things that need to be polished on our json files.
>


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mpuci-test-labgrid.py --]
[-- Type: text/x-python; name="mpuci-test-labgrid.py", Size: 7626 bytes --]

#!/usr/bin/python3

import sys
import os
import argparse
import hashlib
import subprocess
import time
import json
import socket

def labgrid_test(coord, board, image, test, test_json, bootrr):
    cmd = "labgrid-client -x {} reserve board={} --shell --wait".format(coord,board)    
    result = subprocess.check_output(cmd, stderr=subprocess.STDOUT,shell=True)
    token = result.decode("utf-8").strip().split("=")[-1] 
    
    cmd = "labgrid-client -x {} -p +{} lock".format(coord, token)
    result = subprocess.check_output(cmd, stderr=subprocess.STDOUT,shell=True)
    lab = result.decode("utf-8").strip().split()[-1]
    print("Labgrid resources locked ")
    
    #get lab name 
    lab = "remote-{}.yaml".format(lab)
    print("Execute test in lab {}".format(lab))
    
    currpwd = os.getcwd()

    if bootrr is not  None:
        bootrr = " --bootrr {} ".format(bootrr)
        

    os.chdir("labgrid-test/release/{}".format(board))
    os.system("pwd")
    os.system("rm console_main_ttyUSB0")
    start_test =time.time()
    try:
        cmd = "pytest -v -s --json-report --lg-coordinator {} --lg-log --lg-env {} {} --file {} {}".format(coord, lab, test, image, bootrr)
        print(cmd)
        sys.stdout.flush()
        ret= os.system(cmd)
    except:
        cmd = "labgrid-client -x {} -p +{} release".format(coord, token)
        print(cmd)
        ret =os.system(cmd)
        return 1, []


    if ret == 0:
        status = 'PASS'
    else:
        status = 'FAIL'
    end_test = time.time()
   
    if os.path.isfile("console_main_ttyUSB0"):
        logfile = open("console_main_ttyUSB0", "r", encoding='utf8',errors='ignore')
        test_json["log"] = logfile.read()
    else:
        print ("Can not find pytest console log")
    
    #Add baseline test results from json report
    testlist= ["test_login", "test_dmesg_alert", "test_dmesg_crit", "test_dmesg_err", "test_bootrr"]
    try:
        if os.path.isfile(".report.json"):
            with open(".report.json", "r") as report:
                data = json.load(report)
                for test in data['tests']:
                    if  "metadata" in test:
                        if  'result' in test['metadata']:
                            result = test["metadata"]['result']
                            result["job"] = test_json["job"]
                            result["time"] = test['call']['duration']
                            test_json["test_cases"].append(result) 
                        else:
                            result = { "name" : test["nodeid"] }
                            result = { "status" : "fail" }
                            test_json["test_cases"].append(result) 
                    else:                         
                        if errtest in testlist:
                            name= errtest.split('_')[-1]
                            result = { "name" : name, 'status' : 'fail' }
                        else:
                            continue
                                
                        test_json["test_cases"].append(result)                         
    except:
        print("Error parsing results from labgrid!")        
        ret=1, test_json
                               


    
    cmd = "labgrid-client -x {} -p +{} release".format(coord, token)
    os.system(cmd)
    os.chdir(currpwd) #restore path
    return ret,test_json
        

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description = 'Linux build test tool.')
    parser.add_argument('-p','--path',
                        required = False,
                        help ='Path to kernel to test', 
                        default ="../linux-at91/_install_/fit/")

    parser.add_argument('-k','--kernel',
                        required = False,
                        help ='Kernel image to test.', 
                        default ="sama5d2_xplained.itb")

    parser.add_argument('-b','--board',
                        required = True,
                        help ='Farm board to test with.', 
                        default ="sama5d2_xplained")
    
    parser.add_argument('-c','--coord',
                        required = False,
                        help ='Labgrid cooordinator"', 
                        default = "ws://mpuci-dev.mchp-main.com:20408/ws" )

    parser.add_argument('-t','--test',
                        required = False,
                        help ='Pytest file to execute"', 
                        default = "test_baseline.py" )

    parser.add_argument('-l','--lab',
                        required = False,
                        help ='KCI test lab', 
                        default = "lab-test" )
    
    parser.add_argument('--bootrr',
                        required = False,
                        help ='bootrr script to test', 
                        default = "" )

    #parse arguments
    args = parser.parse_args()


    #Retrieve build data
    sp = args.path.split("/")
    sp.pop(-1)
    buildpath = "/".join(sp) 
    sp.append("tmeta_{}.json".format(args.board))
    outfile = "/".join(sp)
    
    with open(buildpath + "/bmeta.json", 'r') as json_file:
        data = json.load(json_file)

    #Deploy to nfs/tftp shared folder
    image = args.path + args.kernel
    if os.path.isfile(image):
        imagepath = "/opt/tftpd/kernelci/{}/{}".format(socket.gethostname(),data["git_describe_v"])
        os.system("ssh labgrid@mpuci-dev.mchp-main.com mkdir -p {}".format(imagepath))

        imagepath = "{}/{}".format(imagepath, data["build_environment"])
        os.system("ssh labgrid@mpuci-dev.mchp-main.com mkdir -p {}".format(imagepath))

        imagepath = "{}/{}".format(imagepath, args.board)
        os.system("ssh labgrid@mpuci-dev.mchp-main.com mkdir -p {}".format(imagepath))

        print("Copy image {} to tftpd server {} -->  {}".format(image, "mpuci-dev", imagepath ))
        ret = os.system("scp {} labgrid@mpuci-dev.mchp-main.com:{}/".format(image,imagepath))
    else:
        print("Kernel Image file {} does not exist!".format(image))
        exit(1)

    if ret != 0:
        print("Error copying file to tftp server.")      
        exit(ret)

    #create test group dict.
    test_group = {
        "arch"  : data["arch"],
        "job"   : data["job"],
        "kernel": data["git_describe"],
        "defconfig": data["defconfig"],
        "defconfig_full": data["defconfig_full"],
        "build_environment": data["build_environment"],
        "git_branch": data["git_branch"],
        "git_commit": data["git_commit"],
        "lab_name": args.lab,
        "endian": "little",
        "mach"  : "at91",
        "board_instance" : "at91-"+args.board,
        "device_type" : "at91-"+args.board,
        "time": 0,
        "name": "baseline",
        "kernel_image": args.kernel,
        "file_server_resource": data["file_server_resource"],
        "test_cases" : []
    }

    start = time.time()
    #Run test
    kernel = imagepath +"/"+ args.kernel.split('/')[-1]
    result, test_group =labgrid_test(args.coord, args.board, kernel , args.test, test_group, args.bootrr )
    end = time.time()
    test_group["time"]= end - start
    
    if result == 0:        
        print("Write Json test result file") 
        with open(outfile, 'w') as wr_file:
            json.dump(test_group,wr_file, indent=4)
    else:
        print("Failed to execute test")
        exit(result)



[-- Attachment #3: tmeta_sama5d2_xplained.json --]
[-- Type: application/json, Size: 23587 bytes --]