Reference browser for testing purposes

Reference browser for testing purposes

[Barros Pena, Belen]

We should probably have raised this question earlier and had a plan in
place, but hey, better late ... The question is: which browsers should we
be using as a reference for QA purposes? Our guideline here is decent
HTML5 compatibility, but we never qualified what 'decent' means.

The other reference we could use is traffic to the Yocto Project website.
Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
and 27 on Linux. I can put together more detailed numbers if anybody wants
to see them. 

Those might be a bit too cutting edge, but could guide our decision
somehow. QA is currently testing with Firefox 11: that is probably too
old. 

In light of the above, any suggestions about which browsers we should use
for testing?

Thanks!

Belén

Re: Reference browser for testing purposes

[Georgescu, Alexandru C]

I agree with Chrome and Firefox. Don't know about the versions yet (I'm not so familiar with web testing) but from what I know, Chrome and Firefox has a quick pace on releasing new versions so in my opinion makes sense just to test the latest versions. When we'll have something automated than we will cover more browsers and distros.

Regards,
--
Alexandru Georgescu

-----Original Message-----
From: toaster-bounces@yoctoproject.org [mailto:toaster-bounces@yoctoproject.org] On Behalf Of Barros Pena, Belen
Sent: Wednesday, March 19, 2014 6:24 PM
To: toaster@yoctoproject.org
Cc: Sun, Yuan (Wind River); Paul Eggleton
Subject: [Toaster] Reference browser for testing purposes

We should probably have raised this question earlier and had a plan in place, but hey, better late ... The question is: which browsers should we be using as a reference for QA purposes? Our guideline here is decent
HTML5 compatibility, but we never qualified what 'decent' means.

The other reference we could use is traffic to the Yocto Project website.
Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26 and 27 on Linux. I can put together more detailed numbers if anybody wants to see them. 

Those might be a bit too cutting edge, but could guide our decision somehow. QA is currently testing with Firefox 11: that is probably too old. 

In light of the above, any suggestions about which browsers we should use for testing?

Thanks!

Belén

--
_______________________________________________
toaster mailing list
toaster@yoctoproject.org
https://lists.yoctoproject.org/listinfo/toaster

Re: Reference browser for testing purposes

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 2046 bytes --]

Our goal is not "decent" but complete HTML5 compatibility.

The target is that our HTML output is to be validated by HTML5 validators
with no errors shown. We already selected the industry-standard HTML5
validators to verify this.
Specifically, we are using in development http://validator.w3.org/ through
a browser extension. This MUST be automated at a certain point.

What I'm trying to avoid here is coding specifically for a target browser
or platform. I suggest to not restrict testing to a certain
browser/platform/version, but use what ever the tester uses in real life.
In case of presentation bugs are discovered, first we have to rule out an
issue with the browser of choice by testing visual reproducibility with
another browser on the same page and verifying browser's HTML5
compatibility.

For visual reference, widgets in the page change across different platforms
and browsers. Do we have test cases for the appearance ?

The test cases should not be dependent or executed with a specified
platform/browser version.

Cheers,
Alex



On Wed, Mar 19, 2014 at 4:23 PM, Barros Pena, Belen <
belen.barros.pena@intel.com> wrote:

> We should probably have raised this question earlier and had a plan in
> place, but hey, better late ... The question is: which browsers should we
> be using as a reference for QA purposes? Our guideline here is decent
> HTML5 compatibility, but we never qualified what 'decent' means.
>
> The other reference we could use is traffic to the Yocto Project website.
> Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
> and 27 on Linux. I can put together more detailed numbers if anybody wants
> to see them.
>
> Those might be a bit too cutting edge, but could guide our decision
> somehow. QA is currently testing with Firefox 11: that is probably too
> old.
>
> In light of the above, any suggestions about which browsers we should use
> for testing?
>
> Thanks!
>
> Belén
>
>


-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 3700 bytes --]

Re: Reference browser for testing purposes

[Yuan Sun]

[-- Attachment #1: Type: text/plain, Size: 2971 bytes --]


Hi all,
     From a tester's views, I choose Ubuntu 12.04 as a test host. Firefox 11.0 
is a default web browser in Ubuntu 12.04. Ubuntu 12.04 is a long-term support 
release. It has continuous hardware support improvements as well as guaranteed 
security and support updates until April 2017. However, Ubuntu 13.10(the latest 
version) will be supported for *ONLY* 9 months. So I think that Ubuntu 12.04 is 
a popular version and so many users would use it. Choose it as a test 
environment would covers a  great percentage of users.
     Maybe we can create a recommendation list of web-browser for customers so 
that it can lead to good user experience.
     Thanks.
                 Yuan

On 14-03-20 02:14 AM, Damian, Alexandru wrote:
>
> Our goal is not "decent" but complete HTML5 compatibility.
>
> The target is that our HTML output is to be validated by HTML5 validators with 
> no errors shown. We already selected the industry-standard HTML5 validators to 
> verify this.
> Specifically, we are using in development http://validator.w3.org/ through a 
> browser extension. This MUST be automated at a certain point.
>
> What I'm trying to avoid here is coding specifically for a target browser or 
> platform. I suggest to not restrict testing to a certain 
> browser/platform/version, but use what ever the tester uses in real life.
> In case of presentation bugs are discovered, first we have to rule out an 
> issue with the browser of choice by testing visual reproducibility with 
> another browser on the same page and verifying browser's HTML5 compatibility.
>
> For visual reference, widgets in the page change across different platforms 
> and browsers. Do we have test cases for the appearance ?
>
> The test cases should not be dependent or executed with a specified 
> platform/browser version.
>
> Cheers,
> Alex
>
>
>
> On Wed, Mar 19, 2014 at 4:23 PM, Barros Pena, Belen 
> <belen.barros.pena@intel.com <mailto:belen.barros.pena@intel.com>> wrote:
>
>     We should probably have raised this question earlier and had a plan in
>     place, but hey, better late ... The question is: which browsers should we
>     be using as a reference for QA purposes? Our guideline here is decent
>     HTML5 compatibility, but we never qualified what 'decent' means.
>
>     The other reference we could use is traffic to the Yocto Project website.
>     Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
>     and 27 on Linux. I can put together more detailed numbers if anybody wants
>     to see them.
>
>     Those might be a bit too cutting edge, but could guide our decision
>     somehow. QA is currently testing with Firefox 11: that is probably too
>     old.
>
>     In light of the above, any suggestions about which browsers we should use
>     for testing?
>
>     Thanks!
>
>     Belén
>
>
>
>
> -- 
> Alex Damian
> Yocto Project
> SSG / OTC


[-- Attachment #2: Type: text/html, Size: 6021 bytes --]

Re: Reference browser for testing purposes

[Barros Pena, Belen]

On 19/03/2014 18:14, "Damian, Alexandru" <alexandru.damian@intel.com>
wrote:

>
>
>Our goal is not "decent" but complete HTML5 compatibility.
>
>
>
>The target is that our HTML output is to be validated by HTML5 validators
>with no errors shown. We already selected the industry-standard HTML5
>validators to verify this.

Validating is a great and necessary thing, but does not address
compatibility. The markup might be valid, but the browser rendering it
might not support some of its features


>
>Specifically, we are using in development
>http://validator.w3.org/ through a browser extension. This MUST be
>automated at a certain point.
>
>
>What I'm trying to avoid here is coding specifically for a target browser
>or platform. I suggest to not restrict testing to a certain
>browser/platform/version, but use what ever the tester uses in real life.

I am not sure this is the right approach, to be honest. What the tester
uses might be completely out of sync with the technology and goals of the
application we are developing.

>In case of presentation bugs are discovered, first we have to rule out an
>issue with the browser of choice by testing visual reproducibility with
>another browser on the same page and verifying browser's HTML5
>compatibility.

In order to address compatibility, we need to clarify what we mean by it.
Compatibility with what: just with the markup language, or with all or
some of the technologies that make html5? If we decide that with the
markup language only (like so http://caniuse.com/#cats=HTML5), and
cross-checking with the traffic data from yoctoproject.org, we should
probably be testing with Firefox 26 and Chrome 31. But bearing in mind
that Toaster might be used with oldish distros (Yuan's point about Ubuntu
12.04 I think is a valid one), maybe we should push down the versions a
bit farther.

>
>
>For visual reference, widgets in the page change across different
>platforms and browsers. Do we have test cases for the appearance ?

I think the criteria here is not the presentation, but if the UI
components are functional (for example, can I select and deselect columns
using the Edit columns widget, or can I toggle the errors and warnings
content, not if the labels are not correctly aligned).

>
>
>The test cases should not be dependent or executed with a specified
>platform/browser version.

I am sorry, but I disagree: I believe we need some kind of reference.

>
>
>Cheers,
>Alex
> 
>
>
>
>
>On Wed, Mar 19, 2014 at 4:23 PM, Barros Pena, Belen
><belen.barros.pena@intel.com> wrote:
>
>We should probably have raised this question earlier and had a plan in
>place, but hey, better late ... The question is: which browsers should we
>be using as a reference for QA purposes? Our guideline here is decent
>HTML5 compatibility, but we never qualified what 'decent' means.
>
>The other reference we could use is traffic to the Yocto Project website.
>Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
>and 27 on Linux. I can put together more detailed numbers if anybody wants
>to see them.
>
>Those might be a bit too cutting edge, but could guide our decision
>somehow. QA is currently testing with Firefox 11: that is probably too
>old.
>
>In light of the above, any suggestions about which browsers we should use
>for testing?
>
>Thanks!
>
>Belén
>
>
>
>
>
>
>
>
>-- 
>Alex Damian
>Yocto Project
>
>SSG / OTC 
>
>

Re: Reference browser for testing purposes

[Barros Pena, Belen]

Just in case is useful, a list of the supported distros and their browsers
(the ones I could find):

* Ubuntu 12.04 (LTS) - Firefox 11.0

* Ubuntu 12.10 - Firefox 16.0.1

* Ubuntu 13.04 - Firefox 20.0

* Fedora release 18 (Spherical Cow) - Firefox 17.01

* Fedora release 19 (Schrödinger's Cat) - Firefox 21.0

* CentOS release 6.4 - ?

* Debian GNU/Linux 6.0.7 (Squeeze) - ?

* Debian GNU/Linux 7.0 (Wheezy) - Chromium 26.0.1410.43 and Firefox 10.0.12

* Debian GNU/Linux 7.1 (Wheezy) - ?

* openSUSE 12.2 - Firefox 14.0.1

* openSUSE 12.3 - Firefox 19.0





On 20/03/2014 11:09, "Barros Pena, Belen" <belen.barros.pena@intel.com>
wrote:

>On 19/03/2014 18:14, "Damian, Alexandru" <alexandru.damian@intel.com>
>wrote:
>
>>
>>
>>Our goal is not "decent" but complete HTML5 compatibility.
>>
>>
>>
>>The target is that our HTML output is to be validated by HTML5 validators
>>with no errors shown. We already selected the industry-standard HTML5
>>validators to verify this.
>
>Validating is a great and necessary thing, but does not address
>compatibility. The markup might be valid, but the browser rendering it
>might not support some of its features
>
>
>>
>>Specifically, we are using in development
>>http://validator.w3.org/ through a browser extension. This MUST be
>>automated at a certain point.
>>
>>
>>What I'm trying to avoid here is coding specifically for a target browser
>>or platform. I suggest to not restrict testing to a certain
>>browser/platform/version, but use what ever the tester uses in real life.
>
>I am not sure this is the right approach, to be honest. What the tester
>uses might be completely out of sync with the technology and goals of the
>application we are developing.
>
>>In case of presentation bugs are discovered, first we have to rule out an
>>issue with the browser of choice by testing visual reproducibility with
>>another browser on the same page and verifying browser's HTML5
>>compatibility.
>
>In order to address compatibility, we need to clarify what we mean by it.
>Compatibility with what: just with the markup language, or with all or
>some of the technologies that make html5? If we decide that with the
>markup language only (like so http://caniuse.com/#cats=HTML5), and
>cross-checking with the traffic data from yoctoproject.org, we should
>probably be testing with Firefox 26 and Chrome 31. But bearing in mind
>that Toaster might be used with oldish distros (Yuan's point about Ubuntu
>12.04 I think is a valid one), maybe we should push down the versions a
>bit farther.
>
>>
>>
>>For visual reference, widgets in the page change across different
>>platforms and browsers. Do we have test cases for the appearance ?
>
>I think the criteria here is not the presentation, but if the UI
>components are functional (for example, can I select and deselect columns
>using the Edit columns widget, or can I toggle the errors and warnings
>content, not if the labels are not correctly aligned).
>
>>
>>
>>The test cases should not be dependent or executed with a specified
>>platform/browser version.
>
>I am sorry, but I disagree: I believe we need some kind of reference.
>
>>
>>
>>Cheers,
>>Alex
>> 
>>
>>
>>
>>
>>On Wed, Mar 19, 2014 at 4:23 PM, Barros Pena, Belen
>><belen.barros.pena@intel.com> wrote:
>>
>>We should probably have raised this question earlier and had a plan in
>>place, but hey, better late ... The question is: which browsers should we
>>be using as a reference for QA purposes? Our guideline here is decent
>>HTML5 compatibility, but we never qualified what 'decent' means.
>>
>>The other reference we could use is traffic to the Yocto Project website.
>>Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
>>and 27 on Linux. I can put together more detailed numbers if anybody
>>wants
>>to see them.
>>
>>Those might be a bit too cutting edge, but could guide our decision
>>somehow. QA is currently testing with Firefox 11: that is probably too
>>old.
>>
>>In light of the above, any suggestions about which browsers we should use
>>for testing?
>>
>>Thanks!
>>
>>Belén
>>
>>
>>
>>
>>
>>
>>
>>
>>-- 
>>Alex Damian
>>Yocto Project
>>
>>SSG / OTC 
>>
>>
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: Reference browser for testing purposes

[Paul Eggleton]

On Friday 28 March 2014 12:12:22 Barros Pena, Belen wrote:
> >>On Wed, Mar 19, 2014 at 4:23 PM, Barros Pena, Belen
> >><belen.barros.pena@intel.com> wrote:
> >>
> >>We should probably have raised this question earlier and had a plan in
> >>place, but hey, better late ... The question is: which browsers should we
> >>be using as a reference for QA purposes? Our guideline here is decent
> >>HTML5 compatibility, but we never qualified what 'decent' means.
> >>
> >>The other reference we could use is traffic to the Yocto Project website.
> >>Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
> >>and 27 on Linux. I can put together more detailed numbers if anybody
> >>wants to see them.
> >>
> >>Those might be a bit too cutting edge, but could guide our decision
> >>somehow. QA is currently testing with Firefox 11: that is probably too
> >>old.
> >>
> >>In light of the above, any suggestions about which browsers we should use
> >>for testing?
>
> Just in case is useful, a list of the supported distros and their browsers
> (the ones I could find):
> 
> * Ubuntu 12.04 (LTS) - Firefox 11.0
> 
> * Ubuntu 12.10 - Firefox 16.0.1
> 
> * Ubuntu 13.04 - Firefox 20.0

We'll need to add Ubuntu 13.10 here, and probably drop Ubuntu 12.10

> * Fedora release 18 (Spherical Cow) - Firefox 17.01

Fedora 18 is out of support so we'll be dropping it for 1.6.

> * Fedora release 19 (Schrödinger's Cat) - Firefox 21.0

We'll need to add Fedora 20 (Firefox 28.0 here).
 
> * CentOS release 6.4 - ?
>
> * Debian GNU/Linux 6.0.7 (Squeeze) - ?

I wouldn't bother with these two to be honest.
 
> * Debian GNU/Linux 7.0 (Wheezy) - Chromium 26.0.1410.43 and Firefox 10.0.12
> 
> * Debian GNU/Linux 7.1 (Wheezy) - ?
> 
> * openSUSE 12.2 - Firefox 14.0.1

Drop openSUSE 12.2, we should be dropping it from our support list
 
> * openSUSE 12.3 - Firefox 19.0
 
We probably want openSUSE 13.1 as well.

(FWIW I've just sent a note to QA to get our tested distro list updated.)

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Reference browser for testing purposes

[Gary Thomas]

On 2014-03-28 12:41, Paul Eggleton wrote:
> On Friday 28 March 2014 12:12:22 Barros Pena, Belen wrote:
>>>> On Wed, Mar 19, 2014 at 4:23 PM, Barros Pena, Belen
>>>> <belen.barros.pena@intel.com> wrote:
>>>>
>>>> We should probably have raised this question earlier and had a plan in
>>>> place, but hey, better late ... The question is: which browsers should we
>>>> be using as a reference for QA purposes? Our guideline here is decent
>>>> HTML5 compatibility, but we never qualified what 'decent' means.
>>>>
>>>> The other reference we could use is traffic to the Yocto Project website.
>>>> Visits are mainly coming from Chrome 32 and 33 on Windows, and Firefox 26
>>>> and 27 on Linux. I can put together more detailed numbers if anybody
>>>> wants to see them.
>>>>
>>>> Those might be a bit too cutting edge, but could guide our decision
>>>> somehow. QA is currently testing with Firefox 11: that is probably too
>>>> old.
>>>>
>>>> In light of the above, any suggestions about which browsers we should use
>>>> for testing?
>>
>> Just in case is useful, a list of the supported distros and their browsers
>> (the ones I could find):
>>
>> * Ubuntu 12.04 (LTS) - Firefox 11.0

Ubuntu 12.04 is actually on Firefox 28.0  (normal fully updated system)

You also don't seem to have much representation of Google Chrome which
is one of the more common optional browsers, currently at 33.0.1750.152

Even if 12.04 is LTS, some things _are_ kept fairly current :-)

>>
>> * Ubuntu 12.10 - Firefox 16.0.1
>>
>> * Ubuntu 13.04 - Firefox 20.0
> 
> We'll need to add Ubuntu 13.10 here, and probably drop Ubuntu 12.10
> 
>> * Fedora release 18 (Spherical Cow) - Firefox 17.01
> 
> Fedora 18 is out of support so we'll be dropping it for 1.6.
> 
>> * Fedora release 19 (Schrödinger's Cat) - Firefox 21.0
> 
> We'll need to add Fedora 20 (Firefox 28.0 here).
>  
>> * CentOS release 6.4 - ?
>>
>> * Debian GNU/Linux 6.0.7 (Squeeze) - ?
> 
> I wouldn't bother with these two to be honest.
>  
>> * Debian GNU/Linux 7.0 (Wheezy) - Chromium 26.0.1410.43 and Firefox 10.0.12
>>
>> * Debian GNU/Linux 7.1 (Wheezy) - ?
>>
>> * openSUSE 12.2 - Firefox 14.0.1
> 
> Drop openSUSE 12.2, we should be dropping it from our support list
>  
>> * openSUSE 12.3 - Firefox 19.0
>  
> We probably want openSUSE 13.1 as well.
> 
> (FWIW I've just sent a note to QA to get our tested distro list updated.)
> 
> Cheers,
> Paul
> 

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Reference browser for testing purposes

[Barros Pena, Belen]

On 28/03/2014 18:56, "Gary Thomas" <gary@mlbassoc.com> wrote:

>>>* Ubuntu 12.04 (LTS) - Firefox 11.0
>
>Ubuntu 12.04 is actually on Firefox 28.0  (normal fully updated system)
>
>You also don't seem to have much representation of Google Chrome which
>is one of the more common optional browsers, currently at 33.0.1750.152
>
>Even if 12.04 is LTS, some things _are_ kept fairly current :-)

Thanks, Gary. This is very good news: I was having nightmares about FF11 ;)

Cheers

Belén