Advertising the Git User's Survey 2011

Advertising the Git User's Survey 2011

[Ævar Arnfjörð Bjarmason]

On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:

> Currenly the survey has more than 5000 responses (in a not whole month)

That seems somewhat low, but maybe people just aren't that interested
in taking surveys.

This may have to do with advertising however. Have you contacted sites
like GitHub, Gitorious etc. asking them to advertise it? Is there
maybe a list of people that have been contacted somewhere?

I bet if we could get e.g. GitHub to advertise it in their regular
news distribution system we could raise that number of 5000
respondents significantly.

Re: Advertising the Git User's Survey 2011

[Jacob Helwig]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

On Wed, 22 Sep 2010 18:57:35 +0000, Ævar Arnfjörð Bjarmason wrote:
> 
> On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > Currenly the survey has more than 5000 responses (in a not whole month)
> 
> That seems somewhat low, but maybe people just aren't that interested
> in taking surveys.
> 
> This may have to do with advertising however. Have you contacted sites
> like GitHub, Gitorious etc. asking them to advertise it? Is there
> maybe a list of people that have been contacted somewhere?

You mean, this?

http://github.com/blog/717-git-user-survey-2010

> 
> I bet if we could get e.g. GitHub to advertise it in their regular
> news distribution system we could raise that number of 5000
> respondents significantly.

-- 
Jacob Helwig

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 665 bytes --]

Re: Advertising the Git User's Survey 2011

[Ævar Arnfjörð Bjarmason]

On Wed, Sep 22, 2010 at 19:12, Jacob Helwig <jacob.helwig@gmail.com> wrote:
> On Wed, 22 Sep 2010 18:57:35 +0000, Ævar Arnfjörð Bjarmason wrote:
>>
>> On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
>>
>> > Currenly the survey has more than 5000 responses (in a not whole month)
>>
>> That seems somewhat low, but maybe people just aren't that interested
>> in taking surveys.
>>
>> This may have to do with advertising however. Have you contacted sites
>> like GitHub, Gitorious etc. asking them to advertise it? Is there
>> maybe a list of people that have been contacted somewhere?
>
> You mean, this?
>
> http://github.com/blog/717-git-user-survey-2010

Ah, I hadn't seen it because GitHub Jobs advertisements hide the
regular GitHub broadcasts, but if I dismiss those I can see it. That's
some new "feature" of their system I wasn't taking into account.

Re: Advertising the Git User's Survey 2011

[Jakub Narebski]

On Wed, 22 Sep 2010, Ævar Arnfjörð Bjarmason wrote:
> On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > Currenly the survey has more than 5000 responses (in a not whole month)

6217 responses in 45 days now, see 
  https://www.survs.com/results/33Q0OZZE/MV653KSPI2

> 
> That seems somewhat low, but maybe people just aren't that interested
> in taking surveys.

That might be it.
 
> This may have to do with advertising however. Have you contacted sites
> like GitHub, Gitorious etc. asking them to advertise it? Is there
> maybe a list of people that have been contacted somewhere?
> 
> I bet if we could get e.g. GitHub to advertise it in their regular
> news distribution system we could raise that number of 5000
> respondents significantly.

It is announced both on GitHub Blog and in GitHub announcements, see
  http://github.com/blog/717-git-user-survey-2010
  http://img.skitch.com/20100914-q4wuyf6i8f2yt9ef2tnxaifc1a.png

Besides that, it is announced on Git Wiki, on Git Homepage, and also on
repo.or.cz, git.kernel.org, GitHub, Gitorious, InDefero, Codaset,
Codebase, Unfuddle, Savannah, SourceForge and Beanstalk.  It would be
(or is) announced on Bettercodes.org.  I have send request for 
announcement also to Codesion, Assembla, The Chaw, ProjectLocker,
CipherHive and GitFarm, but I didn't get any response yet (I might
not found and used correct contact information).

I am still sending announcements to other git hosting sites...
-- 
Jakub Narebski
Poland

Re: Advertising the Git User's Survey 2010 - report

[Jakub Narebski]

On Wed, 22 Sep 2010, Jakub Narębski wrote:
> On Wed, 22 Sep 2010, Ævar Arnfjörð Bjarmason wrote:
> > On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
> > 
> > > Currenly the survey has more than 5000 responses (in a not whole month)
> 
> 6217 responses in 22 days now, see 
>   https://www.survs.com/results/33Q0OZZE/MV653KSPI2

[...]
> > This may have to do with advertising however. Have you contacted sites
> > like GitHub, Gitorious etc. asking them to advertise it? Is there
> > maybe a list of people that have been contacted somewhere?
> > 
> > I bet if we could get e.g. GitHub to advertise it in their regular
> > news distribution system we could raise that number of 5000
> > respondents significantly.
> 
> It is announced both on GitHub Blog and in GitHub announcements, see
>   http://github.com/blog/717-git-user-survey-2010
>   http://img.skitch.com/20100914-q4wuyf6i8f2yt9ef2tnxaifc1a.png
> 
> Besides that, it is announced on Git Wiki, on Git Homepage, and also on
> repo.or.cz, git.kernel.org, GitHub, Gitorious, InDefero, Codaset,
> Codebase, Unfuddle, Savannah, SourceForge and Beanstalk.  It would be
> (or is) announced on Bettercodes.org.  I have send request for 
> announcement also to Codesion, Assembla, The Chaw, ProjectLocker,
> CipherHive and GitFarm, but I didn't get any response yet (I might
> not found and used correct contact information).
> 
> I am still sending announcements to other git hosting sites...

The Git User's Survey 2010 was announced on the following channels:

1. Mailing lists:
   * git mailing list
   * LKML (linux kernel mailing list), as the Linux kernel development
     was the inspiration for creating Git, and it is major project using
     Git as version control system
   * Ævar forwarded announcement to perl5-porters mailing list
   * I have also send announcement to (developers') mailing list for
     various projects using Git.  Many of them are subscribe-only and
     are awaiting moderation, some  outright refused non-subscribe
     contributions.  If you plan announcing Git User's Survey 2010
     in such channel, plese check if it wasn't already sent.

2. Google Groups:
   * "Git for Human Beings" and msysGit Google Groups cover git-related
     mailing list channel
   * I have also send announcements to GitHub and Gitorious Google Groups.

3. The Git User's Survey 2010 is announced on Git homepage, on Git Wiki,
   and in the #git IRC channel topic (channel description)

4. Blogs (not counting blogs for Git hosting sites)
   * gitster's journal (blog of Junio C Hamano)
     http://gitster.livejournal.com/47343.html
   * git blog - A git blog looking to be official
     http://gitlog.wordpress.com/2010/09/01/git-user%E2%80%99s-survey-2010/
   * and probably other blogs (many thanks to bloggers who posted 
     announcements about Git User's Survey 2010)
   - I asked to have it announced on Perlbuzz, and therefore on Perl blog
     planets, and on Newren's blog, and therefore on GNOME planets, but
     without results

5. Social news sites:
   * Hacker News (posted by Ævar)
     http://news.ycombinator.com/item?id=1652126
   * Reddit programming (posted by Ævar)
     http://www.reddit.com/r/programming/comments/d835u/git_users_survey_2010/

6. Git hosting sites and software hosting sites with Git support
   * Those that replied to my request for announcing the survey,
     in alphabetical order
     - A2 Hosting
     - Beanstalk
     - BerliOS
     - Bettercodes
     - CodaSet
     - Codebase
     - GitHub
     - Gitorious
     - InDefero
     - git.kernel.org
     - repo.or.cz
     - Savannah
     - SourceForge
     - SourceRepo
     - Unfuddle
     - XP-Dev
   * Those that didn't reply, and I don't think they posted announcement,
     in alphabetical order
     - Assembla
     - The Chaw (posted on its Google Group, though)
     - CipherHive
     - Codesion
     - GitFarm
     - ProjectLocker
     - repositoryhosting.com
     - SSH Control
     - Tux Family
   * Those that I didn't know how to contact,
     in alphabetical order
     - Alioth / git.debian.org
     - debian-hosted
     - Fedora Hosted
     - Project Kenai (if it is still active)
     - USLA-Tracs (not in English)

   If there are Git hosting sites that are missing from the above list,
   that didn't already announced Git User's Survey 2010 somehow, and you
   know how to contact them, please try.  There are 2 weeks of survey
   yet.

7. Microblogging platforms: I announced Git User's Survey 2010 on Twitter
   and Plurk, but I don't have much of following.  Some of git hosting
   sites also announced survey via Twitter.

8. Other: I have tried to announce the survey for Debian developers.
   In previous years this announcement was relayed to debian-devel-announce
   mailing list through http://wiki.debian.org/DeveloperNews wiki page,
   e.g.:
     http://lists.debian.org/debian-devel-announce/2009/08/msg00002.html
     http://lists.debian.org/debian-devel-announce/2008/09/msg00003.html
     http://wiki.debian.org/ProjectNews/Issues/2008/11

   But unfortunately not this year.


Are there channels that should be utilized missing from the above list?
How next year Git User's Survey 2011 (if there would be one) should be
announced?

P.S. There are 7598 non-test responses to the survey now, see
     https://www.survs.com/results/33Q0OZZE/MV653KSPI2

-- 
Jakub Narebski
Poland

Re: Advertising the Git User's Survey 2010 - report

[Nguyen Thai Ngoc Duy]

On 9/30/10, Jakub Narebski <jnareb@gmail.com> wrote:
>  Are there channels that should be utilized missing from the above list?
>  How next year Git User's Survey 2011 (if there would be one) should be
>  announced?

>From the report, 77% answers question 21 as "this is my first survey",
I think you have done a fantastic job.

By the way, for the next surveys maybe you should list git-lol [1] as
a Git "GUI". I rarely use gitk. Ever since I discovered git-lol, I
don't know if I need gitk again.

[1] http://andyjeffries.co.uk/articles/25-tips-for-intermediate-git-users
(13. Viewing a Log)
-- 
Duy

Re: Advertising the Git User's Survey 2010 - report

[Junio C Hamano]

Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:

> On 9/30/10, Jakub Narebski <jnareb@gmail.com> wrote:
>>  Are there channels that should be utilized missing from the above list?
>>  How next year Git User's Survey 2011 (if there would be one) should be
>>  announced?
>
> From the report, 77% answers question 21 as "this is my first survey",
> I think you have done a fantastic job.

That, or (more likely) these people have started to seriously look at git
within the past year.  There doesn't seem to be a more direct "How long
have you been using git?" question, but look at #22 as a proxy.  54% of
the people answer they cannot compare the current version of git and the
one from one year ago.

Re: Advertising the Git User's Survey 2010 - report

[Štěpán Němec]

Junio C Hamano <gitster@pobox.com> writes:

> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:
>
>> On 9/30/10, Jakub Narebski <jnareb@gmail.com> wrote:
>>>  Are there channels that should be utilized missing from the above list?
>>>  How next year Git User's Survey 2011 (if there would be one) should be
>>>  announced?
>>
>> From the report, 77% answers question 21 as "this is my first survey",
>> I think you have done a fantastic job.
>
> That, or (more likely) these people have started to seriously look at git
> within the past year.  There doesn't seem to be a more direct "How long
> have you been using git?" question, but look at #22 as a proxy.  54% of
> the people answer they cannot compare the current version of git and the
> one from one year ago.

Heh... I actually misunderstood that question as asking about the
_survey_, not the Git version... I suspect I'm not the only one, so
please make the question less ambiguous in future versions. :-)

Re: Advertising the Git User's Survey 2011

[Ramkumar Ramachandra]

Hi Jakub and Ævar,

Ævar Arnfjörð Bjarmason writes:
> On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > Currenly the survey has more than 5000 responses (in a not whole month)
> 
> That seems somewhat low, but maybe people just aren't that interested
> in taking surveys.

Nice work on the survey! This is our best year by far. Some general
observations:

Interesting statistic: 24508 people viewed it, 7821 people completed
it, but 0 people started filling out information and later decided not
to submit it. It could mean that many people clicked through and found
the survey, but probably left because it looked too long at a glance?
The average time spent on the survery is 34 minutes
- I think we can bring that down to 10~15 minutes if we design
questions to extract more information. Also, there's little incentive
for taking the survey: while many companies actually give out
discounts/ coupons for taking surveys, the least we can do is present
real-time results in the most interesting manner possible ie. survey
takers should see the "results so far" immediately after taking the
survey; some visualizations such as pie charts?

In questions 5, 10, 12, 13, 16, cut down out the options that have
very few respondents and let them all go into "other". It probably
doesn't actually save the survey taker any time, but I think seeing a
long page with many options can be scary.

1. Country of residence: we can probably make this a nice click-on-map
   interface as opposed to freeform text. It'll be more useful to us,
   and more interesting to users when we advertise the results.
2. Age: Maybe we restrict the input to 2-digit integers and draw a
   graph with all these integers to show a mean, median etc?
11. Just change this to an optional sometimes/ often? Why should users
    spend time clicking on "never"?
17, 18: Merge perhaps?
24, 25: Merge into single question with options: "Yes, and my problems
        are solved more often than not", "Yes, but my problems often
        remain unresolved", "No, I don't go to others for help".
26, 27: Merge into "How do you talk to other people using Git, either
        for technical help or otherwise?"

Ofcourse, I understand that there must be some technical constraints
due to which some things are not implementable (eg. survs doesn't
provide the feature?), but I've not taken that into consideration.

-- Ram

Re: Advertising the Git User's Survey 2011

[Jakub Narebski]

On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:
> Ævar Arnfjörð Bjarmason writes:
> > On Mon, Sep 20, 2010 at 13:38, Jakub Narebski <jnareb@gmail.com> wrote:
> > 
> > > Currenly the survey has more than 5000 responses (in a not whole month)
> > 
> > That seems somewhat low, but maybe people just aren't that interested
> > in taking surveys.
> 
> Nice work on the survey! This is our best year by far. Some general
> observations:
> 
> Interesting statistic: 24508 people viewed it, 7821 people completed
> it, but 0 people started filling out information and later decided not
> to submit it. It could mean that many people clicked through and found
> the survey, but probably left because it looked too long at a glance?

Without the knowledge how those two numbers are calculated we can only
speculate what do they mean.

I think that '0' in 'Incomplete' statistics is here because this survey
doesn't have compulsory questions: answering all questions are optional,
so it might mean that even if one question in the survey is answered,
then the survey is considered complete by Survs.com statistics.

I personally do not like the "wizard" formatting of surveys, i.e. 
dividing survey into page so you are not presented with very long page,
but are presented withc chunks of survey at glance.  Even if you see
how much survey did you fill in (how many pages there are in total),
and even if you can go back to previous page.

I'd prefer to create a better information about survey upfront.  We say
that all questions are optional ("Note that you may skip questions as
you like").  It is also stated that you can fill only a part of survey,
and later go back to finish it... hmmm, I wonder if those cases where
one edited his/her survey responses multiple times are counted as one
finished survey, but multiple views.

We could also write how much time it takes on average to fil the survey.

> The average time spent on the survery is 34 minutes

It would be interesting to have more detailed statistics of time spent
on the survey that only average time, a single number.  When one is
filling open-form essay-length question, it would obviously take much
more time than for one who doesn't.

But Survs.com currently doesn't provide it.  I can try to ask for it,
though (via feedback).

> - I think we can bring that down to 10~15 minutes if we design
> questions to extract more information. Also, there's little incentive
> for taking the survey: while many companies actually give out
> discounts/ coupons for taking surveys, the least we can do is present
> real-time results in the most interesting manner possible ie. survey
> takers should see the "results so far" immediately after taking the
> survey; some visualizations such as pie charts?

What we can do is after finishing the survey to redirect to the
survey analysis page:

  https://www.survs.com/results/33Q0OZZE/MV653KSPI2
  http://tinyurl.com/GitSurvey2010Analysis

instead of IIRC currently used redirect to

  https://git.wiki.kernel.org/index.php/GitSurvey2010


As far as I know Survs.com doesn't provide any API for extracting data
or survey statistics required for creating such visualization.  Neither
we have a place where such app could be created, I think.

> 
> In questions 5, 10, 12, 13, 16, cut down out the options that have
> very few respondents and let them all go into "other". It probably
> doesn't actually save the survey taker any time, but I think seeing a
> long page with many options can be scary.

The "5. Which Git version(s) are you using?" is not that long.  We could
create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or
even earlier, we could remove alternate implementations answers 
(git-bigfiles, JGit, other implementations), or even concatenate 'master',
'next', 'pu' into single response... but would it buy us much?

The other side of removing choices, relying instead on "other, please
specify" response is that it makes it harder to analyze results of survey:
different people use different words for the same thing (and there are
also spelling mistakes), and make results less reliable: people do not
fill "other" if there is at least partial match, or do not know how to
specify their version.


In "10. What Git interfaces, implementations and frontends do you use?"
we can remove tools marked as deprecated... if not for the fact that it
is actively interesting to know how many people use such deprecated
tools.  Besides, the list of answers to this question is not overly long,
I don't think.


We can remove those choices in "12. What Git GUIs (graphical user
interfaces) do you use?" that got less than 1% rounded, or less than
10 responses.  On the other hand some people stated earlier that the
list of possible choices in the survey (not necessarily about this
question in specific) serve as reminder / information about possible
choices.


The other side of removing options from "13. Which git hosting site(s)
do you use for your project(s)?" is that when sending requests to 
announce the survey to those git hosting sites that are not on this
list, some of them requested to be added (which is impossible after
starting the survey; and before survey begins it is little sense to
send announcements).

Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
CipherHive) are also those that I didn't get response to request for
announcing Git User's Survey 2010...
 
We could make it more organized though, e.g. by sorting list of options
alphabetically, or something like that.


Removing options from "16. Which of the following features do you use?"
would make it harder to analyze and less reliable.  Especially in this
question different people consider different features important enough
to mention, and describe feature in many different ways.

Besides, each option except of "git cvsserver" got more than 1% rounded,
and having "git cvsserver" is interesting on its own (perhaps in other
question?).

> 1. Country of residence: we can probably make this a nice click-on-map
>    interface as opposed to freeform text. It'll be more useful to us,
>    and more interesting to users when we advertise the results.

It would be nice to have click-on-map (Google Maps or Bing Maps based),
something like Ohloh provides, resulting in map of survey responders
similar to the map of git users and git contributors on Ohloh

  http://www.ohloh.net/p/git/map

it isn't something that Survs.com offers currently.  I can only ask for
it to be provided...


Another solution would be to have pre-filled combo box (<select> field)
with the list of countries to choose from, with GeoIP ised to pre-select
the country.  I can generate list of all countries myself

  $ perl -MLocale::Country \
    -wle 'print join("\n", sort (all_country_names()))'

as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
to survey questions.

> 2. Age: Maybe we restrict the input to 2-digit integers and draw a
>    graph with all these integers to show a mean, median etc?

Restricting input doesn't give us much.

There is nice histogram of responder's age for Git User's Survey 2009

  https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F

and tabularization of responses.  We can calculate mean, median, mode
(aka modal score, i.e. most common response), perhaps after eliminating
outliers, but would it give us much information?

> 11. Just change this to an optional sometimes/ often? Why should users
>     spend time clicking on "never"?

"Never" is here because you can't un-click response in given row.
I'm also not sure how not answering row is represented in the export
of survey data (which we use for further analysis).

> 17, 18: Merge perhaps?

Those questions are split because of limitation of Survs.com; the 
"other, please specify" results in limited width _text field_, while
it is much easier to write in-depth response in large _textarea_ field
that wuestion 18 provides.

> 24, 25: Merge into single question with options: "Yes, and my problems
>         are solved more often than not", "Yes, but my problems often
>         remain unresolved", "No, I don't go to others for help".

Good idea.  Will do.

> 26, 27: Merge into "How do you talk to other people using Git, either
>         for technical help or otherwise?"

I don't think it is good idea.  Those two issues, namely requesting
help and being (perhaps silent) part of git community are two unrelated
issues.

> Ofcourse, I understand that there must be some technical constraints
> due to which some things are not implementable (eg. survs doesn't
> provide the feature?), but I've not taken that into consideration.

Note that as it currently stands we can use Survs.com account only for
2011 survey, provided that it is done earlier than this year (perhaps
1 June -- 31 July?), as our Premium account which we got thanks to 
generosity of Survs.com admins (after Survs.com got out of beta) will
downgrade to the Free plan (which is offersn much too low limits)
on Sep 22, 2011.

-- 
Jakub Narebski
Poland

Re: Advertising the Git User's Survey 2010 - report

[Jakub Narebski]

Dnia sobota 2. października 2010 03:07, Štěpán Němec napisał:
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:
>>
>>> On 9/30/10, Jakub Narebski <jnareb@gmail.com> wrote:
>>>>  Are there channels that should be utilized missing from the above list?
>>>>  How next year Git User's Survey 2011 (if there would be one) should be
>>>>  announced?
>>>
>>> From the report, 77% answers question 21 as "this is my first survey",
>>> I think you have done a fantastic job.
>>
>> That, or (more likely) these people have started to seriously look at git
>> within the past year.  There doesn't seem to be a more direct "How long
>> have you been using git?" question, but look at #22 as a proxy.  54% of
>> the people answer they cannot compare the current version of git and the
>> one from one year ago.

Hmmm... such question was removed from the survey, but perhaps we should
re-introduce it, perhaps in tabularized (1 week, 1 month, few months,
year, few years, 5 years or more) rather than free-form question?

> 
> Heh... I actually misunderstood that question as asking about the
> _survey_, not the Git version... I suspect I'm not the only one, so
> please make the question less ambiguous in future versions. :-)

Will fix.  It will be "How do you compare the current _Git_ version
with the version from one year ago?".  

Putting it just after "21. Did you participate in previous Git User's
Surveys?" probably just increased the confusion, even though below
this question is description what changed in *Git* since year ago.

-- 
Jakub Narebski
Poland

Re: Advertising the Git User's Survey 2011

[Ramkumar Ramachandra]

Hi Jakub,

Thanks for taking the time to go through my long (somewhat
unstructured) note.

Jakub Narebski writes:
> On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:
> > Interesting statistic: 24508 people viewed it, 7821 people completed
> > it, but 0 people started filling out information and later decided not
> > to submit it. It could mean that many people clicked through and found
> > the survey, but probably left because it looked too long at a glance?
> 
> Without the knowledge how those two numbers are calculated we can only
> speculate what do they mean.
> 
> I think that '0' in 'Incomplete' statistics is here because this survey
> doesn't have compulsory questions: answering all questions are optional,
> so it might mean that even if one question in the survey is answered,
> then the survey is considered complete by Survs.com statistics.

I just opened up survs.com, signed up for a free account and created a
survey with two compulsory questions- I couldn't submit the results
without filling up both of them. Then I looked at the statistics and I
saw a '0' in incomplete- I'm confused. Perhaps we can email survs.com
and ask them?

> I personally do not like the "wizard" formatting of surveys, i.e. 
> dividing survey into page so you are not presented with very long page,
> but are presented withc chunks of survey at glance.  Even if you see
> how much survey did you fill in (how many pages there are in total),
> and even if you can go back to previous page.

Same. I also think "wizard" is a bad idea for this- it would make it
look like some sort of online test :p

> I'd prefer to create a better information about survey upfront.  We say
> that all questions are optional ("Note that you may skip questions as
> you like").  It is also stated that you can fill only a part of survey,
> and later go back to finish it... hmmm, I wonder if those cases where
> one edited his/her survey responses multiple times are counted as one
> finished survey, but multiple views.

Sure. Again, we should perhaps email survs.com for such
clarifications.

> We could also write how much time it takes on average to fil the survey.

Sounds good.

> > The average time spent on the survery is 34 minutes
> 
> It would be interesting to have more detailed statistics of time spent
> on the survey that only average time, a single number.  When one is
> filling open-form essay-length question, it would obviously take much
> more time than for one who doesn't.

Hm, yes. I didn't think about this.

> > - I think we can bring that down to 10~15 minutes if we design
> > questions to extract more information. Also, there's little incentive
> > for taking the survey: while many companies actually give out
> > discounts/ coupons for taking surveys, the least we can do is present
> > real-time results in the most interesting manner possible ie. survey
> > takers should see the "results so far" immediately after taking the
> > survey; some visualizations such as pie charts?
> 
> What we can do is after finishing the survey to redirect to the
> survey analysis page:
> 
>   https://www.survs.com/results/33Q0OZZE/MV653KSPI2
>   http://tinyurl.com/GitSurvey2010Analysis
> 
> instead of IIRC currently used redirect to
> 
>   https://git.wiki.kernel.org/index.php/GitSurvey2010
> 
> 
> As far as I know Survs.com doesn't provide any API for extracting data
> or survey statistics required for creating such visualization.  Neither
> we have a place where such app could be created, I think.

Oh, ok- we should take too much trouble though; we should keep
cost-to-benefit ratio in mind.

> > In questions 5, 10, 12, 13, 16, cut down out the options that have
> > very few respondents and let them all go into "other". It probably
> > doesn't actually save the survey taker any time, but I think seeing a
> > long page with many options can be scary.
> 
> The "5. Which Git version(s) are you using?" is not that long.  We could
> create a cut off a bit earlier, perhaps on 1.4 (i.e. have "pre 1.4") or
> even earlier, we could remove alternate implementations answers 
> (git-bigfiles, JGit, other implementations), or even concatenate 'master',
> 'next', 'pu' into single response... but would it buy us much?
> 
> The other side of removing choices, relying instead on "other, please
> specify" response is that it makes it harder to analyze results of survey:
> different people use different words for the same thing (and there are
> also spelling mistakes), and make results less reliable: people do not
> fill "other" if there is at least partial match, or do not know how to
> specify their version.

Ofcourse. We should only remove options that don't necessarily tell us
very much.

> We can remove those choices in "12. What Git GUIs (graphical user
> interfaces) do you use?" that got less than 1% rounded, or less than
> 10 responses.  On the other hand some people stated earlier that the
> list of possible choices in the survey (not necessarily about this
> question in specific) serve as reminder / information about possible
> choices.

Hm, I hadn't thought of that. I suppose it can be used to advertise
various applications.

> The other side of removing options from "13. Which git hosting site(s)
> do you use for your project(s)?" is that when sending requests to 
> announce the survey to those git hosting sites that are not on this
> list, some of them requested to be added (which is impossible after
> starting the survey; and before survey begins it is little sense to
> send announcements).

I see. How are we going to tackle this in future?

> Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
> CipherHive) are also those that I didn't get response to request for
> announcing Git User's Survey 2010...

Interesting.

> We could make it more organized though, e.g. by sorting list of options
> alphabetically, or something like that.

Sounds good.

> > 1. Country of residence: we can probably make this a nice click-on-map
> >    interface as opposed to freeform text. It'll be more useful to us,
> >    and more interesting to users when we advertise the results.
> 
> It would be nice to have click-on-map (Google Maps or Bing Maps based),
> something like Ohloh provides, resulting in map of survey responders
> similar to the map of git users and git contributors on Ohloh
> 
>   http://www.ohloh.net/p/git/map
> 
> it isn't something that Survs.com offers currently.  I can only ask for
> it to be provided...
> 
> 
> Another solution would be to have pre-filled combo box (<select> field)
> with the list of countries to choose from, with GeoIP ised to pre-select
> the country.  I can generate list of all countries myself
> 
>   $ perl -MLocale::Country \
>     -wle 'print join("\n", sort (all_country_names()))'
> 
> as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
> to survey questions.

I suppose we could always work out a way to display the results from
the information Survs.com gives us.

> > 2. Age: Maybe we restrict the input to 2-digit integers and draw a
> >    graph with all these integers to show a mean, median etc?
> 
> Restricting input doesn't give us much.

I just meant it as a sanity check in case people enter "34 years old"
and the like.

> There is nice histogram of responder's age for Git User's Survey 2009
> 
>   https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F
> 
> and tabularization of responses.  We can calculate mean, median, mode
> (aka modal score, i.e. most common response), perhaps after eliminating
> outliers, but would it give us much information?

Nice histogram! How did we manage to do this in 2009? Did we use a
custom-made application to do the survey?

> > 11. Just change this to an optional sometimes/ often? Why should users
> >     spend time clicking on "never"?
> 
> "Never" is here because you can't un-click response in given row.
> I'm also not sure how not answering row is represented in the export
> of survey data (which we use for further analysis).

Oh, right. I didn't think about this.

> > 17, 18: Merge perhaps?
> 
> Those questions are split because of limitation of Survs.com; the 
> "other, please specify" results in limited width _text field_, while
> it is much easier to write in-depth response in large _textarea_ field
> that wuestion 18 provides.

Ah, I thought so.

> > Ofcourse, I understand that there must be some technical constraints
> > due to which some things are not implementable (eg. survs doesn't
> > provide the feature?), but I've not taken that into consideration.
> 
> Note that as it currently stands we can use Survs.com account only for
> 2011 survey, provided that it is done earlier than this year (perhaps
> 1 June -- 31 July?), as our Premium account which we got thanks to 
> generosity of Survs.com admins (after Survs.com got out of beta) will
> downgrade to the Free plan (which is offersn much too low limits)
> on Sep 22, 2011.

I see. Any thoughts on long-term plans? Do we pay for the premium
account or do we build a custom application?

-- Ram

Re: Advertising the Git User's Survey 2011

[Jakub Narebski]

On Sun, 3 Oct 2010, Ramkumar Ramachandra wrote:
> Jakub Narebski writes:
> > On Sat, 2 Oct 2010, Ramkumar Ramachandra wrote:

[...]
> Perhaps we can email survs.com and ask them?

Should I do it, or would you do it?

[...]
> > We could also write how much time it takes on average to fill the survey.
> 
> Sounds good.

Note however that such information is available only after survey is
opened for a bit (unless we use test run statistics).

> > > The average time spent on the survery is 34 minutes

[...]
> > The other side of removing options from "13. Which git hosting site(s)
> > do you use for your project(s)?" is that when sending requests to 
> > announce the survey to those git hosting sites that are not on this
> > list, some of them requested to be added (which is impossible after
> > starting the survey; and before survey begins it is little sense to
> > send announcements).
> 
> I see. How are we going to tackle this in future?
> 
> > Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
> > CipherHive) are also those that I didn't get response to request for
> > announcing Git User's Survey 2010...
> 
> Interesting.

We can always remove (and do not add) those choices for which we don't
get much responses for, and which do not respond to attemts to contact.  
Conversely, we probably should add those git hosting sites that we have
many replies in the "other, please specify".

> > We could make it more organized though, e.g. by sorting list of options
> > alphabetically, or something like that.
> 
> Sounds good.

Will do, perhaps starting with repo.or.cz (first site), GitHub (most
popular), Gitorious and perhaps Gitolite (OSS engine), and ending with
non-generic sites such as git.kernel.org, Alioth, Fedora Hosted, etc.

> > > 1. Country of residence: we can probably make this a nice click-on-map
> > >    interface as opposed to freeform text. It'll be more useful to us,
> > >    and more interesting to users when we advertise the results.
> > 
> > It would be nice to have click-on-map (Google Maps or Bing Maps based),
> > something like Ohloh provides, resulting in map of survey responders
> > similar to the map of git users and git contributors on Ohloh
> > 
> >   http://www.ohloh.net/p/git/map
> > 
> > it isn't something that Survs.com offers currently.  I can only ask for
> > it to be provided...
> > 
> > 
> > Another solution would be to have pre-filled combo box (<select> field)
> > with the list of countries to choose from, with GeoIP ised to pre-select
> > the country.  I can generate list of all countries myself
> > 
> >   $ perl -MLocale::Country \
> >     -wle 'print join("\n", sort (all_country_names()))'
> > 
> > as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
> > to survey questions.
> 
> I suppose we could always work out a way to display the results from
> the information Survs.com gives us.

Do you have any idea how to display such geographical information, and
what tool to use for that visualization?

> > > 2. Age: Maybe we restrict the input to 2-digit integers and draw a
> > >    graph with all these integers to show a mean, median etc?
> > 
> > Restricting input doesn't give us much.
> 
> I just meant it as a sanity check in case people enter "34 years old"
> and the like.

We can quite easily parse this, see below.

> > There is nice histogram of responder's age for Git User's Survey 2009
> > 
> >   https://git.wiki.kernel.org/index.php/GitSurvey2009#02._How_old_are_you_.28in_years.29.3F
> > 
> > and tabularization of responses.  We can calculate mean, median, mode
> > (aka modal score, i.e. most common response), perhaps after eliminating
> > outliers, but would it give us much information?
> 
> Nice histogram! How did we manage to do this in 2009? Did we use a
> custom-made application to do the survey?

I used a Perl script, which uses Text::CSV to parse data exported from
Survs.com in CSV format (and PerlIO::gzip to not have to decompress it).
Each survey page on Git Wiki, except for the very first survey, contains
link to file with such exported data.

For example for age it extracts digits from the response, and assumes
that it is number of years.  It also creates this nice table of ranges
that you can see in the mentioned section of GitSurvey2009 page.

I can publish this script, e.g. the one used for 2009 survey on the
GitSurvey2009 page, but it is rough'n'dirty script.

> > > Of course, I understand that there must be some technical constraints
> > > due to which some things are not implementable (eg. survs doesn't
> > > provide the feature?), but I've not taken that into consideration.
> > 
> > Note that as it currently stands we can use Survs.com account only for
> > 2011 survey, provided that it is done earlier than this year (perhaps
> > 1 June -- 31 July?), as our Premium account which we got thanks to 
> > generosity of Survs.com admins (after Survs.com got out of beta) will
> > downgrade to the Free plan (which is offersn much too low limits)
> > on Sep 22, 2011.
> 
> I see. Any thoughts on long-term plans? Do we pay for the premium
> account or do we build a custom application?

Dunno.

-- 
Jakub Narebski
Poland

Re: Advertising the Git User's Survey 2011

[Ramkumar Ramachandra]

Hi again,

Jakub Narebski writes:
> On Sun, 3 Oct 2010, Ramkumar Ramachandra wrote:
> > Perhaps we can email survs.com and ask them?
> 
> Should I do it, or would you do it?

Er, why will they know who I am and give me this information? You've
been in touch with them before, I hope?

> > > We could also write how much time it takes on average to fill the survey.
> > 
> > Sounds good.
> 
> Note however that such information is available only after survey is
> opened for a bit (unless we use test run statistics).

I suppose that's alright.

> > > The other side of removing options from "13. Which git hosting site(s)
> > > do you use for your project(s)?" is that when sending requests to 
> > > announce the survey to those git hosting sites that are not on this
> > > list, some of them requested to be added (which is impossible after
> > > starting the survey; and before survey begins it is little sense to
> > > send announcements).
> > 
> > I see. How are we going to tackle this in future?
> > 
> > > Besides all of those below 1% rounded (Codesion, GitFarm, The Chaw,
> > > CipherHive) are also those that I didn't get response to request for
> > > announcing Git User's Survey 2010...
> > 
> > Interesting.
> 
> We can always remove (and do not add) those choices for which we don't
> get much responses for, and which do not respond to attemts to contact.  
> Conversely, we probably should add those git hosting sites that we have
> many replies in the "other, please specify".

*nod*

> > > We could make it more organized though, e.g. by sorting list of options
> > > alphabetically, or something like that.
> > 
> > Sounds good.
> 
> Will do, perhaps starting with repo.or.cz (first site), GitHub (most
> popular), Gitorious and perhaps Gitolite (OSS engine), and ending with
> non-generic sites such as git.kernel.org, Alioth, Fedora Hosted, etc.

*nod*

> > > Another solution would be to have pre-filled combo box (<select> field)
> > > with the list of countries to choose from, with GeoIP ised to pre-select
> > > the country.  I can generate list of all countries myself
> > > 
> > >   $ perl -MLocale::Country \
> > >     -wle 'print join("\n", sort (all_country_names()))'
> > > 
> > > as far as I know Survs.com doesn't offer GeoIP nor any API to hook it
> > > to survey questions.
> > 
> > I suppose we could always work out a way to display the results from
> > the information Survs.com gives us.
> 
> Do you have any idea how to display such geographical information, and
> what tool to use for that visualization?

A quick Google search pointed me to several tools that parse a
plaintext file of (lattitude, longitude) entries and use the Google
Maps API to plot them. I'm sorry, but I don't know much more about
this.

> > Nice histogram! How did we manage to do this in 2009? Did we use a
> > custom-made application to do the survey?
> 
> I used a Perl script, which uses Text::CSV to parse data exported from
> Survs.com in CSV format (and PerlIO::gzip to not have to decompress it).
> Each survey page on Git Wiki, except for the very first survey, contains
> link to file with such exported data.
> 
> For example for age it extracts digits from the response, and assumes
> that it is number of years.  It also creates this nice table of ranges
> that you can see in the mentioned section of GitSurvey2009 page.
> 
> I can publish this script, e.g. the one used for 2009 survey on the
> GitSurvey2009 page, but it is rough'n'dirty script.

Sounds good- we should create a small repository that contains all the
tools used, notes made, and results (in semantic format) of previous
surveys. Embarrassingly enough, I can't read Perl myself, but I'm sure
the others will find it useful. Also, isn't there some Perl module to
use Google Maps API to draw that map?

-- Ram

Re: Advertising the Git User's Survey 2011

[Jakub Narebski]

Ramkumar Ramachandra wrote:
> Jakub Narebski writes:
>> On Sun, 3 Oct 2010, Ramkumar Ramachandra wrote:
>>> Perhaps we can email survs.com and ask them?
>> 
>> Should I do it, or would you do it?
> 
> Er, why will they know who I am and give me this information? You've
> been in touch with them before, I hope?

Well, I sent a bit of feedback, bugreports and feature requests while
Survs.com was still in beta...

[...]
>>> I suppose we could always work out a way to display the results from
>>> the information Survs.com gives us.
>> 
>> Do you have any idea how to display such geographical information, and
>> what tool to use for that visualization?
> 
> A quick Google search pointed me to several tools that parse a
> plaintext file of (lattitude, longitude) entries and use the Google
> Maps API to plot them. I'm sorry, but I don't know much more about
> this.

What we have is the names of countries (which can be quite large) rather
than geographical coordinates.  Perhaps use color to mark countries on
the political map, or something?

>>> Nice histogram! How did we manage to do this in 2009? Did we use a
>>> custom-made application to do the survey?
>> 
>> I used a Perl script, which uses Text::CSV to parse data exported from
>> Survs.com in CSV format (and PerlIO::gzip to not have to decompress it).
>> Each survey page on Git Wiki, except for the very first survey, contains
>> link to file with such exported data.
>> 
>> For example for age it extracts digits from the response, and assumes
>> that it is number of years.  It also creates this nice table of ranges
>> that you can see in the mentioned section of GitSurvey2009 page.
>> 
>> I can publish this script, e.g. the one used for 2009 survey on the
>> GitSurvey2009 page, but it is rough'n'dirty script.
> 
> Sounds good- we should create a small repository that contains all the
> tools used, notes made, and results (in semantic format) of previous
> surveys. Embarrassingly enough, I can't read Perl myself, but I'm sure
> the others will find it useful. Also, isn't there some Perl module to
> use Google Maps API to draw that map?

Good idea.

-- 
Jakub Narebski
Poland

Re: Advertising the Git User's Survey 2011

[Ramkumar Ramachandra]

Hi Jakub,

Jakub Narebski writes:
> Ramkumar Ramachandra wrote:
> > Jakub Narebski writes:
> >> On Sun, 3 Oct 2010, Ramkumar Ramachandra wrote:
> >>> Perhaps we can email survs.com and ask them?
> >> 
> >> Should I do it, or would you do it?
> > 
> > Er, why will they know who I am and give me this information? You've
> > been in touch with them before, I hope?
> 
> Well, I sent a bit of feedback, bugreports and feature requests while
> Survs.com was still in beta...

Do contact them then, and let us know what the results are.

> >>> I suppose we could always work out a way to display the results from
> >>> the information Survs.com gives us.
> >> 
> >> Do you have any idea how to display such geographical information, and
> >> what tool to use for that visualization?
> > 
> > A quick Google search pointed me to several tools that parse a
> > plaintext file of (lattitude, longitude) entries and use the Google
> > Maps API to plot them. I'm sorry, but I don't know much more about
> > this.
> 
> What we have is the names of countries (which can be quite large) rather
> than geographical coordinates.  Perhaps use color to mark countries on
> the political map, or something?

Right. Sure.

> >>> Nice histogram! How did we manage to do this in 2009? Did we use a
> >>> custom-made application to do the survey?
> >> 
> >> I used a Perl script, which uses Text::CSV to parse data exported from
> >> Survs.com in CSV format (and PerlIO::gzip to not have to decompress it).
> >> Each survey page on Git Wiki, except for the very first survey, contains
> >> link to file with such exported data.
> >> 
> >> For example for age it extracts digits from the response, and assumes
> >> that it is number of years.  It also creates this nice table of ranges
> >> that you can see in the mentioned section of GitSurvey2009 page.
> >> 
> >> I can publish this script, e.g. the one used for 2009 survey on the
> >> GitSurvey2009 page, but it is rough'n'dirty script.
> > 
> > Sounds good- we should create a small repository that contains all the
> > tools used, notes made, and results (in semantic format) of previous
> > surveys. Embarrassingly enough, I can't read Perl myself, but I'm sure
> > the others will find it useful. Also, isn't there some Perl module to
> > use Google Maps API to draw that map?
> 
> Good idea.

I don't know if the Maps API allows for you to mark numbers on the
map. After all, we don't needs markers for specific coordinates, but
just a numerical label for each country. There's probably something to
even contour-color the countries based on the number- I'll let you
know if I find something we can use.

-- Ram

Re: Advertising the Git User's Survey 2011

[Jakub Narebski]

On Tue, 5 Oct 2010, Ramkumar Ramachandra wrote:
> Jakub Narebski writes:
>> Ramkumar Ramachandra wrote:
>>> Jakub Narebski writes:
>>>> On Sun, 3 Oct 2010, Ramkumar Ramachandra wrote:
>>>>>
>>>>> Perhaps we can email survs.com and ask them?
>>>> 
>>>> Should I do it, or would you do it?
>>> 
>>> Er, why will they know who I am and give me this information? You've
>>> been in touch with them before, I hope?
>> 
>> Well, I sent a bit of feedback, bugreports and feature requests while
>> Survs.com was still in beta...
> 
> Do contact them then, and let us know what the results are.

For the moment I've received information about the meaning of the
numbers in the "Survey Completion Statistics" box on Analyze page:

  Viewed - number of respondents that have opened survey page.
  [not sure if it is somewhat tracked to count users, not clicks]

  Incomplete - number of respondents that have navigated the survey,
  but have not finished it.  This means that these respondents have at
  least navigated to another page; this means that it is 0 for 
  single-page surveys.

  Finished - number of respondents that pressed the finish button at
  the end of the survey.

-- 
Jakub Narebski
Poland