Closed Bug 976735 Opened 10 years ago Closed 10 years ago

Create funnelcake builds for Australis onboarding experience testing

Categories

(Release Engineering :: Release Requests, defect)

Product:
Release Engineering
Component:
Release Requests
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cmore, Unassigned)

References

Details

Attachments

(3 files, 2 obsolete files)

[partner-repacks] Funnelcake 30 & 31, https urls
1.96 KB, patch
catlee
: review+
nthomas
: checked-in-
Details | Diff | Splinter Review
patch for stub installers for funnelcake
2.00 KB, patch
Details | Diff | Splinter Review
pr.diff
628 bytes, patch
catlee
: review+
Details | Diff | Splinter Review 
We have been building and testing onboarding tours (bug 939305) for new and existing users who first experience Firefox Australis. We want to be able to measure product usage over time to see if users who see any tour performs better or worse than users that don't see a tour.

This means we will need a funnelcake build for users that don't see a tour and one for each variation of the tour. We will have some data in FHR/telemetry about tours, but ADI data from funnelcakes will provide additional long-term insight into effectiveness of user education.

This is my first pass at the number of funnelcakes that we will need. I believe the last funnelcake we did was #29.

Funnelcake 30 - no tour - existing user
Funnelcake 31 - tour 1 - existing user
Funnelcake 32 - tour 2 - existing user*
Funnelcake 33 - no tour - new user
Funnelfake 34 - tour 1 - new user
Funnelcake 35 - tour 2 - new user*

* - we may not have different variations of the tours for both new and existing users.

The logic to serve up the funnelcakes would have to be modified so that Firefox browsers get served funnelcake 30-32 and 33-35 would be served to non-Firefox visitors. After the funnelcakes are installed, Firefox is opened and the f=[30-35] is added to the /firstrun/ or /whatsnew/ page, we can redirect each of those users to one of two possible tours.

The launch date for all of this would have to coincide with Firefox 29 on April 29th. How soon before April 29th could the funnelcakes be created? Even if they were repacked on merge date (April 28th), having them early for testing would be very helpful.
Blocks: 939305
We usually create the release build about a week before it is shipped, so the full installers could be available for testing on April 22nd. Creating stub installers and setting up download.m.o would be done manually around the same time.

How will you know if a user has seen the whole tour, or at least most of it ?
> How will you know if a user has seen the whole tour, or at least most of it ?

We have GA on all of the steps of the tour and we can tell how far people make it through and when they cancel. Also, since the f=[x] will be in the URL, we can also associate that with the funnelcakes. Blair also added data to telemetery that records which tour users see. It is all anonymous as expected, and we can use the ADI pings, FHR, and telemetry data to help determine if user education (the tour) has a positive impact on product adoption. That's the goal at least.
We will know how many funnelcakes we need for the GA release of Australis after the content A/B test running on beta via bug 979496.
No longer blocks: 978746
Ok, please let us know the platform & locale requirements then too.

RelEng: We can base this on http://hg.mozilla.org/build/partner-repacks/rev/ee947081baf4 from last time, with updated ID's, about, etc.
(In reply to Nick Thomas [:nthomas] from comment #4)
> Ok, please let us know the platform & locale requirements then too.
> 
> RelEng: We can base this on
> http://hg.mozilla.org/build/partner-repacks/rev/ee947081baf4 from last time,
> with updated ID's, about, etc.

Hi Nick.

Here's what I know so far:

* Locale: en-US only
* Platforms: All, win32, mac, linux
* Best case scenario: 4 funnelcakes (2 for existing users, 2 for new), 4 x 3 platforms = 12 total.
* Worse case scenario: 6 funnelcakes (3 for existing users, 3 for new), 6 x 3 platforms = 18 total.

While funnelcakes for existing users doesn't make much sense, it is difficult to know if the user will be shipped off to /whatsnew/ or /firstrun/. If a user on Chrome that has Firefox installed Firefox Australis and has a local profile, they will get /whatsnew/. If a Chrome user that doesn't have Firefox installed or a local profile installs Firefox Australis, they will get /firstrun/. Since we can't check from a website perspective if they have a profile installed and the /(whatsnew|firstrun)/ logic exist in the product, we can't say for certain their destination URL.

We would distribute 1 of 6 funnelcakes (worse case scenario) via /firefox/new/ using Google Content experiments and appending f=[30-35] to the URL. The user would then download 1 of the 6 funnelcakes, install, and then on launch of Firefox, it would open either /(whatsnew|firstrun)/ depending if they have a local profile and append f=[30-35] to the URL. Depending on what F= value is on the (firstrun|whatsnew) page, we would show one of 3 pages (no tour, tour a, tour b). The challenge is here is that some users will get a funnelcake build ID that is for the /firstrun/ experience, but they are existing user who viewed /whatsnew/. We will probably have to throw away those mismatches and show no tour for them or we stand to corrupt the data by users being a cohort and seeing a different tour. 

This would be so much easier if we were focusing only on /firstrun/, but we have different tours on /whatsnew/ and we want to use the same funnelcake infrastructure to server up the variations to all users.
I just had a quick meeting with Saptarshi about the mismatch concern from my previous comment. He said we will be able to see the destination URL and profile creation dates and thus we can separate in the data post-analysis which funnelcake saw what URL and tour. If a mismatch happens we should pick one tour (if there are multiples) that all mismatch cohort users see. 

For example, if a user downloads funnelcake 33 (no tour, new user), they have Firefox installed or are using Firefox, and /whatsnew/ is displayed (mismatch), we show that user the tour 1 of /whatsnew/. We have to as a team decide what tour (tour 1 or 2) is displayed when a mismatch happens.
No longer blocks: 939305
Blocks: 984972
Update: Habber/Jbertsch/cmore have decided there will be only one tour and not multiple tours for release. This will reduce scope some. Nick: this means less builds needed and I will provide more clarity soon.
When can we make the funnelcake builds?

This is what we need:

Funnelcake 30 - no tour - existing user
Funnelcake 31 - tour - existing user
Funnelcake 32 - no tour - new user
Funnelfake 33 - tour - new user

4 funnelcakes for 3 platforms (win32, mac, linux) for a total of 12 builds.
Flags: needinfo?(nthomas)
The scope sounds OK. I'd assumed we would create the builds during the 29.0 release process, which would mean they were available on April 22nd or so. Do I sense a desire to test earlier than that ?
Flags: needinfo?(nthomas) → needinfo?(chrismore.bugzilla)
(In reply to Nick Thomas [:nthomas] from comment #9)
> The scope sounds OK. I'd assumed we would create the builds during the 29.0
> release process, which would mean they were available on April 22nd or so.
> Do I sense a desire to test earlier than that ?

We just re-scoped this a bit further. Only two funnelcakes now:

Funnelcake 30 - no tour - existing user
Funnelcake 31 - tour - existing user

Can you confirm that 30 and 31 are the next two funnelcake numbers that we can use? We need to confirm this now as there will be logic on our the website to display conditional content depending on the value of the F= query parameter.

Two funnelcakes, three platforms, for a total of 6 builds.

If there are available on April 22-23, that would be great. We are doing to unit test the websites and we feel confident that we won't need them any earlier.
Flags: needinfo?(chrismore.bugzilla)
Also, en-US only builds.
We used funnelcake26-29 for bug 933847, so 30 & 31 are next. Config for win32/mac/linux32 and en-US only, as requested. 

Please review the whatsnew url is what you want (swap to https?). 

How is the delivery of this going to work ? Will we need stub installers for windows ?
Attachment #8404564 - Flags: feedback?(chrismore.bugzilla)
(In reply to Nick Thomas [:nthomas] from comment #13)
> Created attachment 8404564 [details] [diff] [review]
> [partner-repacks] Funnelcake 30 & 31
> 
> We used funnelcake26-29 for bug 933847, so 30 & 31 are next. Config for
> win32/mac/linux32 and en-US only, as requested. 

Yes, funnelcake 30,31, win32/mac/linux32 and en-US only.

> Please review the whatsnew url is what you want (swap to https?). 

Yes, that attachment looked right and yes, change it to https.

> How is the delivery of this going to work ? Will we need stub installers for
> windows ?

Well, if they are running windows, the stub installer they download, should fetch the associated funnelcake. All other platforms, will just get the full funnelcake installer.

Let me know if there are any issues with this. Thanks!
Can someone confirm that funnelcake isn't used for in-product updates? A quick google search didn't answer this clearly. I want to make sure that the whatsnew page offered in the update snippet won't override the funnelcake one (if it is used for background updates).
Funnelcakes are only for the install case (we have bug 976689 to explore updates but nothing happening yet/if at all). So sounds (bug 987480, bug 987407) like we're using prefs to get a whatsnew page for everyone, with similar urls for the three cases:

update: 
    https://www.mozilla.org/%LOCALE%/firefox/%VERSION%/whatsnew/?oldversion=%OLD_VERSION%
funnelcake30 paveover install:
    https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/whatsnew/?oldversion=%OLD_VERSION%&f=30
funnelcake31 paveover install:
    https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/whatsnew/?oldversion=%OLD_VERSION%&f=31

        Attachment #8404564 -
        Attachment is obsolete: true

        Attachment #8404564 -
        Flags: feedback?(chrismore.bugzilla)

        Attachment #8406617 -
        Flags: review?(catlee)

        Attachment #8406617 -
        Flags: review?(catlee) → review+

    
    

    
  
The ideal time to land this is after 29.0b9, and before 29.0, so early next week.

        Attachment #8406617 -
        Flags: checked-in+

    
    

    
  
(In reply to Nick Thomas [:nthomas] (PTO Apr 17-27 PST) from comment #18)
> The ideal time to land this is after 29.0b9, and before 29.0, so early next
> week.

When will funnelcake 30 and 31 for all three platforms be built and the products created in bouncer? We have the website experiences on our demo servers and I want to test the experience. Thanks!
Flags: needinfo?(nthomas)

    
    

    
  
The installers are built, and are currently available here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/29.0-candidates/build1/partner-repacks/funnelcake30/

I'm working on getting them into bouncer, and creating the stub installers.
Flags: needinfo?(nthomas)

    
    

    
  
Turns out we wanted linux64 as well, so I'll be doing that manually.

Right now I've set up the Firefox-29.0-SSL-f30 and Firefox-29.0-SSL-f31 products in bouncer, so these URLs should work:

http://download.mozilla.org/?product=firefox-29.0-ssl-f30&os=win&lang=en-US
http://download.mozilla.org/?product=firefox-29.0-ssl-f31&os=win&lang=en-US  

(repeat for os=osx, os=linux. linux64 will be added later)

Still TODO: creating stub installers for windows for f30 and f31

    
    

    
  

      
      
      
      

        Attached patch
          
          
        pr.diff
        — 
          — Splinter Review
      

    


  
I generated repacks for linux-x86_64 manually. Adding those for the posterity.

        Attachment #8411342 -
        Flags: review?(catlee)

    
    

    
  
Comment on attachment 8411342 [details] [diff] [review]
pr.diff

Review of attachment 8411342 [details] [diff] [review]:
-----------------------------------------------------------------

we should probably back these out soon though so we don't generate funnelcake builds on the next release.

        Attachment #8411342 -
        Flags: review?(catlee) → review+

    
    

    
  
(In reply to Chris AtLee [:catlee] from comment #25)
> we should probably back these out soon though so we don't generate
> funnelcake builds on the next release.

No need to push my changes than.

    
    

    
  
How's it looking?
Flags: needinfo?(chrismore.bugzilla)

    
  
Flags: needinfo?(mozbugs.retornam)

    
    

    
  
Raymond/Gareth: We will have to verify users are downloading FC 30 and 31 from all 4 platforms listed in comment 29 *after* Gareth turns on the funnelcake download in GA.
Flags: needinfo?(garethcull.bugs)

    
    

    
  
Ok Chris. Everything is ready to go on the GA side. I'll run an os report after the test is launched.
Flags: needinfo?(garethcull.bugs)

    
    

    
  
(In reply to Gareth Cull [:garethc] from comment #31)
> Ok Chris. Everything is ready to go on the GA side. I'll run an os report
> after the test is launched.

Launch schedule here: https://etherpad.mozilla.org/australis-release-steps-websites

    
    

    
  
FYI, RelEng need to put the files into firefox/releases/29.0, and update the download.m.o configuration. We'll do that today.

    
    

    
  
Comment on attachment 8411340 [details] [diff] [review]
patch for stub installers for funnelcake

For the future, we can make the stub changes with a one liner patch:

diff --git a/browser/installer/windows/nsis/defines.nsi.in b/browser/installer/windows/nsis/defines.nsi.in
--- a/browser/installer/windows/nsis/defines.nsi.in
+++ b/browser/installer/windows/nsis/defines.nsi.in
@@ -5,13 +5,13 @@

 # Defining FunnelcakeVersion will append the value of StubURLVersionAppend to
 # StubURLVersion, append the value of URLManualDownloadAppend to
 # URLManualDownload, and append the value of URLStubDownloadAppend to
 # URLStubDownload. The value of FunnelcakeVersion should not be defined when it
 # is not used and when it is defined its value should never be empty.
-# !define FunnelcakeVersion        "999"
+!define FunnelcakeVersion        "30"

 !ifdef FunnelcakeVersion
 !define URLManualDownloadAppend  "&f=${FunnelcakeVersion}"
 !define URLStubDownloadAppend    "-f${FunnelcakeVersion}"
 !define StubURLVersionAppend     "-${FunnelcakeVersion}"
 !else

    
    

    
  
(In reply to Nick Thomas [:nthomas] (PTO Apr 17-27 PST) from comment #33)
> FYI, RelEng need to put the files into firefox/releases/29.0, and update the
> download.m.o configuration. We'll do that today.

Done, but into firefox/releases/29.0-funnelcake{30,31}. Also verified that 
  https://download.mozilla.org/?product=firefox-stub-f31&os=win&lang=en-US
yields a first run at
  https://www.mozilla.org/en-US/firefox/29.0/firstrun/?f=31
with tour. Similarly -f30 which doesn't. All the links in comment 29 are 302 --> 200 as before.

    
    

    
  
Gareth: We need to shut off the funnelcake experiment on the afternoon of May 8th 2014.
Flags: needinfo?(garethcull.bugs)

    
    

    
  
Sounds good. I'll shut it off on the 8th.
Flags: needinfo?(garethcull.bugs)

    
    

    
  
Comment on attachment 8406617 [details] [diff] [review]
[partner-repacks] Funnelcake 30 & 31, https urls

Backed out so we don't keep creating these builds on the latest version:
https://hg.mozilla.org/build/partner-repacks/rev/e92e49571d39

        Attachment #8406617 -
        Flags: checked-in+ → checked-in-

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: