Ponderings on Private Beta

Hi all,

I’ve been asked my by PO to put out a request about what a good private beta looks like in terms of size, ramp up et? I would think it would vary from service to service.

Any advice / links to examples would be most welcome.

Thanks in advance.




Really interesting question. I’ve not seen any formal guidance on it –

I guess my answer would be yes it varies between services. Big enough so you can get some useful insight and Small enough that it is low key and no one notices if you have to pull it?

The last thing you want for your ‘Private Beta’ service is to spend ages setting up a private beta panel/community only to not have enough users going through to gain any useful insight, this would be very expensive means of testing and validating a thing

For the new NHS Manage Your Referral Service (part of NHS e-Referral Service). We ran a very very small public beta. The old national service has around 1 million transactions per year, we used a tiny link on the homepage of the old service to redirect about 1% of users, we later made a more prominent link which redirected about 10%.

We had (for a few months) only a few dozen users per day — around 10% provided survey feedback (5–10 users per day) but the survey and analytics base sizes weren’t robust enough to say if it was statistically an improvement (e.g. Satisfaction/completion rate) over an old service it was due to replace — was this just by chance due to low sample and high margin of error.

What you can do though with only a few users is analyse individual users and see why people drop off and the errors and make some informed changes based on analytics / database queries, that needs to be part of the private or public beta plan. Until you scale up you won’t know for sure if the private beta data is robust and the service is cooking on gas as many weird errors/alternative use cases that you won’t find in private beta testing are much less likely to occur with only a handful of users!

I guess it comes back to your service, questions for the PO and UR to think about might be:

  • what are the objectives are for private beta?
  • what is the means for validating incremental improvements?
  • is there is a drive/need to go for public beta?
  • how many changes are you planning to test? as you will need much larger numbers e.g. if using AB or multi variate testing,
  • how quickly can we resolve issues/make changes?
  • when should we scale up? how long will it take to gather data to validate that it works?

Hope this helps.


Previously posted on Medium

Categorized as GDS Tagged