Sixty days of dailyprog

dailyprog has published a puzzle every day since 19 June. Sixty days, sixty puzzles, no gaps. Eleven more are already written and sitting in the catalogue, with the safety net to catch any gaps.

That’s long enough that the analytics say something. Not all of it is flattering.

The numbers

Growth is nothing to write home about, but steady. Eleven days ago it was 1,372 visitors, so the site adds around 140 a week without me doing much to make that happen. It’s not even indexed by a search engine yet. There’s no launch spike left in these numbers, so whatever this is, it’s the baseline.

Retention

Day-7 return: 18.6%. Eleven of the fifty-nine people who had a first day came back a week later.

This one took real work to get, and the story is a decent argument for not believing your dashboard. Umami has a returning-visitor metric. It reported 2.1%, which I stared at for a while wondering what was so wrong with the site. Nothing was. Umami derives its session ID from IP and user agent, and that doesn’t survive a day boundary, so a person coming back tomorrow is a brand new visitor by definition. The 2.1% was an honest count of something nobody asked about.

So the site now writes a lifetime_day number onto its own events, from data it already has locally, and the cohort is day:7 / day:1. It shipped on 6 August and the first honest reading landed a week later at 16.3%. It’s 18.6% now. Nothing before 13 August is comparable to anything, which is annoying and correct.

18.6% of people who try a daily puzzle site come back the following week. I have no idea whether that’s good. I’ve looked for a public benchmark and mostly found blog posts quoting each other.

The funnel

Split every session by whether the person typed anything:

Half the traffic (52%) arrives, does nothing measurable, and leaves at a median dwell of zero seconds. The other group sits on the page for twenty-five minutes.

I expected a spread. There isn’t one. You either bounce off the front door or you spend most of a lunch break there, and the 226 who wandered in between are the smallest of the three groups. That’s a strange shape for a product to have and I don’t fully understand it yet. The optimistic reading is that the puzzle is doing exactly what it’s supposed to for the people who start it. The pessimistic one is that everything I could tune in the middle of the funnel is tuning for a population that barely exists.

The single biggest loss in the whole funnel is the step from “opened a puzzle” to “touched the editor”. 545 of 1,325 sessions, so 41%.

I wrote that up twice as a first-impression problem. Bad copy, unclear prompt, something about the hero. Both times I ruled things out by traffic source, found no bad referrer, and concluded it was broad and expensive and hard.

Then I split it by device, which is one JOIN against a table Umami had been populating the whole time.

I think the mobile number is flattered, because some of those buckets are one devoted person each. Take the regulars out and everyone else on mobile reaches the editor 13.7% of the time, against laptop’s 45%.

xychart-beta
    title "Reached the editor, by device"
    x-axis ["desktop", "laptop", "mobile", "mobile minus regulars"]
    y-axis "% of puzzle-page sessions" 0 --> 70
    bar [61.3, 45.1, 23.2, 13.7]

The mechanism, once you look, is not subtle. On desktop the editor is in the right-hand column, visible on arrival. Below the md: breakpoint the layout collapses to a single stack, and the reading order is title, prompt, every worked example, then the visualiser if the puzzle has one, and only then the editor. On a 390px-wide phone that’s somewhere between one and a half and two and a half screens of prose before you reach anything you can touch. Visitors are handed an article, so they read it like an article and leave. Nothing has shipped against this yet. If those 350 non-regular mobile sessions converted like laptop does, the overall figure goes from 41% to about 48%.

Solving

(That accounts for 724 of the 739 submissions. Fifteen of them reached Umami without an attempt number attached and I haven’t chased down why.)

Three quarters of the people who submit pass on the first go. After a failed first attempt, fewer than half ever get there.

The same table as a ladder, with the third attempt onward rolled into one rung:

---
config:
  sankey:
    nodeAlignment: left
    labelStyle: outlined
    height: 420
    nodeColors:
      "attempt 1": "#86b6ef"
      "attempt 2": "#3987e5"
      "attempt 3+": "#1c5cab"
      "passed": "#008300"
      "stopped": "#52514e"
      "out of attempts": "#e66767"
---
sankey-beta

attempt 1,passed,365
attempt 1,stopped,10
attempt 1,attempt 2,122
attempt 2,passed,57
attempt 2,stopped,8
attempt 2,attempt 3+,57
attempt 3+,passed,39
attempt 3+,stopped,6
attempt 3+,out of attempts,12

I read that as reassuring and slightly worrying at the same time. Reassuring because the hidden tests aren’t springing gotchas: if you understood the puzzle, you pass immediately. Worrying because the debugging loop is doing almost no work. A failed submission tells you timeout or wrong or error and deliberately nothing else, no hidden input and not even which case broke, because the hidden tests are the answers. That constraint is vital and I’m not loosening it. But it does mean the second attempt is a much colder start than I’d assumed, and the table is what that costs. Only 15 sessions in the site’s entire history have run out of attempts, so people aren’t grinding into the wall either. They just stop.

Per-puzzle, the range is enormous. Puzzle 29 is the most-attempted in the catalogue at 47 submissions and passes at 15%. Puzzle 60, from this morning, passes at 91%.

Each puzzle carried an easy / medium / hard tier. I deleted it yesterday.

Across the 32 puzzles with enough submissions to say anything, the correlation between the authored tier and the observed pass rate was -0.10. Faintly backwards. Four of 32 puzzles landed inside the bands the tier was supposed to promise, and an easy-labelled puzzle (number 8, 29%) turned out harder than every hard one.

The reason is boring and took me two full analytics passes to spot: there were two definitions and nobody had reconciled them. The generator’s design playbook defines the tiers in solve-time minutes, and my calibration bands defined them in pass rate. Both documents were internally consistent. They just weren’t about the same thing.

The label was also barely visible to players, which is why deleting it cost nothing. It survived as long as it did because it sat in a JSON field that every read sliced by, so it looked like a measurement instead of a note to myself.

The puzzles also get rated. Mean 4.34 out of 5 across 173 ratings, up from 4.27 at the start of the month.

pie showData
    title 173 ratings
    "★★★★★" : 99
    "★★★★" : 48
    "★★★" : 14
    "★★" : 9
    "★" : 3

I don’t believe it much, and neither should you. The rating widget appears after you solve. Of the first 116 ratings, 114 came from people who had just passed and two from people who had run out of attempts. It measures how people feel right after winning, and it turns out they feel 4.34 stars about it.

The number that would actually be useful is the one I can’t collect, because the people who’d give it have already closed the tab.

The regulars

The site has no accounts (yet?). There’s an optional identity, a keypair in your browser, that exists so streaks survive and so a solve can be attributed. 37 people have made one. Nine of them have played on ten or more separate days.

The live streaks right now:

30d  30d  30d  28d  7d  4d  3d  3d

Three people have solved every single puzzle for a month. Further down that list, one of the 3-day runs has a best-ever of 26 behind it, so somebody broke a long streak and started again from nothing. I find that more encouraging than the thirties.

Between them, those 37 identities account for 278 verified solves, plus 134 practice completions on puzzles whose day had already ended.

Now the uncomfortable part. Pass rates have climbed steadily with puzzle number: the correlation is +0.52, and the median has gone 58% to 73% to 86% across the catalogue’s three eras. Two explanations predict that table identically. Either the puzzles have drifted easy, or the audience that stuck around got better.

I think it’s the audience, and that’s a judgement from watching the actual people rather than from anything in the data. But if it’s the audience, then a stable 86% pass rate means the puzzles are getting easier relative to the people who stayed, and those people are precisely the day-7 cohort the whole site is measured on. A player on a 30-day streak clearing every puzzle on the first attempt has stopped being challenged, and nothing in this post would warn me about that in advance. The first signal is a streak ending, at which point it’s already happened.

So the top of the difficulty range goes up, while every metric still says things are fine.

Growth

139 share clicks, from 61 sessions. That’s 18.5% of the people who solved, down from 24% at the start of the month while the number of solvers grew.

Sharing is the only mechanism by which this site grows on its own, and it is the one thing moving in the wrong direction. 35 identities existed on 15 August and 37 exist now, against a much faster-growing pile of solvers.

I have no good theory here yet. The share button hasn’t changed, the copy hasn’t changed, and the puzzles people are sharing pass at higher rates than ever, which if anything should make sharing more appealing. Two more solvers signing up in nine days is not a rounding error I can explain away.

Where the traffic is from: US 377 sessions, Netherlands 335, Germany 203, Brazil 76.

The Netherlands being second is no mystery: I live here, and a good chunk of those 335 sessions is probably me. The site is localised into ten languages, and 88% of sessions are English, including the overwhelming majority of the Dutch ones. Dutch-language sessions are about 3% of the Dutch traffic. Country volume and locale demand turn out to be almost unrelated, which was worth learning before I spent more on translation.

German is the largest non-English language at 80 sessions, which is 5%. The localisation is staying, because it’s built and it costs nothing to keep. The policy around it has loosened though: English is the source of truth now and a missing key falls back to English rather than blocking the feature. Carrying nine translations in every commit stopped being worth the friction at roughly one non-English session in ten.

What I’m doing about it

In order:

1. Get the engagement up on mobile, with the design settled first this time. Biggest available win, just needs a good nudge.
2. Step up the top of the difficulty range, before a streak breaks rather than after.
3. Work out what happened to sharing.

Every finding in this post was already sitting in data I’d been collecting for weeks. Most of them took a second or third pass over the same tables to see. The one I’m annoyed about is the mobile conversion, which I wrote up twice as a copy problem before I thought to join one extra table and found out it was a viewport.