Playtest Analysis

With two playtest sessions now completed I can compare the average times from each session to see how the changes I made impacted the game. In general, I want the difficulty of the second level to be higher than the first. Puzzle 1-4 can be solved two different ways and I want the majority of players to solve the puzzle the easy way the first time.

Graph 1

The graph above shows the average time players took to complete each puzzle in Playtest One. As we can see, players spent much longer in 1-2 than they did in 2-2. This is to be expected because players are being introduced to the Lift ability during 1-2. However, the tutorial was unclear and many testers got stuck at this point. I expect 1-2 to be higher than 2-2, but the discrepancy is still too high. We can also see that players took longer to complete 1-3 than they did 2-3 which shouldn’t be the case. Puzzle 2-3 involves the player having to jump past falling pins which the companion would have otherwise protected them from. Needless to say, this shouldn’t be easier to accomplish without the companion. Partly to blame for this was the poor checkpoint placement which allowed players to die on the final pin but still progress in the level. Puzzles 1-1, 2-1 and 1-4, 2-4 show the kind of difference in times I would expect.

Graph 2

The graph above shows, that on average, players spent less time to complete a puzzle during the second level than they did during the first. This is the complete opposite of what I would like. However, if we compare the results from Playtest Two:

Graph 3

We can see that this is no longer the case. During Playtest Two players spent less time to solve a puzzle during the first level than they did during the second. Now if we compare both playtests directly we can see which puzzles account for this change:

Graph 4

In the graph above, the solid lines represent Playtest Two and the dotted lines, Playtest One. If we look at puzzle 1-2 (the blue line) we can see that players spent significantly less time during this puzzle than they did during Playtest One. If we look at puzzle 2-3 (the orange line) we can see that players spent much more time to complete this puzzle than they did in Playtest One. Based on this comparison, it’s clear that the changes I made to these puzzles after the first playtest have had the desired effect I was looking for. However, this graph includes an outlier. One of the playtester in Playtest Two took much longer than average to complete puzzle 2-4:

Graph 5

The graph above shows the same comparison but with this outlier excluded. We can see that puzzle 2-4 (the orange line) takes less time to complete than puzzle 1-4 (the blue line). If we compare that to Playtest One (the dotted line), we can see that players took significantly less time to complete the final puzzle during Playtest Two than they did in Playtest One. From this we can see that the changes I’ve made to this puzzle have had the opposite effect I was looking for and that now it would seem to be easier than it was prior to my changes. I have also confirmed that when excluding the outlier, only the average time for puzzle 2-4 is significantly affected. This indicates that this playtester just got stuck on puzzle 2-4, which can happen to anyone.

In the first level, puzzle 1-4 can be solved in two different ways, an easier way and a harder way. In the second level puzzle 2-4 can only be solved in the hard way, the same harder solution to puzzle 1-4. This means that players who solve 1-4 the harder way, can solve 2-4 by just repeating the solution. Interestingly, during Playtest One, 50% of players explored the harder section of 1-4 before the easier section, with 80% of players solving the puzzle the easy way and only 20% the hard way. However, during Playtest Two, 70% of players explored the harder section first, with 60% solving it the easy way and 40% the hard way. It would seem that the changes I’ve made to puzzle 1-4 have caused more people to explore the harder section first and solved it the harder way. Because 40% of players solved it the hard way in 1-4, they simply had to repeat the puzzle again in 2-4. This may account for the decrease in time in Playtest Two.

After Playtest One, to discourage players from exploring the harder section, I changed the camera position to show more of the easier section and less of the harder section. This may have actually encouraged players to explore rather than discouraged them. If someone tells you not to think of penguins, what do you think about? Along with this change, I also increased the size of the harder section and made it easier to interact with elements of the puzzle. One, or all of these changes could be responsible, but it might also be the case that my sample size is just not big enough. I may have just asked more people who were better at puzzle games to playtest during Playtest Two than I did in Playtest One.

Either way, puzzle 1-4 still needs some work and my plan is to implement more changes going forward. I will be continuing to playtest, but my playtest sessions are now concluded. I feel like I have a significantly better game than I did prior to this testing and it’s thanks to the feedback I was given. Next, I want to finally finish my NPC designs and get rid of those white rectangles!

Phase Four: Keeper Design