About Encode Bench

This benchmark was created out of curiosity about the asymmetrical ability for large language models to proficiently read but not generate encoded text.

I'd like to lead with two acknowledgments:

1. Like any benchmark, Goodharts Law applies and since agents helped write this benchmark the best before date is already approaching.
2. I recognize that models aren't trained to generate encoded responses, but they aren't explicitly trained to read them either.

The hypothesis

In my early testing I noticed that models generally recognized as "weaker" had more difficulty generating an encoded and correct response. This piqued my curiosity whether the proficiency at generating encoded responses correlated with a models other proficiencies or weaknesses such as prompt injection.

If it turns out that the ability to generate complex encoded responses correlatets more strongly than not to "intelligence", the second part of my hypothesis is that as Goodharts Law runs its course, the correlation will strenghten.

However, decoupling actual model improvements on this benchmark through generalization and specialization when this inevitably enters training data is difficult. Here I defer to the community.