OpenAI published a coding evaluation across twenty languages. Here is how it checks database behavior and why its score does not establish COBOL migration readiness.
OpenAI introduced MultiLangBench on October 19. It asks a model to build a SQLite-compatible engine in one of twenty languages. The same announcement describes a separate task: migrating existing COBOL programs while preserving their behavior. These measure different outcomes.
The public repository describes a database-file interface. A candidate executes a query; reference SQLite 3.40.1 reads the modified database and checks the expected result. The README recommends a 272-test subset, with the full suite available separately. That recommendation does not establish which subset produced every published score.
We read one concrete test: its query creates two tables and inserts a row. Validation checks that row, the second table’s row count and a named index. This was a source-file audit, not a model run or independent reproduction of its score.
OpenAI’s separate COBOL migration evaluation uses hidden tests requiring byte-for-byte parity of observable outputs, with every test passing. A SQL-test pass rate therefore is not a success rate for migrating production systems. The vendor also calls for expert review of business rules and dependencies before production use.