Fault in Schema Migration Outage: Author or Reviewer?

Once upon a time, a developer named John was modifying the name column in a large table from VARCHAR(20) to VARCHAR(512). This change was intended to accommodate more diversity in name lengths, a request that had come down from the product team as they expanded into new, international markets. The exact SQL is:

ALTER TABLE person MODIFY name VARCHAR(512);

Bob, the Database Administrator (DBA), reviewed John's migration script. On the surface, it seemed like a straightforward change, one that he had seen and approved many times in his career. Bob gave his blessing to the change.

With Bob's approval, John executed the migration. As soon as the migration was deployed, alarms blared throughout the office.

The root cause is subtle. Even MySQL 8.0 has increased the online DDL coverage, this particular statement is still not covered. Because the statement enlarges the byte to encode the size value from 1 to 2 as described in the official doc.

Thus all other transactions were blocked while waiting for this DDL to finish. And for large tables, that would take hours.

Though the industry is advocating the blameless culture, someone needs to be accountable. Otherwise, no one will own the initiative to fix the process and the chances are it will happen again.

Is this the author, John’s fault? Maybe. However, it’s a bit more ask for an average developer to know the execution plan of a particular statement from a particular database at a particular version.

Is this the reviewer, Bob’s fault? Plausible. Since Bob has more knowledge and was hired exactly for catching this type of error. However, if Bob takes the bullet, there will be less incentive for the developers to improve their database knowledge.

The good thing is, that such dilemmas have become less nowadays:

1. Companies are moving more DB responsibilities to the developers. There will still be a DB person or a platform team taking care of the DB infrastructure, but the day-to-day DB development tasks are managed by the developer team entirely.
2. Companies may use an external DBA consulting service like Vettabase, Percona, Enterprise DB for expert advice.
3. Companies may adopt database development tools like Bytebase to improve the collaboration between the developers and the DB team.

Welcome to the era of Database DevOps.