I’ve created a monster
It sounds like a simple and reasonable request: just create a database table to track hits to a certain feature against the correct affiliate partners and store a few other pieces of user information. Every bone in my body knew that this wouldn't end well, I made a vain attempt to push for a more future proof solution but with the reality of the urgent timeframe and all the usual non development politics, it was going ahead as planned. We can always improve it later right?
A few months after it's set into production, this simple single table is now 95% of the database with millions of records a month being pumped into it, a simple database backup now requires gigabytes of bandwidth to download and its requiring more server resources every month that passes. There's now all sorts of reporting systems relying on this raw data and the prospects of archiving/denormalizing the data are getting worse and worse each day, meanwhile the database is becoming harder to manage.
This thing is literally an out of control monster with over 100 million records.. Denormalizing the data and creating an action plan to switch over to the new system in real time is going to be a fairly tedious operation requiring more server resources temporarily and modification to a number of existing systems throughout the business. The work required now to get it on track would far eclipse the initial investment of having built a more robust system from the start.
If this innocent value add feature was core to the client’s business there would have been no question of putting the time and resources in to upgrade the system now or in the beginning. However this system is still important enough that they can’t do without it, but not important enough to divert significant time away from other projects. This is what has fed the monster for so long.
The nature of modern web development is to follow the MVP (Minimum Viable Product) approach which I usually agree with, however some things should still be set in concrete from the start. Core database schema in my opinion is just as important as selecting the right technology.
