Moving twenty years of records into a new system
By John Marta, Principal & Senior IT Architect at GO Build Labs
When a business replaces the system it's run on for ten or twenty years, the new software gets all the attention. Whether anyone ever trusts it is decided somewhere less exciting, by what happens to the old records. We've written about replacing an old system in pieces. This post is about moving the records over.
Look at the old records in the first week
Most project plans put moving the data near the very end, after the new system is built. That's the wrong way round, because your old records are the best description of how your business works that exists anywhere. A field that got used for something else back in 2011, a status that should be impossible but shows up 400 times, a customer entered under three slightly different names: each of those is a business rule nobody ever wrote down. Better to find them while there's still time to plan for them, so pull the data out of the old system in the first week, even if the switchover is months away.
Sort everything into three piles
The first pile is active records, such as open orders, current customers and this year's jobs. These have to arrive in the new system complete and correct, because people will be working on them Monday morning.
The second is history. People need to be able to search it and run reports on it, but nobody needs to edit it. Forcing ten years of closed records through all the new system's checks is how these projects fall behind schedule.
The third is the stuff that doesn't fit anywhere, like the module everyone stopped using in 2014 or the fields nobody can explain. Keep it in an archive you can still search, and leave it out of the new system. Moving it across "just to be safe" only makes the confusion permanent.
Expect some arguments about that third pile. Have them at the start, with the people who use the system every day, rather than the week you switch over.
Keep the words your office already uses
Job numbers, product codes, categories and abbreviations are twenty years of how your office gets things done, and keeping them is the difference between moving to a new system and retraining everyone. When we rebuilt a system from around 2003 for a sourcing broker, the capability codes and job numbering came across unchanged. More than any new feature, that's why the office felt they were using a faster version of a system they knew, rather than starting a new job. If a vendor's plan involves renumbering everything to suit their software, your staff pay for that every day afterward, one lookup at a time.
Check the numbers instead of taking someone's word
"The move went fine" is a feeling. Checking it should be simple arithmetic. The old system's own reports tell you how many customers, orders and order lines it holds, and the new system should show those counts next to what it brought in. That way, "are we missing anything?" takes a glance rather than an anxious afternoon. Then spot-check the money, because the counts can match while a decimal point has moved. Pick real records, especially awkward ones like a credit memo or a partial shipment, and compare them field by field on both screens.
Switch over on an ordinary day
Don't agree to a big weekend where everyone is offline while the data moves and everyone hopes for the best. Run both systems for a while instead. Each night, copy the day's changes from the old system into the new one, and have anything unclear flagged for a person to check rather than overwritten. We explained why connections between systems deserve suspicion in what breaks when systems integrate, and this one needs extra care. Because it's temporary, nobody wants to build it properly, and because it carries your entire history, it's the one that most needs to be. Done this way, the switchover is just the day people stop opening the old system, and that can be an ordinary Tuesday.
Keep a full copy of the old data
Whoever does this work for you, insist on getting a complete export of everything in the old system, in a standard format, with notes explaining what's in it, and kept by you. That means all of it, not just what made it into the new system. It's your record of what was there, your answer when someone asks a question in 2031, and your way out if the relationship goes wrong, for the same reasons you should own the code. It's also the cheapest thing on the whole project, as long as you ask for it at the start.
Moving the records is most of the risk in replacing an old system, so we treat it as first-week work and put the record counts on screen where the client can check them. If you're looking at twenty years of data and a vendor who calls moving it "a final step," it's worth a conversation before you sign anything.