How do you test a large amount of data

1. check if there is specification of the data to see if you can narrow down the format/definition of the data so you can run your program with no/little error.
2. write your code to write in the log when running into exception of the specification.
3. run your test over PART of the data until there is no exception thrown.
4. estimate based on 2 number of errors/exception you might see in production.
5. later on, you will be finding rare exceptions that might only occur a couple of times in the input. Just solve these by hand

if there is no spec, try to see if you can normalize the data