Is there any smart way to test for Speed (Code performance) to catch speed regressions. In DUnit or other. I have some ideas what it should measure and how.
Is there any smart way to test for Speed (Code performance) to catch speed regressions. In DUnit or other. I have some ideas what it should measure and how.
What I can say just on top of my head there should be:
1. It should run the test many times and record the time with highest accuracy possible
2. Maybe filter out slowest if seems to be interference of other processes
3. Should run the test in Real-time priority
4. Save results with info OF Computer and CPU model (And maybe more info). Maybe some Application of Library Version Number
And Have some Graphs on the results.
5. Maybe Some warning report if Some suddenly Changes quite lot (If something important was taken out and it is faster but does not work) or something is suddenly very slow and you could check what is actually causing it.
I am been thinking of and library and now I have to figure out some very core internals and it affects the speed and memory usage. And without measuring I can't make any Choice what so ever :D
What I can say just on top of my head there should be:
1. It should run the test many times and record the time with highest accuracy possible
2. Maybe filter out slowest if seems to be interference of other processes
3. Should run the test in Real-time priority
4. Save results with info OF Computer and CPU model (And maybe more info). Maybe some Application of Library Version Number
And Have some Graphs on the results.
5. Maybe Some warning report if Some suddenly Changes quite lot (If something important was taken out and it is faster but does not work) or something is suddenly very slow and you could check what is actually causing it.
I am been thinking of and library and now I have to figure out some very core internals and it affects the speed and memory usage. And without measuring I can't make any Choice what so ever :D
Old tech analysis trick - run the tests multiple times and use two WMA (weighted moving averages) - one short, one long - and look for crossing points. If the short crosses from below and up - you have a significant increase. This means you need a way to save old test timings, though...
ReplyDeleteA project I know of ran speed regression suits as part of the CI an plotted the main speed index on a graph, showing progress over the last few months. The graph was prominent on the CI dashboard.
ReplyDelete6. Filter out functions that don't change between the tests. To do that you will need to create a map file, parse it to find each function's address then parse the compiled module to find the end of the function and calculate a hash of the procedure's code. Obviously you have to pass the same data to each function for the results to have a meaning.
ReplyDelete