You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently timeit has significant iterating overhead when tests fast statements. Such overhead makes hard to measure effects of microoptimizations. To decrease overhead and get more precise results we should repeat tested statement many times:
Looks good, but I think it is better to have an "unroll" option rather than do it automatically. I'm okay with the default being to unroll, but sometimes I want to compare the speed between different versions of Python, and having unroll=False to ensure the same behaviour between versions would be good.
Indeed, what's good for CPython may be quite annoying for e.g. a JIT-enabled Python implementation. I wonder what the PyPy developers think about this.
I think this is likely to make timeit less representative of how code actually performs in the real world on systems with a JIT. This is because of the cost of sequential operations is not strictly "additive" on PyPy.
If you have statements a and b, and you run a; b on PyPy, the performance of a; b is usually faster than the sum of a, b, assuming they are not 100% independent.
This is because the JIT will be able to remove type checks that were already performed. Since this just repeats the same statement, the cost of the unrolled iterations beyond the first will be massively lower in many cases, producing confusing results.
The opposite argument might be relevant too: in some cases, a tracing JIT compiler seeing a long block of code might perform artificially worse. If each repeated line creates a branching path with two outcomes of roughly equal likeliness, then if the line is repeated 20 times, the JIT will need to compile 2**20 different paths before it has fully warmed up. In practice, it will never fully warm up and will run with the constant huge overhead of finding and compiling more paths.
arigo commented on Jul 16, 2014
arigomannequin
Mannequin
More actions
...but I don't think PyPy should be by itself a good enough reason to reject this patch. It would be fine if timeit detects which interpreter it runs on, and only tries to unroll on CPython, for example.
On Wed, Jul 16, 2014 at 02:49:31PM +0000, Armin Rigo wrote:
...but I don't think PyPy should be by itself a good enough reason to
reject this patch. It would be fine if timeit detects which
interpreter it runs on, and only tries to unroll on CPython, for
example.
I would *much* rather a parameter to timeit which controls whether or
not to unroll, rather than timeit trying to guess whether you want it to
unroll or not. PyPy can default to off, CPython to on, and other
implementations can choose whichever default makes sense for them.
I would *much* rather a parameter to timeit which controls whether or
not to unroll, rather than timeit trying to guess whether you want it to
unroll or not. PyPy can default to off, CPython to on, and other
implementations can choose whichever default makes sense for them.
I think it is overkill. Apart from rather silly microbenchmarks, there
isn't much point in adding the loop unrolling facility. In real world,
even cheap operations such as "x = x + 1" will be surrounded by less
cheap operations, so if an improvement cannot yield tangible benefits
inside a simple for loop, then it doesn't deserve to be committed.
I'm afraid "microoptimizations" aren't worth measuring to begin with, since, well, they're "micro" ;-) Seriously, switch compilers, compilation flags, or move to a new release of a single compiler, and a micro-optimization often turns into a micro-pessimization. If I _want_ to measure something unrolled, I'll unroll it myself. But since I almost never want to measure something unrolled anyway, an option to do so would just be "yet another attractive nuisance" to me.
Activity
serhiy-storchaka commentedon Jul 16, 2014
Currently timeit has significant iterating overhead when tests fast statements. Such overhead makes hard to measure effects of microoptimizations. To decrease overhead and get more precise results we should repeat tested statement many times:
Proposed patch makes it automatically for user. It unrolls and vectorize the loop, and decreases iterating overhead 1000 times:
An user gets precision value without explicit cumbersome repeating.
[-]Decrease iterating overhead it timeit[/-][+]Decrease iterating overhead in timeit[/+]stevendaprano commentedon Jul 16, 2014
Looks good, but I think it is better to have an "unroll" option rather than do it automatically. I'm okay with the default being to unroll, but sometimes I want to compare the speed between different versions of Python, and having unroll=False to ensure the same behaviour between versions would be good.
pitrou commentedon Jul 16, 2014
Indeed, what's good for CPython may be quite annoying for e.g. a JIT-enabled Python implementation. I wonder what the PyPy developers think about this.
alex commentedon Jul 16, 2014
I think this is likely to make timeit less representative of how code actually performs in the real world on systems with a JIT. This is because of the cost of sequential operations is not strictly "additive" on PyPy.
If you have statements
aandb, and you runa; bon PyPy, the performance ofa; bis usually faster than the sum ofa,b, assuming they are not 100% independent.This is because the JIT will be able to remove type checks that were already performed. Since this just repeats the same statement, the cost of the unrolled iterations beyond the first will be massively lower in many cases, producing confusing results.
pitrou commentedon Jul 16, 2014
Thanks, then I guess I'm -1 on the patch.
arigo commentedon Jul 16, 2014
The opposite argument might be relevant too: in some cases, a tracing JIT compiler seeing a long block of code might perform artificially worse. If each repeated line creates a branching path with two outcomes of roughly equal likeliness, then if the line is repeated 20 times, the JIT will need to compile 2**20 different paths before it has fully warmed up. In practice, it will never fully warm up and will run with the constant huge overhead of finding and compiling more paths.
arigo commentedon Jul 16, 2014
...but I don't think PyPy should be by itself a good enough reason to reject this patch. It would be fine if timeit detects which interpreter it runs on, and only tries to unroll on CPython, for example.
stevendaprano commentedon Jul 16, 2014
On Wed, Jul 16, 2014 at 02:49:31PM +0000, Armin Rigo wrote:
I would *much* rather a parameter to timeit which controls whether or
not to unroll, rather than timeit trying to guess whether you want it to
unroll or not. PyPy can default to off, CPython to on, and other
implementations can choose whichever default makes sense for them.
pitrou commentedon Jul 16, 2014
Le 16/07/2014 12:43, Steven D'Aprano a รฉcrit :
I think it is overkill. Apart from rather silly microbenchmarks, there
isn't much point in adding the loop unrolling facility. In real world,
even cheap operations such as "x = x + 1" will be surrounded by less
cheap operations, so if an improvement cannot yield tangible benefits
inside a simple for loop, then it doesn't deserve to be committed.
tim-one commentedon Jul 17, 2014
I'm afraid "microoptimizations" aren't worth measuring to begin with, since, well, they're "micro" ;-) Seriously, switch compilers, compilation flags, or move to a new release of a single compiler, and a micro-optimization often turns into a micro-pessimization. If I _want_ to measure something unrolled, I'll unroll it myself. But since I almost never want to measure something unrolled anyway, an option to do so would just be "yet another attractive nuisance" to me.
gvanrossum commentedon Jul 17, 2014
I don't see the value in this complication. Please close as won't fix.
gvanrossum commentedon Jul 17, 2014
(Had to switch identities to close it.)
bitdancer commentedon Jul 18, 2014
Guido: I've added developer privs to your Guido.van.Rossum account.
serhiy-storchaka commentedon Jul 22, 2014
OK. In any case I don't like this patch, it breaks simplicity and elegance of current code.