use the following search parameters to narrow your results:
e.g.(and 'dog' reddit:'aww' site:'imgur.com')
(and 'dog' reddit:'aww' site:'imgur.com')
see the search faq for details.
advanced search: by author, community...
What is the fastest way to do fast on-screen pixel-by-pixel image manipulation? (self.Python)
submitted 9 months ago by ReallyGoodAdvice
I'm making something like a cellular automata visualization. I want to be able to do tens of thousands of pixel color changes per second. What would be the best way to approach this?
[–]Cixelyn 8 points9 points10 points 9 months ago*
Drawing pixels via pygame seems like a reasonable way to achieve what you're doing.
In pygame, draw the pixels via direct array access as the basic drawing commands will probably be too slow for your needs.
[–]ocrow 0 points1 point2 points 9 months ago
Agreed - a pygame pixelarray should be pretty quick. If you're doing lots of data manipulation before writing the pixels that part might also be a bottleneck.
I don't think you can use PyPy with pygame yet, or that would be an interesting path.
I would start with python and pygame, and then profile it. Then you can see which parts are slow, and think about other optimizing strategies, like tweaking how you store + manipulate data or even perhaps pushing some of your lowest level code into a C extension.
[–]cecilkorik 0 points1 point2 points 9 months ago
I think I've seen PyPy projects using pygame. pyopengl definitely doesn't work yet though.
[–]hyljeDjango guy 2 points3 points4 points 9 months ago
pyglet works on pypy by the virtue of only using ctypes opengl
[–]Mob_Of_One 0 points1 point2 points 9 months ago
Pretty much anything that talks to C and hasn't been hand-converted isn't going to work.
[–]UloPe 0 points1 point2 points 9 months ago
Except for ctypes
[–]Mob_Of_One 1 point2 points3 points 9 months ago
Yes, which they're eager to use of late.
[–]jamesshuang 3 points4 points5 points 9 months ago
You should probably phrase your problem in terms of vectorized math, then output the images via pygame. Straight up Python won't do pixel-by-pixel manipulations very quickly, but if you can think of it as a convolution, it can be very fast.
[–]Doormatty 2 points3 points4 points 9 months ago
Your idea sounds interesting, but I would have no idea even where to start. Could you suggest an article or something on this concept?
[–]vplatt 1 point2 points3 points 9 months ago
Well, it looks like Pygame has NumPy integrated now, so I would start there. I did some Pygame a few years ago, but I haven't used NumPy with it, so I can't personally vouch for that.
http://www.pygame.org/wiki/FrequentlyAskedQuestions
Like stoph points out, Python may not be up to the task of "tens of thousands of pixel color changes per second", but this is where I would start with trying to make that happen in Python.
[–]Doormatty 1 point2 points3 points 9 months ago
Sorry, I wasn't clear.
I meant the entire concept of "phrase your problem in terms of vectorized math".
[–]mackstann 1 point2 points3 points 9 months ago
Basically, you do operations on arrays of values all at once, instead of one single value at a time. Modern CPUs are optimized for this, so you can significantly increase the amount of work done in the same amount of time, but you are limited to certain mathematical operations. Here is a list of the vector operations NumPy supports, and here is an example of how you'd use one of them.
[–]Doormatty 0 points1 point2 points 9 months ago
Many thanks! I would have never made that leap. I still think of vectors in the physics sense.
[–]fishdicks 2 points3 points4 points 9 months ago
Here's another example, which performs one step of the Game of Life in a fully vectorized fashion:
def iterate(Z): # find number of neighbours that each square has N = np.zeros(Z.shape) N[1:, 1:] += Z[:-1, :-1] N[1:, :-1] += Z[:-1, 1:] N[:-1, 1:] += Z[1:, :-1] N[:-1, :-1] += Z[1:, 1:] N[:-1, :] += Z[1:, :] N[1:, :] += Z[:-1, :] N[:, :-1] += Z[:, 1:] N[:, 1:] += Z[:, :-1] # a live cell is killed if it has fewer than 2 or more than 3 neighbours. part1 = ((Z == 1) & (N < 4) & (N > 1)) # a new cell forms if a square has exactly three members part2 = ((Z == 0) & (N == 3)) return (part1 | part2).astype(int)
You might also want to try to combine it with this to reduce the number of Python-to-numpy calls.
Also, google "numpy cellular automata", "numpy convolutions" etc for more examples.
[–]vplatt 0 points1 point2 points 9 months ago
That examples page you posted rocks. Makes me want to write something, anything at all, that uses all of the operations on that page.
[–]lucasvb 1 point2 points3 points 9 months ago
The answer really depends of what, exactly, you are trying to achieve, how many colors you are going to need, what kind of pixel manipulation, is it on the entire screen or only a few regions, is it bounded or unbounded, etc.
[–]crocowhile 1 point2 points3 points 9 months ago
A 800 * 600 image has 480k pixels and can be refreshed easily at 25Hz using opencv or probably even PIL. They both support Numpy.
Treat your image as a numpy array and make sure all the operations you make on them are actually made on the array, entirely or sliced. NEVER USE LOOPS. You can make everything you want on entire arrays but it takes some mental effort to come up with the way to do it if you have not experience on it.
For a safe bet, I'd go numpy & scipy + cv. They both work very well with python 2.7 and they'll let you do more than what you want to achieve. They all have sections on stackoverflow too.
[–]takluyverIPython, Py3, etc 1 point2 points3 points 9 months ago
Probably not a practical solution, but out of interest, PyPy recently had a demo that it was fast enough to run some real-time image processing, written in pure Python.
http://morepypy.blogspot.com/2011/07/realtime-image-processing-in-python.html
[–]blatheringDolt 0 points1 point2 points 9 months ago
Out of curiosity, are you making something that spins or moves very fast?
[–]servetus 0 points1 point2 points 9 months ago
Depending on what you are doing you may want to use your GPU. Pyopengl has a decent APO for compiling and using GLSL shaders. The performance increase can be several orders of magnitude.
[–]mkor 0 points1 point2 points 9 months ago
To visualize Langton's loops we were using OpenCV for Python and friend of mine succeed with quite good speed.
[–]irve 0 points1 point2 points 9 months ago
Have done quite similar thing: I ditched real-time and rendered an amount of .png files which I later stitched automatically using ffmpeg.
[–]stoph 0 points1 point2 points 9 months ago
Once you follow the advice given by others, you may find that a lower level language is still more suitable to your optimization needs. Try direct OpenGL commands in C++ or D language.
[–]thantik 0 points1 point2 points 9 months ago
Wouldn't optimization in Cython also be just as good?
Well... first of all, the author probably doesn't want to do tens of thousands of color changes per second. Most monitors can only display up to about 80 frames per second.
But if they need to do intense computation to figure out what needs to be displayed, there is still plenty of room to do that in C++ or whatever compiled language floats your boat. If you really love Python, you could write a DLL interface to C++ to offload the computationally intensive parts.
I suppose the submitter could have meant "tens of thousands of pixel changes per second" in reference to the resolution of the rendering, as opposed to the frame rate. Still, there's nothing wrong with recognizing Python's limitations and using C++ where appropriate.
[–]thantik 1 point2 points3 points 9 months ago*
Wouldn't a DLL interface in C++ be a level harder than Cython though? (I'm not talking about CPython...I'm talking about Cython.)
Cython allows static typing, massively speeding up any kind of computation you need to do, while maintaining a python-like syntax.
http://prabhuramachandran.blogspot.com/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig.html
There's a good comparison -- C++ and Cython are in the same ballpark while allowing the use of Pythons native containers and (most of the) syntax.
I think we might be talking about the same thing. Calling fast, compiled code from Python sounds a lot like Cython. Cython might put a pretty wrapper on things but I consider both approaches to be the same.
Here's an awesome example of Cython code:
http://wiki.cython.org/examples/mandelbrot
That's pretty sweet. I am not a huge Python expert but it sort of reminds me of how RPython is using a restricted language to boost speeds on core functions in PyPy.
all it takes is a username and password
create account
is it really that easy? only one way to find out...
already have an account and just want to login?
login
[–]Cixelyn 8 points9 points10 points ago*
[–]ocrow 0 points1 point2 points ago
[–]cecilkorik 0 points1 point2 points ago
[–]hyljeDjango guy 2 points3 points4 points ago
[–]Mob_Of_One 0 points1 point2 points ago
[–]UloPe 0 points1 point2 points ago
[–]Mob_Of_One 1 point2 points3 points ago
[–]jamesshuang 3 points4 points5 points ago
[–]Doormatty 2 points3 points4 points ago
[–]vplatt 1 point2 points3 points ago
[–]Doormatty 1 point2 points3 points ago
[–]mackstann 1 point2 points3 points ago
[–]Doormatty 0 points1 point2 points ago
[–]fishdicks 2 points3 points4 points ago
[–]vplatt 0 points1 point2 points ago
[–]lucasvb 1 point2 points3 points ago
[–]crocowhile 1 point2 points3 points ago
[–]takluyverIPython, Py3, etc 1 point2 points3 points ago
[–]blatheringDolt 0 points1 point2 points ago
[–]servetus 0 points1 point2 points ago
[–]mkor 0 points1 point2 points ago
[–]irve 0 points1 point2 points ago
[–]stoph 0 points1 point2 points ago
[–]thantik 0 points1 point2 points ago
[–]stoph 0 points1 point2 points ago
[–]thantik 1 point2 points3 points ago*
[–]stoph 0 points1 point2 points ago
[–]thantik 0 points1 point2 points ago
[–]stoph 0 points1 point2 points ago