Vector graphics on the GPU with Stage3D

Hey guys and gals,

Started playing with vectors on the GPU last year but it sort of ground to a halt really, I didn’t like the ugly aliasing and let face it vectors should look smooth as a peach. Anyway, skip forward until flash player 11.6 beta was released supporting new shader op codes (namely ddx and ddy) woo, now I had the ability to add in that sweet sweet anti aliasing by leveraging the screen space derivatives!

Cool so now that I was able to render filled triangles and bezier curves (and soooooo close to cubic curves too – need help on this one though), I thought it should be easy to tie it in to the new readGraphicsData command as seen here:
Query Graphics Data

Sadly not quite as straight forward as you might think!
The readGraphicsData method returns all the information to serialize the graphics of any object with graphics in it in flash! But there are still a few difficulties with parsing this data for use on the GPU.

Here are some of the issues:

  • Determining if a bezier curve is concave or convex (this depends on what side of the closed region the curve is on)


(of the two curves on the left shape one is concave and one is convex, blue and red respectively and the shader needs tweaking dependant on which one it is)

  • Determining if two curves overlap (i.e. in long sweeping thin curves… in this case the curve may need to be broken down into smaller curves)
  • Gradients fills! A couple of problems here, 1 is reversing the gradient matrix and the other is to replicate it on the GPU. Simple gradient fills are possible but can get complex easily as you start added more than two colours (maybe at this point a 1×256 pixel texture could be used as a lookup) . That said I have not got round to this yet so might not be to bad.
  • TRIANGULATION!!!!!! This is the real problem here… (at least to me it is). So from the output of the readGraphicsData we have extracted all the curves and along the way we have collected a series of points. These points make up the triangles that we use to fill the solid sections of the shape. Things get tricky however because these points do NOT automatically make up a nice sequence of triangles, you will get overlapping issues and a whole host of other problems. So this is where the triangulation comes in. At first I tried Delaunay triangulation but it was too greedy making triangles outside of the actual shape, so no good. Then I tried some ear clipping examples that I found but only 1 of the 3 I tried kind of worked and I say “kind of” because  it goes into a number of infinite loops that I had to hardcode exceptions for :( and as such it misses a few triangles. (Also every now and again it would reject a complete path for no apparent reason). Not only that but the ear clipping algorithm is SLOW and doesn’t scale very well.

True vector graphics are great, they are something we love about the platform and will miss when the next major release of actionscript comes about. This is the reason why I though it would be great if we could emulate it on the GPU and I am sure it’s possible. It would however be 100 times easier if Adobe could extend their api to expose the result of the internal triangulation that they have already implemented in the player. That way they are still leaving it up to us to handle the rendering but we won’t have to spend an AGE trying to do work that they already have done a long time ago.

 

Anyway am bored of typing so I will post a small demo video (best viewed @ high resolution + full screen):

..Interactive demo coming soon.

 

 

If anyone wants to know more, discuss the topic, contribute, point me at an awesome triangulation library for as3, or anything else.. just drop me a message or reply to this post.

Related link(s) of justice:
http://www.bytearray.org/?p=5013

 

TODOs:

Use edge lists to enable anti aliasing on straight edges of standard (non curve) triangles.

Use edges to also determine if curve is convex or concave.

Look into cubic curve  - quadratic bezier curve conversion

 

UPDATE:

In an ideal world I would like this to become a small Open Source project. One that is not geared to any specific engine or renderer just a simple tool that can be used to generate the required data from any flash display object. If anyone is interested get in touch, I will be more than happy so share the code once its a bit more optimized and would love to see this become something useful not just for the developers of a particular engine but for all flash devs :-)

30 Responses to “Vector graphics on the GPU with Stage3D”

  1. Nice work Ben! So when will B3D be finally released ?

  2. bwhiting says:

    If I stick the current rate of development should be sometime between the apocalypse and hell freezing over :(

  3. It is amazing. As I understand, all work done in pixel shader?

  4. bwhiting says:

    Yeah and a pretty light shader at that ;)

  5. Dude, that is freaking amazing!!!

    Adobe should absolutely be doing anything and everything it can to help you along with the work you’ve already started. Definitely get this under Thibault’s nose and see if he can help out in any way.

  6. This looks like a really great way to preserve legacy assets and even continue using Flash Pro as an asset production tool. Using ddx and ddy produces much better results, but could there still be a mode where they’re not used for times where the extended profile isn’t available? Even if the quality is worse it’s better to have something than nothing.

    Keep up the good work!

  7. bwhiting says:

    Yeah that would not be a problem at all, the screen space derivatives are only used for the anti aliasing not as an integral part of the shaders themselves so can easily be omitted.

  8. [...] on blog.bwhiting.co.uk Share this:TwitterFacebookLike this:LikeBe the first to like [...]

  9. radu birsan says:

    Are you saying that vector graphics will not be supported in AS4?

  10. bwhiting says:

    There will still be the vector graphics… just no display list to render them. The idea being you can still create nice graphics in the IDE or with code and then render the result to a bitmap for use on the GPU. So no more native display list as we know it.

  11. Wilson Silva says:

    This looks great! I sent an email to Thibault, asking if Adobe could help you out on this, because they could benefit from it too.

  12. Jon says:

    RE: So no more native display list as we know it.

    Where did you read/hear that?

  13. Elliot Geno says:

    Awesome work dude! I can’t wait until things settle down with the platform. I can envision the entire displayList converted to the GPU in the near future. If FlashPro could just export it, this would be amazing!

  14. Gil Amran says:

    No Native Display list!?!?
    HUH!? where did you hear that?

  15. HB says:

    Yeah, would also like to hear more about that “no more classic display list” statement, as that means most UI frameworks out there will be useless without extensive rewrites when AS 4 will be out late this year.

  16. Om says:

    Ben,

    I came across this blog post and what you have demoed here looks amazing. I am playing with creating Flex components with GPU based rendering and thought I could take a look at your code on how you rendered vectors using the GPU. Can you please post your code somewhere or drop me an email?

    Thanks,
    Om

  17. Mem's says:

    Nice job!

    Like, Om, before waiting a nice & clean release of a lib, could you publish your raw code used to convert QueryGraphicData to GPU data? It’s should be really nice! ;)

    Thanks

  18. nicoptere says:

    hi,
    great job (all of it) :)

    I belieive the Flash rendering engine is not triangulating anything – it’s rather a “draw this line until you reach another shape” approach – hence it’s speed and some memorable glitches :)

    about triangulation, as you mentioned, Delaunay won’t be of any help.
    to perform a constrained triangulation, you’ll need to compute the Voronoi diagram and check wether each edge belongs to (or intersects) both sets in which case, it would be part of the “original” polygon (or so I vaguely remember).
    quite an overhead anyway, there are articles on how to compute a triangulation on the GPU but I think it’s not what you’re after

    ear cutting is more suited, this was helpful in a former project :
    http://forum.unity3d.com/threads/27223-Polygon-triangulation-code and http://www.unifycommunity.com/wiki/index.php?title=Triangulator

    as the ones you tried, this might just fail miserably on simple cases (“complex” polygons namely), if it can be of any help, I ported it to AS3.

    the https://code.google.com/p/poly2tri/ library can triangulate complex polygons pretty well.
    I don’t think this can be ported to AGAL :)

    finally, I’m pretty sure you’ve seen it but this paper addresses the subject in depth : http://http.developer.nvidia.com/GPUGems3/gpugems3_ch25.html

    cheers

  19. bwhiting says:

    Hey nicoptere, I have followed your work for ages :)

    Thanks for the comment/links/info! Yeah I am not trying to triangulate on the GPU just provide an easy way to convert graphics data to triangles that can be more easily stored and reused at a later date, i.e. for game ui’s, assets, animations and stuff like that.

    A lot of what I did was indeed based on that paper but they don’t go into enough detail on the triangulation side of things for what I still need to figure out.

    I will check out that poly2tri lib, it looks quite extensive and will probably be better than what I am currently using.
    I have been INSANELY busy at work these last two weeks but things should calm down soon, and I will try to integrate that library and lob everything so far on github – see if I can’t get some fellow flashers to dive in and have a play.

    b

  20. Fill says:

    Hi man, what about open source?) Your demo is amazing!

  21. bwhiting says:

    It will be happening soon! Have made a start but am mental busy at the mo. Was hoping to have something ready for last weekend but got overrun. I will get there though! This post will be updated when it is there.

  22. Nimai says:

    I’m very interested in this! Would love to contribute as soon as you have time to make the project available. I’ve implemented many 3D exporters and triangulation routines and found this post by searching to see if anyone had done it for Flash yet. Thanks for your work on this!

  23. bwhiting says:

    There really isn’t that much, so no excuse for me not having sorted this out sooner. My only problem is that it makes use of the “ddx” and “ddy” shader opcodes and I wasn’t sure if adobe was dropping support for them or not. Without them the vector graphics cannot be anti-aliased as and far as I am concerned that is an absolute must!!

    I will get it on github this week in some form or other so folks can grab it and work on improving it.

  24. Acrossfy says:

    Hi, i tried to make my own vector renderer on GPU, but i have many troubles, which you have no. My method described in video description – http://www.youtube.com/watch?v=vh3jlAgGVnE

  25. Michi_Ki says:

    Hi there!

    Amazing piece of work you did!

    I’ve been digging into graphics development for a while now and I’m sure
    I can contribute something to your project as soon as it goes open source.
    Could you perhaps give me access to the source code of your demo?
    (I don’t mind if it’s not polished yet.)

    Keep up the good posts coming!!

  26. Andy says:

    Please answer just for one question! How did you get the anchor coordinates from (IGraphics) GraphicsPath for your bezier curve?
    Thanks in advance!)

  27. aoakenfo says:

    What’s the status of this project? And why isn’t Adobe working on this? Flash is all about vectors, I don’t know why they continue to go down the bitmap path.

  28. radu birsan says:

    I think Adobe is buying everybody that wants to tap in to the potential of this approach

  29. Rob says:

    Hey Ben!

    Very interested in the work you are doing – would love to talk more about some similar areas we are currently working on for Away3D. Could you drop me a mail some time?

    cheers

    Rob

  30. xaibot says:

    Hi Ben!

    I just found this post while looking for “stage3d vector graphics”. It is very interesting, so thanks a lot for sharing it.

    I noticed you created a repository for it on github: https://github.com/bwhiting/b3d but the code is missing. It would be great if you could push the remaining files for others to continue your work on it. (Thanks a lot in advance).

    Best Regards.

Leave a Reply