I'm currently working on an marker-based Augmented Reality project. The AR system is based on OpenCV and works well so far. I have a working prototype here that uses OpenGL to display simple colored quads on top of found markers.
The basic approach is that a 3D projection matrix is calculated from some camera properties. A marker gets detected and it's 3D pose is estimated. An OpenGL matrix for its transform information is calculated. A "scale factor" is multiplied to scale the coordinate system so that 1 OpenGL unit corresponds to a marker side length. This 3 pieces of information (projection matrix, transform matrix, and scale) can be used inside an OpenGL scene to correctly place the marker overlay in the scene.
I want to display sprites later and take advantage of some simple effects and animations and I don't intend to develop this on my own, that's why I came across Cocos2D. I see the "2D" in the name and I know that there is Cocos3D. However, Cocos3D hasn't seen active development in the last months and I'm not sure if it has what I need. I also don't want to use Unity3D, since it is too bloated for this case.
I researched a bit and found out that Cocos2D has the ability to set a custom projection matrix via
[CCDirector setProjection:CCDirectorProjectionCustom]. Bingo. Now my idea was that I have a special "3D Node" called
CCNodeAR which contains the 3D transform matrix for a marker to place it correctly in the scene. Now each
CCSprite that I add to such a node should be placed on a plane specified by this special node's 3D transform matrix. I furthermore set a scale factor on this sprite so that 1 OpenGL unit corresponds to the marker side length. By this, I can have 2D sprites that are correctly placed in the 3D scene on top of the markers.
At least this was the theory. In practice, however, I basically have the problem that there seems to be something between the projection matrix and the 3D transform matrix so that the real marker pose does not correspond to the OpenGL overlay. Large "real" camera movements become very small marker overlay movements in the OpenGL overlay. Furthermore, objects are clipped on the "near plane" and the sprites are displayed as being huge, if I do not scale them down further. Last but not least, the application always starts in "portrait" mode although I disabled it and the iPad needs to be turned so that it switches to "landscape" mode -- but this is a minor problem for now.
I tried to understand the way how projection and transform matrices are handled in Cocos2D but I could not dig down to the point where the 3D transform matrix of my
CCNodeAR gets multiplied with something that makes it behave oddly. Does anybody have an advice for me? You are very invited to have look at the code that I have created so far for the project. It's not so much yet, it's well documented and it's on github: https://github.com/htw-inka/ocv_ar-examples/tree/cocos2d/examples/ios/OcvARCocos2D
I'm using Cocos2D 3.1 and I'm running my tests on an iPad 3.