What you are asking for is called Arcball rotation. Quaternions are the easy solution only if you understand how they work. You can achieve the same without quaternions however.

Pre-requisites

Do you know how to rotate objects in general? Let’s say you have an object at the origin. Do you know how you rotate it (hint: multiply by some rotation matrix)? If yes, then I am assuming you know what will happen if you translate the object first and then rotate it?

You must know how to calculate a rotation matrix from angle-axis (easy as pie, look at the myriad of equations online, a lot of them give you the code as well)

Solution

• Get the camera’s up and right vectors. Note that they should be normalized.
• Get the vector from the focus point to the camera (camPosition – Focus). This is the vector that you are going to be rotating. Let’s call this camFocusVector.
• Decide how much you want to rotate in yaw/pitch in relation to the camera
• Create two rotation matrices. The 1st rotation matrix will use the up of the camera as the axis and yaw angle that you decided. The 2nd rotation matrix will use the right of the camera as the axis and pitch angle that you decided.
• Now rotate the camFocusVector with the new rotation matrices. This is now your new position of the camera relative to the origin. We of course, want it to be relative to the focus point…
• Add the focus point position to camFocusVector. This is now your camera’s new position. Translate your camera accordingly.
• Finally, ask the camera to focus on the focus point by calling your lookAt() function

Caveats

You will have to lookout for certain cases or singularities at which your camera will stop working. Looking straight down/up for example. I will let you figure out how to deal with those.

EDIT1: How to recalculate the orthonormal vectors of the camera

You already know the direction of the camera ( (cameraPos – focusPoint).normalize() ). Now assume that your camera’s up is +Y (or w/e your world’s current up axis is… that is up to you). Now simply cross the direction with the up to get the right. Done? Nope! Your up vector is no longer orthogonal to the other two. To fix that, cross right with direction and you get your new up.

Note that the Gram-Schmidt is really what should be used to orthonormalize vectors.

Again, take a note of the caveats as this will fail to work in some cases (direction is parallel to up for example).