I know its old, but maybe somebody else is facing this painful challange:

So a camera represented only by:

glm::quat mRotation;
glm::vec3 mPosition;

needs also keep track the Pitch and Yaw separately, and apply it one by one to the main mRotation quaternion. I haven’t found any other way to limit a quaternion otherwise.

So all in all the Camera class must have these members initialized to 0.0f:

glm::quat mRotation;
glm::vec3 mPosition;
float mPitch, mYaw;

During the frame by frame call, where you pass in the delta mouse movement values, the Pitch keeps track the cumulated mouse movement, and thats what you can clamp, then pass it into the quaternion as angel axis, here is how:

void Camera::rotate(const float horizontalMouseMove, const float verticalMouseMove)
{
    mPitch -= verticalMouseMove;
    mYaw -= horizontalMouseMove;

    //TODO: clamp Yaw?
    mRotation = glm::angleAxis(mYaw, axis::positiveY);

    //clamp Pitch
    constexpr float PITCH_LIMIT = 1.4f;//magic number, 1.0f is too flat to my taste
    if (mPitch < -PITCH_LIMIT)
    {
        mPitch = -PITCH_LIMIT;
    }
    else
    if (mPitch > PITCH_LIMIT)
    {
        mPitch = PITCH_LIMIT;
    }

    //quaternion multiplication is not commutative, and we have YAW rotation already applied mRotation
    mRotation = mRotation * glm::angleAxis(mPitch, getRight());
}