This includes read-only buffers.

The code split between bytes and numpy array is removed and all objects
are now handled equally.

Furthermore, it seems to be more correct to increase the ref count for
None objects as well, while a pointer to that object is used.