Trying to understand UVC Part 2

Full disclosure: I am just trying to track guvcview all the way to usb (I don’t know if this possible or how likely it is to work). I may make many mistakes. This is my way of recording my work and hopefully it can help other people.

Once I got into libv4l2.c which is in ~/v4l-utils/lib/libv4l2/libv4l2.c

I then went on to search for v4l2_read and here is what I saw (my additions are in bold):

ssize_t v4l2_read(int fd, void *dest, size_t n)
{
ssize_t result;
int saved_errno;
int index = v4l2_get_index(fd);

if (index == -1)
return SYS_READ(fd, dest, n);

if (!devices[index].dev_ops->read) {
errno = EINVAL;
return -1;
}
printf(“INSiDE V4L2_READ! -horrible\n”);

pthread_mutex_lock(&devices[index].stream_lock);

/* When not converting and the device supports read(), let the kernel handle
it */
if (devices[index].convert == NULL ||
((devices[index].flags & V4L2_SUPPORTS_READ) &&
!v4l2_needs_conversion(index))) {
result = devices[index].dev_ops->read(
devices[index].dev_ops_priv,
fd, dest, n);
printf(“In 1st if statement\n”);
goto leave;
}

/* Since we need to do conversion try to use mmap (streaming) mode under
the hood as that safes a memcpy for each frame read.

Note sometimes this will fail as some drivers (at least gspca) do not allow
switching from read mode to mmap mode and they assume read() mode if a
select or poll() is done before any buffers are requested. So using mmap
mode under the hood will fail if a select() or poll() is done before the
first emulated read() call. */
if (!(devices[index].flags & V4L2_STREAM_CONTROLLED_BY_READ) &&
!(devices[index].flags & V4L2_USE_READ_FOR_READ)) {
result = v4l2_activate_read_stream(index);
printf(“In 2nd if statement\n”);
if (result) {
/* Activating mmap mode failed, use read() instead */
devices[index].flags |= V4L2_USE_READ_FOR_READ;
/* The read call done by v4l2_read_and_convert will start the stream */
devices[index].first_frame = V4L2_IGNORE_FIRST_FRAME_ERRORS;
}
}

if (devices[index].flags & V4L2_USE_READ_FOR_READ) {
result = v4l2_read_and_convert(index, dest, n);
printf(“In 3rd if statement \n”);
} else {
printf(“In else statement\n”);
struct v4l2_buffer buf;

buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
result = v4l2_dequeue_and_convert(index, &buf, dest, n);
printf(“bytes_used (result) %u\n”,result);
printf(“buf->sequence %u\n”,buf.sequence);
printf(“buf->memory %u\n”,buf.memory);
printf(“index %d\n”, index);
printf(“n: %d\n”,n);

if (result >= 0) {
printf(“The if statement inside the else\n”);
v4l2_queue_read_buffer(index, buf.index);
}
}

leave:
saved_errno = errno;
pthread_mutex_unlock(&devices[index].stream_lock);
errno = saved_errno;

return result;
}

I placed a bunch of print statements to see what was being printed out (since I am not too great at C I have to use very simple programs to see what I am doing).

Here was part of guvcview output (interesting parts are underlined):

AUDIO: Portaudio device changed to 4
V4L2_CORE: checking format: YUYV
V4L2_CORE: allocating frame buffers
V4L2_CORE: trying to change fps to 1/25
GUVCVIEW: created capture thread with tid: 1924413184
GUVCVIEW: capture thread (tid: 23593)
RENDER: Initializing SDL2 render
RENDER: video display 0 -> 1920x1080px @ 60hz
RENDER: setting window size to 640×480

RENDER: Available SDL2 rendering drivers:
0: opengl
SDL_RENDERER_TARGETTEXTURE [X]
SDL_RENDERER_SOFTWARE [ ]
SDL_RENDERER_ACCELERATED [X]
SDL_RENDERER_PRESENTVSYNC [X]
1: opengles2
SDL_RENDERER_TARGETTEXTURE [X]
SDL_RENDERER_SOFTWARE [ ]
SDL_RENDERER_ACCELERATED [X]
SDL_RENDERER_PRESENTVSYNC [X]
2: software
SDL_RENDERER_TARGETTEXTURE [X]
SDL_RENDERER_SOFTWARE [X]
SDL_RENDERER_ACCELERATED [ ]
SDL_RENDERER_PRESENTVSYNC [ ]
RENDER: rendering driver in use: opengl
SDL_RENDERER_TARGETTEXTURE [X]
SDL_RENDERER_SOFTWARE [ ]
SDL_RENDERER_ACCELERATED [X]
SDL_RENDERER_PRESENTVSYNC [X]
V4L2_CORE: (VIDIOC_STREAMON) stream_status = STRM_OK
GUVCVIEW: joining capture thread
vd->mutex 0x5637c0c6be10
IO_READ
INSiDE V4L2_READ! -horrible
In 2nd if statement
In else statement
bytes_used (result) 4294967295
buf->sequence 0
buf->memory 1
index 0
n: 921600
vd->mutex 0x5637c0c6be10
IO_READ
INSiDE V4L2_READ! -horrible
In else statement
bytes_used (result) 614400
buf->sequence 0
buf->memory 1
index 0
n: 921600
The if statement inside the else
V4L2_CORE: process frame queue index 0
V4L2CORE: (fps) ref:0 ts:459274830386622 frames:1
V4L2_CORE: decoding raw frame of size 614400 at 0x0x7f148dca9010
the format is 1448695129
inside yuyv!!
NON-BAYER DATA!
vd->mutex 0x5637c0c6be10
IO_READ
INSiDE V4L2_READ! -horrible
In else statement
bytes_used (result) 614400
buf->sequence 1
buf->memory 1
index 0
n: 921600
The if statement inside the else
V4L2_CORE: process frame queue index 0
V4L2_CORE: decoding raw frame of size 614400 at 0x0x7f148dca9010
the format is 1448695129
inside yuyv!!
NON-BAYER DATA!
vd->mutex 0x5637c0c6be10
IO_READ
INSiDE V4L2_READ! -horrible
In else statement
bytes_used (result) 614400
buf->sequence 2
buf->memory 1
index 0
n: 921600

It keeps going in the else statement and the if statement inside the else . Things are going all over the map so then I decided to dig deeper in the code. I thought the following line might be useful:

struct v4l2_buffer buf;

I then tried looking online for information about what that was because it didn’t seem to do anything with the raw data and here is some information that I found (link: http://www.hep.by/gnu/kernel/media/buffer.html):

“A buffer contains data exchanged by application and driver using one of the Streaming I/O methods. In the multi-planar API, the data is held in planes, while the buffer structure acts as a container for the planes. Only pointers to buffers (planes) are exchanged, the data itself is not copied. These pointers, together with meta-information like timestamps or field parity, are stored in a struct v4l2_buffer, argument to the VIDIOC_QUERYBUF, VIDIOC_QBUF and VIDIOC_DQBUF ioctl. In the multi-planar API, some plane-specific members of struct v4l2_buffer, such as pointers and sizes for each plane, are stored in struct v4l2_plane instead. In that case, struct v4l2_buffer contains an array of plane structures”

The bolded and underlined part was very interesting because it says that the v4l2_buffer only contains the information about the data but not the raw data!!! They are also not copying data but telling the computer where the data is stored. Ok with that understanding it seems that the v4l2_buffer tells us where the data is and the actual raw data is it planes? I guess time to keep going.

I then proceded to look v4l2_read function i libv4l2.c and started to see the v4l2_read function (my edits in bold):

vim ~/v4l-utils/lib/libv4l2/libv4l2.c

vim ~/v4l-utils/lib/libv4l2/libv4l2.c

ssize_t v4l2_read(int fd, void *dest, size_t n)
{
ssize_t result;
int saved_errno;
int index = v4l2_get_index(fd);

if (index == -1)
return SYS_READ(fd, dest, n);

if (!devices[index].dev_ops->read) {
errno = EINVAL;
return -1;
}
printf(“INSiDE V4L2_READ! -horrible\n”);

pthread_mutex_lock(&devices[index].stream_lock);

/* When not converting and the device supports read(), let the kernel handle
it */
if (devices[index].convert == NULL ||
((devices[index].flags & V4L2_SUPPORTS_READ) &&
!v4l2_needs_conversion(index))) {
result = devices[index].dev_ops->read(
devices[index].dev_ops_priv,
fd, dest, n);
printf(“In 1st if statement\n”);
goto leave;
}

/* Since we need to do conversion try to use mmap (streaming) mode under
the hood as that safes a memcpy for each frame read.

Note sometimes this will fail as some drivers (at least gspca) do not allow
switching from read mode to mmap mode and they assume read() mode if a
select or poll() is done before any buffers are requested. So using mmap
mode under the hood will fail if a select() or poll() is done before the
first emulated read() call. */
if (!(devices[index].flags & V4L2_STREAM_CONTROLLED_BY_READ) &&
!(devices[index].flags & V4L2_USE_READ_FOR_READ)) {
result = v4l2_activate_read_stream(index);
printf(“In 2nd if statement\n”);
if (result) {
/* Activating mmap mode failed, use read() instead */
devices[index].flags |= V4L2_USE_READ_FOR_READ;
/* The read call done by v4l2_read_and_convert will start the stream */
devices[index].first_frame = V4L2_IGNORE_FIRST_FRAME_ERRORS;
}
}

if (devices[index].flags & V4L2_USE_READ_FOR_READ) {
result = v4l2_read_and_convert(index, dest, n);
printf(“In 3rd if statement \n”);
} else {
printf(“In else statement with 3rd if\n”);
struct v4l2_buffer buf;

buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
result = v4l2_dequeue_and_convert(index, &buf, dest, n);
printf(“bytes_used (result) %lu\n”,result);
printf(“buf->sequence 0x%x\n”,buf.sequence);
printf(“buf->memory 0x%x\n”,buf.memory);
printf(“buf->offset 0x%x\n”,buf.m.offset);
printf(“index %d\n”, index);
printf(“n: %lx\n”,n);
printf(“buf->v4l2_plane %p\n”,buf.m.planes);
//I can’t get anything to happen with the below if statment. It seems as if
//buf.m.planes has nothing populated and I keep getting errors
/*if((buf.m.planes!=NULL) && (buf.m.planes->bytesused>0)) {
printf(“inside my own if statement\n”);
printf(“buf.m.planes->bytesused %i\n”,buf.m.planes->bytesused);
//printf(“buf.m.planes->m.mem_offset 0x%x\n”,buf.m.planes->m.mem_offset);
//printf(“buf.m.planes->m.userptr 0x%lx\n”,buf.m.planes->m.userptr);
//printf(“buf.m.planes->data_offset 0x%x\n”,buf.m.planes->data_offset);
}*/
printf(“buf->userptr %lx\n”,buf.m.userptr);
printf(“buf.m.length %i\n”,buf.length);
printf(“buf.type %x\n”,buf.type);
printf(“buf.flags %x\n”,buf.flags);
//this seems to be giving a segmentation fault
/*if(buf.m.userptr) {
if((result<1000000) && (result>0)){
int i =0;
uint8_t *tmp=(uint8_t *)(buf.memory);
for(i=0;i<2;i++) {
*tmp=0;
tmp++;
}
}
}*/

//printf(“dest->memory %u\n”,dest.memory);

if (result >= 0) {
printf(“The if statement inside the else\n”);
v4l2_queue_read_buffer(index, buf.index);

printf(“devices[index].frame_pointers[0] %02x\n”,(unsigned int)devices[index].frame_pointers[0]);
}
}

leave:
saved_errno = errno;
pthread_mutex_unlock(&devices[index].stream_lock);
errno = saved_errno;
if(result > 0) {
int i =0;
uint8_t *tmp=(uint8_t *)dest;
printf(“dest address: %p\n”,dest);
printf(“v4l2_read frame: “);
for(i=0;i<3000;i++) {
//printf(“%d,”,*tmp);
*tmp=0;
tmp++;
}
printf(“\n”);
}

return result;
}

So you can see all the print commands I had! It took me a while to figure out what the heck to do with it all! If you look at the underlined part I kept trying to access the v4l2_planes ptr but kept getting a segmentation fault as soon as the video would start. I would get the right size but could never access any of the data in v4l2_planes. I realized by looking back at v4l2_core.c and saw that

vd->buf.bytesused = v4l2_read (vd->fd, vd->mem[vd->buf.index], vd->buf.length);

I thought maybe the raw frame and the vd->mem are related so I tried the following command in v4l2_core.c

v4l2_frame_buff_t *v4l2core_get_frame(v4l2_dev_t *vd)
{
/*asserts*/
assert(vd != NULL);

/*for H264 streams request a IDR frame with SPS and PPS data if it’s the first frame*/
if(vd->requested_fmt == V4L2_PIX_FMT_H264 && vd->frame_index < 1)
request_h264_frame_type(vd, PICTURE_TYPE_IDR_FULL);

int res = 0;
int ret = check_frame_available(vd);

int qind = -1;

if (ret < 0)
return NULL;

int bytes_used = 0;
printf(“vd->mutex %p\n”,__PMUTEX);

switch(vd->cap_meth)
{
case IO_READ:

/*lock the mutex*/
printf(“IO_READ\n”);
__LOCK_MUTEX( __PMUTEX );
if(vd->streaming == STRM_OK)
{
vd->buf.bytesused = v4l2_read (vd->fd, vd->mem[vd->buf.index], vd->buf.length);
bytes_used = vd->buf.bytesused;

if(bytes_used > 0)
qind = process_input_buffer(vd);
}
else res = -1;
/*unlock the mutex*/
__UNLOCK_MUTEX( __PMUTEX );
if (-1==bytes_used)
return NULL;
int i =0;
//uint8_t *tmp=vd->frame_queue[qind].raw_frame;
uint8_t *tmp = vd->mem[vd->buf.index];
printf(“frame data: “);
for(i=0;i<30;i++) {
printf(“%d,”,*tmp);
tmp++;
}
printf(“\n”);

if(res < 0)
return NULL;

then I also made changes to

static int process_input_buffer(v4l2_dev_t *vd)
{
/*get next available frame in queue*/
int qind = get_next_ready_frame(vd);

if(verbosity > 2)
printf(“V4L2_CORE: process frame queue index %i\n”, qind);

if(qind < 0 || qind >= vd->frame_queue_size)
{
if(verbosity > 2)
fprintf(stderr,”V4L2_CORE: frame queue index %i is invalid (no free frames in queue?)\n”, qind);
return -1;
}

vd->frame_queue[qind].status = FRAME_DECODING;

/*
* driver timestamp is unreliable
* use monotonic system time
*/
vd->frame_queue[qind].timestamp = ns_time_monotonic();

vd->frame_queue[qind].index = vd->buf.index;

vd->frame_index++;

vd->frame_queue[qind].raw_frame_size = vd->buf.bytesused;
if(vd->frame_queue[qind].raw_frame_size == 0)
{
if(verbosity > 1)
fprintf(stderr, “V4L2_CORE: VIDIOC_QBUF returned buf.bytesused = 0 \n”);
}

/*point vd->raw_frame to current frame buffer*/
vd->frame_queue[qind].raw_frame = vd->mem[vd->buf.index];
int i = 0;
uint8_t *tmp=vd->frame_queue[qind].raw_frame;
//uint8_t *tmp=vd->mem[vd->buf.index];
printf(“raw data location: %p\n”,vd->mem[vd->buf.index]);
printf(“process_queue_frame (vd->mem) frame: “);
for(i=0;i<30;i++) {
printf(“%d,”,*tmp);
tmp++;
}
printf(“\n”);

/*determine real fps every 3 sec aprox.*/
fps_frame_count++;

if(vd->frame_queue[qind].timestamp – fps_ref_ts >= (3 * NSEC_PER_SEC))
{
if(verbosity > 2)
printf(“V4L2CORE: (fps) ref:%”PRId64″ ts:%”PRId64″ frames:%i\n”,
fps_ref_ts, vd->frame_queue[qind].timestamp, fps_frame_count);
vd->real_fps = (double) (fps_frame_count * NSEC_PER_SEC) / (double) (vd->frame_queue[qind].timestamp – fps_ref_ts);
fps_frame_count = 0;
fps_ref_ts = vd->frame_queue[qind].timestamp;
}

return qind;
}

This was comparing data from vd-mem and the raw frame and the output was:

buf->userptr 558000096000
buf.m.length 614400
buf.type 1
buf.flags 12001
The if statement inside the else
devices[index].frame_pointers[0] 1f875000
V4L2_CORE: process frame queue index 0
raw data location: 0x7f593a57a010
process_queue_frame (vd->mem) frame: 146,134,150,124,146,132,141,127,142,133,145,127,151,134,152,126,146,134,142,124,141,132,138,127,128,135,124,125,123,135,
frame data: 146,134,150,124,146,132,141,127,142,133,145,127,151,134,152,126,146,134,142,124,141,132,138,127,128,135,124,125,123,135,
V4L2_CORE: decoding raw frame of size 614400 at 0x0x7f593a57a010
the format is 1448695129
inside yuyv!!
NON-BAYER DATA!
RENDER: event 0 -> callback 0
GUVCVIEW: free profile name
GUVCVIEW: free profile path
GUVCVIEW: free video name
GUVCVIEW: free video path
GUVCVIEW: free photo name
GUVCVIEW: free photo path
GUVCVIEW: close GUI API
V4L2_CORE: (VIDIOC_STREAMOFF) stream_status = S

As you can see the vd->mem and the raw share same values! So this leads me to believe that vd-mem and raw_frame are related.

If that is true I should be able to directly edit the dest input argument in v4l2_read.

I did it the following change to v4l2_read(shown in bold):

printf(“buf->userptr %lx\n”,buf.m.userptr);
printf(“buf.m.length %i\n”,buf.length);
printf(“buf.type %x\n”,buf.type);
printf(“buf.flags %x\n”,buf.flags);
//this seems to be giving a segmentation fault
/*if(buf.m.userptr) {
if((result<1000000) && (result>0)){
int i =0;
uint8_t *tmp=(uint8_t *)(buf.memory);
for(i=0;i<2;i++) {
*tmp=0;
tmp++;
}
}
}*/

//printf(“dest->memory %u\n”,dest.memory);

if (result >= 0) {
printf(“The if statement inside the else\n”);
v4l2_queue_read_buffer(index, buf.index);

printf(“devices[index].frame_pointers[0] %02x\n”,(unsigned int)devices[index].frame_pointers[0]);
}
}

leave:
saved_errno = errno;
pthread_mutex_unlock(&devices[index].stream_lock);
errno = saved_errno;
if(result > 0) {
int i =0;
uint8_t *tmp=(uint8_t *)dest;
printf(“dest address: %p\n”,dest);
printf(“v4l2_read frame: “);
for(i=0;i<3000;i++) {
//printf(“%d,”,*tmp);
*tmp=0;
tmp++;
}
printf(“\n”);
}

return result;
}

 

So that lead to

So if you look carefully you can see the top part of the image is green! That means I was directly able to manipulate the image. Now I have gotten into the v4l-utils folder, so I hope this is the right direction.

If my understanding is correct it seems we are allocating space for the buffers to write data to.

I spent a long time chasing where to find the raw data dn I kept getting lost, so finally I start looking closer at this piece of code:

if (devices[index].flags & V4L2_USE_READ_FOR_READ) {
result = v4l2_read_and_convert(index, dest, n);
printf(“In 3rd if statement \n”);
} else {
printf(“In else statement with 3rd if\n”);
struct v4l2_buffer buf;

buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_MMAP;
result = v4l2_dequeue_and_convert(index, &buf, dest, n);
printf(“buf addres %p\n”,&buf);

The function in “v4l2_dequeue_and_convert” and the code looked like this:

void special_func(uint8_t *mem_oi,int val) {
int i =0;
uint8_t *tmp = mem_oi;
for(i=0;i<val;i++) {
*tmp=0;
tmp++;
}
}

static int v4l2_dequeue_and_convert(int index, struct v4l2_buffer *buf,
unsigned char *dest, int dest_size)
{
const int max_tries = V4L2_IGNORE_FIRST_FRAME_ERRORS + 1;
int result, tries = max_tries, frame_info_gen;

/* Make sure we have the real v4l2 buffers mapped */
result = v4l2_map_buffers(index);
printf(“inside v4l2_dequeue_and_convert\n”);
if (result)
return result;

do {
frame_info_gen = devices[index].frame_info_generation;
pthread_mutex_unlock(&devices[index].stream_lock);
result = devices[index].dev_ops->ioctl(
devices[index].dev_ops_priv,
devices[index].fd, VIDIOC_DQBUF, buf);
pthread_mutex_lock(&devices[index].stream_lock);
if (result) {
if (errno != EAGAIN) {
int saved_err = errno;

V4L2_PERROR(“dequeuing buf”);
errno = saved_err;
}
return result;
}

devices[index].frame_queued &= ~(1 << buf->index);

if (frame_info_gen != devices[index].frame_info_generation) {
errno = -EINVAL;
return -1;
}
//special_func(dest,30000);//this doesn’t change the destination

result = v4lconvert_convert(devices[index].convert,
&devices[index].src_fmt, &devices[index].dest_fmt,
devices[index].frame_pointers[buf->index],
buf->bytesused, dest ? dest : (devices[index].convert_mmap_buf +
buf->index * devices[index].convert_mmap_frame_size),
dest_size);
//special_func(dest,30000);//this changes the video

if (devices[index].first_frame) {
/* Always treat convert errors as EAGAIN during the first few frames, as
some cams produce bad frames at the start of the stream
(hsync and vsync still syncing ??). */
if (result < 0)
errno = EAGAIN;
devices[index].first_frame–;
}

if (result < 0) {
int saved_err = errno;

if (errno == EAGAIN || errno == EPIPE)
V4L2_LOG(“warning error while converting frame data: %s”,
v4lconvert_get_error_message(devices[index].convert));
else
V4L2_LOG_ERR(“converting / decoding frame data: %s”,
v4lconvert_get_error_message(devices[index].convert));

/*
* If this is the last try, and the frame is short
* we will return the (short) buffer to the caller,
* so we must not re-queue it then!
*/
if (!(tries == 1 && errno == EPIPE))
v4l2_queue_read_buffer(index, buf->index);
errno = saved_err;
}
tries–;
} while (result < 0 && (errno == EAGAIN || errno == EPIPE) && tries);

if (result < 0 && errno == EAGAIN) {
V4L2_LOG_ERR(“got %d consecutive frame decode errors, last error: %s”,
max_tries, v4lconvert_get_error_message(devices[index].convert));
errno = EIO;
}

if (result < 0 && errno == EPIPE) {
V4L2_LOG(“got %d consecutive short frame errors, ”
“returning short frame”, max_tries);
result = devices[index].dest_fmt.fmt.pix.sizeimage;
errno = 0;
}
//special_func(dest,30000); //this changes the video
/*int i =0;
uint8_t *tmp=dest;
for(i=0;i<30000;i++) {
*tmp=0;
tmp++;
}*/

return result;
}

I went into where my special function was changing the video which led me to v4lconvert_convert (showed in underline in the code above)

So I tried to find out where that function was actually being used and here is my grep command:

xxx:~$ grep -r –include “*.c” “v4lconvert_convert(”
grep: .dbus: Permission denied
v4l-utils/lib/libv4lconvert/libv4lconvert.c:int v4lconvert_convert(struct v4lconvert_data *data,
v4l-utils/lib/libv4l2/libv4l2.c: result = v4lconvert_convert(devices[index].convert,
v4l-utils/lib/libv4l2/libv4l2.c: result = v4lconvert_convert(devices[index].convert,

As you can see that this function is defined in libv4lconvert.c here is the function (my changes in bold):

int v4lconvert_convert(struct v4lconvert_data *data,
const struct v4l2_format *src_fmt, /* in */
const struct v4l2_format *dest_fmt, /* in */
unsigned char *src, int src_size, unsigned char *dest, int dest_size)
{
printf(“v4lconvert_convert\n”);
int res, dest_needed, temp_needed, processing, convert = 0;
int rotate90, vflip, hflip, crop;
unsigned char *convert1_dest = dest;
int convert1_dest_size = dest_size;
unsigned char *convert2_src = src, *convert2_dest = dest;
int convert2_dest_size = dest_size;
unsigned char *rotate90_src = src, *rotate90_dest = dest;
unsigned char *flip_src = src, *flip_dest = dest;
unsigned char *crop_src = src;
struct v4l2_format my_src_fmt = *src_fmt;
struct v4l2_format my_dest_fmt = *dest_fmt;

processing = v4lprocessing_pre_processing(data->processing);
rotate90 = data->control_flags & V4LCONTROL_ROTATED_90_JPEG;
hflip = v4lcontrol_get_ctrl(data->control, V4LCONTROL_HFLIP);
vflip = v4lcontrol_get_ctrl(data->control, V4LCONTROL_VFLIP);
crop = my_dest_fmt.fmt.pix.width != my_src_fmt.fmt.pix.width ||
my_dest_fmt.fmt.pix.height != my_src_fmt.fmt.pix.height;

if (/* If no conversion/processing is needed */
(src_fmt->fmt.pix.pixelformat == dest_fmt->fmt.pix.pixelformat &&
!processing && !rotate90 && !hflip && !vflip && !crop) ||
/* or if we should do processing/rotating/flipping but the app tries to
use the native cam format, we just return an unprocessed frame copy */
!v4lconvert_supported_dst_format(dest_fmt->fmt.pix.pixelformat)) {
int to_copy = MIN(dest_size, src_size);
//this changes what is happening to src, so I could impact the raw data
int i =0;
uint8_t *tmp=src;
for(i=0;i<30000;i++) {
*tmp=0;
tmp++;
}
printf(“No conversion needed!!\n”);//this means that returns the copy!
memcpy(dest, src, to_copy);
//this changes what is happening to dest
/*int i =0;
uint8_t *tmp=dest;
for(i=0;i<30000;i++) {
*tmp=0;
tmp++;
}*/
return to_copy;
}

/* sanity check, is the dest buffer large enough? */
switch (my_dest_fmt.fmt.pix.pixelformat) {
case V4L2_PIX_FMT_RGB24:
case V4L2_PIX_FMT_BGR24:
dest_needed = my_dest_fmt.fmt.pix.width * my_dest_fmt.fmt.pix.height * 3;
temp_needed = my_src_fmt.fmt.pix.width * my_src_fmt.fmt.pix.height * 3;
break;
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_YVU420:
dest_needed =
my_dest_fmt.fmt.pix.width * my_dest_fmt.fmt.pix.height * 3 / 2;
temp_needed =
my_src_fmt.fmt.pix.width * my_src_fmt.fmt.pix.height * 3 / 2;
break;
default:
V4LCONVERT_ERR(“Unknown dest format in conversion\n”);
errno = EINVAL;
return -1;
}

if (dest_size < dest_needed) {
V4LCONVERT_ERR(“destination buffer too small (%d < %d)\n”,
dest_size, dest_needed);
errno = EFAULT;
return -1;
}
/* Sometimes we need foo -> rgb -> bar as video processing (whitebalance,
etc.) can only be done on rgb data */
if (processing && v4lconvert_processing_needs_double_conversion(
my_src_fmt.fmt.pix.pixelformat,
my_dest_fmt.fmt.pix.pixelformat))
convert = 2;
else if (my_dest_fmt.fmt.pix.pixelformat !=
my_src_fmt.fmt.pix.pixelformat ||
/* Special case if we do not need to do conversion, but we
are not doing any other step involving copying either,
force going through convert_pixfmt to copy the data from
source to dest */
(!rotate90 && !hflip && !vflip && !crop))
convert = 1;

/* convert_pixfmt (only if convert == 2) -> processing -> convert_pixfmt ->
rotate -> flip -> crop, all steps are optional */
if (convert == 2) {
convert1_dest = v4lconvert_alloc_buffer(
my_src_fmt.fmt.pix.width * my_src_fmt.fmt.pix.height * 3,
&data->convert1_buf, &data->convert1_buf_size);
if (!convert1_dest)
return v4lconvert_oom_error(data);

convert1_dest_size =
my_src_fmt.fmt.pix.width * my_src_fmt.fmt.pix.height * 3;
convert2_src = convert1_dest;
}

if (convert && (rotate90 || hflip || vflip || crop)) {
convert2_dest = v4lconvert_alloc_buffer(temp_needed,
&data->convert2_buf, &data->convert2_buf_size);
if (!convert2_dest)
return v4lconvert_oom_error(data);

convert2_dest_size = temp_needed;
rotate90_src = flip_src = crop_src = convert2_dest;
}

if (rotate90 && (hflip || vflip || crop)) {
rotate90_dest = v4lconvert_alloc_buffer(temp_needed,
&data->rotate90_buf, &data->rotate90_buf_size);
if (!rotate90_dest)
return v4lconvert_oom_error(data);

flip_src = crop_src = rotate90_dest;
}

if ((vflip || hflip) && crop) {
flip_dest = v4lconvert_alloc_buffer(temp_needed, &data->flip_buf,
&data->flip_buf_size);
if (!flip_dest)
return v4lconvert_oom_error(data);

crop_src = flip_dest;
}

/* Done setting sources / dest and allocating intermediate buffers,
real conversion / processing / … starts here. */
if (convert == 2) {
res = v4lconvert_convert_pixfmt(data, src, src_size,
convert1_dest, convert1_dest_size,
&my_src_fmt,
V4L2_PIX_FMT_RGB24);
if (res)
return res;

src_size = my_src_fmt.fmt.pix.sizeimage;
}

if (processing)
v4lprocessing_processing(data->processing, convert2_src, &my_src_fmt);

if (convert) {
res = v4lconvert_convert_pixfmt(data, convert2_src, src_size,
convert2_dest, convert2_dest_size,
&my_src_fmt,
my_dest_fmt.fmt.pix.pixelformat);
if (res)
return res;

src_size = my_src_fmt.fmt.pix.sizeimage;

/* We call processing here again in case the source format was not
rgb, but the dest is. v4lprocessing checks it self it only actually
does the processing once per frame. */
if (processing)
v4lprocessing_processing(data->processing, convert2_dest, &my_src_fmt);
}

if (rotate90)
v4lconvert_rotate90(rotate90_src, rotate90_dest, &my_src_fmt);

if (hflip || vflip)
v4lconvert_flip(flip_src, flip_dest, &my_src_fmt, hflip, vflip);

if (crop) {
v4lconvert_crop(crop_src, dest, &my_src_fmt, &my_dest_fmt);}

printf(“end of v4lconvert_convert\n”);

return dest_needed;
}

I was able to change the dest data and the src data.

I did not see printf(“end of v4lconvert_convert\n”); so I assumed something was happening before that. I saw that there was something that didnt’ require any processing. Here is what the screenshot looks like:

(ignore the movie in the background)

After I was able to change the src I saw the following image:

(ignore the movie)

I have gotten step closer. Next to dig more into

That’s all for today, will keep updating with more: