Adding cursor and vectors

[aukondk]

Some notes on what I have tried so far. I am not familiar with gstreamer and have only dabbled with python before so I am likely barking up the wrong tree on these things.

I have managed to parse the cursor.xml and make another layer which displays a png of a red dot in the right place at the right time. It takes a long time to process and something in the default compression settings causes a lot of artifacts. Also forget about importing it into Pitivi.

For vectors I thought I would look at using cairo and rsvg to create pngs of each element which are then added to a new layer. Hit a bump there as rsvg doesn't seem to accept an svg string, only a file or "raw data".

[jhenstridge]

If you've got red dot cursor support going, please make a pull request. Even if it doesn't quite work right, it could be a useful starting point.

For the whiteboard scribbles, I think the right option would be to dissect shapes.svg into one SVG for each point in time where the scene changes, each only showing the elements that are visible during that stretch of time. These separate image files could then be arranged sequentially in a layer with the appropriate time codes. If GES accepts SVG files as a source, then that might be all that's needed. If it needs bitmap images, then we probably will need to use rsvg as you mention.

One thing I haven't checked is whether any coordinate transformation will be needed when switching between slides and screen share, which are at different resolutions and could represent different aspect ratios. I guess it will be pretty obvious if there are problems surrounding this.

[aukondk]

I just realized that my code in #7 had the layers in the wrong order. Fixed now.

I've attempted to change the h264 quality profile to high-10 but it hasn't changed the issue with the cursor.

It repeats and glitches when moving fast.
The xges file is massive when created and the final mp4 takes a lot more than the total video time to process.

Some good progress for vectors in PR #8; lines, paths and shapes are working but not text.

[jhenstridge]

I've merged #7 with some clean-ups. Namely:

• Scale cursor positions based on the dimensions of the current slide (which is my reading of what BBB's JavaScript is trying to do).
• Position the cursor clip so that it is centred on the cursor position, rather than the top left corner.

I didn't notice any ghosting with the default codec settings in my small amount of testing. I still need to have a read over the annotations PR.

[jhenstridge]

@aukondk: I've built on the work done in #8 here:

https://github.com/plugorgau/bbb-render/tree/annotation-support

This version is using a different SVG renderer, which should handle text annotations. If you've got time, I'd appreciate if you'd test it against your recording to see if it improves matters.

[aukondk]

I'm getting an error.

File "./make-xges.py", line 173, in add_webcams self._add_clip(layer, asset, 0, 0, asset.props.duration, File "./make-xges.py", line 106, in _add_clip clip = layer.add_asset_full(asset, start, inpoint, duration, AttributeError: 'Layer' object has no attribute 'add_asset_full'

I'm guessing it's a version problem. I'm running this on my Ubuntu 20.04 box with the main repos. Gstreamer is 1.16 here but add_asset_full is new to 1.18. I'll see what I can do.

[jhenstridge]

I guess it is new API. It should be safe to change it back to an add_asset() call -- the only difference between the two is that add_asset_full tells you what went wrong when it fails rather than just returning None. I'll make sure to change it back before merging.

As mentioned in the PR, this probably won't help with the missing text annotations. The shapes.svg file that drives the recording relies on embedded HTML for text annotations, which is not supported by either of the SVG renderers we tried. I've got some ideas that should handle simple text annotations, but will misrender any that rely on line wrapping. I think it can probably wait for a second PR though.

[aukondk]

I tried rendering the test recording @symptog posted in #8
https://webroom.hrz.tu-chemnitz.de/playback/presentation/2.0/playback.html?meetingId=ef153938d8b15587eff96c475d2c355c6161a4b8-1616591890688

annotest.mp4

As you can see, it has a lot more artifacts compared to your render so I guess this might be to do with my version of Gstreamer or some other library.

annotations-test2.mp4

[aukondk]

I'm currently rendering some of my big recordings to see how they do.

[jhenstridge]

If it makes a difference, the video I rendered was produced on an Ubuntu 21.04 system, so slightly newer software. One other potential difference is that GStreamer will try to use a hardware codec if available (e.g. via VA-API or VDPAU). The system I was running on would have been using the software encoder though.

I've merged #8 to master now, and made the red dot cursor support conditional on --annotations. I figure if you want the cursor you probably also want the scribbles drawn with the cursor.

[jhenstridge]

There is a first go at text support in #9. It will almost certainly misrender annotations that depend on line wrapping though.