It all started when a customer asked me to create a tool where he can draw lines, circles, arrow, squares and whatever he wants over videos that have been uploaded by the player on his already existing web application. He showed me a mobile application doing this already, and it is shaaaarrrp. Trying to do the same on a small budget is kind of a challenge, but who risk nothing has nothing.

And knowledge is worth more than money. Also, using the power of the web to build great portable things is what I look for.

I've tried to build something from existing solutions over the web and I came to an interesting proof of concept in less than an hour.

Not a huge challenge you would say? The real challenge is not there, but is to end up with a product that has all functionalities that the user wants with a great user experience. And I am far from there.

What would someone want with that kind of Sketch-over-video tool?

First, I've used a tool called Literally (http://literallycanvas.com). It is a small tool built with ReactJS that creates a Canvas with a couple of drawing tools, something like paint but smaller. The UI is far from what I want, but at least for a small tweak I have a working demo.

Then, I used VideoJS (https://videojs.com), which is a well known product, to be able to easily create a control bar. For the moment, it is only for the pause and play button. We need this because the canvas is over the video player and we cannot interact with the controls.

It would be possible to just subtract dynamically a couple pixel from the height of the canvas and still be able to use the buttons and the seekbar, but there is 2 problems with this:

• the html5 video default skin has controls all over the video and when you pause it, the play button is in the middle of the video, which is not so good for the video analysis.
• the play button from the custom VideoJS skin is on the top left corner of the player, we cannot click it with the canvas on.

So, we still need to play the video from outside the player. My solution is to create a custom control bar, but I still need to see if I can move the video smoothly with a custom seekbar, as I need this when analysing the video.

Next, I would want to be able to save the sketching for later consultation and usage. It would be nice to be able to load multiple sketches over a video and toggle them when we want or even on a timeline that fits the video. It will be easy to save a canvas state, but I am not sure yet if I can easily reload it in the Literally plugin or any other sketching tool, existing or custom made. It currently keep the drawing history and I did not dig into it for now.

Last thing, the Literally sketch pad skin do not fit what I see of a video analysis tool, so if I continue with this javascript library, I will need for sure to create a new skin. More convenient with the toolbar outside of the video or fitting the video inside. It may be complex to have something that fits all screens well. We have to consider that this tool will be used inside a web application that can already compare 2 videos side to side. The UI can becoming overwhelming.

Here is the small demo: https://mentortechies.github.io/video-sketch-html5/

Here is the source code: https://github.com/mentortechies/video-sketch-html5

Have you ever built or experienced this kind of thing?

P.S. Bootstrap 3 have been used as it is the current CSS Framework used by the project.