![]() ![]() This function just moves through the data looking for where there is no red or blue and where green is larger than some floor value gFloor. If you are used to dealing with multi-dimensional image arrays, the canvas getImageData is weird in that it is just a long string of raw bits without any object structure. The simplest way I found to do that is just to draw the image to a canvas and run through it pixel-by-pixel and add the transparency anywhere there is green. WebRTC won’t send transparency (confirmation and thread on discuss-webrtc), but we add that back in. In any case, it’s green – now we just need to make it transparent. I am not sure why the values are different after or if shading the background green was by design (note: this looks a bit like issue 762443). I tried this the next day and found the green background was a different color – RGBA( 74, 255, 20, 254): Sending that transparent-background image of a PeerConnection – green color 2 A, standing for alpha where 0 is fully transparent and 255 is fully opaque. Notice it isn’t transparent? What happened? Well, the WebRTC encoder doesn’t process the alpha channel. ![]() ![]() You’ll see it looks like this: Sending that transparent-background image of a PeerConnection – green color 1 I just take the incoming stream and add it to an element. See my code for the mechanics of how I set up and send this over a peer connection. Now let’s send that over an RTCPeerConnection Sending transparency over a WebRTC Peer Connection (kind of works?) The segmentation could use some improvement in that particular frame, but that wasn’t too much work. That should give you an image that looks like this: MediaPipe segmentation after adding compositing the user’s video MediaPipe includes a Selfie Segmentation library, so I used that. Fippo showed MediaPipe’s use in Making Zoom’s Smart Gallery on the Web with MediaPipe and BreakoutBox. MediaPipe builds on TensorFlow and is faster. That example uses individual body segmentation but you can also use it to segment the user from the background and it even has a built-in Bokeh effect option (i.e. I showed how to use the tensorflow.js BodyPix library at the beginning of the pandemic in Stop touching your face using a browser and TensorFlow.js. Fortunately, the ML mechanisms are relatively easy. To do this, you can either ask your users to set up a physical green screen or use a Machine Learning library to do this. One of the first things you need to do is segment the user from the background. (Speaking of Kranky Geek, make sure you sign up for our next live stream broadcast on November 18 here). In my examples below I intentionally made my environment a bit difficult – I turned off my forward-facing lighting, used my usual high-back chair, and wore my green Kranky Geek t-shirt to confuse any poorly tuned green screen effects. Play with the playground yourself at playground.html (note there is a streamlined example covered further down too). I started out with some proof of concept code and ended up expanding that into a more comprehensive playground as I experimented with different things. ![]() I walk through methods and findings below or jump here to see the demos.ĭemonstration of the webrtcHacks virtual background transparency playground on a classic video call Proof of Concept WebRTC does make this as easy as it could be, but fortunately new APIs like WebRTC Insertable Streams / Breakout box help to make this a lot more performant. I wanted to not only show the user’s self-view as transparent but also have the user’s background removed for anyone they send their image to over a WebRTC peer connection. I decided to figure out how to do this in the browser. Virtual green screens or newscaster mode are other names. There doesn’t seem to be a universal name for this feature. Rather than inserting a new background behind user(s), transparency removes the background altogether, allowing the app to place users over a screen share or together in a shared environment. Virtual backgrounds have been around for a while. There is a cool new feature everyone has been trying to implement – background transparency. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |