Patent
System and method for managing encoded information in a real-time screen-to-camera communication environment
| العنوان: | System and method for managing encoded information in a real-time screen-to-camera communication environment |
|---|---|
| Patent Number: | 12114,003 |
| تاريخ النشر: | October 08, 2024 |
| Appl. No: | 18/321829 |
| Application Filed: | May 23, 2023 |
| مستخلص: | A system and method for managing encoded information in a real-time screen-to-camera communication environment are disclosed. The system converts information into a pre-defined number of characters and generates data symbols in shapes and pilot symbols corresponding to the characters. Further, the system embeds the data symbols in media content frames and modulates pixels and boundaries for display of display device, based on luminance, and adaptively displays frames as temporal-complementary frames. Furthermore, the system detects frames from recorded content, extracts data symbols based on grid and fixed pattern, and detects bit values by analyzing color differences. Additionally, the system generates information based on the detected bit values and outputs the information on an user device display, including products, recommendations, services, and relevant information related to the media content. |
| Inventors: | Kudekar, Shrinivas (Roswell, GA, US) |
| Assignees: | Kudekar, Shrinivas (Roswell, GA, US) |
| Claim: | 1. A computer-implemented system for managing encoded information in a real-time screen-to-camera communication environment, the computer-implemented system comprising: one or more hardware processors; and a memory coupled to the one or more hardware processors, wherein the memory comprises a plurality of modules in form of programmable instructions executable by the one or more hardware processors, wherein the plurality of modules comprises: an information encoding module, wherein the information encoding module comprises: a character converting submodule configured to convert information into a pre-defined number of characters; a symbol generating submodule configured to generate at least one of one or more data symbols, one or more pilot symbols, and one or more scannable barcodes corresponding to the pre-defined number of characters, wherein the one or more data symbols are generated in one or more shapes; a symbol embedding submodule configured to embed, for a pre-defined duration of media content, the generated at least one of the one or more data symbols, the one or more pilot symbols, and the one or more scannable barcodes in each of one or more frames of the media content, wherein the one or more data symbols are embedded as at least one of a grid and a fixed pattern on each of the one or more frames, and the one or more pilot symbols are embedded as a column of at least one of discrete pilot symbols and a single continuous column of a color on each of the one or more frames; a pixel modulating submodule configured to adaptively modulate, over at least one of a color channel and a luminance channel, a lightness channel, an intensity channel, one or more pixels and a boundary in each of the one or more frames for the pre-defined duration, for the embedded one or more data symbols and the one or more pilot symbols; and a frame displaying submodule configured to adaptively display, each pixel, on a screen associated with a display device, at least one of the scannable barcodes and the modulated one or more pixels, and the boundary in each of the one or more frames for the pre-defined duration, based on a luminance of each of one or more regions in each of one or more frames, wherein the one or more frames are adaptively displayed as one or more temporal-complementary frames (TCF); and an information decoding module, wherein the information decoding module comprises: a content receiving submodule configured to receive recorded content corresponding to the media content, recorded through a camera associated with a user device; a frame detecting submodule configured to detect the one or more frames from the recorded content, wherein the detected one or more frames are converted into a color space; a symbol detecting submodule configured to detect at least one of the one or more data symbols, the one or more pilot symbols, and the one or more scannable barcodes in the detected the one or more frames; a symbol extracting submodule configured to extract the one or more data symbols from the one or more frames based on at least one of the grid and the fixed pattern, wherein the one or more data symbols are extracted based on a position of the column of the one or more pilot symbols in each of the one or more frames; a bit value detecting submodule configured to detect one or more bit values of each data symbol, by analyzing a color difference between each of the one or more frames and a shape of the one or more data symbols, wherein the color difference of the one or more data symbols and the one or more pilot symbols is cumulatively accumulated over the one or more frames for an energy accumulation to analyze the color difference between each of the one or more frames and the one or more shapes of the one or more data symbols; an information generating submodule configured to generate the information based on at least one of the detected bit values of the one or more frames and the detected one or more scannable barcodes; and an information outputting submodule configured to output the generated information through at least one of displaying on a display of the user device and transmitting the information to one or more digital delivery destination platforms, wherein outputting the generated information comprises displaying at least one of one or more products, one or more recommendations, one or more services, and relevant information regarding the media content to a user of the user device for interacting with the information in a unobtrusive mode. |
| Claim: | 2. The computer-implemented system of claim 1 , wherein for adaptively displaying, each pixel, on the screen associated with the display device, for the modulated one or more pixels, and the boundary in each of the one or more frames, the information encoding module further comprises: a luminance determining submodule configured to determine a luminance value of each of the adaptively modulated one or more pixels; a luminance modulating submodule configured to modulate the luminance value, over the luminance channel, of the one or more pixels, when the determined luminance value of each of the adaptively modulated one or more pixels is lesser than a first pre-defined threshold; the luminance modulating submodule configured to modulate the luminance value, over the color space, of the one or more pixels, when the determined luminance value of each of the adaptively modulated one or more pixels is lesser than a second pre-defined threshold; and a pixel flickering submodule configured to flickering a region of pixels to transmit the one or more data symbols, wherein the one or more data symbols encodes a plurality of bits of the information, based on flickering the region of pixels; wherein the region of pixels encoding the one or more data symbols is an arbitrary two-dimensional shape belonging to a set of at least one of orthogonal shapes or non-orthogonal shapes. |
| Claim: | 3. The computer-implemented system of claim 2 , wherein to flicker the region of pixels, the information encoding module further comprises: the pixel flickering submodule configured to: perform at least one of an intensity flickering and a color flickering the region of pixels based on determining if each region of the region of pixels comprises an intensity above a pre-defined intensity threshold; determine the shape of the region of pixels for flickering at least one of a boundary and a rim of the region of pixels; modify one or more pixel values the region of pixels based on different values in one or more consecutive frames using an symmetrical modulation technique; flicker the boundary and an interior of the region of pixels to transmit an additional bit of information, wherein the additional bit of information corresponds to a direction of flickering of the region of pixels, wherein the one or more pilot symbols comprises a fixed direction of flickering, wherein the direction of flickering of the one or more pilot symbols is determined using the additional bit of information; wherein the one or more pilot symbols are placed in fixed position known to the information decoding module, wherein at least one of the one or more data symbols and the one or more pilot symbols are packed in a fixed pattern to implement a plurality of bits of information. |
| Claim: | 4. The computer-implemented system of claim 1 , wherein to convert the one or more frames into a color space, the frame detecting submodule is further configured to: convert from a standard color space to the color space; cumulatively accumulate over the one or more frames for the energy accumulation; correct one or more perspective distortions using the detected frame boundary; detect the one or more shapes of each of the one or more data symbols, using contour detection techniques; and determine one or more coded bits based on the detected one or more shapes and decode the one or more coded bits using a forward-error-correcting-code technique. |
| Claim: | 5. The computer-implemented system of claim 1 , wherein the symbol extracting submodule is further configured to: detect lines in the one or more frames using a line detection technique to extract one or more quadrilaterals corresponding to a region of interest in each of the one or more frames; detect one or more edges and one or more contours in the detected lines to determine the one or more shapes of the one or more data symbols; determine a degree of angle of tilt of the one or more contours; analyze iteratively one or more errors in determining the one or more data symbols; and output correctly determined one or more data symbols. |
| Claim: | 6. The computer-implemented system of claim 1 , wherein the symbol embedding submodule is further configured to perform at least one of an up-sample and an interpolate the media content with a frame rate lower than a pre-defined frame rate to a higher frame rate of the pre-defined frame rate using at least one of an optical flow based techniques, a sample-and-hold interpolation technique, a linear interpolation technique, an optical flow interpolation technique, and a deep neural network based interpolation technique, prior to embedding the one or more data symbols. |
| Claim: | 7. The computer-implemented system of claim 1 , wherein the frame displaying submodule is further configured to apply opponent colors, for one bit of the information, for each of the one or more data symbols for each of subsequent frames in the one or more frames. |
| Claim: | 8. The computer-implemented system of claim 1 , wherein the frame displaying submodule is further configured to display an indicia on the screen of the display device, upon adaptively displaying at least one of the scannable barcodes and the modulated one or more pixels, and the boundary in each of the one or more frames for the pre-defined duration. |
| Claim: | 9. The computer-implemented system of claim 1 , wherein each of the one or more frames is adaptively modulated by converting a standard color space to at least one of an orthogonal color space and a perceptual color space, wherein the standard color space comprise a red green blue (RGB) color space, and wherein the orthogonal color space comprises an OKlab color space, wherein the luminance channel, the lightness channel, the intensity channel are orthogonal to the color channel, in which the changing value on one channel does not affect the other channel. |
| Claim: | 10. The computer-implemented system of claim 1 , wherein the one or more data symbols are modulated by modifying each pixel value within a shape of the one or more data symbols using at least one of a fixed modulation depth, a pyramidU technique and a rim technique, wherein the pyramidU technique comprises varied modulation depth based on the region of each of the one or more frames, and wherein the rim technique comprises modulating along the boundary of the one or more data symbols. |
| Claim: | 11. The computer-implemented system of claim 1 , wherein the column of the one or more pilot symbols are positioned on the one or more frame in at least one of a left, a right, and a middle of each of the one or more frames based on the content of the frame, with the position fixed for the pre-defined duration of the media content. |
| Claim: | 12. A computer-implemented method for managing encoded information in a real-time screen-to-camera communication environment, the computer-implemented method comprising: converting, by one or more hardware processors, information into a pre-defined number of characters; generating, by the one or more hardware processors, at least one of one or more data symbols and one or more pilot symbols, and scannable barcodes corresponding to the pre-defined number of characters, wherein the one or more data symbols are generated in one or more shapes; embedding, by the one or more hardware processors, for a pre-defined duration of media content, the generated at least one of the one or more data symbols the one or more pilot symbols, and the scannable barcodes in each of one or more frames of the media content, wherein the one or more data symbols are embedded on at least one of a grid and a fixed pattern, on each of the one or more frames, and the one or more pilot symbols are embedded as a column of at least one of discrete pilot symbols and a single continuous column of a color on each of the one or more frames; adaptively modulating, by the one or more hardware processors, over at least one of a color channel and a luminance channel, a lightness channel, an intensity channel, one or more pixels, and a boundary in each of the one or more frames for the pre-defined duration, for the embedded one or more data symbols and the one or more pilot symbols; adaptively displaying, by the one or more hardware processors, each pixel, on a screen associated with a display device, at least one of the scannable barcodes and the modulated one or more pixels, and the boundary in each of the one or more frames for the pre-defined duration, based on a luminance of each of one or more regions in each of one or more frames, wherein the one or more frames are adaptively displayed as one or more temporal-complementary frames (TCF); receiving, by the one or more hardware processors, recorded content corresponding to the media content, recorded through a camera associated with a user device; detecting, by the one or more hardware processors, the one or more frames from the recorded content, wherein the detected one or more frames are converted into a color space; detecting, by the one or more hardware processors, at least one of the one or more data symbols, the one or more pilot symbols, and the one or more scannable barcodes in the detected the one or more frames; extracting, by the one or more hardware processors, the one or more data symbols from the one or more frames based on at least one of the grid and the fixed pattern, wherein the one or more data symbols are extracted based on a position of the column of the one or more pilot symbols in each of the one or more frames; detecting, by the one or more hardware processors, one or more bit values of each data symbol, by analyzing a color difference between each of the one or more frames and a shape of the one or more data symbols, wherein the color difference of the one or more data symbols and the pilot symbols is cumulatively accumulated over the one or more frames for an energy accumulation to analyze the color difference between each of the one or more frames and the one or more shapes of the one or more data symbols; generating, by the one or more hardware processors, the information based on at least one of the detected bit values of the one or more frames and the detected one or more scannable barcodes; and outputting, by the one or more hardware processors, the generated information through at least one of displaying on a display of the user device and transmitting the information to one or more digital delivery destination platforms, wherein outputting the generated information comprises displaying at least one of one or more products, one or more recommendations, one or more services, and relevant information regarding the media content to a user of the user device for interacting with the information in a unobtrusive mode. |
| Claim: | 13. The computer-implemented method of claim 12 , wherein adaptively displaying, each pixel, on the screen associated with the display device, for the modulated one or more pixels, and the boundary in each of the one or more frames, further comprises: determining, by the one or more hardware processors, a luminance value of each of the adaptively modulated one or more pixels; modulating, by the one or more hardware processors, the luminance value, over the luminance channel, of the one or more pixels, when the determined luminance value of each of the adaptively modulated one or more pixels is lesser than a first pre-defined threshold; modulating, by the one or more hardware processors, the luminance value, over the color channel, of the one or more pixels, when the determined luminance value of each of the adaptively modulated one or more pixels is lesser than a second pre-defined threshold; and flickering, by the one or more hardware processors, a region of pixels to transmit the one or more data symbols, wherein the one or more data symbols encodes a plurality of bits of the information, based on flickering the region of pixels; wherein the region of pixels encoding the one or more data symbols is an arbitrary two-dimensional shape belonging to a set of at least one of orthogonal shapes or non-orthogonal shapes. |
| Claim: | 14. The computer-implemented method of claim 13 , wherein flickering the region of pixels, further comprises: performing, by the one or more hardware processors, at least one of an intensity flickering and a color flickering the region of pixels based on determining if each region of the region of pixels comprises an intensity above a pre-defined intensity threshold; determining, by the one or more hardware processors, the shape of the region of pixels for flickering at least one of a boundary and a rim of the region of pixels; modifying, by the one or more hardware processors, one or more pixel values the region of pixels based on different values in one or more consecutive frames using an symmetrical modulation technique; flickering by the one or more hardware processors, the boundary and an interior of the region of pixels to transmit an additional bit of information, wherein the additional bit of information corresponds to a direction of flickering of the region of pixels, wherein the one or more pilot symbols comprises a fixed direction of flickering, wherein the direction of flickering of the one or more pilot symbols is determined using the additional bit of information; wherein the one or more pilot symbols are placed in fixed position known to the information decoding module, wherein at least one of the one or more data symbols and the one or more pilot symbols are packed in a fixed pattern to implement a plurality of bits of information. |
| Claim: | 15. The computer-implemented method of claim 13 further comprising: detecting, by the one or more hardware processors, lines in the one or more frames using a line detection technique to extract one or more quadrilaterals corresponding to a region of interest in each of the one or more frames; detecting, by the one or more hardware processors, one or more edges and one or more contours in the detected lines to determine the one or more shapes of the one or more data symbols; determining, by the one or more hardware processors, a degree of angle of tilt of the one or more contours; analyzing, by the one or more hardware processors, iteratively one or more errors in determining the one or more data symbols; and outputting, by the one or more hardware processors, correctly determined one or more data symbols. |
| Claim: | 16. The computer-implemented method of claim 13 further comprising: performing at least one of a up-sampling and an interpolating, by the one or more hardware processors, the media content with a frame rate lower than a pre-defined frame rate to a higher frame rate of the pre-defined frame rate using at least one of an optical flow based techniques, a sample-and-hold interpolation technique, a linear interpolation technique, an optical flow interpolation technique, and a deep neural network-based interpolation technique, prior to embedding the one or more data symbols. |
| Claim: | 17. The computer-implemented method of claim 13 further comprising: applying, by the one or more hardware processors, opponent colors, for one bit of the information, for each of the one or more data symbols for each of subsequent frames in the one or more frames. |
| Claim: | 18. The computer-implemented method of claim 13 further comprising: displaying, by the one or more hardware processors, an indicia on the screen of the display device, upon adaptively displaying at least one of the scannable barcodes and the modulated one or more pixels, and the boundary in each of the one or more frames for the pre-defined duration. |
| Claim: | 19. The computer-implemented method of claim 13 , wherein each of the one or more frames is adaptively modulated by converting a standard color space to at least one of an orthogonal color space and a perceptual color space, wherein the standard color space comprise a red green blue (RGB) color space, and wherein the orthogonal color space comprises an OKlab color space, wherein the luminance channel, the lightness channel, the intensity channel are orthogonal to the color channel, in which the changing value on one channel does not affect the other channel. |
| Claim: | 20. The computer-implemented method of claim 13 , wherein the one or more data symbols are modulated by modifying each pixel value within a shape of the one or more data symbols using at least one of a fixed modulation depth, a pyramidU technique, and a rim technique, wherein the pyramidU technique comprises varied modulation depth based on the region of each of the one or more frames, and wherein the rim technique comprises modulating along the boundary of the one or more data symbols. |
| Claim: | 21. The computer-implemented method of claim 13 , wherein the column of the one or more pilot symbols are positioned on the one or more frame in at least one of a left, a right, and a middle of each of the one or more frames based on the content of the frame, with the position fixed for the pre-defined duration of the media content. |
| Claim: | 22. The computer-implemented method of claim 12 , wherein converting the one or more frames into a color space, further comprises: converting by the one or more hardware processors, from a standard color space to the color space; cumulatively accumulating, by the one or more hardware processors, over the one or more frames for the energy accumulation; correcting by the one or more hardware processors, one or more perspective distortions using the detected frame boundary; detecting by the one or more hardware processors, the one or more shapes of each of the one or more data symbols, using contour detection techniques; and determining by the one or more hardware processors, one or more coded bits based on the detected one or more shapes and decode the one or more coded bits using a forward-error-correcting-code technique. |
| Claim: | 23. A non-transitory computer-readable storage medium having instructions stored therein that, when executed by one or more hardware processors, cause the one or more hardware processors to: convert information into a pre-defined number of characters; generate at least one of one or more data symbols and one or more pilot symbols, and scannable barcodes corresponding to the pre-defined number of characters, wherein the one or more data symbols are generated in one or more shapes; embed, for a pre-defined duration of media content, the generated at least one of the one or more data symbols the one or more pilot symbols, and the scannable barcodes in each of one or more frames of the media content, wherein the one or more data symbols are embedded as at least one of a grid, and a fixed pattern on each of the one or more frames, and the one or more pilot symbols are embedded as a column of at least one of discrete pilot symbols and a single continuous column of a color on each of the one or more frames; adaptively modulate, over at least one of a color channel and a luminance channel, a lightness channel, an intensity channel, one or more pixels and a boundary in each of the one or more frames for the pre-defined duration, for the embedded one or more data symbols and the one or more pilot symbols; adaptively display, each pixel, on a screen associated with a display device, at least one of the scannable barcodes and the modulated one or more pixels, and the boundary in each of the one or more frames for the pre-defined duration, based on a luminance of each of one or more regions in each of one or more frames, wherein the one or more frames are adaptively displayed as one or more temporal-complementary frames (TCF); receive recorded content corresponding to the media content, recorded through a camera associated with a user device; detect the one or more frames from the recorded content, wherein the detected one or more frames are converted into a color space; detect at least one of the one or more data symbols, the one or more pilot symbols, and the one or more scannable barcodes in the detected the one or more frames; extract the one or more data symbols from the one or more frames based on at least one of the grid, and the fixed pattern, wherein the one or more data symbols are extracted based on a position of the column of the one or more pilot symbols in each of the one or more frames; detect one or more bit values of each data symbol, by analyzing a color difference between each of the one or more frames and a shape of the one or more data symbols, wherein the color difference of the one or more data symbols and the pilot symbols is cumulatively accumulated over the one or more frames for an energy accumulation to analyze the color difference between each of the one or more frames and the one or more shapes of the one or more data symbols; generate the information based on at least one of the detected bit values of the one or more frames and the detected one or more scannable barcodes; and output the generated information through at least one of displaying on a display of the user device and transmitting the information to one or more digital delivery destination platforms, wherein outputting the generated information comprises displaying at least one of one or more products, one or more recommendations, one or more services, and relevant information regarding the media content to a user of the user device for interacting with the information in a unobtrusive mode. |
| Claim: | 24. The non-transitory computer-readable storage medium of claim 23 , wherein for adaptively displaying, each pixel, on the screen associated with the display device, for the modulated one or more pixels, and the boundary in each of the one or more frames, the one or more hardware processors are further configured to: determine a luminance value of each of the adaptively modulated one or more pixels; modulate the luminance value, over the luminance channel, of the one or more pixels, when the determined luminance value of each of the adaptively modulated one or more pixels is lesser than a first pre-defined threshold; modulate the luminance value, over the color channel, of the one or more pixels, when the determined luminance value of each of the adaptively modulated one or more pixels is greater than a second pre-defined threshold; and flicker a region of pixels to transmit the one or more data symbols, wherein the one or more data symbols encodes a plurality of bits of the information, based on flickering the region of pixels; wherein the region of pixels encoding the one or more data symbols is an arbitrary two-dimensional shape belonging to a set of at least one of orthogonal shapes or non-orthogonal shapes. |
| Patent References Cited: | 8229159 July 2012 Tourapis 10785495 September 2020 Enigma 20230109762 April 2023 Suyama |
| Primary Examiner: | Ren, Zhubing |
| Attorney, Agent or Firm: | Cameron, Jason C. |
| رقم الانضمام: | edspgr.12114003 |
| قاعدة البيانات: | USPTO Patent Grants |
| الوصف غير متاح. |