Link to sketch: LINK
I’ve always been captivated by remnants of broken web, finding glitches that create beautiful and seemingly temporal effects. For this project, I wanted to experiment with datamoshing, the process of changing the data within media files to create distorted visual or auditory effects. Currently, many people achieve this effect by using softwares that run scripts within the media files’ data, but is it possible to make something glitch-like using what I’ve learned through code in p5?
I started with a createCapture(); and created a Region class, with region objects generated randomly. The region would capture a small area of pixels and hold it across multiple frames (and slowly move towards the lower right). The movement of the region was slightly random (it can either increase, decrease or stay the same in x or y dimensions). I added a posterize filter so that the region’s image could degrade over time.
Next, I thought the image felt too clear and wanted to play around with the capture on a pixel level, so I created a for loop that would pick a random pixel in the pixel array, and draw it on top of the previous image.
Next, I added an if statement that measures the brightness of each random pixel, and makes it so that the square is only drawn if the brightness goes past a certain threshold. I liked how fragmented this made the capture look. It created floating windows and spaces, feeling dream-like and surreal.
I’d love to do more research into mimicking datamoshing at one point though. According to Wikipedia (copying and pasting so I can reread this later a couple times),
“In the field of video compression a video frame is compressed using different algorithms with different advantages and disadvantages, centered mainly around amount of data compression. These different algorithms for video frames are called picture types or frame types. The three major picture types used in the different video algorithms are I, P and B.[1] They are different in the following characteristics:
I‑frames are the least compressible but don't require other video frames to decode.
P‑frames can use data from previous frames to decompress and are more compressible than I‑frames.
B‑frames can use both previous and forward frames for data reference to get the highest amount of data compression.”
According to this website datamoshing.com,
“Modern compressed video files have very complex methods of reducing the amount of storage or bandwidth needed to display the video. To do this most formats don’t store the entire image for each frame. Frames which store an entire picture are called I-frames (Intra-coded), and can be displayed without any additional information.
Frames which don’t contain the entire picture require information from other frames in order to be displayed, either previous or subsequent frames, these frames are called P-frames (Predicted) and B-frames (Bi-predictive). Instead of storing full pictures these P-frames and B-frames contain data describing only the differences in the picture from the preceding frame, and/or from the next frame, this data is much smaller compared to storing the entire picture — especially in videos where there isn’t much movement.
If an I-frame is corrupted, removed or replaced the data contained in the following P-frames is applied to the wrong picture. In the above video I-frames have been removed and so instead of scenes changing properly you see the motion from a new scene applied to a picture from a previous frame. This process of corrupting, removing or replacing I-frames is a very popular video datamoshing technique and what this tutorial will focus on.
Another video datamoshing technique involves selecting one or more P-frames and duplicating them multiple times consecutively. This results in the same P-frame data being applied to one picture over and over again, accentuating the movement and creating what’s known as a Bloom effect.”
Maybe next time, I can follow this tutorial to try pixel sorting, using this tutorial as a template: http://datamoshing.com/2016/06/16/how-to-glitch-images-using-pixel-sorting/.