Audio/Video out of sync in rendered video, but not in VSE

Question

First let me apologize to ask this question, as this has been basically already asked, for instance here, here and here. However these answers don't seem to provide an understanding what's really going on, so I will pose my question anyway.

Every once in a while the video in a rendered video file will playback much faster than the audio, despite audio and video being perfectly in sync in the VSE. This problem seems always to pop up when then input source video is "mostly static", i.e. when the change from one frame to the next is miniscule (e.g. a screen recording). I should also add that I record the videos in question using a lossless codec (FRAPS or lagarith).

From the other questions linked above, I get the impression that this is due to Blender (or rather its version libavcodec) incorrectly decoding the compressed source video. It is also mentioned in one answer to the above questions that this is related to the use of b-frames in the encoding algorithm. However I had this problem also using the lagarith codec, which apparently uses only keyframes/i-frames.

Now in order to turn this cry for help into a question, let me ask: What the hell is going on here?

More precisely:

1. Why can Blender (or its version of libavcodec) apparently correctly decode and show the video in the VSE, but not render it correctly? Is the decoding in the VSE done differently than in rendering?
2. What version of libavcodec is used in Blender 2.74? Is the implementation of e.g. lagarith faulty in this version? If not, then why does Blender mess up the rendering?
3. It has been suggested that one should use "Blender-friendly" codecs for compression. Which lossless codecs are known to be so?

Edit: It seems there are several error sources here, which just makes everything much more confusing. The problem of the video playing back at twice the normal speed I had with this one particular lagarith encoded source seemed to stem from a corrupted (??) .blend file. Namely during editing Blender got stuck and I had to force it quit. When I re-opened this file Blender didn't give me any complains, but comparing its file size with the sizes of some other .blend files which were similar, I noticed that it seemed about 3 kB smaller than the rest. Opening this file and saving it to a new file ("Save as") got rid of this playback issue. However the video is still out of sync, even more subtly. Instead of the video globally playing back faster, it sometimes seems to "get stuck" and then "catching up again".

These issues are also already visible in the VSE, so it seems that my problem at least in this instance is solved. On the other hand this makes the behaviour of Blender seem even more mysterious, as I do not understand how a corrupted (was it even corrupted??) .blend file can lead to such an effect.

As I wrote, I experienced this issue with several videos, one of them being a video encoded in FRAPS. There the audio and video seemed in sync in the VSE, but wasn't in the rendered video. Strangely, after re-creating this .blend file from scratch, the audio and video now was out of sync even in the VSE. I tried transcoding it with ffmpeg, and this also gave an out of sync video. However transcoding it with Windows Movie Maker gave perfect result. So I suppose that for this particular video, the problem is that the fraps codec is implemented imperfectly in ffmpeg/libavcodec.

2nd Edit: The problem is back. Apparently it didn't have anything to do with the file being 'corrupted'. Running blender again on the very same file this time produced again an out-of-sync video. This issue seems to appear and disappear randomly.

3rd Edit: Still no clue why it only appears sometimes. But I noticed something else. The audio in the rendered video is as it should be, i.e. its start and end are as defined in the .blend file. However the last frame in the rendered video is actually way beyond the end frame. For some reason Blender consumes more video frames than it is supposed to be. I am going to file a bug report on this.

Answer 1

I was asked to take a look at this post. Codecs can be rather puzzling to diagnose from a distance, but it does seem that some information can be offered.

This problem seems always to pop up when then input source video is "mostly static", i.e. when the change from one frame to the next is miniscule (e.g. a screen recording). I should also add that I record the videos in question using a lossless codec (FRAPS or lagarith).

The “mostly static” comment here is interesting. This does indeed sound like an I versus P / B frame issue, as the “mostly static” frames would be subject to many more instances of P / B frames. While I am unfamiliar with Lagarith or FRAPS, I would be extremely surprised if a non-GOP codec had this issue, as your solution hints toward.

Why can Blender (or its version of libavcodec) apparently correctly decode and show the video in the VSE, but not render it correctly? Is the decoding in the VSE done differently than in rendering?

The VSE is using certain techniques to display the video that, when decoded on a shot by shot basis for rendering are different. This is why non-GOP codecs are more robust, or time code indices will yield more consistent results. This is also why most always-online NLEs will transcode upon ingestion; to avoid the codec problems as well as try to reduce the always-online image degradation issues.

What version of libavcodec is used in Blender 2.74? Is the implementation of e.g. lagarith faulty in this version? If not, then why does Blender mess up the rendering?

FFMPEG's version was a bit nasty on some platforms as it previously used a bundled version on some while on others it used a local version. If you check the ffcompat code, you can see many version checks to support generic versions of FFMPEG.

It has been suggested that one should use "Blender-friendly" codecs for compression. Which lossless codecs are known to be so?

The easiest way to avoid all of the codec mess is to rely on still frame intermediates. If you must for whatever reason rely on codecs, you are likely going to be constantly faced with these sorts of issues unless you manage to coax one of the codecs to behave under generated timecode indices. I would also speculate that FFv1 should be a reasonable choice to test and put under duress, given the nature of its design. If you try FFv1, it would be interesting if you reported your results back here for future reference.

Best of luck.

Answer 2

Unfortunately I can't really answer my questions 1, 2, 3, but at least I figured out how to get rid of said problem. It seems that Blender cannot be trusted to render a video correctly, if said video is not endowed with a timecode, even though most of the times Blender seems to do well without one.

TL;DR: Generate a "timecode" for each video strip before rendering, by selecting the strip, checking the "Proxy/Timecode" checkbox in the "Edit Strip" menu, choosing a timecode option e.g. "Record Run" and finally running "Rebuild Proxy and Timecode Indices".

Update 2017-04-17: Unfortunately, my solution isn't universal. I just had this problem re-appear today, despite generating timecode as proposed above. Not only that, but in fact the a/v-sync issue disappeared when I disabled the use of timecodes! There seems to be something seriously wrong with Blender.

Update 2017-04-21 This was with Blender 2.76 (the last XP-supporting version). I tried reproducing this issue today with Blender 2.78c, but there it was fine. So perhaps, this issue only affects older versions of Blender.

Update 2017-05-24 The problem reoccured today: I can confirm that this issue still exists in Blender 2.78c.

Answer 3

Well, I once posted a question identical to this, which, funny enough wasn't in your list of duplicates. :D I found after the whole that they actually weren't out of sync, but rather the software I used (Windows Media Player) was being laggy and played it back funny, while other players (VLC media player & YouTube) would do it fine! This is a possibility of you see this occuring after your video is rendered.

If this happens to you before rendering and only in Blender's live playback, most likely you are dropping frames. To fix this, in the toolbar if the timeline window there is a tab called "playback". Open it up and click "AV-SYNCING". Only click it if the checkbox next to the label isn't already checked.

Answer 4

**edit: The problem went away on its own for some reason. I converted alot of my project to 16 bit png as I added more edits. So also believe this is the solution for fixing the problem, that the video originally used (an h.264 mkv) was to blame. And others have replied stating the same here and elsewhere. Animation cache doesn't seem to to fix audio issues like it used to, potentially improves quality slightly though. Honestly the quality is bad to begin with using audio/video renders....just use mixdown and recombine with ffmpeg or some other program that won't re-encode (tossing it back in blender re-encodes remember so dont do that).

I don't see any solution here, so here is mine :) I had the same problem. My fix for 2.79b was 1. update the animation cache right before u render under scene->audio ;2. render my video without an audio codec, specifically try it as matroska h.264 ;3. mixdown ur audio as a .wav ;4. combine the audio and video in ffmpeg. Use command: ffmpeg -i videoname.mkv -i audioname.wav -c copy makeupanoutputname.mkv; you guys can go through on ur own and determine what step specifically solves the problem and if other codecs/containers work as well all the same. its probably just NOT rendering with an audio codec that solves the problem.

This is clearly a blender problem and not an ffmpeg problem. I'm very upset that the chosen solution doesn't give people a way to fix anything and is telling us something that the evidence points to being a lie.

VSE: rendered audio is offset (audio+video render)

Answer 5

So after a lot of online searching, I did find an answer to fix this problem, but not in Blender. If you are like me and would like to use Blender for video editing and still get around the issue, then I found a workaround, but you need Shotcut for this. Shotcut is another great free and open-source video editor

1. Export the entire long video from Blender (the rendered video has desync issues as expected).
2. Open the video in Shotcut and detach the audio from it.
3. Use the audio properties to make very fine adjustments to the audio playback speed to suit your requirements (make fine adjustments until video and audio are in sync).

Follow the GIF attached. (I am using a shorter video in the GIF but you get the idea)

Explanation

• Blender has issues while rendering long videos and I noticed that the video is exported at 1.0x speed but the audio is sometimes faster (1.00400x or something like that) and hence the rendered video has audio not in sync with the video.
• Another bad thing is that Blender does not really allow very fine playback speed adjustment just to the audio.
• One trick is to adjust the pitch of the audio in Blender which in turn changes the playback speed but this is only allowed up to 2 decimal places (not what we want for long videos) and it makes the audio sound funny (since it actually changes the pitch).
• Shotcut is a great tool that allows fine playback adjustment, and it also has a pitch compensation feature so that your pitch is kind of unaffected (since we don't want the characters to be sounding funny in our edited video).
• Shotcut allows playback speed adjustment up to 6 decimal places.

I had posted a similar question earlier, link to that post can be found here.

I have also created a short video about this, which can be found here.