Have you tried using seperate audio and video sources? Some avi's need to go through another process.

I use dvd2avi to convert the audio stream to a wave or mp file.

Use the newly created audio file and original video file in TMPGEnc and see what happens.