Video formats are a bit of a mess for the lay person.
AVI, WMV, ASF, MKV, and (confusingly) MP4 are containers. They let you combine a video file, audio file, chapter info, subtitles, etc. into a single "video file" for playback. If you double-click a file with these extensions and get video and audio, then it's just a container file. The video inside it could be encoded in any variety of formats.
MPEG1 (not used anymore), MPEG2 (used to encode DVDs), h.264 (aka MPEG4, often confusingly abbreviated MP4), h.265 (aka HEVC), VP9, AV1, are video formats (codecs). Methods for compressing video (and only video).
The trick is to get the video encoded in a video codec and container format the car recognizes. MP4 (MPEG4 container with h.264 video) should work since it's on the list. But since MP4 is just a container, it can actually contain video encoded with a different codec. If someone manually did the encode, it may not be a h.264 MP4 file.
I haven't played around with it on my Treg, but more than likely the limitation is the 720x576 resolution limit. If you have videos encoded at a larger resolution, you'll need to re-encode it at the lower resolution (720x576 for 4:3 movies, 720x404 or 720x400 for 16:9 movies). Handbrake can indeed be used to do this. Make sure on the video tab, you click the Fast Decode option, as that generates an output file with maximum compatibility (at the cost of slightly larger file size). Handbrake should default to encoding with h.264 (MPEG4), but you'll probably want to double-check it to make sure it's not using a newer codec like h.265 that the car doesn't support. The other suggestions RocketTech made are all good too.
One place I would not recommend Handbrake for is converting DVDs. DVDs use interlaced video, and Handbrake has a terrible deinterlacer/decomb filter which will turn diagonal lines jagged. It's fine if you don't care about these things, but it annoys me to no end. The program I use (StaxRip) is way too complicated for the lay person. Try videohelp.com for suggestions on an all-in-one DVD conversion program and guides.
Video forums, video software downloads, guides, blu-ray players and media.
www.videohelp.com
Adding chapters requires a chapter file. It's just a text file with the chapter name, and the time or frame index. Unfortunately, Handbrake doesn't support creating them. If the original video had chapters, Handbrake will import them and will let you rename then. But it doesn't have any way for you to create new chapter markers. Try some of the other chapter editing tools available. (Also note that the container format and playback device needs to support chapters for it to work. MP4 does. The original AVI format doesn't, though it was added later as an extension. I dunno if the RNS850 supports chapters.)
Download free video and audio software. Old versions, user reviews, version history, screenshots.
www.videohelp.com
AAC is preferable for audio encoding (it was supposed to be the upgraded replacement for MP3). In addition to not being as good, AC3 suffers from licensing issues - if your player is not licensed for AC3 (hasn't paid a fee to the patent holders), the video will play but the AC3 audio will not.
I don't see AAC on the supported audio format list though. It may work though. Some of the music on my SD card in FLC format worked (others came out distorted, I assume because of too high a bitrate).
Usually multiple audio streams are included for different languages, or one is stereo (2-channel left/right) and the other is 5.1 ( center front-left/front-right/rear-left/rear-right and subwoofer). I have no idea if the RNS850 supports 5.1, although if it does it would definitely be preferable over stereo.