They are both "container" files extentions. By container, I mean that they contain and allow for playback of audio (multiple tracks in some cases), video, and subtitles (and future developement might see other interesting interactivity such as Flash or web interaction). These are NOT compression codecs.
.AVI - Audio/Video Interleave
.mkv - Matroska
AVI is older and the most accepted container, while MKV is newer tech, but might not play right away with every video player on the market. Quality between the two is arguable, as the compression method is really the determining factor.