When should I add a custom field?
The question to ask yourself is, "Do I need to search on this data or do metrics on it?" If you don't, you should not be adding a custom field. Instead, you should just be creating a field that puts data into the first comment. (See the section on "bug/create/create.html.tmpl and bug/create/comment.txt.tmpl".)
When should I use the template custom/ directory, and when should I modify files in the default/ directory?
Actually, the ideal thing to do is to use template hooks. Hooks aren't very likely to change between versions of Bugzilla, so you can just write your customizations once and it won't be very much work to bring them forward into new versions of Bugzilla. You can find hooks by searching for "Hook.process" in all the templates.
Sometimes you can't use hooks, though, or it would be inconvenient. In that case, here's when you should use custom/ and when you should use default/:
custom/ is good for entirely new files--ones that aren't actually a normal part of Bugzilla. Putting them in custom/ makes it easy to see that you added those files, and that they're not normally a part of Bugzilla.
custom/ is also good for a few files like bug/create/user-message.html.tmpl that are designed to be entirely replaced. (Say, they just contain some text for you to override.)
Otherwise, it's actually best to just modify the already-existing files in default/. That allows you to just do a "cvs diff" and see what you've changed, which makes it much easier to bring your customizations forward into new versions.