Most typically a dialog tree is used when the player talks to a non-playable character in the game: Typically a list of sentences will be shown for the player to choose, and the NPC will then respond to it. Especially if there may then be further options based on what you chose (and sometimes based on what the NPC responded), it makes it an actual "tree" (because the conversation path can branch to different directions depending on which option you chose.)
I don't know if and what these may be named, but I would divide such dialog trees broadly into three categories: Trivial exclusive trees, subtler exclusive trees, and fully inclusive trees.
An "exclusive" dialog tree is one where you have to make a choice and it will be "locked in". You can't go back and change it. The other branches will usually be "lost forever" in practice. Typically such choice will have an effect on further events.
A "trivial" exclusive tree is one that's almost stereotypically two-sided, of the type:
NPC: "What should we do with the children in that burning house?"
- "Let's go save them!"
- "Let them burn! HAHAHAA!"
A much subtler exclusive tree is one where there are more options, most or all of them being morally ambiguous (or overall ambiguous and unclear in how they will affect subsequent events.)
A "fully inclusive tree" is one that has little to no consequence to the subsequent events, and which you are allowed to traverse through. Typically no branch of the tree will ever be "lost forever". (Often these dialog trees are very shallow, consisting mostly or even solely of "sentence choice" - "NPC response" pairs, with no further branches from there, with few exceptions.)
The inclusive trees often feel rather pointless. Most often they are even clearly designed to be traversed in the listed order. You could traverse all the options in some other order but it's usually pointless and, moreover, in some cases it may even result in an unnatural-sounding conversation (because the conversation does not follow a logical progression). They don't seem to play any kind of important gameplay role at all.
Exclusive trees, especially the subtler ones, are supposed to make the game experience more profound, with conversation choices having non-trivial consequences that may be hard to predict (just like in real life.) Doing this naturally is hard to pull off and only few games have any kind of success in this. However, the major problem with them is that they often leave the player wondering what would have happened if they had chosen something else. The "lost forever" nature of this kind of dialog tree can in itself be annoying because the player cannot experience everything that the game has to offer, as the game locks some paths out. (This is supposed to give the game replayability value, but not all gamers want to play the same game through more than once.)
One has to wonder why games still have such dialog trees. Some feel rather pointless, others lock the player out of branching events. Only few games succeed in implementing it in such a manner that it avoids both problems.