I definitely believe there's an internal struggle inside Sega, and politics 100% being the reason why Shenmue 4 hasn't been greenlit. But I get the sense there's some sort of animosity between YS and some key members at Sega. Somewhat similar to what happened between Kojima and Konami. Main difference being that YS hasn't completely burned all of his bridges at Sega.
But avoiding competition between their own internal franchises is a reason that I don't buy. The fact that Sega resourced RGG to create a Fist of the Northstar game that nobody asked for shows that they were willing to use their engine for games outside of the RGG series. I don't know what the sales are, but I'm confident that Shenmue III probably outsold that FOTN game that came out.
Sega could easily have made it to Shenmue 7 if they would just use the already polished Dragon Engine to make some new Shenmue games instead.
I really loved FOTN Lost Paradise, for fans of the manga and sega/yakuza series, it's a dream come true.
But I think you're aren't that far for sales, I remember that Sega
expected 400k in Japan, can't remember it it reached those numbers at the end counting the west, but I found an old topic where they mentioned 167,000 copies in Japan after some months, so similar to other yakuza games in Japan.
With Shenmue 3 we should be in the realm of 300k minimum (worldwide), so we aren't that far in case.
I don't think it's really a competition, both Shenmue and Yakuza are Sega games, each do a different thing.
They could be alternated to avoid fatigue in Yakuza franchise (one year Yakuza, one year Shenmue, third year a spin off and so on).
Also RGG has an already established "factory" that is able to make open world yakuza games in less than a year.
Of course with Shenmue you can't recycle too much, but all the cutscenes technology, mocap, engines used in RGG etc. are ready and adapting those to UE4 / UE5 for shenmue 4 and 5 would greatly reduce overall costs, compared to Ysnet working with another company.