From what I observed since all these years, many fans never really appreciated the Shenmue combat system. It was an "ok" thing, somehow like the gunfights were in GTA III. But while Rockstar made a very clunky combat system, Sega AM2's one was rather esoteric. Very stiff feeling, an insane load of technical moves, a hidden skill three for a very few combats. It goes without saying that a lot of people actually struggled to master this part, and naturally never really enjoyed it while they were very fond of the rest of the game.
It's pretty much the core nature of Virtua Fighter that cannot be appreciated if you don't master your character. While they were including the VF2 system into Shenmue, Sega AM2 were aware of this and allowed button smashing to further help average players. This decision has brought some drawbacks like making the mastering pretty useless but the sake of the aesthetic. So even for the more experienced players, enjoy this part was not a given.
I definitely think we overrate the importance people attach to it, because if it was the case, the Shenmue fanbase wouldn't be as large as it is today.