The only time the bags are on my bike is when on a trip. Otherwise they are stored in my motorcycle gear room. Sometimes I ride with the trunk, sometimes not. When my wife rides with me she likes it back there. Or if I'm running to the store to pick something up, it comes in handy. Otherwise I'd just as soon leave it off too, and often do. Stripped it sorta reminds you of an FJ 1200.
The bike feels lighter, and seems to handle better to me without that stuff back there, but admittedly I'm no road racer, just a rider, and if in my head it feels better, well it feels better, so I ride without it. Same goes for my suspension settings. Undoubtedly too soft for those who know better, but it feels good to me, so that's how I have it set, and going to leave it like that.
Also really like my shorty windshield. I like lots of air, especially when it's hot and humid like it is around here from June to October. When fall comes, or heading out on a trip, then the Yamaha touring shield goes back on. Switching from sport mode to tour mode takes less than 15 mins and that includes the walk upstairs to retrieve the saddlebags, and changing out the shield.
This bike really adapts well to the situation or wants of the owner, which is great. It can be a stripped sporty bike, or it can be a full blown tourer or something in between depending on the desire of the owner, changeable back and fort on a moment's notice. Set up your bike how it suits you, for the situation at hand, and enjoy the ride.
Out on a tour