Having just "downgraded" to an SV myself I feel I can give an honest response to this post also. I fully agree with the response concerning the bike and your "ego". Think about the damage you can do to someone else's ego??? I'm finding the underdog situation to be fun.
Seriously though I went from a 1000RR (wrecked), to a Honda F3 (B to A, couple wrecks), rode my fat pig VFR street bike in A a couple times and then bought my SV. I can say without a doubt that the SV has been the most fun even though they are all fun! I rode the SV on the track for the second time in A yesterday on the NJMP Lightening course, what a blast! Believe it or not there were many, many times where I was being held up in the tight sections. Now the straights, that's a different story!
The biggest thing I realized yesterday was I could ride that thing hard and never really feel like I was getting close to the edge and potentially crashing. The bike simply takes a beating, does what you want and puts a smile on your face. At this stage in my riding career, going home without pain is a major motivator. In my opinion, an SV is as close to "safe" as it gets.
FYI, I bought my '02 (naked) for $3000 with a bunch of riding gear which I sold. The bike had a penske rear shock, worked fronts, pazzos, woodcraft rearsets, sliders, full exhaust, fully wired....ready to go! Deals can be had on these bikes.
Go for it!