Madrid isn't for me, and I couldn't hack the summers there (they can't either) but I really liked being there for a few months and it has a vibrant art scene. I find most Spanish cities that aren't super touristy to be very livable even if they aren't spectacular.
It's definitely not a glamorous city, but the cities in Spain I like more aren't glamorous either. Most modern construction in Spain is ugly, because Spain had no money, but most modern city building is ugly because of cars, outside of northern Europe. I prefer smaller cities in general.
Anyhow, I have a soft spot for Lavapiés. I went to a manu chao concert and went home with a girl who couldn't speak a word of English, but was filthy and enthusiastic. It was the first and only time anyone ever licked my arsehole.
I suppose that's the soft spot, my anus.
They should really get to putting a historic placard up there.