Even though I don't actually have any bookcases in my house, I am always looking at them in awe. The ones that I have had in the past were always boring and I never knew how to put them together- they always stumped me. Over time, I have seen more and more pics, and paid more and more attention to other peoples and learned a few clues for myself (now I just need one of my own so I can put these clues to work!).
I also found this great post by Young House Love on tips for Styling a Bookcase
I think that one of the most important things to know when it comes to styling a bookcase is to not only limit the contents to books. To create a visually stimulating look, include a variety of items- candle holders, bowls, boxes, statues, figurines, pictures, frames etc.
Pepper Design Blog
Try grouping similar things together. Collections, books, or similar colors and shapes work well together.
Everything doesn't have to match exactly, or even relate to each other which is the glory of styling!
You also want to try to give the decorations height, and kind of layer them. For smaller objects find a box or some books to put in on and give it that extra height, as well as maybe a tall vase next to something shorter or a framed picture leaning against the back.
This bookshelf has only books on it, but I just love how they aren't all stacked in a traditional way. They are layered and placed in different ways which keeps the eye still interested.
I think texture is important as well. Pairing something tall and smooth with a short, rough object adds extra interest. Or maybe putting blankets next to some smooth books. Baskets are great for this purpose too- a nice place to stick random knick knacks and keep them out of the main view while giving the texture of the basket to look at.
Wallpaper, or a decorative backing gives a great look and also helps with not needing to fit something into every space. On the flip side, a solid color backing can give your bookcase extra flair! Just let the paint/paper do the talking!