That's the way moving is for me too. My books-to-everything-else ratio is crazy enough that I only need to label the non-book boxes.
I don't label the book boxes "books" so much as "scifi/fantasy," "ya lit," "ya scifi/fantasy," "picture books," "middle grades," "history/politics/sociology," "travel," "art/architecture," and the like. It makes setting up my shelves so much easier.
And then there are books that are still in boxes because I don't have enough bookshelves.
Which lead to numbering boxes - rather than descriptive labeling - and then keeping track of the location of specific titles via excel spreadsheet.