Agreed.
If the story will make no sense without art notes, i.e. the opposite thing is happening on the page than the text states, etc., then include the notes. I've read a blog post somewhere where the writer states that she considers art notes more like "action notes." What agents and editors don't want to see are notes indicating a color of something if it's not essential to the story, how the character looks, etc.
Just be absolutely sure your notes are ESSENTIAL for the story to make sense. And be sparse with them. Agents, editors, and illustrators are professionals and they can probably understand the story more than you think. :-)