I wouldn't be too concerned about that. Other than Thanksgiving Day, I can't think of a much worse day and time to use as a barometer of a diner's success.
We passed them a few times last week and all was packed as usual. I think afternoons are their down times, when they get to have a little breather. We've been in for lunch and dinner and can't wait to go back. The bar is small but inviting and I want to just go sit there and have a few.