My vote...
A little war torn is probably a good thing, but...
A flat chunk of steel bar stock...around 4" x 2" x 5/8" is my favorite dolly for hammering away at a Willys. You would be surprised at how smooth you can get that panel. Add in a little heat shrinking then a little more hammer/dolly work. Smooth with a light coat of filler...just to even out the surfaces hiding behind your hat channels, then call it good.
You're doing such a nice job on the repairs. This surface will be one of the most seen on your Willys. Smooth it to at least the level you will smooth out the sides. Otherwise, I think you might regret not cleaning it up a bit.
My humble opinion only.
Tim