I only saw one small black wire coming from the fire wall that connects to the same spot on the starter as the positive battery cable
That is not a ground wire. The other end of that wire should be connected to you Ammeter.
For proper grounds, confirm that:
The cable coming from the Negative battery terminal is securely grounded to the frame. You will see a hole on the front of the battery tray where it should be bolted too. Make sure that area is free of paint or grease. You want good bare metal contact.
Since your system is 6V make sure you are using heavy gauge battery cables. The smaller ones typically found on 12V systems will not work and may even melt (ask me how I know this).
You should also have a braided ground strap going from the engine plate to one of the motor mount bolts.
Additionally, you'll want to make sure that any of the gauges that require it are also grounded. Filing or sanding the paint off the insides of the dash mounting holes will usually do the trick.
I had a similar issue as you are having now and solved it by improving the grounding as stated above. I'll keep my fingers crossed that yours is as simple a fix.