Withers was born in Karachi, then part of British India, to Edgar Withers, a captain in the Royal Navy, and a Dutch-German mother named Zitette. She acquired the name "Googie" (Little Pigeon) at a young age from her ayah (nanny). As a child, she learned Urdu. Her father left the Royal Navy to manage a foundry in Birmingham, England, and Googie was sent to a boarding school near Dover, Kent. She began acting at the age of twelve. A student at the Italia Conti Academy of Theatre Arts, she was a dancer in a West End production when she was offered work as a film extra in Michael Powell's The Girl in the Crowd (1935). She arrived on the set to find one of the major players in the production had been dismissed, and she was immediately asked to step into the role in her place.