Harry Phillips served as a member of the Tennessee State House of Representatives from 1935 to 1937, and then as an assistant state attorney general of Tennessee from 1937 to 1943. He was in the United States Navy during World War II, from 1943 to 1946, achieving the rank of Lieutenant Commander. He was again an assistant state attorney general of Tennessee from 1946 to 1950.
On June 4, 1963, Phillips was nominated by President John F. Kennedy to a seat on the United States Court of Appeals for the Sixth Circuit vacated by John Donelson Martin, Sr.. Phillips was confirmed by the United States Senate on June 28, 1963, and received his commission on July 3, 1963. He served as chief judge from 1969 to 1979, assuming senior status on January 15, 1979 and serving in that capacity until his death.