One of the reasons I moved to Taiwan is due to their impeccable public transportation. As soon as I found out that Taiwan has a High Speed Rail (HSR), I booked my ticket right away. I decided to go to Kaohsiung, which is the second largest city in Taiwan. The fastest way to Kaohsiung is by the HSR, which is about 2 hours. I also took a bus to Kaohsiung recently, which is about 3-4 hours.
Price (Round Trip)
HSR = 2,900 NT ($100)
Bus = around 700 NT ($25)
Depending on your budget you can either choose from the quickest or slower way.
Personally, I like the bus option because it's cheaper and I don't mind sitting on a bus for 3-4 hours. The bus is also comfortable. My recent trip to Kaohsiung, I had my own seat without anyone sitting next to me and it also reclines. Tickets can be purchased online or at Taipei Main Station. I found that it's easier to purchase the bus tickets in person because the website is all in Chinese.
HSR and bus lines are located at Taipei Main Station (blue line)